So i tried to watch a tutorial and make a web scraper, i combined like 5 tutorial worth code into my code so i suppose it's messy.
This is the code:
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
const fs = require('fs');
const { html } = require('cheerio');
const cheerio = require('cheerio');
const { setInterval } = require('timers/promises');
require('dotenv').config();
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
const client = require('twilio')(accountSid,authToken);
puppeteer.use(StealthPlugin());
const oldData = "oldData";
async function scrape(){
const browser = await puppeteer.launch({
headless: true,
defaultViewport:{
width: 1920,
height: 1080
}
});
const page = await browser.newPage();
await page.goto('page');
const pageData = await page.evaluate(() => {
return{
width: document.documentElement.clientWidth,
height: document.documentElement.clientHeight,
html: document.documentElement.innerHTML,
};
})
const $ = cheerio.load(pageData.html);
const element = $(".accordion__header-inner:last");
console.log(element.text());
};
const handle = setInterval(scrape(), 4000);
scrape();
I hide some parts that i didn't want to be seen.
So, when i run the code function it logged the element thing twice. When i delete the interval thing it works only one. That is the only difference, i want to log the element thing in every 5 seconds.
Any help is appreciated and i hope this isn't a poorly written question.
In your setInterval you are executing the function and passing it's return value as argument, instead of passing function itself as argument.
It should be:
const handle = setInterval(scrape, 4000);
Note, you won't be use this inside the function, because it will be called inside setInterval scope.
If you need to use this, then use bind():
const handle = setInterval(scrape.bind(scrape), 4000);
You seem to be calling the "scrape()" method in two places. Once in the SetInterval() method and then immediately again on the line after. That looks like your problem.