I want to scrape a list of posts while the page scroll is infinite loading. I want scrollIntoView() for each element in the loop. my code is temporarily like this and the result when the page is loaded, it bounces out but no error.
for (let i = 0; i < 20; i++) {
const selector = 'div[role="feed"]:nth-child(2) div.sjgh65i0'
await page.evaluate((selector, i) => {
setTimeout(() => {
const element = document.querySelectorAll(selector)[i]
if(element) {
element.scrollIntoView();
}
}, 2000);
}, selector, i)
}
For example, I want to take 20 posts and then the scraper will take the posts one by one.
With your current flow, all timeouts are set almost at once and then all fire after the same 2 sec.
Try something like this:
const selector = 'div[role="feed"]:nth-child(2) div.sjgh65i0';
for (let i = 0; i < 20; i++) {
await page.waitForTimeout(2000);
await page.evaluate((selector, i) => {
const element = document.querySelectorAll(selector)[i];
if(element) {
element.scrollIntoView();
}
}, selector, i);
}