I am trying to scrape the data from this page. The page has infinite scroll within an element in the DOM. I have the following puppeteer script to scroll all the way to the bottom to reveal all data so that I can scrape it. The first page.evaluate call is working as expected, scrolling to the bottom and revealing the further contents. However, subsequent calls are not working. I have tried using await page.waitFor(2000); to wait before I can scroll further but it does not seem to work. Kindly help someone!
// this script parses the data from http://covid.rcmedicrew.org/ using their /scripts/getSearch.php API
// install nodejs then npm install axios
const puppeteer = require('puppeteer');
const siteURL = "https://www.powerbi.com/view?r=eyJrIjoiOTcyM2JkNTQtYzA5ZS00MWI4LWIxN2UtZjY1NjFhYmFjZDBjIiwidCI6ImQ1ZmE3M2I0LTE1MzgtNGRjZi1hZGIwLTA3NGEzNzg4MmRkNiJ9";
async function callPowerBiApi(apiUrl) {
const browser = await puppeteer.launch({headless: false, args: ['--auto-open-devtools-for-tabs']});
// const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
page.setDefaultNavigationTimeout(0);
await page.goto(apiUrl, {waitUntil: 'networkidle0', timeout: 0});
await page.waitForSelector('.innerContainer');
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1].querySelector(".bodyCells>div>div div:last-child").scrollIntoView());
//await page.waitFor(2000);
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1].querySelector(".bodyCells>div>div div:last-child").scrollIntoView());
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1].querySelector(".bodyCells>div>div div:last-child").scrollIntoView());
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1].querySelector(".bodyCells>div>div div:last-child").scrollIntoView());
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1].querySelector(".bodyCells>div>div div:last-child").scrollIntoView());
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1].querySelector(".bodyCells>div>div div:last-child").scrollIntoView());
}()
I was scrolling to the wrong element, fixed it by using:
await page.evaluate(() => document.querySelectorAll(".innerContainer")[1]
.querySelector(".bodyCells>div div:last-of-type div:last-of-type div:last-of-type")
.scrollIntoView()
);
However, it seems that the previous results are getting cleared when you scroll down so will need another strategy to scrape it all.