I am trying to scrape this web https://poe.ninja/challenge/builds?time-machine=day-6 using Puppeteer. I tried Puppeteer page.evaluate querySelectorAll return empty objects and saw lot of similar question here. but none of them solve my problem.
Here is my code:
const scrapeNinja = async () => {
const browser = await puppeteer.launch({headless: false})
const page = await browser.newPage()
await page.goto(`https://poe.ninja/challenge/builds?time-machine=day-6`, {
waitUntil: 'domcontentloaded',
})
const getArray = await page.evaluate(() => {
return Array.from(document.querySelectorAll(
'#openSidebar > div > section:nth-child(3) > div > div > div > ul li .css-1h2ruwl'
)).map(e => e.textContent)
})
console.log(getArray)
}
I know the values returned from page.evaluate should be serializeable. isn't this Array.from(document.querySelectorAll('#openSidebar > div > section:nth-child(3) > div > div > div > ul li .css-1h2ruwl')).map(e => e.textContent) not a serializeable value? I tried use this on the dev tool section it return exacully what i want, but back to node.js, it only return empty array...
Am i do something wrong?
Looks like the problem is really with waiting, you are looking for elements even if full dom content isnt fully loaded.
const scrapeNinja = async () => {
const browser = await puppeteer.launch({headless: false})
const page = await browser.newPage()
await page.goto(`https://poe.ninja/challenge/builds?time-machine=day-6`, {
waitUntil: 'networkidle2',
})
const getArray = await page.$$eval('#openSidebar > div > section:nth-child(3) > div > div > div > ul li .css-1h2ruwl',
el => el.map(item => item.textContent))
console.log(getArray)
}
scrapeNinja()
This code works perfectly for me, even you dont have to initialize array. In the future use networkidle2 in waitUntil option