I am trying to use Puppeteer to scrape data from https://pagespeed.web.dev - I need to be able to take screenshots of the results and while it would be much simpler to use Google's own API to get the raw data, I can't screenshot the actual results that way. The challenge I'm having is filtering out DOM elements while still retaining the ElementHandle nature of the objects. For example, getting the "This URL" / "Origin" buttons:
In a normal JS console, I would run this:
[...document.querySelectorAll('button')].filter(b => b.innerText === 'This URL')
This would give me an Array of DOM elements that I could then run click() on or whatever.
I have tried a number of ways to get Puppeteer to give me a usable ElementHandle object and they have all returned an array of objects, with the sole member of that object being an __incrementalDOMData object:
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto(`https://pagespeed.web.dev/report?url=${url}`)
await page.waitForSelector(homepageWaitSelector, { visible: true })
// here's where the fun starts
const buttons = await page.$$eval('button', buttons => buttons.filter(button => button.innerText === 'This URL'))
const buttons = await page.$$eval('button', buttons => buttons.map(b => b.innerText).filter(t => t === 'This URL'))
// This one seems to run because the list of elements returned has the right length, but I can never get a breakpoint to catch inside the `evaluate` method, nor does a `console.log` statement actually print.
const buttons = await page.evaluate(() => {
const b = [...document.querySelectorAll('button')].filter(b => b.innerText === 'This URL')
return b
})
All of those methods end up returning something like this:
[{
__incrementalDOMData: {
j: ['class', 'the-classes-used'],
key: 'a key',
v: true
}]
Because there are so many buttons, and because the class names are all random and reused, I can't just target the ones I want (I mean I suppose I could build a super precise selector), filter them, and then return not just the filtered data but the actual elements themselves. Is what I'm asking for even possible?
Thanks to ggorlen's comment above I simplified my approach and got something working without an overly complex selector or logic:
const buttonContainer = await page.$('div.R3HzDf.ucyZQe')
const thisUrlButton = await buttonContainer.$(':first-child')
await thisUrlButton.evaluate(b => b.click())
await page.screenshot({ path: 'homepage.png' })
const originButton = await buttonContainer.$(':last-child')
await originButton.evaluate(b => b.click())
await page.screenshot({ path: 'origin.png' })