I'm trying to scrape (this) product page, specifically the modal that shows up when you click "View all bids".
The html structure is just a simple table, I'm trying to get every "Size" element. The problem is that whenever I run my code, it opens up the modal but only returns a few random shoe sizes that are not in order.
Example:
shoeSizeBids: [
'14', '11.5', '10.5',
'11', '8.5', '11',
'9', '9', '7',
'13'
]
My code:
const bidsChartSel =
'#market-summary > div.ask.ask-button-b > div.sale-size > div:nth-child(2)';
await Promise.all([page.click(bidsChartSel)]);
// Get all the shoe size bids
const shoeSizeBids= await page.evaluate(() =>
Array.from(
document.querySelectorAll('tbody > tr > td:nth-child(1)'),
(element) => element.textContent
)
);
The sorting order comes from that page, i.e. the sizes are rendered in that order. To get them sorted properly, you'd need to:
That can be achieved with the following:
const uniqueSortedSizes = Array.from(new Set(shoeSizeBids))
.map(s => parseFloat(s, 10))
.sort((a, b) => a > b ? 1: a < b ? -1 : 0);
You are matching multiple HTML tables with the current selector (tbody > tr > td:nth-child(1)). the one inside the modal uses:
.activity-table > tbody > tr > td:nth-child(1)
You can also use page.$$eval as a puppeteer shorthand for Array.from(document.querySelectorAll(selector)):
const shoeSizeBids = await page.$$eval('.activity-table > tbody > tr > td:nth-child(1)', elems => elems.map(el => el.innerText))