I'm trying to scrape a product page using pupeteer and Cheerio. (this page)
I'm using a data id to scrape the title and image. The problem is that the title never gets scraped while the image does every time.
I've tried scraping the title by class name but that doesn't work either. Does this have something to do with the specfic website I'm trying to scrape? Thank you.
my code:
// Load cheerio
const $ = cheerio.load(data);
/* Scrape Product Page */
const product = [];
// Title
$('[data-testid="product-name"]').each(() => {
product.push({
title: $(this).text(),
});
});
// Image
$('[data-testid="product-detail-image"]').each((index, value) => {
const imgSrc = $(value).attr('src');
product.push({
image: imgSrc,
});
});
As I mentioned in the comments, there's almost no use case I can think of that makes sense with both Puppeteer and Cheerio at once. If the data is static, use Cheerio alongside a simple request library like Axios, otherwise use Puppeteer and skip Cheerio entirely in favor of native Puppeteer selectors.
One other potential reason to use Puppeteer is if your requests library is being blocked by the server's robot detector, as appears to be the case here.
This script worked for me:
const puppeteer = require("puppeteer");
let browser;
(async () => {
browser = await puppeteer.launch({headless: true});
const [page] = await browser.pages();
const url = "https://stockx.com/nike-air-force-1-low-white-07";
await page.goto(url);
const nameSel = '[data-testid="product-name"]';
await page.waitForSelector(nameSel, {timeout: 60000});
const name = await page.$eval(nameSel, el => el.textContent);
const imgSel = '[data-testid="product-detail-image"]';
await page.waitForSelector(imgSel);
const src = await page.$eval(imgSel, el => el.src);
console.log(name, src);
})()
.catch(err => console.error(err))
.finally(() => browser?.close())
;