0
542
ViewsCannot read properties of null (reading 'textContent') in Puppeteer evaluate
I am trying to use Puppeteer to scrape this element from eBay:
However, when I run my code, I get an error that says "Cannot read properties of null (reading 'textContent')". This is my code:
async function scrape() {
const browser = await puppeteer.launch({
headless: false
});
const page = await browser.newPage();
await page.goto('https://www.ebay.com/sch/i.html?_from=R40&_nkw=Blastoise+%282%2F102%29+%5BCelebrations%3A+Classic+Collection%5D&_sacat=0&Graded=No&_dcat=183454&rt=nc&LH_Sold=1&LH_Complete=1');
await page.waitForSelector('.s-item');
let cards = await page.evaluate(() => {
let cardElement = document.body.querySelectorAll('.s-item')
let cards = Object.values(cardElement).map(x => {
return {
date: x.querySelector('.s-item__title--tagblock span.POSITIVE').textContent ? ? null
}
})
return cards
})
console.log(cards)
})()
How can I solve this?
1 answers
Answer question0
The first item in the result list is sort of a dummy/template/placeholder node that has no data in it. You can use .slice(1) or .s-item:not(:first-child) to skip that first node.
While you're at it, you might as well make your waitForSelector more precise to match the element you're about to select (although this line isn't necessary, as I'll explain below), use domcontentloaded to speed up goto and use ?.textContent so you can see undefined rather than a crash, which is how I debugged the problem. .textContent ?? null doesn't work because the throw happens before ?? null has the chance to evaluate.
const puppeteer = require("puppeteer"); // ^14.1.1
let browser;
(async () => {
browser = await puppeteer.launch({headless: true});
const [page] = await browser.pages();
const url = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=Blastoise+%282%2F102%29+%5BCelebrations%3A+Classic+Collection%5D&_sacat=0&Graded=No&_dcat=183454&rt=nc&LH_Sold=1&LH_Complete=1";
await page.goto(url, {waitUntil: "domcontentloaded"});
await page.waitForSelector(".s-item .s-item__title--tagblock span.POSITIVE");
const cards = await page.evaluate(() =>
[...document.querySelectorAll(".s-item:not(:first-child)")].map(x => ({
date: x
.querySelector(".s-item__title--tagblock span.POSITIVE")
?.textContent
}))
);
console.log(cards);
})()
.catch(err => console.error(err))
.finally(() => browser?.close())
;
Better yet, skip Puppeteer entirely and make an HTTP request with Axios or fetch, then use Cheerio to parse the static HTML, which already has the data you need:
const axios = require("axios");
const cheerio = require("cheerio");
const url = "https://www.ebay.com/sch/i.html?_from=R40&_nkw=Blastoise+%282%2F102%29+%5BCelebrations%3A+Classic+Collection%5D&_sacat=0&Graded=No&_dcat=183454&rt=nc&LH_Sold=1&LH_Complete=1";
axios.get(url).then(({data}) => {
const $ = cheerio.load(data);
const cards = [];
$(".s-item:not(:first-child)").each(function (i, e) {
$(this).find(".s-item__title--tagblock .POSITIVE").each(function (i, e) {
cards.push($(this).text().trim());
});
});
console.log(cards);
})
.catch(err => console.error(err))
;