I am following this tutorial. https://www.digitalocean.com/community/tutorials/how-to-scrape-a-website-using-node-js-and-puppeteer
I am trying to learn how to use puppeteer.
I am trying to get whats written in the anchor tag .
So my code in pageScraper file is this
const scraperObject = {
url: 'http://books.toscrape.com',
async scraper(browser){
let page = await browser.newPage();
console.log(`Navigating to ${this.url}...`);
// Navigate to the selected page
await page.goto(this.url);
// Wait for the required DOM to be rendered
await page.waitForSelector('.page_inner');
let mytext=await page.$$eval('.nav-list li > a',text => text.textContent)
console.log(mytext);
}
}// dataObj['bookTitle'] = await newPage.$eval('.product_main > h1', text => text.textContent);
module.exports = scraperObject;
However I am getting an undefined while printing.
I am trying to get this. Selectors
First of all, http://books.toscrape.com doesn't have any functional JS, so there's no need for Puppeteer. Better to just use a lightweight static request library like fetch or axios and an HTML parser like Cheerio.
That said, for pedagogical purposes,
await page.$$eval('.nav-list li > a',text => text.textContent)
appears to be the problem. $$eval means "select all and run a callback on all selected elements in the browser context", similar to document.querySelectorAll. So the variable you call text is an array of elements, not a single element, and the array property .textContent is undefined.
If you want to select a single element, use $eval. Please call the variable element or el, not text.
On the other hand, if you want to select multiple elements, treat it like an array in the callback.
Getting one element's text:
await page.$eval('.nav-list li > a', el => el.textContent);
Getting multiple elements' text:
await page.$$eval('.nav-list li > a', els => els.map(el => el.textContent));