I'm trying to scrape all of price data from this site https://www.bynogame.com/tr/oyunlar/knight-online/gold-bar using the puppeteer.
I can scrap prices one by one, but I can't get all p elements, null data is returned. Here is my code that works to scrap one by one, and the code below to scrap in all datas which does not work. Where I am doing wrong?
const puppeteer = require("puppeteer");
const gb = async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.bynogame.com/tr/oyunlar/knight-online/gold-bar');
const data = await page.$eval('body > div.container.mb-5 > div > div:nth-child(1) > div.col-md-18.order-1.order-sm-12 > div > div:nth-child(1) > div > div > div.col-md-21 > div > div.col-md-4 > div > div > div > p', el => el.textContent); //Output is true
await browser.close();
console.log(data)
};
gb();
//Here is not work.
const data = await page.$$eval('.col-md-24 mb-2 itemDiv .itemCard .row d-flex align-items-center .col-md-21 .row d-flex align-items-center .col-md-4 .row d-flex flex-column .col .div p', obj => obj.map(p => p.textContent));
There was no element with .col-md-24 mb-2 itemDiv .itemCard .row d-flex align-items-center .col-md-21 .row d-flex align-items-center .col-md-4 .row d-flex flex-column .col .div p selector in https://www.bynogame.com/tr/oyunlar/knight-online/gold-bar page.
You can try it be going to the page, then open up the console and run document.querySelectorAll.
For example:
document.querySelectorAll('.col-md-24 mb-2 itemDiv .itemCard .row d-flex align-items-center .col-md-21 .row d-flex align-items-center .col-md-4 .row d-flex flex-column .col .div p')
It shows me an empty array, which mean none HTML element matched the given selector.
You do not required such long selector. When you at the page, you can see that the price was inside <p> tag, with classes of font-weight-bolder text-black m-0. Firstly, check whether any other element sharing the same selector as the price by querying for <p> tag, with classes of font-weight-bolder text-black m-0 in the console.
document.querySelectorAll('p.font-weight-bolder.text-black.m-0')
--- Output ---
NodeList(9) [ p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0, p.font-weight-bolder.text-black.m-0 ]
By checking the output, I found that the returned elements was the price I was looking for. Therefore, I can substitute the correct selector into the page.$$eval. The final code will be
const puppeteer = require("puppeteer");
const gb = async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto("https://www.bynogame.com/tr/oyunlar/knight-online/gold-bar");
// Get element with <p class="font-weight-bolder text-black m-0">
const data = await page.$$eval("p.font-weight-bolder.text-black.m-0", (obj) =>
obj.map((p) => p.textContent)
);
await browser.close();
console.log(data);
};
gb();