Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

88
Views
Restart while loop when selector for next page is clicked

sorry but I am very new to coding and this is my first project, in which i chose to create a simple scraper for eBay using puppeteer. The program takes in the item to be searched and the desired amount of pages to scraped as a command line argument. However, due to the selector children li:nth-child increasing each time, i had to create an incremental while loop to through said n-th children from 2 to 60. This works for the first page, and outputs all items desired. However, due to having to scrape 2+ pages, the i value used in the loop must reset at the start of the next page to restart said loop. I believe the pagination is correct as in headless mode, the second, third, etc. pages are clicked through.

current loop

let currentPage = 1;

while (currentPage <= pagestoScrape) {
    let content = await page.evaluate(() => {
        let data = [];
        let childnumber = 2;
        let i = 2;

        while (i < 60) {
            let itemids = document.querySelectorAll(`#srp-river-results > ul > li:nth-child(${childnumber}) > div`);
            itemids.forEach((itemid) => {
                let PRODUCT_ID = `#srp-river-results > ul > li:nth-child(${childnumber}) > div > div.s-item__info.clearfix > a > h3`;
                let PRODUCT_PRICE = `#srp-river-results > ul > li:nth-child(${childnumber}) > div > div.s-item__info.clearfix > div.s-item__details.clearfix > div:nth-child(1) > span`;
                let item = itemid.querySelector(PRODUCT_ID).innerText;
                let price = itemid.querySelector(PRODUCT_PRICE).innerText;
                data.push({
                    item,
                    price,
                });
            });
            childnumber++;
            i++;
        }
        return data;
    });

    let urls = [];
    urls = urls.concat(content);
    if (currentPage < pagestoScrape) {
        await Promise.all([
            await page.click('#srp-river-results > ul > div.srp-river-answer.srp-river-answer--BASIC_PAGINATION_V2 > div.s-pagination > span > span > nav > a.pagination__next.icon-link'),
            await page.waitForSelector('#srp-river-results > ul > div.srp-river-answer.srp-river-answer--BASIC_PAGINATION_V2 > div.s-pagination > span > span > nav > a.pagination__next.icon-link'),
        ])
    }

    currentPage++;
    let jsonData = JSON.stringify(content, null, 1);
    fs.writeFileSync(`./output/output${item}.txt`, jsonData);
}

I have tried reavaluating i after the loop, during the promise and changing the loop to a for. I expected this to reset the loop and for the output to contain the results of each page. However, it results in the output of just 1 page, with no error.

about 4 years ago · Juan Pablo Isaza
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!