Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

276
Views
Is there a node.js package for web scraping that will capture content outside of page source-code?

Some sites don't have everything in the page source code, you can only see the new elements when you load the page on a browser. I don't know why is that, I'm thinking maybe it's because a site requires javascript to be enabled? In python, I fixed my issue with the "AsyncHTMLSession" module and this code:

r = await session.get(url)
await r.html.arender()

It let me grab the new elements and do whatever I needed, is there anything like this for node.js?

about 4 years ago · Juan Pablo Isaza
1 answers
Answer question

0

Yes, basically you are looking for a way to access content that shows up when a page loads using JavaScript.

The solution you want is a headless browser - puppeteer does this very well.

Checkout: https://www.toptal.com/puppeteer/headless-browser-puppeteer-tutorial

Or even a quick video: https://www.youtube.com/watch?v=dXjKh66BR2U

about 4 years ago · Juan Pablo Isaza Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!