Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

175
Views
Scrape dynmically rendered by scroll objects

I would like to scrape linkedin's job section, for example the following link:

https://www.linkedin.com/jobs/search/?f_TPR=r86400&geoId=105080838&keywords=Full%20Stack&location=New%20York%2C%20United%20States

As you can see, as soon as you scroll to to bottom the page loads more jobs.

The effect I am looking for is, I want somehow to get an answer from the site after it scrolled all the way to the bottom, containing all the possible jobs, for parsing I will use cheerio.

What I managed to do using js and nodejs is to get only the first page, but not more.

about 4 years ago · Santiago Gelvez
1 answers
Answer question

0

the first step would be to get the number of full-stack jobs returned from your search:

number of job results

now that we know the attributes of the element, we can grab him using cheerio,

let number_of_jobs = $("h1>small[class='jobs-search-results-list__text']").text()

now, as you mentioned,

The way that LinkedIn job postings work is loading more jobs if you scroll down the browser bar

but, when you drag the bar a few times, it will not load automatically, whereas you would have to click a button that says 'See more jobs'.

So we will take advantage of that and use a try/except method. The number of jobs returned from our query will determine the number of times we will drag the handler using selenium (we cant use cheerio here since it's an HTML parser). You can also use puppeteer phantomJS.

let i = 2
while (i <= int(no_of_jobs/25)+1): 
    wd.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
     i = i + 1
     try:
     wd.find_element_by_xpath
(‘/html/body/main/div/section/button’).click()
     time.sleep(5)
     except:
         pass
         time.sleep(5)

To get the jobs list, we would use the class name - jobs-search-results__list, which is exclusive to the jobs list, and we'll loop through its li elements for each job.

jobs list

again, using selenium:

let jobs_list = wd.find_element_by_class_name(‘jobs-search__results-list’)
let jobs = jobs_list.find_elements_by_tag_name(‘li’)

You can check the length of jobs and the variable number_of_jobs we set at the beginning to ensure you got everything.

And that's all.


You can use any chrome controller API with this logic, I would recommend puppeteer, but here I used selenium since you used it as a tag.

about 4 years ago · Santiago Gelvez Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!