Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

177
Vistas
Scrape dynmically rendered by scroll objects

I would like to scrape linkedin's job section, for example the following link:

https://www.linkedin.com/jobs/search/?f_TPR=r86400&geoId=105080838&keywords=Full%20Stack&location=New%20York%2C%20United%20States

As you can see, as soon as you scroll to to bottom the page loads more jobs.

The effect I am looking for is, I want somehow to get an answer from the site after it scrolled all the way to the bottom, containing all the possible jobs, for parsing I will use cheerio.

What I managed to do using js and nodejs is to get only the first page, but not more.

about 4 years ago · Santiago Gelvez
1 Respuestas
Responde la pregunta

0

the first step would be to get the number of full-stack jobs returned from your search:

number of job results

now that we know the attributes of the element, we can grab him using cheerio,

let number_of_jobs = $("h1>small[class='jobs-search-results-list__text']").text()

now, as you mentioned,

The way that LinkedIn job postings work is loading more jobs if you scroll down the browser bar

but, when you drag the bar a few times, it will not load automatically, whereas you would have to click a button that says 'See more jobs'.

So we will take advantage of that and use a try/except method. The number of jobs returned from our query will determine the number of times we will drag the handler using selenium (we cant use cheerio here since it's an HTML parser). You can also use puppeteer phantomJS.

let i = 2
while (i <= int(no_of_jobs/25)+1): 
    wd.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
     i = i + 1
     try:
     wd.find_element_by_xpath
(‘/html/body/main/div/section/button’).click()
     time.sleep(5)
     except:
         pass
         time.sleep(5)

To get the jobs list, we would use the class name - jobs-search-results__list, which is exclusive to the jobs list, and we'll loop through its li elements for each job.

jobs list

again, using selenium:

let jobs_list = wd.find_element_by_class_name(‘jobs-search__results-list’)
let jobs = jobs_list.find_elements_by_tag_name(‘li’)

You can check the length of jobs and the variable number_of_jobs we set at the beginning to ensure you got everything.

And that's all.


You can use any chrome controller API with this logic, I would recommend puppeteer, but here I used selenium since you used it as a tag.

about 4 years ago · Santiago Gelvez Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda