Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

180
Visualizações
Scrape dynmically rendered by scroll objects

I would like to scrape linkedin's job section, for example the following link:

https://www.linkedin.com/jobs/search/?f_TPR=r86400&geoId=105080838&keywords=Full%20Stack&location=New%20York%2C%20United%20States

As you can see, as soon as you scroll to to bottom the page loads more jobs.

The effect I am looking for is, I want somehow to get an answer from the site after it scrolled all the way to the bottom, containing all the possible jobs, for parsing I will use cheerio.

What I managed to do using js and nodejs is to get only the first page, but not more.

about 4 years ago · Santiago Gelvez
1 Respostas
Responde à pergunta

0

the first step would be to get the number of full-stack jobs returned from your search:

number of job results

now that we know the attributes of the element, we can grab him using cheerio,

let number_of_jobs = $("h1>small[class='jobs-search-results-list__text']").text()

now, as you mentioned,

The way that LinkedIn job postings work is loading more jobs if you scroll down the browser bar

but, when you drag the bar a few times, it will not load automatically, whereas you would have to click a button that says 'See more jobs'.

So we will take advantage of that and use a try/except method. The number of jobs returned from our query will determine the number of times we will drag the handler using selenium (we cant use cheerio here since it's an HTML parser). You can also use puppeteer phantomJS.

let i = 2
while (i <= int(no_of_jobs/25)+1): 
    wd.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
     i = i + 1
     try:
     wd.find_element_by_xpath
(‘/html/body/main/div/section/button’).click()
     time.sleep(5)
     except:
         pass
         time.sleep(5)

To get the jobs list, we would use the class name - jobs-search-results__list, which is exclusive to the jobs list, and we'll loop through its li elements for each job.

jobs list

again, using selenium:

let jobs_list = wd.find_element_by_class_name(‘jobs-search__results-list’)
let jobs = jobs_list.find_elements_by_tag_name(‘li’)

You can check the length of jobs and the variable number_of_jobs we set at the beginning to ensure you got everything.

And that's all.


You can use any chrome controller API with this logic, I would recommend puppeteer, but here I used selenium since you used it as a tag.

about 4 years ago · Santiago Gelvez Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda