I would like to scrape linkedin's job section, for example the following link:
As you can see, as soon as you scroll to to bottom the page loads more jobs.
The effect I am looking for is, I want somehow to get an answer from the site after it scrolled all the way to the bottom, containing all the possible jobs, for parsing I will use cheerio.
What I managed to do using js and nodejs is to get only the first page, but not more.
the first step would be to get the number of full-stack jobs returned from your search:
now that we know the attributes of the element, we can grab him using cheerio,
let number_of_jobs = $("h1>small[class='jobs-search-results-list__text']").text()
now, as you mentioned,
The way that LinkedIn job postings work is loading more jobs if you scroll down the browser bar
but, when you drag the bar a few times, it will not load automatically, whereas you would have to click a button that says 'See more jobs'.
So we will take advantage of that and use a try/except method. The number of jobs returned from our query will determine the number of times we will drag the handler using selenium (we cant use cheerio here since it's an HTML parser). You can also use puppeteer phantomJS.
let i = 2
while (i <= int(no_of_jobs/25)+1):
wd.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
i = i + 1
try:
wd.find_element_by_xpath
(‘/html/body/main/div/section/button’).click()
time.sleep(5)
except:
pass
time.sleep(5)
To get the jobs list, we would use the class name - jobs-search-results__list, which is exclusive to the jobs list, and we'll loop through its li elements for each job.
again, using selenium:
let jobs_list = wd.find_element_by_class_name(‘jobs-search__results-list’)
let jobs = jobs_list.find_elements_by_tag_name(‘li’)
You can check the length of jobs and the variable number_of_jobs we set at the beginning to ensure you got everything.
And that's all.
You can use any chrome controller API with this logic, I would recommend puppeteer, but here I used selenium since you used it as a tag.