I am trying to scrape a list of tech Companies in Greater Noida from 'GlassDoor':
but Problem: Page changes its content after some seconds of load (may be using javascript).
you can change value of page in link.. give a try for that. after changing page no. i am rendering same first page content..after changing page no.!
from bs4 import BeautifulSoup
from selenium import webdriver
lst=[]
options = webdriver.ChromeOptions()
options.add_argument('--headless')
browser = webdriver.Chrome(options=options, executable_path='chromedriver.exe')
browser.get("https://www.glassdoor.co.in/Explore/browse-companies.htm?overall_rating_low=3.5&page=8&isHiringSurge=0&locId=4475367&locType=C&locName=Greater%20Noida")
html = browser.page_source
soup = BeautifulSoup(html, features="html.parser")
company_lst = soup.find_all('section', {'class': 'employerCard__EmployerCardStyles__employerCard'})
for item in company_lst:
d={}
d['Name'] = item.find('h2', {'data-test': 'employer-short-name'}).text
d['Rating'] = item.find('span', {'data-test': 'rating'}).text
d['Reviews-Count'] = item.find('div', {'data-test': 'cell-Reviews-count'}).text
d['Salaries-Count'] = item.find('div', {'data-test': 'cell-Salaries-count'}).text
d['Jobs-Count'] = item.find('div', {'data-test': 'cell-Jobs-count'}).text
d['Industry'] = item.find('span', {'data-test': 'employer-industry'}).text
print(d['Name'])
Not sure why the page reloads to the First page. But you can try to click on the page number buttons like below(which works fine).
browser.get("https://www.glassdoor.co.in/Explore/browse-companies.htm?overall_rating_low=3.5&page=1&isHiringSurge=0&locId=4475367&locType=C&locName=Greater%20Noida")
for i in range(2,8):
browser.find_element_by_xpath("//ul[contains(@class,'pagination')]//li/button[text()='{}']".format(i)).click()
print(browser.current_url) # Prints the URL from page 2 to 7.
time.sleep(5)