0
183
ViewsScraping data from a website that URL does not change when clicking on a particular onclick button
Website: http://www.busonlineticket.com/booking/singapore-to-shah-alam-bus-tickets
Basically I am trying to scrape data regarding the bus trips from this website but the data I scrape is dependent on the date selected by the user that my web-app is for.
Does anyone have any idea how I can write a program that can scrape data from the <li class='liDaysNew'> tags
Picture of the website and the DOM in question
My current code is as such:
page = requests.get(bus_URL, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')
buses = soup.find_all('tr', class_='bustr1')
bus_companies = []
depart_times = []
departure_locations = []
arrival_locations = []
prices = []
for bus in buses:
bus_company = bus.find('span', class_='buscompanyname').text
depart_time = bus.find('span', class_='bustime').text
departure_location_div = bus.find('div', class_='mbuspickup1')
departure_location = departure_location_div.find('span').text
arrival_location_div = bus.find('div', class_='mbusdropoff1')
arrival_location = arrival_location_div.find('span').text
price = bus.find('price', class_='mbusprice1').text
I know how to web scrape a normal website it is just the <li class='liDaysNew'> tags with the onclick logic that is throwing me off.
·
Santiago Gelvez
1 answers
Answer question0
import requests
from bs4 import BeautifulSoup
import pandas as pd
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36'}
data = {'deptdate': '2022-07-16', ## your departure date
'rtndate': '2022-07-18', ## your return date
'pax': '1',
'way': '1',
'type': 'bus',
'sbf': 'undefined'}
r = requests.post("https://www.busonlineticket.com/booking/singapore-to-shah-alam-bus-tickets", headers=headers, data=data)
soup = BeautifulSoup(r.text)
table_div = soup.select_one('#subtab1')
tables = pd.read_html(str(table_div))
df = tables[0]
print(df)
This will return a pandas dataframe:
Of course, you can use BeautifulSoup to extract data in a different way than a dataframe, like separate table elements etc.
·
Santiago Gelvez
Report
Answer question