to begin, i wanted to scrape this table that load from java script and i only want to scrape specific row of data such as base on Fund abbreviation or search base on that, but my basic script here is not working well. please help
from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()
url = 'https://www.publicmutual.com.my/Our-Products/UT-Fund-Prices'
r = session.get(url)
r.html.render()
soup=BeautifulSoup(r.html.html,'html.parser')
table = soup.find('table', attrs={'class':'fundtable col-sm-12'})
print(table)
how do i make this work and scrape information only certain rows that i wanted based on the fund name or fund abbreviation column?
If you go in chromedevtools and check for the url that render the table you will see the html of the table as follow: HTML
The website perform a post request to get the data
If you need that your scraping application been fast you have to map the payload and scrapy with python-requests and bs4. If speed is not a problem you can use python-Selenium
