I am seeking help running requests-html in Python with ExpressVPN. I am trying to scrape page title data from a site that renders content in JavaScript. I am able to load the webpages in my browser with my VPN on but I am unable to get page title data when I try to run the code below in Python with the VPN on. I have searched other threads but have been unable to solve this issue so far, any advice would be very much appreciated!
I have tried copying the header information as suggested on this post, but defining the header data as found on the network dev tools tab hasn't solved my issue. Python Requests Does Not get website that opens on browser
from requests_html import HTMLSession
from bs4 import BeautifulSoup
session = HTMLSession(browser_args=["--no-sandbox", "--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36"])
url = "https://www.url.com"
resp = session.get(url)
resp.html.render(sleep=10)
resp.html.html
soup = BeautifulSoup(resp.html.html, "lxml")
option_tags = soup.find_all("title")
print(option_tags)