The website I'm looking to scrape has a button which asks for dates of the data you want to download as a .csv file. I have automated this process to download this data via python but need to manually get a token each time from the browser under ctrl,shift,i->cookies->request cookies and then copy and paste the token into the python file each time. The problem is the token times out after 1 day and modifying the expiry and/or issue date does not work so I assume this token is being generated by client side javascript.
I have tried using:
response = requests.get(www.exampleurl.com)
print(response.cookies)
And I get an empty response.
I've also tried:
print(response.headers)
But I do not get the cookie or token data at all.
I want to know where I may find the client side javascript which is generating the token so I can copy the algorithm or atleast run this javascript algorithm each time I need a token.
I'm trying to avoid selenium due to it having a 1GB overhead in comparison to the script which would be 10MB.