I'm total noob when it comes to javascript, html, web-scrapping and web data mining.
I'm trying to perform the following via Python script:


So far I have the following code:
from bs4 import BeautifulSoup
from requests_html import HTMLSession
def AccessScienceDirect(credentials):
# this url gives me search results page with links to articles
_url = f"https://www.sciencedirect.com/search?authors={credentials}"
session = HTMLSession()
try:
response = session.get(_url)
response.html.render()
except Exception as e:
return (f'Error for {_url}', e)
soup = BeautifulSoup(response.html.html, 'html.parser' )
_text = soup.get_text()
return _text
AccessScienceDirect("Albert Einstein")
Unfortunately I don't know what I should do next.
The _text doesn't seems to hold any information that resembles web page I can access by inserting link into browser:
"https://www.sciencedirect.com/search?authors=Albert Einstein"
Not mentioning accessing any article links. I assume this is because this web page uses javascript.
Offcourse I don't need to perform actions 1-6 in sequence. If it would require just one url request to get that text out of the 'pop-up' window I'd be more than happy.
So i'm stuck on step: 3 :-(
Thank you in advance for any help/advice.