Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

323
Views
Get text from window popup on ScienceDirect web page using Python requests-html

I'm total noob when it comes to javascript, html, web-scrapping and web data mining.

I'm trying to perform the following via Python script:

  1. Go to: https://www.sciencedirect.com/
  2. Search by author name and last name i.e: "Albert Einstein".
  3. Get links to the articles that are in search results page. enter image description here
  4. Access those links.
  5. On the article page, I need to access link with author credentials just below article title, when clicked, a window will appear from the right. enter image description here
  6. I want all the text from that window and I need it via Python script. Preferable by using some form of url requests.

So far I have the following code:

from bs4 import BeautifulSoup
from requests_html import HTMLSession

def AccessScienceDirect(credentials):

    # this url gives me search results page with links to articles
    _url = f"https://www.sciencedirect.com/search?authors={credentials}"
    session = HTMLSession()
    
    try:
        response = session.get(_url)
        response.html.render()
    except Exception as e:
        return (f'Error for {_url}', e)

    soup = BeautifulSoup(response.html.html, 'html.parser' )
    _text = soup.get_text()
    return _text

AccessScienceDirect("Albert Einstein")

Unfortunately I don't know what I should do next.

The _text doesn't seems to hold any information that resembles web page I can access by inserting link into browser: "https://www.sciencedirect.com/search?authors=Albert Einstein"

Not mentioning accessing any article links. I assume this is because this web page uses javascript.

Offcourse I don't need to perform actions 1-6 in sequence. If it would require just one url request to get that text out of the 'pop-up' window I'd be more than happy.

So i'm stuck on step: 3 :-(

Thank you in advance for any help/advice.

about 4 years ago · Juan Pablo Isaza
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!