Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

320
Vistas
Get text from window popup on ScienceDirect web page using Python requests-html

I'm total noob when it comes to javascript, html, web-scrapping and web data mining.

I'm trying to perform the following via Python script:

  1. Go to: https://www.sciencedirect.com/
  2. Search by author name and last name i.e: "Albert Einstein".
  3. Get links to the articles that are in search results page. enter image description here
  4. Access those links.
  5. On the article page, I need to access link with author credentials just below article title, when clicked, a window will appear from the right. enter image description here
  6. I want all the text from that window and I need it via Python script. Preferable by using some form of url requests.

So far I have the following code:

from bs4 import BeautifulSoup
from requests_html import HTMLSession

def AccessScienceDirect(credentials):

    # this url gives me search results page with links to articles
    _url = f"https://www.sciencedirect.com/search?authors={credentials}"
    session = HTMLSession()
    
    try:
        response = session.get(_url)
        response.html.render()
    except Exception as e:
        return (f'Error for {_url}', e)

    soup = BeautifulSoup(response.html.html, 'html.parser' )
    _text = soup.get_text()
    return _text

AccessScienceDirect("Albert Einstein")

Unfortunately I don't know what I should do next.

The _text doesn't seems to hold any information that resembles web page I can access by inserting link into browser: "https://www.sciencedirect.com/search?authors=Albert Einstein"

Not mentioning accessing any article links. I assume this is because this web page uses javascript.

Offcourse I don't need to perform actions 1-6 in sequence. If it would require just one url request to get that text out of the 'pop-up' window I'd be more than happy.

So i'm stuck on step: 3 :-(

Thank you in advance for any help/advice.

about 4 years ago · Juan Pablo Isaza
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda