Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

194
Vistas
How to get Python to scrape web page generated by JavaScript files

I have a website I want to automate some actions on but the page is generated by 2 JavaScript files and is defined like this in the html:

<script src="/build/runtime.js"></script><script src="/build/app.js"></script>

runtime.js is about 70 lines and app.js is about 40k lines... I have no idea how to read the code as I don't know any JavaScript and my Pyton knowledge is a mere atom more ;)

I'd share the particular site but the page is behind a login. So I've managed to get to the page using 2 different methods but can't find a way to press buttons within this next page generated by the JS.

Method 1 - Requests & BeautifulSoup but got stuck on the JS bit so switched to method2.

import requests
from bs4 import BeautifulSoup

# Site & creds
LOGIN_URL = 'https://website.com/login'
USERNAME = 'user'
PASSWORD = 'pass'

# Pretend to be browser
headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
}
# Start session
session = requests.session()
# Get login page
response = session.get(LOGIN_URL, headers=headers, verify=False)
# Get csrf token
soup = BeautifulSoup(response.content, 'html.parser')
csrf_token = (soup.find(id="login_form__token")["value"])
# Set creds with csfr token
payload = {
    'login_form[username]': USERNAME,
    'login_form[password]': PASSWORD,
    'login_form[login]': '',
    'login_form[_token]': csrf_token
}
# Login & do something else with cookies I don't understand
response = session.post(LOGIN_URL, data=payload, verify=False)
response = session.get('https://website.com/pageIWant', verify=False)

soup = BeautifulSoup(response.content, 'html.parser')
print(soup.prettify())

Method 2 - Selenium & ChromeDriver

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")
options.add_argument('--disable-gpu')
options.add_argument('--disable-software-rasterizer')

driver = webdriver.Chrome(options=options, executable_path='chromedriver.exe')
driver.get("https://website.com/login")

driver.find_element_by_id("login_form_username").send_keys('user')
driver.find_element_by_id("login_form_password").send_keys('pass')
driver.find_element_by_id("login_form_login").click()

driver.get("https://website.com/pageIWant")

html = driver.page_source
print(html)

So I thought method 2 would make things easier but pretty much stuck at the same point. The page generated that I want contains buttons I'd need to press in order to access downstream pages. Read a lot about accessing elements but can't see anything within this 40k worth of JS jibberish. Where is a good place to start?

about 4 years ago · Santiago Gelvez
1 Respuestas
Responde la pregunta

0

"Where is a good place to start?"

Regardless of how the page is generated (HTML or JS), ultimately what you have to address in Selenium is the page's live DOM. So "where to start" is inspecting the page's DOM in browser dev tools, and from the DOM, figure out how to find the button elements in Selenium.

about 4 years ago · Santiago Gelvez Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda