Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

242
Vistas
Scraping text from a website when the text does not appear in the source

I am trying to retrieve the 'Now Playing' information from http://radioplayer.magic.co.uk/live using Python and Beautiful Soup.

I can see the text in a web browser and can copy and paste it so I am assuming this text is downloaded from somewhere, when I look at the page from Beautiful Soup I can't see the text or even where it might be coming from.

I am a beginner at this so please be gentle!

Thanks in advance for sharing your knowledge and experience.

ADDITIONAL INFORMATION: I am using Python 3 on a raspberry pi

about 4 years ago · Santiago Trujillo
3 Respuestas
Responde la pregunta

0

The content of Now Playing div is loaded dynamically by making an AJAX request and that is why it is not included in the source page you will received.

What you can do is imitating the ajax request made and fetching the response from there.

This is how you can achieve this :

import requests
import json

main_url = "http://radioplayer.magic.co.uk/live/"
ajax_url = "http://ps1.pubnub.com/subscribe/sub-eff4f180-d0c2-11e1-bee3-1b5222fb6268/np_4/0/14901814159272341?uuid=ef978c6c-2edf-4ff5-910a-39765d038427"
re = requests.get(ajax_url).content
playing_list = json.loads(re)[0]
max_time = 0
playing_now_dict = {}

for playings in playing_list : 
    if int(playings['start_time']) > max_time  : 
        playing_now_dict = playings
print(playing_now_dict.get('title', ''))
print(playing_now_dict.get('artist', ''))

This currently prints :

Young Hearts Run Free
Candi Staton
about 4 years ago · Santiago Trujillo Denunciar

0

It seems like a task for python and selenium: http://selenium-python.readthedocs.io/ (this enables you to control the browser and do whatever you can do manually, e.g. select displayed text)

(Warinng - the Firefox plugin is somewhat picky about the version, last stable version in Ubuntu works only with Firefox up to 45)

about 4 years ago · Santiago Trujillo Denunciar

0

If you want to stick to using a headless browser (e.g. urllib, requests) then you will have to monitor the network calls while loading the website and get the exact URI (& necessary form data?) to use in python.

OR you could use python-selenium which will work exactly like the browser. Once you load the page, you can use driver.page_source to parse the source through BeautifulSoup.

Also, if you are lucky, maybe the website has an API (json/xml) that lets you fetch what you want without going through the hassle of parsing the raw source.

about 4 years ago · Santiago Trujillo Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda