Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

241
Views
Scraping text from a website when the text does not appear in the source

I am trying to retrieve the 'Now Playing' information from http://radioplayer.magic.co.uk/live using Python and Beautiful Soup.

I can see the text in a web browser and can copy and paste it so I am assuming this text is downloaded from somewhere, when I look at the page from Beautiful Soup I can't see the text or even where it might be coming from.

I am a beginner at this so please be gentle!

Thanks in advance for sharing your knowledge and experience.

ADDITIONAL INFORMATION: I am using Python 3 on a raspberry pi

about 4 years ago · Santiago Trujillo
3 answers
Answer question

0

The content of Now Playing div is loaded dynamically by making an AJAX request and that is why it is not included in the source page you will received.

What you can do is imitating the ajax request made and fetching the response from there.

This is how you can achieve this :

import requests
import json

main_url = "http://radioplayer.magic.co.uk/live/"
ajax_url = "http://ps1.pubnub.com/subscribe/sub-eff4f180-d0c2-11e1-bee3-1b5222fb6268/np_4/0/14901814159272341?uuid=ef978c6c-2edf-4ff5-910a-39765d038427"
re = requests.get(ajax_url).content
playing_list = json.loads(re)[0]
max_time = 0
playing_now_dict = {}

for playings in playing_list : 
    if int(playings['start_time']) > max_time  : 
        playing_now_dict = playings
print(playing_now_dict.get('title', ''))
print(playing_now_dict.get('artist', ''))

This currently prints :

Young Hearts Run Free
Candi Staton
about 4 years ago · Santiago Trujillo Report

0

It seems like a task for python and selenium: http://selenium-python.readthedocs.io/ (this enables you to control the browser and do whatever you can do manually, e.g. select displayed text)

(Warinng - the Firefox plugin is somewhat picky about the version, last stable version in Ubuntu works only with Firefox up to 45)

about 4 years ago · Santiago Trujillo Report

0

If you want to stick to using a headless browser (e.g. urllib, requests) then you will have to monitor the network calls while loading the website and get the exact URI (& necessary form data?) to use in python.

OR you could use python-selenium which will work exactly like the browser. Once you load the page, you can use driver.page_source to parse the source through BeautifulSoup.

Also, if you are lucky, maybe the website has an API (json/xml) that lets you fetch what you want without going through the hassle of parsing the raw source.

about 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!