I am trying to retrieve the 'Now Playing' information from http://radioplayer.magic.co.uk/live using Python and Beautiful Soup.
I can see the text in a web browser and can copy and paste it so I am assuming this text is downloaded from somewhere, when I look at the page from Beautiful Soup I can't see the text or even where it might be coming from.
I am a beginner at this so please be gentle!
Thanks in advance for sharing your knowledge and experience.
ADDITIONAL INFORMATION: I am using Python 3 on a raspberry pi
The content of Now Playing div is loaded dynamically by making an AJAX request and that is why it is not included in the source page you will received.
What you can do is imitating the ajax request made and fetching the response from there.
This is how you can achieve this :
import requests
import json
main_url = "http://radioplayer.magic.co.uk/live/"
ajax_url = "http://ps1.pubnub.com/subscribe/sub-eff4f180-d0c2-11e1-bee3-1b5222fb6268/np_4/0/14901814159272341?uuid=ef978c6c-2edf-4ff5-910a-39765d038427"
re = requests.get(ajax_url).content
playing_list = json.loads(re)[0]
max_time = 0
playing_now_dict = {}
for playings in playing_list :
if int(playings['start_time']) > max_time :
playing_now_dict = playings
print(playing_now_dict.get('title', ''))
print(playing_now_dict.get('artist', ''))
This currently prints :
Young Hearts Run Free
Candi Staton
It seems like a task for python and selenium: http://selenium-python.readthedocs.io/ (this enables you to control the browser and do whatever you can do manually, e.g. select displayed text)
(Warinng - the Firefox plugin is somewhat picky about the version, last stable version in Ubuntu works only with Firefox up to 45)
If you want to stick to using a headless browser (e.g. urllib, requests) then you will have to monitor the network calls while loading the website and get the exact URI (& necessary form data?) to use in python.
OR you could use python-selenium which will work exactly like the browser. Once you load the page, you can use driver.page_source to parse the source through BeautifulSoup.
Also, if you are lucky, maybe the website has an API (json/xml) that lets you fetch what you want without going through the hassle of parsing the raw source.