I have little to no knowledge on dynamic page, so, this might come across as duplicate questions. But so far I have not been able to find a solution.

What I want to do is the scrape prices of financial products (index, futures, VI) from this website. (It is in Japanese though.)

https://nikkei225jp.com/chart/

If I look at the network tab, I can see that it is sending a request to the server every second to retrieve a payload, which is then used to update the prices. The end of the string looks suspiciously like epoch time, but it is not exactly it either, I think it has about 4 extra characters.

Sample request URL: https://nikkei225jp.com/_data/_nfsWEB/ajax_nikkei.js?_=1652153479901

Payload: A[111]="25978.44_-340.90_-1.30_12:35_1_26193.44_25773.83"; A[112]="1854.24_-24.15_-1.29_12:35_1_1865.04_1840.00"; ...

I thought if I could just have a way to generate the timestamp, and using the request from Python, I could retrieve the payload, but website engineering is much more complicated than what I was thinking.

I could use selenium, but it was a bit too slow. Is headless browser the only way to achieve what I want? Or am I missing some other methods? Also, this webpage is considered as Javascript, but I heard it has something to do with websocket, which I have little to no idea.

Currently, I am using request and beautifulsoup to obtain the index price, but I needed other prices, which is not available in the html code.

Thanks in advance.