I would like to use Python to retrieve some content behind the JavaScript. I have a website that is structured like this: (I have added line breaks for the ease of reading):
'<!DOCTYPE html>\n<html lang="ja">\n <head>\n <meta charset="utf-8"/>\n <title>HRMOS CORE Employee</title>\n
<base href="/"/>\n <meta name="viewport" content="width=device-width, initial-scale=1"/>\n
<link rel="icon" type="image/x-icon" href="assets/favicon.ico"/>\n <link rel="stylesheet" href="styles.ea399ef1dfe4d0c1ba1f.css"></head>\n
<body class="pol">\n
<ess-root></ess-root>\n <script src="runtime-es2015.9968ad6faaffe0609121.js" type="module"></script><script src="runtime-es5.9968ad6faaffe0609121.js" nomodule defer></script><script src="polyfills-es5.1271ae52020e40d8a200.js" nomodule defer></script><script src="polyfills-es2015.04cc7054da2685d59767.js" type="module"></script><script src="main-es2015.07a2df0e93ddc04b405f.js" type="module"></script><script src="main-es5.07a2df0e93ddc04b405f.js" nomodule defer></script></body>\n</html>\n'
The actual html content is an interactive table which is rendered by the js scripts in the previous code.
I am trying to retrieve the content of the table using the request module. I used the the following code (I am sure the authorization is working because the request module is returning status 200):
from requests_html import AsyncHTMLSession
import lxml.html
session = AsyncHTMLSession()
r = await session.get('https://ess.hrmos.co/', auth=('user', 'password'))
await r.html.arender(wait=10, sleep=10, keep_page=True)
tree = lxml.html.fromstring(r.text)
title_elem = tree.xpath("/html/body/ess-root/core-ui-theme/ess-core-layout/pol-layout/pol-content/ess-employee-list-page/div/div/ess-employee-cards-container/cdk-virtual-scroll-viewport/div[1]/div[1]/a[1]/ess-employee-card/pol-card/div[1]")
print("title tag:", title_elem.tag)
But even if I use wait in the code, the actual html I get from request is empty without any content in it. Is there a way to make sure I retrieve the content? Or do I have to use selenium?