I'm using puppeteer on my telegram bot to fetch stuff from the web. Locally, it's all work just fine, but here is the catch:
When trying to fetch the URL's HTML, the given response is not the HTML file, but the web app with all the js files. On my local machine - I'm getting the HTML file with all needed links.
That's my code:
return await puppeteer.launch({
args: ['--no-sandbox', '--disable-setuid-sandbox'],
headless: true,
}).then(async browser => {
const page = await browser.newPage()
page.setJavaScriptEnabled(true)
await page.goto(targetUrl)
await page.waitForTimeout(2000)
const data = await page.$$eval(SELECTORS, (res) => {
return res.map(r => {
return r.getAttribute('ATRR')
})
}) as never as string[]
.......
})PS -------------- already added this repo to Heroku:
https://github.com/jontewks/puppeteer-heroku-buildpack
Apparently, when you are using puppeteer-extra, you need to install puppeteer library too.
For some reason, they both depend BUT not auto-installed, so the Chromium engine didn't install on Heroku, and it does explain how it does on my local machine - which has it by default.