This is a little scraper for SPA's , my final goal would be to render the results and have a download as excel feature. However I am still far from that, so far, I can get the scraped results logged to console, but I am having issues getting them rendered at all to the front end.
The scraping part is done via IIFE, but the Express res.render does not have access to the returned values from the IIFE, it throws the following error >> "ReferenceError: jobTitles is not defined" atr line 11(where the res.render tries to get access to this bit:
let jobTitles = await page.$$eval('ul li h3 a', titles => {
return titles.map(title => title.innerText);
The full code:
const puppeteer = require('puppeteer');
const express = require('express');
const app = express();
const path = require('path');
const router = express.Router();
app.set('view engine', 'ejs');
app.get('/', function(req, res) {
res.render('pages/index', { key: `Job Titles on first page of Workable are:
${jobTitles.join(', ')}` });
});
app.use('/', router);
app.listen(process.env.port || 3000);
console.log('Running at Port 3000');
(async () => {
try {
const browser = await puppeteer.launch();
const page = await browser.newPage();
const navigationPromise = page.waitForNavigation();
await page.goto('https://jobs.workable.com/');
await page.setViewport({ width: 1440, height: 744 });
await navigationPromise;
await page.waitForSelector('ul li h3 a');
let jobTitles = await page.$$eval('ul li h3 a', titles => {
return titles.map(title => title.innerText);
});
console.log(`Job Titles on first page of Workable are: ${jobTitles.join(', ')}`);
await browser.close();
} catch (e) {
console.log(`Error while fetching workable job titles ${e.message}`);
}
})();
The console log statement runs fine, returns the results fine, so how can I structure this to get it rendered with ExpressJS?