i've developed a scraper that retrieve some information for me, and i have this on heroku. it works fine except that i can't see the real-time updates (my app shows the first fetched values, not the actual one)
my code:
const axios = require('axios');
const cheerio = require('cheerio');
const express = require('express');
const PORT = process.env.PORT || 8000;
const app = express();
const myarr= []
app.get('/mylink', (req, res) =>{
axios.get(myUrl,{
responseEncoding: 'binary'
})
.then((response) => {
const html = response.data
const $ = cheerio.load(html)
$('tr', html).each(function (parentIdx) {
const title = $(this).find('a.tw-hidden.lg\\:tw-flex.font-bold.tw-items-center.tw-justify-between').text()
myarr.push({
title
})
})
res.json(myarr)
}).catch((err) => console.log(err))
})
app.listen(PORT, () => console.log(`Server avviato ed in ascolto sulla PORTA ${PORT}`))
how can i get updated results?
You cannot get immediate updated results when the site updates unless you own the website, in which case you can use websockets or a web hook. Failing that, your only other choice is to schedule a cronjob (as others have said) to check the website for changes at a specified interval (once per day, once per hour, etc.). For heroku, you can easily setup this up like so:
#!/usr/bin/env node. The rest of it can be your code./bin folder (create it if does not exist) at root level.myScript.js so that the full command is $ myScript.js. Obviously, myScript.js referrers to whatever your script's file name is.console.log output.