How can I further automate my web-scrapper? Currently it can search singular constant urls. How do I add a function that makes it search multiple pages within a constant website.
Here is my code
const PORT = 8000
const axios = require('axios')
const cheerio = require('cheerio')
const express = require('express')
const app = express ()
const url = 'heresurdata'
axios(url)
.then(response => {
const html = response.data
console.log(html)
})
app.listen (PORT, () => console.log('server running on PORT ${PORT}'))
you could make a request to the first page find all the anchor tags get their links make request to them and repeat the process
const {data} = await axios.get(url);
const $ = cheerio.load(data);
const allLinkTags = $('body a').map((index, el) => $(el).attr('href')).get();
// Now look through all the link tags and scrape them