I have recently run into an issue I cannot solve using Axios and Cheerio. What I am trying to do is scrape a coffee store's webpage that contains multiple different coffee products. So far I have been able to scrape each of the the coffee names, countries, its flavors, and the urls to each single product page. What I would like to do is take it a step further and scrape each of the single product urls and return its stock availability. On each single product page there is a description div that contains some text saying whether it is in stock or not. However I have run into the issue of scraping each url for that text.
So far this is what I have:
const links = [];
axios(coffeeStoreURL)
.then(response => {
const html = response.data;
const coffeePageHTML = cheerio.load(html);
const coffees = [];
coffeePageHTML('.coffeeBadge', html).each(function() {
const coffeeName = coffeePageHTML(this).find('.coffeeBadgeBottom a').text()
const coffeeCountry = coffeePageHTML(this).find('.country-label').text()
const coffeeFlavors = coffeePageHTML(this).find('.flavors').text()
const coffeeImage = coffeePageHTML(this).find('img').attr('src')
const coffeeURL = coffeePageHTML(this).find('a').attr('href')
coffees.push({
coffeeImage,
coffeeName,
coffeeCountry,
coffeeFlavors,
coffeeURL
});
links.push((
coffeeURL
))
});
})
.then( () => {
for (let i = 0; i < links.length; i++) {
axios(links[i])
//This is where I am stuck on how to scrape each link in my links array and use cheerio to grab the div data from each page
}
}).catch(err => console.log(err));
I assume I am on the right path but if not then any guidance would be much appreciated.
Thank You
The quick but less elegant way to do this is by calling a self-executing anonymous function (AKA an IIFE) that holds each iteration inside your for loop.
for (let i = 0; i < links.length; i++) {
(function(link) {
// do your axios calls here, "link" is holding the current URL
})(links[i]);
}
If you want something more elegant then check out this question: Axios.get().then() in a for loop