I using following code, but some URL from URL List, get 500 Error because of structure of the page.
I get the error exactly on
.map((htmlOnePage, index) =>line when some URLs page not valid, and the flow control of program goes at Catch Part. How I can find which URL is invalid?
const requestPromise = require('request-promise');
const Promise = require('bluebird');
const cheerio = require('cheerio');
for (var i = 1; i <= 250; i++) {
p = "https://mywebsite.com/" + i.toString()
urls[i - 1] = p
}
Promise.map(urls, requestPromise)
.map((htmlOnePage, index) => {
const $ = cheerio.load(htmlOnePage);
$('.txtSearch1').each(function() {
var h = "";
h = $(this).text()
h = h.replace(/(\r\n|\n|\r)/gm, "")
html44.push(h)
})
shareTuple[urls[index]] = html44;
html44 = []
fs.writeFileSync("data.json", JSON.stringify(shareTuple))
}, {
concurrency: 1
})
.then()
.catch((e) => console.log('We encountered an error' + e));
in other word how I can show which URL get into catch?
You should add try catch phrases inside all iterating functions to pin point the problem. and do the logging for every one based on op that they are catching.
For example i would wrap it like this:
try {
const $ = cheerio.load(htmlOnePage);
$('.txtSearch1').each(function () {
try {
var h="";
h=$(this).text()
h= h.replace(/(\r\n|\n|\r)/gm, "")
html44.push (h)
}catch (error) {
console.log('Error in getting p text');
console.log(error);
}
} catch (error) {
console.log('Error in loading'+ htmlOnePage);
console.log(error);
}
You can break down the error more by looking up through error object values to find the problem if you want to manualy remove it.
The other way to programaticaly remove it is to try doing the request for the page after creating it and also wrapping it in try catch. If it throws an exception you can add in catch to remove it from the list.
That text before console log of error is just so you can see where it broke.