I have an array of URLs I scraped from a webpage and then make an API call to validate the URLs to see if they are malicious. The only problem is I am limited to a certain amount of API calls per day and the array contains duplicate URLs. I am trying to loop through the array and used a saved API call for duplicate values and I am struggling to find the best way to do it since there could be multiple duplicates. If the loop encounters a duplicate value I want it to not make the API call and just return the already saved values from the previous API call. I included some basic sudo code inside the code below and I am unsure of what to populate the sudo code with.
/* urlToValidate is a list of URLS */
urlToValidate = ["ups.com", "redfin.com", "ups.com", "redfin.com", "redfin.com", "redfin.com"];
var isValid = false;
/* API Overview https://www.ipqualityscore.com/documentation/malicious-url-scanner-api/overview */
for (let i = 0; i < urlToValidate.length; i++) {
if (i == 0 || Is Not A DUPLICATE) {
$.getJSON('https://ipqualityscore.com/api/json/url/<API_KEY>/' + urlToValidate[i], function( json ) {
if (!json.phishing && !json.spamming && json.risk_score < 80) {
isValid = true;
returnMessage(isValid, json.risk_score, i)
} else {
isValid = false;
returnMessage(isValid, json.risk_score, i)
}
});
} else {
returnMessage(alreadySaved duplicateValue, alreadySaved duplicate risk_score, i)
}
}
Desired Output:
URL Valid: true Risk Rating: 0 Position: 7
or
Duplicate URL: true Risk Rating: 0 Position: 7
This is a simple matter of caching.
Outside of your for loop, maintain some kind of mapping of URLs to their corresponding fetch results. That way, you can store not only whether that URL has been called but also the result, if it exists. An easy way to do that is with a basic object, where the "keys" are strings corresponding to the URLs, and the "values" are the results of your fetches.
const resultCache = {};
Inside of your loop, before you do a fetch you should first check whether the cache already has a result for that URL.
let result;
if (resultCache[urlToFetch]) {
result = resultCache[urlToFetch];
} else {
// use the previous result
result = await fetch(/* whatever */);
// remember to also store result in cache
resultCache[urlToFetch] = result;
}
You have a few options.
First you could convert your urls to a Set which prevents any duplicates from occurring at all.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set
Another option would be to store the return in an object with the key being the url and in your if statement check to see if the value is not null.
*** UPDATE using a set ***
/* urlToValidate is a list of URLS */
urlToValidate = ["ups.com", "redfin.com", "ups.com", "redfin.com", "redfin.com", "redfin.com"];
var urls = new Set(urlToValidate);
var isValid = false;
/* API Overview https://www.ipqualityscore.com/documentation/malicious-url-scanner-api/overview */
for (let i = 0; i < urls.length; i++) {
$.getJSON('https://ipqualityscore.com/api/json/url/<API_KEY>/' + urls[i], function( json ) {
if (!json.phishing && !json.spamming && json.risk_score < 80) {
isValid = true;
returnMessage(isValid, json.risk_score, i)
} else {
isValid = false;
returnMessage(isValid, json.risk_score, i)
}
});
}
}