I'm trying to form an url array to then perform parallel fetches usign .map and Promise.all The code in itself works if I try it with a manually made array but not if I try to form an array (for pagination) with a for loop. I get the maxnumber of pages from the headers of the first fetch and with that number I use a for loop to add an increasing numbered page until the final amount of pages.
I'm realizing trying to make a minimally reproducible example here that this isn't 'working' in a sense, when I do this in my server with this sample code, it works and
console.log(urls);
actually shows the looped urls and it does seem to be in array format, since I used push seems impossible not to be? But then again, when performing the urls.map it doesn't work at all and trying to do something else, like .slice to urls it doesn't work either, so I'm sensing I'm not forming the array I think I am? I can't figure it out.
async function smt() {
var url = 'https://jsonplaceholder.typicode.com/todos/';
var urls = [];
var firstFetch = fetch(url)
.then(function(e) {
var maxpages = 5;
//var maxpages = e.get.headers('TotalPages'); < I would get the maxpages here
for (let i = 1; i < maxpages; i++) {
urls.push(url + i)
}
});
console.log(urls);
var data = await Promise.all(urls.map(async url => {
const res = await fetch(url);
return res.json();
}))
console.log(data);
var other = [
'https://jsonplaceholder.typicode.com/todos/1',
'https://jsonplaceholder.typicode.com/todos/2',
'https://jsonplaceholder.typicode.com/todos/3',
'https://jsonplaceholder.typicode.com/todos/4',
]
var data2 = await Promise.all(other.map(async url => {
const res = await fetch(url);
return res.json();
}))
console.log(data2);
}
smt();The problem was with the .then() function I had no clue but I tested everypart until it worked.
async function smt() {
var url = 'https://jsonplaceholder.typicode.com/todos/';
var urls = [];
var firstFetch = await fetch(url)
var maxpages = 5;
//var maxpages = firstFetch.get.headers('TotalPages'); < I would get the maxpages here
for (let i = 1; i < maxpages; i++) {
urls.push(url + i)
}
console.log(urls);
var data = await Promise.all(urls.map(async url => {
const res = await fetch(url);
return res.json();
}))
console.log(data);
var other = [
'https://jsonplaceholder.typicode.com/todos/1',
'https://jsonplaceholder.typicode.com/todos/2',
'https://jsonplaceholder.typicode.com/todos/3',
'https://jsonplaceholder.typicode.com/todos/4',
]
var data2 = await Promise.all(other.map(async url => {
const res = await fetch(url);
return res.json();
}))
console.log(data2);
}
smt();/**** Updated Answer ****/
async function smt() {
var url = 'https://jsonplaceholder.typicode.com/todos/';
var urls = [];
//var firstFetch = await fetch(url) < would get headers
var maxpages = 5;
//var maxpages = firstFetch.get.headers('TotalPages'); < I would get the maxpages here
for (let i = 1; i < maxpages; i++) {
urls.push(url + i)
}
console.log(urls);
let promises = urls.map(async url => {
let a = await fetch(url);
return a.json()
})
var data = await Promise.all(promises)
console.log(data);
}
smt();I think this is a better approach since you can limit concurrency when you do the urls.map outside the Promise.all.
I have done it with p-limit in node it works great.
const limit = pLimit(8);
let promises = urls.map(url => {
return limit(async() => {
let a = await fetch(url, optionsGet)
return a.json()
})
});