I am working with large arrays where I need to make a call for each item in an array. I am mapping the array to create an array of axios calls and then using Promise.all to await all the promises at once. However, axios returns a large object and all I really need is what is under the data key. Is there a way to destructure this within a Promise.all to just get the returned body?
I know I could probably map right after the Promise.all, but that has overhead as well and the arrays are really large and I want to save as much memory as possible.
const largeArrOfIds = [1,2,3,4...]
const mappedCalls = largeArrOfIds.map(x=> axios({
url: `exampleurl.com/${x}`,
headers
}))
const result = await Promise.all(mappedCalls)
// Result here looks like this:
result = [{
config: {11},
data: {1},
headers: {17},
request: {38},
status: 200,
...
},{another}, {another}, ...]
// But I really just want an array of all the "data" and
// I don't want the memory to be used to hold all the other data.
Thank you all!
The best way to do this would be to change the backend to only serve you the values you want, instead of having to request everything and then filter for only the property you want in your local script.
Lacking that - no matter what approach you use, all values will be loaded into memory at some point during the script execution. If the amount of data really is so large that it's a problem, the only approach to reduce the load I can think of would be to put a .then right after the axios call, and extract only the property you want - this way, as soon as a single request resolves, only the values you need are saved semi-permanently, and the rest are free to be garbage collected.
const mappedCalls = largeArrOfIds.map(x=>
axios({
url: `exampleurl.com/${x}`,
headers
})
.then(result => result.data)
)
But this isn't likely to have much effect, because garbage collectors usually run seconds apart, not milliseconds apart - and even if you have a lot of requests, it's not likely to take more than a second or two for all of them to resolve (so there may not be time for the GC to run before all are finished and everything is in memory anyway).
If you really wanted to reduce overall memory usage at any given time, I suppose you could make the requests serially, instead of using Promise.all (and use FinalizationRegistry if you wanted to be really fancy), but that would have the side effect of making the script take a lot longer to complete.