Let's say I have collection of events. There will be much of them, like 10-20k. For each these events I need to make additional request to get eventDetails. Base implementation would looks like:
const eventsWithDetails = await Promise.all(
events.map(async(event) => {
const eventDetails = await event.getEventDetails();
return {
...event,
...eventDetails
}
})
);And actually this is what I was needed with one clarification about the API I interact with: with a large number of consecutive requests, the API periodically throws errors for some of them due to overloading the endpoint. One of the brute force solution - is a just make sequential requests for bunches per N items (of course it's ugly and completely unscalable. But it works!):
const slice1 = await Promise.all(
events.slice(0, 500).map(mapEventWithTimestamp)
);
await sleep(4000);
const slice2 = await Promise.all(
events.slice(500, 1000).map(mapEventWithTimestamp)
);
await sleep(4000);
...slices3,
...sliceN
return [...slice1, ...slice2, ...sliceN]As part of the search for a robust solution, I'm trying to wrap my head around implementation with usage of async iterator. Having stateful object which will also has async iteration interface. The point here to make delay of N seconds after each 1000 requests. Something like that:
let mapper = {
start: 0,
stop: events.length,
step: 1000,
result: [],
async * [Symbol.asyncIterator]() {
for (let current = this.start; current <= this.stop; current += this.step) {
await sleep(4000);
let slice = await Promise.all(events.slice(current, current + this.step);
this.result = [...this.result, ...slice];
yield slice;
}
}
// is there way to return this.result to external usage??
};
}Is there any way to accomplish this task with elegant and scalable solution with async iteration? To have definitive interface like that:
const eventsWithDetails = await mapEventsWithDetails(events); (which will make internal iteration and then returns mapped data)
You can try the following, using iter-ops library, to process everything sequentially, as an iterable:
import {pipe, toAsync, map, wait} from 'iter-ops';
const i = pipe(
toAsync(events), // make the list asynchronous
map(event => event.getEventDetails()), // remap into requests
wait() // resolve each promise inside iterable
);
// this is where it will start execution;
for await(const a of i) {
console.log(a); // print whatever data you're getting
}
And you can add delay or throttle operators to the pipeline, as needed.
And if you actually want to process data in bulk, you can make use of the page operator, to split requests into pages:
import {pipe, toAsync, map, page, wait} from 'iter-ops';
const i = pipe(
toAsync(events),
page(500), // split into pages of 500 items in each
map(page => Promise.all(page.map(a => a.getEventDetails()))),
wait() // resolve each page
);
// this will trigger processing one page at a time;
for await(const page of i) {
console.log(page); // print a whole page of data
}