I am trying to fetch all records from a remote api that has a limitation on how many can be fetched at once. I am trying to use recursion, but the general idea of what works in Python I can't get to work in JS as well.
This is what I am currently doing, and it will fetch 100 records, but then keeps going in the background. I have tried many different ways to do it and nothing seems to work at expected.
Any help would be great, thanks.
async getBooks(offset = 0) {
return await bookGenie.find(
'fiction',
100,
offset,
)
}
I suggest this one:
async function getBooks(offset = 0, result = []) {
try {
// get new page
const page = await bookGenie.find(
'fiction',
100,
offset,
);
// add page to result array
result = result.concat(page);
// check is this the end
if (page.length !== 100) { // or what condition do you need
// return result array
return result;
} else {
// or increase offset and continue processing
return getBooks(offset + 100, result);
}
} catch (err) {
throw err; // or handling somehow
}
}
Now recursive getBook returns promise with all records found merged to one array on resolve and throwed error on reject.
All requests will be executed one-by-one increasing offset by 100 on each step until request return less then 100 records (not full page = end of list).
Usage:
const list = await getBooks();
console.log(list);
// [ ... ]
// List of all records
Extra: same method but compact
async function getBooks(offset = 0, result = []) {
try {
const page = await bookGenie.find('fiction', 100, offset);
return page.length !== 100 ? result.concat(page) : getBooks(offset + 100, result.concat(page));
} catch (err) {
throw err; // or handling somehow
}
}
Your await getBooks() call:
getBooks((offset = offset += 100))
should probably be written like:
getBooks((offset += 100))
And the if statement before that looks wrong to me. Maybe try:
if (result.length < 1000) { // will continue until result.length is 1000
result.push(await getBooks(offset += 1)); /* + 1 so the .push will add one element to the array each time. + 100 may work too */
}
I wasn't able to actually test this, but this (or something similar to this) is more conventional for JS. Here is a link that might help you out https://www.javascripttutorial.net/javascript-recursive-function/