I want to constantly sync data from a website, however I only have 300 calls/15 minutes. Thus I thought i could put all my sync requests (around 1000) into an array and then resolve just 300 of them every 15 minutes till the requests array is empty and then start again. However when I do the following:
let requests = []
params = 'invoice-items?invoice_id='
let positions = null
for (const invoice of invoices) {
requests.push(new Promise(async (resolve, reject) => {
positions = await getBillomatData(params + invoice.id, null, 0, null)
await updateDatabase(positions, models.billomat.Position)
}))
}
console.log(requests[0])
await requests[0]
console.log(requests[0])As soon as I wait for the request at requests[0] it executes all of the requests and I go over the limit of calls.
The most simplest approach (not optimised though) would be
let batch = []
// imagine urls is a array of url of 900 items
urls.map(async (url)=>{
batch.push(somePromiseFuctionToDoTheApiCall(url))
if(batch.length >= 300){
await Promise.all(batch)
// sleep is a promisified settimeout function, ref: https://stackoverflow.com/a/56520579/3359432
await sleep(calculateTimeToWaitBeforeProceedingToNextBatch)
batch = []
}
})
// there might be some leftovers at the end of batch you should process them also
if you are ok with using libraries and stop reinventing the wheel then have a look at lodash.chunk, bluebird.map, bluebird.each, bluebird.delay
All async calls are executed at once because JavaScript executes ALL code as soon as each call is processed. Await awaits the result, it does not await execution.
You'll need to use a tool such as bottleneck to rate limit your requests.
Processing 300 requests of 1000 every 15 minutes will take an hour to complete. This is a long time to keep the node job running doing nothing.
A basic a basic limiter might keep track of all the requests you want to make in an external file or database and then use a cron job to execute your JavaScript code every 15 minutes to process another 300 requests. If there are no additional requests, your app could just terminate. However, it will wake up and run every 15 minutes as long as the cron job keeps running.