I currently have a script that update datas on a website using api calls. I loop through all the products in my database and I do my api calls to update the data on the website.
Sometimes an error occurs in the middle of the script execution so I'm not able to complete all the updates since I can't go through all the products in the database. And when I restart my script it always start from the beginning (first products in the database) so I'm never completing my bulk update .
How can I continue updates after an error occurred ? For example, if error occurred at the product number 10, how can I keep doing the updates for the product number 11 without restarting my script from the beginning ? Is there a mongoDb function that tracks errors ?
I'm a beginner and I'm trying to understand the logics around bulk updates. I've thought about saving the ID of the products where the error occurred and then start updating after this ID. But, I'm sure there's a standard procedure for managing bulk updates error and failures.
I'm using javascript, nodeJS and cron for my script.
When you bulkupdate, you receive errors in BulkWriteError and if the operation is ordered, it will be stopped executing.
First of all, divide your updates into some batches may be 1000 or less or more depending upon how many in total you have.
Secondly, running the batch updates without order makes performance improvements as the next update doesn't wait for the current one to finish.
db.collection.bulkWrite(
[
{ insertOne : <document> },
{ updateOne : <document> },
{ updateMany : <document> },
],
{ ordered : false }
)
Thirdly, check for errors for the ids and push them to the next batch that you create or keep all the errors in one batch and run once you complete all the bulk updates.
We are following the same approach for more than 100k records and it works well today. If you or anyone else have a better solution I would definitely hear it and implement it.