I need to insert
10K documents as fast as possible but it's taking a long time.
I am currently using
Model.create([<huge array here>]) to do this.
Would it help to use multiple connections to the database? For example have 10 connections saving
You can use
Some stuff to note below.
10 connections is usually sufficient, but it greatly depends on your hardware. Opening up more connections may slow down your server.
In some cases, the number of connections between the applications and the database can overwhelm the ability of the server to handle requests.
There are a couple of options for insertMany that can speed up insertion.
[options.lean «Boolean» = false] if true, skips hydrating and validating the documents. This option is useful if you need the extra performance, but Mongoose won't validate the documents before inserting.
[options.limit «Number» = null] this limits the number of documents being processed (validation/casting) by mongoose in parallel, this does NOT send the documents in batches to MongoDB. Use this option if you're processing a large number of documents and your app is running out of memory.
Setting options on
writeConcern in options can also affect performance.
If applications specify write concerns that include the j option, mongod will decrease the duration between journal writes, which can increase the overall write load.