I need to insert 10K documents as fast as possible but it's taking a long time.
I am currently using Model.create([<huge array here>]) to do this.
Would it help to use multiple connections to the database? For example have 10 connections saving 1K each?
Santiago Trujillo
You can use model.insertMany(doc, options).
Some stuff to note below.
10 connections is usually sufficient, but it greatly depends on your hardware. Opening up more connections may slow down your server.
In some cases, the number of connections between the applications and the database can overwhelm the ability of the server to handle requests.
There are a couple of options for insertMany that can speed up insertion.
[options.lean «Boolean» = false] if true, skips hydrating and validating the documents. This option is useful if you need the extra performance, but Mongoose won't validate the documents before inserting.
[options.limit «Number» = null] this limits the number of documents being processed (validation/casting) by mongoose in parallel, this does NOT send the documents in batches to MongoDB. Use this option if you're processing a large number of documents and your app is running out of memory.
Setting options on writeConcern in options can also affect performance.
If applications specify write concerns that include the j option, mongod will decrease the duration between journal writes, which can increase the overall write load.
Use db.collection.insertMany([])
insertMany will accept array of objects and this is use to perform bulk insertions.