0
346
ViewsHow to save a large array to MongoDB using mongoose in Node.js
I'm wanting to save a very large array (over 40k strings) in MongoDB.
const allWords = require("an-array-of-english-words");
const { patterns } = require("./models/pattern");
const mongoose = require("mongoose");
// Create a model for the Words object
const Words = mongoose.model(
"Words",
new mongoose.Schema({
words: {
type: Array,
required: true,
},
})
);
/*
Filters the list of words to use only words greater than 4 and less than 6
*/
const array = allWords.filter(function (text) {
return text.length >= 4 && text.length <= 6;
});
let words = [...array];
for (let i = 0; i < array.length; i++) {
for (let j = 0; j < patterns.length; j++) {
let result = patterns[j].test(array[i]);
if (result) {
let index = array.indexOf(array[i]);
array.splice(index, 1);
}
}
}
async function saveWords(words) {
console.log("start");
const array = new Words({ words });
console.log("mid");
console.log(array);
//it's successfully making the array object but it's having trouble saving it
await array.save();
console.log("done");
}
saveWords(words);
console.log("array length: " + array.length + " " + allWords.length);
Everything works up until the call to save the array, then a time out error is logged on the console. This is my first project working on my with Node.js and I'm sure I'm making a small easily fixable mistake but I'm just not sure what it is.
1 answers
Answer question0
I would say you don't need to save these words in the DB instead create an algorithm that creates a random string first, then you can add randomness to the created string if you want like adding numbers or special characters.
You could use a third-party npm package. You can check this.
Also, it would be good if you look at how others achieve it. This will be a good practice to improve on how you can implement it efficiently. This can be a good package.