I am currently working on a project where I have to read a large .csv file and upload its contents in a MongoDB collection. To read the CSV files, I am using the @fast-csv/parse npm module. Also, I am using node js stream API to pause the streaming and insert 200 documents per batch into a MongoDB collection. After that I resume the streaming. Here is the code.

  module.exports.seedProductCollection = (filepath) => new Promise((resolve, reject) => {
  let documents = [];
  let counter = 0;
  const stream = fs.createReadStream(filepath);
  stream.pipe(csv.parse({ headers: true }))
    .on('error', (error) => reject(error))
    .on('data', async (row) => {
      if (documents.length >= 200) {
        stream.pause();
        try {
          documents.push(row);
          await db.Product.insertMany(documents);
          counter += documents.length;
          console.log('counter: ', counter);
          documents = [];
          stream.resume();
        } catch (error) {
          reject(error);
        }
      } else {
        documents.push(row);
      }
    })
    .on('end', async () => {
      if (documents.length > 0) {
        try {
          await db.Product.insertMany(documents);
          counter += documents.length;
        } catch (error) {
          reject(error);
        }
      }
      resolve(counter);
    });
});
   

So, basically, I have created a readStream from the .csv file and inserted every row of the .csv file as a JavaSCript object into the documents array. When the documents array has 200 objects, I paused the streaming using stream.pause(). Then inserting these 200 objects into a MongoDB collection and then resuming again the stream using stream.resume(). Also, assigning the documents array an empty array documents = [] . After the read streaming is done, in the stream.on('end') event listener, I am checking if there are more objects in the documents array and if it has any, I am inserting them again into the collection. Now, the issue is, when I check the collection in MongoDB, I noticed after every 200 documents, the documents are repeating again like this.

   { id: 1,
     name: 'product1',
     description: 'this is product1' 
   }, 
   { id: 2,
     name: 'product2',
     description: 'this is product2' 
   },
   .....
   { id: 200,
     name: 'product200',
     description: 'this is product200' 
   },
   { id: 1,
     name: 'product1',
     description: 'this is product1' 
   }, 
   { id: 2,
     name: 'product2',
     description: 'this is product2' 
   },
   

As you can see, after the productId 200, the documents are repeating again and the streams enter an infinite loop. If I change the length of the documents array from 200 to 100, it happens after every 100 items. I am really stuck at this point. Tried a couple of things, but nothing worked. Any help would be greatly appreciated. Thanks in advance.