I have a folder with 98 .csv files that I need to read and save on a Postgres database. I'm running this code:
// Files in folder
const filesInFolder = fs.readdirSync('../pages-csv');
for (let i = 0; i < filesInFolder.length; i++) {
console.log(`Saving file ${filesInFolder[i]}`)
await fs.promises.readFile(path.join(__dirname, `../pages-csv/${filesInFolder[i]}`), 'utf8').then(async (data) => {
await createOne(data) // database function
})
}
But it leads to this error:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
It seems like files are not closing or not being flushed from memory after inserting in database.
Have you tried streaming it? Something like:
var csv = require('csv-parser')
var data = []
for (let i = 0; i < filesInFolder.length; i++) {
console.log(`Saving file ${filesInFolder[i]}`)
enter code here
// add the 'file path here'
fs.createReadStream(filesInFolder[i])
.pipe(csv())
.on('data', function (row) {
createOne(row)
})
.on('end', function () {
console.log('End cvs')
})
}
Or maybe there is a way to clear the garbage collection: global.gc(), after you expose it: node --expose_gc
I found the problem, it was the createOne function, it had a forEach loop, removed it and now it works!