Suppose I have code as below:
const fs = require('node:fs')
const csvParser = require('csv-parser')
const rs = fs.createReadStream(dir)
const ws = fs.createWriteStream(outDir)
ws.write('[' + '\n')
rs.pipe(
csvParser({
mapHeaders: mapper,
}),
)
.on('data', (data) => {
ws.write('\t' + JSON.stringify({ ...data, provider } + ',\n'))
})
.on('end', () => {
ws.write(']')
console.log('writing to a file has been completed >> ', OUT_DIR)
})
Basically what this code does is to read from large csv file then stream this into write a json file. As seen in the code, I append ',\n' as separator for each row such that all the json code is formatted once streaming has been completed.
The issue with such method is that it leaves comma at the end of each entry that you would have to manually format the file (i.e. removing trailing comma) each time document write is completed.
For instance, the result would come out be as such:
[
{"key1":"value1", "key2":"value2"},
{"key1":"value1", "key2":"value2"}, // <- note the trailing comma here.
]
Any suggestion as to handling the trailing comma? Also, what would be alternative way of handling read and write stream better, please let me know. Thanks in advance.
I'd reverse the problem and include the comma before every item except the first.
let first = true;
ws.write('[' + '\n')
// ...
.on('data', (data) => {
if (!first) {
ws.write(',\n');
}
first = false;
ws.write('\t' + JSON.stringify({ ...data, provider }))
})
[EDIT] as mentioned in comments, this defeats the purpose of using stream. Don't use this on really large files
I would not handle the trailling coma but here is how I would do it.
The idea is to push each row into an array and the JSON.stringify()thee whole array.
Something like this
const rows = []
rs.pipe(
csvParser({
mapHeaders: mapper,
}),
)
.on('data', (data) => {
rows.push({ row content })
})
.on('end', () => {
ws.write(JSON.stringify(rows))
console.log('writing to a file has been completed >> ', OUT_DIR)
})
There is probably a better way to do it but I would do something like this:
const fs = require('node:fs')
const csvParser = require('csv-parser')
const rs = fs.createReadStream(dir)
const ws = fs.createWriteStream(outDir)
let buffer = null;
ws.write('[' + '\n')
rs.pipe(
csvParser({
mapHeaders: mapper,
}),
)
.on('data', (data) => {
if (buffer) {
ws.write(buffer + ',\n'))
}
buffer = '\t' + JSON.stringify({ ...data, provider })
})
.on('end', () => {
ws.write(buffer + '\n'))
ws.write(']')
console.log('writing to a file has been completed >> ', OUT_DIR)
})