I'm new to Javascript and NodeJS. Im trying to read multiple CSV files before doing some processing on them. My current issue is when I run the code it tries to execute the processing before the reading of the file is complete. I would like to load both the CSVs before I start doing any processing on them.
Could some explain why this happens and how I can solve the problem in Javascript/NodeJS.
function readCSV(path){
var events = []
fs.createReadStream(path).pipe(csv()).on('data', (row) => {
events.push(row);
}).on('end', () => {
console.log('CSV file successfully processed. Length: ' + events.length);
});
return events
}
function app(){
var file_list = listFiles(folder_path);
for (let i = 0; i < file_list.length; i++) {
const file = file_list[i];
var events = readCSV(file)
}
processCSV(events) // Some processing
}
app();
Any help would be great and any explanation on how I can control when code is executed would be much appreciated.
Sorry, you code cannot be compiled, so a can answer only with untested code
My current issue is when I run the code it tries to execute the processing before the reading of the file is complete.
The main problem that fs.createReadStream don't read file, - it request file system to start reading and inform your callbacks when some chunks was read
so event 'end' will be called match later, when readCSV will be completed and returned empty result
You code was written as if you expect synchronous answer, and you can make it work correctly with use of sync methods like fs.readFileSync
How to fix it in asynchronous way? write csv processing in "on(end)" callback or Use promises
Promises are match more simple and linear
First make readCSV to return Promise
function readCSV(path: string){ //return Promise<any[]>
return new Promise((resolve) => {
var events = [];
fs.createReadStream(path)
.pipe(csv())
.on('data', (row) => {
// this code called in future
events.push(row);
}).on('end', () => {
// this code called in future to,
console.log('CSV file successfully processed. Length: ' + events.length);
resolve(events); //return csv parsed result
});
})
}
then in main app, use Promise.all to wait all fileReading promises
function app(){
// i don't know what is listFiles,
// i hope it returns sync result
var file_list = fs.listFiles(folder_path);
const dataPromises: Promise[] = []
for (let i = 0; i < file_list.length; i++) {
const file = file_list[i];
//launch reading
dataPromises.push(readCSV(file))
}
Promise.all(dataPromises).then(result => {
//this code will be called in future after all readCSV Promises call resolve(..)
for(const events of result){
processCSV(events);
}
})
}