My code is nearly working well my problem is that only the first CSV data is logged and the lambda function ends.
I guess I need someway to wait for all stream pipes to end.
myCsvList.forEach((myElem) => {
const data = [];
// setup params
const csvFile = s3.getObject(params).createReadStream(); // works fine
csvFile
.pipe(csv())
.on('data', function(entry) {
data.push(entry);
})
.on('end', () => {
console.log(data); // after the log of the first element on myCsvList, the code finishes. It should log all csvFiles from myCsvList
});
});I guess I need a promises or something?
you can change every item to Promise, then use Promise.all():
const getItem = (myElem) => {
const data = [];
// setup params
const csvFile = s3.getObject(params).createReadStream(); // works fine
return new Promise((resolve) => {
csvFile
.pipe(csv())
.on('data', function (entry) {
data.push(entry);
})
.on('end', () => {
resolve(data); // after the log of the first element on myCsvList, the code finishes. It should log all csvFiles from myCsvList
});
});
};
const start = () => {
// all promises have been finished
return Promise.all(myCsvList.map(myElem => getItem(myElem)));
}
myCsvList.map(myElem => getItem(myElem)) can get a promise list, when all promises have been resolved, it will trigger Promise.all().