I have 2 S3 buckets. Bucket A is where a zip file gets uploaded to and bucket B contains the extracted data from the latest zip that was uploaded to bucket A. I have a Lambda that triggers upon a object create event from bucket A, it then extracts the zip as a stream and uploads to bucket B. Here is the code:
const aws = require('aws-sdk');
const unzipper = require('unzipper');
const s3Client = new aws.S3({ apiVersion: '2006-03-01' });
exports.handler = async (event, context) => {
const sourceBucket = event.Records[0].s3.bucket.name;
const zip = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
const s3Params = {
Bucket: sourceBucket,
Key: zip
};
const zip = s3Client.getObject(s3Params).createReadStream().pipe(unzipper.Parse({ forceStream: true }));
const promises = [];
for await (const file of zip) {
const type = file.type;
if (type === 'File') {
const fileName = file.path;
const params = {
Bucket: process.env.DEST_BUCKET,
Key: `${fileName}`,
Body: file,
};
promises.push(s3Client.upload(params).promise());
} else {
file.autodrain();
}
}
await Promise.all(promises);
};
The problem is the zip is about 200MB with over 10,000 files and folders in it. The lambda, even set at 900 seconds, is timing out. Is there a more efficient way to write the above code to utilize something like parallel streams or rewriting of the for-loop?