I need to create a zip file with around 1GB of data (but that amount will grow in the future). Initially I built this with AdmZip, but since our VM doesn't have a whole lot of memory, (nor does it need it, except for when it's creating this zip file, so I don't want to scale up), it runs out of memory and crashes.
I then rebuilt the process in order to use command line zip, but after getting that all set up and releasing to the server, I had to find out the hard way that that command is not available on an Azure WebApp.
So now I need to rebuild it again. Can anyone recommend me the right node library/libraries to do zipping and unzipping on disk instead of in memory?
(the zip file format is not a hard requirement, so if any other file formats are better for this, I'm open to suggestions!)
I'd have a look at Archiver JS, this is pretty powerful, allowing a streaming interface to .zip files, for example:
const fs = require('fs');
const archiver = require('archiver');
const zipFile = archiver('zip', { zlib: { level: 9 }});
zipFile.on('warning', error => {
console.log('warning:', error)
});
zipFile.on('error', error => {
console.error('error occurred :', error)
});
const writeStream = fs.createWriteStream(__dirname + '/test.zip');
zipFile.pipe(writeStream);
// Append test file
zipFile.append(fs.createReadStream('file1.txt'), { name: 'file_1.txt' });
// Append another file...
zipFile.append(fs.createReadStream('file2.txt'), { name: 'file_2.txt' });
zipFile.finalize();
Full docs are here: https://www.archiverjs.com/docs/archiver