I am trying to stream process a large file (few GBs) with the following code, but it outputs only the very first line of the uncompressed file.
const instream = fs.createReadStream('logs.gz');
const unzip = zlib.createGunzip();
const outstream = process.stdout;
instream.pipe(unzip).pipe(outstream);
// <first line of file>>
I have not compressed the file, but from the API documentation of the log provider, it should be gunzipped.
If i decompress it manually using 7zip, it decompresses fine. If I then compress it again with 7zip, the code works as expected on this 7zip processed file.
If I hook on error|finish|data event of the unzip stream, no errors, data prints the one line and then the finish. Any clues what might be wrong?
Tried createUnzip too.
Node 5.5.0 on Win10, same behaviour on MacOSX node v5.6.0.
It's related to the Node version you're using: prior to Node v5.9.0, zlib didn't properly handle files that contained multiple gzipped blocks. With those files, only the first block would be decompressed.
To replicate this issue, I used the following to create a file that contains multiple blocks:
$ echo foo | gzip > logs.gz
$ echo bar | gzip >> logs.gz
My guess is that your logfile was created in a similar manner, where each log line is compressed separately and appended to the compressed logfile.
Probably the only reasonable solution would be to upgrade to at least v5.9.0 (or v6, even).