I have a library, published via npm as "by-request", which can, among other things, auto-decompress web content. Part of the code to handle this situation looks like this:
if (!options.dontDecompress || !binary) {
if (contentEncoding === 'gzip' || (options.autoDecompress && /\b(gzip|gzipped|gunzip)\b/.test(contentType))) {
source = zlib.createGunzip();
res.pipe(source);
}
// *** start of temporary new code ***
else if (contentEncoding === 'x-gzip' && options.autoDecompress) {
source = zlib.createGunzip(); // zlib.createUnzip() doesn't work either
res.pipe(source);
}
// *** end of temporary new code ***
else if (contentEncoding === 'deflate' || (options.autoDecompress && /\bdeflate\b/.test(contentType))) {
source = zlib.createInflate();
res.pipe(source);
}
else if (contentEncoding === 'br') {
source = zlib.createBrotliDecompress();
res.pipe(source);
}
else if (contentEncoding && contentEncoding !== 'identity') {
reject(UNSUPPORTED_MEDIA_TYPE);
return;
}
}
The code had been working pretty well until I tried to read a file of astronomical info from here: https://cdsarc.cds.unistra.fr/viz-bin/nph-Cat/txt.gz?VII/118/names.dat
I was hitting the reject(UNSUPPORTED_MEDIA_TYPE) error handler because I hadn't specifically handled the Content-Type of x-gzip. Simply adding a check for x-gzip, however, didn't fix the problem.
zlib is choking on the data, coming back with this error:
Error: incorrect header check
at Zlib.zlibOnError [as onerror] (node:zlib:190:17)
Is there a different decompression library I need? I've searched around, but haven't found a good solution yet. According to this previous Stack Overflow answer: Difference between "x-gzip" and "gzip" for content-encoding
...gzip and x-gzip should be the same. It's not working out that way. On the other hand, no web browser I've tried has any trouble at all getting and displaying the text from the cdsarc.cds.unistra.fr URL.
The following solution is working for me, substituting a shell gzip decompression operation for that provided by zlib and createGunzip(). The only reason I can think of for this fix to work might be that there's something a bit quirky about the zipped data stream provided by the particular web site that was causing the failure, something for which the shell command is tolerant, but zlib is not.
if (!checkedGzipShell) {
checkedGzipShell = true;
hasGzipShell = true;
const gzipProc = spawn('gzip', ['-L']);
await new Promise<void>(resolve => {
gzipProc.once('error', () => { hasGzipShell = false; resolve(); });
gzipProc.stdout.once('end', resolve);
});
}
if (contentEncoding === 'gzip' || contentEncoding === 'x-gzip' ||
(options.autoDecompress && /\b(gzip|gzipped|gunzip)\b/.test(contentType))) {
if (hasGzipShell) {
const gzipProc = spawn('gzip', ['-dc']);
source = gzipProc.stdout;
res.pipe(gzipProc.stdin);
}
else {
source = zlib.createGunzip();
res.pipe(source);
}
}