I have a function unzip() on my local Node.js server that downloads a .json.gz file from an AWS CloudFront URL and unzips it to .json, when the server is run with node app.js.
The posts from that json file are loaded before responding to routes -
app.all('*', function(req, res, next){
fs.readFile('s3posts.json', function(err, data){
res.locals.posts = JSON.parse(data);
next();
});
});
The problem is that the file is downloaded and unzipped when the server starts running, and serves that same data to the user as long as it's running. The file is not re-downloaded unless the server is restarted. So, any changes to that data is not reflected until the restart.
I've tried calling the unzip() function inside of the above mentioned app.all() route before the fs.readFile() function that reads from the json file, but that throws an error when I request any page -
undefined:1
undefined
^
SyntaxError: Unexpected token u in JSON at position 0
at JSON.parse (<anonymous>)
at ReadFileContext.callback (/Users/Anish/Workspace/NodeJS/unzipper/app.js:48:27)
at FSReqWrap.readFileAfterOpen [as oncomplete] (fs.js:359:13)
When I look in the directory, I see that it downloaded the file but failed to unzip it. Even the downloaded file is corrupt and can't be unzipped manually.
NOTE - The file downloads, unzips and gets served fine if it's placed outside of app.all().
How can I download and unzip the file on every request from the user?
I recommend reading about middlewares in https://expressjs.com/en/guide/using-middleware.html
app.use(function(request)... should help you deal with all request regardless of the routing.
I was able to use the request module to do this -
app.all('*', function(req, res, next) {
request({
method: 'GET',
uri: 'http://post.s3post.cf/s3posts.json.gz',
gzip: true
}, function(error, response, body) {
res.locals.posts = JSON.parse(body);
next();
});
});
This also eliminated the need for me to have a custom unzip() function that downloaded and unzipped the file.
You should try this,
app.all('*', function(req, res, next){
fs.readFile('http://post.s3post.cf/s3posts.json.gz', function(err, data){
res.locals.posts = JSON.parse(data);
next();
});
});