We have an app build on nodejs (Server A), which is fetching data from another nodejs app(Server B) using REST API's.
Similarly for downloading files Server A make request on Server B and Server B fetches files from AWS S3 and pipes the data back to server A.
This system works perfectly and we are able to download files upto 1GB.
Issue is sometimes Server A crashes with below error:
buffer.js:488
throw new Error('toString failed');
^
Error: toString failed
at Buffer.toString (buffer.js:488:11)
at Request.<anonymous> (/var/www/sky/portal/node_modules/request/request.js:1145:39)
at emitOne (events.js:95:20)
at Request.emit (events.js:182:7)
at IncomingMessage.<anonymous> (/var/www/sky/portal/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:273:16)
at emitNone (events.js:85:20)
at IncomingMessage.emit (events.js:179:7)
at endReadableNT (_stream_readable.js:913:12)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)
Server A is calling server B using below code:
exports.downloadAsset = function(req, res, assetUrl, cb) {
res.cookie('fileDownload', 'true');
request(
config.api.baseUrl + assetUrl
,
function(error, response, body) {
}
).pipe(res);
}
We are not sure from where app is getting crashed. Please suggest.
The Solution I used to sort above issue: I had to modify our flow. We were streaming data from Server B to Server A, but now Server B is sending signed URL to Server A and file is being downloaded directly from S3 using that signed url.