We are planning to migrate our NodeJS platform from plain EC2 to ElasticBeanstalk. During these process, after some struggles, we have deployed our app and able to access and perform actions. However, for some requests, we received 502 error.
After checking the logs we found below;
2020/03/16 06:12:09 [error] 3009#0: *119488 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xx.xx.xxx, server: , request: "POST /www_auth/register HTTP/1.1", upstream: "http://127.0.0.1:8081/register", host: "****.us-east-2.elasticbeanstalk.com"
It occurs in randomly and I don't have any clue. I feel some configuration level changes I missed/need to add with nginx.
If you have any steps/suggestions to solve this, appreciate it!
AWS Elastic Load Balancer pre-connects to backend servers, and it can cause a race condition where ELB thinks a connection is open, but the Node.js backend has already closed it, due to the server.keepAliveTimeout of 5 seconds of idle inactivity, the default value in Node.js 8.x and newer.
Disable server.keepAliveTimeout and server.headersTimeout to work around this issue, or set these timeouts to a ms value larger than the AWS ELB's Idle Timeout value.
const app = express();
// Set up the app...
const server = app.listen(8080);
// Disable both timeouts
server.keepAliveTimeout = 0;
server.headersTimeout = 0;
Credit for this solution goes to Shuhei Kagawa:
https://shuheikagawa.com/blog/2019/04/25/keep-alive-timeout/