I am getting a few 502 errors in nginx:

2020/05/05 20:40:05 [error] 13425#0: *1360447 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.31.12.13, server: , request: "POST /app/functionHTTP/1.1", upstream: "http://127.0.0.1:8080/app/function", host: "example.com"

Architecture:

client - load balancer - instances (each with nginx proxy for NodeJS http server)

Timeout values:

• client:

  • request timeout: 60s

• load balancer:

  • keepalive timeout: 60s

• nginx:

  • http keepalive_timeout: 65s
  • upstream keepalive_timeout: 60s
  • upstream keepalive: 256 connections
  • proxy_connect_timeout: 60s
  • proxy_read_timeout: 60s
  • proxy_send_timeout: 60s

• server (NodeJS):

  • request timeout: 50s
  • keepAliveTimeout: 65s
  • headersTimeout: 70s

The 502 errors occur rarely (~0.01% of requests) but consistently.

What I've checked so far:

• Limit of file descriptors not reached
• Limit of nginx connections not reached
• Limit of TCP ports not reached
• Limit of CPU, memory, disk I/O not reached

It looks much like a race condition where the NodeJS server closed the connection and nginx is assuming the connection still exists.

What would be proper nginx timeout values in the current setting?

(I am unsure whether this isn't rather a question for ServerFault?)