I know the this question is asked multiple times and not related to aws.
2020/07/29 10:23:17 [error] 6#6: *37749 connect() failed (113: Host is unreachable) while connecting to upstream, client:
I am facing this issue while I have deployed nginx in aws cloud.
localtion configuration
location /test {
proxy_pass http://test-service;
proxy_set_header HOST $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
Upstream section like below
upstream test-service {
server test-service.internal:38102;
keepalive 10;
}
Here test-service.internal is my route53 hosted zone entry and it is pointing to some internal application load balancer of aws.
When I deploy/restart nginx server, it works well but after few days (around two/three days) it will hang in proxy pass only. When I load html content, it works perfectly but proxy pass call stuck.
Any solution would be helpful?
Thanks.
After long debugging, we found that nginx will cache test-service.internal ips. And aws will chang it's internal load balancer's ips.
So nginx cached ips are no more exist. so we need to provide new ips.
Solution:
nginx has provided resolver directive
location /test {
resolver 10.0.0.2 127.0.0.1 valid=30s;
set $backend_servers test-service.internal;
proxy_pass http://$backend_servers:38102;
#proxy_pass http://test-service;
proxy_set_header HOST $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
We have changed two things.
Added resolver.
Removed upstream (resolver is not supported in nginx. nginx-plus support the upstream with resolver)
resolver 10.0.0.2 127.0.0.1 valid=30s;
set $backend_servers test-service.internal;
proxy_pass http://$backend_servers:38102;
Now we are using aws dns server 10.0.0.2 to resolve test-service.internal after every 30s