I am using nginx as a reverse proxy to prometheus UI to add authentication to prometheus UI. Both Prometheus and Nginx are deployed as separate kubernetes pods and communicate with each other via kubernetes services.

Sometimes I see error messages like this when I receive 502 gateway error displayed when tried to launch Prometheus UI proxied via Nginx.

2021/08/09 15:17:20 [error] 22#22: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.105.0, server: nginx-prometheus-proxy, request: "GET / HTTP/1.1", upstream: "http://192.168.43.215:8080/", host: "k8s-master01.example.com:30856"
192.168.105.0 - nanda [09/Aug/2021:15:17:20 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0" "-" 
2021/08/09 15:17:22 [error] 22#22: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.105.0, server: nginx-prometheus-proxy, request: "GET / HTTP/1.1", upstream: "http://192.168.43.215:8080/", host: "k8s-master01.example.com:30856" 192.168.105.0 - nanda [09/Aug/2021:15:17:22 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0" "-"

But more often it fails with below error messages in Nginx logs.

2021/08/09 15:42:56 [error] 22#22: *22 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.105.0, server: nginx-prometheus-proxy, request: "GET / HTTP/1.1", upstream: "http://192.168.43.215:8080/", host: "k8s-master01.example.com:30856"

Tried many solutions suggested but still the 502 bad gateway issue persists. Any solutions are appreciated