I've been trying to implement a new communication flow in a working production ready NGINX server. The server is running inside a DaemonSet in kubernetes with hostNetwork true. This new flow will forward de income UDP connections from the clients to one backend preserving the original port. I've tried with this config (inside a stream block):
server {
listen 12000-12100 udp;
proxy_pass backend_name:$server_port;
proxy_bind $remote_addr:$remote_port transparent;
proxy_responses 0;
resolver 1.2.2.10;
}
When the UDP packet come to the nginx, the log output is:
2021/07/01 06:19:55 [alert] 30#30: *24650 sendmsg() failed (1: Operation not permitted) while proxying and sending to upstream, udp client: 172.21.2.114, server: 0.0.0.0:12037, upstream: "1.2.123.101:12037", bytes from/to client:1946/0, bytes from/to upstream:0/0
2021/07/01 06:19:56 [alert] 30#30: *24652 sendmsg() failed (1: Operation not permitted) while proxying and sending to upstream, udp client: 172.21.2.114, server: 0.0.0.0:12037, upstream: "1.2.123.101:12037", bytes from/to client:973/0, bytes from/to upstream:0/0
2021/07/01 06:19:58 [alert] 30#30: *24654 sendmsg() failed (1: Operation not permitted) while proxying and sending to upstream, udp client: 172.21.2.114, server: 0.0.0.0:12037, upstream: "1.2.123.101:12037", bytes from/to client:973/0, bytes from/to upstream:0/0
2021/07/01 06:20:02 [alert] 30#30: *24656 sendmsg() failed (1: Operation not permitted) while proxying and sending to upstream, udp client: 172.21.2.114, server: 0.0.0.0:12037, upstream: "1.2.123.101:12037", bytes from/to client:973/0, bytes from/to upstream:0/0
2021/07/01 06:20:10 [alert] 30#30: *24658 sendmsg() failed (1: Operation not permitted) while proxying and sending to upstream, udp client: 172.21.2.114, server: 0.0.0.0:12037, upstream: "1.2.123.101:12037", bytes from/to client:973/0, bytes from/to upstream:0/0
I'm using user root and worker_processes auto in the main config file.
I have read this guide to help me with no results: [https://www.nginx.com/blog/ip-transparency-direct-server-return-nginx-plus-transparent-proxy/][1]
I can't find any issue like this happened to somebody else, the message "while proxying and sending to upstream" have no coincidences in the web.
Somebody can help me to address what is happening?
Finally, I understood that the problem was related with the OS and not only with Nginx. When the OS try to open the socket with the sendmsg function, the OS firewall (iptables) deny this operation. iptables had a reject rule for the upstream servers. This rule was created by the kubelet service because the destination pod endpoints were not reachable. Getting the pods configured in the destination port, made them available;thus erase the iptables reject rules and the traffic was properly forwarded.