I have a K3s (v1.20.4+k3s1) cluster with 3 nodes, each with two interfaces. The default interface has a public IP, the second one a 10.190.1.0 address. I installed K3s with and without the -flannel-backend=none option and then deployed flannel via " kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml", previously binding the kube-flannel container to the internal interface via the args "--iface=". In this setup the kube-flannel pods get the node-ip of the internal interface, but I can't reach the pods on the other nodes via ICPM. If I deploy flannel without -iface arg, the kube-flannel pods get an address from the 10.42.0.0 network. Then I can reach the pods of the other hosts, but the traffic will be routed through the public interfaces, which I want to avoid. Does anyone have a tip for me?
The problem was resolved in the comments section but for better visibility I decided to provide an answer.
As we can see in the K3s documentation, K3s uses flannel as the CNI by default:
By default, K3s will run with flannel as the CNI, using VXLAN as the default backend. To change the CNI, refer to the section on configuring a custom CNI.
By default, flannel selects the first interface on a host (look at the flannel documentation), but we can override this behavior with the --flannel-iface flag.
Additionally we can explicitly set IP address to advertise for node using the --node-ip flag.
I've created a simple example to illustrate how it works.
On my host machine I have two network interfaces (ens4 and ens5):
kmaster:~# ip a s | grep -i "UP\|inet"
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
inet 10.156.15.197/32 brd 10.156.15.197 scope global dynamic ens4
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
inet 192.168.0.2/32 brd 192.168.0.2 scope global dynamic ens5
Without setting the --flannel-iface and --node-ip flags, flannel will select the first interface (ens4: 10.156.15.197):
kmaster:~# curl -sfL https://get.k3s.io | sh -
[INFO] Finding release for channel stable
[INFO] Using v1.20.4+k3s1 as release
...
[INFO] systemd: Starting k3s
kmaster:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
kmaster Ready control-plane,master 97s v1.20.4+k3s1 10.156.15.197
But as I mentioned before we are able to override default flannel interface with the --flannel-iface flag:
kmaster:~# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-ip=192.168.0.2 --flannel-iface=ens5" sh -
[INFO] Finding release for channel stable
[INFO] Using v1.20.4+k3s1 as release
...
[INFO] systemd: Starting k3s
kmaster:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
kmaster Ready control-plane,master 64s v1.20.4+k3s1 192.168.0.2