I am trying to run a traefik container on my docker swarm cluster. Because we are using TLS encrypted communication, I want the traefik dashboard to be available via https.
In my browser, I try to access traefik via the docker swarm manager hostname via https://my.docker.manager and therefor I mounted my hosts certificate and key into the traefik service.
When I try to open https://my.docker.manager in my browser, I get a timeout.
When I try to curl https://my.docker.manager directly on the host (my.docker.manager) I get HTTP code 403 as response
My traefik config:
debug=true
logLevel = "DEBUG"
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/etc/traefik/certs/my.docker.manager.crt"
keyFile = "/etc/traefik/certs/my.docker.manager.key"
[entryPoints.https.tls.defaultCertificate]
certFile = "/etc/traefik/certs/my.docker.manager.crt"
keyFile = "/etc/traefik/certs/my.docker.manager.key"
[api]
address = ":8080"
[docker]
watch = true
swarmMode = true
My traefik compose file:
version: "3.7"
services:
traefik:
image: traefik
ports:
- 80:80
- 443:443
networks:
- devops-net
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /mnt/docker-data/secrets/certs/:/etc/traefik/certs/
configs:
- source: traefik.conf
target: /etc/traefik/traefik.toml
deploy:
placement:
constraints:
- node.role == manager
labels:
- "traefik.docker.network=devops-net"
- "traefik.frontend.rule=Host:my.docker.manager"
- "traefik.port=8080"
networks:
devops-net:
driver: overlay
external: true
configs:
traefik.conf:
external: true
As described in this article (https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-16-04), I expected to see the traefik dashboard, when I call https://my.docker.manager in my browser. But I only get a timeout. When using curl https://my.docker.manager I get HTTP code 403. I followed the mentioned article except two differences:
1) I did not configure credentials
2) I used my hosts own certificates instead of letsencrypt
In the meantime I found the reason for my problem (not sure what is the best solution). For the case someone is interested, I will try to explain it.
I have a Swarm of three nodes, in the network prod.company.de
My client is in another network intranet.comnpany.de
My swarm manager is adressed by docker-manager.prod.company.de. On this host, I have deployed the traefik service, that I want to access via https://docker-manager.prod.company.de (This is port 443 and because of traefik config forwarded to the traefik dashboard on 8080 inside the container).
If I track my network traffic, I can see that the https request (https://docker-manager.prod.company.de) from my client browser reaches the server and than the traffic is forwarded to the docker_gwbridge address 17.18.0.2. But the answer does not find the way back to my client because of the docker NAT configuration.
iptables -t nat -L -v
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
8 504 MASQUERADE all -- any docker_gwbridge anywhere anywhere ADDRTYPE match src-type LOCAL
0 0 MASQUERADE all -- any !docker0 172.17.0.0/16 anywhere
0 0 MASQUERADE all -- any !docker_gwbridge 172.18.0.0/16 anywhere
MASQERADE says that the source IP of the request should be replaced with the bridge IP (in my case 172.18.0.1) so that answers are routed back to this IP. IN the case above, the rule 8 504 MASQUERADE all -- any docker_gwbridge anywhere anywhere would do this, but it is limited to requests from LOCAL by ADDRTYPE match src-type LOCAL, which means that using a browser on the docker host would work, but my connections from client do not work, because the answer will not find the way back to my client address.
Currently, I added one more NAT rule to my iptables:
iptables -t nat -A POSTROUTING -o docker_gwbridge -j MASQUERADE
which results in
1 52 MASQUERADE all -- any docker_gwbridge anywhere anywhere
After that, I can see the traefik Dashboard, when opening https://docker-manager.prod.company.de in my browser on the client.
But I do not understand, why I have to do this, because I found nothing about that in any documentation and I don't think that my usecase is really rare. Thats why I would be happy, if someone could have a look at this post and maybe check if I did some other thing wrong or explain to my, why I have to do such things to get a standard usecase working.
Kind regards