I have a stack with 10 application containers with frontend nginx . While deploying new application container images in the stack i see ~1-2% failed requests if i run a apachebenchmark or gatling test against it . Around 2 requests fails if i send ~100 req/sec during deployment .
While going through the logs it seems like the difference is just ~1-2ms for the failure . Nginx sends the request & swarm take the application container down & nginx gives 502 error .
Error log (see the response time - 0.002) -
23 Mar 2017 11:07:35.047 192.168.50.68 - - [23/Mar/2017:05:37:34 +0000] "GET http://dev-hapdocker-01/feeds HTTP/1.1" 502 173 "-" "-" 0.002- Context
I suspect that this can be an issue if your machine is bit on slower side .
Let me know if i am missing something . Just curious to know how is the design made for swarm mode during rolling updates or deployments, do they say that it's zero downtime? (I have enabled all the healthchecks) .
We do this with DotNet Core, but it means a lot of work on both ends (giggity). You need to make sure that your image is ready to launch the moment it's instantiated. In a compiled language, you'll want to make sure all the build operations are done before the container is instantiated, IE in the build stage.
This means that whenever the service updates the image, it's basically running at that point in time. The next thing you need to look into is adding the update-delay flag to your service update statement.
If you do docker service update --image new/image:version --update-delay 10s,
The service will rebuild one of the containers on the service, but will wait for 10 seconds before moving to the next one. It takes some planning, and I would always still have an application load balancer in front of the swarm hosts, but we manage it without any issues.