I'm testing the new feature graceful shutdown in Spring Boot 2.3 (server.shutdown=graceful and Tomcat Web server) with Kubernetes terminationGracePeriodSeconds field.
When the graceful shutdown phase starts, new HTTP requests are rejected as expected and in this time the current requests should be elaborated until configurable timeout (spring.lifecycle.timeout-per-shutdown-phase).
The strange behavior is the Spring boot actuator liveness result because, in this scenario, the liveness endpoint is unreachable.
So kubelet cannot know if the microservice is still alive during the shutdown or stuck for other things. Since the K8s liveness probe doesn't rely on terminationGracePeriodSeconds field, the POD will restart according to its K8s liveness configuration and Spring boot microservice cannot perform the graceful shutdown due to brutal restart.
Am I missing something? How could I manage this scenario?
In meanwhile, I've adopted another solution, I'm invoking the liveness probe of spring boot via JMX using RMI protocol. In this case, the liveness gives an "UP" response during the graceful shutdown.
To do this:
enable JMX setting the property spring.jmx.enabled to true and the liveness invoke JMX Health indicator using a JMX client.