TL;DR - gke 1.20 preemptible nodes cause pods to zombie into Failed/Shutdown
We have been using GKE for a few years with clusters containing a mixture of both stable and preemptible node pools. Recently, since gke v1.20, we have started seeing preempted pods enter into a weird zombie state where they are described as:
Status: Failed
Reason: Shutdown
Message: Node is shutting, evicting pods
When this started occurring we were convinced it was related to our pods failing to properly handle the SIGTERM at preemption. We decided to eliminate our service software as a source of a problem by boiling it down to a simple service that mostly sleeps:
/* eslint-disable no-console */
let exitNow = false
process.on( 'SIGINT', () => {
console.log( 'INT shutting down gracefully' )
exitNow = true
} )
process.on( 'SIGTERM', () => {
console.log( 'TERM shutting down gracefully' )
exitNow = true
} )
const sleep = ( seconds ) => {
return new Promise( ( resolve ) => {
setTimeout( resolve, seconds * 1000 )
} )
}
const Main = async ( cycles = 120, delaySec = 5 ) => {
console.log( `Starting ${cycles}, ${delaySec} second cycles` )
for ( let i = 1; i <= cycles && !exitNow; i++ ) {
console.log( `---> ${i} of ${cycles}` )
await sleep( delaySec ) // eslint-disable-line
}
console.log( '*** Cycle Complete - exiting' )
process.exit( 0 )
}
Main()This code is built into a docker image using the tini init to spawn the pod process running under nodejs (fermium-alpine image). No matter how we shuffle the signal handling it seems the pods never really shutdown cleanly, even though the logs suggest they are.
Another oddity to this is that according to the Kubernetes Pod logs, we see the pod termination start and then gets cancelled:
2021-08-06 17:00:08.000 EDT Stopping container preempt-pod
2021-08-06 17:02:41.000 EDT Cancelling deletion of Pod preempt-pod
We have also tried adding a preStop 15 second delay just to see if that has any effect, but nothing we try seems to matter - the pods become zombies. New replicas are started on the other nodes that are available in the pool, so it always maintains the minimum number of successfully running pods on the system.
We are also testing the preemption cycle using a sim maintenance event:
gcloud compute instances simulate-maintenance-event node-id
After poking around various posts I finally relented to running a cronjob every 9 minutes to avoid the alertManager trigger that occurs after pods have been stuck in shutdown for 10+ minutes. This still feels like a hack to me, but it works, and it forced me to dig in to k8s cronjob and RBAC.
This post started me on the path: How to remove Kubernetes 'shutdown' pods
And the resultant cronjob spec:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-accessor-role
namespace: default
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "delete", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: pod-access
namespace: default
subjects:
- kind: ServiceAccount
name: cronjob-sa
namespace: default
roleRef:
kind: Role
name: pod-accessor-role
apiGroup: ""
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: cronjob-sa
namespace: default
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: cron-zombie-killer
namespace: default
spec:
schedule: "*/9 * * * *"
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
template:
metadata:
name: cron-zombie-killer
namespace: default
spec:
serviceAccountName: cronjob-sa
restartPolicy: Never
containers:
- name: cron-zombie-killer
imagePullPolicy: IfNotPresent
image: bitnami/kubectl
command:
- "/bin/sh"
args:
- "-c"
- "kubectl get pods -n default --field-selector='status.phase==Failed' -o name | xargs kubectl delete -n default 2> /dev/null"
status: {}
Note that the redirect of stderr to /dev/null is to simply avoid the error output from kubectl delete when the kubectl get doesn't find any pods in the failed state.
Update added missing "delete" verb from the role, and added the missing RoleBinding
Update added imagePullPolicy
Starting with GKE 1.20.5 and later, the kubelet graceful node shutdown feature is enabled preemptible nodes. From the note on the feature page:
When pods were evicted during the graceful node shutdown, they are marked as failed. Running kubectl get pods shows the status of the the evicted pods as Shutdown. And kubectl describe pod indicates that the pod was evicted because of node shutdown:
Status: Failed Reason: Shutdown Message: Node is shutting, evicting pods Failed pod objects will be preserved until explicitly deleted or cleaned up by the GC. This is a change of behavior compared to abrupt node termination.
These pods should eventually be garbage collected, although I'm not sure of the threshold value.