3 nodes Kubernetes cluster created using kubeadm v1.19.9. The nodes are VMs (using KVM hypervisor on Ubuntu 20.04).
The usage of this Kubernetes cluster is for development and exercises on Kubernetes. I'd like to stop and restart the cluster where it was left off. In the same fashion as the stop and start commands available with minikube or microk8s.
EDIT: clarify the question to avoid suggested duplicated posts. I am looking for an elegant solution to stop and restart the same cluster. NOT to destroy / reset / uninstall the cluster.
I couldn't find a simple solution from various web searches. There are solutions which suggested to tear down the cluster which is not my use case here. An answer dating 3 years ago, proper shutdown of a kubernetes cluster, is closer to what I want but this sounds quite complicate. Another solution How to Setup & Recover a Self-hosted Kubeadm Kubernetes Cluster After Reboot doesn't explain well enough about the principle used.
I hope there is a simpler solution now.
EDIT (2021-04-11): Kubernetes 1.21 release notes:
Kubelet Graceful Node Shutdown feature graduates to Beta and enabled by default.
kubernetes/enhancements Graceful node shutdown #2000
Enhancement target (which target equals to which milestone):
- Alpha release target (1.20)
- Beta release target (1.21)
- Stable release target (1.23)
To summarize:
k8s should be able to handle shutdowns. What may not be able to handle it are the applications/containers that you run - just make sure containers start on their own and don't require manual intervention and you should be fine.
I mentioned in comments about flushing etcd data to disk but (after some research) this should not be neccessary since etcd does it itself and implements strong consistency model to make sure it doesn't lose the data. But this doesn't mean you should not be doing your backups - it's better to have a backup and don't ever use it then don't have one when needed.
The solution mentioned in How to Setup & Recover a Self-hosted Kubeadm Kubernetes Cluster After Reboot is relevant only if you use SelfHosting.
Also (for convenience) make sure that all configs are persisting between reboots e.g. swap partition should be disabled and if you only run swapoff -a it won't persist after reboot - it's much better to make changes in fstab so that when rebooted you don't have to disable anything again manually.
Here are some links: