0
189
ViewsAKS log format changed
we recently updated our AKS cluster from 1.17.x to 1.19.x and recognised that the format of our custom application logs in /var/lib/docker/containers changed.
Before the update it looked like this:
Afterwards it looks like this:
I can find some notes in the changelog that kubernetes changed from just text logs to structured logs (for system components) but I don't see how this correlates to how our log format changed.
https://kubernetes.io/docs/concepts/cluster-administration/system-logs/
Is there a chance to still get valid json logs to /var/lib/docker/containers in AKS > 1.19.x?
Background: We send our application logs to Splunk and don't use the Azure stack for log analysis. Our Splunk setup cannot parse that new log format as of now.
1 answers
Answer question0
The format of the logs are defined by the container runtime. It seems before you were parsing logs from docker container runtime, and now it is containerd (https://azure.microsoft.com/en-us/updates/azure-kubernetes-service-aks-support-for-containerd-runtime-is-in-preview/).
Based on the article - you can still choose moby (which is docker) as the container runtime.
To take that also from your shoulders, you should look into using one of those (considering that they will automatically detect the log format and container runtime for you).
Splunk Connect for Kubernetes https://github.com/splunk/splunk-connect-for-kubernetes
Collectord https://www.outcoldsolutions.com