I'm running into issues because my containers are producing too much log output. This apparently causes the reserved space to run out and evicts the pods with message The node was low on resource: ephemeral-storage. Container XYZ was using 1412Ki, which exceeds its request of 0.
Is there any way to tell kubernetes to just forget about old logs? I would be perfectly fine with saying "10MB of logs per pod are more than enough", but I can't find any place to specify such a thing.
Whether this is a limit in time or size does not matter to me that much, as long as I can keep my pods alive. This is for a development environment, so log retention is not crucial at all.
I'm assuming the container engine is docker in which case, you can configure docker to limit the logs written by your pod(s) container(s).
https://docs.docker.com/config/containers/logging/configure/
This is a Community Wiki answer so feel free to edit it and add any additional details you consider important.
As William has already answered this question, I'm only going to add a few details to make this more complete.
Please take a look at Logging Architecture in the official kubernetes docs. As you can read here "Kubernetes currently is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that."
If your container runtime happens to be docker, you can set up your log rotation policy by editing/creating:
/etc/docker/daemon.json
(on Linux host)
or:
C:\ProgramData\docker\config\daemon.json
(on Windows host)
which may look as follows:
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3",
"labels": "production_status",
"env": "os,customer"
}
}