I'm trying to find out and understand how OOM-killer works on the container.
To figuring it out, I've read lots of articles and I found out that OOM-killer kills container based on the oom_score. And oom_score is determined by oom_score_adj and memory usage of that process.
And there're two metrics container_memory_working_set_bytes, container_memory_rss from the cAdvisor for monitoring memory usage of the container.
It seems that RSS memory (container_memory_rss) has impact on oom_score so I can understand that with the container_memory_rss metric, if that metric reached to memory limit, the OOM-killer will kill the process.
But from the articles like below:
The better metric is
container_memory_working_set_bytesas this is what the OOM killer is watching for.
I cannot understand the fact that OOM-killer is watching for container's working set memory. I think I'm not understand the meaning of the working set memory on the container which is 'total usage - inactive file'.
Where can I find the reference? Or could you explain the relationship between working set memory and OOM-kill on the container?
As you already know, container_memory_working_set_bytes is:
the amount of working set memory and it includes recently accessed memory, dirty memory, and kernel memory. Therefore, Working set is (lesser than or equal to) </= "usage".
The container_memory_working_set_bytes is being used for OoM decisions because it excludes cached data (Linux Page Cache) that can be evicted in memory pressure scenarios.
So, if the container_memory_working_set_bytes is increased to the limit, it will lead to oomkill.
You can find the fact that when Linux kernel checking available memory, it calls vm_enough_memory() to find out how many pages are potentially available.
Then when the machine is low on memory, old page frames including cache will be reclaimed but kernel still may find that it was unable free enough pages to satisfy a request. Now it's time to call out_of_memory() to kill the process. To determine the candidate process to be killed it uses oom_score.
So when Working Set bytes reached to limits, it means that kernel cannot find availables pages even after reclaiming old pages including cache so kernel will trigger OOM-killer to kill the process.
You can find more details on the Linux kernel documents: