I launch our spring boot application in docker container on AWS Fargate service, so once the CPU consumption is reached more then 100% the container is stopped Docker OOM-killer with error
Reason: OutOfMemoryError: Container killed due to memory usage
On metrics we can see that CPU becomes more then 100%. It seems after some time of profiling we found CPU consuming code, but my question is, how CPU can be grater than 100%?
Is it some way to say JVM use only 100%?
I remember we had similar issue with memory consumption. I read a lot of articles about cgroups, and the solution was found to specify
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
So when you launch docker with option -m=512 heap size will be 1/4 of mac size. The heap size can also be tuned with option
-XX:MaxRAMFraction=2
which will allocate 1/2 of docker memory for heap. Should I use something similar for CPU? I read article https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits, but it tells that
As of Java SE 8u131, and in JDK 9, the JVM is Docker-aware with respect to Docker CPU limits transparently. That means if -XX:ParalllelGCThreads, or -XX:CICompilerCount are not specified as command line options, the JVM will apply the Docker CPU limit as the number of CPUs the JVM sees on the system. The JVM will then adjust the number of GC threads and JIT compiler threads just like it would as if it were running on a bare metal system with number of CPUs set as the Docker CPU limit.
Docker command is used to start
docker run -d .... -e JAVA_OPTS='-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:+PrintFlagsFinal -XshowSettings:vm' -m=512 -c=256 ...
Java version is used
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
Some additional info on app during start up
VM settings:
Max. Heap Size (Estimated): 123.75M
Ergonomics Machine Class: client
Using VM: OpenJDK 64-Bit Server VM
ParallelGCThreads = 0
CICompilerCount := 2
CICompilerCountPerCPU = true
I found answer to my question. The behaviour to identify number of processors to use was fixed in https://bugs.openjdk.java.net/browse/JDK-8146115
Number of CPUs
Use a combination of number_of_cpus() and cpu_sets() in order to determine how many processors are available to the process and adjust the JVMs os::active_processor_count appropriately. The number_of_cpus() will be calculated based on the cpu_quota() and cpu_period() using this formula: number_of_cpus() = cpu_quota() / cpu_period(). If cpu_shares has been setup for the container, the number_of_cpus() will be calculated based on cpu_shares()/1024. 1024 is the default and standard unit for calculating relative cpu usage in cloud based container management software.
Also add a new VM flag (-XX:ActiveProcessorCount=xx) that allows the number of CPUs to be overridden. This flag will be honored even if UseContainerSupport is not enabled.
So on AWS you generally setup cpu_shares on task definition level. Before jvm fix it was calculated incorrectly.
On java8 version < 191: cpu_shares()/1024 = 256/1024 = was identified as 2
After migration on java8 version > 191: cpu_shares()/1024 = 256/1024 = was identified as 1
The code to test
val version = System.getProperty("java.version")
val runtime = Runtime.getRuntime()
val processors = runtime.availableProcessors()
logger.info("========================== JVM Info ==========================")
logger.info("Java version is: {}", version)
logger.info("Available processors: {}", processors)
The sample output
"Java version is: 1.8.0_212"
"Available processors: 1"
I hope it will help someone, as I can't find the answer anywhere (spring-issues-tracker, AWS support, etc.)