While running the scala compiler using the JVM, I have noticed that if enough (8) instances of the JVM are spawned concurrently, some of the JVMs will exit with EMFILE. These invocations all succeed when run in sequence. I have also strace'd the jobs and I can see that they only have 64 out of 1024 file descriptors open when their open() fails with EMFILE. This also corresponds to what I can see when I inspect /proc/xyz/fd while they run.

The only cause of EMFILE I can find in the documentation is that the rlimit 'NOFILE' is too low. However, the soft limit is set to 1024 (the hard limit is 1048576). Even if you add up all the file descriptors of the 8 invocations, 8*64 < 1024. Furthermore, it is my understanding (confirmed with a small C test program) that rlimit should apply to processes independently.

The JVM is heavily multi-threaded, if that is somehow relevant. The process is being run from an ext4 LV from a locally attached PV. The OS is Ubuntu 16.04, though I have had a similar failure on RHEL8. The system has 128GiB of RAM and had no other competing processes.

Also, bizarrely, if I increase the soft ulimit to 102400 in the shell script which launches the jobs, the 8 concurrent jobs will complete successfully.

What other conditions can lead the kernel to return EMFILE for open?

For reference, here is the trace of the open/close calls at the point of failure:

11665 open("<real filename censored for stackoverflow>", O_RDONLY) = 63
12659 open("/sys/fs/cgroup/memory/user.slice/memory.limit_in_bytes", O_RDONLY) = 64
11665 open("<real filename censored for stackoverflow>", O_RDONLY <unfinished ...>
12659 close(64)                         = 0
11665 <... open resumed> )              = -1 EMFILE (Too many open files)
12659 open("/sys/fs/cgroup/memory/user.slice/memory.stat", O_RDONLY) = 64
12659 close(64)                         = 0
11790 open("/sys/fs/cgroup/memory/user.slice/memory.limit_in_bytes", O_RDONLY <unfinished ...>
12658 open("/sys/fs/cgroup/memory/user.slice/memory.limit_in_bytes", O_RDONLY <unfinished ...>
11796 open("/sys/fs/cgroup/memory/user.slice/memory.limit_in_bytes", O_RDONLY <unfinished ...>
12658 <... open resumed> )              = 65
11796 <... open resumed> )              = 66

I've just learned that the JVM is manipulating the ulimit on its own anyway! It immediately maximizes the soft limit upon startup:

3059  getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024*1024}) = 0
3059  setrlimit(RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0

I still haven't figured why this happens. I've given up for now. Just limiting it to at most 6 builds at once.