Is there a place where the Linux kernel passively logs SIGKILL (kill -9) shutdown requests?
I have a JVM running that is arbitrarily being shut down and I suspect that, based on the evidence available, is being shut down by a stray process that is somehow issuing a shutdown of the JVM process. I have robust logging in place but in order to confirm my suspicion, I'd have to turn up the logging level to overwhelming levels.
I've researched exhaustively through /var/log and can't seem to find any place that might capture and log these SIGKILL events. Any ideas where I might find these events, if they exist?
Option 1:
If your kernel has ftrace support (very likely) try the killsnoop tool from Brendan Gregg's perf-tools:
wget https://raw.githubusercontent.com/brendangregg/perf-tools/master/killsnoop
chmod +x killsnoop
sudo ./killsnoop -s
More usage examples in the killsnoop_example.txt file.
Option 2: (passive capture)
If your kernel has no ftrace support you can use the kernel-siglog kernel module from https://github.com/nfedera/kernel-siglog :
git clone https://github.com/nfedera/kernel-siglog.git
cd kernel-siglog/
make
sudo insmod siglog.ko
Once inserted the siglog kernel module will record the last 10,000 signals in /proc/siglog
I had a similar issue and found the culprit using this kernel module. I had it inserted on a customer's server for some weeks and when the service was killed I logged in, did a cat /proc/siglog and found that my service was killed by a customer's own buggy watchdog script.