Starting yesterday (Sunday) morning my production app fails to start, with no code changes from my side. It's running Springboot 2.3.4, Liquibase-core 3.8.0 and is hosted on Amazon linux2. Funny thing is there are no exceptions locally, only when deployed.
Here is the relevant stack trace:
Caused by: liquibase.exception.UnexpectedLiquibaseException: java.nio.file.NoSuchFileException: /tmp/agent12302722365010540729.jar
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:129)
at liquibase.servicelocator.ServiceLocator.<init>(ServiceLocator.java:69)
at liquibase.servicelocator.CustomResolverServiceLocator.<init>(CustomResolverServiceLocator.java:16)
at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener$LiquibasePresent.replaceServiceLocator(LiquibaseServiceLocatorApplicationListener.java:55)
at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener.onApplicationEvent(LiquibaseServiceLocatorApplicationListener.java:44)
at org.springframework.boot.liquibase.LiquibaseServiceLocatorApplicationListener.onApplicationEvent(LiquibaseServiceLocatorApplicationListener.java:36)
...
Caused by: java.nio.file.NoSuchFileException: /tmp/agent801508645517312012.jar
at liquibase.resource.ClassLoaderResourceAccessor.getResourcesAsStream(ClassLoaderResourceAccessor.java:53)
at liquibase.servicelocator.ServiceLocator.setResourceAccessor(ServiceLocator.java:115)
I double checked all application related files and env variables and they are all the same. The file in question is in no way related to my app.
Do you have any idea what this file is and why is Liquibase trying to find it all of the sudden?
I had the same problem. On the startup of the amazon linux 2, there is a security patch that is installed.
The package causing the problem is log4j-cve-2021-44228-hotpatch.noarch (you can check that in /var/log/yum.log)
A temporary solution is to uninstall the patch and install another java version.
yum remove log4j-cve-2021-44228-hotpatch.noarch
yum install java-11-openjdk-11.0.12.0.7-0.amzn2.0.2.x86_64
Thanks to @mihristov for the solution.
This is a temporary issue with: java-11-amazon-corretto-headless-11.0.13+8-2.amzn2.aarch64
To fix:
yum remove log4j-cve-2021-44228-hotpatch.noarch
yum install java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.aarch64
Thanks to @Reda and @mihristov for pointing the issue out.
Our actual fix was just to install a specific version of corretto instead of the general package
yum install java-11-amazon-corretto-headless-11.0.13+8-1.amzn2.aarch64
You should only do this temporarily until a fix is out!
For us moving to latest liquibase (4+) resolved this. They use a more standard ServiceLocator in 4+ and that seems to make the difference.
To elaborate a bit more on what's happening here:
AWS has developed a systemd service that, every second, looks for any JVMs running on the system. For all that it finds and supports it connect a java agent that at runtime replaces the log4j JNDI lookup() method with a hardcoded safe string.
Why liquibase fails on this is not clear to me. I do see that the already given answers seem to be too invasive. There's no need to remove/reinstall/update your JVM. You can work around this by:
Stopping the systemd patch service: sudo service log4j-cve-2021-44228-hotpatch status|stop|start
Best solution - create the kill file that, if read by the service, stops applying the patch: sudo touch /etc/log4j-cve-2021-44228-hotpatch.kill