Found out that you can create the Jetty Server instance yourself and configure it. In Kotlin:

    val port = Integer.parseInt(System.getProperty("PORT", "8080"))
    val jettyServer = JettyServerUtil.defaultServer()
    jettyServer.apply {
        connectors = arrayOf(ServerConnector(jettyServer).apply {
            this.host = System.getProperty("HOST", "0.0.0.0")
            this.port = port
        })
    }
    val app = Javalin.create()
            .port(port)
            .server { jettyServer }
            .start()

Here's how to do the same in Java:

int port = Integer.parseInt(System.getProperty("PORT", "8080"))
org.eclipse.jetty.server.Server server = new org.eclipse.jetty.server.Server();
ServerConnector connector = new ServerConnector(server);
connector.setHost(System.getProperty("HOST", "0.0.0.0"));
connector.setPort(port);
server.setConnectors(new ServerConnector[] { connector });

Javalin app = Javalin.create(config -> {
    config.server(() -> server);
}).start(port);

Reference: https://github.com/tipsy/javalin/issues/138