I am running Apache Kafka inside a docker network. The container of the Kafka broker is called docker_kafka. What I want is to allow connections from Kafka clients from another docker container as well as from docker host. Port 9092 of the docker_kafka has been mapped to the docker host.
I have been trying KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS environment variables. For example when I use KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092, then another docker kafka client cannot connect to kafka by PLAINTEXT://docker_kafka:9092 and vice versa. It does not allow multiple address with same port number and protocol.
Is there a way to allow both type of connections?
It is possible to configure multiple listeners in Kafka. An option listener.security.protocol.map allows to use same security protocol multiple times in the listeners config.
Using environment variables:
1.Define the mappings to as many listener aliases as necessary:
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER1:PLAINTEXT,LISTENER2:PLAINTEXT
2.Then, configure the listeners using the aliases. Sure enough, for this to work each listener needs to listen on its own host+port:
KAFKA_LISTENERS: LISTENER1://0.0.0.0:9092,LISTENER2://0.0.0.0:9093
3.Configure according advertised listeners. For ex.:
KAFKA_ADVERTISED_LISTENERS: LISTENER1://hostname1:9092,LISTENER2://hostname2:9093
4.Finally, for the cluster to work it needs to know which listener to use for the communication between the nodes:
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER1
Note that I only was able to make this configuration work if the listener is declared on the broadcast address (0.0.0.0). I attempted to declare the listener to listen on a fixed host name, but this rendered my configuration inoperable (got connection refused errors). I would gladly appreciate a comment explaining this part.
I'm not the author of this idea (and not taking the credit for it). I've found this solution in this blog post: https://www.kaaproject.org/kafka-docker/
Kafka configuration documentation: https://kafka.apache.org/documentation/#configuration (search for listener.security.protocol.map option).