Below is my docker-compose file:
version: "3"
services:
app:
image: app
restart: always
ports:
- "8001:8081"
depends_on:
- kafka
- zookeeper
- consumer
environment:
- KAFKA_HOST = kafka
zookeeper:
image: "wurstmeister/zookeeper:latest"
ports:
- "2181:2181"
hostname: zookeeper
kafka:
image: "wurstmeister/kafka:2.12-2.2.0"
ports:
- "9092:9092"
hostname: kafka
links:
- zookeeper:zookeeper
environment:
KAFKA_CREATE_TOPICS: "Topic01:2:2" #TOPIC:PARTITON:REPLICATION
KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: "60000"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_LISTENERS: 'PLAINTEXT://:9092'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
consumer:
image: consumer:latest
build:
context: ./consumer
ports:
- "8283:8283"
Text stream is generated using container for app image and is able to produce the messages.
It is verified by navigating inside the kafka container:
docker exec -it <CONTAINER ID of Kafka Image> /bin/bash
when I manually run the below script inside the container
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Topic01 --from-beginning
I can see the chunk of text that is being produced.
But when I try to read the same data from the container of the image consumer, It gives me blank,
Method tried:
docker exec inside the consumer container, and ran the below python code:
from kafka import KafkaConsumer
consumer = KafkaConsumer('Topic01', bootstrap_servers='kafka:9092')
for messages in consumer:
print(messages)
It prints nothing.
Is it because my docker-compose file is wrong or the mistake in the python code?
According to Kafka documentation for listeners:
Listener List - Comma-separated list of URIs we will listen on and the listener names. If the listener name is not a security protocol, listener.security.protocol.map must also be set. Specify hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to bind to default interface. Examples of legal listener lists: PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093
You are setting KAFKA_LISTENERS to PLAINTEXT://:9092, so it binds to default interface, which might not be accessible from the outside of your Kafka container (while at the same time it works fine with console consumer using localhost). Give it a try and specify KAFKA_LISTENERS as PLAINTEXT://0.0.0.0:9092, to check if you consumer starts to consume messages.