Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

208
Views
Consume Kafka messages from a different container in python

Below is my docker-compose file:

   version: "3"
    services:
      app:
        image: app
        restart: always
        ports:
          - "8001:8081"
        depends_on:
          - kafka
          - zookeeper
          - consumer
        environment:
          - KAFKA_HOST = kafka

  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    ports:
        - "2181:2181"
    hostname: zookeeper   

  kafka:
    image: "wurstmeister/kafka:2.12-2.2.0"
    ports:
      - "9092:9092"
    hostname: kafka
    links: 
     - zookeeper:zookeeper
    environment:
      KAFKA_CREATE_TOPICS: "Topic01:2:2" #TOPIC:PARTITON:REPLICATION
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: "60000"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_LISTENERS: 'PLAINTEXT://:9092'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  consumer:
    image: consumer:latest
    build:
      context: ./consumer
    ports:
      - "8283:8283"

Text stream is generated using container for app image and is able to produce the messages.

It is verified by navigating inside the kafka container:

docker exec -it <CONTAINER ID of Kafka Image> /bin/bash

when I manually run the below script inside the container

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Topic01 --from-beginning

I can see the chunk of text that is being produced.

But when I try to read the same data from the container of the image consumer, It gives me blank,

Method tried:

docker exec inside the consumer container, and ran the below python code:

from kafka import KafkaConsumer
consumer = KafkaConsumer('Topic01', bootstrap_servers='kafka:9092')
for messages in consumer:
        print(messages)

It prints nothing.

Is it because my docker-compose file is wrong or the mistake in the python code?

over 4 years ago · Santiago Trujillo
1 answers
Answer question

0

According to Kafka documentation for listeners:

Listener List - Comma-separated list of URIs we will listen on and the listener names. If the listener name is not a security protocol, listener.security.protocol.map must also be set. Specify hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to bind to default interface. Examples of legal listener lists: PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093

You are setting KAFKA_LISTENERS to PLAINTEXT://:9092, so it binds to default interface, which might not be accessible from the outside of your Kafka container (while at the same time it works fine with console consumer using localhost). Give it a try and specify KAFKA_LISTENERS as PLAINTEXT://0.0.0.0:9092, to check if you consumer starts to consume messages.

over 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!