Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

209
Visualizações
Consume Kafka messages from a different container in python

Below is my docker-compose file:

   version: "3"
    services:
      app:
        image: app
        restart: always
        ports:
          - "8001:8081"
        depends_on:
          - kafka
          - zookeeper
          - consumer
        environment:
          - KAFKA_HOST = kafka

  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    ports:
        - "2181:2181"
    hostname: zookeeper   

  kafka:
    image: "wurstmeister/kafka:2.12-2.2.0"
    ports:
      - "9092:9092"
    hostname: kafka
    links: 
     - zookeeper:zookeeper
    environment:
      KAFKA_CREATE_TOPICS: "Topic01:2:2" #TOPIC:PARTITON:REPLICATION
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: "60000"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_LISTENERS: 'PLAINTEXT://:9092'
      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  consumer:
    image: consumer:latest
    build:
      context: ./consumer
    ports:
      - "8283:8283"

Text stream is generated using container for app image and is able to produce the messages.

It is verified by navigating inside the kafka container:

docker exec -it <CONTAINER ID of Kafka Image> /bin/bash

when I manually run the below script inside the container

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic Topic01 --from-beginning

I can see the chunk of text that is being produced.

But when I try to read the same data from the container of the image consumer, It gives me blank,

Method tried:

docker exec inside the consumer container, and ran the below python code:

from kafka import KafkaConsumer
consumer = KafkaConsumer('Topic01', bootstrap_servers='kafka:9092')
for messages in consumer:
        print(messages)

It prints nothing.

Is it because my docker-compose file is wrong or the mistake in the python code?

over 4 years ago · Santiago Trujillo
1 Respostas
Responde à pergunta

0

According to Kafka documentation for listeners:

Listener List - Comma-separated list of URIs we will listen on and the listener names. If the listener name is not a security protocol, listener.security.protocol.map must also be set. Specify hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to bind to default interface. Examples of legal listener lists: PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093

You are setting KAFKA_LISTENERS to PLAINTEXT://:9092, so it binds to default interface, which might not be accessible from the outside of your Kafka container (while at the same time it works fine with console consumer using localhost). Give it a try and specify KAFKA_LISTENERS as PLAINTEXT://0.0.0.0:9092, to check if you consumer starts to consume messages.

over 4 years ago · Santiago Trujillo Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda