Hola, actualmente estoy configurando Kafka con Docker. Logré configurar Zookeeper y Kafka con la imagen confluente publicada, consulte el siguiente archivo docker-compose:
version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:3.2.0 container_name: zookeeper hostname: zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 restart: always kafka: image: confluentinc/cp-kafka:3.2.0 hostname: kafka container_name: kafka depends_on: - zookeeper ports: - '9092:9092' environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.99.100:9092 LISTENERS: PLAINTEXT://0.0.0.0:9092 restart: always kafka-rest: image: confluentinc/cp-kafka-rest:3.2.0 container_name: kafka-rest depends_on: - kafka ports: - '8082:8082' environment: KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_REST_LISTENERS: http://kafka-rest:8082 KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081 KAFKA_REST_HOST_NAME: kafka-rest restart: always schema-registry: image: confluentinc/cp-schema-registry:3.2.0 container_name: schema-registry depends_on: - kafka ports: - '8081' environment: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181' SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081 restart: always connect: image: confluentinc/cp-kafka-connect:3.2.0 container_name: kafka-connect depends_on: - zookeeper - kafka - schema-registry ports: - "8083:8083" restart: always environment: CONNECT_BOOTSTRAP_SERVERS: 'kafka:9092' CONNECT_REST_ADVERTISED_HOST_NAME: connect CONNECT_REST_PORT: 8083 CONNECT_GROUP_ID: compose-connect-group CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081 CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081 CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_ZOOKEEPER_CONNECT: "zookeeper:2181"Ahora logré exponer correctamente el contenedor de Kafka a mis aplicaciones no dockerizadas configurando correctamente la propiedad adverted.listener en PLAINTEXT://{DOCKER_MACHINE_IP}:9092, pero como puede ver, también agregué otras aplicaciones confluentes a extender mi configuración de Kafka (Kafka REST, Schema-Registry). Estos ya no pueden conectarse a mi instancia de Kafka debido a la propiedad adverted.listener.
Podría cambiarlo al nombre de host del contenedor correcto --> PLAINTEXT://kafka:9092 pero luego pierdo la capacidad de llegar a la instancia de kafka una vez más con mis otras aplicaciones. ¿Hay alguna manera fácil de resolver este problema?
Omar, tal vez ya hayas resuelto tu problema, pero para futuras referencias, el comentario de Hans Jespersen me funcionó, incluso en Windows.
Como administrador, abra C:\Windows\System32\drivers\etc\hosts y agregue la siguiente línea para exponer el agente kafka como localhost. 127.0.0.1 broker
Y mi archivo docker-compose.yml tiene el siguiente aspecto:
--- version: '2' services: zookeeper: image: confluentinc/cp-zookeeper hostname: zookeeper extra_hosts: - "moby:127.0.0.1" ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: image: confluentinc/cp-kafka hostname: broker extra_hosts: - "moby:127.0.0.1" depends_on: - zookeeper ports: - '9092:9092' environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:9092' KAFKA_DEFAULT_REPLICATION_FACTOR: 1 schema_registry: image: confluentinc/cp-schema-registry hostname: schema_registry # extra_hosts: # - "moby:127.0.0.1" depends_on: - zookeeper - broker ports: - '8081:8081' environment: SCHEMA_REGISTRY_HOST_NAME: schema_registry SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181' kafka-rest: image: confluentinc/cp-kafka-rest container_name: kafka-rest extra_hosts: - "moby:127.0.0.1" depends_on: - zookeeper - broker ports: - '8082:8082' environment: KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_REST_LISTENERS: http://kafka-rest:8082 KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081 KAFKA_REST_HOST_NAME: kafka-rest Alternativamente, exponer la dirección IP actual de mi computadora portátil (usando ipconfig /all) también funciona, pero esto tiene la desventaja de que, cada vez que mi red cambie, también tendría que cambiar el archivo docker-compose.yml .
Suponiendo que esta es una configuración destinada a su entorno de desarrollo local, aquí hay una solución para ejecutar en una red Docker.
version: '2' services: zookeeper: image: confluentinc/cp-zookeeper:3.2.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 restart: always ports: ['2181:2181'] kafka: image: confluentinc/cp-kafka:3.2.0 depends_on: - zookeeper ports: ['29092:29092'] environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 restart: always kafka-rest: image: confluentinc/cp-kafka-rest:3.2.0 depends_on: - kafka ports: ['8082:8082'] environment: KAFKA_REST_ZOOKEEPER_CONNECT: 'zookeeper:2181' KAFKA_REST_LISTENERS: http://0.0.0.0:8082 KAFKA_REST_SCHEMA_REGISTRY_URL: http://schema-registry:8081 KAFKA_REST_HOST_NAME: localhost restart: always schema-registry: image: confluentinc/cp-schema-registry:3.2.0 depends_on: - kafka ports: ['8081:8081'] environment: SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181' SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081 restart: always connect: image: confluentinc/cp-kafka-connect:3.2.0 depends_on: - zookeeper - kafka - schema-registry ports: ['8083:8083'] restart: always environment: CONNECT_BOOTSTRAP_SERVERS: 'kafka:29092' CONNECT_REST_ADVERTISED_HOST_NAME: connect CONNECT_REST_PORT: 8083 CONNECT_GROUP_ID: compose-connect-group CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status CONNECT_KEY_CONVERTER: io.confluent.connect.avro.AvroConverter CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081 CONNECT_VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema-registry:8081 CONNECT_INTERNAL_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_INTERNAL_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter CONNECT_ZOOKEEPER_CONNECT: "zookeeper:2181"Actualizado aquí: https://github.com/confluentinc/examples/blob/5.3.1-post/cp-all-in-one/docker-compose.yml
Solo Kafka con solo Zookepper
version: '2.1' services: zookeeper: image: wurstmeister/zookeeper hostname: zookeeper container_name: zookeeper ports: - "2181:2181" environment: ZOO_MY_ID: 1 ZOO_PORT: 2181 ZOO_SERVERS: server.1=zookeeper:2888:3888 restart: always volumes: - ./zk-single-kafka-single/zookeeper/data:/data - ./zk-single-kafka-single/zookeeper/datalog:/datalog kafka: image: wurstmeister/kafka hostname: kafka container_name: kafka ports: - "9092:9092" environment: HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2" KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092,LISTENER_DOCKER_EXTERNAL://_{HOSTNAME_COMMAND}:9094 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181" KAFKA_LISTENERS: LISTENER_DOCKER_INTERNAL://:9092,LISTENER_DOCKER_EXTERNAL://:9094 KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL KAFKA_ADVERTISED_HOST_NAME: 172.19.0.1 KAFKA_BROKER_ID: 1 KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO" KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 restart: always volumes: - ./zk-single-kafka-single/kafka/data:/var/lib/kafka/data depends_on: - zookeeper