How to logs (stdout / stderr) from all container pods azure Kubernetes to the event hub. I can able to see all logs by Log Analytics workspaces >> Logs using an Azure query language.
I want to send all logs to the event hub.
Can anyone suggest on this?
You can easily forward container logs to Event Hubs via Fluent-Bit's Kafka output.
Here is Fluent-Bit documentation for Kafka - https://docs.fluentbit.io/manual/pipeline/outputs/kafka
And here is Kafka client integration with Event Hubs - https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
Worked for me using fluentbit kafka output to Azure EventHub
td-agent-bit.conf
[INPUT]
Name tail
Path xxx.log
Refresh_Interval 10
[OUTPUT]
Name kafka
Match *
brokers xxx.xxx.windows.net:9093
topics xxx
rdkafka.security.protocol SASL_SSL
rdkafka.sasl.username $ConnectionString
rdkafka.sasl.password Endpoint=sb://xxx.xxx.windows.net/;SharedAccessKeyName=xxx;SharedAccessKey=xxx
rdkafka.sasl.mechanism PLAIN
[OUTPUT]
name stdout
match *
Inside docker container (MUST HAVE or broker down/ssl fail)
docker-compose.yml
version: "3.7"
services:
fluent-bit:
image: fluent/fluent-bit:1.6.2
container_name: fluentbit
restart: always
volumes:
- ./td-agent-bit.conf:/fluent-bit/etc/fluent-bit.conf
- ./xxx.log:/fluent-bit/etc/xxx.log:ro