kafka_example

Description

this repository was created for the trial use of apache kafka

• 2 python clients are used as an example (confluent-kafka and aiokafka)
• Inside contains scripts for producing and consuming messages in kafka
• There is also an example of a service using confluent-kafka, and a notebook using it
• In addition to using kafka, the repository contains an example of kafka monitoring using grafana and prometheus
• I hope you find this repository useful and learn something new for yourself

TODO

• Custom Kafka handlers (produce, consume)

Application installation

• Clone repository

• Set up a virtual environment and install dependicies:

python -m venv venv

or

pyenv virtualenv 3.12.3 kafka_example

poetry install

• You can view the available commands with:

make help

• Setup settings for docker

make setup

• Run kafka and kafka-ui in docker (by default is kafka-ui from redpanda, but you can uncomment another ui or run your own.)

make kafka

make kafka-ui

UI endpoint - http://localhost:8080/overview

• Setup environment variables

source set_env.sh

Run producer and consumer examples

• Aiokafka producer and consumer run:

make aiokafka-produce

make aiokafka-consume

• Confluent-kafka producer and consumer run:

make confluent-kafka-produce

make confluent-kafka-consume

• Example of topic creation with confluent admin instance

make confluent-kafka-admin

Services example

• Example of service for kafka administartion more info in documentation:

• You can try use notebooks in ./notebooks directory

Kafka monitoring

• Clone repository grafana_example

• Run grafana

make grafana

• Copy example of dashboard for kafka - ./dashboards/kafka_dashboards_example.json

• Create new dashboard

• Running a script to create data in the dashboard

make kafka-loop