An attemp to write a ligthweight kafka-like distributed log in golang.
My main goal is to learn how to build a distributed log and Golang allows me to bypass a lot of "language logic" by providing great concurrency support, mutexes, etc... When the implementation is done, I may consider rewriting the project in a more performant language.
- Message serialization and storage
- Log Segments
- Partitions
- Rentention Logic
- time (retention.ms)
- size (retention.bytes)
- cleanup policies
- delete
- compact (latest message per key)
- Topics
- TCP protocol
- Concurrent request handling
- Commands
- Run server
- Edit server
- Create topics
- Get topic
- List topics
- Delete topics
- Edit topics
- Produce
- Consume
- Commit
- List consumer groups
- Get consumer group
- Create consumer group
- Delete consumer group
- Create consumer
- Delete consumer
- Consumer groups
- Consumer rebalancing
- Consumer heartbeats
- Autocommit
- Consumer groups persistence
- Rebalancing notif to consumers
- Full concurrency support (mutexes)
- Topic deletion in background
- Empty key partition rotation (round robin)
- Write Ahead Log
- Log segments sparse indexes (faster random access)
- CRC on messages/batches (corruption detection)
- Message batching (increased write performance)
- Fully Distributed
- Raft consensus
- Partitions distribution
- Consumer groups distribution
- Replication
- Disaster recovery
- Golang client library
- Client
- Topics CRUD
- Produce
- Comsumer
- Fully functional CLI
- Node.js client library
- Python client library
- Dockerfile (basic)
- Kubernetes manifests
- decouple consumer creation and consume action
- simplify protocol files
- removal notif to removed consumer
- decouple consumer group creation and consumer creation
- show error in cli when topic doesn't exist
- check message size for absolute maximum size (max uint32)
- cli consumer not consuming in order (probably client side)
- short write response when consuming (server side)