Mini Kafka
Project Overview
A distributed message queue system built from scratch in Go. Developed a custom binary protocol to facilitate high-throughput communication between Producers, Brokers, and Consumers. The system supports multi-topic partitioning, offset commit mechanisms for reliable processing, and sophisticated Consumer Groups with automatic rebalancing and multi-consume capabilities.
Challenges & Solutions
Independent research to implement a production-style message broker from scratch. Key challenges included designing a low-latency protocol for Broker-Producer-Consumer communication, managing persistent data segments, and handling complex state in Consumer Groups with auto-rebalancing support.
Results & Impact
Successfully built a functional distributed message queue featuring Topic-level partitioning, high-performance Producers/Consumers, and offset Commit mechanisms. Implemented a robust Consumer Group system that supports parallel consumption and automatic group management.