seaweedfs/weed/mq/KAFKA_PHASE3_PLAN.md

Phase 3: Consumer Groups & Advanced Kafka Features

Overview

Phase 3 transforms the Kafka Gateway from a basic producer/consumer system into a full-featured, production-ready Kafka-compatible platform with consumer groups, advanced APIs, and enterprise features.

Goals

• Consumer Group Coordination: Full distributed consumer support
• Advanced Kafka APIs: Offset management, group coordination, heartbeats
• Performance & Scalability: Connection pooling, batching, compression
• Production Features: Metrics, monitoring, advanced configuration
• Enterprise Ready: Security, observability, operational tools

Core Features

1. Consumer Group Coordination

New Kafka APIs to Implement:

• JoinGroup (API 11): Consumer joins a consumer group
• SyncGroup (API 14): Coordinate partition assignments
• Heartbeat (API 12): Keep consumer alive in group
• LeaveGroup (API 13): Clean consumer departure
• OffsetCommit (API 8): Commit consumer offsets
• OffsetFetch (API 9): Retrieve committed offsets
• DescribeGroups (API 15): Get group metadata

Consumer Group Manager:

• Group membership tracking
• Partition assignment strategies (Range, RoundRobin)
• Rebalancing coordination
• Offset storage and retrieval
• Consumer liveness monitoring

2. Advanced Record Processing

Record Batch Improvements:

• Full Kafka record format parsing (v0, v1, v2)
• Compression support (gzip, snappy, lz4, zstd) — IMPLEMENTED
• Proper CRC validation — IMPLEMENTED
• Transaction markers handling
• Timestamp extraction and validation

Performance Optimizations:

• Record batching for SeaweedMQ
• Connection pooling to Agent
• Async publishing with acknowledgment batching
• Memory pooling for large messages

3. Enhanced Protocol Support

Additional APIs:

• FindCoordinator (API 10): Locate group coordinator
• DescribeConfigs (API 32): Get broker/topic configs
• AlterConfigs (API 33): Modify configurations
• DescribeLogDirs (API 35): Storage information
• CreatePartitions (API 37): Dynamic partition scaling

Protocol Improvements:

• Multiple API version support
• Better error code mapping
• Request/response correlation tracking
• Protocol version negotiation

4. Operational Features

Metrics & Monitoring:

• Prometheus metrics endpoint
• Consumer group lag monitoring
• Throughput and latency metrics
• Error rate tracking
• Connection pool metrics

Health & Diagnostics:

• Health check endpoints
• Debug APIs for troubleshooting
• Consumer group status reporting
• Partition assignment visualization

Configuration Management:

• Dynamic configuration updates
• Topic-level settings
• Consumer group policies
• Rate limiting and quotas

Implementation Plan

Step 1: Consumer Group Foundation (2-3 days)

1. Consumer group state management
2. Basic JoinGroup/SyncGroup APIs
3. Partition assignment logic
4. Group membership tracking

Step 2: Offset Management (1-2 days)

1. OffsetCommit/OffsetFetch APIs
2. Offset storage in SeaweedMQ
3. Consumer position tracking
4. Offset retention policies

Step 3: Consumer Coordination (1-2 days)

1. Heartbeat mechanism
2. Group rebalancing
3. Consumer failure detection
4. LeaveGroup handling

Step 4: Advanced Record Processing (2-3 days)

1. Full record parsing and real Fetch batch construction
2. Compression codecs and CRC (done) — focus on integration and tests
3. Performance optimizations
4. Memory management

Step 5: Enhanced APIs (1-2 days)

1. FindCoordinator implementation
2. DescribeGroups functionality
3. Configuration APIs
4. Administrative tools

Step 6: Production Features (2-3 days)

1. Metrics and monitoring
2. Health checks
3. Operational dashboards
4. Performance tuning

Architecture Changes

Consumer Group Coordinator

┌─────────────────────────────────────────────────┐
│                Gateway Server                    │
├─────────────────────────────────────────────────┤
│  Protocol Handler                               │
│  ├── Consumer Group Coordinator                 │
│  │   ├── Group State Machine                    │
│  │   ├── Partition Assignment                   │
│  │   ├── Rebalancing Logic                     │
│  │   └── Offset Manager                        │
│  ├── Enhanced Record Processor                  │
│  └── Metrics Collector                         │
├─────────────────────────────────────────────────┤
│  SeaweedMQ Integration Layer                    │
│  ├── Connection Pool                            │
│  ├── Batch Publisher                           │
│  └── Offset Storage                            │
└─────────────────────────────────────────────────┘

Consumer Group State Management

Consumer Group States:
- Empty: No active consumers
- PreparingRebalance: Waiting for consumers to join
- CompletingRebalance: Assigning partitions
- Stable: Normal operation
- Dead: Group marked for deletion

Consumer States:  
- Unknown: Initial state
- MemberPending: Joining group
- MemberStable: Active in group
- MemberLeaving: Graceful departure

Success Criteria

Functional Requirements

• ✅ Consumer groups work with multiple consumers
• ✅ Automatic partition rebalancing
• ✅ Offset commit/fetch functionality
• ✅ Consumer failure handling
• ✅ Full Kafka record format support (v2 with real records)
• ✅ Compression support for major codecs (already available)

Performance Requirements

• ✅ Handle 10k+ messages/second per partition
• ✅ Support 100+ consumer groups simultaneously
• ✅ Sub-100ms consumer group rebalancing
• ✅ Memory usage < 1GB for 1000 consumers

Compatibility Requirements

• ✅ Compatible with kafka-go, Sarama, and other Go clients
• ✅ Support Kafka 2.8+ client protocol versions
• ✅ Backwards compatible with Phase 1&2 implementations

Testing Strategy

Unit Tests

• Consumer group state transitions
• Partition assignment algorithms
• Offset management logic
• Record parsing and validation

Integration Tests

• Multi-consumer group scenarios
• Consumer failures and recovery
• Rebalancing under load
• SeaweedMQ storage integration

End-to-End Tests

• Real Kafka client libraries (kafka-go, Sarama)
• Producer/consumer workflows
• Consumer group coordination
• Performance benchmarking

Load Tests

• 1000+ concurrent consumers
• High-throughput scenarios
• Memory and CPU profiling
• Failure recovery testing

Deliverables

1. Consumer Group Coordinator - Full group management system
2. Enhanced Protocol Handler - 13+ Kafka APIs supported
3. Advanced Record Processing - Compression, batching, validation
4. Metrics & Monitoring - Prometheus integration, dashboards
5. Performance Optimizations - Connection pooling, memory management
6. Comprehensive Testing - Unit, integration, E2E, and load tests
7. Documentation - API docs, deployment guides, troubleshooting

Risk Mitigation

Technical Risks

• Consumer group complexity: Start with basic Range assignment, expand gradually
• Performance bottlenecks: Profile early, optimize incrementally
• SeaweedMQ integration: Maintain compatibility layer for fallback

Operational Risks

• Breaking changes: Maintain Phase 2 compatibility throughout
• Resource usage: Implement proper resource limits and monitoring
• Data consistency: Ensure offset storage reliability

Post-Phase 3 Vision

After Phase 3, the SeaweedFS Kafka Gateway will be:

• Production Ready: Handle enterprise Kafka workloads
• Highly Compatible: Work with major Kafka client libraries
• Operationally Excellent: Full observability and management tools
• Performant: Meet enterprise throughput requirements
• Reliable: Handle failures gracefully with strong consistency guarantees

This positions SeaweedFS as a compelling alternative to traditional Kafka deployments, especially for organizations already using SeaweedFS for storage and wanting unified message queue capabilities.