# Kafka Gateway Development Plan

## Overview
Implementation plan for SeaweedFS Kafka Gateway feature completion, organized by difficulty and dependencies.

## Phase 0: Quick Wins and Hygiene (2-3 days)
**Goal**: Foundation cleanup and basic test coverage

### Tasks
- [ ] Unit tests: gateway connection/refusal in `test/kafka/unit/gateway_test.go`
- [ ] Schema Registry connectivity test in `test/kafka/integration/docker_test.go`
- [ ] Offsets: legacy key parsing in `weed/mq/kafka/offset/smq_storage.go`
- [ ] Offsets: leaderEpoch placeholder (return 0 instead of -1)
- [ ] ApiVersions audit: align return set with validation
- [ ] Admin polish: error codes in DeleteTopics/CreateTopics
- [ ] Logging: replace fmt.Printf with structured logging
- [ ] Throttle: consistent throttle_time_ms across responses

**Exit Criteria**: All unit/integration tests green, no debug prints, consistent throttling

## Phase 1: Data Plane + Admin Visibility (1 week)
**Goal**: Core fetch/produce robustness and group management

### Tasks
- [ ] Fetch: record batch concatenation in `fetch.go:getMultipleRecordBatches`
- [ ] Fetch: GZIP compression implementation
- [ ] Group introspection: DescribeGroups (15) and ListGroups (16) handlers
- [ ] Client compatibility tests expansion
- [ ] Metrics: basic request/error/latency tracking

**Exit Criteria**: Multi-batch fetch works, gzip validated, group introspection functional

## Phase 2: Consumer Groups Robustness (1 week)
**Goal**: Stable consumer group rebalancing

### Tasks
- [ ] Rebalancing test with new framework
- [ ] Cooperative-sticky assignment implementation (feature-flagged)
- [ ] Consumer group state management hardening

**Exit Criteria**: Deterministic rebalances, cooperative-sticky tests passing

## Phase 3: Schema Management Readiness (2 weeks)
**Goal**: Schema validation and encoding fixes

### Tasks
- [ ] Avro union re-encoding fix (re-enable skipped tests)
- [ ] Protobuf index varints + descriptor field APIs
- [ ] Schema topic detection and metadata lookup
- [ ] Produce: per-topic schema validation

**Exit Criteria**: Schema decode/encode tests pass, produce validation enabled

## Phase 4: Schematized Fetch Integration (1-2 weeks)
**Goal**: End-to-end schema round-trip

### Tasks
- [ ] fetchSchematizedRecords SeaweedMQ integration
- [ ] Batch reconstruction with compression/CRC
- [ ] End-to-end schema tests

**Exit Criteria**: Round-trip produce/fetch for all schema formats

## Risk Mitigation
- Avro fix blocks schema features - prioritize early
- Cooperative-sticky can be feature-flagged
- Compression beyond gzip can be deferred
- Performance targets: 100k messages, p99 < reasonable latency

## Release Criteria
- All test suites green (unit/integration/e2e/client-compat)
- Consumer groups stable under churn
- Schema topics round-trip correctly
- No debug logging in production paths