mirror of
https://github.com/chrislusf/seaweedfs
synced 2025-09-18 17:20:22 +02:00
3847984566
- Add unit tests for gateway connection/refusal in test/kafka/unit/gateway_test.go - Add Schema Registry connectivity test in test/kafka/integration/docker_test.go - Implement legacy offset key parsing in weed/mq/kafka/offset/smq_storage.go - Fix leaderEpoch placeholder to return 0 instead of -1 in offset_management.go - Add comprehensive test coverage for parseTopicPartitionKey function All tests passing. Ready for Phase 0 Part 2 (API cleanup and logging).
74 lines
2.8 KiB
Markdown
74 lines
2.8 KiB
Markdown
# Kafka Gateway Development Plan
|
|
|
|
## Overview
|
|
Implementation plan for SeaweedFS Kafka Gateway feature completion, organized by difficulty and dependencies.
|
|
|
|
## Phase 0: Quick Wins and Hygiene (2-3 days)
|
|
**Goal**: Foundation cleanup and basic test coverage
|
|
|
|
### Tasks
|
|
- [ ] Unit tests: gateway connection/refusal in `test/kafka/unit/gateway_test.go`
|
|
- [ ] Schema Registry connectivity test in `test/kafka/integration/docker_test.go`
|
|
- [ ] Offsets: legacy key parsing in `weed/mq/kafka/offset/smq_storage.go`
|
|
- [ ] Offsets: leaderEpoch placeholder (return 0 instead of -1)
|
|
- [ ] ApiVersions audit: align return set with validation
|
|
- [ ] Admin polish: error codes in DeleteTopics/CreateTopics
|
|
- [ ] Logging: replace fmt.Printf with structured logging
|
|
- [ ] Throttle: consistent throttle_time_ms across responses
|
|
|
|
**Exit Criteria**: All unit/integration tests green, no debug prints, consistent throttling
|
|
|
|
## Phase 1: Data Plane + Admin Visibility (1 week)
|
|
**Goal**: Core fetch/produce robustness and group management
|
|
|
|
### Tasks
|
|
- [ ] Fetch: record batch concatenation in `fetch.go:getMultipleRecordBatches`
|
|
- [ ] Fetch: GZIP compression implementation
|
|
- [ ] Group introspection: DescribeGroups (15) and ListGroups (16) handlers
|
|
- [ ] Client compatibility tests expansion
|
|
- [ ] Metrics: basic request/error/latency tracking
|
|
|
|
**Exit Criteria**: Multi-batch fetch works, gzip validated, group introspection functional
|
|
|
|
## Phase 2: Consumer Groups Robustness (1 week)
|
|
**Goal**: Stable consumer group rebalancing
|
|
|
|
### Tasks
|
|
- [ ] Rebalancing test with new framework
|
|
- [ ] Cooperative-sticky assignment implementation (feature-flagged)
|
|
- [ ] Consumer group state management hardening
|
|
|
|
**Exit Criteria**: Deterministic rebalances, cooperative-sticky tests passing
|
|
|
|
## Phase 3: Schema Management Readiness (2 weeks)
|
|
**Goal**: Schema validation and encoding fixes
|
|
|
|
### Tasks
|
|
- [ ] Avro union re-encoding fix (re-enable skipped tests)
|
|
- [ ] Protobuf index varints + descriptor field APIs
|
|
- [ ] Schema topic detection and metadata lookup
|
|
- [ ] Produce: per-topic schema validation
|
|
|
|
**Exit Criteria**: Schema decode/encode tests pass, produce validation enabled
|
|
|
|
## Phase 4: Schematized Fetch Integration (1-2 weeks)
|
|
**Goal**: End-to-end schema round-trip
|
|
|
|
### Tasks
|
|
- [ ] fetchSchematizedRecords SeaweedMQ integration
|
|
- [ ] Batch reconstruction with compression/CRC
|
|
- [ ] End-to-end schema tests
|
|
|
|
**Exit Criteria**: Round-trip produce/fetch for all schema formats
|
|
|
|
## Risk Mitigation
|
|
- Avro fix blocks schema features - prioritize early
|
|
- Cooperative-sticky can be feature-flagged
|
|
- Compression beyond gzip can be deferred
|
|
- Performance targets: 100k messages, p99 < reasonable latency
|
|
|
|
## Release Criteria
|
|
- All test suites green (unit/integration/e2e/client-compat)
|
|
- Consumer groups stable under churn
|
|
- Schema topics round-trip correctly
|
|
- No debug logging in production paths
|