mirror of
https://github.com/chrislusf/seaweedfs
synced 2025-09-18 17:20:22 +02:00
3847984566
- Add unit tests for gateway connection/refusal in test/kafka/unit/gateway_test.go - Add Schema Registry connectivity test in test/kafka/integration/docker_test.go - Implement legacy offset key parsing in weed/mq/kafka/offset/smq_storage.go - Fix leaderEpoch placeholder to return 0 instead of -1 in offset_management.go - Add comprehensive test coverage for parseTopicPartitionKey function All tests passing. Ready for Phase 0 Part 2 (API cleanup and logging).
2.8 KiB
2.8 KiB
Kafka Gateway Development Plan
Overview
Implementation plan for SeaweedFS Kafka Gateway feature completion, organized by difficulty and dependencies.
Phase 0: Quick Wins and Hygiene (2-3 days)
Goal: Foundation cleanup and basic test coverage
Tasks
- Unit tests: gateway connection/refusal in
test/kafka/unit/gateway_test.go - Schema Registry connectivity test in
test/kafka/integration/docker_test.go - Offsets: legacy key parsing in
weed/mq/kafka/offset/smq_storage.go - Offsets: leaderEpoch placeholder (return 0 instead of -1)
- ApiVersions audit: align return set with validation
- Admin polish: error codes in DeleteTopics/CreateTopics
- Logging: replace fmt.Printf with structured logging
- Throttle: consistent throttle_time_ms across responses
Exit Criteria: All unit/integration tests green, no debug prints, consistent throttling
Phase 1: Data Plane + Admin Visibility (1 week)
Goal: Core fetch/produce robustness and group management
Tasks
- Fetch: record batch concatenation in
fetch.go:getMultipleRecordBatches - Fetch: GZIP compression implementation
- Group introspection: DescribeGroups (15) and ListGroups (16) handlers
- Client compatibility tests expansion
- Metrics: basic request/error/latency tracking
Exit Criteria: Multi-batch fetch works, gzip validated, group introspection functional
Phase 2: Consumer Groups Robustness (1 week)
Goal: Stable consumer group rebalancing
Tasks
- Rebalancing test with new framework
- Cooperative-sticky assignment implementation (feature-flagged)
- Consumer group state management hardening
Exit Criteria: Deterministic rebalances, cooperative-sticky tests passing
Phase 3: Schema Management Readiness (2 weeks)
Goal: Schema validation and encoding fixes
Tasks
- Avro union re-encoding fix (re-enable skipped tests)
- Protobuf index varints + descriptor field APIs
- Schema topic detection and metadata lookup
- Produce: per-topic schema validation
Exit Criteria: Schema decode/encode tests pass, produce validation enabled
Phase 4: Schematized Fetch Integration (1-2 weeks)
Goal: End-to-end schema round-trip
Tasks
- fetchSchematizedRecords SeaweedMQ integration
- Batch reconstruction with compression/CRC
- End-to-end schema tests
Exit Criteria: Round-trip produce/fetch for all schema formats
Risk Mitigation
- Avro fix blocks schema features - prioritize early
- Cooperative-sticky can be feature-flagged
- Compression beyond gzip can be deferred
- Performance targets: 100k messages, p99 < reasonable latency
Release Criteria
- All test suites green (unit/integration/e2e/client-compat)
- Consumer groups stable under churn
- Schema topics round-trip correctly
- No debug logging in production paths