seaweedfs/KAFKA_SMQ_INTEGRATION_SUMMARY.md

# Kafka-SMQ Integration Implementation Summary

## Overview

This implementation provides **SMQ native offset management** for the Kafka Gateway, leveraging SeaweedMQ's existing distributed infrastructure instead of building custom persistence layers.

## Recommended Approach: SMQ Native Offset Management

### Core Concept

Instead of implementing separate consumer offset persistence, leverage SMQ's existing distributed offset management system where offsets are replicated across brokers and survive client disconnections and broker failures.

### Architecture

```
┌─ Kafka Gateway ─────────────────────────────────────┐
│ ┌─ Kafka Abstraction ──────────────────────────────┐ │
│ │ • Presents Kafka-style sequential offsets       │ │
│ │ • Translates to/from SMQ consumer groups         │ │
│ └──────────────────────────────────────────────────┘ │
│                        ↕                           │
│ ┌─ SMQ Native Offset Management ──────────────────┐ │
│ │ • Consumer groups with SMQ semantics            │ │
│ │ • Distributed offset tracking                   │ │
│ │ • Automatic replication & failover              │ │
│ │ • Built-in persistence                          │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
```

## Key Components

### 1. Message Offset Persistence (Implemented)
- **File**: `weed/mq/kafka/offset/persistence.go`
- **Purpose**: Maps Kafka sequential offsets to SMQ timestamps
- **Storage**: `kafka-system/offset-mappings` topic
- **Status**: Fully implemented and working

### 2. SMQ Native Consumer Groups (Recommended)
- **File**: `weed/mq/kafka/offset/smq_native_offset_management.go`
- **Purpose**: Use SMQ's built-in consumer group offset management
- **Benefits**: 
  - Automatic persistence and replication
  - Survives broker failures and restarts
  - No custom persistence code needed
  - Leverages battle-tested SMQ infrastructure

### 3. Offset Translation Layer
- **Purpose**: Lightweight cache for Kafka offset ↔ SMQ timestamp mapping
- **Implementation**: In-memory cache with LRU eviction
- **Scope**: Only recent mappings needed for active consumers

## Kafka to SMQ Mapping

| Kafka Concept | SMQ Concept | Implementation |
|---------------|-------------|----------------|
| Consumer Group | SMQ Consumer Group | Direct 1:1 mapping with "kafka-cg-" prefix |
| Topic-Partition | SMQ Topic-Partition Range | Map Kafka partition N to SMQ ring [N*32, (N+1)*32-1] |
| Sequential Offset | SMQ Timestamp + Index | Use SMQ's natural timestamp ordering |
| Offset Commit | SMQ Consumer Position | Let SMQ track position natively |
| Offset Fetch | SMQ Consumer Status | Query SMQ for current position |

## Long-Term Disconnection Handling

### Scenario: 30-Day Disconnection
```
Day 0:   Consumer reads to offset 1000, disconnects
         → SMQ consumer group position: timestamp T1000

Day 1-30: System continues, messages 1001-5000 arrive
         → SMQ stores messages with timestamps T1001-T5000
         → SMQ preserves consumer group position at T1000

Day 15:  Kafka Gateway restarts
         → In-memory cache lost
         → SMQ consumer group position PRESERVED

Day 30:  Consumer reconnects
         → SMQ automatically resumes from position T1000
         → Consumer receives messages starting from offset 1001
         → Perfect resumption, no manual recovery needed
```

## Benefits vs Custom Persistence

| Aspect | Custom Persistence | SMQ Native |
|--------|-------------------|------------|
| Implementation | High complexity (dual systems) | Medium (SMQ integration) |
| Reliability | Custom code risks | Battle-tested SMQ |
| Performance | Dual writes penalty | Native SMQ speed |
| Operational | Monitor 2 systems | Single system |
| Maintenance | Custom bugs/fixes | SMQ team handles |
| Scalability | Custom scaling | SMQ distributed arch |

## Implementation Roadmap

### Phase 1: SMQ API Assessment (1-2 weeks)
- Verify SMQ consumer group persistence behavior
- Test `RESUME_OR_EARLIEST` functionality
- Confirm replication across brokers

### Phase 2: Consumer Group Integration (2-3 weeks)
- Implement SMQ consumer group wrapper
- Map Kafka consumer lifecycle to SMQ
- Handle consumer group coordination

### Phase 3: Offset Translation (2-3 weeks)
- Lightweight cache for Kafka ↔ SMQ offset mapping
- Handle edge cases (rebalancing, etc.)
- Performance optimization

### Phase 4: Integration & Testing (3-4 weeks)
- Replace current offset management
- Comprehensive testing with long disconnections
- Performance benchmarking

**Total Estimated Effort: ~10 weeks**

## Current Status

### Completed
- Message offset persistence (Kafka offset → SMQ timestamp mapping)
- SMQ publishing integration with offset tracking
- SMQ subscription integration with offset mapping
- Comprehensive integration tests
- Architecture design for SMQ native approach

### Next Steps
1. Assess SMQ consumer group APIs
2. Implement SMQ native consumer group wrapper
3. Replace current in-memory offset management
4. Test long-term disconnection scenarios

## Key Advantages

- **Leverage Existing Infrastructure**: Use SMQ's proven distributed systems
- **Reduced Complexity**: No custom persistence layer to maintain
- **Better Reliability**: Inherit SMQ's fault tolerance and replication
- **Operational Simplicity**: One system to monitor instead of two
- **Performance**: Native SMQ operations without translation overhead
- **Automatic Edge Cases**: SMQ handles network partitions, broker failures, etc.

## Recommendation

The **SMQ Native Offset Management** approach is strongly recommended over custom persistence because:

1. **Simpler Implementation**: ~10 weeks vs 15+ weeks for custom persistence
2. **Higher Reliability**: Leverage battle-tested SMQ infrastructure
3. **Better Performance**: Native operations without dual-write penalty
4. **Lower Maintenance**: SMQ team handles distributed systems complexity
5. **Operational Benefits**: Single system to monitor and maintain

This approach solves the critical offset persistence problem while minimizing implementation complexity and maximizing reliability.