syslog-harness/LITELLM-MIGRATION-PLAN.md

# LiteLLM Integration Migration Plan
## Syslog Solution LLC — June 13, 2026

---

## Executive Summary

**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness — without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.

**Architecture Decision:** Two-layer architecture.

```
                  ┌─────────────────────────────────┐
                  │     LiteLLM Gateway (Layer 1)    │
                  │     Port 4000 — Policy & UX      │
                  │  ┌─────────────────────────────┐ │
                  │  │ Admin UI (/ui)              │ │
                  │  │ Virtual Keys & Permissions  │ │
                  │  │ Teams, Users, SSO (OIDC)    │ │
                  │  │ Spend Tracking & Budgets    │ │
                  │  │ Usage Analytics Dashboard   │ │
                  │  │ Request Audit Trail         │ │
                  │  │ Global Rate Limiting        │ │
                  │  └─────────────────────────────┘ │
                  │           │                       │
                  │    Pass-through to router         │
                  └──────────┬──────────────────────┘
                             │
                  ┌──────────▼──────────────────────┐
                  │  Custom Router (Layer 2)          │
                  │  Port 9000 — Intelligence & HW   │
                  │  ┌─────────────────────────────┐ │
                  │  │ 5-Tier Content-Based Routing│ │
                  │  │ GPU Slot Management (Redis) │ │
                  │  │ Agent Spread Prevention     │ │
                  │  │ GPU Health Scoring (40/30/30)│ │
                  │  │ Sidecar VRAM/Temp/Power     │ │
                  │  │ Circuit Breaker             │ │
                  │  │ Context Window Tracking      │ │
                  │  │ Per-Request Perf Recording  │ │
                  │  │ Hardware Rate Limiting      │ │
                  │  └─────────────────────────────┘ │
                  └──────────┬──────────────────────┘
                             │
          ┌──────────────────┼──────────────────────┐
          │                  │                       │
  ┌───────▼──────┐  ┌────────▼───────┐  ┌───────────▼──────┐
  │ qwen3.6-35B  │  │ qwen3.6-27B   │  │ gemma-4-12b      │
  │ MoE/Strix    │  │ Dense/RTX3090 │  │ VLM/RTX 5070     │
  │ :8080 (llama)│  │ :8080 (llama) │  │ :8080 (llama)    │
  │ :8090 (side) │  │ :8090 (side)  │  │ :8090 (sidecar)  │
  └──────────────┘  └───────────────┘  └──────────────────┘
```

---

## 1. Current State Baseline

### 1.1 Router (`router-fixed.py` — port 9000, deployed on CT 116 / docker-vm)

| Feature | Implementation |
|---------|---------------|
| **Routing Engine** | 5-tier content-based: lightweight → simple_conv → medium → heavy_reasoning → default |
| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset |
| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health |
| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs |
| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 |
| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation |
| **Performance** | Per-request latency/tokens/tps → Redis lists (perf:recent, perf:model:X, perf:agent:X) |
| **Context Tracking** | Session-level token accumulation with compaction warnings in headers |
| **SSE Streaming** | Real-time dashboard updates, per-model timeseries |
| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` |

### 1.2 GPU Backends

| GPU | Host | llama.cpp | Sidecar | VRAM | Context |
|-----|------|-----------|---------|------|---------|
| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K |
| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K |
| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K |

### 1.3 Existing LiteLLM Attempt
- `/root/litellm-fix.sh` — previous setup script for docker-vm
- Configured with Postgres, host networking, master key
- **Never productionized** — still in exploratory phase

---

## 2. What LiteLLM Brings (That We Don't Have)

| Feature | Our Router | LiteLLM | Value Add |
|---------|-----------|---------|-----------|
| **Admin UI** | ❌ | ✅ Full dashboard at /ui | Non-technical users can manage keys, view spend |
| **Virtual Key Permissions** | ❌ (binary key→tier) | ✅ Granular: per-model, per-team, budget caps | Fine-grained access control |
| **Spend Tracking** | ❌ | ✅ Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing |
| **Teams & Orgs** | ❌ | ✅ Multi-tenant: org→team→user hierarchy | Segregate clients/projects |
| **SSO/OIDC** | ❌ | ✅ Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration |
| **Budget Alerts** | ❌ | ✅ Per-key, per-user, per-team budget with webhooks | Prevent overspend |
| **Usage Analytics** | ⚠️ (custom /metrics) | ✅ Built-in: daily trends, model breakdown, per-customer | Better visualization |
| **100+ Provider Support** | ❌ (3 local GPUs) | ✅ OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access |
| **Fallback Chains** | ❌ | ✅ Multi-provider: OpenAI→Azure→Together | External model resilience |
| **RPM/TPM Weighted LB** | ❌ | ✅ Weighted load balancing across deployments | Fine-grained traffic shaping |

---

## 3. What We Keep (That LiteLLM Doesn't Have)

| Feature | Why We Must Keep It |
|---------|---------------------|
| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints |
| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power — our 40/30/30 scoring prevents routing to overheating GPUs |
| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading |
| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle |
| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers |
| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature — exposed via /metrics and SSE dashboard |
| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular |

---

## 4. Migration Architecture

### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room"

- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI
- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking

### 4.2 Flow

```
Agent Request
     │
     ▼
┌─────────────────────────────────────────────┐
│ LiteLLM Gateway (:4000)                      │
│                                              │
│ 1. Authenticate virtual key (sk-litellm-...) │
│ 2. Check key permissions (model access)      │
│ 3. Check budget (per-key, per-user, per-team)│
│ 4. Check team rate limits                    │
│ 5. Log request metadata                      │
│ 6. Forward to custom router as OpenAI-compat  │
│    POST http://router:9000/v1/chat/completions│
│    Headers: Authorization: Bearer <agent-key> │
│             X-LiteLLM-User: <user-id>         │
│             X-LiteLLM-Team: <team-id>         │
│             X-Session-Id: <session>           │
│                                              │
│ 7. On response: log spend, update budgets     │
│ 8. Return response to agent                   │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│ Custom Router (:9000)                         │
│                                               │
│ 1. Authenticate agent key (sk-syslog-...)     │
│ 2. Hardware rate limit (per-tier RPM)         │
│ 3. Content-based tier routing                 │
│    - Estimate tokens, detect system msg       │
│    - Count turns, check routing_hints         │
│ 4. GPU slot availability (Redis counter)      │
│ 5. GPU health check (sidecar)                 │
│ 6. Agent spread logic (select_best_gpu)       │
│ 7. Queue if saturated (with timeout)          │
│ 8. Forward to selected llama.cpp GPU          │
│ 9. Track context window, set compaction header│
│ 10. Record performance metrics                │
│ 11. Return response (with routing metadata)   │
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│ llama.cpp GPU (:8080)                         │
└──────────────────────────────────────────────┘
```

### 4.3 LiteLLM Config (`config.yaml`)

```yaml
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm
  store_model_in_db: true

model_list:
  # All three GPUs exposed as a single virtual "syslog-router" model
  # LiteLLM passes through to our router, which handles actual GPU selection
  - model_name: syslog-auto           # Default auto-routing
    litellm_params:
      model: openai/syslog-auto       # Using OpenAI-compatible format
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
      rpm: 600                         # Cap total RPM across all GPUs
  
  # Individual GPU pass-through (for explicit model requests)
  - model_name: qwen3.6-35B-A3B
    litellm_params:
      model: openai/qwen3.6-35B-A3B
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
  
  - model_name: qwen3.6-27B-code
    litellm_params:
      model: openai/qwen3.6-27B-code
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

  - model_name: gemma-4-12b
    litellm_params:
      model: openai/gemma-4-12b
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

litellm_settings:
  num_retries: 0                     # Disabled — our router handles retry
  request_timeout: 600               # Match our 10-min llama-server timeout
  set_verbose: true
  failure_callback: ["prometheus"]   # Optional: export to Prometheus

router_settings:
  routing_strategy: "usage-based-routing"  # For external models only
  # Note: All local GPU routing is handled by custom router
  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
  allowed_fails: 100                       # Don't cooldown — our circuit breaker handles

# Cost tracking: map model names to per-token pricing
# These are passed through from our router's X-Usage-Tokens header
```

### 4.4 Router Modifications (Light Touch)

Minimal changes to `router-fixed.py` — the router remains largely unchanged:

1. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent — already works)
2. **New endpoint for health passthrough**: `GET /v1/models` already works
3. **Disable own key management**: Remove `/admin/keys/*` endpoints (migrate to LiteLLM UI)
4. **Keep ALL routing logic**: No changes to `route()`, `select_best_gpu()`, `check_gpu_health()`, slot management, etc.
5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs

```python
# ADD to router-fixed.py chat() response:
resp.headers["X-Usage-Tokens"] = json.dumps({
    "prompt_tokens": prompt_tokens,
    "completion_tokens": completion_tokens,
    "model": model
})
```

---

## 5. Deployment Plan (3 Phases)

### Phase 1: Shadow Mode (Week 1) — Zero Risk

**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode.

```
Agent → LiteLLM (:4000) → Router (:9000) → GPU
         (new, testing)    (existing, unchanged)
         
Agent can also directly hit :9000 as fallback
```

**Tasks:**
1. **Deploy Postgres + LiteLLM on docker-vm**
   ```bash
   cd /opt/litellm
   # Apply litellm-fix.sh (already prepared)
   docker compose up -d
   ```

2. **Create config.yaml** with router as upstream (see §4.3)

3. **Create virtual keys for test agents** via LiteLLM UI
   - Mirror existing API_KEYS in LiteLLM's key store
   - Set per-key budgets (test with $100 cap)

4. **Verify pass-through works**
   ```bash
   curl -X POST http://docker-vm:4000/v1/chat/completions \
     -H "Authorization: Bearer sk-litellm-test-key" \
     -H "Content-Type: application/json" \
     -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
   ```

5. **Run 24-hour shadow**: Both :4000 and :9000 active, agents use :9000
   - Monitor LiteLLM spend logs vs router metrics — confirm parity
   - Verify GPU health metrics unaffected

### Phase 2: Cutover (Week 2) — Gradual Migration

**Goal:** Move agents one-by-one to LiteLLM endpoint.

**Tasks:**
1. **Migrate API keys to LiteLLM virtual keys:**
   - Create virtual key per agent in LiteLLM UI
   - Set model access: `syslog-auto` (default), plus individual GPU models
   - Set per-agent budget limits
   - Create teams: "Core Agents" (Abiba, Mumuni, Tanko), "Dev Agents" (Kagenz0, Koby, Koonimo)

2. **Update agent configs:**
   - Change `OPENAI_API_BASE` from `http://docker-vm:9000/v1` → `http://docker-vm:4000/v1`
   - Replace agent API keys with LiteLLM virtual keys
   - Test each agent one at a time

3. **Migrate admin functions:**
   - Key creation/revocation → LiteLLM UI
   - Rate limit management → LiteLLM per-key RPM + router hardware RPM (dual enforcement)
   - Deprecated key tracking → LiteLLM UI key list

4. **Enable SSO** (optional, Phase 2+):
   ```yaml
   general_settings:
     litellm_dashboard_sso: true
     sso_provider: "google"  # or github, microsoft, keycloak
     sso_client_id: os.environ/SSO_CLIENT_ID
     sso_client_secret: os.environ/SSO_CLIENT_SECRET
   ```

5. **Keep router :9000 accessible** as emergency fallback for 48 hours

### Phase 3: Production Hardening (Week 3+) — Optimize

**Goal:** Lock down, optimize, monitor.

**Tasks:**
1. **Remove deprecated router endpoints:**
   - Drop `/admin/keys/*` — fully migrated to LiteLLM UI
   - Drop Phase 0 dual-key logic (LiteLLM handles key rotation)
   - Simplify `API_KEYS` to single `ROUTER_API_KEY`

2. **Add LiteLLM observability:**
   - Prometheus metrics export
   - Slack/email budget alerts
   - Daily spend report webhook

3. **Enable LiteLLM caching** (Redis, shared with router):
   ```yaml
   router_settings:
     redis_host: os.environ/REDIS_HOST
     redis_port: 6379
     cache: true
     cache_ttl: 3600
   ```

4. **Optional: External model fallbacks**
   - Add Anthropic Claude as fallback for code-heavy requests
   - Add OpenAI GPT-4o as fallback for reasoning overflow
   - LiteLLM's native fallback chains handle this cleanly

5. **Router slim-down:** Extract GPU health metrics to dedicated /health only
   - Keep: routing, slots, health checks, performance recording
   - Remove: key management, dual-key logic, admin endpoints

---

## 6. Nginx Configuration

The existing nginx config routes `/admin/` → router :9000. This MUST change:

```nginx
# OLD (remove)
# location /admin/ {
#     proxy_pass http://127.0.0.1:9000/admin/;
# }

# NEW
location /ui/ {
    proxy_pass http://127.0.0.1:4000/ui/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

location /v1/ {
    # Primary: LiteLLM gateway
    proxy_pass http://127.0.0.1:4000/v1/;
    proxy_set_header Host $host;
    proxy_read_timeout 600s;
    # Fallback: direct router (if LiteLLM down)
    # error_page 502 = @router_fallback;
}

location @router_fallback {
    proxy_pass http://127.0.0.1:9000/v1/;
}

# Keep router metrics accessible (not behind LiteLLM)
location /router/ {
    proxy_pass http://127.0.0.1:9000/;
    # Rewrite /router/stream → :9000/stream
    # Rewrite /router/metrics → :9000/metrics
}

# Health check — combines both layers
location /health {
    # Check LiteLLM first, then router
    proxy_pass http://127.0.0.1:4000/health;
}
```

---

## 7. Docker Compose (`docker-compose.yml` on docker-vm)

```yaml
services:
  # Layer 1: LiteLLM Gateway (Policy & Admin)
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    network_mode: "host"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - UI_USERNAME=admin
      - UI_PASSWORD=${UI_PASSWORD}
      - DATABASE_URL=postgresql://litellm:${POSTGRES_PASSWORD}@localhost:5432/litellm
      - STORE_MODEL_IN_DB=True
      - ROUTER_API_KEY=${ROUTER_API_KEY}
    command:
      - --config
      - /app/config.yaml
      - --port
      - "4000"
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  # Database for LiteLLM
  postgres:
    image: postgres:16-alpine
    network_mode: "host"
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  # Layer 2: Custom Router (Intelligence & Hardware)
  # Already deployed separately — not in this compose file
  # The router is managed by the existing harness deployment on CT 116

volumes:
  pgdata:
```

---

## 8. Risk Mitigation

| Risk | Mitigation |
|------|------------|
| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable for admin features. LiteLLM is a thin proxy. |
| LiteLLM down = all agents down | Nginx fallback to router :9000 direct (see §6). Agents can also be configured with dual endpoints. |
| Key sync drift (LiteLLM keys ≠ router keys) | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` from LiteLLM's perspective. Agent keys live in LiteLLM only. |
| Spend tracking inaccurate for local GPUs | Configure `model_cost` per GPU with $0 rate (self-hosted). Optionally track "internal cost" via custom pricing. |
| Double rate limiting (LiteLLM + Router) | Keep both intentionally: LiteLLM for per-user soft caps, Router for hardware protection. Non-overlapping concerns. |
| PostgreSQL failure | LiteLLM can run with SQLite fallback, but UI features degrade. Postgres is the recommended path. |
| Router custom logic becomes a black box to LiteLLM | Acceptable trade-off. LiteLLM sees router as opaque OpenAI endpoint. GPU-level routing decisions are router's domain. |

---

## 9. Success Metrics

| Metric | Before | After |
|--------|--------|-------|
| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy |
| Spend visibility | None | Per-agent, per-team, per-model $ tracking |
| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped |
| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key |
| Admin UX | curl + JSON responses | Visual dashboard, graphs, search |
| Audit trail | Router logs (stdout only) | Database-backed with UI search |
| SSO | None | Google/GitHub/Microsoft OIDC |
| Budget enforcement | None | Automatic: key suspended at $limit |
| GPU routing intelligence | Full (unchanged) | Full (unchanged) |
| GPU health monitoring | Full (unchanged) | Full (unchanged) |

---

## 10. Migration Commands (Quick Reference)

```bash
# On docker-vm (CT 116):

# 1. Deploy LiteLLM stack
cd /opt/litellm
docker compose down -v   # Clean slate
docker compose up -d     # Postgres + LiteLLM

# 2. Verify
curl http://localhost:4000/health
curl http://localhost:4000/ui   # Admin dashboard

# 3. Create first virtual key via UI or CLI
docker compose exec litellm litellm-proxy keys create \
  --key-alias "abiba-test" \
  --models "syslog-auto" \
  --max-budget 10.0 \
  --team-id "core-agents"

# 4. Test end-to-end
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer <virtual-key>" \
  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"Hello"}]}'

# 5. Update nginx (see §6)
nginx -t && nginx -s reload

# 6. Monitor both layers
curl http://localhost:4000/global/spend/logs   # LiteLLM spend
curl http://localhost:9000/metrics             # Router GPU metrics
curl http://localhost:9000/stream              # Router SSE dashboard
```

---

## Appendix A: Router Slim-Down (Phase 3)

After full migration, `router-fixed.py` can be simplified by removing:

```python
# REMOVE (migrated to LiteLLM):
- API_KEYS validation logic (keep single ROUTER_API_KEY)
- Dual-key deprecation tracking
- /admin/keys, /admin/keys/generate, /admin/keys/revoke
- /admin/keys/deprecation-summary
- Phase 0 deprecated key logging
- check_rate_limit() (optional — keep as hardware safety net)

# KEEP:
- route() — all 5 tiers
- select_best_gpu()
- check_gpu_health()
- is_gpu_busy(), gpu_active_count(), gpu_incr/decr()
- estimate_tokens()
- store_perf_record()
- GPU_SIDECARS, GPU_URLS, GPU_MAX_CONCURRENT, GPU_CONTEXT
- counter_audit_loop()
- /v1/chat/completions — core routing endpoint
- /v1/models
- /health
- /metrics, /metrics/performance, /metrics/scatter, /metrics/timeseries
- /stream — SSE dashboard
```

## Appendix B: LiteLLM Cost Config for Local GPUs

```yaml
# In config.yaml — map models to per-token pricing for spend tracking
litellm_settings:
  model_cost:
    qwen3.6-35B-A3B:
      input_cost_per_token: 0.0        # Self-hosted, no external cost
      output_cost_per_token: 0.0
    qwen3.6-27B-code:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
    gemma-4-12b:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
  # For internal cost allocation, set symbolic rates:
  # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
```

---

*Plan drafted: 2026-06-13 by Abiba 🦊⚡*
*Status: Ready for Kwame review*