From d901235c0373d82655a57bc4a2566a2a76410fb2 Mon Sep 17 00:00:00 2001 From: Abiba Date: Sun, 14 Jun 2026 00:39:59 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20LiteLLM=20migration=20plan=20=E2=80=94?= =?UTF-8?q?=20two-layer=20architecture=20with=20model=20identity=20gap=20a?= =?UTF-8?q?nalysis?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architecture review identifying metric accuracy issue where router silently reroutes explicit model requests. Proposes Option A: strict passthrough for explicit models with LiteLLM-native fallback chains. Keeps syslog-auto for content-based routing. Awaiting Mumuni and Kagenz0 review. --- LITELLM-MIGRATION-PLAN.md | 577 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 577 insertions(+) create mode 100644 LITELLM-MIGRATION-PLAN.md diff --git a/LITELLM-MIGRATION-PLAN.md b/LITELLM-MIGRATION-PLAN.md new file mode 100644 index 0000000..fcf560d --- /dev/null +++ b/LITELLM-MIGRATION-PLAN.md @@ -0,0 +1,577 @@ +# LiteLLM Integration Migration Plan +## Syslog Solution LLC — June 13, 2026 + +--- + +## Executive Summary + +**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness — without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring. + +**Architecture Decision:** Two-layer architecture. + +``` + ┌─────────────────────────────────┐ + │ LiteLLM Gateway (Layer 1) │ + │ Port 4000 — Policy & UX │ + │ ┌─────────────────────────────┐ │ + │ │ Admin UI (/ui) │ │ + │ │ Virtual Keys & Permissions │ │ + │ │ Teams, Users, SSO (OIDC) │ │ + │ │ Spend Tracking & Budgets │ │ + │ │ Usage Analytics Dashboard │ │ + │ │ Request Audit Trail │ │ + │ │ Global Rate Limiting │ │ + │ └─────────────────────────────┘ │ + │ │ │ + │ Pass-through to router │ + └──────────┬──────────────────────┘ + │ + ┌──────────▼──────────────────────┐ + │ Custom Router (Layer 2) │ + │ Port 9000 — Intelligence & HW │ + │ ┌─────────────────────────────┐ │ + │ │ 5-Tier Content-Based Routing│ │ + │ │ GPU Slot Management (Redis) │ │ + │ │ Agent Spread Prevention │ │ + │ │ GPU Health Scoring (40/30/30)│ │ + │ │ Sidecar VRAM/Temp/Power │ │ + │ │ Circuit Breaker │ │ + │ │ Context Window Tracking │ │ + │ │ Per-Request Perf Recording │ │ + │ │ Hardware Rate Limiting │ │ + │ └─────────────────────────────┘ │ + └──────────┬──────────────────────┘ + │ + ┌──────────────────┼──────────────────────┐ + │ │ │ + ┌───────▼──────┐ ┌────────▼───────┐ ┌───────────▼──────┐ + │ qwen3.6-35B │ │ qwen3.6-27B │ │ gemma-4-12b │ + │ MoE/Strix │ │ Dense/RTX3090 │ │ VLM/RTX 5070 │ + │ :8080 (llama)│ │ :8080 (llama) │ │ :8080 (llama) │ + │ :8090 (side) │ │ :8090 (side) │ │ :8090 (sidecar) │ + └──────────────┘ └───────────────┘ └──────────────────┘ +``` + +--- + +## 1. Current State Baseline + +### 1.1 Router (`router-fixed.py` — port 9000, deployed on CT 116 / docker-vm) + +| Feature | Implementation | +|---------|---------------| +| **Routing Engine** | 5-tier content-based: lightweight → simple_conv → medium → heavy_reasoning → default | +| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset | +| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health | +| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs | +| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 | +| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation | +| **Performance** | Per-request latency/tokens/tps → Redis lists (perf:recent, perf:model:X, perf:agent:X) | +| **Context Tracking** | Session-level token accumulation with compaction warnings in headers | +| **SSE Streaming** | Real-time dashboard updates, per-model timeseries | +| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` | + +### 1.2 GPU Backends + +| GPU | Host | llama.cpp | Sidecar | VRAM | Context | +|-----|------|-----------|---------|------|---------| +| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K | +| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K | +| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K | + +### 1.3 Existing LiteLLM Attempt +- `/root/litellm-fix.sh` — previous setup script for docker-vm +- Configured with Postgres, host networking, master key +- **Never productionized** — still in exploratory phase + +--- + +## 2. What LiteLLM Brings (That We Don't Have) + +| Feature | Our Router | LiteLLM | Value Add | +|---------|-----------|---------|-----------| +| **Admin UI** | ❌ | ✅ Full dashboard at /ui | Non-technical users can manage keys, view spend | +| **Virtual Key Permissions** | ❌ (binary key→tier) | ✅ Granular: per-model, per-team, budget caps | Fine-grained access control | +| **Spend Tracking** | ❌ | ✅ Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing | +| **Teams & Orgs** | ❌ | ✅ Multi-tenant: org→team→user hierarchy | Segregate clients/projects | +| **SSO/OIDC** | ❌ | ✅ Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration | +| **Budget Alerts** | ❌ | ✅ Per-key, per-user, per-team budget with webhooks | Prevent overspend | +| **Usage Analytics** | ⚠️ (custom /metrics) | ✅ Built-in: daily trends, model breakdown, per-customer | Better visualization | +| **100+ Provider Support** | ❌ (3 local GPUs) | ✅ OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access | +| **Fallback Chains** | ❌ | ✅ Multi-provider: OpenAI→Azure→Together | External model resilience | +| **RPM/TPM Weighted LB** | ❌ | ✅ Weighted load balancing across deployments | Fine-grained traffic shaping | + +--- + +## 3. What We Keep (That LiteLLM Doesn't Have) + +| Feature | Why We Must Keep It | +|---------|---------------------| +| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints | +| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power — our 40/30/30 scoring prevents routing to overheating GPUs | +| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading | +| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle | +| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers | +| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature — exposed via /metrics and SSE dashboard | +| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular | + +--- + +## 4. Migration Architecture + +### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room" + +- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI +- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking + +### 4.2 Flow + +``` +Agent Request + │ + ▼ +┌─────────────────────────────────────────────┐ +│ LiteLLM Gateway (:4000) │ +│ │ +│ 1. Authenticate virtual key (sk-litellm-...) │ +│ 2. Check key permissions (model access) │ +│ 3. Check budget (per-key, per-user, per-team)│ +│ 4. Check team rate limits │ +│ 5. Log request metadata │ +│ 6. Forward to custom router as OpenAI-compat │ +│ POST http://router:9000/v1/chat/completions│ +│ Headers: Authorization: Bearer │ +│ X-LiteLLM-User: │ +│ X-LiteLLM-Team: │ +│ X-Session-Id: │ +│ │ +│ 7. On response: log spend, update budgets │ +│ 8. Return response to agent │ +└──────────────┬──────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────┐ +│ Custom Router (:9000) │ +│ │ +│ 1. Authenticate agent key (sk-syslog-...) │ +│ 2. Hardware rate limit (per-tier RPM) │ +│ 3. Content-based tier routing │ +│ - Estimate tokens, detect system msg │ +│ - Count turns, check routing_hints │ +│ 4. GPU slot availability (Redis counter) │ +│ 5. GPU health check (sidecar) │ +│ 6. Agent spread logic (select_best_gpu) │ +│ 7. Queue if saturated (with timeout) │ +│ 8. Forward to selected llama.cpp GPU │ +│ 9. Track context window, set compaction header│ +│ 10. Record performance metrics │ +│ 11. Return response (with routing metadata) │ +└──────────────┬───────────────────────────────┘ + │ + ▼ +┌──────────────────────────────────────────────┐ +│ llama.cpp GPU (:8080) │ +└──────────────────────────────────────────────┘ +``` + +### 4.3 LiteLLM Config (`config.yaml`) + +```yaml +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY + database_url: postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm + store_model_in_db: true + +model_list: + # All three GPUs exposed as a single virtual "syslog-router" model + # LiteLLM passes through to our router, which handles actual GPU selection + - model_name: syslog-auto # Default auto-routing + litellm_params: + model: openai/syslog-auto # Using OpenAI-compatible format + api_base: http://router:9000/v1 + api_key: os.environ/ROUTER_API_KEY + rpm: 600 # Cap total RPM across all GPUs + + # Individual GPU pass-through (for explicit model requests) + - model_name: qwen3.6-35B-A3B + litellm_params: + model: openai/qwen3.6-35B-A3B + api_base: http://router:9000/v1 + api_key: os.environ/ROUTER_API_KEY + + - model_name: qwen3.6-27B-code + litellm_params: + model: openai/qwen3.6-27B-code + api_base: http://router:9000/v1 + api_key: os.environ/ROUTER_API_KEY + + - model_name: gemma-4-12b + litellm_params: + model: openai/gemma-4-12b + api_base: http://router:9000/v1 + api_key: os.environ/ROUTER_API_KEY + +litellm_settings: + num_retries: 0 # Disabled — our router handles retry + request_timeout: 600 # Match our 10-min llama-server timeout + set_verbose: true + failure_callback: ["prometheus"] # Optional: export to Prometheus + +router_settings: + routing_strategy: "usage-based-routing" # For external models only + # Note: All local GPU routing is handled by custom router + enable_loadbalancing_on_proxy: false # Disable LiteLLM's internal LB + allowed_fails: 100 # Don't cooldown — our circuit breaker handles + +# Cost tracking: map model names to per-token pricing +# These are passed through from our router's X-Usage-Tokens header +``` + +### 4.4 Router Modifications (Light Touch) + +Minimal changes to `router-fixed.py` — the router remains largely unchanged: + +1. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent — already works) +2. **New endpoint for health passthrough**: `GET /v1/models` already works +3. **Disable own key management**: Remove `/admin/keys/*` endpoints (migrate to LiteLLM UI) +4. **Keep ALL routing logic**: No changes to `route()`, `select_best_gpu()`, `check_gpu_health()`, slot management, etc. +5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs + +```python +# ADD to router-fixed.py chat() response: +resp.headers["X-Usage-Tokens"] = json.dumps({ + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "model": model +}) +``` + +--- + +## 5. Deployment Plan (3 Phases) + +### Phase 1: Shadow Mode (Week 1) — Zero Risk + +**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. + +``` +Agent → LiteLLM (:4000) → Router (:9000) → GPU + (new, testing) (existing, unchanged) + +Agent can also directly hit :9000 as fallback +``` + +**Tasks:** +1. **Deploy Postgres + LiteLLM on docker-vm** + ```bash + cd /opt/litellm + # Apply litellm-fix.sh (already prepared) + docker compose up -d + ``` + +2. **Create config.yaml** with router as upstream (see §4.3) + +3. **Create virtual keys for test agents** via LiteLLM UI + - Mirror existing API_KEYS in LiteLLM's key store + - Set per-key budgets (test with $100 cap) + +4. **Verify pass-through works** + ```bash + curl -X POST http://docker-vm:4000/v1/chat/completions \ + -H "Authorization: Bearer sk-litellm-test-key" \ + -H "Content-Type: application/json" \ + -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}' + ``` + +5. **Run 24-hour shadow**: Both :4000 and :9000 active, agents use :9000 + - Monitor LiteLLM spend logs vs router metrics — confirm parity + - Verify GPU health metrics unaffected + +### Phase 2: Cutover (Week 2) — Gradual Migration + +**Goal:** Move agents one-by-one to LiteLLM endpoint. + +**Tasks:** +1. **Migrate API keys to LiteLLM virtual keys:** + - Create virtual key per agent in LiteLLM UI + - Set model access: `syslog-auto` (default), plus individual GPU models + - Set per-agent budget limits + - Create teams: "Core Agents" (Abiba, Mumuni, Tanko), "Dev Agents" (Kagenz0, Koby, Koonimo) + +2. **Update agent configs:** + - Change `OPENAI_API_BASE` from `http://docker-vm:9000/v1` → `http://docker-vm:4000/v1` + - Replace agent API keys with LiteLLM virtual keys + - Test each agent one at a time + +3. **Migrate admin functions:** + - Key creation/revocation → LiteLLM UI + - Rate limit management → LiteLLM per-key RPM + router hardware RPM (dual enforcement) + - Deprecated key tracking → LiteLLM UI key list + +4. **Enable SSO** (optional, Phase 2+): + ```yaml + general_settings: + litellm_dashboard_sso: true + sso_provider: "google" # or github, microsoft, keycloak + sso_client_id: os.environ/SSO_CLIENT_ID + sso_client_secret: os.environ/SSO_CLIENT_SECRET + ``` + +5. **Keep router :9000 accessible** as emergency fallback for 48 hours + +### Phase 3: Production Hardening (Week 3+) — Optimize + +**Goal:** Lock down, optimize, monitor. + +**Tasks:** +1. **Remove deprecated router endpoints:** + - Drop `/admin/keys/*` — fully migrated to LiteLLM UI + - Drop Phase 0 dual-key logic (LiteLLM handles key rotation) + - Simplify `API_KEYS` to single `ROUTER_API_KEY` + +2. **Add LiteLLM observability:** + - Prometheus metrics export + - Slack/email budget alerts + - Daily spend report webhook + +3. **Enable LiteLLM caching** (Redis, shared with router): + ```yaml + router_settings: + redis_host: os.environ/REDIS_HOST + redis_port: 6379 + cache: true + cache_ttl: 3600 + ``` + +4. **Optional: External model fallbacks** + - Add Anthropic Claude as fallback for code-heavy requests + - Add OpenAI GPT-4o as fallback for reasoning overflow + - LiteLLM's native fallback chains handle this cleanly + +5. **Router slim-down:** Extract GPU health metrics to dedicated /health only + - Keep: routing, slots, health checks, performance recording + - Remove: key management, dual-key logic, admin endpoints + +--- + +## 6. Nginx Configuration + +The existing nginx config routes `/admin/` → router :9000. This MUST change: + +```nginx +# OLD (remove) +# location /admin/ { +# proxy_pass http://127.0.0.1:9000/admin/; +# } + +# NEW +location /ui/ { + proxy_pass http://127.0.0.1:4000/ui/; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; +} + +location /v1/ { + # Primary: LiteLLM gateway + proxy_pass http://127.0.0.1:4000/v1/; + proxy_set_header Host $host; + proxy_read_timeout 600s; + # Fallback: direct router (if LiteLLM down) + # error_page 502 = @router_fallback; +} + +location @router_fallback { + proxy_pass http://127.0.0.1:9000/v1/; +} + +# Keep router metrics accessible (not behind LiteLLM) +location /router/ { + proxy_pass http://127.0.0.1:9000/; + # Rewrite /router/stream → :9000/stream + # Rewrite /router/metrics → :9000/metrics +} + +# Health check — combines both layers +location /health { + # Check LiteLLM first, then router + proxy_pass http://127.0.0.1:4000/health; +} +``` + +--- + +## 7. Docker Compose (`docker-compose.yml` on docker-vm) + +```yaml +services: + # Layer 1: LiteLLM Gateway (Policy & Admin) + litellm: + image: ghcr.io/berriai/litellm:main-stable + network_mode: "host" + volumes: + - ./config.yaml:/app/config.yaml:ro + environment: + - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY} + - UI_USERNAME=admin + - UI_PASSWORD=${UI_PASSWORD} + - DATABASE_URL=postgresql://litellm:${POSTGRES_PASSWORD}@localhost:5432/litellm + - STORE_MODEL_IN_DB=True + - ROUTER_API_KEY=${ROUTER_API_KEY} + command: + - --config + - /app/config.yaml + - --port + - "4000" + depends_on: + postgres: + condition: service_healthy + restart: unless-stopped + + # Database for LiteLLM + postgres: + image: postgres:16-alpine + network_mode: "host" + environment: + - POSTGRES_DB=litellm + - POSTGRES_USER=litellm + - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} + volumes: + - pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U litellm"] + interval: 5s + timeout: 3s + retries: 5 + restart: unless-stopped + + # Layer 2: Custom Router (Intelligence & Hardware) + # Already deployed separately — not in this compose file + # The router is managed by the existing harness deployment on CT 116 + +volumes: + pgdata: +``` + +--- + +## 8. Risk Mitigation + +| Risk | Mitigation | +|------|------------| +| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable for admin features. LiteLLM is a thin proxy. | +| LiteLLM down = all agents down | Nginx fallback to router :9000 direct (see §6). Agents can also be configured with dual endpoints. | +| Key sync drift (LiteLLM keys ≠ router keys) | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` from LiteLLM's perspective. Agent keys live in LiteLLM only. | +| Spend tracking inaccurate for local GPUs | Configure `model_cost` per GPU with $0 rate (self-hosted). Optionally track "internal cost" via custom pricing. | +| Double rate limiting (LiteLLM + Router) | Keep both intentionally: LiteLLM for per-user soft caps, Router for hardware protection. Non-overlapping concerns. | +| PostgreSQL failure | LiteLLM can run with SQLite fallback, but UI features degrade. Postgres is the recommended path. | +| Router custom logic becomes a black box to LiteLLM | Acceptable trade-off. LiteLLM sees router as opaque OpenAI endpoint. GPU-level routing decisions are router's domain. | + +--- + +## 9. Success Metrics + +| Metric | Before | After | +|--------|--------|-------| +| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy | +| Spend visibility | None | Per-agent, per-team, per-model $ tracking | +| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped | +| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key | +| Admin UX | curl + JSON responses | Visual dashboard, graphs, search | +| Audit trail | Router logs (stdout only) | Database-backed with UI search | +| SSO | None | Google/GitHub/Microsoft OIDC | +| Budget enforcement | None | Automatic: key suspended at $limit | +| GPU routing intelligence | Full (unchanged) | Full (unchanged) | +| GPU health monitoring | Full (unchanged) | Full (unchanged) | + +--- + +## 10. Migration Commands (Quick Reference) + +```bash +# On docker-vm (CT 116): + +# 1. Deploy LiteLLM stack +cd /opt/litellm +docker compose down -v # Clean slate +docker compose up -d # Postgres + LiteLLM + +# 2. Verify +curl http://localhost:4000/health +curl http://localhost:4000/ui # Admin dashboard + +# 3. Create first virtual key via UI or CLI +docker compose exec litellm litellm-proxy keys create \ + --key-alias "abiba-test" \ + --models "syslog-auto" \ + --max-budget 10.0 \ + --team-id "core-agents" + +# 4. Test end-to-end +curl -X POST http://localhost:4000/v1/chat/completions \ + -H "Authorization: Bearer " \ + -d '{"model":"syslog-auto","messages":[{"role":"user","content":"Hello"}]}' + +# 5. Update nginx (see §6) +nginx -t && nginx -s reload + +# 6. Monitor both layers +curl http://localhost:4000/global/spend/logs # LiteLLM spend +curl http://localhost:9000/metrics # Router GPU metrics +curl http://localhost:9000/stream # Router SSE dashboard +``` + +--- + +## Appendix A: Router Slim-Down (Phase 3) + +After full migration, `router-fixed.py` can be simplified by removing: + +```python +# REMOVE (migrated to LiteLLM): +- API_KEYS validation logic (keep single ROUTER_API_KEY) +- Dual-key deprecation tracking +- /admin/keys, /admin/keys/generate, /admin/keys/revoke +- /admin/keys/deprecation-summary +- Phase 0 deprecated key logging +- check_rate_limit() (optional — keep as hardware safety net) + +# KEEP: +- route() — all 5 tiers +- select_best_gpu() +- check_gpu_health() +- is_gpu_busy(), gpu_active_count(), gpu_incr/decr() +- estimate_tokens() +- store_perf_record() +- GPU_SIDECARS, GPU_URLS, GPU_MAX_CONCURRENT, GPU_CONTEXT +- counter_audit_loop() +- /v1/chat/completions — core routing endpoint +- /v1/models +- /health +- /metrics, /metrics/performance, /metrics/scatter, /metrics/timeseries +- /stream — SSE dashboard +``` + +## Appendix B: LiteLLM Cost Config for Local GPUs + +```yaml +# In config.yaml — map models to per-token pricing for spend tracking +litellm_settings: + model_cost: + qwen3.6-35B-A3B: + input_cost_per_token: 0.0 # Self-hosted, no external cost + output_cost_per_token: 0.0 + qwen3.6-27B-code: + input_cost_per_token: 0.0 + output_cost_per_token: 0.0 + gemma-4-12b: + input_cost_per_token: 0.0 + output_cost_per_token: 0.0 + # For internal cost allocation, set symbolic rates: + # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens +``` + +--- + +*Plan drafted: 2026-06-13 by Abiba 🦊⚡* +*Status: Ready for Kwame review*