# LiteLLM Integration Migration Plan ## Syslog Solution LLC June 14, 2026 **Deployment Target:** CT 116 `syslog-api` (192.168.68.116) on minipve all services co-located. **DNS Strategy:** Option A /etc/hosts + Docker extra_hosts for internal resolution of `auth.sysloggh.net` 192.168.68.11. **GitOps:** This plan lives in `SyslogSolution/syslog-harness` on Gitea. All changes tracked via git with conventional commits. --- ## Executive Summary **Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring. **Architecture Decision:** Two-layer architecture. ``` LiteLLM Gateway (Layer 1) Port 4000 Policy & UX Admin UI (/ui) Virtual Keys & Permissions Teams, Users, SSO (OIDC) Spend Tracking & Budgets Usage Analytics Dashboard Request Audit Trail Global Rate Limiting Pass-through to router Custom Router (Layer 2) Port 9000 Intelligence & HW 5-Tier Content-Based Routing GPU Slot Management (Redis) Agent Spread Prevention GPU Health Scoring Sidecar VRAM/Temp/Power Circuit Breaker Context Window Tracking Per-Request Perf Recording Hardware Rate Limiting qwen3.6-35B qwen3.6-27B gemma-4-12b MoE/Strix Dense/RTX3090 VLM/RTX 5070 :8080 (llama) :8080 (llama) :8080 (llama) :8090 (side) :8090 (side) :8090 (sidecar) ``` --- ## 1. Current State Baseline ### 1.1 Router (`router-fixed.py` port 9000, deployed on CT 116 / syslog-api) **Deployment Host:** CT 116 `syslog-api` on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host. | Feature | Implementation | |---------|---------------| | **Routing Engine** | 5-tier content-based: lightweight simple_conv medium heavy_reasoning default | | **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset | | **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health | | **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs | | **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 | | **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation | | **Performance** | Per-request latency/tokens/tps Redis lists (perf:recent, perf:model:X, perf:agent:X) | | **Context Tracking** | Session-level token accumulation with compaction warnings in headers | | **SSE Streaming** | Real-time dashboard updates, per-model timeseries | | **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` | | **Strict Passthrough** | Explicit model requests go to that GPU exactly (no silent fallback). LiteLLM owns failover. | ### 1.2 GPU Backends | GPU | Host | llama.cpp | Sidecar | VRAM | Context | |-----|------|-----------|---------|------|---------| | qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K | | qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K | | gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K | ### 1.3 Existing LiteLLM POC on CT 116 CT 116 already has a LiteLLM container running (POC, 6 days uptime): ``` harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081->4000 harness-redis | redis:7-alpine | 127.0.0.1:6379 harness-router | inference-harness-router | 127.0.0.1:9000 harness-nginx | nginx:alpine | 0.0.0.0:80 harness-dashboard | inference-harness-dashboard | 127.0.0.1:3000 ``` - `/opt/litellm/` previous setup directory on CT 116 - Configured with Postgres, host networking, master key - Currently bypassed router routes directly to GPUs - **Goal: Productionize with two-layer architecture on this same host** ### 1.4 DNS Routing (Split-Horizon) For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally: **Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net` 72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily. **Solution Option A: /etc/hosts on CT 116 host:** ```bash # On CT 116 (syslog-api) echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts ``` **Docker containers** also need this resolution add to docker-compose.yml: ```yaml services: nginx: extra_hosts: - "auth.sysloggh.net:192.168.68.11" litellm: extra_hosts: - "auth.sysloggh.net:192.168.68.11" ``` **DNS Servers:** CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS. --- ## 2. What LiteLLM Brings (That We Don't Have) | Feature | Our Router | LiteLLM | Value Add | |---------|-----------|---------|-----------| | **Admin UI** | | Full dashboard at /ui | Non-technical users can manage keys, view spend | | **Virtual Key Permissions** | (binary key->tier) | Granular: per-model, per-team, budget caps | Fine-grained access control | | **Spend Tracking** | | Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing | | **Teams & Orgs** | | Multi-tenant: org->team->user hierarchy | Segregate clients/projects | | **SSO/OIDC** | | Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration | | **Budget Alerts** | | Per-key, per-user, per-team budget with webhooks | Prevent overspend | | **Usage Analytics** | (custom /metrics) | Built-in: daily trends, model breakdown, per-customer | Better visualization | | **100+ Provider Support** | (3 local GPUs) | OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access | | **Fallback Chains** | (silent rerouting) | Explicit multi-provider failover with per-model logging | Accurate per-model tracking, visible failover | | **RPM/TPM Weighted LB** | | Weighted load balancing across deployments | Fine-grained traffic shaping | --- ## 3. What We Keep (That LiteLLM Doesn't Have) | Feature | Why We Must Keep It | |---------|---------------------| | **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints | | **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power our scoring prevents routing to overheating GPUs | | **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading | | **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle | | **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers | | **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature exposed via /metrics and SSE dashboard | | **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular | --- ## 4. Migration Architecture ### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room" - **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI - **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking ### 4.2 Flow ``` Agent Request LiteLLM Gateway (:4000) 1. Authenticate virtual key (sk-litellm-...) 2. Check key permissions (model access) 3. Check budget (per-key, per-user, per-team) 4. Check team rate limits 5. Log request metadata 6. Forward to custom router as OpenAI-compat POST http://router:9000/v1/chat/completions Headers: Authorization: Bearer *** X-LiteLLM-User: X-LiteLLM-Team: X-Session-Id: 7. On response: log spend, update budgets 8. If router returns 503 (GPU saturated): consult fallback chain, retry next model 9. Return response to agent Custom Router (:9000) 1. Authenticate agent key (sk-syslog-...) 2. Hardware rate limit (per-tier RPM) 3. Content-based tier routing (for syslog-auto) OR strict passthrough (for explicit models) 4. GPU slot availability (Redis counter) 5. GPU health check (sidecar) 6. Agent spread logic (select_best_gpu) 7. Queue if saturated (with timeout) 8. Forward to selected llama.cpp GPU 9. Track context window, set compaction header 10. Record performance metrics 11. Return response (with routing metadata) llama.cpp GPU (:8080) ``` ### 4.3 LiteLLM Config (`config.yaml`) ```yaml general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: postgresql://litellm:***@postgres:5432/litellm store_model_in_db: true model_list: # Content-based auto-routing (router picks GPU via 5-tier analysis) - model_name: syslog-auto litellm_params: model: openai/syslog-auto api_base: http://router:9000/v1 api_key: os.environ/ROUTER_API_KEY rpm: 600 # Individual GPU strict passthrough (exact GPU, no silent fallback) - model_name: qwen3.6-35B-A3B litellm_params: model: openai/qwen3.6-35B-A3B api_base: http://router:9000/v1 api_key: os.environ/ROUTER_API_KEY - model_name: qwen3.6-27B-code litellm_params: model: openai/qwen3.6-27B-code api_base: http://router:9000/v1 api_key: os.environ/ROUTER_API_KEY - model_name: gemma-4-12b litellm_params: model: openai/gemma-4-12b api_base: http://router:9000/v1 api_key: os.environ/ROUTER_API_KEY # Guardrails: Pre-call and post-call content moderation guardrails: - guardrail_name: "input-moderation" litellm_params: guardrail: openai_moderation mode: "pre_call" - guardrail_name: "output-moderation" litellm_params: guardrail: openai_moderation mode: "post_call" - guardrail_name: "harmful-content-filter" litellm_params: guardrail: litellm_content_filter mode: "pre_call" categories: - category: "harmful_self_harm" enabled: true action: "BLOCK" severity_threshold: "medium" - category: "harmful_violence" enabled: true action: "BLOCK" severity_threshold: "medium" - category: "harmful_illegal_weapons" enabled: true action: "BLOCK" severity_threshold: "medium" litellm_settings: num_retries: 0 # Disabled our router handles retry request_timeout: 600 # Match our 10-min llama-server timeout set_verbose: true failure_callback: ["prometheus"] # Optional: export to Prometheus router_settings: routing_strategy: "usage-based-routing" # For external models only enable_loadbalancing_on_proxy: false # Disable LiteLLM's internal LB allowed_fails: 100 # Router returns 503 on saturated GPUs cooldown disabled # Fallback chains: LiteLLM retries down the chain when router returns saturated # This gives accurate per-model metrics because router no longer silently reroutes fallbacks: - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"] - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"] - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"] # Cost tracking: map model names to per-token pricing for spend tracking litellm_settings: model_cost: qwen3.6-35B-A3B: input_cost_per_token: 0.0 output_cost_per_token: 0.0 qwen3.6-27B-code: input_cost_per_token: 0.0 output_cost_per_token: 0.0 gemma-4-12b: input_cost_per_token: 0.0 output_cost_per_token: 0.0 # For internal cost allocation, set symbolic rates: # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens ``` ### 4.4 Router Modifications To accommodate LiteLLM, `router-fixed.py` requires the following updates: 1. **Strict passthrough for explicit models** (DEPLOYED): ```python # In route(), the explicit model section changed from silent fallback to strict: req = rd.get("model","auto") if req != "auto": # STRICT MODE: no silent fallback LiteLLM handles failover chains. # This keeps per-model metrics accurate. Returns saturated if busy. target = req if req in avail else avail[0] if req not in avail: return {"model": req, "reason": "explicit_unavailable", "saturated": True} if is_gpu_busy(target): return {"model": target, "reason": "explicit_saturated", "saturated": True} return {"model": target, "reason": "explicit"} ``` 2. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent already works) 3. **New endpoint for health passthrough**: `GET /v1/models` already works 4. **Keep ALL routing logic**: No changes to `select_best_gpu()`, `check_gpu_health()`, slot management, etc. Content-based routing for `syslog-auto` is fully intact. 5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs ```python resp.headers["X-Usage-Tokens"] = json.dumps({ "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens, "model": model }) ``` ### 4.5 Router Logic Refinements **GPU Health Scoring (Updated):** We are updating the scoring algorithm to include Power metrics: ```python def gpu_health_score(model): h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1) if h.get("status") == "down": return 999 # never pick down GPUs vram_pct = h.get("vram_pct") or 50 temp_c = h.get("temp_c") or 50 power_w = h.get("power_w") or 50 active = gpu_active_count(model) max_c = GPU_MAX_CONCURRENT.get(model, 1) load_pct = (active / max_c) * 100 if max_c > 0 else 0 # Score: lower = better score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2)) return score ``` --- ## 5. Deployment Plan (4 Phases) Zero-Downtime Strategy ### Phase 0: Infrastructure Prep (Current Week) Zero Downtime **Goal:** Prepare CT 116 infrastructure without affecting running agents. **Tasks:** 1. **Set up DNS split-horizon on CT 116** ```bash echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts ``` 2. **Deploy Postgres container** alongside existing services 3. **Replace LiteLLM config** with production config.yaml (see 4.3) - All 4 models `http://router:9000/v1` - Fallback chains for explicit models - Guardrails (pre-call, post-call, content filter) - `allowed_fails: 100` (router returns 503 on saturated) - `num_retries: 0` (LiteLLM retries handled by fallback chains) 4. **Deploy custom_sso.py** for Authentik OIDC integration 5. **Restart LiteLLM container** with new config 6. **Verify internal routing** **Verification Checklist:** - [ ] DNS resolution: `getent hosts auth.sysloggh.net` 192.168.68.11 - [ ] Postgres container healthy - [ ] LiteLLM `/health` returns 200 - [ ] LiteLLM Router pass-through returns valid chat completion - [ ] GPU health metrics unaffected - [ ] Explicit model request returns saturated (not silently rerouted) when GPU busy ### Phase 1: Shadow Mode (Week 1) Zero Risk, Zero Downtime **Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. **Agents continue using :9000 directly.** **Tasks:** 1. Create virtual keys for test agents via LiteLLM UI 2. Verify pass-through works for all 4 models 3. Validate fallback chains: saturate MoE confirm LiteLLM retries Dense confirm VLM 4. Run 24-hour shadow: monitor LiteLLM spend logs vs router metrics 5. Verify GPU health metrics unaffected 6. Check guardrails not generating false positives ### Phase 2: Cutover (Week 2) Gradual Agent Migration **Goal:** Move agents one-by-one to LiteLLM endpoint. **Tasks:** 1. Migrate API keys to LiteLLM virtual keys 2. Create teams: "Core Agents" (enterprise), "Dev Agents" (professional) 3. Update agent configs one at a time: `OPENAI_API_BASE` `:4000` 4. Test each agent individually 5. Enable SSO via Authentik + custom_sso.py 6. Keep router :9000 as emergency fallback for 48 hours ### Phase 3: Production Hardening (Week 3+) **Goal:** Lock down, optimize, monitor. **Tasks:** 1. Remove deprecated router endpoints (after all agents migrated) 2. Add LiteLLM observability (Prometheus, Slack/email alerts) 3. Enable LiteLLM caching (shared Redis) 4. Add external model fallbacks for client-facing services 5. Router slim-down: keep routing/slots/health/perf, remove key management 6. Multi-tenancy setup for client-facing inference services --- ## 6. Nginx Configuration (with Authentik OIDC Forward Auth) ```nginx # OLD (remove) # location /admin/ { # proxy_pass http://127.0.0.1:9000/admin/; # } # === Authentik auth subrequest endpoint === location /authentik/auth { internal; proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx; proxy_pass_request_body off; proxy_set_header Content-Length ""; proxy_set_header X-Original-URL $scheme://$http_host$request_uri; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } # === LiteLLM Admin UI Authentik-protected === location /ui/ { auth_request /authentik/auth; auth_request_set $auth_user $upstream_http_x_authentik_username; auth_request_set $auth_email $upstream_http_x_authentik_email; proxy_set_header X-Authentik-Username $auth_user; proxy_set_header X-Authentik-Email $auth_email; proxy_pass http://127.0.0.1:4000/ui/; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; } # === LiteLLM SSO callback === location /sso/callback { proxy_pass http://127.0.0.1:4000/sso/callback; proxy_set_header Host $host; } # === API endpoint Bearer token auth === location /v1/ { proxy_pass http://127.0.0.1:4000/v1/; proxy_set_header Host $host; proxy_read_timeout 600s; error_page 502 = @router_fallback; } location @router_fallback { proxy_pass http://127.0.0.1:9000/v1/; proxy_set_header Host $host; } # === Key management API === location /key/ { proxy_pass http://127.0.0.1:4000/key/; proxy_set_header Host $host; proxy_set_header Authorization $http_authorization; } # Keep router metrics accessible (not behind LiteLLM) location /router/ { proxy_pass http://127.0.0.1:9000/; } location /health { proxy_pass http://127.0.0.1:4000/health; } ``` --- ## 7. Docker Compose (`docker-compose.yml` on CT 116) ```yaml services: # Layer 1: LiteLLM Gateway (Policy & Admin) litellm: image: ghcr.io/berriai/litellm:main-stable network_mode: "host" extra_hosts: - "auth.sysloggh.net:192.168.68.11" volumes: - ./config.yaml:/app/config.yaml:ro - ./custom_sso.py:/app/custom_sso.py:ro environment: - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY} - DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm - STORE_MODEL_IN_DB=True - ROUTER_API_KEY=${ROUTER_API_KEY} - OPENAI_API_KEY=*** - ANTHROPIC_API_KEY=${ANTH...KEY} - PROXY_BASE_URL=https://litellm.sysloggh.net command: - --config - /app/config.yaml - --port - "4000" depends_on: postgres: condition: service_healthy restart: unless-stopped postgres: image: postgres:16-alpine network_mode: "host" environment: - POSTGRES_DB=litellm - POSTGRES_USER=litellm - POSTGRES_PASSWORD=${POST...} volumes: - pgdata:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U litellm"] interval: 5s timeout: 3s retries: 5 restart: unless-stopped nginx: image: nginx:alpine extra_hosts: - "auth.sysloggh.net:192.168.68.11" volumes: - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro ports: - "80:80" restart: unless-stopped volumes: pgdata: ``` --- ## 8. Risk Mitigation | Risk | Mitigation | |------|------------| | LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable | | LiteLLM down = all agents down | NGINX fallback to router :9000 direct (see 6) | | Explicit GPU saturated no fallback available | LiteLLM fallback chains try all 3 GPUs in order before failing | | Fallback chain masking real GPU failures | Router returns `saturated: true` only for capacity, `down` returns different error | | Key sync drift | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` | | Spend tracking inaccurate for local GPUs | `model_cost` per GPU with $0 rate; optional symbolic pricing for internal billing | | Double rate limiting | Intentional: LiteLLM for per-user caps, Router for hardware protection | | PostgreSQL failure | LiteLLM can run with SQLite fallback; UI features degrade | | Per-model metrics accuracy with syslog-auto | `syslog-auto` is opaque by design (content-based routing). Explicit models are accurate. Use explicit models for per-GPU billing. | --- ## 9. Success Metrics | Metric | Before | After | |--------|--------|-------| | Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy | | Spend visibility | None | Per-agent, per-team, per-model $ tracking | | Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped | | New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key | | Admin UX | curl + JSON responses | Visual dashboard, graphs, search | | Audit trail | Router logs (stdout only) | Database-backed with UI search | | SSO | None | Authentik OIDC | | Budget enforcement | None | Automatic: key suspended at $limit | | GPU failover | Silent (inaccurate metrics) | Explicit (LiteLLM fallback chains, per-model logs) | | GPU routing intelligence | Full (unchanged) | Full (unchanged) | --- ## 10. Migration Commands (Quick Reference) ```bash # On CT 116 (SSH via minipve: pct exec 116 bash): # Phase 0: Infrastructure Prep echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts cd /opt/litellm docker compose up -d postgres # Replace config.yaml with production version (see 4.3) docker compose restart litellm # Verify curl http://127.0.0.1:4000/health curl -X POST http://127.0.0.1:4000/v1/chat/completions \ -H "Authorization: Bearer ***" \ -H "Content-Type: application/json" \ -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}' ``` --- ## 11. GitOps Workflow - `main` production-ready code - `feature/litellm-migration` current development branch **Conventional Commits:** ``` feat(plan): add fallback chains and strict passthrough for model identity gap fix(router): strict passthrough for explicit models no silent rerouting feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC docs: LiteLLM migration plan two-layer architecture with model identity gap analysis ``` --- ## 12. Zero-Downtime Migration Strategy **Per-Agent Cutover (<2 minutes):** 1. Create LiteLLM virtual key in UI 2. Update agent's `OPENAI_API_BASE` to `:4000` 3. Verify routing works 4. Monitor LiteLLM logs for errors **Global Rollback:** 1. If LiteLLM :4000 fails, revert all agents to `:9000` 2. NGINX `router_fallback` handles automatic failover 3. Monitor GPU metrics for health checks --- ## Appendix A: Model Identity Gap Analysis (RESOLVED) ### Problem Identified (Abiba, June 14) The original architecture had a metrics accuracy gap: when an agent requested `qwen3.6-35B-A3B` and MoE was busy, the router silently rerouted to Dense. LiteLLM logged it as MoE usage, corrupting per-model spend/usage tracking. ### Root Cause ```python # OLD router code (router-fixed.py): if is_gpu_busy(target) and req in allowed: alts = [m for m in avail if m != target and m in allowed] if alts: alt = select_best_gpu(alts, "explicit", agent) if alt: return alt # silently changed GPU ``` ### Resolution: Strict Passthrough + LiteLLM Fallback Chains Two changes deployed: **1. Router strict passthrough:** ```python # NEW: strict mode no silent fallback if req != "auto": target = req if req in avail else avail[0] if req not in avail: return {"model": req, "reason": "explicit_unavailable", "saturated": True} if is_gpu_busy(target): return {"model": target, "reason": "explicit_saturated", "saturated": True} return {"model": target, "reason": "explicit"} ``` **2. LiteLLM fallback chains (in config.yaml):** ```yaml router_settings: allowed_fails: 100 fallbacks: - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"] - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"] - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"] ``` ### Result | Scenario | Before | After | |----------|--------|-------| | Agent asks for MoE, MoE available | MoE used, metrics OK | MoE used, metrics OK | | Agent asks for MoE, MoE busy | Router Dense silently, metrics WRONG | Router 503, LiteLLM Dense, metrics show BOTH attempts | | Agent uses syslog-auto | Router picks GPU, LiteLLM sees opaque | Same (syslog-auto is opaque by design) | | All 3 GPUs saturated | Router queues (30s), then 503 | Same, LiteLLM sees 503 after fallback chain exhausted | --- ## Appendix B: LiteLLM Virtual Key Migration | Agent | Old Key | New LiteLLM Key | Tier | Budget | |-------|---------|-----------------|------|--------| | Abiba | sk-***-*** | sk-litellm-*** | enterprise | $1000 | | Mumuni | sk-***-*** | sk-litellm-*** | enterprise | $1000 | | Tanko | sk-***-*** | sk-litellm-*** | enterprise | $1000 | | Kagenz0 | sk-***-*** | sk-litellm-*** | professional | $500 | | Koby | sk-***-*** | sk-litellm-*** | professional | $500 | | Koonimo | sk-***-*** | sk-litellm-*** | professional | $500 | --- ## Appendix C: Authentik SSO Integration **Authentik Provider Setup:** 1. Create OAuth2 application in Authentik 2. Set redirect URI: `http:///sso/callback` 3. Configure `client_id` and `client_secret` 4. Mount `custom_sso.py` to LiteLLM container 5. Update config.yaml with provider details --- ## Appendix D: Prometheus Monitoring **Metrics Export:** - LiteLLM metrics `http://127.0.0.1:4000/metrics` - Router metrics `http://127.0.0.1:9000/metrics` - GPU health metrics `http://127.0.0.1:9000/metrics/gpu` **Alerts:** - GPU health score > 70 alert - Circuit breaker trip alert - LiteLLM spend > $100/day alert - LiteLLM latency > 1000ms alert