SyslogSolution/syslog-harness
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ac13ecaaf7d3d7d5f8002a4665a1add9de857254
syslog-harness/LITELLM-MIGRATION-PLAN.md
T
Abiba 776343f2ab feat(plan): add fallback chains and resolve model identity gap 
- Added LiteLLM fallback chains for explicit GPU models
- Changed allowed_fails: 0 -> 100 (router returns 503 on saturated)
- Documented strict passthrough router change (already deployed)
- Rewrote Appendix A as actual Model Identity Gap Analysis
- Added risk mitigation for fallback chain masking real failures
- Updated success metrics to reflect accurate per-model tracking
- Reviewed and approved by Mumuni and Kagenz0
2026-06-14 22:48:53 +00:00

760 lines
27 KiB
Markdown
Raw Blame History

# LiteLLM Integration Migration Plan
## Syslog Solution LLC June 14, 2026
**Deployment Target:** CT 116 `syslog-api` (192.168.68.116) on minipve all services co-located.
**DNS Strategy:** Option A /etc/hosts + Docker extra_hosts for internal resolution of `auth.sysloggh.net` 192.168.68.11.
**GitOps:** This plan lives in `SyslogSolution/syslog-harness` on Gitea. All changes tracked via git with conventional commits.
---
## Executive Summary
**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.
**Architecture Decision:** Two-layer architecture.
```
LiteLLM Gateway (Layer 1)
Port 4000 Policy & UX
Admin UI (/ui)
Virtual Keys & Permissions
Teams, Users, SSO (OIDC)
Spend Tracking & Budgets
Usage Analytics Dashboard
Request Audit Trail
Global Rate Limiting
Pass-through to router
Custom Router (Layer 2)
Port 9000 Intelligence & HW
5-Tier Content-Based Routing
GPU Slot Management (Redis)
Agent Spread Prevention
GPU Health Scoring
Sidecar VRAM/Temp/Power
Circuit Breaker
Context Window Tracking
Per-Request Perf Recording
Hardware Rate Limiting
qwen3.6-35B qwen3.6-27B gemma-4-12b
MoE/Strix Dense/RTX3090 VLM/RTX 5070
:8080 (llama) :8080 (llama) :8080 (llama)
:8090 (side) :8090 (side) :8090 (sidecar)
```
---
## 1. Current State Baseline
### 1.1 Router (`router-fixed.py` port 9000, deployed on CT 116 / syslog-api)
**Deployment Host:** CT 116 `syslog-api` on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.
| Feature | Implementation |
|---------|---------------|
| **Routing Engine** | 5-tier content-based: lightweight simple_conv medium heavy_reasoning default |
| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset |
| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health |
| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs |
| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 |
| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation |
| **Performance** | Per-request latency/tokens/tps Redis lists (perf:recent, perf:model:X, perf:agent:X) |
| **Context Tracking** | Session-level token accumulation with compaction warnings in headers |
| **SSE Streaming** | Real-time dashboard updates, per-model timeseries |
| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` |
| **Strict Passthrough** | Explicit model requests go to that GPU exactly (no silent fallback). LiteLLM owns failover. |
### 1.2 GPU Backends
| GPU | Host | llama.cpp | Sidecar | VRAM | Context |
|-----|------|-----------|---------|------|---------|
| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K |
| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K |
| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K |
### 1.3 Existing LiteLLM POC on CT 116
CT 116 already has a LiteLLM container running (POC, 6 days uptime):
```
harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081->4000
harness-redis | redis:7-alpine | 127.0.0.1:6379
harness-router | inference-harness-router | 127.0.0.1:9000
harness-nginx | nginx:alpine | 0.0.0.0:80
harness-dashboard | inference-harness-dashboard | 127.0.0.1:3000
```
- `/opt/litellm/` previous setup directory on CT 116
- Configured with Postgres, host networking, master key
- Currently bypassed router routes directly to GPUs
- **Goal: Productionize with two-layer architecture on this same host**
### 1.4 DNS Routing (Split-Horizon)
For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally:
**Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net` 72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.
**Solution Option A: /etc/hosts on CT 116 host:**
```bash
# On CT 116 (syslog-api)
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
```
**Docker containers** also need this resolution add to docker-compose.yml:
```yaml
services:
nginx:
extra_hosts:
- "auth.sysloggh.net:192.168.68.11"
litellm:
extra_hosts:
- "auth.sysloggh.net:192.168.68.11"
```
**DNS Servers:** CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.
---
## 2. What LiteLLM Brings (That We Don't Have)
| Feature | Our Router | LiteLLM | Value Add |
|---------|-----------|---------|-----------|
| **Admin UI** | | Full dashboard at /ui | Non-technical users can manage keys, view spend |
| **Virtual Key Permissions** | (binary key->tier) | Granular: per-model, per-team, budget caps | Fine-grained access control |
| **Spend Tracking** | | Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing |
| **Teams & Orgs** | | Multi-tenant: org->team->user hierarchy | Segregate clients/projects |
| **SSO/OIDC** | | Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration |
| **Budget Alerts** | | Per-key, per-user, per-team budget with webhooks | Prevent overspend |
| **Usage Analytics** | (custom /metrics) | Built-in: daily trends, model breakdown, per-customer | Better visualization |
| **100+ Provider Support** | (3 local GPUs) | OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access |
| **Fallback Chains** | (silent rerouting) | Explicit multi-provider failover with per-model logging | Accurate per-model tracking, visible failover |
| **RPM/TPM Weighted LB** | | Weighted load balancing across deployments | Fine-grained traffic shaping |
---
## 3. What We Keep (That LiteLLM Doesn't Have)
| Feature | Why We Must Keep It |
|---------|---------------------|
| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints |
| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power our scoring prevents routing to overheating GPUs |
| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading |
| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle |
| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers |
| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature exposed via /metrics and SSE dashboard |
| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular |
---
## 4. Migration Architecture
### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room"
- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI
- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking
### 4.2 Flow
```
Agent Request
LiteLLM Gateway (:4000)
1. Authenticate virtual key (sk-litellm-...)
2. Check key permissions (model access)
3. Check budget (per-key, per-user, per-team)
4. Check team rate limits
5. Log request metadata
6. Forward to custom router as OpenAI-compat
POST http://router:9000/v1/chat/completions
Headers: Authorization: Bearer ***
X-LiteLLM-User: <user-id>
X-LiteLLM-Team: <team-id>
X-Session-Id: <session>
7. On response: log spend, update budgets
8. If router returns 503 (GPU saturated):
consult fallback chain, retry next model
9. Return response to agent
Custom Router (:9000)
1. Authenticate agent key (sk-syslog-...)
2. Hardware rate limit (per-tier RPM)
3. Content-based tier routing (for syslog-auto)
OR strict passthrough (for explicit models)
4. GPU slot availability (Redis counter)
5. GPU health check (sidecar)
6. Agent spread logic (select_best_gpu)
7. Queue if saturated (with timeout)
8. Forward to selected llama.cpp GPU
9. Track context window, set compaction header
10. Record performance metrics
11. Return response (with routing metadata)
llama.cpp GPU (:8080)
```
### 4.3 LiteLLM Config (`config.yaml`)
```yaml
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: postgresql://litellm:***@postgres:5432/litellm
store_model_in_db: true
model_list:
# Content-based auto-routing (router picks GPU via 5-tier analysis)
- model_name: syslog-auto
litellm_params:
model: openai/syslog-auto
api_base: http://router:9000/v1
api_key: os.environ/ROUTER_API_KEY
rpm: 600
# Individual GPU strict passthrough (exact GPU, no silent fallback)
- model_name: qwen3.6-35B-A3B
litellm_params:
model: openai/qwen3.6-35B-A3B
api_base: http://router:9000/v1
api_key: os.environ/ROUTER_API_KEY
- model_name: qwen3.6-27B-code
litellm_params:
model: openai/qwen3.6-27B-code
api_base: http://router:9000/v1
api_key: os.environ/ROUTER_API_KEY
- model_name: gemma-4-12b
litellm_params:
model: openai/gemma-4-12b
api_base: http://router:9000/v1
api_key: os.environ/ROUTER_API_KEY
# Guardrails: Pre-call and post-call content moderation
guardrails:
- guardrail_name: "input-moderation"
litellm_params:
guardrail: openai_moderation
mode: "pre_call"
- guardrail_name: "output-moderation"
litellm_params:
guardrail: openai_moderation
mode: "post_call"
- guardrail_name: "harmful-content-filter"
litellm_params:
guardrail: litellm_content_filter
mode: "pre_call"
categories:
- category: "harmful_self_harm"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "harmful_violence"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
- category: "harmful_illegal_weapons"
enabled: true
action: "BLOCK"
severity_threshold: "medium"
litellm_settings:
num_retries: 0 # Disabled our router handles retry
request_timeout: 600 # Match our 10-min llama-server timeout
set_verbose: true
failure_callback: ["prometheus"] # Optional: export to Prometheus
router_settings:
routing_strategy: "usage-based-routing" # For external models only
enable_loadbalancing_on_proxy: false # Disable LiteLLM's internal LB
allowed_fails: 100 # Router returns 503 on saturated GPUs cooldown disabled
# Fallback chains: LiteLLM retries down the chain when router returns saturated
# This gives accurate per-model metrics because router no longer silently reroutes
fallbacks:
- qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
- qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
- gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
# Cost tracking: map model names to per-token pricing for spend tracking
litellm_settings:
model_cost:
qwen3.6-35B-A3B:
input_cost_per_token: 0.0
output_cost_per_token: 0.0
qwen3.6-27B-code:
input_cost_per_token: 0.0
output_cost_per_token: 0.0
gemma-4-12b:
input_cost_per_token: 0.0
output_cost_per_token: 0.0
# For internal cost allocation, set symbolic rates:
# e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
```
### 4.4 Router Modifications
To accommodate LiteLLM, `router-fixed.py` requires the following updates:
1. **Strict passthrough for explicit models** (DEPLOYED):
```python
# In route(), the explicit model section changed from silent fallback to strict:
req = rd.get("model","auto")
if req != "auto":
# STRICT MODE: no silent fallback LiteLLM handles failover chains.
# This keeps per-model metrics accurate. Returns saturated if busy.
target = req if req in avail else avail[0]
if req not in avail:
return {"model": req, "reason": "explicit_unavailable", "saturated": True}
if is_gpu_busy(target):
return {"model": target, "reason": "explicit_saturated", "saturated": True}
return {"model": target, "reason": "explicit"}
```
2. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent already works)
3. **New endpoint for health passthrough**: `GET /v1/models` already works
4. **Keep ALL routing logic**: No changes to `select_best_gpu()`, `check_gpu_health()`, slot management, etc. Content-based routing for `syslog-auto` is fully intact.
5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs
```python
resp.headers["X-Usage-Tokens"] = json.dumps({
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"model": model
})
```
### 4.5 Router Logic Refinements
**GPU Health Scoring (Updated):**
We are updating the scoring algorithm to include Power metrics:
```python
def gpu_health_score(model):
h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
if h.get("status") == "down":
return 999 # never pick down GPUs
vram_pct = h.get("vram_pct") or 50
temp_c = h.get("temp_c") or 50
power_w = h.get("power_w") or 50
active = gpu_active_count(model)
max_c = GPU_MAX_CONCURRENT.get(model, 1)
load_pct = (active / max_c) * 100 if max_c > 0 else 0
# Score: lower = better
score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2))
return score
```
---
## 5. Deployment Plan (4 Phases) Zero-Downtime Strategy
### Phase 0: Infrastructure Prep (Current Week) Zero Downtime
**Goal:** Prepare CT 116 infrastructure without affecting running agents.
**Tasks:**
1. **Set up DNS split-horizon on CT 116**
```bash
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
```
2. **Deploy Postgres container** alongside existing services
3. **Replace LiteLLM config** with production config.yaml (see 4.3)
- All 4 models `http://router:9000/v1`
- Fallback chains for explicit models
- Guardrails (pre-call, post-call, content filter)
- `allowed_fails: 100` (router returns 503 on saturated)
- `num_retries: 0` (LiteLLM retries handled by fallback chains)
4. **Deploy custom_sso.py** for Authentik OIDC integration
5. **Restart LiteLLM container** with new config
6. **Verify internal routing**
**Verification Checklist:**
- [ ] DNS resolution: `getent hosts auth.sysloggh.net` 192.168.68.11
- [ ] Postgres container healthy
- [ ] LiteLLM `/health` returns 200
- [ ] LiteLLM Router pass-through returns valid chat completion
- [ ] GPU health metrics unaffected
- [ ] Explicit model request returns saturated (not silently rerouted) when GPU busy
### Phase 1: Shadow Mode (Week 1) Zero Risk, Zero Downtime
**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. **Agents continue using :9000 directly.**
**Tasks:**
1. Create virtual keys for test agents via LiteLLM UI
2. Verify pass-through works for all 4 models
3. Validate fallback chains: saturate MoE confirm LiteLLM retries Dense confirm VLM
4. Run 24-hour shadow: monitor LiteLLM spend logs vs router metrics
5. Verify GPU health metrics unaffected
6. Check guardrails not generating false positives
### Phase 2: Cutover (Week 2) Gradual Agent Migration
**Goal:** Move agents one-by-one to LiteLLM endpoint.
**Tasks:**
1. Migrate API keys to LiteLLM virtual keys
2. Create teams: "Core Agents" (enterprise), "Dev Agents" (professional)
3. Update agent configs one at a time: `OPENAI_API_BASE` `:4000`
4. Test each agent individually
5. Enable SSO via Authentik + custom_sso.py
6. Keep router :9000 as emergency fallback for 48 hours
### Phase 3: Production Hardening (Week 3+)
**Goal:** Lock down, optimize, monitor.
**Tasks:**
1. Remove deprecated router endpoints (after all agents migrated)
2. Add LiteLLM observability (Prometheus, Slack/email alerts)
3. Enable LiteLLM caching (shared Redis)
4. Add external model fallbacks for client-facing services
5. Router slim-down: keep routing/slots/health/perf, remove key management
6. Multi-tenancy setup for client-facing inference services
---
## 6. Nginx Configuration (with Authentik OIDC Forward Auth)
```nginx
# OLD (remove)
# location /admin/ {
# proxy_pass http://127.0.0.1:9000/admin/;
# }
# === Authentik auth subrequest endpoint ===
location /authentik/auth {
internal;
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# === LiteLLM Admin UI Authentik-protected ===
location /ui/ {
auth_request /authentik/auth;
auth_request_set $auth_user $upstream_http_x_authentik_username;
auth_request_set $auth_email $upstream_http_x_authentik_email;
proxy_set_header X-Authentik-Username $auth_user;
proxy_set_header X-Authentik-Email $auth_email;
proxy_pass http://127.0.0.1:4000/ui/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# === LiteLLM SSO callback ===
location /sso/callback {
proxy_pass http://127.0.0.1:4000/sso/callback;
proxy_set_header Host $host;
}
# === API endpoint Bearer token auth ===
location /v1/ {
proxy_pass http://127.0.0.1:4000/v1/;
proxy_set_header Host $host;
proxy_read_timeout 600s;
error_page 502 = @router_fallback;
}
location @router_fallback {
proxy_pass http://127.0.0.1:9000/v1/;
proxy_set_header Host $host;
}
# === Key management API ===
location /key/ {
proxy_pass http://127.0.0.1:4000/key/;
proxy_set_header Host $host;
proxy_set_header Authorization $http_authorization;
}
# Keep router metrics accessible (not behind LiteLLM)
location /router/ {
proxy_pass http://127.0.0.1:9000/;
}
location /health {
proxy_pass http://127.0.0.1:4000/health;
}
```
---
## 7. Docker Compose (`docker-compose.yml` on CT 116)
```yaml
services:
# Layer 1: LiteLLM Gateway (Policy & Admin)
litellm:
image: ghcr.io/berriai/litellm:main-stable
network_mode: "host"
extra_hosts:
- "auth.sysloggh.net:192.168.68.11"
volumes:
- ./config.yaml:/app/config.yaml:ro
- ./custom_sso.py:/app/custom_sso.py:ro
environment:
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
- DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm
- STORE_MODEL_IN_DB=True
- ROUTER_API_KEY=${ROUTER_API_KEY}
- OPENAI_API_KEY=***
- ANTHROPIC_API_KEY=${ANTH...KEY}
- PROXY_BASE_URL=https://litellm.sysloggh.net
command:
- --config
- /app/config.yaml
- --port
- "4000"
depends_on:
postgres:
condition: service_healthy
restart: unless-stopped
postgres:
image: postgres:16-alpine
network_mode: "host"
environment:
- POSTGRES_DB=litellm
- POSTGRES_USER=litellm
- POSTGRES_PASSWORD=${POST...}
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm"]
interval: 5s
timeout: 3s
retries: 5
restart: unless-stopped
nginx:
image: nginx:alpine
extra_hosts:
- "auth.sysloggh.net:192.168.68.11"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- "80:80"
restart: unless-stopped
volumes:
pgdata:
```
---
## 8. Risk Mitigation
| Risk | Mitigation |
|------|------------|
| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable |
| LiteLLM down = all agents down | NGINX fallback to router :9000 direct (see 6) |
| Explicit GPU saturated no fallback available | LiteLLM fallback chains try all 3 GPUs in order before failing |
| Fallback chain masking real GPU failures | Router returns `saturated: true` only for capacity, `down` returns different error |
| Key sync drift | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` |
| Spend tracking inaccurate for local GPUs | `model_cost` per GPU with $0 rate; optional symbolic pricing for internal billing |
| Double rate limiting | Intentional: LiteLLM for per-user caps, Router for hardware protection |
| PostgreSQL failure | LiteLLM can run with SQLite fallback; UI features degrade |
| Per-model metrics accuracy with syslog-auto | `syslog-auto` is opaque by design (content-based routing). Explicit models are accurate. Use explicit models for per-GPU billing. |
---
## 9. Success Metrics
| Metric | Before | After |
|--------|--------|-------|
| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy |
| Spend visibility | None | Per-agent, per-team, per-model $ tracking |
| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped |
| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key |
| Admin UX | curl + JSON responses | Visual dashboard, graphs, search |
| Audit trail | Router logs (stdout only) | Database-backed with UI search |
| SSO | None | Authentik OIDC |
| Budget enforcement | None | Automatic: key suspended at $limit |
| GPU failover | Silent (inaccurate metrics) | Explicit (LiteLLM fallback chains, per-model logs) |
| GPU routing intelligence | Full (unchanged) | Full (unchanged) |
---
## 10. Migration Commands (Quick Reference)
```bash
# On CT 116 (SSH via minipve: pct exec 116 bash):
# Phase 0: Infrastructure Prep
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
cd /opt/litellm
docker compose up -d postgres
# Replace config.yaml with production version (see 4.3)
docker compose restart litellm
# Verify
curl http://127.0.0.1:4000/health
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer ***" \
-H "Content-Type: application/json" \
-d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
```
---
## 11. GitOps Workflow
- `main` production-ready code
- `feature/litellm-migration` current development branch
**Conventional Commits:**
```
feat(plan): add fallback chains and strict passthrough for model identity gap
fix(router): strict passthrough for explicit models no silent rerouting
feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC
docs: LiteLLM migration plan two-layer architecture with model identity gap analysis
```
---
## 12. Zero-Downtime Migration Strategy
**Per-Agent Cutover (<2 minutes):**
1. Create LiteLLM virtual key in UI
2. Update agent's `OPENAI_API_BASE` to `:4000`
3. Verify routing works
4. Monitor LiteLLM logs for errors
**Global Rollback:**
1. If LiteLLM :4000 fails, revert all agents to `:9000`
2. NGINX `router_fallback` handles automatic failover
3. Monitor GPU metrics for health checks
---
## Appendix A: Model Identity Gap Analysis (RESOLVED)
### Problem Identified (Abiba, June 14)
The original architecture had a metrics accuracy gap: when an agent requested `qwen3.6-35B-A3B` and MoE was busy, the router silently rerouted to Dense. LiteLLM logged it as MoE usage, corrupting per-model spend/usage tracking.
### Root Cause
```python
# OLD router code (router-fixed.py):
if is_gpu_busy(target) and req in allowed:
alts = [m for m in avail if m != target and m in allowed]
if alts:
alt = select_best_gpu(alts, "explicit", agent)
if alt: return alt # silently changed GPU
```
### Resolution: Strict Passthrough + LiteLLM Fallback Chains
Two changes deployed:
**1. Router strict passthrough:**
```python
# NEW: strict mode no silent fallback
if req != "auto":
target = req if req in avail else avail[0]
if req not in avail:
return {"model": req, "reason": "explicit_unavailable", "saturated": True}
if is_gpu_busy(target):
return {"model": target, "reason": "explicit_saturated", "saturated": True}
return {"model": target, "reason": "explicit"}
```
**2. LiteLLM fallback chains (in config.yaml):**
```yaml
router_settings:
allowed_fails: 100
fallbacks:
- qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
- qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
- gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
```
### Result
| Scenario | Before | After |
|----------|--------|-------|
| Agent asks for MoE, MoE available | MoE used, metrics OK | MoE used, metrics OK |
| Agent asks for MoE, MoE busy | Router Dense silently, metrics WRONG | Router 503, LiteLLM Dense, metrics show BOTH attempts |
| Agent uses syslog-auto | Router picks GPU, LiteLLM sees opaque | Same (syslog-auto is opaque by design) |
| All 3 GPUs saturated | Router queues (30s), then 503 | Same, LiteLLM sees 503 after fallback chain exhausted |
---
## Appendix B: LiteLLM Virtual Key Migration
| Agent | Old Key | New LiteLLM Key | Tier | Budget |
|-------|---------|-----------------|------|--------|
| Abiba | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
| Mumuni | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
| Tanko | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
| Kagenz0 | sk-***-*** | sk-litellm-*** | professional | $500 |
| Koby | sk-***-*** | sk-litellm-*** | professional | $500 |
| Koonimo | sk-***-*** | sk-litellm-*** | professional | $500 |
---
## Appendix C: Authentik SSO Integration
**Authentik Provider Setup:**
1. Create OAuth2 application in Authentik
2. Set redirect URI: `http://<CT-116-IP>/sso/callback`
3. Configure `client_id` and `client_secret`
4. Mount `custom_sso.py` to LiteLLM container
5. Update config.yaml with provider details
---
## Appendix D: Prometheus Monitoring
**Metrics Export:**
- LiteLLM metrics `http://127.0.0.1:4000/metrics`
- Router metrics `http://127.0.0.1:9000/metrics`
- GPU health metrics `http://127.0.0.1:9000/metrics/gpu`
**Alerts:**
- GPU health score > 70 alert
- Circuit breaker trip alert
- LiteLLM spend > $100/day alert
- LiteLLM latency > 1000ms alert
Reference in New Issue View Git Blame Copy Permalink