inference-harness/LITELLM-MIGRATION-PLAN.md

LiteLLM Integration Migration Plan

Syslog Solution LLC June 14, 2026

Deployment Target: CT 116 syslog-api (192.168.68.116) on minipve all services co-located. DNS Strategy: Option A /etc/hosts + Docker extra_hosts for internal resolution of auth.sysloggh.net 192.168.68.11. GitOps: This plan lives in SyslogSolution/syslog-harness on Gitea. All changes tracked via git with conventional commits.

---

Executive Summary

Goal: Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.

Architecture Decision: Two-layer architecture.

                  
                       LiteLLM Gateway (Layer 1)    
                       Port 4000  Policy & UX      
                     
                     Admin UI (/ui)               
                     Virtual Keys & Permissions   
                     Teams, Users, SSO (OIDC)     
                     Spend Tracking & Budgets     
                     Usage Analytics Dashboard    
                     Request Audit Trail          
                     Global Rate Limiting         
                     
                                                    
                      Pass-through to router         
                  
                             
                  
                    Custom Router (Layer 2)          
                    Port 9000  Intelligence & HW   
                     
                     5-Tier Content-Based Routing 
                     GPU Slot Management (Redis)  
                     Agent Spread Prevention      
                     GPU Health Scoring            
                     Sidecar VRAM/Temp/Power      
                     Circuit Breaker              
                     Context Window Tracking       
                     Per-Request Perf Recording   
                     Hardware Rate Limiting       
                     
                  
                             
          
                                                   
      
   qwen3.6-35B     qwen3.6-27B      gemma-4-12b      
   MoE/Strix       Dense/RTX3090    VLM/RTX 5070     
   :8080 (llama)   :8080 (llama)    :8080 (llama)    
   :8090 (side)    :8090 (side)     :8090 (sidecar)  
      

---

1. Current State Baseline

1.1 Router (router-fixed.py port 9000, deployed on CT 116 / syslog-api)

Deployment Host: CT 116 syslog-api on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.

Feature	Implementation
Routing Engine	5-tier content-based: lightweight simple_conv medium heavy_reasoning default
GPU Slot Mgmt	Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset
Health Checks	Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health
Agent Spreading	select_best_gpu() prefers GPUs with 0 other agents, then non-self GPUs
Rate Limiting	Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20
Auth	Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation
Performance	Per-request latency/tokens/tps Redis lists (perf:recent, perf:model:X, perf:agent:X)
Context Tracking	Session-level token accumulation with compaction warnings in headers
SSE Streaming	Real-time dashboard updates, per-model timeseries
Admin	/admin/keys, /admin/keys/generate, /admin/keys/revoke, /admin/keys/deprecation-summary
Strict Passthrough	Explicit model requests go to that GPU exactly (no silent fallback). LiteLLM owns failover.

1.2 GPU Backends

GPU	Host	llama.cpp	Sidecar	VRAM	Context
qwen3.6-35B-A3B (MoE)	192.168.68.15	:8080	:8090	Strix Halo	262K
qwen3.6-27B-code (Dense)	192.168.68.8	:8080	:8090	RTX 3090	262K
gemma-4-12b (VLM)	192.168.68.110	:8080	:8090	RTX 5070	262K

1.3 Existing LiteLLM POC on CT 116

CT 116 already has a LiteLLM container running (POC, 6 days uptime):

harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081->4000
harness-redis   | redis:7-alpine                      | 127.0.0.1:6379
harness-router  | inference-harness-router             | 127.0.0.1:9000
harness-nginx   | nginx:alpine                        | 0.0.0.0:80
harness-dashboard | inference-harness-dashboard        | 127.0.0.1:3000

• /opt/litellm/ previous setup directory on CT 116
• Configured with Postgres, host networking, master key
• Currently bypassed router routes directly to GPUs
• Goal: Productionize with two-layer architecture on this same host

1.4 DNS Routing (Split-Horizon)

For OIDC SSO with Authentik, CT 116 must resolve auth.sysloggh.net internally:

Problem: auth.sysloggh.net CNAMEs to netbird.sysloggh.net 72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.

Solution Option A: /etc/hosts on CT 116 host:

# On CT 116 (syslog-api)
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts

Docker containers also need this resolution add to docker-compose.yml:

services:
  nginx:
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
  litellm:
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"

DNS Servers: CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.

---

2. What LiteLLM Brings (That We Don't Have)

Feature	Our Router	LiteLLM	Value Add
Admin UI		Full dashboard at /ui	Non-technical users can manage keys, view spend
Virtual Key Permissions	(binary key->tier)	Granular: per-model, per-team, budget caps	Fine-grained access control
Spend Tracking		Per-request $ cost with model-specific pricing	Billing, cost allocation, client invoicing
Teams & Orgs		Multi-tenant: org->team->user hierarchy	Segregate clients/projects
SSO/OIDC		Google, GitHub, Microsoft, Okta, Keycloak	Enterprise auth integration
Budget Alerts		Per-key, per-user, per-team budget with webhooks	Prevent overspend
Usage Analytics	(custom /metrics)	Built-in: daily trends, model breakdown, per-customer	Better visualization
100+ Provider Support	(3 local GPUs)	OpenAI, Anthropic, Bedrock, Vertex, etc.	Future cloud model access
Fallback Chains	(silent rerouting)	Explicit multi-provider failover with per-model logging	Accurate per-model tracking, visible failover
RPM/TPM Weighted LB		Weighted load balancing across deployments	Fine-grained traffic shaping

---

3. What We Keep (That LiteLLM Doesn't Have)

Feature	Why We Must Keep It
Content-based 5-tier routing	LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints
GPU hardware health scoring	LiteLLM doesn't monitor VRAM, temp, power our scoring prevents routing to overheating GPUs
GPU slot management	LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading
Agent spread prevention	Our select_best_gpu() spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle
Cross-turn context tracking	Session-level token accumulation with compaction warnings via X-Context-Warning headers
GPU sidecar metrics	VRAM %, GPU utilization %, power draw, temperature exposed via /metrics and SSE dashboard
Circuit breaker	39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular

---

4. Migration Architecture

4.1 Principle: "LiteLLM is the lobby, our router is the engine room"

• LiteLLM handles everything a user/admin touches: keys, teams, budgets, spend logs, SSO, the UI
• Custom Router handles everything the GPUs need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking

4.2 Flow

Agent Request
     
     

 LiteLLM Gateway (:4000)                      
                                              
 1. Authenticate virtual key (sk-litellm-...) 
 2. Check key permissions (model access)      
 3. Check budget (per-key, per-user, per-team)
 4. Check team rate limits                    
 5. Log request metadata                      
 6. Forward to custom router as OpenAI-compat  
    POST http://router:9000/v1/chat/completions
    Headers: Authorization: Bearer *** 
             X-LiteLLM-User: <user-id>         
             X-LiteLLM-Team: <team-id>         
             X-Session-Id: <session>           
                                              
 7. On response: log spend, update budgets     
 8. If router returns 503 (GPU saturated):     
    consult fallback chain, retry next model    
 9. Return response to agent                   

               
               

 Custom Router (:9000)                         
                                               
 1. Authenticate agent key (sk-syslog-...)     
 2. Hardware rate limit (per-tier RPM)         
 3. Content-based tier routing (for syslog-auto)
    OR strict passthrough (for explicit models) 
 4. GPU slot availability (Redis counter)      
 5. GPU health check (sidecar)                 
 6. Agent spread logic (select_best_gpu)       
 7. Queue if saturated (with timeout)          
 8. Forward to selected llama.cpp GPU          
 9. Track context window, set compaction header
 10. Record performance metrics                
 11. Return response (with routing metadata)   

               
               

 llama.cpp GPU (:8080)                         

4.3 LiteLLM Config (config.yaml)

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: postgresql://litellm:***@postgres:5432/litellm
  store_model_in_db: true

model_list:
  # Content-based auto-routing (router picks GPU via 5-tier analysis)
  - model_name: syslog-auto
    litellm_params:
      model: openai/syslog-auto
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
      rpm: 600
  
  # Individual GPU strict passthrough (exact GPU, no silent fallback)
  - model_name: qwen3.6-35B-A3B
    litellm_params:
      model: openai/qwen3.6-35B-A3B
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
  
  - model_name: qwen3.6-27B-code
    litellm_params:
      model: openai/qwen3.6-27B-code
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

  - model_name: gemma-4-12b
    litellm_params:
      model: openai/gemma-4-12b
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

# Guardrails: Pre-call and post-call content moderation
guardrails:
  - guardrail_name: "input-moderation"
    litellm_params:
      guardrail: openai_moderation
      mode: "pre_call"

  - guardrail_name: "output-moderation"
    litellm_params:
      guardrail: openai_moderation
      mode: "post_call"

  - guardrail_name: "harmful-content-filter"
    litellm_params:
      guardrail: litellm_content_filter
      mode: "pre_call"
      categories:
        - category: "harmful_self_harm"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"
        - category: "harmful_violence"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"
        - category: "harmful_illegal_weapons"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"

litellm_settings:
  num_retries: 0                     # Disabled  our router handles retry
  request_timeout: 600               # Match our 10-min llama-server timeout
  set_verbose: true
  failure_callback: ["prometheus"]   # Optional: export to Prometheus

router_settings:
  routing_strategy: "usage-based-routing"  # For external models only
  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
  allowed_fails: 100                       # Router returns 503 on saturated GPUs  cooldown disabled
  # Fallback chains: LiteLLM retries down the chain when router returns saturated
  # This gives accurate per-model metrics because router no longer silently reroutes
  fallbacks:
    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]

# Cost tracking: map model names to per-token pricing for spend tracking
litellm_settings:
  model_cost:
    qwen3.6-35B-A3B:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
    qwen3.6-27B-code:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
    gemma-4-12b:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
    # For internal cost allocation, set symbolic rates:
    # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens

4.4 Router Modifications

To accommodate LiteLLM, router-fixed.py requires the following updates:

1. Strict passthrough for explicit models (DEPLOYED):

   # In route(), the explicit model section changed from silent fallback to strict:
   req = rd.get("model","auto")
   if req != "auto":
       # STRICT MODE: no silent fallback  LiteLLM handles failover chains.
       # This keeps per-model metrics accurate. Returns saturated if busy.
       target = req if req in avail else avail[0]
       if req not in avail:
           return {"model": req, "reason": "explicit_unavailable", "saturated": True}
       if is_gpu_busy(target):
           return {"model": target, "reason": "explicit_saturated", "saturated": True}
       return {"model": target, "reason": "explicit"}
   

2. New header passthrough: Forward X-LiteLLM-* headers to GPU (transparent already works)

3. New endpoint for health passthrough: GET /v1/models already works

4. Keep ALL routing logic: No changes to select_best_gpu(), check_gpu_health(), slot management, etc. Content-based routing for syslog-auto is fully intact.

5. Add LiteLLM-compatible response: Return X-Usage-Tokens header so LiteLLM can track token costs

   resp.headers["X-Usage-Tokens"] = json.dumps({
       "prompt_tokens": prompt_tokens,
       "completion_tokens": completion_tokens,
       "model": model
   })
   

4.5 Router Logic Refinements

GPU Health Scoring (Updated): We are updating the scoring algorithm to include Power metrics:

def gpu_health_score(model):
    h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
    if h.get("status") == "down":
        return 999  # never pick down GPUs
    vram_pct = h.get("vram_pct") or 50
    temp_c = h.get("temp_c") or 50
    power_w = h.get("power_w") or 50
    active = gpu_active_count(model)
    max_c = GPU_MAX_CONCURRENT.get(model, 1)
    load_pct = (active / max_c) * 100 if max_c > 0 else 0
    # Score: lower = better
    score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2))
    return score

---

5. Deployment Plan (4 Phases) Zero-Downtime Strategy

Phase 0: Infrastructure Prep (Current Week) Zero Downtime

Goal: Prepare CT 116 infrastructure without affecting running agents.

Tasks:

1. Set up DNS split-horizon on CT 116

   echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
   

2. Deploy Postgres container alongside existing services

3. Replace LiteLLM config with production config.yaml (see 4.3)

   • All 4 models http://router:9000/v1
   • Fallback chains for explicit models
   • Guardrails (pre-call, post-call, content filter)
   • allowed_fails: 100 (router returns 503 on saturated)
   • num_retries: 0 (LiteLLM retries handled by fallback chains)

4. Deploy custom_sso.py for Authentik OIDC integration

5. Restart LiteLLM container with new config

6. Verify internal routing

Verification Checklist:

• DNS resolution: getent hosts auth.sysloggh.net 192.168.68.11
• Postgres container healthy
• LiteLLM /health returns 200
• LiteLLM Router pass-through returns valid chat completion
• GPU health metrics unaffected
• Explicit model request returns saturated (not silently rerouted) when GPU busy

Phase 1: Shadow Mode (Week 1) Zero Risk, Zero Downtime

Goal: Deploy LiteLLM alongside existing router, test in shadow mode. Agents continue using :9000 directly.

Tasks:

1. Create virtual keys for test agents via LiteLLM UI
2. Verify pass-through works for all 4 models
3. Validate fallback chains: saturate MoE confirm LiteLLM retries Dense confirm VLM
4. Run 24-hour shadow: monitor LiteLLM spend logs vs router metrics
5. Verify GPU health metrics unaffected
6. Check guardrails not generating false positives

Phase 2: Cutover (Week 2) Gradual Agent Migration

Goal: Move agents one-by-one to LiteLLM endpoint.

Tasks:

1. Migrate API keys to LiteLLM virtual keys
2. Create teams: "Core Agents" (enterprise), "Dev Agents" (professional)
3. Update agent configs one at a time: OPENAI_API_BASE :4000
4. Test each agent individually
5. Enable SSO via Authentik + custom_sso.py
6. Keep router :9000 as emergency fallback for 48 hours

Phase 3: Production Hardening (Week 3+)

Goal: Lock down, optimize, monitor.

Tasks:

1. Remove deprecated router endpoints (after all agents migrated)
2. Add LiteLLM observability (Prometheus, Slack/email alerts)
3. Enable LiteLLM caching (shared Redis)
4. Add external model fallbacks for client-facing services
5. Router slim-down: keep routing/slots/health/perf, remove key management
6. Multi-tenancy setup for client-facing inference services

---

6. Nginx Configuration (with Authentik OIDC Forward Auth)

# OLD (remove)
# location /admin/ {
#     proxy_pass http://127.0.0.1:9000/admin/;
# }

# === Authentik auth subrequest endpoint ===
location /authentik/auth {
    internal;
    proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

# === LiteLLM Admin UI  Authentik-protected ===
location /ui/ {
    auth_request /authentik/auth;
    auth_request_set $auth_user $upstream_http_x_authentik_username;
    auth_request_set $auth_email $upstream_http_x_authentik_email;

    proxy_set_header X-Authentik-Username $auth_user;
    proxy_set_header X-Authentik-Email $auth_email;

    proxy_pass http://127.0.0.1:4000/ui/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

# === LiteLLM SSO callback ===
location /sso/callback {
    proxy_pass http://127.0.0.1:4000/sso/callback;
    proxy_set_header Host $host;
}

# === API endpoint  Bearer token auth ===
location /v1/ {
    proxy_pass http://127.0.0.1:4000/v1/;
    proxy_set_header Host $host;
    proxy_read_timeout 600s;
    error_page 502 = @router_fallback;
}

location @router_fallback {
    proxy_pass http://127.0.0.1:9000/v1/;
    proxy_set_header Host $host;
}

# === Key management API ===
location /key/ {
    proxy_pass http://127.0.0.1:4000/key/;
    proxy_set_header Host $host;
    proxy_set_header Authorization $http_authorization;
}

# Keep router metrics accessible (not behind LiteLLM)
location /router/ {
    proxy_pass http://127.0.0.1:9000/;
}

location /health {
    proxy_pass http://127.0.0.1:4000/health;
}

---

7. Docker Compose (docker-compose.yml on CT 116)

services:
  # Layer 1: LiteLLM Gateway (Policy & Admin)
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    network_mode: "host"
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - ./custom_sso.py:/app/custom_sso.py:ro
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm
      - STORE_MODEL_IN_DB=True
      - ROUTER_API_KEY=${ROUTER_API_KEY}
      - OPENAI_API_KEY=***
      - ANTHROPIC_API_KEY=${ANTH...KEY}
      - PROXY_BASE_URL=https://litellm.sysloggh.net
    command:
      - --config
      - /app/config.yaml
      - --port
      - "4000"
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    network_mode: "host"
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POST...}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  nginx:
    image: nginx:alpine
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:80"
    restart: unless-stopped

volumes:
  pgdata:

---

8. Risk Mitigation

Risk	Mitigation
LiteLLM adds latency overhead	Shadow mode measures: <50ms extra is acceptable
LiteLLM down = all agents down	NGINX fallback to router :9000 direct (see 6)
Explicit GPU saturated no fallback available	LiteLLM fallback chains try all 3 GPUs in order before failing
Fallback chain masking real GPU failures	Router returns saturated: true only for capacity, down returns different error
Key sync drift	Single-source: LiteLLM is key authority. Router uses one ROUTER_API_KEY
Spend tracking inaccurate for local GPUs	model_cost per GPU with $0 rate; optional symbolic pricing for internal billing
Double rate limiting	Intentional: LiteLLM for per-user caps, Router for hardware protection
PostgreSQL failure	LiteLLM can run with SQLite fallback; UI features degrade
Per-model metrics accuracy with syslog-auto	syslog-auto is opaque by design (content-based routing). Explicit models are accurate. Use explicit models for per-GPU billing.

---

9. Success Metrics

Metric	Before	After
Key management	Manual CLI + env vars + redeploy	UI-based, instant, no redeploy
Spend visibility	None	Per-agent, per-team, per-model $ tracking
Access control	Tier-based (3 levels)	Per-key, per-model, budget-capped
New agent onboarding	Generate key, update env var, redeploy router	Create in UI, share key
Admin UX	curl + JSON responses	Visual dashboard, graphs, search
Audit trail	Router logs (stdout only)	Database-backed with UI search
SSO	None	Authentik OIDC
Budget enforcement	None	Automatic: key suspended at $limit
GPU failover	Silent (inaccurate metrics)	Explicit (LiteLLM fallback chains, per-model logs)
GPU routing intelligence	Full (unchanged)	Full (unchanged)

---

10. Migration Commands (Quick Reference)

# On CT 116 (SSH via minipve: pct exec 116 bash):

# Phase 0: Infrastructure Prep
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts

cd /opt/litellm
docker compose up -d postgres
# Replace config.yaml with production version (see 4.3)
docker compose restart litellm

# Verify
curl http://127.0.0.1:4000/health
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'

---

11. GitOps Workflow

• main production-ready code
• feature/litellm-migration current development branch

Conventional Commits:

feat(plan): add fallback chains and strict passthrough for model identity gap
fix(router): strict passthrough for explicit models  no silent rerouting
feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC
docs: LiteLLM migration plan  two-layer architecture with model identity gap analysis

---

12. Zero-Downtime Migration Strategy

Per-Agent Cutover (<2 minutes):

1. Create LiteLLM virtual key in UI
2. Update agent's OPENAI_API_BASE to :4000
3. Verify routing works
4. Monitor LiteLLM logs for errors

Global Rollback:

1. If LiteLLM :4000 fails, revert all agents to :9000
2. NGINX router_fallback handles automatic failover
3. Monitor GPU metrics for health checks

---

Appendix A: Model Identity Gap Analysis (RESOLVED)

Problem Identified (Abiba, June 14)

The original architecture had a metrics accuracy gap: when an agent requested qwen3.6-35B-A3B and MoE was busy, the router silently rerouted to Dense. LiteLLM logged it as MoE usage, corrupting per-model spend/usage tracking.

Root Cause

# OLD router code (router-fixed.py):
if is_gpu_busy(target) and req in allowed:
    alts = [m for m in avail if m != target and m in allowed]
    if alts:
        alt = select_best_gpu(alts, "explicit", agent)
        if alt: return alt   # silently changed GPU

Resolution: Strict Passthrough + LiteLLM Fallback Chains

Two changes deployed:

1. Router strict passthrough:

# NEW: strict mode  no silent fallback
if req != "auto":
    target = req if req in avail else avail[0]
    if req not in avail:
        return {"model": req, "reason": "explicit_unavailable", "saturated": True}
    if is_gpu_busy(target):
        return {"model": target, "reason": "explicit_saturated", "saturated": True}
    return {"model": target, "reason": "explicit"}

2. LiteLLM fallback chains (in config.yaml):

router_settings:
  allowed_fails: 100
  fallbacks:
    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]

Result

Scenario	Before	After
Agent asks for MoE, MoE available	MoE used, metrics OK	MoE used, metrics OK
Agent asks for MoE, MoE busy	Router Dense silently, metrics WRONG	Router 503, LiteLLM Dense, metrics show BOTH attempts
Agent uses syslog-auto	Router picks GPU, LiteLLM sees opaque	Same (syslog-auto is opaque by design)
All 3 GPUs saturated	Router queues (30s), then 503	Same, LiteLLM sees 503 after fallback chain exhausted

---

Appendix B: LiteLLM Virtual Key Migration

Agent	Old Key	New LiteLLM Key	Tier	Budget
Abiba	sk--	sk-litellm-***	enterprise	$1000
Mumuni	sk--	sk-litellm-***	enterprise	$1000
Tanko	sk--	sk-litellm-***	enterprise	$1000
Kagenz0	sk--	sk-litellm-***	professional	$500
Koby	sk--	sk-litellm-***	professional	$500
Koonimo	sk--	sk-litellm-***	professional	$500

---

Appendix C: Authentik SSO Integration

Authentik Provider Setup:

1. Create OAuth2 application in Authentik
2. Set redirect URI: http://<CT-116-IP>/sso/callback
3. Configure client_id and client_secret
4. Mount custom_sso.py to LiteLLM container
5. Update config.yaml with provider details

---

Appendix D: Prometheus Monitoring

Metrics Export:

• LiteLLM metrics http://127.0.0.1:4000/metrics
• Router metrics http://127.0.0.1:9000/metrics
• GPU health metrics http://127.0.0.1:9000/metrics/gpu

Alerts:

• GPU health score > 70 alert
• Circuit breaker trip alert
• LiteLLM spend > $100/day alert
• LiteLLM latency > 1000ms alert