SyslogSolution/syslog-harness
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
492a4fe68bd2507d2d3dfc154976a221a850551a
syslog-harness/LITELLM-MIGRATION-PLAN.md
T

31 KiB
Raw Blame History

LiteLLM Integration Migration Plan

Syslog Solution LLC June 14, 2026

Deployment Target: CT 116 syslog-api (192.168.68.116) on minipve all services co-located. DNS Strategy: Option A /etc/hosts + Docker extra_hosts for internal resolution of auth.sysloggh.net 192.168.68.11. GitOps: This plan lives in SyslogSolution/syslog-harness on Gitea. All changes tracked via git with conventional commits.

Executive Summary

Goal: Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.

Architecture Decision: Two-layer architecture.

                  
                       LiteLLM Gateway (Layer 1)    
                       Port 4000  Policy & UX      
                     
                     Admin UI (/ui)               
                     Virtual Keys & Permissions   
                     Teams, Users, SSO (OIDC)     
                     Spend Tracking & Budgets     
                     Usage Analytics Dashboard    
                     Request Audit Trail          
                     Global Rate Limiting         
                     
                                                    
                      Pass-through to router         
                  
                             
                  
                    Custom Router (Layer 2)          
                    Port 9000  Intelligence & HW   
                     
                     5-Tier Content-Based Routing 
                     GPU Slot Management (Redis)  
                     Agent Spread Prevention      
                     GPU Health Scoring (40/30/30) 
                     Sidecar VRAM/Temp/Power      
                     Circuit Breaker              
                     Context Window Tracking       
                     Per-Request Perf Recording   
                     Hardware Rate Limiting       
                     
                  
                             
          
                                                   
      
   qwen3.6-35B     qwen3.6-27B      gemma-4-12b      
   MoE/Strix       Dense/RTX3090    VLM/RTX 5070     
   :8080 (llama)   :8080 (llama)    :8080 (llama)    
   :8090 (side)    :8090 (side)     :8090 (sidecar)

1. Current State Baseline

1.1 Router (router-fixed.py port 9000, deployed on CT 116 / syslog-api)

Deployment Host: CT 116 syslog-api on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.

Feature Implementation
Routing Engine 5-tier content-based: lightweight simple_conv medium heavy_reasoning default
GPU Slot Mgmt Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset
Health Checks Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health
Agent Spreading select_best_gpu() prefers GPUs with 0 other agents, then non-self GPUs
Rate Limiting Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20
Auth Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation
Performance Per-request latency/tokens/tps Redis lists (perf:recent, perf:model:X, perf:agent:X)
Context Tracking Session-level token accumulation with compaction warnings in headers
SSE Streaming Real-time dashboard updates, per-model timeseries
Admin /admin/keys, /admin/keys/generate, /admin/keys/revoke, /admin/keys/deprecation-summary

1.2 GPU Backends

GPU Host llama.cpp Sidecar VRAM Context
qwen3.6-35B-A3B (MoE) 192.168.68.15 :8080 :8090 Strix Halo 262K
qwen3.6-27B-code (Dense) 192.168.68.8 :8080 :8090 RTX 3090 262K
gemma-4-12b (VLM) 192.168.68.110 :8080 :8090 RTX 5070 262K

1.3 Existing LiteLLM POC on CT 116

CT 116 already has a LiteLLM container running (POC, 6 days uptime):

harness-redis   | redis:7-alpine                      | 127.0.0.1:6379
harness-router  | inference-harness-router             | 127.0.0.1:9000
harness-nginx   | nginx:alpine                        | 0.0.0.0:80
harness-dashboard | inference-harness-dashboard        | 127.0.0.1:3000
```\

- `/opt/litellm/`  previous setup directory on CT 116
- Configured with Postgres, host networking, master key
- Currently bypassed  router routes directly to GPUs
- **Goal: Productionize with two-layer architecture on this same host**

### 1.4 DNS Routing (Split-Horizon)

For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally:

**Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net`  72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.

**Solution  Option A: /etc/hosts on CT 116 host:**
```bash
# On CT 116 (syslog-api)
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts

Docker containers also need this resolution add to docker-compose.yml:

services:
  nginx:
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
  litellm:
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"

DNS Servers: CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.

2. What LiteLLM Brings (That We Don't Have)

Feature Our Router LiteLLM Value Add
Admin UI Full dashboard at /ui Non-technical users can manage keys, view spend
Virtual Key Permissions (binary keytier) Granular: per-model, per-team, budget caps Fine-grained access control
Spend Tracking Per-request $ cost with model-specific pricing Billing, cost allocation, client invoicing
Teams & Orgs Multi-tenant: orgteamuser hierarchy Segregate clients/projects
SSO/OIDC Google, GitHub, Microsoft, Okta, Keycloak Enterprise auth integration
Budget Alerts Per-key, per-user, per-team budget with webhooks Prevent overspend
Usage Analytics (custom /metrics) Built-in: daily trends, model breakdown, per-customer Better visualization
100+ Provider Support (3 local GPUs) OpenAI, Anthropic, Bedrock, Vertex, etc. Future cloud model access
Fallback Chains Multi-provider: OpenAIAzureTogether External model resilience
RPM/TPM Weighted LB Weighted load balancing across deployments Fine-grained traffic shaping

3. What We Keep (That LiteLLM Doesn't Have)

Feature Why We Must Keep It
Content-based 5-tier routing LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints
GPU hardware health scoring LiteLLM doesn't monitor VRAM, temp, power our 40/30/30 scoring prevents routing to overheating GPUs
GPU slot management LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading
Agent spread prevention Our select_best_gpu() spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle
Cross-turn context tracking Session-level token accumulation with compaction warnings via X-Context-Warning headers
GPU sidecar metrics VRAM %, GPU utilization %, power draw, temperature exposed via /metrics and SSE dashboard
Circuit breaker 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular

4. Migration Architecture

4.1 Principle: "LiteLLM is the lobby, our router is the engine room"

  • LiteLLM handles everything a user/admin touches: keys, teams, budgets, spend logs, SSO, the UI
  • Custom Router handles everything the GPUs need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking

4.2 Flow

Agent Request
     
     

 LiteLLM Gateway (:4000)                      
                                              
 1. Authenticate virtual key (sk-litellm-...) 
 2. Check key permissions (model access)      
 3. Check budget (per-key, per-user, per-team)
 4. Check team rate limits                    
 5. Log request metadata                      
 6. Forward to custom router as OpenAI-compat  
    POST http://router:9000/v1/chat/completions
    Headers: Authorization: Bearer *** 
             X-LiteLLM-User: <user-id>         
             X-LiteLLM-Team: <team-id>         
             X-Session-Id: <session>           
                                              
 7. On response: log spend, update budgets     
 8. Return response to agent                   

               
               

 Custom Router (:9000)                         
                                               
 1. Authenticate agent key (sk-syslog-...)     
 2. Hardware rate limit (per-tier RPM)         
 3. Content-based tier routing                 
    - Estimate tokens, detect system msg       
    - Count turns, check routing_hints         
 4. GPU slot availability (Redis counter)      
 5. GPU health check (sidecar)                 
 6. Agent spread logic (select_best_gpu)       
 7. Queue if saturated (with timeout)          
 8. Forward to selected llama.cpp GPU          
 9. Track context window, set compaction header
 10. Record performance metrics                
 11. Return response (with routing metadata)   

               
               

 llama.cpp GPU (:8080)

4.3 LiteLLM Config (config.yaml)

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: postgresql://litellm:***@postgres:5432/litellm
  store_model_in_db: true

model_list:
  # All three GPUs exposed as a single virtual "syslog-router" model
  # LiteLLM passes through to our router, which handles actual GPU selection
  - model_name: syslog-auto           # Default auto-routing
    litellm_params:
      model: openai/syslog-auto       # Using OpenAI-compatible format
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
      rpm: 600                         # Cap total RPM across all GPUs
  
  # Individual GPU pass-through (for explicit model requests)
  - model_name: qwen3.6-35B-A3B
    litellm_params:
      model: openai/qwen3.6-35B-A3B
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
  
  - model_name: qwen3.6-27B-code
    litellm_params:
      model: openai/qwen3.6-27B-code
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

  - model_name: gemma-4-12b
    litellm_params:
      model: openai/gemma-4-12b
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

# Guardrails: Pre-call and post-call content moderation
guardrails:
  - guardrail_name: "input-moderation"
    litellm_params:
      guardrail: openai_moderation
      mode: "pre_call"

  - guardrail_name: "output-moderation"
    litellm_params:
      guardrail: openai_moderation
      mode: "post_call"

  - guardrail_name: "harmful-content-filter"
    litellm_params:
      guardrail: litellm_content_filter
      mode: "pre_call"
      categories:
        - category: "harmful_self_harm"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"
        - category: "harmful_violence"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"
        - category: "harmful_illegal_weapons"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"

litellm_settings:
  num_retries: 0                     # Disabled  our router handles retry
  request_timeout: 600               # Match our 10-min llama-server timeout
  set_verbose: true
  failure_callback: ["prometheus"]   # Optional: export to Prometheus

router_settings:
  routing_strategy: "usage-based-routing"  # For external models only
  # Note: All local GPU routing is handled by custom router
  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
  allowed_fails: 0                       # Set to 0  router's circuit breaker is authoritative

# Cost tracking: map model names to per-token pricing
# These are passed through from our router's X-Usage-Tokens header

4.4 Router Modifications (Engine Room Updates)

To accommodate LiteLLM, the router-fixed.py requires the following updates:

  1. New header passthrough: Forward X-LiteLLM-* headers to GPU (transparent already works)
  2. New endpoint for health passthrough: GET /v1/models already works
  3. Disable own key management: Remove /admin/keys/* endpoints (migrate to LiteLLM UI)
  4. Keep ALL routing logic: No changes to route(), select_best_gpu(), check_gpu_health(), slot management, etc.
  5. Add LiteLLM-compatible response: Return X-Usage-Tokens header so LiteLLM can track token costs
# ADD to router-fixed.py chat() response:
resp.headers["X-Usage-Tokens"] = json.dumps({
    "prompt_tokens": prompt_tokens,
    "completion_tokens": completion_tokens,
    "model": model
})

4.5 Router Logic Refinements

GPU Health Scoring (Updated): We are updating the scoring algorithm to include Power metrics (30% weight):

def gpu_health_score(model):
    h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
    if h.get("status") == "down":
        return 999  # never pick down GPUs
    vram_pct = h.get("vram_pct") or 50
    temp_c = h.get("temp_c") or 50
    power_w = h.get("power_w") or 50
    active = gpu_active_count(model)
    max_c = GPU_MAX_CONCURRENT.get(model, 1)
    load_pct = (active / max_c) * 100 if max_c > 0 else 0
    # Score: lower = better (more headroom, cooler, less power-constrained, less loaded)
    score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2))
    return score

5. Deployment Plan (4 Phases) Zero-Downtime Strategy

Phase 0: Infrastructure Prep (Current Week) Zero Downtime

Goal: Prepare CT 116 infrastructure without affecting running agents.

Tasks:

  1. Set up DNS split-horizon on CT 116

    # On CT 116 host (syslog-api)
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts

    Add to docker-compose.yml (see 7):

    extra_hosts:
  - "auth.sysloggh.net:192.168.68.11"

  2. Deploy Postgres container alongside existing services:

    cd /opt/litellm
# Add postgres to docker-compose.yml
docker compose up -d postgres

    Note: Use a dedicated volume for pgdata to ensure LiteLLM database persistence.

  3. Replace LiteLLM config with production config.yaml (see 4.3)

    • All 4 models http://router:9000/v1
    • Add guardrails (pre-call, post-call, content filter)
    • Set num_retries: 0 (router handles retry)
    • Set request_timeout: 600

  4. Deploy custom_sso.py for Authentik OIDC integration

    • Mount to LiteLLM container volume
    • Reference in config.yaml: custom_ui_sso_sign_in_handler: custom_sso.custom_ui_sso_sign_in_handler

  5. Restart LiteLLM container with new config

    docker compose restart litellm

  6. Verify internal routing LiteLLM Router pass-through works:

    curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer *** \
  -H "Content-Type: application/json" \
  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'

Verification Checklist:

  • DNS resolution: getent hosts auth.sysloggh.net 192.168.68.11
  • Postgres container healthy
  • LiteLLM /health returns 200
  • LiteLLM Router pass-through returns valid chat completion
  • GPU health metrics unaffected (watch /metrics)

Phase 1: Shadow Mode (Week 1) Zero Risk, Zero Downtime

Goal: Deploy LiteLLM alongside existing router, test in shadow mode. Agents continue using :9000 directly.

         (new, testing)    (existing, unchanged)
         
Agent can also directly hit :9000 as fallback (unchanged)

Tasks:

  1. Create virtual keys for test agents via LiteLLM UI or API:

    curl -X POST http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer *** \
  -H "Content-Type: application/json" \
  -d '{"models":["syslog-auto"],"metadata":{"agent":"test"}}'
    • Mirror existing API_KEYS in LiteLLM's key store
    • Set per-key budgets (test with $100 cap)

  2. Verify pass-through works

    curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer *** \
  -H "Content-Type: application/json" \
  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'

  3. Run 24-hour shadow: Both :4000 and :9000 active, agents use :9000

    • Monitor LiteLLM spend logs vs router metrics confirm parity
    • Verify GPU health metrics unaffected
    • Check guardrails not generating false positives

Zero-Downtime Guarantee: Router :9000 remains the primary agent endpoint. LiteLLM :4000 is tested in parallel with no impact on production traffic.

Phase 2: Cutover (Week 2) Gradual Agent Migration, Zero Cumulative Downtime

Goal: Move agents one-by-one to LiteLLM endpoint. Each agent is migrated individually other agents unaffected.

Tasks:

  1. Migrate API keys to LiteLLM virtual keys:

    • Create virtual key per agent in LiteLLM UI
    • Set model access: syslog-auto (default), plus individual GPU models
    • Set per-agent budget limits
    • Create teams:
      • "Core Agents" (Abiba, Mumuni, Tanko) enterprise tier
      • "Dev Agents" (Kagenz0, Koby, Koonimo) professional tier

  2. Update agent configs one at a time:

    • Change OPENAI_API_BASE from http://192.168.68.116:9000/v1 http://192.168.68.116:4000/v1
    • Replace agent API keys with LiteLLM virtual keys
    • Test each agent individually verify routing works
    • Per-agent downtime: <2 minutes

  3. Migrate admin functions:

    • Key creation/revocation LiteLLM UI
    • Rate limit management LiteLLM per-key RPM + router hardware RPM (dual enforcement)
    • Deprecated key tracking LiteLLM UI key list

  4. Enable SSO via Authentik + custom_sso.py:

    • Authentik OAuth2 provider configured
    • NGINX auth_request for /ui/* paths
    • Custom UI SSO sign-in handler reads x-authentik-* headers
    • Users authenticate with existing Authentik credentials

  5. Keep router :9000 accessible as emergency fallback for 48 hours

    • NGINX configured with router_fallback for 502 errors
    • Agents can revert by changing OPENAI_API_BASE back to :9000

Zero-Downtime Guarantee: Each agent has <2 minutes downtime during config update. Router :9000 stays online throughout. Fallback path available for instant rollback.

Phase 3: Production Hardening (Week 3+) Optimize & Scale

Goal: Lock down, optimize, monitor. Prepare for client-facing services.

Tasks:

  1. Remove deprecated router endpoints (only after all agents migrated):

    • Drop /admin/keys/* fully migrated to LiteLLM UI
    • Drop Phase 0 dual-key logic (LiteLLM handles key rotation)
    • Simplify API_KEYS to single ROUTER_API_KEY

  2. Add LiteLLM observability:

    • Prometheus metrics export
    • Slack/email budget alerts via webhook Hermes
    • Daily spend report webhook

  3. Enable LiteLLM caching (Redis, shared with router):

router_settings: redis_host: os.environ/REDIS_HOST redis_port: 6379 cache: true cache_ttl: 3600


4. **Add external model fallbacks** for client-facing services:
   - Add Anthropic Claude as fallback for code-heavy requests
   - Add OpenAI GPT-4o as fallback for reasoning overflow
   - LiteLLM's native fallback chains handle this cleanly

5. **Router slim-down:** Extract GPU health metrics to dedicated /health only
   - Keep: routing, slots, health checks, performance recording
   - Remove: key management, dual-key logic, admin endpoints

6. **Multi-tenancy setup** for client-facing inference services:
   - Organization  Team  User hierarchy per client
   - Per-client budget limits and guardrail policies
   - Self-service onboarding via Authentik SSO  LiteLLM UI

**Zero-Downtime Guarantee:** All changes are additive  new features added while existing routing continues uninterrupted. Router remains the GPU intelligence layer throughout.

---

---

## 6. Nginx Configuration (with Authentik OIDC Forward Auth)

The existing nginx config routes `/admin/`  router :9000. This MUST change:

```nginx
# OLD (remove)
# location /admin/ {
#     proxy_pass http://127.0.0.1:9000/admin/;
# }

# === Authentik auth subrequest endpoint ===
location /authentik/auth {
    internal;
    # Proxy to Authentik's outpost on acerpve (192.168.68.11)
    # Uses internal /etc/hosts resolution: auth.sysloggh.net  192.168.68.11
    proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

# === LiteLLM Admin UI  Authentik-protected ===
location /ui/ {
    auth_request /authentik/auth;
    auth_request_set $auth_user $upstream_http_x_authentik_username;
    auth_request_set $auth_email $upstream_http_x_authentik_email;

    proxy_set_header X-Authentik-Username $auth_user;
    proxy_set_header X-Authentik-Email $auth_email;

    proxy_pass http://127.0.0.1:4000/ui/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

# === LiteLLM SSO callback (for OIDC redirect flow) ===
location /sso/callback {
    proxy_pass http://127.0.0.1:4000/sso/callback;
    proxy_set_header Host $host;
}

# === API endpoint  Bearer token auth (no Authentik) ===
location /v1/ {
    # Primary: LiteLLM gateway
    proxy_pass http://127.0.0.1:4000/v1/;
    proxy_set_header Host $host;
    proxy_read_timeout 600s;
    # Fallback: direct router (if LiteLLM down)
    error_page 502 = @router_fallback;
}

location @router_fallback {
    proxy_pass http://127.0.0.1:9000/v1/;
    proxy_set_header Host $host;
}

# === Key management API (needs master_key, not Authentik) ===
location /key/ {
    proxy_pass http://127.0.0.1:4000/key/;
    proxy_set_header Host $host;
    proxy_set_header Authorization $http_authorization;
}

# Keep router metrics accessible (not behind LiteLLM)
location /router/ {
    proxy_pass http://127.0.0.1:9000/;
}

# Health check  combines both layers
location /health {
    proxy_pass http://127.0.0.1:4000/health;
}

7. Docker Compose (docker-compose.yml on CT 116)

Deployment host: CT 116 syslog-api (192.168.68.116) on minipve. All services co-located.

services:
  # Layer 1: LiteLLM Gateway (Policy & Admin)
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    network_mode: "host"
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - ./custom_sso.py:/app/custom_sso.py:ro
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm
      - STORE_MODEL_IN_DB=True
      - ROUTER_API_KEY=${ROUTER_API_KEY}
      - OPENAI_API_KEY=***            # Optional: external fallback
      - ANTHROPIC_API_KEY=${ANTH...KEY}       # Optional: external fallback
      - PROXY_BASE_URL=https://litellm.sysloggh.net
    command:
      - --config
      - /app/config.yaml
      - --port
      - "4000"
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  # Database for LiteLLM
  postgres:
    image: postgres:16-alpine
    network_mode: "host"
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POST...}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  # Nginx also needs auth resolution
  nginx:
    image: nginx:alpine
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:80"
    restart: unless-stopped

  # Layer 2: Custom Router (Intelligence & Hardware)
  # Already deployed separately  not in this compose file
  # The router is managed by the existing harness deployment on CT 116

volumes:
  pgdata:

8. Risk Mitigation

Risk Mitigation
LiteLLM adds latency overhead Shadow mode measures: <50ms extra is acceptable for admin features. LiteLLM is a thin proxy.
LiteLLM down = all agents down NGINX fallback to router :9000 direct (see 6). Agents can also be configured with dual endpoints.
Key sync drift (LiteLLM keys router keys) Single-source: LiteLLM is key authority. Router uses one ROUTER_API_KEY from LiteLLM's perspective. Agent keys live in LiteLLM only.
Spend tracking inaccurate for local GPUs Configure model_cost per GPU with $0 rate (self-hosted). Optionally track "internal cost" via custom pricing.
Double rate limiting (LiteLLM + Router) Keep both intentionally: LiteLLM for per-user soft caps, Router for hardware protection. Non-overlapping concerns.
PostgreSQL failure LiteLLM can run with SQLite fallback, but UI features degrade. Postgres is the recommended path.
Router custom logic becomes a black box to LiteLLM Acceptable trade-off. LiteLLM sees router as opaque OpenAI endpoint. GPU-level routing decisions are router's domain.

9. Success Metrics

Metric Before After
Key management Manual CLI + env vars + redeploy UI-based, instant, no redeploy
Spend visibility None Per-agent, per-team, per-model $ tracking
Access control Tier-based (3 levels) Per-key, per-model, budget-capped
New agent onboarding Generate key, update env var, redeploy router Create in UI, share key
Admin UX curl + JSON responses Visual dashboard, graphs, search
Audit trail Router logs (stdout only) Database-backed with UI search
SSO None Google/GitHub/Microsoft OIDC
Budget enforcement None Automatic: key suspended at $limit
GPU routing intelligence Full (unchanged) Full (unchanged)
GPU health monitoring Full (unchanged) Full (unchanged)

10. Migration Commands (Quick Reference) CT 116

Target: CT 116 syslog-api (192.168.68.116), minipve. All services co-located.

# On CT 116 (SSH via minipve: pct exec 116 bash):

# === Phase 0: Infrastructure Prep ===

# 0. DNS split-horizon fix
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
getent hosts auth.sysloggh.net  # Verify  192.168.68.11

# 1. Navigate to harness deployment directory
cd /opt/litellm

# 2. Deploy Postgres (LiteLLM DB)
docker compose up -d postgres

# 3. Replace LiteLLM config with production config.yaml (see 4.3)
#    - All models  http://router:9000/v1
#    - Add guardrails (pre-call, post-call, content filter)
#    - Set num_retries: 0 (router handles retry)
#    - Set request_timeout: 600

# 4. Deploy custom_sso.py
#    - Mount to LiteLLM container volume
#    - Reference in config.yaml: custom_ui_sso_sign_in_handler: custom_sso.custom_ui_sso_sign_in_handler

# 5. Restart LiteLLM container
docker compose restart litellm

# 6. Verify internal routing
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
    -H "Authorization: Bearer *** \
    -H "Content-Type: application/json" \
    -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'

11. GitOps Workflow

Branching Strategy:

  • main production-ready code
  • feature/litellm-migration current development branch
  • deploy/phase-0, deploy/phase-1, etc. deployment-specific branches

Conventional Commits:

<type>(<scope>): <description>

- feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC + zero-downtime strategy
- fix(router): handle None temp_c/vram_pct in gpu_health_score
- chore(docker): add postgres volume for LiteLLM database persistence
- docs: LiteLLM migration plan  two-layer architecture with model identity gap analysis

Deployment Flow:

  1. Commit changes to feature/litellm-migration
  2. Open PR to main
  3. Review with git diff --stat
  4. Merge to main after approval
  5. Run deployment scripts in order: Phase 0 Phase 1 Phase 2 Phase 3

12. Zero-Downtime Migration Strategy

Per-Agent Cutover (2 minutes):

  1. Create LiteLLM virtual key in UI
  2. Update agent's OPENAI_API_BASE to :4000
  3. Verify routing works
  4. Monitor LiteLLM logs for errors

Global Rollback:

  1. If LiteLLM :4000 fails, revert all agents to :9000
  2. NGINX router_fallback handles automatic failover
  3. Monitor GPU metrics for health checks

Verification:

  • Check LiteLLM /health endpoint
  • Verify GPU metrics via router /metrics
  • Test each agent individually
  • Confirm no 502/503 errors in logs

Appendix A: Model Identity Gap Analysis

Model GPU VRAM Context Status
qwen3.6-35B-A3B MoE/Strix 65GB 262K Healthy
qwen3.6-27B-code Dense/RTX3090 24GB 262K Healthy
gemma-4-12b VLM/RTX 5070 12GB 262K Healthy

Notes:

  • All three GPUs are operational and available for LiteLLM routing
  • GPU health scoring (40/30/30) prevents routing to unhealthy GPUs
  • Router handles all GPU-level routing decisions LiteLLM sees them as opaque endpoints

Appendix B: LiteLLM Virtual Key Migration

Current API Keys LiteLLM Virtual Keys:

Agent Old Key New LiteLLM Key Tier Budget
Abiba sk-- sk-litellm-*** enterprise $1000
Mumuni sk-- sk-litellm-*** enterprise $1000
Tanko sk-- sk-litellm-*** enterprise $1000
Kagenz0 sk-- sk-litellm-*** professional $500
Koby sk-- sk-litellm-*** professional $500
Koonimo sk-- sk-litellm-*** professional $500

Appendix C: Authentik SSO Integration

Authentik Provider Setup:

  1. Create OAuth2 application in Authentik
  2. Set redirect URI: http://<CT-116-IP>/sso/callback
  3. Configure client_id and client_secret
  4. Mount custom_sso.py to LiteLLM container
  5. Update config.yaml with provider details

Appendix D: Prometheus Monitoring

Metrics Export:

  • LiteLLM metrics http://127.0.0.1:4000/metrics
  • Router metrics http://127.0.0.1:9000/metrics
  • GPU health metrics http://127.0.0.1:9000/metrics/gpu
  • Circuit breaker metrics http://127.0.0.1:9000/metrics/circuit-breaker

Alerts:

  • GPU health score > 70 alert to Hermes
  • Circuit breaker trip alert to Hermes
  • LiteLLM spend > $100/day alert to Hermes
  • LiteLLM latency > 1000ms alert to Hermes
Reference in New Issue View Git Blame Copy Permalink