inference-harness/LITELLM-MIGRATION-PLAN.md

LiteLLM Integration Migration Plan

Syslog Solution LLC — June 14, 2026

Deployment Target: CT 116 syslog-api (192.168.68.116) on minipve — all services co-located. DNS Strategy: Option A — /etc/hosts + Docker extra_hosts for internal resolution of auth.sysloggh.net → 192.168.68.11. GitOps: This plan lives in SyslogSolution/syslog-harness on Gitea. All changes tracked via git with conventional commits.

---

Executive Summary

Goal: Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness — without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.

Architecture Decision: Two-layer architecture.

                  ┌─────────────────────────────────┐
                  │     LiteLLM Gateway (Layer 1)    │
                  │     Port 4000 — Policy & UX      │
                  │  ┌─────────────────────────────┐ │
                  │  │ Admin UI (/ui)              │ │
                  │  │ Virtual Keys & Permissions  │ │
                  │  │ Teams, Users, SSO (OIDC)    │ │
                  │  │ Spend Tracking & Budgets    │ │
                  │  │ Usage Analytics Dashboard   │ │
                  │  │ Request Audit Trail         │ │
                  │  │ Global Rate Limiting        │ │
                  │  └─────────────────────────────┘ │
                  │           │                       │
                  │    Pass-through to router         │
                  └──────────┬──────────────────────┘
                             │
                  ┌──────────▼──────────────────────┐
                  │  Custom Router (Layer 2)          │
                  │  Port 9000 — Intelligence & HW   │
                  │  ┌─────────────────────────────┐ │
                  │  │ 5-Tier Content-Based Routing│ │
                  │  │ GPU Slot Management (Redis) │ │
                  │  │ Agent Spread Prevention     │ │
                  │  │ GPU Health Scoring (40/30/30)│ │
                  │  │ Sidecar VRAM/Temp/Power     │ │
                  │  │ Circuit Breaker             │ │
                  │  │ Context Window Tracking      │ │
                  │  │ Per-Request Perf Recording  │ │
                  │  │ Hardware Rate Limiting      │ │
                  │  └─────────────────────────────┘ │
                  └──────────┬──────────────────────┘
                             │
          ┌──────────────────┼──────────────────────┐
          │                  │                       │
  ┌───────▼──────┐  ┌────────▼───────┐  ┌───────────▼──────┐
  │ qwen3.6-35B  │  │ qwen3.6-27B   │  │ gemma-4-12b      │
  │ MoE/Strix    │  │ Dense/RTX3090 │  │ VLM/RTX 5070     │
  │ :8080 (llama)│  │ :8080 (llama) │  │ :8080 (llama)    │
  │ :8090 (side) │  │ :8090 (side)  │  │ :8090 (sidecar)  │
  └──────────────┘  └───────────────┘  └──────────────────┘

---

1. Current State Baseline

1.1 Router (router-fixed.py — port 9000, deployed on CT 116 / syslog-api)

Deployment Host: CT 116 syslog-api on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.

Feature	Implementation
Routing Engine	5-tier content-based: lightweight → simple_conv → medium → heavy_reasoning → default
GPU Slot Mgmt	Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset
Health Checks	Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health
Agent Spreading	select_best_gpu() prefers GPUs with 0 other agents, then non-self GPUs
Rate Limiting	Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20
Auth	Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation
Performance	Per-request latency/tokens/tps → Redis lists (perf:recent, perf:model:X, perf:agent:X)
Context Tracking	Session-level token accumulation with compaction warnings in headers
SSE Streaming	Real-time dashboard updates, per-model timeseries
Admin	/admin/keys, /admin/keys/generate, /admin/keys/revoke, /admin/keys/deprecation-summary

1.2 GPU Backends

GPU	Host	llama.cpp	Sidecar	VRAM	Context
qwen3.6-35B-A3B (MoE)	192.168.68.15	:8080	:8090	Strix Halo	262K
qwen3.6-27B-code (Dense)	192.168.68.8	:8080	:8090	RTX 3090	262K
gemma-4-12b (VLM)	192.168.68.110	:8080	:8090	RTX 5070	262K

1.3 Existing LiteLLM POC on CT 116

CT 116 already has a LiteLLM container running (POC, 6 days uptime):

harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081→4000
harness-redis   | redis:7-alpine                      | 127.0.0.1:6379
harness-router  | inference-harness-router             | 127.0.0.1:9000
harness-nginx   | nginx:alpine                        | 0.0.0.0:80
harness-dashboard | inference-harness-dashboard        | 127.0.0.1:3000

• /opt/litellm/ — previous setup directory on CT 116
• Configured with Postgres, host networking, master key
• Currently bypassed — router routes directly to GPUs
• Goal: Productionize with two-layer architecture on this same host

1.4 DNS Routing (Split-Horizon)

For OIDC SSO with Authentik, CT 116 must resolve auth.sysloggh.net internally:

Problem: auth.sysloggh.net CNAMEs to netbird.sysloggh.net → 72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.

Solution — Option A: /etc/hosts on CT 116 host:

# On CT 116 (syslog-api)
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts

Docker containers also need this resolution — add to docker-compose.yml:

services:
  nginx:
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
  litellm:
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"

DNS Servers: CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.

---

2. What LiteLLM Brings (That We Don't Have)

Feature	Our Router	LiteLLM	Value Add
Admin UI	❌	✅ Full dashboard at /ui	Non-technical users can manage keys, view spend
Virtual Key Permissions	❌ (binary key→tier)	✅ Granular: per-model, per-team, budget caps	Fine-grained access control
Spend Tracking	❌	✅ Per-request $ cost with model-specific pricing	Billing, cost allocation, client invoicing
Teams & Orgs	❌	✅ Multi-tenant: org→team→user hierarchy	Segregate clients/projects
SSO/OIDC	❌	✅ Google, GitHub, Microsoft, Okta, Keycloak	Enterprise auth integration
Budget Alerts	❌	✅ Per-key, per-user, per-team budget with webhooks	Prevent overspend
Usage Analytics	⚠️ (custom /metrics)	✅ Built-in: daily trends, model breakdown, per-customer	Better visualization
100+ Provider Support	❌ (3 local GPUs)	✅ OpenAI, Anthropic, Bedrock, Vertex, etc.	Future cloud model access
Fallback Chains	❌	✅ Multi-provider: OpenAI→Azure→Together	External model resilience
RPM/TPM Weighted LB	❌	✅ Weighted load balancing across deployments	Fine-grained traffic shaping

---

3. What We Keep (That LiteLLM Doesn't Have)

Feature	Why We Must Keep It
Content-based 5-tier routing	LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints
GPU hardware health scoring	LiteLLM doesn't monitor VRAM, temp, power — our 40/30/30 scoring prevents routing to overheating GPUs
GPU slot management	LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading
Agent spread prevention	Our select_best_gpu() spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle
Cross-turn context tracking	Session-level token accumulation with compaction warnings via X-Context-Warning headers
GPU sidecar metrics	VRAM %, GPU utilization %, power draw, temperature — exposed via /metrics and SSE dashboard
Circuit breaker	39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular

---

4. Migration Architecture

4.1 Principle: "LiteLLM is the lobby, our router is the engine room"

• LiteLLM handles everything a user/admin touches: keys, teams, budgets, spend logs, SSO, the UI
• Custom Router handles everything the GPUs need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking

4.2 Flow

Agent Request
     │
     ▼
┌─────────────────────────────────────────────┐
│ LiteLLM Gateway (:4000)                      │
│                                              │
│ 1. Authenticate virtual key (sk-litellm-...) │
│ 2. Check key permissions (model access)      │
│ 3. Check budget (per-key, per-user, per-team)│
│ 4. Check team rate limits                    │
│ 5. Log request metadata                      │
│ 6. Forward to custom router as OpenAI-compat  │
│    POST http://router:9000/v1/chat/completions│
│    Headers: Authorization: Bearer <agent-key> │
│             X-LiteLLM-User: <user-id>         │
│             X-LiteLLM-Team: <team-id>         │
│             X-Session-Id: <session>           │
│                                              │
│ 7. On response: log spend, update budgets     │
│ 8. Return response to agent                   │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│ Custom Router (:9000)                         │
│                                               │
│ 1. Authenticate agent key (sk-syslog-...)     │
│ 2. Hardware rate limit (per-tier RPM)         │
│ 3. Content-based tier routing                 │
│    - Estimate tokens, detect system msg       │
│    - Count turns, check routing_hints         │
│ 4. GPU slot availability (Redis counter)      │
│ 5. GPU health check (sidecar)                 │
│ 6. Agent spread logic (select_best_gpu)       │
│ 7. Queue if saturated (with timeout)          │
│ 8. Forward to selected llama.cpp GPU          │
│ 9. Track context window, set compaction header│
│ 10. Record performance metrics                │
│ 11. Return response (with routing metadata)   │
└──────────────┬───────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────┐
│ llama.cpp GPU (:8080)                         │
└──────────────────────────────────────────────┘

4.3 LiteLLM Config (config.yaml)

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm
  store_model_in_db: true

model_list:
  # All three GPUs exposed as a single virtual "syslog-router" model
  # LiteLLM passes through to our router, which handles actual GPU selection
  - model_name: syslog-auto           # Default auto-routing
    litellm_params:
      model: openai/syslog-auto       # Using OpenAI-compatible format
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
      rpm: 600                         # Cap total RPM across all GPUs
  
  # Individual GPU pass-through (for explicit model requests)
  - model_name: qwen3.6-35B-A3B
    litellm_params:
      model: openai/qwen3.6-35B-A3B
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY
  
  - model_name: qwen3.6-27B-code
    litellm_params:
      model: openai/qwen3.6-27B-code
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

  - model_name: gemma-4-12b
    litellm_params:
      model: openai/gemma-4-12b
      api_base: http://router:9000/v1
      api_key: os.environ/ROUTER_API_KEY

# Guardrails: Pre-call and post-call content moderation
guardrails:
  - guardrail_name: "input-moderation"
    litellm_params:
      guardrail: openai_moderation
      mode: "pre_call"

  - guardrail_name: "output-moderation"
    litellm_params:
      guardrail: openai_moderation
      mode: "post_call"

  - guardrail_name: "harmful-content-filter"
    litellm_params:
      guardrail: litellm_content_filter
      mode: "pre_call"
      categories:
        - category: "harmful_self_harm"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"
        - category: "harmful_violence"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"
        - category: "harmful_illegal_weapons"
          enabled: true
          action: "BLOCK"
          severity_threshold: "medium"

litellm_settings:
  num_retries: 0                     # Disabled — our router handles retry
  request_timeout: 600               # Match our 10-min llama-server timeout
  set_verbose: true
  failure_callback: ["prometheus"]   # Optional: export to Prometheus

router_settings:
  routing_strategy: "usage-based-routing"  # For external models only
  # Note: All local GPU routing is handled by custom router
  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
  allowed_fails: 100                       # Don't cooldown — our circuit breaker handles

# Cost tracking: map model names to per-token pricing
# These are passed through from our router's X-Usage-Tokens header

4.4 Router Modifications (Light Touch)

Minimal changes to router-fixed.py — the router remains largely unchanged:

1. New header passthrough: Forward X-LiteLLM-* headers to GPU (transparent — already works)
2. New endpoint for health passthrough: GET /v1/models already works
3. Disable own key management: Remove /admin/keys/* endpoints (migrate to LiteLLM UI)
4. Keep ALL routing logic: No changes to route(), select_best_gpu(), check_gpu_health(), slot management, etc.
5. Add LiteLLM-compatible response: Return X-Usage-Tokens header so LiteLLM can track token costs

# ADD to router-fixed.py chat() response:
resp.headers["X-Usage-Tokens"] = json.dumps({
    "prompt_tokens": prompt_tokens,
    "completion_tokens": completion_tokens,
    "model": model
})

---

5. Deployment Plan (4 Phases) — Zero-Downtime Strategy

Phase 0: Infrastructure Prep (Current Week) — Zero Downtime

Goal: Prepare CT 116 infrastructure without affecting running agents.

Tasks:

1. Set up DNS split-horizon on CT 116

   # On CT 116 host (syslog-api)
   echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
   

   Add to docker-compose.yml (see §7):

   extra_hosts:
     - "auth.sysloggh.net:192.168.68.11"
   

2. Deploy Postgres container alongside existing services:

   cd /opt/litellm
   # Add postgres to docker-compose.yml
   docker compose up -d postgres
   

3. Replace LiteLLM config with production config.yaml (see §4.3)

   • All 4 models → http://router:9000/v1
   • Add guardrails (pre-call, post-call, content filter)
   • Set num_retries: 0 (router handles retry)
   • Set request_timeout: 600

4. Deploy custom_sso.py for Authentik OIDC integration

   • Mount to LiteLLM container volume
   • Reference in config.yaml: custom_ui_sso_sign_in_handler: custom_sso.custom_ui_sso_sign_in_handler

5. Restart LiteLLM container with new config

   docker compose restart litellm
   

6. Verify internal routing — LiteLLM → Router pass-through works:

   curl -X POST http://127.0.0.1:4000/v1/chat/completions \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
   

Verification Checklist:

• DNS resolution: getent hosts auth.sysloggh.net → 192.168.68.11
• Postgres container healthy
• LiteLLM /health returns 200
• LiteLLM → Router pass-through returns valid chat completion
• GPU health metrics unaffected (watch /metrics)

Phase 1: Shadow Mode (Week 1) — Zero Risk, Zero Downtime

Goal: Deploy LiteLLM alongside existing router, test in shadow mode. Agents continue using :9000 directly.

Agent → LiteLLM (:4000) → Router (:9000) → GPU
         (new, testing)    (existing, unchanged)
         
Agent can also directly hit :9000 as fallback (unchanged)

Tasks:

1. Create virtual keys for test agents via LiteLLM UI or API:

   curl -X POST http://127.0.0.1:4000/key/generate \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H "Content-Type: application/json" \
     -d '{"models":["syslog-auto"],"metadata":{"agent":"test"}}'
   

   • Mirror existing API_KEYS in LiteLLM's key store
   • Set per-key budgets (test with $100 cap)

2. Verify pass-through works

   curl -X POST http://127.0.0.1:4000/v1/chat/completions \
     -H "Authorization: Bearer sk-litellm-test-key" \
     -H "Content-Type: application/json" \
     -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
   

3. Run 24-hour shadow: Both :4000 and :9000 active, agents use :9000

   • Monitor LiteLLM spend logs vs router metrics — confirm parity
   • Verify GPU health metrics unaffected
   • Check guardrails not generating false positives

Zero-Downtime Guarantee: Router :9000 remains the primary agent endpoint. LiteLLM :4000 is tested in parallel with no impact on production traffic.

Phase 2: Cutover (Week 2) — Gradual Agent Migration, Zero Cumulative Downtime

Goal: Move agents one-by-one to LiteLLM endpoint. Each agent is migrated individually — other agents unaffected.

Tasks:

1. Migrate API keys to LiteLLM virtual keys:

   • Create virtual key per agent in LiteLLM UI
   • Set model access: syslog-auto (default), plus individual GPU models
   • Set per-agent budget limits
   • Create teams:
     • "Core Agents" (Abiba, Mumuni, Tanko) — enterprise tier
     • "Dev Agents" (Kagenz0, Koby, Koonimo) — professional tier

2. Update agent configs one at a time:

   • Change OPENAI_API_BASE from http://192.168.68.116:9000/v1 → http://192.168.68.116:4000/v1
   • Replace agent API keys with LiteLLM virtual keys
   • Test each agent individually — verify routing works
   • Per-agent downtime: <2 minutes

3. Migrate admin functions:

   • Key creation/revocation → LiteLLM UI
   • Rate limit management → LiteLLM per-key RPM + router hardware RPM (dual enforcement)
   • Deprecated key tracking → LiteLLM UI key list

4. Enable SSO via Authentik + custom_sso.py:

   • Authentik OAuth2 provider configured
   • NGINX auth_request for /ui/* paths
   • Custom UI SSO sign-in handler reads x-authentik-* headers
   • Users authenticate with existing Authentik credentials

5. Keep router :9000 accessible as emergency fallback for 48 hours

   • NGINX configured with router_fallback for 502 errors
   • Agents can revert by changing OPENAI_API_BASE back to :9000

Zero-Downtime Guarantee: Each agent has <2 minutes downtime during config update. Router :9000 stays online throughout. Fallback path available for instant rollback.

Phase 3: Production Hardening (Week 3+) — Optimize & Scale

Goal: Lock down, optimize, monitor. Prepare for client-facing services.

Tasks:

1. Remove deprecated router endpoints (only after all agents migrated):

   • Drop /admin/keys/* — fully migrated to LiteLLM UI
   • Drop Phase 0 dual-key logic (LiteLLM handles key rotation)
   • Simplify API_KEYS to single ROUTER_API_KEY

2. Add LiteLLM observability:

   • Prometheus metrics export
   • Slack/email budget alerts via webhook → Hermes
   • Daily spend report webhook

3. Enable LiteLLM caching (Redis, shared with router):

   router_settings:
     redis_host: os.environ/REDIS_HOST
     redis_port: 6379
     cache: true
     cache_ttl: 3600
   

4. Add external model fallbacks for client-facing services:

   • Add Anthropic Claude as fallback for code-heavy requests
   • Add OpenAI GPT-4o as fallback for reasoning overflow
   • LiteLLM's native fallback chains handle this cleanly

5. Router slim-down: Extract GPU health metrics to dedicated /health only

   • Keep: routing, slots, health checks, performance recording
   • Remove: key management, dual-key logic, admin endpoints

6. Multi-tenancy setup for client-facing inference services:

   • Organization → Team → User hierarchy per client
   • Per-client budget limits and guardrail policies
   • Self-service onboarding via Authentik SSO → LiteLLM UI

Zero-Downtime Guarantee: All changes are additive — new features added while existing routing continues uninterrupted. Router remains the GPU intelligence layer throughout.

---

---

6. Nginx Configuration (with Authentik OIDC Forward Auth)

The existing nginx config routes /admin/ → router :9000. This MUST change:

# OLD (remove)
# location /admin/ {
#     proxy_pass http://127.0.0.1:9000/admin/;
# }

# === Authentik auth subrequest endpoint ===
location /authentik/auth {
    internal;
    # Proxy to Authentik's outpost on acerpve (192.168.68.11)
    # Uses internal /etc/hosts resolution: auth.sysloggh.net → 192.168.68.11
    proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

# === LiteLLM Admin UI — Authentik-protected ===
location /ui/ {
    auth_request /authentik/auth;
    auth_request_set $auth_user $upstream_http_x_authentik_username;
    auth_request_set $auth_email $upstream_http_x_authentik_email;

    proxy_set_header X-Authentik-Username $auth_user;
    proxy_set_header X-Authentik-Email $auth_email;

    proxy_pass http://127.0.0.1:4000/ui/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

# === LiteLLM SSO callback (for OIDC redirect flow) ===
location /sso/callback {
    proxy_pass http://127.0.0.1:4000/sso/callback;
    proxy_set_header Host $host;
}

# === API endpoint — Bearer token auth (no Authentik) ===
location /v1/ {
    # Primary: LiteLLM gateway
    proxy_pass http://127.0.0.1:4000/v1/;
    proxy_set_header Host $host;
    proxy_read_timeout 600s;
    # Fallback: direct router (if LiteLLM down)
    error_page 502 = @router_fallback;
}

location @router_fallback {
    proxy_pass http://127.0.0.1:9000/v1/;
    proxy_set_header Host $host;
}

# === Key management API (needs master_key, not Authentik) ===
location /key/ {
    proxy_pass http://127.0.0.1:4000/key/;
    proxy_set_header Host $host;
    proxy_set_header Authorization $http_authorization;
}

# Keep router metrics accessible (not behind LiteLLM)
location /router/ {
    proxy_pass http://127.0.0.1:9000/;
}

# Health check — combines both layers
location /health {
    proxy_pass http://127.0.0.1:4000/health;
}

---

7. Docker Compose (docker-compose.yml on CT 116)

Deployment host: CT 116 syslog-api (192.168.68.116) on minipve. All services co-located.

services:
  # Layer 1: LiteLLM Gateway (Policy & Admin)
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    network_mode: "host"
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - ./custom_sso.py:/app/custom_sso.py:ro
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - DATABASE_URL=postgresql://litellm:${POSTGRES_PASSWORD}@localhost:5432/litellm
      - STORE_MODEL_IN_DB=True
      - ROUTER_API_KEY=${ROUTER_API_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}            # Optional: external fallback
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}       # Optional: external fallback
      - PROXY_BASE_URL=https://litellm.sysloggh.net
    command:
      - --config
      - /app/config.yaml
      - --port
      - "4000"
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped

  # Database for LiteLLM
  postgres:
    image: postgres:16-alpine
    network_mode: "host"
    environment:
      - POSTGRES_DB=litellm
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  # Nginx also needs auth resolution
  nginx:
    image: nginx:alpine
    extra_hosts:
      - "auth.sysloggh.net:192.168.68.11"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:80"
    restart: unless-stopped

  # Layer 2: Custom Router (Intelligence & Hardware)
  # Already deployed separately — not in this compose file
  # The router is managed by the existing harness deployment on CT 116

volumes:
  pgdata:

---

8. Risk Mitigation

Risk	Mitigation
LiteLLM adds latency overhead	Shadow mode measures: <50ms extra is acceptable for admin features. LiteLLM is a thin proxy.
LiteLLM down = all agents down	Nginx fallback to router :9000 direct (see §6). Agents can also be configured with dual endpoints.
Key sync drift (LiteLLM keys ≠ router keys)	Single-source: LiteLLM is key authority. Router uses one ROUTER_API_KEY from LiteLLM's perspective. Agent keys live in LiteLLM only.
Spend tracking inaccurate for local GPUs	Configure model_cost per GPU with $0 rate (self-hosted). Optionally track "internal cost" via custom pricing.
Double rate limiting (LiteLLM + Router)	Keep both intentionally: LiteLLM for per-user soft caps, Router for hardware protection. Non-overlapping concerns.
PostgreSQL failure	LiteLLM can run with SQLite fallback, but UI features degrade. Postgres is the recommended path.
Router custom logic becomes a black box to LiteLLM	Acceptable trade-off. LiteLLM sees router as opaque OpenAI endpoint. GPU-level routing decisions are router's domain.

---

9. Success Metrics

Metric	Before	After
Key management	Manual CLI + env vars + redeploy	UI-based, instant, no redeploy
Spend visibility	None	Per-agent, per-team, per-model $ tracking
Access control	Tier-based (3 levels)	Per-key, per-model, budget-capped
New agent onboarding	Generate key, update env var, redeploy router	Create in UI, share key
Admin UX	curl + JSON responses	Visual dashboard, graphs, search
Audit trail	Router logs (stdout only)	Database-backed with UI search
SSO	None	Google/GitHub/Microsoft OIDC
Budget enforcement	None	Automatic: key suspended at $limit
GPU routing intelligence	Full (unchanged)	Full (unchanged)
GPU health monitoring	Full (unchanged)	Full (unchanged)

---

10. Migration Commands (Quick Reference) — CT 116

Target: CT 116 syslog-api (192.168.68.116), minipve. All services co-located.

# On CT 116 (SSH via minipve: pct exec 116 bash):

# === Phase 0: Infrastructure Prep ===

# 0. DNS split-horizon fix
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
getent hosts auth.sysloggh.net  # Verify → 192.168.68.11

# 1. Navigate to harness deployment directory
cd /opt/litellm

# 2. Deploy Postgres (LiteLLM DB)
docker compose up -d postgres

# 3. Replace LiteLLM config with production config.yaml (see §4.3)
#    - All models → http://router:9000/v1
#    - Guardrails enabled
#    - custom_sso.py mounted

# 4. Restart LiteLLM with new config
docker compose restart litellm

# 5. Verify core health
curl http://127.0.0.1:4000/health           # LiteLLM alive
curl http://127.0.0.1:9000/health           # Router alive
curl http://127.0.0.1:4000/health/liveliness  # LiteLLM readiness

# 6. Test LiteLLM → Router pass-through
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"ping"}]}'
# Expected: valid chat completion with GPU routing metadata

# === Phase 1: Shadow Mode ===

# 7. Create virtual key for testing
curl -X POST http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models":["syslog-auto","qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"],"metadata":{"agent":"test"},"max_budget":100}'

# 8. Test with virtual key
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer <virtual-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"Hello from LiteLLM!"}]}'

# 9. Verify NGINX → Authentik routing (after Phase 2)
curl -I http://127.0.0.1/ui/               # Should return 302 → Authentik login
curl -I http://127.0.0.1/v1/chat/completions  # Should proxy to LiteLLM :4000

# 10. Monitor both layers
curl http://127.0.0.1:4000/global/spend/logs   # LiteLLM spend
curl http://127.0.0.1:9000/metrics             # Router GPU metrics
curl http://127.0.0.1:9000/stream              # Router SSE dashboard

# 11. NGINX reload after config changes
nginx -t && nginx -s reload

# 12. View logs
docker compose logs -f litellm    # LiteLLM gateway logs
docker compose logs -f nginx      # Reverse proxy logs

---

11. GitOps Workflow

All harness changes tracked in SyslogSolution/syslog-harness on Gitea (http://192.168.68.17:3000). Multiple agents (Abiba, Mumuni, Kagenz0, Agent Zero) collaborate on this repo.

Branching Strategy:

• main — Production-ready, deployed on CT 116
• dev/<feature> — Feature branches for each migration phase
• PR required for main merges (allows cross-agent review)

Conventional Commits:

feat(router): add X-Usage-Tokens header for LiteLLM cost tracking
fix(docker): add extra_hosts for Authentik internal DNS
chore(plan): update LITELLM-MIGRATION-PLAN.md with Phase 0-3 zero-downtime
refactor(router): remove deprecated /admin/keys/* endpoints (Phase 3)
infra(dns): configure /etc/hosts split-horizon for CT 116

Commit Signatures (for audit trail):

• Agent commits include source: Co-authored-by: Abiba <abiba@sysloggh.net>
• Human commits: Signed-off-by: Jerome Tabiri <jerome@sysloggh.net>

Deployment Flow:

1. PR opened on Gitea (feature → main)
2. Review by at least one other agent or Jerome
3. Merge to main
4. git pull on CT 116 (/opt/litellm/)
5. docker compose up -d (rolling update)

Post-Deployment Verification:

• Check LiteLLM /health — 200 OK
• Check Router /metrics — GPU health unchanged
• Check Dashboard SSE — streaming live
• Test agent inference — valid chat completion

---

12. Zero-Downtime Migration Strategy

Principle: Agents must experience zero cumulative downtime throughout the migration. Router :9000 is the safety net — never taken offline until all agents verified on LiteLLM.

Per-Agent Migration (2 min downtime each):

1. Create LiteLLM virtual key for agent (no impact on running agent)
2. Update agent config: OPENAI_API_BASE → :4000, OPENAI_API_KEY → virtual key
3. Restart agent (2 min)
4. Verify inference works via LiteLLM → Router → GPU
5. If issue: revert OPENAI_API_BASE → :9000, restart (30 sec rollback)

Dual-Path Architecture (throughout migration):

Path A: Agent → LiteLLM :4000 → Router :9000 → GPU  (new, for migrated agents)
Path B: Agent → Router :9000 → GPU                  (legacy, for unmigrated agents)

Both paths work simultaneously. Router unchanged in both paths.

Rollback Procedure (per agent, 30 seconds):

1. Set OPENAI_API_BASE=http://192.168.68.116:9000/v1
2. Set OPENAI_API_KEY=<original-agent-key>
3. Restart agent

Emergency Global Rollback (if LiteLLM catastrophic failure):

1. NGINX auto-fallback: error_page 502 = @router_fallback → traffic routes to :9000
2. Manual: Update all agent configs back to :9000 (scripted)
3. Restart all agents (5 min)

Health Checks (continuously monitored):

• /health — LiteLLM + Router combined status
• /router/metrics — GPU health, circuit breaker, active slots
• /stream — SSE dashboard (real-time)
• Redis: circuit:* keys for tripped breakers

---

Appendix A: Router Slim-Down (Phase 3)

After full migration, router-fixed.py can be simplified by removing:

# REMOVE (migrated to LiteLLM):
- API_KEYS validation logic (keep single ROUTER_API_KEY)
- Dual-key deprecation tracking
- /admin/keys, /admin/keys/generate, /admin/keys/revoke
- /admin/keys/deprecation-summary
- Phase 0 deprecated key logging
- check_rate_limit() (optional — keep as hardware safety net)

# KEEP:
- route() — all 5 tiers
- select_best_gpu()
- check_gpu_health()
- is_gpu_busy(), gpu_active_count(), gpu_incr/decr()
- estimate_tokens()
- store_perf_record()
- GPU_SIDECARS, GPU_URLS, GPU_MAX_CONCURRENT, GPU_CONTEXT
- counter_audit_loop()
- /v1/chat/completions — core routing endpoint
- /v1/models
- /health
- /metrics, /metrics/performance, /metrics/scatter, /metrics/timeseries
- /stream — SSE dashboard

Appendix B: LiteLLM Cost Config for Local GPUs

# In config.yaml — map models to per-token pricing for spend tracking
litellm_settings:
  model_cost:
    qwen3.6-35B-A3B:
      input_cost_per_token: 0.0        # Self-hosted, no external cost
      output_cost_per_token: 0.0
    qwen3.6-27B-code:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
    gemma-4-12b:
      input_cost_per_token: 0.0
      output_cost_per_token: 0.0
  # For internal cost allocation, set symbolic rates:
  # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens

---

Plan drafted: 2026-06-13 by Abiba 🦊⚡ Updated: 2026-06-14 by Agent Zero (kagentz-bot) — CT 116 deployment target, DNS split-horizon, Authentik OIDC, Zero-Downtime strategy, GitOps workflow Status: Ready for Phase 0 deployment