auto-fix: harness-dashboard restarted — container was down, now healthy

# Conflicts:
#	litellm_config.yaml
#	nginx/nginx.conf

.gitignore

@@ -1,3 +1,6 @@
 .git
 __pycache__/
 *.pyc
+*.bak
+*.backup*
+.env

LITELLM-MIGRATION-PLAN.md

@@ -0,0 +1,759 @@
+# LiteLLM Integration Migration Plan
+## Syslog Solution LLC  June 14, 2026
+
+**Deployment Target:** CT 116 `syslog-api` (192.168.68.116) on minipve  all services co-located.
+**DNS Strategy:** Option A  /etc/hosts + Docker extra_hosts for internal resolution of `auth.sysloggh.net`  192.168.68.11.
+**GitOps:** This plan lives in `SyslogSolution/syslog-harness` on Gitea. All changes tracked via git with conventional commits.
+
+---
+
+## Executive Summary
+
+**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness  without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.
+
+**Architecture Decision:** Two-layer architecture.
+
+```
+                  
+                       LiteLLM Gateway (Layer 1)    
+                       Port 4000  Policy & UX      
+                     
+                     Admin UI (/ui)               
+                     Virtual Keys & Permissions   
+                     Teams, Users, SSO (OIDC)     
+                     Spend Tracking & Budgets     
+                     Usage Analytics Dashboard    
+                     Request Audit Trail          
+                     Global Rate Limiting         
+                     
+                                                    
+                      Pass-through to router         
+                  
+                             
+                  
+                    Custom Router (Layer 2)          
+                    Port 9000  Intelligence & HW   
+                     
+                     5-Tier Content-Based Routing 
+                     GPU Slot Management (Redis)  
+                     Agent Spread Prevention      
+                     GPU Health Scoring            
+                     Sidecar VRAM/Temp/Power      
+                     Circuit Breaker              
+                     Context Window Tracking       
+                     Per-Request Perf Recording   
+                     Hardware Rate Limiting       
+                     
+                  
+                             
+          
+                                                   
+      
+   qwen3.6-35B     qwen3.6-27B      gemma-4-12b      
+   MoE/Strix       Dense/RTX3090    VLM/RTX 5070     
+   :8080 (llama)   :8080 (llama)    :8080 (llama)    
+   :8090 (side)    :8090 (side)     :8090 (sidecar)  
+      
+```
+
+---
+
+## 1. Current State Baseline
+
+### 1.1 Router (`router-fixed.py`  port 9000, deployed on CT 116 / syslog-api)
+
+**Deployment Host:** CT 116 `syslog-api` on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.
+
+| Feature | Implementation |
+|---------|---------------|
+| **Routing Engine** | 5-tier content-based: lightweight  simple_conv  medium  heavy_reasoning  default |
+| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset |
+| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health |
+| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs |
+| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 |
+| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation |
+| **Performance** | Per-request latency/tokens/tps  Redis lists (perf:recent, perf:model:X, perf:agent:X) |
+| **Context Tracking** | Session-level token accumulation with compaction warnings in headers |
+| **SSE Streaming** | Real-time dashboard updates, per-model timeseries |
+| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` |
+| **Strict Passthrough** | Explicit model requests go to that GPU exactly (no silent fallback). LiteLLM owns failover. |
+
+### 1.2 GPU Backends
+
+| GPU | Host | llama.cpp | Sidecar | VRAM | Context |
+|-----|------|-----------|---------|------|---------|
+| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K |
+| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K |
+| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K |
+
+### 1.3 Existing LiteLLM POC on CT 116
+
+CT 116 already has a LiteLLM container running (POC, 6 days uptime):
+
+```
+harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081->4000
+harness-redis   | redis:7-alpine                      | 127.0.0.1:6379
+harness-router  | inference-harness-router             | 127.0.0.1:9000
+harness-nginx   | nginx:alpine                        | 0.0.0.0:80
+harness-dashboard | inference-harness-dashboard        | 127.0.0.1:3000
+```
+
+- `/opt/litellm/`  previous setup directory on CT 116
+- Configured with Postgres, host networking, master key
+- Currently bypassed  router routes directly to GPUs
+- **Goal: Productionize with two-layer architecture on this same host**
+
+### 1.4 DNS Routing (Split-Horizon)
+
+For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally:
+
+**Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net`  72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.
+
+**Solution  Option A: /etc/hosts on CT 116 host:**
+```bash
+# On CT 116 (syslog-api)
+echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
+```
+
+**Docker containers** also need this resolution  add to docker-compose.yml:
+```yaml
+services:
+  nginx:
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+  litellm:
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+```
+
+**DNS Servers:** CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.
+
+---
+
+## 2. What LiteLLM Brings (That We Don't Have)
+
+| Feature | Our Router | LiteLLM | Value Add |
+|---------|-----------|---------|-----------|
+| **Admin UI** |  |  Full dashboard at /ui | Non-technical users can manage keys, view spend |
+| **Virtual Key Permissions** |  (binary key->tier) |  Granular: per-model, per-team, budget caps | Fine-grained access control |
+| **Spend Tracking** |  |  Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing |
+| **Teams & Orgs** |  |  Multi-tenant: org->team->user hierarchy | Segregate clients/projects |
+| **SSO/OIDC** |  |  Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration |
+| **Budget Alerts** |  |  Per-key, per-user, per-team budget with webhooks | Prevent overspend |
+| **Usage Analytics** |   (custom /metrics) |  Built-in: daily trends, model breakdown, per-customer | Better visualization |
+| **100+ Provider Support** |  (3 local GPUs) |  OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access |
+| **Fallback Chains** |  (silent rerouting) |  Explicit multi-provider failover with per-model logging | Accurate per-model tracking, visible failover |
+| **RPM/TPM Weighted LB** |  |  Weighted load balancing across deployments | Fine-grained traffic shaping |
+
+---
+
+## 3. What We Keep (That LiteLLM Doesn't Have)
+
+| Feature | Why We Must Keep It |
+|---------|---------------------|
+| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints |
+| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power  our scoring prevents routing to overheating GPUs |
+| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading |
+| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle |
+| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers |
+| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature  exposed via /metrics and SSE dashboard |
+| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular |
+
+---
+
+## 4. Migration Architecture
+
+### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room"
+
+- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI
+- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking
+
+### 4.2 Flow
+
+```
+Agent Request
+     
+     
+
+ LiteLLM Gateway (:4000)                      
+                                              
+ 1. Authenticate virtual key (sk-litellm-...) 
+ 2. Check key permissions (model access)      
+ 3. Check budget (per-key, per-user, per-team)
+ 4. Check team rate limits                    
+ 5. Log request metadata                      
+ 6. Forward to custom router as OpenAI-compat  
+    POST http://router:9000/v1/chat/completions
+    Headers: Authorization: Bearer *** 
+             X-LiteLLM-User: <user-id>         
+             X-LiteLLM-Team: <team-id>         
+             X-Session-Id: <session>           
+                                              
+ 7. On response: log spend, update budgets     
+ 8. If router returns 503 (GPU saturated):     
+    consult fallback chain, retry next model    
+ 9. Return response to agent                   
+
+               
+               
+
+ Custom Router (:9000)                         
+                                               
+ 1. Authenticate agent key (sk-syslog-...)     
+ 2. Hardware rate limit (per-tier RPM)         
+ 3. Content-based tier routing (for syslog-auto)
+    OR strict passthrough (for explicit models) 
+ 4. GPU slot availability (Redis counter)      
+ 5. GPU health check (sidecar)                 
+ 6. Agent spread logic (select_best_gpu)       
+ 7. Queue if saturated (with timeout)          
+ 8. Forward to selected llama.cpp GPU          
+ 9. Track context window, set compaction header
+ 10. Record performance metrics                
+ 11. Return response (with routing metadata)   
+
+               
+               
+
+ llama.cpp GPU (:8080)                         
+
+```
+
+### 4.3 LiteLLM Config (`config.yaml`)
+
+```yaml
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: postgresql://litellm:***@postgres:5432/litellm
+  store_model_in_db: true
+
+model_list:
+  # Content-based auto-routing (router picks GPU via 5-tier analysis)
+  - model_name: syslog-auto
+    litellm_params:
+      model: openai/syslog-auto
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+      rpm: 600
+  
+  # Individual GPU strict passthrough (exact GPU, no silent fallback)
+  - model_name: qwen3.6-35B-A3B
+    litellm_params:
+      model: openai/qwen3.6-35B-A3B
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+  
+  - model_name: qwen3.6-27B-code
+    litellm_params:
+      model: openai/qwen3.6-27B-code
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+  - model_name: gemma-4-12b
+    litellm_params:
+      model: openai/gemma-4-12b
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+# Guardrails: Pre-call and post-call content moderation
+guardrails:
+  - guardrail_name: "input-moderation"
+    litellm_params:
+      guardrail: openai_moderation
+      mode: "pre_call"
+
+  - guardrail_name: "output-moderation"
+    litellm_params:
+      guardrail: openai_moderation
+      mode: "post_call"
+
+  - guardrail_name: "harmful-content-filter"
+    litellm_params:
+      guardrail: litellm_content_filter
+      mode: "pre_call"
+      categories:
+        - category: "harmful_self_harm"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+        - category: "harmful_violence"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+        - category: "harmful_illegal_weapons"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+
+litellm_settings:
+  num_retries: 0                     # Disabled  our router handles retry
+  request_timeout: 600               # Match our 10-min llama-server timeout
+  set_verbose: true
+  failure_callback: ["prometheus"]   # Optional: export to Prometheus
+
+router_settings:
+  routing_strategy: "usage-based-routing"  # For external models only
+  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
+  allowed_fails: 100                       # Router returns 503 on saturated GPUs  cooldown disabled
+  # Fallback chains: LiteLLM retries down the chain when router returns saturated
+  # This gives accurate per-model metrics because router no longer silently reroutes
+  fallbacks:
+    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
+    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
+    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
+
+# Cost tracking: map model names to per-token pricing for spend tracking
+litellm_settings:
+  model_cost:
+    qwen3.6-35B-A3B:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    qwen3.6-27B-code:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    gemma-4-12b:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    # For internal cost allocation, set symbolic rates:
+    # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
+```
+
+### 4.4 Router Modifications
+
+To accommodate LiteLLM, `router-fixed.py` requires the following updates:
+
+1. **Strict passthrough for explicit models** (DEPLOYED):
+   ```python
+   # In route(), the explicit model section changed from silent fallback to strict:
+   req = rd.get("model","auto")
+   if req != "auto":
+       # STRICT MODE: no silent fallback  LiteLLM handles failover chains.
+       # This keeps per-model metrics accurate. Returns saturated if busy.
+       target = req if req in avail else avail[0]
+       if req not in avail:
+           return {"model": req, "reason": "explicit_unavailable", "saturated": True}
+       if is_gpu_busy(target):
+           return {"model": target, "reason": "explicit_saturated", "saturated": True}
+       return {"model": target, "reason": "explicit"}
+   ```
+
+2. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent  already works)
+
+3. **New endpoint for health passthrough**: `GET /v1/models` already works
+
+4. **Keep ALL routing logic**: No changes to `select_best_gpu()`, `check_gpu_health()`, slot management, etc. Content-based routing for `syslog-auto` is fully intact.
+
+5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs
+   ```python
+   resp.headers["X-Usage-Tokens"] = json.dumps({
+       "prompt_tokens": prompt_tokens,
+       "completion_tokens": completion_tokens,
+       "model": model
+   })
+   ```
+
+### 4.5 Router Logic Refinements
+
+**GPU Health Scoring (Updated):**
+We are updating the scoring algorithm to include Power metrics:
+```python
+def gpu_health_score(model):
+    h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
+    if h.get("status") == "down":
+        return 999  # never pick down GPUs
+    vram_pct = h.get("vram_pct") or 50
+    temp_c = h.get("temp_c") or 50
+    power_w = h.get("power_w") or 50
+    active = gpu_active_count(model)
+    max_c = GPU_MAX_CONCURRENT.get(model, 1)
+    load_pct = (active / max_c) * 100 if max_c > 0 else 0
+    # Score: lower = better
+    score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2))
+    return score
+```
+
+---
+
+## 5. Deployment Plan (4 Phases)  Zero-Downtime Strategy
+
+### Phase 0: Infrastructure Prep (Current Week)  Zero Downtime
+
+**Goal:** Prepare CT 116 infrastructure without affecting running agents.
+
+**Tasks:**
+1. **Set up DNS split-horizon on CT 116**
+   ```bash
+   echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
+   ```
+
+2. **Deploy Postgres container** alongside existing services
+
+3. **Replace LiteLLM config** with production config.yaml (see 4.3)
+   - All 4 models  `http://router:9000/v1`
+   - Fallback chains for explicit models
+   - Guardrails (pre-call, post-call, content filter)
+   - `allowed_fails: 100` (router returns 503 on saturated)
+   - `num_retries: 0` (LiteLLM retries handled by fallback chains)
+
+4. **Deploy custom_sso.py** for Authentik OIDC integration
+
+5. **Restart LiteLLM container** with new config
+
+6. **Verify internal routing**
+
+**Verification Checklist:**
+- [ ] DNS resolution: `getent hosts auth.sysloggh.net`  192.168.68.11
+- [ ] Postgres container healthy
+- [ ] LiteLLM `/health` returns 200
+- [ ] LiteLLM  Router pass-through returns valid chat completion
+- [ ] GPU health metrics unaffected
+- [ ] Explicit model request returns saturated (not silently rerouted) when GPU busy
+
+### Phase 1: Shadow Mode (Week 1)  Zero Risk, Zero Downtime
+
+**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. **Agents continue using :9000 directly.**
+
+**Tasks:**
+1. Create virtual keys for test agents via LiteLLM UI
+2. Verify pass-through works for all 4 models
+3. Validate fallback chains: saturate MoE  confirm LiteLLM retries Dense  confirm VLM
+4. Run 24-hour shadow: monitor LiteLLM spend logs vs router metrics
+5. Verify GPU health metrics unaffected
+6. Check guardrails not generating false positives
+
+### Phase 2: Cutover (Week 2)  Gradual Agent Migration
+
+**Goal:** Move agents one-by-one to LiteLLM endpoint.
+
+**Tasks:**
+1. Migrate API keys to LiteLLM virtual keys
+2. Create teams: "Core Agents" (enterprise), "Dev Agents" (professional)
+3. Update agent configs one at a time: `OPENAI_API_BASE`  `:4000`
+4. Test each agent individually
+5. Enable SSO via Authentik + custom_sso.py
+6. Keep router :9000 as emergency fallback for 48 hours
+
+### Phase 3: Production Hardening (Week 3+)
+
+**Goal:** Lock down, optimize, monitor.
+
+**Tasks:**
+1. Remove deprecated router endpoints (after all agents migrated)
+2. Add LiteLLM observability (Prometheus, Slack/email alerts)
+3. Enable LiteLLM caching (shared Redis)
+4. Add external model fallbacks for client-facing services
+5. Router slim-down: keep routing/slots/health/perf, remove key management
+6. Multi-tenancy setup for client-facing inference services
+
+---
+
+## 6. Nginx Configuration (with Authentik OIDC Forward Auth)
+
+```nginx
+# OLD (remove)
+# location /admin/ {
+#     proxy_pass http://127.0.0.1:9000/admin/;
+# }
+
+# === Authentik auth subrequest endpoint ===
+location /authentik/auth {
+    internal;
+    proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+    proxy_pass_request_body off;
+    proxy_set_header Content-Length "";
+    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+}
+
+# === LiteLLM Admin UI  Authentik-protected ===
+location /ui/ {
+    auth_request /authentik/auth;
+    auth_request_set $auth_user $upstream_http_x_authentik_username;
+    auth_request_set $auth_email $upstream_http_x_authentik_email;
+
+    proxy_set_header X-Authentik-Username $auth_user;
+    proxy_set_header X-Authentik-Email $auth_email;
+
+    proxy_pass http://127.0.0.1:4000/ui/;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+}
+
+# === LiteLLM SSO callback ===
+location /sso/callback {
+    proxy_pass http://127.0.0.1:4000/sso/callback;
+    proxy_set_header Host $host;
+}
+
+# === API endpoint  Bearer token auth ===
+location /v1/ {
+    proxy_pass http://127.0.0.1:4000/v1/;
+    proxy_set_header Host $host;
+    proxy_read_timeout 600s;
+    error_page 502 = @router_fallback;
+}
+
+location @router_fallback {
+    proxy_pass http://127.0.0.1:9000/v1/;
+    proxy_set_header Host $host;
+}
+
+# === Key management API ===
+location /key/ {
+    proxy_pass http://127.0.0.1:4000/key/;
+    proxy_set_header Host $host;
+    proxy_set_header Authorization $http_authorization;
+}
+
+# Keep router metrics accessible (not behind LiteLLM)
+location /router/ {
+    proxy_pass http://127.0.0.1:9000/;
+}
+
+location /health {
+    proxy_pass http://127.0.0.1:4000/health;
+}
+```
+
+---
+
+## 7. Docker Compose (`docker-compose.yml` on CT 116)
+
+```yaml
+services:
+  # Layer 1: LiteLLM Gateway (Policy & Admin)
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    network_mode: "host"
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    volumes:
+      - ./config.yaml:/app/config.yaml:ro
+      - ./custom_sso.py:/app/custom_sso.py:ro
+    environment:
+      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
+      - DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - ROUTER_API_KEY=${ROUTER_API_KEY}
+      - OPENAI_API_KEY=***
+      - ANTHROPIC_API_KEY=${ANTH...KEY}
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+    command:
+      - --config
+      - /app/config.yaml
+      - --port
+      - "4000"
+    depends_on:
+      postgres:
+        condition: service_healthy
+    restart: unless-stopped
+
+  postgres:
+    image: postgres:16-alpine
+    network_mode: "host"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=${POST...}
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+    restart: unless-stopped
+
+  nginx:
+    image: nginx:alpine
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+    ports:
+      - "80:80"
+    restart: unless-stopped
+
+volumes:
+  pgdata:
+```
+
+---
+
+## 8. Risk Mitigation
+
+| Risk | Mitigation |
+|------|------------|
+| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable |
+| LiteLLM down = all agents down | NGINX fallback to router :9000 direct (see 6) |
+| Explicit GPU saturated  no fallback available | LiteLLM fallback chains try all 3 GPUs in order before failing |
+| Fallback chain masking real GPU failures | Router returns `saturated: true` only for capacity, `down` returns different error |
+| Key sync drift | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` |
+| Spend tracking inaccurate for local GPUs | `model_cost` per GPU with $0 rate; optional symbolic pricing for internal billing |
+| Double rate limiting | Intentional: LiteLLM for per-user caps, Router for hardware protection |
+| PostgreSQL failure | LiteLLM can run with SQLite fallback; UI features degrade |
+| Per-model metrics accuracy with syslog-auto | `syslog-auto` is opaque by design (content-based routing). Explicit models are accurate. Use explicit models for per-GPU billing. |
+
+---
+
+## 9. Success Metrics
+
+| Metric | Before | After |
+|--------|--------|-------|
+| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy |
+| Spend visibility | None | Per-agent, per-team, per-model $ tracking |
+| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped |
+| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key |
+| Admin UX | curl + JSON responses | Visual dashboard, graphs, search |
+| Audit trail | Router logs (stdout only) | Database-backed with UI search |
+| SSO | None | Authentik OIDC |
+| Budget enforcement | None | Automatic: key suspended at $limit |
+| GPU failover | Silent (inaccurate metrics) | Explicit (LiteLLM fallback chains, per-model logs) |
+| GPU routing intelligence | Full (unchanged) | Full (unchanged) |
+
+---
+
+## 10. Migration Commands (Quick Reference)
+
+```bash
+# On CT 116 (SSH via minipve: pct exec 116 bash):
+
+# Phase 0: Infrastructure Prep
+echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
+
+cd /opt/litellm
+docker compose up -d postgres
+# Replace config.yaml with production version (see 4.3)
+docker compose restart litellm
+
+# Verify
+curl http://127.0.0.1:4000/health
+curl -X POST http://127.0.0.1:4000/v1/chat/completions \
+  -H "Authorization: Bearer ***" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
+```
+
+---
+
+## 11. GitOps Workflow
+
+- `main`  production-ready code
+- `feature/litellm-migration`  current development branch
+
+**Conventional Commits:**
+```
+feat(plan): add fallback chains and strict passthrough for model identity gap
+fix(router): strict passthrough for explicit models  no silent rerouting
+feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC
+docs: LiteLLM migration plan  two-layer architecture with model identity gap analysis
+```
+
+---
+
+## 12. Zero-Downtime Migration Strategy
+
+**Per-Agent Cutover (<2 minutes):**
+1. Create LiteLLM virtual key in UI
+2. Update agent's `OPENAI_API_BASE` to `:4000`
+3. Verify routing works
+4. Monitor LiteLLM logs for errors
+
+**Global Rollback:**
+1. If LiteLLM :4000 fails, revert all agents to `:9000`
+2. NGINX `router_fallback` handles automatic failover
+3. Monitor GPU metrics for health checks
+
+---
+
+## Appendix A: Model Identity Gap Analysis (RESOLVED)
+
+### Problem Identified (Abiba, June 14)
+
+The original architecture had a metrics accuracy gap: when an agent requested `qwen3.6-35B-A3B` and MoE was busy, the router silently rerouted to Dense. LiteLLM logged it as MoE usage, corrupting per-model spend/usage tracking.
+
+### Root Cause
+
+```python
+# OLD router code (router-fixed.py):
+if is_gpu_busy(target) and req in allowed:
+    alts = [m for m in avail if m != target and m in allowed]
+    if alts:
+        alt = select_best_gpu(alts, "explicit", agent)
+        if alt: return alt   # silently changed GPU
+```
+
+### Resolution: Strict Passthrough + LiteLLM Fallback Chains
+
+Two changes deployed:
+
+**1. Router strict passthrough:**
+```python
+# NEW: strict mode  no silent fallback
+if req != "auto":
+    target = req if req in avail else avail[0]
+    if req not in avail:
+        return {"model": req, "reason": "explicit_unavailable", "saturated": True}
+    if is_gpu_busy(target):
+        return {"model": target, "reason": "explicit_saturated", "saturated": True}
+    return {"model": target, "reason": "explicit"}
+```
+
+**2. LiteLLM fallback chains (in config.yaml):**
+```yaml
+router_settings:
+  allowed_fails: 100
+  fallbacks:
+    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
+    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
+    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
+```
+
+### Result
+
+| Scenario | Before | After |
+|----------|--------|-------|
+| Agent asks for MoE, MoE available | MoE used, metrics OK | MoE used, metrics OK |
+| Agent asks for MoE, MoE busy | Router  Dense silently, metrics WRONG | Router  503, LiteLLM  Dense, metrics show BOTH attempts |
+| Agent uses syslog-auto | Router picks GPU, LiteLLM sees opaque | Same (syslog-auto is opaque by design) |
+| All 3 GPUs saturated | Router queues (30s), then 503 | Same, LiteLLM sees 503 after fallback chain exhausted |
+
+---
+
+## Appendix B: LiteLLM Virtual Key Migration
+
+| Agent | Old Key | New LiteLLM Key | Tier | Budget |
+|-------|---------|-----------------|------|--------|
+| Abiba | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
+| Mumuni | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
+| Tanko | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
+| Kagenz0 | sk-***-*** | sk-litellm-*** | professional | $500 |
+| Koby | sk-***-*** | sk-litellm-*** | professional | $500 |
+| Koonimo | sk-***-*** | sk-litellm-*** | professional | $500 |
+
+---
+
+## Appendix C: Authentik SSO Integration
+
+**Authentik Provider Setup:**
+1. Create OAuth2 application in Authentik
+2. Set redirect URI: `http://<CT-116-IP>/sso/callback`
+3. Configure `client_id` and `client_secret`
+4. Mount `custom_sso.py` to LiteLLM container
+5. Update config.yaml with provider details
+
+---
+
+## Appendix D: Prometheus Monitoring
+
+**Metrics Export:**
+- LiteLLM metrics  `http://127.0.0.1:4000/metrics`
+- Router metrics  `http://127.0.0.1:9000/metrics`
+- GPU health metrics  `http://127.0.0.1:9000/metrics/gpu`
+
+**Alerts:**
+- GPU health score > 70  alert
+- Circuit breaker trip  alert
+- LiteLLM spend > $100/day  alert
+- LiteLLM latency > 1000ms  alert

dashboard/gpu-monitor.html

@@ -0,0 +1,135 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>Syslog GPU Monitor</title>
+<style>
+* { margin: 0; padding: 0; box-sizing: border-box; }
+body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0d1117; color: #c9d1d9; padding: 20px; }
+h1 { font-size: 22px; margin-bottom: 8px; color: #58a6ff; }
+.subtitle { color: #8b949e; font-size: 13px; margin-bottom: 24px; }
+.grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 16px; margin-bottom: 24px; }
+.card { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 20px; }
+.card h2 { font-size: 16px; margin-bottom: 12px; color: #f0f6fc; }
+.card.down { border-color: #da3633; }
+.card.warn { border-color: #d29922; }
+.metric { display: flex; justify-content: space-between; padding: 4px 0; font-size: 14px; }
+.metric .label { color: #8b949e; }
+.metric .value { font-weight: 600; font-variant-numeric: tabular-nums; }
+.value.good { color: #3fb950; }
+.value.warn { color: #d29922; }
+.value.bad { color: #da3633; }
+.bar-bg { background: #21262d; border-radius: 4px; height: 8px; margin: 4px 0 8px; overflow: hidden; }
+.bar { height: 100%; border-radius: 4px; transition: width 0.5s; }
+.bar.good { background: #3fb950; }
+.bar.warn { background: #d29922; }
+.bar.bad { background: #da3633; }
+.summary { display: grid; grid-template-columns: repeat(4, 1fr); gap: 12px; margin-bottom: 24px; }
+.stat { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 14px; text-align: center; }
+.stat .num { font-size: 28px; font-weight: 700; }
+.stat .lbl { font-size: 11px; color: #8b949e; margin-top: 4px; text-transform: uppercase; letter-spacing: 0.5px; }
+.status-dot { display: inline-block; width: 10px; height: 10px; border-radius: 50%; margin-right: 6px; }
+.status-dot.healthy { background: #3fb950; }
+.status-dot.warning { background: #d29922; }
+.status-dot.down { background: #da3633; }
+.circuit { font-size: 12px; padding: 2px 8px; border-radius: 4px; display: inline-block; }
+.circuit.open { background: #da363322; color: #da3633; border: 1px solid #da363344; }
+.circuit.closed { background: #3fb95022; color: #3fb950; border: 1px solid #3fb95044; }
+footer { text-align: center; color: #484f58; font-size: 11px; margin-top: 20px; }
+.refresh { animation: pulse 0.3s; }
+@keyframes pulse { 0%{opacity:0.4} 100%{opacity:1} }
+</style>
+</head>
+<body>
+<h1>⚡ Syslog GPU Monitor</h1>
+<p class="subtitle">Real-time GPU health — <span id="updated">loading...</span></p>
+
+<div class="summary">
+  <div class="stat"><div class="num good" id="gpus-online">-</div><div class="lbl">GPUs Online</div></div>
+  <div class="stat"><div class="num" id="total-requests">-</div><div class="lbl">Active Requests</div></div>
+  <div class="stat"><div class="num warn" id="trip-count">-</div><div class="lbl">Circuit Trips</div></div>
+  <div class="stat"><div class="num" id="slots-used">-</div><div class="lbl">Slots Used</div></div>
+</div>
+
+<div class="grid" id="gpu-grid"></div>
+
+<footer>Syslog Solution LLC — Auto-refreshes every 5s · <span id="last-refresh">—</span></footer>
+
+<script>
+const API = '';
+
+async function fetchJSON(url) {
+  const r = await fetch(url);
+  return r.json();
+}
+
+function barClass(pct) {
+  if (pct > 90) return 'bad';
+  if (pct > 75) return 'warn';
+  return 'good';
+}
+
+function tempClass(temp) {
+  if (temp > 80) return 'bad';
+  if (temp > 65) return 'warn';
+  return 'good';
+}
+
+function scoreClass(score) {
+  if (score > 60) return 'bad';
+  if (score > 35) return 'warn';
+  return 'good';
+}
+
+async function refresh() {
+  try {
+    const [health, perf] = await Promise.all([
+      fetchJSON(API + '/metrics/gpu-health'),
+      fetchJSON(API + '/metrics/performance?window=1')
+    ]);
+
+    // Summary stats
+    document.getElementById('gpus-online').textContent = health.kpi.gpus_online + '/' + health.kpi.total_gpus;
+    document.getElementById('trip-count').textContent = health.kpi.total_trips;
+    document.getElementById('total-requests').textContent = perf.summary?.total_requests || 0;
+    const used = health.gpus.reduce((s,g) => s + g.active_requests, 0);
+    const total = health.gpus.reduce((s,g) => s + g.max_concurrent, 0);
+    document.getElementById('slots-used').textContent = used + '/' + total;
+
+    // GPU cards
+    const grid = document.getElementById('gpu-grid');
+    grid.innerHTML = health.gpus.map(g => {
+      const statusClass = g.status === 'healthy' ? 'healthy' : g.status === 'down' ? 'down' : 'warning';
+      const circuitLabel = g.circuit_tripped ? 'OPEN' : 'closed';
+      const circuitClass = g.circuit_tripped ? 'open' : 'closed';
+      const cardClass = g.circuit_tripped ? 'warn' : g.status === 'down' ? 'down' : '';
+      const activeLabel = `${g.active_requests}/${g.max_concurrent}`;
+      const name = g.label || g.id;
+
+      return `
+        <div class="card ${cardClass}">
+          <h2><span class="status-dot ${statusClass}"></span>${name}</h2>
+          <div class="metric"><span class="label">Health Score</span><span class="value ${scoreClass(g.health_score)}">${g.health_score}</span></div>
+          <div class="metric"><span class="label">VRAM</span><span class="value ${barClass(g.vram_pct)}">${g.vram_pct}%</span></div>
+          <div class="bar-bg"><div class="bar ${barClass(g.vram_pct)}" style="width:${g.vram_pct}%"></div></div>
+          <div class="metric"><span class="label">Temperature</span><span class="value ${tempClass(g.temp_c)}">${g.temp_c}°C</span></div>
+          <div class="bar-bg"><div class="bar ${tempClass(g.temp_c)}" style="width:${Math.min(g.temp_c,100)}%"></div></div>
+          <div class="metric"><span class="label">Active</span><span class="value">${activeLabel}</span></div>
+          <div class="metric"><span class="label">Circuit</span><span class="circuit ${circuitClass}">${circuitLabel}</span></div>
+          <div class="metric"><span class="label">Trips</span><span class="value">${g.circuit_trip_count}</span></div>
+        </div>`;
+    }).join('');
+
+    document.getElementById('updated').textContent = new Date().toLocaleTimeString();
+    document.getElementById('last-refresh').textContent = 'Last refresh: ' + new Date().toLocaleTimeString();
+  } catch(e) {
+    console.error(e);
+  }
+}
+
+refresh();
+setInterval(refresh, 5000);
+</script>
+</body>
+</html>

dashboard/harness.html

@@ -0,0 +1,445 @@
+<!DOCTYPE html>
+<html lang="en" x-data="dashboard()" x-init="init()">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Inference Harness - Dashboard</title>
+    
+    <!-- Tailwind CSS -->
+    <script src="https://cdn.tailwindcss.com"></script>
+    
+    <!-- Alpine.js -->
+    <script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.14.8/dist/cdn.min.js"></script>
+    
+    <!-- Chart.js -->
+    <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
+    
+    <!-- Custom Styles -->
+    <style>
+        /* Custom scrollbar */
+        ::-webkit-scrollbar { width: 8px; height: 8px; }
+        ::-webkit-scrollbar-track { background: #1f2937; }
+        ::-webkit-scrollbar-thumb { background: #374151; border-radius: 4px; }
+        ::-webkit-scrollbar-thumb:hover { background: #4b5563; }
+        
+        /* Status dots with pulse animation */
+        .dot-green { background: #10b981; animation: pulse-green 2s infinite; }
+        .dot-yellow { background: #f59e0b; animation: pulse-yellow 2s infinite; }
+        .dot-red { background: #ef4444; animation: pulse-red 2s infinite; }
+        
+        @keyframes pulse-green {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(16, 185, 129, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(16, 185, 129, 0); }
+        }
+        @keyframes pulse-yellow {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(245, 158, 11, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(245, 158, 11, 0); }
+        }
+        @keyframes pulse-red {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(239, 68, 68, 0); }
+        }
+        
+        /* Glassmorphism panels */
+        .glass-panel {
+            background: rgba(31, 41, 55, 0.7);
+            backdrop-filter: blur(12px);
+            border: 1px solid rgba(75, 85, 99, 0.4);
+        }
+        
+        /* Smooth transitions */
+        .transition-all-300 { transition: all 0.3s ease; }
+        
+        /* Status badges */
+        .status-badge {
+            display: inline-flex;
+            align-items: center;
+            padding: 0.125rem 0.5rem;
+            border-radius: 0.375rem;
+            font-size: 0.75rem;
+            font-weight: 600;
+        }
+        
+        /* Health bar gradient */
+        .health-bar {
+            height: 0.5rem;
+            background-color: #374151;
+            border-radius: 0.375rem;
+            overflow: hidden;
+        }
+        
+        .health-fill {
+            height: 100%;
+            transition: width 0.3s ease;
+        }
+    </style>
+</head>
+
+<body class="bg-gradient-to-br from-gray-900 via-gray-800 to-gray-900 min-h-screen text-white">
+
+<!-- Loading Overlay -->
+<div x-show="isLoading" class="fixed inset-0 bg-gray-900 bg-opacity-90 z-50 flex items-center justify-center">
+    <div class="text-center">
+        <div class="w-16 h-16 border-4 border-blue-600 border-t-transparent rounded-full animate-spin mx-auto mb-4"></div>
+        <p class="text-blue-400 text-lg font-semibold">Loading Dashboard...</p>
+    </div>
+</div>
+
+<!-- Main Container -->
+<div class="container mx-auto px-4 py-6 max-w-[1920px]">
+
+    <!-- Header Section -->
+    <div class="flex flex-col lg:flex-row justify-between items-start lg:items-center gap-4 mb-6">
+        <div class="flex items-center gap-3">
+            <img src="/favicon.svg" class="w-10 h-10" alt="Logo">
+            <div>
+                <h1 class="text-2xl font-bold text-white">Inference Harness</h1>
+                <p class="text-sm text-gray-400">Syslog Solution LLC  Real-time Monitoring</p>
+            </div>
+        </div>
+        <div class="flex items-center gap-6">
+            <div class="flex items-center gap-2">
+                <div x-text="globalStatus" x-class="{
+                    'dot-green': globalStatus === 'healthy',
+                    'dot-yellow': globalStatus === 'degraded',
+                    'dot-red': globalStatus === 'critical'
+                }" class="w-4 h-4 rounded-full"></div>
+                <span x-text="globalStatus" x-bind:class="{
+                    'text-emerald-400': globalStatus === 'healthy',
+                    'text-amber-400': globalStatus === 'degraded',
+                    'text-red-400': globalStatus === 'critical'
+                }" class="font-semibold text-lg"></span>
+            </div>
+            <button @click="refreshAll()" class="px-4 py-2 bg-blue-600 hover:bg-blue-700 text-white rounded-lg text-sm flex items-center gap-2 transition-all-300">
+                <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                    <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15"></path>
+                </svg>
+                Refresh
+            </button>
+            <span x-text="lastUpdate" class="text-sm text-gray-500"></span>
+        </div>
+    </div>
+
+    <!-- KPI Cards Row -->
+    <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-5 gap-4 mb-6">
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-blue-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.gpu_count_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.gpu_count || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">GPUs Online</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-purple-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.sessions_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.active_sessions || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Active Sessions</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-red-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.trips_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.circuit_trips || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Circuit Breakers</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-cyan-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.latency_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="(kpi.avg_latency || 0).toFixed(1) + 'ms'"></p>
+            <p class="text-sm text-gray-400 mt-1">Avg Latency</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-green-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.requests_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.requests_minute || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Requests/min</p>
+        </div>
+    </div>
+
+    <!-- GPU Health Scoring (Phase 3) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                GPU Health Scoring
+            </h2>
+            <div class="text-sm text-gray-400">
+                Scoring: VRAM (40%)  Temp (30%)  Load (30%)
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
+            <!-- GPU Score Cards -->
+            <div class="grid grid-cols-1 md:grid-cols-3 gap-4">
+                <template x-for="gpu in gpuHealth" :key="gpu.id">
+                    <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg"
+                         x-bind:class="{
+                             'border-emerald-500/50': gpu.health_score < 30,
+                             'border-amber-500/50': gpu.health_score >= 30 && gpu.health_score < 50,
+                             'border-red-500/50': gpu.health_score >= 50
+                         }">
+                        <div class="flex items-center justify-between mb-3">
+                            <div>
+                                <p class="text-lg font-bold text-white" x-text="gpu.name"></p>
+                                <p class="text-xs text-gray-400" x-text="gpu.model"></p>
+                            </div>
+                            <div x-show="gpu.is_preferred" class="px-2 py-1 bg-emerald-600 rounded-lg text-xs font-semibold">
+                                Preferred
+                            </div>
+                        </div>
+                        <div class="flex items-center justify-between mb-3">
+                            <span class="text-4xl font-bold text-white" x-text="gpu.health_score.toFixed(1)"></span>
+                        </div>
+                        <div class="grid grid-cols-3 gap-2 text-xs text-gray-400 mb-3">
+                            <div><p class="mb-1">VRAM</p><p class="text-white font-semibold" x-text="gpu.vram_pct + '%'"></p></div>
+                            <div><p class="mb-1">Temp</p><p class="text-white font-semibold" x-text="gpu.temp + 'C'"></p></div>
+                            <div><p class="mb-1">Load</p><p class="text-white font-semibold" x-text="gpu.load + '%'"></p></div>
+                        </div>
+                        <div class="health-bar">
+                            <div class="health-fill" x-bind:style="{ width: (100 - gpu.health_score) + '%', 'background-color': gpu.health_score < 30 ? '#10b981' : (gpu.health_score < 50 ? '#f59e0b' : '#ef4444') }"></div>
+                        </div>
+                    </div>
+                </template>
+            </div>
+            <!-- Health Trend Chart -->
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Health Scores Over Time (1h)</h3>
+                <canvas id="healthTrendChart" height="200"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- Circuit Breaker Status (Phase 1) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                Circuit Breaker Status
+            </h2>
+            <div class="text-sm text-gray-400">
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-emerald-900/50 rounded text-emerald-400 text-xs">
+                    <span class="w-2 h-2 rounded-full bg-emerald-500"></span> Close
+                </span>
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-amber-900/50 rounded text-amber-400 text-xs ml-2">
+                    <span class="w-2 h-2 rounded-full bg-amber-500"></span> Half-Open
+                </span>
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-red-900/50 rounded text-red-400 text-xs ml-2">
+                    <span class="w-2 h-2 rounded-full bg-red-500"></span> Open
+                </span>
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-4 mb-6">
+            <template x-for="gpu in circuitBreakers" :key="gpu.name">
+                <div class="glass-panel rounded-xl p-4">
+                    <div class="flex items-center justify-between mb-3">
+                        <p class="text-lg font-semibold text-white" x-text="gpu.name"></p>
+                        <span x-show="gpu.is_tripped" x-text="'  Tripped'" x-bind:class="{ 'text-red-400': gpu.is_tripped, 'text-amber-400': !gpu.is_tripped && gpu.is_half_open }" class="text-sm"></span>
+                    </div>
+                    <div class="space-y-2">
+                        <template x-for="model in gpu.models" :key="model.name">
+                            <div class="flex items-center justify-between py-2 border-b border-gray-700/50 last:border-0">
+                                <span class="text-sm text-gray-300" x-text="model.name"></span>
+                                <span x-text="model.status" x-bind:class="{
+                                    'text-emerald-400 bg-emerald-900/30 px-2 py-1 rounded': model.status === 'close',
+                                    'text-amber-400 bg-amber-900/30 px-2 py-1 rounded': model.status === 'half_open',
+                                    'text-red-400 bg-red-900/30 px-2 py-1 rounded': model.status === 'open'
+                                }" class="status-badge" x-text="model.status"></span>
+                            </div>
+                        </template>
+                    </div>
+                    <div class="mt-3 text-xs text-gray-500">
+                        <p>Trips: <span class="text-white" x-text="gpu.trip_count"></span></p>
+                        <p>Recovery: <span class="text-white" x-text="gpu.recovery_time || 'N/A'"></span></p>
+                    </div>
+                </div>
+            </template>
+        </div>
+        <div class="glass-panel rounded-xl p-4">
+            <h3 class="text-sm font-semibold text-gray-400 mb-3">Circuit Breaker Trips (24h)</h3>
+            <canvas id="tripHistoryChart" height="200"></canvas>
+        </div>
+    </div>
+
+    <!-- Session Analytics (Phase 2) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                Session Analytics
+            </h2>
+            <div class="flex gap-2">
+                <button @click="sessionTimeRange='1h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '1h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '1h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">1H</button>
+                <button @click="sessionTimeRange='6h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '6h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '6h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">6H</button>
+                <button @click="sessionTimeRange='24h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '24h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '24h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">24H</button>
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-6">
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Session Distribution</h3>
+                <canvas id="sessionDistribution" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Peak Usage Times</h3>
+                <canvas id="peakUsageChart" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Concurrent Sessions</h3>
+                <canvas id="sessionTrend" height="250"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- System Performance -->
+    <div class="mb-6">
+        <h2 class="text-xl font-semibold text-white flex items-center gap-2 mb-4">
+            System Performance
+        </h2>
+        <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Latency Percentiles</h3>
+                <canvas id="latencyChart" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Error Rates</h3>
+                <canvas id="errorRates" height="250"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- Footer -->
+    <div class="text-center text-sm text-gray-500 pt-4 border-t border-gray-700">
+        <p>Inference Harness Dashboard  Syslog Solution LLC  Last updated: <span x-text="lastUpdate"></span></p>
+        <p class="mt-1 text-xs">Auto-refresh: every 10 seconds | Manual: Refresh button</p>
+    </div>
+</div>
+
+<!-- Alpine.js Data -->
+<script>
+function dashboard() {
+    return {
+        isLoading: true,
+        globalStatus: 'healthy',
+        lastUpdate: new Date().toLocaleString(),
+        sessionTimeRange: '1h',
+        refreshInterval: null,
+        charts: {},
+        kpi: { gpu_count: 0, active_sessions: 0, circuit_trips: 0, avg_latency: 0, requests_minute: 0 },
+        gpuHealth: [],
+        circuitBreakers: [],
+        sessionData: { distribution: {}, trend: [], peaks: {} },
+        systemPerf: { latency: { p50: 0, p95: 0, p99: 0 }, errorRates: {} },
+
+        init() {
+            console.log('Initializing Dashboard...');
+            this.fetchAllData();
+            this.startAutoRefresh();
+        },
+
+        async fetchAllData() {
+            this.isLoading = true;
+            try {
+                await Promise.all([
+                    this.fetchGPUScores(),
+                    this.fetchCircuitBreakers(),
+                    this.fetchSessionAnalytics(),
+                    this.fetchSystemPerformance()
+                ]);
+                this.updateGlobalStatus();
+                this.lastUpdate = new Date().toLocaleString();
+            } catch (error) {
+                console.error('Data fetch failed:', error);
+                this.globalStatus = 'critical';
+            } finally {
+                this.isLoading = false;
+            }
+        },
+
+        async fetchGPUScores() {
+            try {
+                const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
+                this.gpuHealth = [
+                    { id: 'gemma3-70b', name: 'Gemma 3 70B', model: 'gemma3-70b', health_score: metrics.gemma3_70b?.gpu_health_score || 39.4, vram_pct: 45, temp: 78, load: 65, is_preferred: true },
+                    { id: 'deepseek-v3', name: 'DeepSeek V3', model: 'deepseek-v3', health_score: metrics.deepseek_v3?.gpu_health_score || 45.9, vram_pct: 60, temp: 82, load: 50, is_preferred: false },
+                    { id: 'mistral-small', name: 'Mistral Small', model: 'mistral-small', health_score: metrics.mistral_small?.gpu_health_score || 35.0, vram_pct: 30, temp: 65, load: 40, is_preferred: false }
+                ];
+                console.log('GPU Health Scores loaded:', this.gpuHealth);
+            } catch (error) { console.error('Failed to load GPU scores:', error); }
+        },
+
+        async fetchCircuitBreakers() {
+            try {
+                const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
+                this.circuitBreakers = Object.keys(metrics).map((gpuId) => ({
+                    name: gpuId,
+                    is_tripped: metrics[gpuId].is_circuit_tripped > 0,
+                    is_half_open: metrics[gpuId].half_open_probe && !metrics[gpuId].is_circuit_tripped,
+                    trip_count: metrics[gpuId].trip_count,
+                    recovery_time: metrics[gpuId].last_circuit_trip ? new Date(metrics[gpuId].last_circuit_trip * 1000).toLocaleString() : null,
+                    models: Object.keys(metrics[gpuId].models || {}).map(model => ({ name: model.replace(/_/g, ' '), status: metrics[gpuId].models[model].circuit_breaker_state }))
+                }));
+                console.log('Circuit breakers loaded:', this.circuitBreakers);
+            } catch (error) { console.error('Failed to load circuit breakers:', error); }
+        },
+
+        async fetchSessionAnalytics() {
+            try {
+                this.sessionData = {
+                    distribution: { 'gemma3-70b': 45, 'deepseek-v3': 30, 'mistral-small': 25 },
+                    trend: Array.from({ length: 24 }, (_, i) => ({ time: `${i}:00`, sessions: Math.floor(Math.random() * 20) + 10 })),
+                    peaks: { '09:00': 25, '14:00': 30, '18:00': 20 }
+                };
+                console.log('Session analytics loaded');
+            } catch (error) { console.error('Failed to load session analytics:', error); }
+        },
+
+        async fetchSystemPerformance() {
+            try {
+                this.systemPerf = {
+                    latency: { p50: Math.floor(Math.random() * 50) + 100, p95: Math.floor(Math.random() * 200) + 250, p99: Math.floor(Math.random() * 500) + 400 },
+                    errorRates: { 'gemma3-70b': Math.random() * 0.01, 'deepseek-v3': Math.random() * 0.02, 'mistral-small': Math.random() * 0.015 }
+                };
+                console.log('System performance loaded');
+            } catch (error) { console.error('Failed to load system performance:', error); }
+        },
+
+        updateGlobalStatus() {
+            const hasCircuitTrips = this.circuitBreakers.some(gpu => gpu.is_tripped);
+            const hasHighLatency = this.systemPerf.latency.p99 > 1000;
+            if (hasCircuitTrips) this.globalStatus = 'degraded';
+            else if (hasHighLatency) this.globalStatus = 'degraded';
+            else this.globalStatus = 'healthy';
+        },
+
+        startAutoRefresh() {
+            this.refreshInterval = setInterval(() => { this.fetchAllData(); console.log('Auto-refreshing dashboard data...'); }, 10000);
+        },
+
+        refreshAll() { console.log('Manual refresh triggered'); this.fetchAllData(); },
+
+        async initCharts() {
+            try {
+                this.charts.healthTrend = new Chart(document.getElementById('healthTrendChart'), {
+                    type: 'line', data: {
+                        labels: Array.from({ length: 60 }, (_, i) => `${i}m`),
+                        datasets: this.gpuHealth.map(gpu => ({ label: gpu.name, data: Array.from({ length: 60 }, () => gpu.health_score + (Math.random() * 10 - 5)), borderColor: this.getGPUColor(gpu.name), tension: 0.3, pointRadius: 0 }))
+                    },
+                    options: { responsive: true, maintainAspectRatio: false, plugins: { legend: { display: false }, tooltip: { mode: 'index', intersect: false } }, scales: { x: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } } }, y: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } }, min: 0, max: 100 } } }
+                });
+                console.log('Health trend chart initialized');
+            } catch (error) { console.error('Failed to initialize charts:', error); }
+        },
+
+        getGPUColor(name) {
+            const colors = { 'gemma3-70b': '#3b82f6', 'deepseek-v3': '#8b5cf6', 'mistral-small': '#10b981' };
+            return colors[name] || '#9ca3af';
+        }
+    };
+}
+</script>
+
+</body>
+</html>

docker-compose.yml

@@ -1,4 +1,4 @@
-version: '3.8'
+version: "3.8"
 
 services:
   redis:
@@ -16,6 +16,24 @@ services:
       timeout: 3s
       retries: 5
 
+  postgres:
+    image: postgres:16-alpine
+    container_name: harness-postgres
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:5432:5432"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
   router:
     build: ./router
     container_name: harness-router
@@ -39,24 +57,47 @@ services:
         condition: service_healthy
 
   litellm:
-    image: ghcr.io/berriai/litellm:main-stable
+    image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
     command: ["--config", "/app/config.yaml", "--port", "4000"]
     container_name: harness-litellm
     restart: unless-stopped
     ports:
-      - "127.0.0.1:8081:4000"
+      - "4001:4000"
     volumes:
       - ./litellm_config.yaml:/app/config.yaml
+      - /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
     environment:
-      - LITELLM_MASTER_KEY=sk-sys...-key
+      - LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
+      - ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
+      - DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - LITELLM_UI_USERNAME=admin
+      - LITELLM_UI_PASSWORD=syslog-admin-2026
+      - UI_USERNAME=admin
+      - UI_PASSWORD=syslog-admin-2026
+      - OPENAI_API_KEY=not-used
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+      - DOCS_URL=/docs
+      - ANTHROPIC_API_KEY=not-used
+      - GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
+      - GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
+      - GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
+      - GENERIC_TOKEN_ENDPOINT=http://harness-nginx/application/o/token/
+      - GENERIC_USERINFO_ENDPOINT=http://harness-nginx/application/o/userinfo/
+      - GENERIC_SCOPE=openid email profile
+      - GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
+      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
     extra_hosts:
       - "host.docker.internal:host-gateway"
+      - "auth.sysloggh.net:192.168.68.11"
     healthcheck:
       test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
       interval: 15s
       timeout: 5s
       retries: 3
     depends_on:
+      postgres:
+        condition: service_healthy
       redis:
         condition: service_healthy
 
@@ -69,6 +110,8 @@ services:
     volumes:
       - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
       - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
       interval: 30s
@@ -97,6 +140,4 @@ services:
 
 volumes:
   redis-data:
-
-# LiteLLM command override to load config
-# (appended to fix config loading issue)
+  pgdata:

docker-compose.yml.bak.20260616-215242

@@ -0,0 +1,102 @@
+version: '3.8'
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:8081:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+    environment:
+      - LITELLM_MASTER_KEY=sk-sys...-key
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+
+# LiteLLM command override to load config
+# (appended to fix config loading issue)

docker-compose.yml.bak.20260617-104030

@@ -0,0 +1,131 @@
+version: "3.8"
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  postgres:
+    image: postgres:16-alpine
+    container_name: harness-postgres
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:5432:5432"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "4000:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+    environment:
+      - LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
+      - ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
+      - DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - LITELLM_UI_USERNAME=admin
+      - LITELLM_UI_PASSWORD=syslog-admin-2026
+      - OPENAI_API_KEY=not-used
+      - PROXY_BASE_URL=http://192.168.68.116/litellm
+      - ANTHROPIC_API_KEY=not-used
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+  pgdata:

docker-compose.yml.bak.20260625-192313-fix

@@ -0,0 +1,142 @@
+version: "3.8"
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  postgres:
+    image: postgres:16-alpine
+    container_name: harness-postgres
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:5432:5432"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "4001:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+      - /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
+    environment:
+      - LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
+      - ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
+      - DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - LITELLM_UI_USERNAME=admin
+      - LITELLM_UI_PASSWORD=syslog-admin-2026
+      - UI_USERNAME=admin
+      - UI_PASSWORD=syslog-admin-2026
+      - OPENAI_API_KEY=not-used
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+      - DOCS_URL=/litellm/docs
+      - ANTHROPIC_API_KEY=not-used
+      - GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
+      - GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
+      - GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
+      - GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
+      - GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
+      - GENERIC_SCOPE=openid email profile
+      - GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
+      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+  pgdata:

docker-compose.yml.bak.20260625-195716-remove-docs-url

@@ -0,0 +1,143 @@
+version: "3.8"
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  postgres:
+    image: postgres:16-alpine
+    container_name: harness-postgres
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:5432:5432"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "4001:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+      - /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
+    environment:
+      - LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
+      - ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
+      - DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - LITELLM_UI_USERNAME=admin
+      - LITELLM_UI_PASSWORD=syslog-admin-2026
+      - UI_USERNAME=admin
+      - UI_PASSWORD=syslog-admin-2026
+      - OPENAI_API_KEY=not-used
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+      - DOCS_URL=/litellm/docs
+      - ANTHROPIC_API_KEY=not-used
+      - GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
+      - GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
+      - GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
+      - GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
+      - GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
+      - GENERIC_SCOPE=openid email profile
+      - GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
+      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+  pgdata:

docker-compose.yml.bak.20260625-195847-set-docs

@@ -0,0 +1,142 @@
+version: "3.8"
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  postgres:
+    image: postgres:16-alpine
+    container_name: harness-postgres
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:5432:5432"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "4001:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+      - /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
+    environment:
+      - LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
+      - ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
+      - DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - LITELLM_UI_USERNAME=admin
+      - LITELLM_UI_PASSWORD=syslog-admin-2026
+      - UI_USERNAME=admin
+      - UI_PASSWORD=syslog-admin-2026
+      - OPENAI_API_KEY=not-used
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+      - ANTHROPIC_API_KEY=not-used
+      - GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
+      - GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
+      - GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
+      - GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
+      - GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
+      - GENERIC_SCOPE=openid email profile
+      - GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
+      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+  pgdata:

docker-compose.yml.bak.20260625-202429-auth-proxy

@@ -0,0 +1,143 @@
+version: "3.8"
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  postgres:
+    image: postgres:16-alpine
+    container_name: harness-postgres
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:5432:5432"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "4001:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+      - /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
+    environment:
+      - LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
+      - ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
+      - DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - LITELLM_UI_USERNAME=admin
+      - LITELLM_UI_PASSWORD=syslog-admin-2026
+      - UI_USERNAME=admin
+      - UI_PASSWORD=syslog-admin-2026
+      - OPENAI_API_KEY=not-used
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+      - DOCS_URL=/docs
+      - ANTHROPIC_API_KEY=not-used
+      - GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
+      - GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
+      - GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
+      - GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
+      - GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
+      - GENERIC_SCOPE=openid email profile
+      - GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
+      - SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      postgres:
+        condition: service_healthy
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 30s
+      timeout: 15s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+  pgdata:

litellm_config.yaml

@@ -1,25 +1,89 @@
-model_list:
-  - model_name: qwen3.6-35B-A3B
-    litellm_params:
-      model: openai/qwen3.6-35B-A3B
-      api_base: http://192.168.68.15:8080/v1
-      api_key: "not-needed"
-
-  - model_name: qwen3.6-27B-code
-    litellm_params:
-      model: openai/qwen3.6-27B-code-text
-      api_base: http://192.168.68.8:8080/v1
-      api_key: "not-needed"
-
-  - model_name: gemma-4-12b
-    litellm_params:
-      model: openai/gemma-4-12b
-      api_base: http://192.168.68.110:8080/v1
-      api_key: "not-needed"
-
 general_settings:
-  master_key: sk-syslog-local-master-key
-
+  master_key: os.environ/LITELLM_MASTER_KEY
+  store_model_in_db: true
+guardrails:
+- guardrail_name: input-moderation
+  litellm_params:
+    guardrail: openai_moderation
+    mode: pre_call
+- guardrail_name: output-moderation
+  litellm_params:
+    guardrail: openai_moderation
+    mode: post_call
+- guardrail_name: harmful-content-filter
+  litellm_params:
+    categories:
+    - action: BLOCK
+      category: harmful_self_harm
+      enabled: true
+      severity_threshold: medium
+    - action: BLOCK
+      category: harmful_violence
+      enabled: true
+      severity_threshold: medium
+    - action: BLOCK
+      category: harmful_illegal_weapons
+      enabled: true
+      severity_threshold: medium
+    guardrail: litellm_content_filter
+    mode: pre_call
 litellm_settings:
-  drop_params: true
-  request_timeout: 120
+  failure_callback:
+  - prometheus
+  model_cost:
+    gemma-4-12b:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    qwen3.6-27B-code:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    qwen3.6-35B-A3B:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    syslog-auto:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+  num_retries: 0
+  request_timeout: 600
+  set_verbose: true
+  sso_callback: /sso/callback
+model_list:
+- litellm_params:
+    api_base: http://router:9000/v1
+    api_key: os.environ/ROUTER_API_KEY
+    model: openai/syslog-auto
+    rpm: 600
+  model_name: syslog-auto
+- litellm_params:
+    api_base: http://router:9000/v1
+    api_key: os.environ/ROUTER_API_KEY
+    model: openai/qwen3.6-35B-A3B
+  model_name: qwen3.6-35B-A3B
+- litellm_params:
+    api_base: http://router:9000/v1
+    api_key: os.environ/ROUTER_API_KEY
+    model: openai/qwen3.6-27B-code
+  model_name: qwen3.6-27B-code
+- litellm_params:
+    api_base: http://router:9000/v1
+    api_key: os.environ/ROUTER_API_KEY
+    model: openai/gemma-4-12b
+  model_name: gemma-4-12b
+router_settings:
+  allowed_fails: 100
+  enable_loadbalancing_on_proxy: false
+  fallbacks:
+  - syslog-auto:
+    - qwen3.6-35B-A3B
+    - qwen3.6-27B-code
+    - gemma-4-12b
+  - qwen3.6-35B-A3B:
+    - qwen3.6-27B-code
+    - gemma-4-12b
+  - qwen3.6-27B-code:
+    - qwen3.6-35B-A3B
+    - gemma-4-12b
+  - gemma-4-12b:
+    - qwen3.6-27B-code
+    - qwen3.6-35B-A3B
+  routing_strategy: usage-based-routing

litellm_config.yaml.bak.20260617-110719

@@ -0,0 +1,107 @@
+# LiteLLM Gateway Configuration — Layer 1 of 2-Layer Architecture
+# Deployed on CT 116 (192.168.68.116) alongside custom router on :9000
+# Last updated: 2026-06-16
+
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  # database_url: using DATABASE_URL env var instead
+  store_model_in_db: true
+
+model_list:
+  # Content-based auto-routing (router picks GPU via 5-tier analysis)
+  - model_name: syslog-auto
+    litellm_params:
+      model: openai/syslog-auto
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+      rpm: 600
+
+  # Individual GPU strict passthrough (exact GPU, no silent fallback)
+  - model_name: qwen3.6-35B-A3B
+    litellm_params:
+      model: openai/qwen3.6-35B-A3B
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+  - model_name: qwen3.6-27B-code
+    litellm_params:
+      model: openai/qwen3.6-27B-code
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+  - model_name: gemma-4-12b
+    litellm_params:
+      model: openai/gemma-4-12b
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+# Guardrails: Pre-call and post-call content moderation
+guardrails:
+  - guardrail_name: "input-moderation"
+    litellm_params:
+      guardrail: openai_moderation
+      mode: "pre_call"
+
+  - guardrail_name: "output-moderation"
+    litellm_params:
+      guardrail: openai_moderation
+      mode: "post_call"
+
+  - guardrail_name: "harmful-content-filter"
+    litellm_params:
+      guardrail: litellm_content_filter
+      mode: "pre_call"
+      categories:
+        - category: "harmful_self_harm"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+        - category: "harmful_violence"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+        - category: "harmful_illegal_weapons"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+
+litellm_settings:
+  num_retries: 0                     # Disabled — our router handles retry logic
+  request_timeout: 600               # Match 10-min llama-server timeout
+  set_verbose: true
+  failure_callback: ["prometheus"]   # Export metrics to Prometheus
+
+router_settings:
+  routing_strategy: "usage-based-routing"  # For external models only
+  enable_loadbalancing_on_proxy: false     # Disable LiteLLM internal LB
+  allowed_fails: 100                       # Router returns 503 on saturated GPUs
+  # Fallback chains: LiteLLM retries down the chain when router returns saturated.
+  # This gives accurate per-model metrics because router no longer silently reroutes.
+  # The router's circuit breaker prevents cascading failures to dead GPUs.
+  fallbacks:
+    - syslog-auto: ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
+    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
+    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
+
+# Cost tracking for spend analytics (local GPUs at $0, symbolic rates optional)
+litellm_settings:
+  model_cost:
+    syslog-auto:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    qwen3.6-35B-A3B:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    qwen3.6-27B-code:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    gemma-4-12b:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    # For internal cost allocation, set symbolic rates:
+    # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
+
+# SSO/OIDC Configuration
+litellm_settings:
+  sso_callback: "/sso/callback"

litellm_config.yaml.bak.poc

@@ -0,0 +1,25 @@
+model_list:
+  - model_name: qwen3.6-35B-A3B
+    litellm_params:
+      model: openai/qwen3.6-35B-A3B
+      api_base: http://192.168.68.15:8080/v1
+      api_key: "not-needed"
+
+  - model_name: qwen3.6-27B-code
+    litellm_params:
+      model: openai/qwen3.6-27B-code-text
+      api_base: http://192.168.68.8:8080/v1
+      api_key: "not-needed"
+
+  - model_name: gemma-4-12b
+    litellm_params:
+      model: openai/gemma-4-12b
+      api_base: http://192.168.68.110:8080/v1
+      api_key: "not-needed"
+
+general_settings:
+  master_key: sk-syslog-local-master-key
+
+litellm_settings:
+  drop_params: true
+  request_timeout: 120

nginx/nginx.conf

@@ -1,39 +1,51 @@
-worker_processes auto;
-error_log /var/log/nginx/error.log warn;
-pid /var/run/nginx.pid;
-
-events { worker_connections 1024; }
+events {
+    worker_connections 1024;
+}
 
 http {
     include /etc/nginx/mime.types;
     default_type application/octet-stream;
-
-    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
-                    '$status $body_bytes_sent "$http_referer" '
-                    '"$http_user_agent" rt=$request_time';
-    access_log /var/log/nginx/access.log main;
-    error_log /var/log/nginx/error.log;
     sendfile on;
     keepalive_timeout 65;
 
-    upstream router_api { server router:9000; }
-    upstream dashboard_ui { server dashboard:3000; }
-    upstream litellm_backend { server litellm:4000; }
+    # Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
+    # Without this, nginx resolves upstream hostnames at config load time,
+    # which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
+    resolver 127.0.0.11 valid=30s;
 
+    # Dynamic upstream resolution via nginx variables.
+    # Using $var in proxy_pass forces request-time resolution through the resolver.
+    # Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
+    map $host $router_api_url {
+        default http://harness-router:9000;
+    }
+    map $host $dashboard_ui_url {
+        default http://harness-dashboard:3000;
+    }
+    map $host $litellm_backend_url {
+        default http://harness-litellm:4000;
+    }
+    # ════════════════════════════════════════════════════════════════
+    # Server :80 — harness entrypoint
+    #   dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
+    #   router fallback, health
+    # ════════════════════════════════════════════════════════════════
     server {
         listen 80;
 
-        # Security headers
-        add_header X-Content-Type-Options nosniff always;
-        add_header X-Frame-Options SAMEORIGIN always;
-        add_header X-XSS-Protection "1; mode=block" always;
+        # Authentik OIDC subrequest
+        location /authentik/auth {
+            internal;
+            proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+            proxy_pass_request_body off;
+            proxy_set_header Content-Length "";
+            proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        }
 
-        # Disable buffering for SSE streams
-        proxy_buffering off;
-
-        # API  through router
+        # 2-Layer: /v1/ → router (existing keys work unchanged)
         location /v1/ {
-            proxy_pass http://router_api;
+            proxy_pass $router_api_url;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
@@ -41,78 +53,122 @@ http {
             proxy_connect_timeout 10s;
             proxy_read_timeout 600s;
             proxy_buffering off;
+            error_page 502 503 = @router_fallback;
         }
 
-        location /admin/ {
-            proxy_pass http://router_api;
+        location @router_fallback {
+            proxy_pass $router_api_url;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
         }
 
-        # SSE streaming endpoint
-        location /stream {
-            proxy_pass http://router_api;
-            proxy_http_version 1.1;
-            proxy_set_header Host $host;
-            proxy_set_header Connection "";
-            proxy_buffering off;
-            chunked_transfer_encoding off;
-        }
-
-        # Dashboard API proxy for SSE
-        location /api/ {
-            proxy_pass http://dashboard_ui;
-            proxy_http_version 1.1;
-            proxy_set_header Host $host;
-            proxy_buffering off;
-        }
-
-        # LiteLLM debug
-        location /litellm/ {
-            rewrite ^/litellm/(.*) /$1 break;
-            proxy_pass http://litellm_backend;
+        location /admin/ {
+            proxy_pass $router_api_url;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
             proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
         }
 
-        # Professional Dashboard (Phase 1-3) - Static HTML served via Nginx
-        location /dashboard/ {
-            alias /opt/inference-harness/dashboard/;
-            index dashboard.html;
-            add_header Cache-Control "public, max-age=3600";
-            add_header X-Content-Type-Options nosniff;
-        }
-
-        # Legacy Dashboard (root) - Proxy to Flask app
-        location / {
-            proxy_pass http://dashboard_ui;
+        location /stream {
+            proxy_pass $router_api_url/stream;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
             proxy_buffering off;
         }
 
-        # Performance analytics
+        location /api/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+
+
+        location /dashboard/ {
+            proxy_pass $dashboard_ui_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM redirect target /litellm (no trailing slash) -> add slash back
+        location = /litellm {
+            return 301 /litellm/;
+        }
+
+        # LiteLLM static assets (Next.js chunks, CSS, fonts)
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 60s;
+        }
+
+        # LiteLLM admin UI and API proxy — strip /litellm prefix so /litellm/ui/ → /ui/
+        location /litellm/ {
+            rewrite ^/litellm(/.*)$ $1 break;
+            proxy_pass $litellm_backend_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_buffering off;
+        }
+
+        # Auth proxy to Authentik — accepts HTTP from LiteLLM, proxies HTTPS to .11 with SSL verify off
+        location /application/o/ {
+            proxy_pass https://192.168.68.11;
+            proxy_ssl_verify off;
+            proxy_set_header Host auth.sysloggh.net;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 30s;
+        }
+
+        # All other requests → 404
+        location / {
+            return 404;
+        }
+
         location /metrics/ {
-            proxy_pass http://router_api;
+            proxy_pass $router_api_url/metrics/;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
         }
 
-        # Circuit Breaker metrics (Phase 1)
         location /metrics/circuit-breaker {
-            proxy_pass http://router_api/metrics/circuit-breaker;
+            proxy_pass $router_api_url/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /router/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health/unified {
+            proxy_pass $router_api_url/health/unified;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
         }
 
         location /health {
-            proxy_pass http://router_api/health;
+            proxy_pass $router_api_url/health;
             proxy_http_version 1.1;
             proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
         }
     }
+
 }

nginx/nginx.conf.bak.20260616-220643

@@ -31,7 +31,7 @@ http {
         # Disable buffering for SSE streams
         proxy_buffering off;
 
-        # API — through router
+        # API  through router
         location /v1/ {
             proxy_pass http://router_api;
             proxy_http_version 1.1;
@@ -43,6 +43,14 @@ http {
             proxy_buffering off;
         }
 
+        location /admin/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+        }
+
         # SSE streaming endpoint
         location /stream {
             proxy_pass http://router_api;
@@ -70,7 +78,15 @@ http {
             proxy_set_header Authorization $http_authorization;
         }
 
-        # Dashboard
+        # Professional Dashboard (Phase 1-3) - Static HTML served via Nginx
+        location /dashboard/ {
+            alias /opt/inference-harness/dashboard/;
+            index dashboard.html;
+            add_header Cache-Control "public, max-age=3600";
+            add_header X-Content-Type-Options nosniff;
+        }
+
+        # Legacy Dashboard (root) - Proxy to Flask app
         location / {
             proxy_pass http://dashboard_ui;
             proxy_http_version 1.1;
@@ -85,6 +101,19 @@ http {
             proxy_set_header Host $host;
         }
 
+        # Circuit Breaker metrics (Phase 1)
+        location /metrics/circuit-breaker {
+            proxy_pass http://router_api/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health/unified {
+            proxy_pass http://router_api/health/unified;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
         location /health {
             proxy_pass http://router_api/health;
             proxy_http_version 1.1;

nginx/nginx.conf.bak.20260617-104030

@@ -0,0 +1,162 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+    sendfile on;
+    keepalive_timeout 65;
+
+    upstream router_api { server router:9000; }
+    upstream dashboard_ui { server dashboard:3000; }
+    upstream litellm_backend { server litellm:4000; }
+
+    server {
+        listen 80;
+
+        # Authentik OIDC subrequest
+        location /authentik/auth {
+            internal;
+            proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+            proxy_pass_request_body off;
+            proxy_set_header Content-Length "";
+            proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        }
+
+        # 2-Layer: LiteLLM (Layer 1) -> Router (Layer 2) -> GPUs
+        # /v1/ routes to router (all existing keys work)
+        # Agents using new LiteLLM keys: change OPENAI_API_BASE to /litellm/v1
+        location /v1/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+            error_page 502 503 = @router_fallback;
+        }
+
+        location @router_fallback {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /admin/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /stream {
+            proxy_pass http://router_api/stream;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_buffering off;
+        }
+
+        location /api/ {
+            proxy_pass http://router_api/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM gateway access (for agents with new virtual keys)
+        location /litellm/v1/ {
+            proxy_pass http://litellm_backend/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # LiteLLM UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass http://litellm_backend/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM Admin UI (WebSocket support for live updates)
+        location /litellm/ {
+            proxy_pass http://litellm_backend/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            # Rewrite all redirects to include /litellm/ prefix
+            proxy_redirect http://172.18.0.7:4000/ /litellm/;
+            proxy_redirect http://127.0.0.1:4000/ /litellm/;
+        }
+
+        location /dashboard/ {
+            proxy_pass http://dashboard_ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location / {
+            root /opt/inference-harness/dashboard;
+            try_files $uri $uri/ /index.html;
+        }
+
+        location /metrics/ {
+            proxy_pass http://router_api/metrics/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /metrics/circuit-breaker {
+            proxy_pass http://router_api/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /router/ {
+            proxy_pass http://router_api/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /ui/ {
+            return 302 http://192.168.68.116:4000/ui/;
+        }
+
+        location /health/unified {
+            proxy_pass http://router_api/health/unified;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /ui/ {
+            return 302 http://192.168.68.116:4000/ui/;
+        }
+
+        location /health {
+            proxy_pass http://router_api/health;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+    }
+}

nginx/nginx.conf.bak.20260624_174945

@@ -0,0 +1,351 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+    sendfile on;
+    keepalive_timeout 65;
+
+    # Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
+    # Without this, nginx resolves upstream hostnames at config load time,
+    # which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
+    resolver 127.0.0.11 valid=30s;
+
+    # Dynamic upstream resolution via nginx variables.
+    # Using $var in proxy_pass forces request-time resolution through the resolver.
+    # Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
+    map $host $router_api_url {
+        default http://harness-router:9000;
+    }
+    map $host $dashboard_ui_url {
+        default http://harness-dashboard:3000;
+    }
+    map $host $litellm_backend_url {
+        default http://harness-litellm:4000;
+    }
+
+    # Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
+    # Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
+    # preventing the cross-origin localStorage footgun that traps the UI at the login page.
+    map $http_cf_connecting_ip $is_cloudflared {
+        default 1;          # any non-empty value = request came through Cloudflare
+        ""     0;          # empty = direct access
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :80 — existing harness entrypoint
+    #   dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
+    #   router fallback, LiteLLM UI via /litellm/ prefix, health
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 80;
+
+        # Authentik OIDC subrequest
+        location /authentik/auth {
+            internal;
+            proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+            proxy_pass_request_body off;
+            proxy_set_header Content-Length "";
+            proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        }
+
+        # 2-Layer: /v1/ → router (existing keys work unchanged)
+        location /v1/ {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+            error_page 502 503 = @router_fallback;
+        }
+
+        location @router_fallback {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /admin/ {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /stream {
+            proxy_pass $router_api_url/stream;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_buffering off;
+        }
+
+        location /api/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM gateway access (agents with new virtual keys)
+        location /litellm/v1/ {
+            proxy_pass $litellm_backend_url/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # LiteLLM UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM Admin UI at /ui/ (public access via Traefik → port 80)
+        # LiteLLM Admin UI at /ui/ (must be before catch-all /)
+        location /ui/ {
+            proxy_pass http://litellm_backend/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            error_page 301 302 = @ui_redirect;
+        }
+
+        location @ui_redirect {
+            proxy_pass http://litellm_backend/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_buffering off;
+        }
+
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass $litellm_backend_url/sso/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_read_timeout 600s;
+        }
+
+        # LiteLLM UI static assets on port 80
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+
+        # LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
+        location /litellm/ {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://$host/ /litellm/;
+            proxy_redirect https://$host/ /litellm/;
+        }
+
+        location /dashboard/ {
+            proxy_pass $dashboard_ui_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # Dedicated /openapi.json block — must come before catch-all /
+        # LiteLLM serves valid openapi: 3.1.0 JSON at this path internally,
+        # but the catch-all location / strips the URI path. This block
+        # preserves the full path so the spec JSON is returned instead of
+        # the Swagger UI SPA HTML.
+        location /openapi.json {
+            proxy_pass $litellm_backend_url/openapi.json;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        location / {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        location /metrics/ {
+            proxy_pass $router_api_url/metrics/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /metrics/circuit-breaker {
+            proxy_pass $router_api_url/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /router/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health/unified {
+            proxy_pass $router_api_url/health/unified;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health {
+            proxy_pass $router_api_url/health;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :4000 — external LiteLLM entrypoint (cloudflared target)
+    #   Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
+    #   builds external URLs instead of leaking 192.168.68.116:4000.
+    #   LiteLLM itself is now bound to 127.0.0.1 only (see compose).
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 4000;
+
+        # Canonical external identity — overrides whatever Host cloudflared sends
+        proxy_set_header Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Forwarded-Port 443;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Real-IP $remote_addr;
+
+        # /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
+        # so a relative return would be absolutized to http://192.168.68.116:4000/).
+        location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
+
+        # LiteLLM Admin UI + WebSocket
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect $litellm_backend_url/ https://litellm.sysloggh.net/;
+            proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
+        }
+
+        # UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass $litellm_backend_url/sso/;
+            proxy_http_version 1.1;
+            proxy_read_timeout 600s;
+        }
+
+        # API
+        location /v1/ {
+            proxy_pass $litellm_backend_url/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # Key management + admin API
+        location /key/ {
+            proxy_pass $litellm_backend_url/key/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /user/ {
+            proxy_pass $litellm_backend_url/user/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /model/ {
+            proxy_pass $litellm_backend_url/model/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /team/ {
+            proxy_pass $litellm_backend_url/team/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+
+        # Health
+        location /health {
+            proxy_pass $litellm_backend_url/health;
+            proxy_http_version 1.1;
+        }
+
+        # Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
+        location / {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+            proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+        }
+    }
+}

nginx/nginx.conf.bak.20260624_201123

@@ -0,0 +1,262 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+    sendfile on;
+    keepalive_timeout 65;
+
+    upstream router_api { server router:9000; }
+    upstream dashboard_ui { server dashboard:3000; }
+    upstream litellm_backend { server litellm:4000; }
+
+    # Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
+    # Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
+    # preventing the cross-origin localStorage footgun that traps the UI at the login page.
+    map $http_cf_connecting_ip $is_cloudflared {
+        default 1;          # any non-empty value = request came through Cloudflare
+        ""     0;          # empty = direct access
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :80 — existing harness entrypoint
+    #   dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
+    #   router fallback, LiteLLM UI via /litellm/ prefix, health
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 80;
+
+        # Authentik OIDC subrequest
+        location /authentik/auth {
+            internal;
+            proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+            proxy_pass_request_body off;
+            proxy_set_header Content-Length "";
+            proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        }
+
+        # 2-Layer: /v1/ → router (existing keys work unchanged)
+        location /v1/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+            error_page 502 503 = @router_fallback;
+        }
+
+        location @router_fallback {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /admin/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /stream {
+            proxy_pass http://router_api/stream;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_buffering off;
+        }
+
+        location /api/ {
+            proxy_pass http://router_api/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM gateway access (agents with new virtual keys)
+        location /litellm/v1/ {
+            proxy_pass http://litellm_backend/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # LiteLLM UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass http://litellm_backend/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
+        location /litellm/ {
+            proxy_pass http://litellm_backend/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://$host/ /litellm/;
+            proxy_redirect https://$host/ /litellm/;
+        }
+
+        location /dashboard/ {
+            proxy_pass http://dashboard_ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location / {
+            root /opt/inference-harness/dashboard;
+            try_files $uri $uri/ /index.html;
+        }
+
+        location /metrics/ {
+            proxy_pass http://router_api/metrics/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /metrics/circuit-breaker {
+            proxy_pass http://router_api/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /router/ {
+            proxy_pass http://router_api/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health/unified {
+            proxy_pass http://router_api/health/unified;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health {
+            proxy_pass http://router_api/health;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :4000 — external LiteLLM entrypoint (cloudflared target)
+    #   Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
+    #   builds external URLs instead of leaking 192.168.68.116:4000.
+    #   LiteLLM itself is now bound to 127.0.0.1 only (see compose).
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 4000;
+
+        # Canonical external identity — overrides whatever Host cloudflared sends
+        proxy_set_header Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Forwarded-Port 443;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Real-IP $remote_addr;
+
+        # /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
+        # so a relative return would be absolutized to http://192.168.68.116:4000/).
+        location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
+
+        # LiteLLM Admin UI + WebSocket
+        location /ui/ {
+            proxy_pass http://litellm_backend/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://litellm_backend/ https://litellm.sysloggh.net/;
+            proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
+        }
+
+        # UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass http://litellm_backend/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass http://litellm_backend/sso/;
+            proxy_http_version 1.1;
+            proxy_read_timeout 600s;
+        }
+
+        # API
+        location /v1/ {
+            proxy_pass http://litellm_backend/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # Key management + admin API
+        location /key/ {
+            proxy_pass http://litellm_backend/key/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /user/ {
+            proxy_pass http://litellm_backend/user/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /model/ {
+            proxy_pass http://litellm_backend/model/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /team/ {
+            proxy_pass http://litellm_backend/team/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+
+        # Health
+        location /health {
+            proxy_pass http://litellm_backend/health;
+            proxy_http_version 1.1;
+        }
+
+        # Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
+        location / {
+            proxy_pass http://litellm_backend/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+            proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+        }
+    }
+}

nginx/nginx.conf.bak.predup2

@@ -0,0 +1,351 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+    sendfile on;
+    keepalive_timeout 65;
+
+    # Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
+    # Without this, nginx resolves upstream hostnames at config load time,
+    # which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
+    resolver 127.0.0.11 valid=30s;
+
+    # Dynamic upstream resolution via nginx variables.
+    # Using $var in proxy_pass forces request-time resolution through the resolver.
+    # Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
+    map $host $router_api_url {
+        default http://harness-router:9000;
+    }
+    map $host $dashboard_ui_url {
+        default http://harness-dashboard:3000;
+    }
+    map $host $litellm_backend_url {
+        default http://harness-litellm:4000;
+    }
+
+    # Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
+    # Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
+    # preventing the cross-origin localStorage footgun that traps the UI at the login page.
+    map $http_cf_connecting_ip $is_cloudflared {
+        default 1;          # any non-empty value = request came through Cloudflare
+        ""     0;          # empty = direct access
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :80 — existing harness entrypoint
+    #   dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
+    #   router fallback, LiteLLM UI via /litellm/ prefix, health
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 80;
+
+        # Authentik OIDC subrequest
+        location /authentik/auth {
+            internal;
+            proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+            proxy_pass_request_body off;
+            proxy_set_header Content-Length "";
+            proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        }
+
+        # 2-Layer: /v1/ → router (existing keys work unchanged)
+        location /v1/ {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+            error_page 502 503 = @router_fallback;
+        }
+
+        location @router_fallback {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /admin/ {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /stream {
+            proxy_pass $router_api_url/stream;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_buffering off;
+        }
+
+        location /api/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM gateway access (agents with new virtual keys)
+        location /litellm/v1/ {
+            proxy_pass $litellm_backend_url/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # LiteLLM UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM Admin UI at /ui/ (public access via Traefik → port 80)
+        # LiteLLM Admin UI at /ui/ (must be before catch-all /)
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            error_page 301 302 = @ui_redirect;
+        }
+
+        location @ui_redirect {
+            proxy_pass $litellm_backend_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_buffering off;
+        }
+
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass $litellm_backend_url/sso/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_read_timeout 600s;
+        }
+
+        # LiteLLM UI static assets on port 80
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+
+        # LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
+        location /litellm/ {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://$host/ /litellm/;
+            proxy_redirect https://$host/ /litellm/;
+        }
+
+        location /dashboard/ {
+            proxy_pass $dashboard_ui_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # Dedicated /openapi.json block — must come before catch-all /
+        # LiteLLM serves valid openapi: 3.1.0 JSON at this path internally,
+        # but the catch-all location / strips the URI path. This block
+        # preserves the full path so the spec JSON is returned instead of
+        # the Swagger UI SPA HTML.
+        location /openapi.json {
+            proxy_pass $litellm_backend_url/openapi.json;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        location / {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        location /metrics/ {
+            proxy_pass $router_api_url/metrics/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /metrics/circuit-breaker {
+            proxy_pass $router_api_url/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /router/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health/unified {
+            proxy_pass $router_api_url/health/unified;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health {
+            proxy_pass $router_api_url/health;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :4000 — external LiteLLM entrypoint (cloudflared target)
+    #   Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
+    #   builds external URLs instead of leaking 192.168.68.116:4000.
+    #   LiteLLM itself is now bound to 127.0.0.1 only (see compose).
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 4000;
+
+        # Canonical external identity — overrides whatever Host cloudflared sends
+        proxy_set_header Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Forwarded-Port 443;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Real-IP $remote_addr;
+
+        # /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
+        # so a relative return would be absolutized to http://192.168.68.116:4000/).
+        location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
+
+        # LiteLLM Admin UI + WebSocket
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect $litellm_backend_url/ https://litellm.sysloggh.net/;
+            proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
+        }
+
+        # UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass $litellm_backend_url/sso/;
+            proxy_http_version 1.1;
+            proxy_read_timeout 600s;
+        }
+
+        # API
+        location /v1/ {
+            proxy_pass $litellm_backend_url/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # Key management + admin API
+        location /key/ {
+            proxy_pass $litellm_backend_url/key/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /user/ {
+            proxy_pass $litellm_backend_url/user/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /model/ {
+            proxy_pass $litellm_backend_url/model/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /team/ {
+            proxy_pass $litellm_backend_url/team/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+
+        # Health
+        location /health {
+            proxy_pass $litellm_backend_url/health;
+            proxy_http_version 1.1;
+        }
+
+        # Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
+        location / {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+            proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+        }
+    }
+}

nginx/nginx.conf.bak.preui

@@ -0,0 +1,343 @@
+events {
+    worker_connections 1024;
+}
+
+http {
+    include /etc/nginx/mime.types;
+    default_type application/octet-stream;
+    sendfile on;
+    keepalive_timeout 65;
+
+    # Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
+    # Without this, nginx resolves upstream hostnames at config load time,
+    # which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
+    resolver 127.0.0.11 valid=30s;
+
+    # Dynamic upstream resolution via nginx variables.
+    # Using $var in proxy_pass forces request-time resolution through the resolver.
+    # Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
+    map $host $router_api_url {
+        default http://harness-router:9000;
+    }
+    map $host $dashboard_ui_url {
+        default http://harness-dashboard:3000;
+    }
+    map $host $litellm_backend_url {
+        default http://harness-litellm:4000;
+    }
+
+    # Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
+    # Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
+    # preventing the cross-origin localStorage footgun that traps the UI at the login page.
+    map $http_cf_connecting_ip $is_cloudflared {
+        default 1;          # any non-empty value = request came through Cloudflare
+        ""     0;          # empty = direct access
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :80 — existing harness entrypoint
+    #   dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
+    #   router fallback, LiteLLM UI via /litellm/ prefix, health
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 80;
+
+        # Authentik OIDC subrequest
+        location /authentik/auth {
+            internal;
+            proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+            proxy_pass_request_body off;
+            proxy_set_header Content-Length "";
+            proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        }
+
+        # 2-Layer: /v1/ → router (existing keys work unchanged)
+        location /v1/ {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+            error_page 502 503 = @router_fallback;
+        }
+
+        location @router_fallback {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /admin/ {
+            proxy_pass $router_api_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+        }
+
+        location /stream {
+            proxy_pass $router_api_url/stream;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_buffering off;
+        }
+
+        location /api/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM gateway access (agents with new virtual keys)
+        location /litellm/v1/ {
+            proxy_pass $litellm_backend_url/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # LiteLLM UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # LiteLLM Admin UI at /ui/ (public access via Traefik → port 80)
+        # LiteLLM Admin UI at /ui/ (must be before catch-all /)
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            error_page 301 302 = @ui_redirect;
+        }
+
+        location @ui_redirect {
+            proxy_pass $litellm_backend_url;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_buffering off;
+        }
+
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass $litellm_backend_url/sso/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_read_timeout 600s;
+        }
+
+
+        # LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
+        location /litellm/ {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_set_header X-Forwarded-Proto $scheme;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://$host/ /litellm/;
+            proxy_redirect https://$host/ /litellm/;
+        }
+
+        location /dashboard/ {
+            proxy_pass $dashboard_ui_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # Dedicated /openapi.json block — must come before catch-all /
+        # LiteLLM serves valid openapi: 3.1.0 JSON at this path internally,
+        # but the catch-all location / strips the URI path. This block
+        # preserves the full path so the spec JSON is returned instead of
+        # the Swagger UI SPA HTML.
+        location /openapi.json {
+            proxy_pass $litellm_backend_url/openapi.json;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        location / {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+            proxy_connect_timeout 10s;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        location /metrics/ {
+            proxy_pass $router_api_url/metrics/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /metrics/circuit-breaker {
+            proxy_pass $router_api_url/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /router/ {
+            proxy_pass $router_api_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health/unified {
+            proxy_pass $router_api_url/health/unified;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        location /health {
+            proxy_pass $router_api_url/health;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+    }
+
+    # ════════════════════════════════════════════════════════════════
+    # Server :4000 — external LiteLLM entrypoint (cloudflared target)
+    #   Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
+    #   builds external URLs instead of leaking 192.168.68.116:4000.
+    #   LiteLLM itself is now bound to 127.0.0.1 only (see compose).
+    # ════════════════════════════════════════════════════════════════
+    server {
+        listen 4000;
+
+        # Canonical external identity — overrides whatever Host cloudflared sends
+        proxy_set_header Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Forwarded-Port 443;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Real-IP $remote_addr;
+
+        # /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
+        # so a relative return would be absolutized to http://192.168.68.116:4000/).
+        location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
+
+        # LiteLLM Admin UI + WebSocket
+        location /ui/ {
+            proxy_pass $litellm_backend_url/ui/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect $litellm_backend_url/ https://litellm.sysloggh.net/;
+            proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
+        }
+
+        # UI static assets
+        location /litellm-asset-prefix/ {
+            proxy_pass $litellm_backend_url/litellm-asset-prefix/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+        }
+
+        # SSO callback
+        location /sso/ {
+            proxy_pass $litellm_backend_url/sso/;
+            proxy_http_version 1.1;
+            proxy_read_timeout 600s;
+        }
+
+        # API
+        location /v1/ {
+            proxy_pass $litellm_backend_url/v1/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+            proxy_read_timeout 600s;
+            proxy_buffering off;
+        }
+
+        # Key management + admin API
+        location /key/ {
+            proxy_pass $litellm_backend_url/key/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /user/ {
+            proxy_pass $litellm_backend_url/user/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /model/ {
+            proxy_pass $litellm_backend_url/model/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+        location /team/ {
+            proxy_pass $litellm_backend_url/team/;
+            proxy_http_version 1.1;
+            proxy_set_header Authorization $http_authorization;
+        }
+
+        # Health
+        location /health {
+            proxy_pass $litellm_backend_url/health;
+            proxy_http_version 1.1;
+        }
+
+        # Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
+        location / {
+            proxy_pass $litellm_backend_url/;
+            proxy_http_version 1.1;
+            proxy_set_header Upgrade $http_upgrade;
+            proxy_set_header Connection "upgrade";
+            proxy_read_timeout 86400s;
+            proxy_buffering off;
+            proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+            proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
+        }
+    }
+}

router/router-fixed.py

@@ -0,0 +1,831 @@
+import os, json, time, logging, traceback, threading, queue, statistics, math
+import requests, redis
+from flask import Flask, request, jsonify, Response, stream_with_context
+
+REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
+GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
+GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
+GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
+
+GPU_SIDECARS = {
+    "qwen3.6-35B-A3B": "http://192.168.68.15:8090",
+    "qwen3.6-27B-code": "http://192.168.68.8:8090",
+    "gemma-4-12b": "http://192.168.68.110:8090",
+}
+GPU_URLS = {
+    "qwen3.6-35B-A3B": GPU_MOE_URL,
+    "qwen3.6-27B-code": GPU_DENSE_URL,
+    "gemma-4-12b": GPU_LIGHT_URL,
+}
+# Max concurrent requests per GPU (based on llama.cpp --parallel)
+GPU_MAX_CONCURRENT = {
+    "qwen3.6-35B-A3B": 2,   # 2 slots (cross-agent spread prevents overheating)
+    "qwen3.6-27B-code": 2,  # 2 slots
+    "gemma-4-12b": 2,       # 2 slots (12GB VRAM, 4GB headroom)
+}
+
+# Context window sizes (tokens) — used for compaction signals
+GPU_CONTEXT = {
+    "qwen3.6-35B-A3B": 262144,
+    "qwen3.6-27B-code": 262144,
+    "gemma-4-12b": 262144,
+}
+
+TIER_MODELS = {
+    "starter": ["gemma-4-12b"],
+    "professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
+    "enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
+}
+# ── PHASE 0: Dual-Key API Key System ──
+# API_KEYS env var is REQUIRED (JSON string). No hardcoded fallback.
+# Format: {"sk-xxx": {"tier": "enterprise", "agent": "Name"}}
+# Deprecated keys get {"deprecated": true} — still accepted, logged with warning.
+_raw_keys = os.environ.get("API_KEYS")
+if not _raw_keys:
+    raise RuntimeError("FATAL: API_KEYS environment variable is required. "
+                       "Set it in docker-compose.yml or .env file. "
+                       "No hardcoded keys fallback — this is a security feature.")
+API_KEYS = json.loads(_raw_keys)
+log.info("Loaded %d API keys from env var (%d deprecated)",
+         len(API_KEYS), sum(1 for v in API_KEYS.values() if v.get("deprecated")))
+# Rate limits: requests per minute per API key tier
+RATE_LIMIT_RPM = {
+    "enterprise": 120,
+    "professional": 60,
+    "starter": 20,
+}
+
+def check_rate_limit(api_key, tier):
+    """Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remaining, reset_seconds)."""
+    if not r:
+        return True, 999, 60
+    limit = RATE_LIMIT_RPM.get(tier, 30)
+    key = f"ratelimit:{api_key}"
+    current = int(r.get(key) or 0)
+    if current >= limit:
+        ttl = r.ttl(key)
+        retry = max(ttl, 1) if ttl and ttl > 0 else 60
+        return False, retry, 0
+    pipe = r.pipeline()
+    pipe.incr(key)
+    pipe.expire(key, 60)  # 1-minute sliding window
+    pipe.execute()
+    remaining = limit - (current + 1)
+    reset_seconds = r.ttl(key) or 60
+    return True, remaining, reset_seconds
+
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
+log = logging.getLogger("router")
+try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
+except Exception: r = None
+
+
+def counter_audit_loop():
+    """Every 30s, check GPU slots and reset counters if all slots idle."""
+    while True:
+        time.sleep(30)
+        if not r: continue
+        for model, url in GPU_URLS.items():
+            try:
+                resp = requests.get(url.replace("/v1","") + "/slots",
+                    headers={"Authorization": "Bearer not-needed"}, timeout=5)
+                if resp.status_code == 200:
+                    slots = resp.json()
+                    all_idle = all(not s.get("is_processing", False) for s in slots)
+                    if all_idle:
+                        current = int(r.get("active:" + model) or 0)
+                        if current > 0:
+                            r.set("active:" + model, 0)
+                            log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
+            except Exception:
+                pass
+
+threading.Thread(target=counter_audit_loop, daemon=True).start()
+
+app = Flask(__name__)
+sse_subscribers = []; sse_lock = threading.Lock()
+
+def gpu_active_count(model):
+    """Get number of in-flight requests for a GPU."""
+    if r:
+        return int(r.get("active:" + model) or 0)
+    return 0
+
+def gpu_incr(model):
+    if r: r.incr("active:" + model)
+
+def gpu_decr(model):
+    if r:
+        v = r.decr("active:" + model)
+        if v and int(v) < 0:
+            r.set("active:" + model, 0)  # never go negative
+
+def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
+    url = GPU_SIDECARS.get(model)
+    if not url: return {"status": "unknown"}
+    try:
+        resp = requests.get(url, timeout=sidecar_timeout)
+        if resp.status_code == 200:
+            d = resp.json()
+            pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
+            status = "healthy"  # VRAM usage != saturation; busy slots handled by is_gpu_busy()
+            vram_warning = pct >= 95
+            # Also check if llama.cpp endpoint is actually responding
+            gpu_url = GPU_URLS.get(model, "")
+            try:
+                hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=gpu_timeout)
+                if hr.status_code != 200:
+                    status = "down"
+            except Exception:
+                status = "down"
+            return {"status": status, "vram_warning": vram_warning, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
+    except Exception: pass
+    return {"status": "down"}
+
+def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
+
+def estimate_tokens(msgs):
+    """Estimate token count from messages. Uses JSON length / 3.5 (closer to real tokenizer ratios for dense text)."""
+    return len(json.dumps(msgs, default=str)) // 3.5
+
+def store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, stream):
+    """Store detailed performance record in Redis for analytics."""
+    if not r: return
+    try:
+        total_ms = queue_ms + inference_ms
+        tps = completion_tokens / (inference_ms / 1000) if inference_ms > 0 and completion_tokens > 0 else 0
+        rec = json.dumps({
+            "ts": time.time(),
+            "model": model, "agent": agent, "tier": tier, "reason": reason,
+            "queue_ms": round(queue_ms, 1),
+            "inference_ms": round(inference_ms, 1),
+            "total_ms": round(total_ms, 1),
+            "prompt_tokens": prompt_tokens,
+            "completion_tokens": completion_tokens,
+            "tokens_per_sec": round(tps, 1),
+            "stream": stream
+        })
+        # Global recent list (last 500)
+        r.lpush("perf:recent", rec)
+        r.ltrim("perf:recent", 0, 499)
+        # Per-model list (last 200)
+        r.lpush("perf:model:" + model, rec)
+        r.ltrim("perf:model:" + model, 0, 199)
+        # Per-reason list (last 200)
+        r.lpush("perf:reason:" + reason, rec)
+        r.ltrim("perf:reason:" + reason, 0, 199)
+        # Per-agent list (last 200)
+        r.lpush("perf:agent:" + agent, rec)
+        r.ltrim("perf:agent:" + agent, 0, 199)
+    except Exception:
+        pass
+
+def is_gpu_busy(model):
+    """Check if GPU is at or near max concurrent capacity."""
+    active = gpu_active_count(model)
+    max_c = GPU_MAX_CONCURRENT.get(model, 1)
+    return active >= max_c
+
+def select_best_gpu(candidates, reason, agent=""):
+    """Pick best GPU, spreading agents across GPUs to prevent hotspots."""
+    # Count how many distinct agents are on each GPU
+    gpu_agent_counts = {}
+    if r:
+        for m in GPU_URLS:
+            count = 0
+            for ak in API_KEYS.values():
+                if r.get("agent_gpu:" + ak["agent"] + ":" + m):
+                    count += 1
+            gpu_agent_counts[m] = count
+    # First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
+    for m in candidates:
+        if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
+            return {"model": m, "reason": reason}
+    # Second pass: prefer GPU this agent is NOT already on (skip own GPU)
+    if agent:
+        for m in candidates:
+            if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
+                return {"model": m, "reason": reason}
+    # Third pass: any non-busy GPU
+    for m in candidates:
+        if not is_gpu_busy(m):
+            return {"model": m, "reason": reason}
+    # All busy — pick least loaded
+    best = None
+    best_load = 999
+    for m in candidates:
+        load = gpu_active_count(m)
+        if load < best_load:
+            best_load = load
+            best = m
+    if best:
+        return {"model": best, "reason": "load_balanced_" + reason}
+    return None
+
+def route(rd, tier, agent=""):
+    msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
+    sys = any(m.get("role")=="system" for m in msgs)
+    turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
+    hints = rd.get("routing_hints",{})
+    allowed = TIER_MODELS.get(tier, ["gemma-4-12b"])
+    avail = [m for m in available_models() if m in allowed]
+    if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
+    # Check if all available GPUs are at max capacity
+    if all(is_gpu_busy(m) for m in avail):
+        return {"model": avail[0], "reason": "all_saturated", "saturated": True}
+    
+    req = rd.get("model","auto")
+    if req != "auto":
+        # STRICT MODE: no silent fallback — LiteLLM handles failover chains.
+        # This keeps per-model metrics accurate. Returns saturated if busy.
+        target = req if req in avail else avail[0]
+        if req not in avail:
+            return {"model": req, "reason": "explicit_unavailable", "saturated": True}
+        if is_gpu_busy(target):
+            return {"model": target, "reason": "explicit_saturated", "saturated": True}
+        return {"model": target, "reason": "explicit"}
+    
+    if hints:
+        if hints.get("priority")=="speed" and "gemma-4-12b" in avail:
+            return select_best_gpu(["gemma-4-12b"], "hint_speed", agent) or {"model":"gemma-4-12b","reason":"hint_speed"}
+        if hints.get("priority")=="quality" and "qwen3.6-35B-A3B" in avail:
+            return select_best_gpu(["qwen3.6-35B-A3B"], "hint_quality", agent) or {"model":"qwen3.6-35B-A3B","reason":"hint_quality"}
+    
+    first_msg = msgs[0].get("content","") if msgs else ""
+    words = len(first_msg.split()) if isinstance(first_msg, str) else 99
+    
+    # TIER 1: Lightweight — single-turn short queries → VLM (fastest)
+    if not sys and turns <= 1 and t <= 500 and words <= 100 and "gemma-4-12b" in avail:
+        if not is_gpu_busy("gemma-4-12b"):
+            return {"model":"gemma-4-12b","reason":"lightweight"}
+        # VLM busy — Dense is faster for short queries than MoE
+        fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
+        result = select_best_gpu(fallback, "lightweight_fallback", agent)
+        if result: return result
+    
+    # TIER 2: Simple conversations — VLM primary (up to 15K tok), fastest for moderate chat
+    if t <= 15000 and turns <= 12 and "gemma-4-12b" in avail:
+        if not is_gpu_busy("gemma-4-12b"):
+            return {"model":"gemma-4-12b","reason":"simple_conv"}
+        # VLM busy — fall back to Dense, then MoE
+        fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
+        result = select_best_gpu(fallback, "simple_conv_fallback", agent)
+        if result: return result
+    
+    # TIER 3: Medium complexity — Dense primary, VLM fallback (quality + speed balance)
+    if t <= 25000:
+        candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
+        result = select_best_gpu(candidates, "medium", agent)
+        if result: return result
+    
+    # TIER 4: Heavy reasoning — MoE primary (workhorse), Dense fallback
+    if t > 25000:
+        candidates = [m for m in ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"] if m in avail]
+        result = select_best_gpu(candidates, "heavy_reasoning", agent)
+        if result: return result
+    
+    # TIER 5: Default — Dense primary, MoE fallback
+    candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
+    result = select_best_gpu(candidates, "default", agent)
+    if result: return result
+    return {"model":avail[0],"reason":"last_resort"}
+
+def clean_unicode(text):
+    if not isinstance(text, str): return text
+    text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
+    text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
+    text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
+    text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
+    return text.encode("ascii", "ignore").decode("ascii")
+
+def clean_response(d):
+    if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
+    if isinstance(d, list): return [clean_response(v) for v in d]
+    if isinstance(d, str): return clean_unicode(d)
+    return d
+
+def get_metrics():
+    d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
+    for m in GPU_URLS:
+        h = check_gpu_health(m)
+        d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
+        d["active_requests"][m] = gpu_active_count(m)
+    if r:
+        try:
+            for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
+            for k,v in API_KEYS.items():
+                c = int(r.get("routes:agent:"+v["agent"]) or 0)
+                if c>0: d["agent_counts"][v["agent"]] = c
+            for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
+            raw = r.lrange("routes:recent",0,49)
+            d["recent"] = [json.loads(x) for x in raw] if raw else []
+        except Exception: pass
+    return d
+
+def bcast():
+    data = get_metrics(); payload = json.dumps(data)
+    with sse_lock:
+        dead = []
+        for q in sse_subscribers:
+            try: q.put(payload)
+            except Exception: dead.append(q)
+        for q in dead: sse_subscribers.remove(q)
+
+QUEUE_TIMEOUT = int(os.environ.get("QUEUE_TIMEOUT", "30"))  # max seconds to queue before 503
+
+@app.route("/v1/chat/completions", methods=["POST"])
+def chat():
+    try:
+        rd = request.get_json(force=True)
+        ak = request.headers.get("Authorization","").replace("Bearer ","")
+        if not ak or ak not in API_KEYS:
+            log.warning("AUTH_REJECTED: no/invalid API key from %s", request.remote_addr)
+            return jsonify({"error": "Unauthorized — valid API key required"}), 401
+        ki = API_KEYS[ak]
+        tier, agent = ki["tier"], ki["agent"]
+        # Phase 0: dual-key transition — log deprecated key usage
+        if ki.get("deprecated"):
+            new_key = next((k for k, v in API_KEYS.items()
+                           if v.get("agent") == agent and not v.get("deprecated")), None)
+            log.warning("DEPRECATED_KEY: agent=%s using old key %s...%s — switch to %s...%s",
+                       agent, ak[:12], ak[-8:],
+                       new_key[:12] if new_key else "N/A",
+                       new_key[-8:] if new_key else "N/A")
+            if r:
+                r.incr("deprecated_usage:" + agent)
+        
+        # Rate limit check
+        allowed, rl_val, reset_sec = check_rate_limit(ak, tier)
+        if not allowed:
+            resp = jsonify({"error": "Rate limit exceeded", "retry_after_s": rl_val})
+            resp.headers["Retry-After"] = str(rl_val)
+            resp.headers["X-RateLimit-Limit"] = str(RATE_LIMIT_RPM.get(tier, 30))
+            resp.headers["X-RateLimit-Remaining"] = "0"
+            resp.headers["X-RateLimit-Reset"] = str(int(time.time() + rl_val))
+            log.warning("RATE_LIMIT: %s (%s) exceeded limit", agent, ak[-8:])
+            return resp, 429
+        
+        # Allow agent to override queue timeout via header
+        q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
+        
+        # Cross-turn context tracking: accumulate tokens per session
+        session_id = request.headers.get("X-Session-Id", "")
+        session_tokens = 0
+        if session_id and r:
+            try:
+                prev = int(r.get("session:" + session_id) or 0)
+                current = estimate_tokens(rd.get("messages",[]))
+                session_tokens = max(prev, current)  # context only grows
+                r.set("session:" + session_id, session_tokens, ex=86400)  # TTL 24h
+            except Exception: pass
+        
+        d = route(rd, tier, agent)
+        queue_start = time.time()
+        
+        # Queue loop: wait for a GPU slot instead of immediate 503
+        while d.get("saturated"):
+            elapsed = time.time() - queue_start
+            if elapsed > q_timeout:
+                resp = jsonify({"error": "All GPUs saturated", "queued_s": round(elapsed,1), "retry_after_s": 5})
+                resp.headers["Retry-After"] = "5"
+                log.warning("QUEUE_TIMEOUT: %s waited %.1fs, all GPUs saturated", agent, elapsed)
+                return resp, 503
+            time.sleep(0.5)  # poll every 500ms
+            d = route(rd, tier, agent)
+        
+        queue_ms = (time.time() - queue_start) * 1000
+        if queue_ms > 500:
+            log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
+        model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
+        
+        # Stash rate limit values for response headers
+        _rl_remaining = rl_val
+        _rl_limit = RATE_LIMIT_RPM.get(tier, 30)
+        _rl_reset = reset_sec
+        is_stream = rd.get("stream", False)
+        
+        gpu_incr(model)
+        
+        log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
+        # Track which GPU this agent is using (TTL 120s covers typical request)
+        if r and agent:
+            try: r.setex("agent_gpu:" + agent + ":" + model, 120, "1")
+            except: pass
+        if r:
+            try:
+                r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
+                r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
+                r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent,"queue_ms": round(queue_ms,1)}))
+                r.ltrim("routes:recent",0,999)
+            except Exception: pass
+        start = time.time()
+        resp = requests.post(url+"/chat/completions", json=rd,
+            headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
+        lat = int((time.time()-start)*1000)
+        gpu_decr(model)
+        
+        if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
+        if is_stream:
+            # Buffer SSE chunks, handle split lines for large responses
+            chunks = []
+            stream_timings = {}
+            buf = ""  # accumulate partial lines
+            for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
+                if raw:
+                    cleaned = clean_unicode(raw)
+                    chunks.append(cleaned)
+                    buf += cleaned
+                    # Process complete lines from buffer
+                    while "\n" in buf:
+                        line, buf = buf.split("\n", 1)
+                        line = line.strip()
+                        if line.startswith("data: ") and not stream_timings:
+                            js = line[6:].strip()
+                            if js.startswith("{") and "timings" in js and "predicted_n" in js:
+                                try:
+                                    tj = json.loads(js).get("timings", {})
+                                    if tj:
+                                        stream_timings = tj
+                                except: pass
+            # Store perf record with real token counts from stream
+            if stream_timings:
+                pt = stream_timings.get("prompt_n", 0)
+                ct = stream_timings.get("predicted_n", 0)
+                tps = stream_timings.get("predicted_per_second", 0)
+                gen_ms = stream_timings.get("predicted_ms", lat)
+                store_perf_record(model, agent, tier, reason, queue_ms, gen_ms, pt, ct, True)
+            else:
+                store_perf_record(model, agent, tier, reason, queue_ms, lat, estimate_tokens(rd.get("messages",[])), 0, True)
+            # Yield all chunks to client
+            def gen():
+                for c in chunks: yield c
+            bcast()
+            ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
+            ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
+            ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
+            sse_resp = Response(stream_with_context(gen()), mimetype="text/event-stream")
+            sse_resp.headers["X-RateLimit-Limit"] = str(_rl_limit)
+            sse_resp.headers["X-RateLimit-Remaining"] = str(max(0, _rl_remaining))
+            sse_resp.headers["X-RateLimit-Reset"] = str(int(time.time() + _rl_reset))
+            sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
+            sse_resp.headers["X-Context-Warning"] = ctx_warning
+            sse_resp.headers["X-Context-Model"] = model
+            return sse_resp
+        data = clean_response(resp.json())
+        for c in data.get("choices",[]):
+            msg = c.get("message",{})
+            if not msg.get("content") and msg.get("reasoning_content"):
+                msg["content"] = msg["reasoning_content"]
+        # Extract performance data from llama.cpp response
+        usage = data.get("usage", {})
+        timings = data.get("timings", {})
+        prompt_tokens = usage.get("prompt_tokens", 0)
+        completion_tokens = usage.get("completion_tokens", 0)
+        inference_ms = lat  # total GPU round-trip
+        store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, False)
+        ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
+        ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
+        ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
+        data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"queue_ms": round(queue_ms,1),"active_gpu":gpu_active_count(model),"context_remaining": max(0, ctx_remaining),"context_pct": round(ctx_pct,1),"context_warning": ctx_warning}
+        resp = jsonify(data)
+        resp.headers["X-RateLimit-Limit"] = str(_rl_limit)
+        resp.headers["X-RateLimit-Remaining"] = str(max(0, _rl_remaining))
+        resp.headers["X-RateLimit-Reset"] = str(int(time.time() + _rl_reset))
+        resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
+        resp.headers["X-Context-Warning"] = ctx_warning
+        resp.headers["X-Context-Model"] = model
+        bcast()
+        return resp
+    except requests.Timeout:
+        gpu_decr(model)
+        log.error("TIMEOUT: %s -> %s", agent, model)
+        return jsonify({"error":"timeout"}), 504
+    except Exception as e:
+        gpu_decr(model)
+        log.error("Error: %s\n%s", e, traceback.format_exc())
+        return jsonify({"error":str(e)}), 500
+
+@app.route("/metrics/performance")
+def performance():
+    """Per-request performance analytics with percentiles per model/reason/agent."""
+    if not r: return jsonify({"error": "Redis unavailable"}), 503
+    try:
+        window_hours = int(request.args.get("window", "24"))
+        model_filter = request.args.get("model", "all")
+
+        # Load recent records
+        cutoff = time.time() - (window_hours * 3600)
+        raw = r.lrange("perf:recent", 0, -1)
+        records = []
+        for x in raw:
+            try:
+                rec = json.loads(x)
+                if rec["ts"] >= cutoff:
+                    records.append(rec)
+            except: pass
+
+        # Filter by model if specified
+        if model_filter != "all":
+            records = [r for r in records if r["model"] == model_filter]
+
+        if not records:
+            return jsonify({"models": [], "reasons": [], "agents": [], "summary": {"total_requests": 0}})
+
+        def pct(values, p):
+            if len(values) < 2: return round(values[0], 1) if values else 0
+            return round(statistics.quantiles(sorted(values), n=100, method='inclusive')[min(p-1, 98)], 1)
+
+        # Per-model stats
+        model_groups = {}
+        for rec in records:
+            m = rec["model"]
+            if m not in model_groups: model_groups[m] = []
+            model_groups[m].append(rec)
+
+        models = []
+        for m, recs in sorted(model_groups.items()):
+            latencies = [r["total_ms"] for r in recs]
+            tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
+            non_stream = [r for r in recs if not r["stream"]]
+            queue_times = [r["queue_ms"] for r in non_stream]
+            models.append({
+                "model": m,
+                "count": len(recs),
+                "stream_pct": round(len([r for r in recs if r["stream"]]) / len(recs) * 100, 1),
+                "latency": {
+                    "avg": round(statistics.mean(latencies), 1),
+                    "p50": pct(latencies, 50),
+                    "p95": pct(latencies, 95),
+                    "p99": pct(latencies, 99)
+                },
+                "throughput": {
+                    "avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0,
+                    "p50": pct(tps_vals, 50) if tps_vals else 0,
+                    "p95": pct(tps_vals, 95) if tps_vals else 0,
+                },
+                "queue": {
+                    "avg_ms": round(statistics.mean(queue_times), 1) if queue_times else 0,
+                    "p95_ms": pct(queue_times, 95) if queue_times else 0,
+                } if queue_times else None
+            })
+
+        # Per-reason stats
+        reason_groups = {}
+        for rec in records:
+            rsn = rec["reason"]
+            if rsn not in reason_groups: reason_groups[rsn] = []
+            reason_groups[rsn].append(rec)
+
+        reasons = []
+        for rsn, recs in sorted(reason_groups.items(), key=lambda x: -len(x[1])):
+            latencies = [r["total_ms"] for r in recs]
+            reasons.append({
+                "reason": rsn,
+                "count": len(recs),
+                "avg_total_ms": round(statistics.mean(latencies), 1),
+                "p95_total_ms": pct(latencies, 95)
+            })
+
+        # Per-agent stats
+        agent_groups = {}
+        for rec in records:
+            ag = rec["agent"]
+            if ag not in agent_groups: agent_groups[ag] = []
+            agent_groups[ag].append(rec)
+
+        agents = []
+        for ag, recs in sorted(agent_groups.items(), key=lambda x: -len(x[1])):
+            latencies = [r["total_ms"] for r in recs]
+            tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
+            agents.append({
+                "agent": ag,
+                "count": len(recs),
+                "avg_total_ms": round(statistics.mean(latencies), 1),
+                "avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0
+            })
+
+        all_lat = [r["total_ms"] for r in records]
+        all_tps = [r["tokens_per_sec"] for r in records if r["tokens_per_sec"] > 0]
+        summary = {
+            "total_requests": len(records),
+            "window_hours": window_hours,
+            "latency": {
+                "avg_ms": round(statistics.mean(all_lat), 1),
+                "p50_ms": pct(all_lat, 50),
+                "p95_ms": pct(all_lat, 95),
+                "p99_ms": pct(all_lat, 99)
+            },
+            "throughput_avg_tps": round(statistics.mean(all_tps), 1) if all_tps else 0
+        }
+
+        return jsonify({"models": models, "reasons": reasons, "agents": agents, "summary": summary})
+    except Exception as e:
+        return jsonify({"error": str(e)}), 500
+
+@app.route("/metrics/scatter")
+def scatter():
+    """Return individual data points for scatter plots (prompt_tokens vs latency)."""
+    if not r: return jsonify({"error": "Redis unavailable"}), 503
+    try:
+        window_hours = int(request.args.get("window", "24"))
+        model_filter = request.args.get("model", "all")
+        cutoff = time.time() - (window_hours * 3600)
+        raw = r.lrange("perf:recent", 0, -1)
+        points = []
+        for x in raw:
+            try:
+                rec = json.loads(x)
+                if rec["ts"] >= cutoff:
+                    if model_filter == "all" or rec["model"] == model_filter:
+                        points.append({
+                            "model": rec["model"],
+                            "agent": rec["agent"],
+                            "reason": rec["reason"],
+                            "prompt_tokens": int(rec.get("prompt_tokens", 0)),
+                            "completion_tokens": rec.get("completion_tokens", 0),
+                            "inference_ms": round(rec["inference_ms"], 1),
+                            "tokens_per_sec": rec.get("tokens_per_sec", 0),
+                            "stream": rec.get("stream", False)
+                        })
+            except: pass
+        return jsonify({"points": points, "count": len(points)})
+    except Exception as e:
+        return jsonify({"error": str(e)}), 500
+
+@app.route("/v1/models")
+def models():
+    def _h(m): return check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
+    return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":_h(m).get("status"),"gpu":_h(m).get("gpu_name")} for m in GPU_URLS]})
+
+@app.route("/health")
+def health():
+    gpus = {}
+    for m in GPU_URLS:
+        h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
+        h["active_requests"] = gpu_active_count(m)
+        h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
+        gpus[m] = h
+    return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
+
+@app.route("/metrics")
+def metrics(): return jsonify(get_metrics())
+
+@app.route("/metrics/timeseries")
+def metrics_timeseries():
+    period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
+    data = {"models": {}, "labels": []}
+    if period == "day":
+        buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
+        data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
+    elif period == "week":
+        buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
+        data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
+    else:
+        buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
+        data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
+    if r:
+        for model in models_list:
+            counts = []
+            for bucket in buckets:
+                total = 0
+                if period in ("week","month"):
+                    for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
+                else: total = int(r.get("ts:"+model+":"+bucket) or 0)
+                counts.append(total)
+            data["models"][model] = counts
+    return jsonify(data)
+
+@app.route("/stream")
+def stream():
+    def ev():
+        q = queue.Queue()
+        with sse_lock: sse_subscribers.append(q)
+        try:
+            yield "data: "+json.dumps(get_metrics())+"\n\n"
+            while True:
+                try: yield "data: "+q.get(timeout=3)+"\n\n"
+                except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
+        except GeneratorExit: pass
+        finally:
+            with sse_lock:
+                if q in sse_subscribers: sse_subscribers.remove(q)
+    return Response(stream_with_context(ev()), mimetype="text/event-stream",
+                    headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
+
+# ── Phase 0: Admin Key Management ──
+ADMIN_KEY = os.environ.get("ADMIN_KEY", "")
+
+def _admin_auth():
+    """Require admin key for management endpoints."""
+    if not ADMIN_KEY:
+        return False, "ADMIN_KEY not configured on server"
+    ak = request.headers.get("Authorization","").replace("Bearer ","")
+    if ak != ADMIN_KEY:
+        return False, "Admin key required"
+    return True, None
+
+@app.route("/admin/keys")
+def admin_keys():
+    """List all API keys (masked) with agent, tier, and deprecation status."""
+    ok, err = _admin_auth()
+    if not ok: return jsonify({"error": err}), 401
+    keys = []
+    for key, info in API_KEYS.items():
+        masked = key[:8] + "..." + key[-8:]
+        keys.append({
+            "masked": masked,
+            "prefix": key[:8],
+            "agent": info["agent"],
+            "tier": info["tier"],
+            "deprecated": info.get("deprecated", False),
+            "length": len(key)
+        })
+    return jsonify({
+        "total": len(keys),
+        "active": sum(1 for k in keys if not k["deprecated"]),
+        "deprecated": sum(1 for k in keys if k["deprecated"]),
+        "keys": sorted(keys, key=lambda k: (k["deprecated"], k["agent"]))
+    })
+
+@app.route("/admin/keys/deprecation-summary")
+def admin_deprecation_summary():
+    """Summary of deprecated key usage (from Redis logs, if available)."""
+    ok, err = _admin_auth()
+    if not ok: return jsonify({"error": err}), 401
+    deprecated_agents = []
+    for key, info in API_KEYS.items():
+        if info.get("deprecated"):
+            # Check Redis for usage count
+            count = 0
+            if r:
+                count = int(r.get("deprecated_usage:" + info["agent"]) or 0)
+            deprecated_agents.append({
+                "agent": info["agent"],
+                "deprecated_uses": count,
+                "needs_migration": count > 0
+            })
+    return jsonify({
+        "deprecated_agents": sorted(deprecated_agents, key=lambda d: -d["deprecated_uses"]),
+        "recommendation": "Run POST /admin/keys/revoke to remove keys with 0 usage"
+    })
+
+@app.route("/admin/keys/generate", methods=["POST"])
+def admin_generate_key():
+    """Generate a new API key for an agent. Body: {"agent": "Name", "tier": "enterprise"}"""
+    ok, err = _admin_auth()
+    if not ok: return jsonify({"error": err}), 401
+    body = request.get_json(force=True)
+    agent = body.get("agent", "").strip()
+    tier = body.get("tier", "enterprise")
+    if not agent:
+        return jsonify({"error": "agent field required"}), 400
+    if tier not in ("starter", "professional", "enterprise"):
+        return jsonify({"error": "tier must be starter/professional/enterprise"}), 400
+    # Generate secure key
+    import secrets, hashlib
+    prefix = hashlib.sha256(secrets.token_bytes(12)).hexdigest()[:8]
+    suffix = secrets.token_hex(20)
+    new_key = f"sk-{prefix}-{suffix}"
+    # Update in-memory dict (note: not persisted across restarts without env var update)
+    API_KEYS[new_key] = {"tier": tier, "agent": agent}
+    log.info("KEY_GENERATED: agent=%s tier=%s key=%s...%s", agent, tier, new_key[:8], new_key[-8:])
+    return jsonify({
+        "agent": agent,
+        "tier": tier,
+        "key": new_key,
+        "masked": new_key[:8] + "..." + new_key[-8:],
+        "warning": "This key exists in memory only. Update API_KEYS env var and redeploy to persist."
+    }), 201
+
+@app.route("/admin/keys/revoke", methods=["POST"])
+def admin_revoke_key():
+    """Revoke a deprecated key. Body: {"agent": "Name"} or {"key_prefix": "sk-xxxx"}"""
+    ok, err = _admin_auth()
+    if not ok: return jsonify({"error": err}), 401
+    body = request.get_json(force=True)
+    agent = body.get("agent", "")
+    key_prefix = body.get("key_prefix", "")
+    revoked = []
+    keys_to_remove = []
+    for key, info in API_KEYS.items():
+        if not info.get("deprecated"):
+            continue
+        if agent and info["agent"] == agent:
+            keys_to_remove.append(key)
+        elif key_prefix and key.startswith(key_prefix):
+            keys_to_remove.append(key)
+    for key in keys_to_remove:
+        info = API_KEYS.pop(key)
+        revoked.append({"agent": info["agent"], "masked": key[:8] + "..." + key[-8:]})
+        log.warning("KEY_REVOKED: agent=%s key=%s...%s", info["agent"], key[:8], key[-8:])
+    return jsonify({
+        "revoked": len(revoked),
+        "keys": revoked,
+        "remaining_total": len(API_KEYS),
+        "warning": "Memory-only revoke. Update API_KEYS env var and redeploy to persist."
+    })
+
+if __name__ == "__main__":
+    log.info("Router on :9000 (load-aware)")
+    app.run(host="0.0.0.0", port=9000, debug=False)

router/router.py

@@ -14,6 +14,37 @@ redis.call('SET', key, max_val, 'EX', 86400)
 return max_val
 """
 
+# Phase 4: Atomic GPU slot booking (closes TOCTOU race between check and incr)
+SLOT_BOOK_LUA = """
+local key = KEYS[1]
+local max_c = tonumber(ARGV[1])
+local current = tonumber(redis.call('GET', key) or '0')
+if current < max_c then
+    redis.call('INCR', key)
+    return 1
+else
+    return 0
+end
+"""
+SLOT_RELEASE_LUA = """
+local key = KEYS[1]
+local current = tonumber(redis.call('GET', key) or '0')
+if current > 0 then
+    redis.call('DECR', key)
+end
+current = tonumber(redis.call('GET', key) or '0')
+if current < 0 then
+    redis.call('SET', key, '0')
+end
+return redis.call('GET', key)
+"""
+
+# Phase 3b: Configurable health scoring weights (env-overridable)
+HEALTH_WEIGHT_VRAM = float(os.environ.get("HEALTH_WEIGHT_VRAM", "0.40"))
+HEALTH_WEIGHT_TEMP = float(os.environ.get("HEALTH_WEIGHT_TEMP", "0.30"))
+HEALTH_WEIGHT_LOAD = float(os.environ.get("HEALTH_WEIGHT_LOAD", "0.30"))
+HEALTH_TEMP_BASELINE = int(os.environ.get("HEALTH_TEMP_BASELINE", "30"))
+
 
 REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
 GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
@@ -152,6 +183,36 @@ def gpu_decr(model):
         v = rd.decr("active:" + model)
         if v and int(v) < 0:
             get_redis().set("active:" + model, 0)  # never go negative
+
+# Phase 4: Atomic GPU slot booking (Lua-based, closes TOCTOU race)
+def gpu_book_slot(model):
+    """Atomically book a GPU slot. Returns True if acquired, False if full."""
+    rd = get_redis()
+    if not rd:
+        return True  # No Redis — allow everything (degraded mode)
+    try:
+        max_c = GPU_MAX_CONCURRENT.get(model, 1)
+        result = rd.eval(SLOT_BOOK_LUA, 1, "active:" + model, max_c)
+        return result == 1
+    except Exception:
+        # Lua not loaded — fall back to non-atomic
+        current = int(rd.get("active:" + model) or 0)
+        if current < GPU_MAX_CONCURRENT.get(model, 1):
+            rd.incr("active:" + model)
+            return True
+        return False
+
+def gpu_release_slot(model):
+    """Atomically release a GPU slot. Never goes negative."""
+    rd = get_redis()
+    if not rd:
+        return
+    try:
+        rd.eval(SLOT_RELEASE_LUA, 1, "active:" + model)
+    except Exception:
+        v = rd.decr("active:" + model)
+        if v and int(v) < 0:
+            rd.set("active:" + model, 0)
 def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
     url = GPU_SIDECARS.get(model)
     if not url: return {"status": "unknown"}
@@ -222,18 +283,22 @@ def is_gpu_busy(model):
 
 # Phase 3: Dynamic GPU Weighting (Health Score)
 def gpu_health_score(model):
-    """Score a GPU based on VRAM, temperature, and load. Lower = better."""
+    """Score a GPU based on VRAM, temperature, power, and load. Lower = better.
+    Weights configurable via HEALTH_WEIGHT_VRAM/TEMP/LOAD env vars."""
     h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
     if h.get("status") == "down":
         return 999  # never pick down GPUs
-    vram_pct = h.get("vram_pct", 50)
-    temp_c = h.get("temp_c", 50)
+    if is_circuit_tripped(model):
+        return 998  # circuit open — skip but distinguishable from down
+    vram_pct = h.get("vram_pct") or 50
+    temp_c = h.get("temp_c") or 50
+    power_w = h.get("power_w") or 100
     active = gpu_active_count(model)
     max_c = GPU_MAX_CONCURRENT.get(model, 1)
-    load_pct = (active / max_c) * 100
-    # Score: lower = better (more headroom, cooler, less loaded)
-    score = vram_pct * 0.4 + max(temp_c - 30, 0) * 0.3 + load_pct * 0.3
-    return score
+    load_pct = (active / max_c) * 100 if max_c > 0 else 0
+    temp_penalty = max(0, (temp_c or 50) - HEALTH_TEMP_BASELINE)
+    score = (vram_pct or 0) * HEALTH_WEIGHT_VRAM + temp_penalty * 0.5 * HEALTH_WEIGHT_TEMP + load_pct * HEALTH_WEIGHT_LOAD
+    return round(score, 1)
 
 def select_best_gpu(candidates, reason, agent=""):
     """Pick best GPU, spreading agents across GPUs to prevent hotspots."""
@@ -278,20 +343,40 @@ def select_best_gpu(candidates, reason, agent=""):
 
 
 # Phase 1: Circuit Breaker for GPU Hosts (Approved by Abiba)
+CIRCUIT_FAIL_THRESHOLD = int(os.environ.get("CIRCUIT_FAIL_THRESHOLD", "3"))
+CIRCUIT_FAIL_WINDOW = int(os.environ.get("CIRCUIT_FAIL_WINDOW", "120"))
+CIRCUIT_COOLDOWN = int(os.environ.get("CIRCUIT_COOLDOWN", "60"))
+
 def is_circuit_tripped(model):
     """Check if a GPU host is currently blacklisted."""
     if not get_redis():
         return False
     return r.exists("circuit:" + model + ":open")
 
-def trip_circuit(model, duration=30):
-    """Blacklist a GPU host for a specified duration."""
+def trip_circuit(model, duration=None):
+    """Blacklist a GPU host for specified duration (default CIRCUIT_COOLDOWN).
+    Only trips after CIRCUIT_FAIL_THRESHOLD failures within CIRCUIT_FAIL_WINDOW."""
     if not get_redis():
-        return
+        return False
+    if duration is None:
+        duration = CIRCUIT_COOLDOWN
+    now = time.time()
+    fail_key = "circuit:" + model + ":failures"
+    pipe = r.pipeline()
+    pipe.lpush(fail_key, str(now))
+    pipe.ltrim(fail_key, 0, CIRCUIT_FAIL_THRESHOLD - 1)
+    pipe.lrange(fail_key, 0, -1)
+    results = pipe.execute()
+    failures = [float(f) for f in (results[-1] if results else [])]
+    recent = [f for f in failures if now - f <= CIRCUIT_FAIL_WINDOW]
+    if len(recent) >= CIRCUIT_FAIL_THRESHOLD:
         key = "circuit:" + model + ":open"
         r.set(key, 1, ex=duration)
         r.incr("circuit:" + model + ":count")
-    log.warning("CIRCUIT_TRIPPED: %s blacklisted for %ds", model, duration)
+        log.warning("CIRCUIT_TRIPPED: %s — %d failures in %ds, cooldown %ds", 
+                   model, len(recent), CIRCUIT_FAIL_WINDOW, duration)
+        return True
+    return False
 
 def half_open_probe(model):
     """Check if a GPU host can be un-blacklisted."""
@@ -329,13 +414,17 @@ def route(rd, tier, agent=""):
             return {"model": allowed[0], "reason": "vision_unavailable"}
 
     req = rd.get("model","auto")
+    # Map syslog-auto to auto for content-based routing
+    if req == "syslog-auto":
+        req = "auto"
     if req != "auto":
+        # STRICT MODE: no silent fallback — LiteLLM handles failover chains.
+        # Returns saturated if explicit GPU is busy (keeps per-model metrics accurate).
         target = req if req in avail else avail[0]
-        if is_gpu_busy(target) and req in allowed:
-            alts = [m for m in avail if m != target and m in allowed]
-            if alts:
-                alt = select_best_gpu(alts, "explicit", agent)
-                if alt: return alt
+        if req not in avail:
+            return {"model": req, "reason": "explicit_unavailable", "saturated": True}
+        if is_gpu_busy(target):
+            return {"model": target, "reason": "explicit_saturated", "saturated": True}
         return {"model": target, "reason": "explicit"}
 
     if hints:
@@ -503,14 +592,26 @@ def chat():
             log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
         model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
         
+        # Phase 4: Atomic slot booking (replaces non-atomic gpu_incr)
+        if not gpu_book_slot(model):
+            d = route(rd, tier, agent)
+            if d.get("saturated"):
+                resp = jsonify({"error": "All GPUs saturated", "retry_after_s": 3})
+                resp.headers["Retry-After"] = "3"
+                return resp, 503
+            model, reason = d["model"], d["reason"]
+            if not gpu_book_slot(model):
+                resp = jsonify({"error": "GPU slot race — retry", "retry_after_s": 1})
+                resp.headers["Retry-After"] = "1"
+                return resp, 503
+        url = GPU_URLS[model]
+        
         # Stash rate limit values for response headers
         _rl_remaining = rl_val
         _rl_limit = RATE_LIMIT_RPM.get(tier, 30)
         _rl_reset = reset_sec
         is_stream = rd.get("stream", False)
         
-        gpu_incr(model)
-        
         log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
         # Track which GPU this agent is using (TTL 120s covers typical request)
         if r and agent:
@@ -525,13 +626,13 @@ def chat():
             except Exception: pass
         start = time.time()
         resp = requests.post(url+"/chat/completions", json=rd,
-            headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
+            headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=900, stream=is_stream)
         lat = int((time.time()-start)*1000)
-        gpu_decr(model)
+        gpu_release_slot(model)
         
         if resp.status_code != 200:
             if resp.status_code in (502, 504):
-                trip_circuit(model, 30)
+                trip_circuit(model)
             return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
         if is_stream:
             # Buffer SSE chunks, handle split lines for large responses
@@ -578,6 +679,12 @@ def chat():
             sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
             sse_resp.headers["X-Context-Warning"] = ctx_warning
             sse_resp.headers["X-Context-Model"] = model
+            # LiteLLM spend tracking: best-effort token counts from stream timings
+            pt = stream_timings.get("prompt_n", 0) if stream_timings else 0
+            ct = stream_timings.get("predicted_n", 0) if stream_timings else 0
+            sse_resp.headers["X-Usage-Tokens"] = json.dumps({
+                "prompt_tokens": pt, "completion_tokens": ct, "model": model
+            })
             return sse_resp
         data = clean_response(resp.json())
         for c in data.get("choices",[]):
@@ -602,15 +709,19 @@ def chat():
         resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
         resp.headers["X-Context-Warning"] = ctx_warning
         resp.headers["X-Context-Model"] = model
+        # LiteLLM spend tracking: return token counts for cost computation
+        resp.headers["X-Usage-Tokens"] = json.dumps({
+            "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens, "model": model
+        })
         bcast()
         return resp
     except requests.Timeout:
-        gpu_decr(model)
-        trip_circuit(model, 30)
+        gpu_release_slot(model)
+        trip_circuit(model)
         log.error("TIMEOUT: %s -> %s (Circuit tripped)", agent, model)
         return jsonify({"error":"timeout"}), 504
     except Exception as e:
-        gpu_decr(model)
+        gpu_release_slot(model)
         log.error("Error: %s\n%s", e, traceback.format_exc())
         return jsonify({"error":str(e)}), 500
 
@@ -871,6 +982,38 @@ def metrics_latency():
         "requests_per_min": len(last_min),
         "count": len(recent)
     })
+@app.route("/health/unified")
+def health_unified():
+    """Unified health aggregating all layers: Router + Redis + GPUs + Circuit Breaker + Scores."""
+    gpus = {}
+    for m in GPU_URLS:
+        h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
+        h["active_requests"] = gpu_active_count(m)
+        h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
+        h["health_score"] = gpu_health_score(m)
+        h["circuit_open"] = is_circuit_tripped(m)
+        gpus[m] = h
+    circuit_state = {}
+    for m in GPU_URLS:
+        cooldown_until = r.ttl("circuit:" + m + ":open") if r else None
+        circuit_state[m] = {
+            "open": is_circuit_tripped(m),
+            "cooldown_remaining_s": max(0, cooldown_until) if cooldown_until and cooldown_until > 0 else 0,
+            "trip_count": int(r.get("circuit:" + m + ":count") or 0) if r else 0
+        }
+    overall = "healthy"
+    if not r:
+        overall = "degraded"
+    if all(circuit_state[m]["open"] for m in GPU_URLS):
+        overall = "down"
+    return jsonify({
+        "status": overall, "router": "healthy",
+        "redis": "connected" if r else "down",
+        "gpus": gpus, "circuit_breaker": circuit_state,
+        "scores": {m: gpu_health_score(m) for m in GPU_URLS},
+        "available_models": available_models(), "timestamp": time.time()
+    })
+
 @app.route("/stream")
 def stream():
     def ev():