Archive 688 GPU self-heal logs from shared memory

- Replaced placeholder keys with actual verified LiteLLM virtual keys
- Tanko & Mumuni: confirmed working keys from agent configs
- Abiba, Kagenz0, Koby, Koonimo: keys regenerated after old values lost
- Added key rotation history
- All stale/blocked/duplicate keys purged
- Added verification status column

.gitignore

@@ -1,3 +1,6 @@
 .git
 __pycache__/
 *.pyc
+*.bak
+*.backup*
+.env

LITELLM-KEY-DISTRIBUTION.md

@@ -0,0 +1,56 @@
+# LiteLLM Virtual Key Distribution — Phase 2 Cutover
+## Syslog Solution LLC — July 1, 2026 (keys regenerated)
+
+### Active Agent Keys (Update agent configs with these)
+
+| Agent | LiteLLM Virtual Key | Tier | Budget | Models | Verified |
+|-------|-------------------|------|--------|--------|----------|
+| **Abiba** | `sk-i7F7rCfgpS0oouOTA2LgMw` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
+| **Mumuni** | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
+| **Tanko** | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
+| **Kagenz0** | `sk-VYKRAVYEG1rKVEMDKEUggg` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
+| **Koby** | `sk-gWDaPAp-FavgKdqKJpzqhQ` | professional | $500 | All 4 | ✅ Jul 1 |
+| **Koonimo** | `sk-1kLC8ZxW-tEZueV3NU4rCQ` | professional | $500 | All 4 | ✅ Jul 1 |
+
+### Configuration Changes Required Per Agent
+
+Each agent must update their `OPENAI_API_BASE` and `OPENAI_API_KEY`:
+
+```
+OPENAI_API_BASE=http://192.168.68.116/v1
+OPENAI_API_KEY=<agent's LiteLLM key from table above>
+```
+
+### LiteLLM Admin Access
+
+| Resource | Key |
+|----------|-----|
+| LiteLLM Admin UI | http://192.168.68.116/ui/ (Authentik SSO — pending) |
+| LiteLLM Master Key | `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` |
+| Router Admin Key | `sk-admin-ee09fffd04978b61a1569ac670c68814` |
+
+### Fallback (if LiteLLM fails)
+
+If LiteLLM is down, NGINX automatically falls back to the router on :9000.
+In that case, agents can also directly use:
+```
+OPENAI_API_BASE=http://192.168.68.116/v1
+OPENAI_API_KEY=<agent's original sk-syslog-* key>
+```
+(Only works when NGINX @router_fallback is active)
+
+### Deprecated Router Keys
+
+These keys still work through the @router_fallback but NOT through LiteLLM:
+- sk-syslog-abiba, sk-syslog-mumuni, sk-syslog-tanko
+- sk-syslog-kagenz0, sk-syslog-koby, sk-syslog-koonimo
+- sk-starter-abc123, sk-professional-xyz789
+- sk-syslog-local-master-key
+
+These have been fully revoked as of July 1, 2026.
+
+### Key Rotation History
+
+- **2026-07-01**: Keys regenerated for Abiba, Kagenz0, Koby, Koonimo (old values unrecoverable).
+  Mumuni and Tanko keys confirmed working, left unchanged. All blocked/stale keys purged.
+- **2026-06-17**: Initial key creation (values in this doc were placeholder/incorrect).

LITELLM-MIGRATION-PLAN.md

@@ -0,0 +1,759 @@
+# LiteLLM Integration Migration Plan
+## Syslog Solution LLC  June 14, 2026
+
+**Deployment Target:** CT 116 `syslog-api` (192.168.68.116) on minipve  all services co-located.
+**DNS Strategy:** Option A  /etc/hosts + Docker extra_hosts for internal resolution of `auth.sysloggh.net`  192.168.68.11.
+**GitOps:** This plan lives in `SyslogSolution/syslog-harness` on Gitea. All changes tracked via git with conventional commits.
+
+---
+
+## Executive Summary
+
+**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness  without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.
+
+**Architecture Decision:** Two-layer architecture.
+
+```
+                  
+                       LiteLLM Gateway (Layer 1)    
+                       Port 4000  Policy & UX      
+                     
+                     Admin UI (/ui)               
+                     Virtual Keys & Permissions   
+                     Teams, Users, SSO (OIDC)     
+                     Spend Tracking & Budgets     
+                     Usage Analytics Dashboard    
+                     Request Audit Trail          
+                     Global Rate Limiting         
+                     
+                                                    
+                      Pass-through to router         
+                  
+                             
+                  
+                    Custom Router (Layer 2)          
+                    Port 9000  Intelligence & HW   
+                     
+                     5-Tier Content-Based Routing 
+                     GPU Slot Management (Redis)  
+                     Agent Spread Prevention      
+                     GPU Health Scoring            
+                     Sidecar VRAM/Temp/Power      
+                     Circuit Breaker              
+                     Context Window Tracking       
+                     Per-Request Perf Recording   
+                     Hardware Rate Limiting       
+                     
+                  
+                             
+          
+                                                   
+      
+   qwen3.6-35B     qwen3.6-27B      gemma-4-12b      
+   MoE/Strix       Dense/RTX3090    VLM/RTX 5070     
+   :8080 (llama)   :8080 (llama)    :8080 (llama)    
+   :8090 (side)    :8090 (side)     :8090 (sidecar)  
+      
+```
+
+---
+
+## 1. Current State Baseline
+
+### 1.1 Router (`router-fixed.py`  port 9000, deployed on CT 116 / syslog-api)
+
+**Deployment Host:** CT 116 `syslog-api` on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.
+
+| Feature | Implementation |
+|---------|---------------|
+| **Routing Engine** | 5-tier content-based: lightweight  simple_conv  medium  heavy_reasoning  default |
+| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset |
+| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health |
+| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs |
+| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 |
+| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation |
+| **Performance** | Per-request latency/tokens/tps  Redis lists (perf:recent, perf:model:X, perf:agent:X) |
+| **Context Tracking** | Session-level token accumulation with compaction warnings in headers |
+| **SSE Streaming** | Real-time dashboard updates, per-model timeseries |
+| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` |
+| **Strict Passthrough** | Explicit model requests go to that GPU exactly (no silent fallback). LiteLLM owns failover. |
+
+### 1.2 GPU Backends
+
+| GPU | Host | llama.cpp | Sidecar | VRAM | Context |
+|-----|------|-----------|---------|------|---------|
+| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K |
+| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K |
+| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K |
+
+### 1.3 Existing LiteLLM POC on CT 116
+
+CT 116 already has a LiteLLM container running (POC, 6 days uptime):
+
+```
+harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081->4000
+harness-redis   | redis:7-alpine                      | 127.0.0.1:6379
+harness-router  | inference-harness-router             | 127.0.0.1:9000
+harness-nginx   | nginx:alpine                        | 0.0.0.0:80
+harness-dashboard | inference-harness-dashboard        | 127.0.0.1:3000
+```
+
+- `/opt/litellm/`  previous setup directory on CT 116
+- Configured with Postgres, host networking, master key
+- Currently bypassed  router routes directly to GPUs
+- **Goal: Productionize with two-layer architecture on this same host**
+
+### 1.4 DNS Routing (Split-Horizon)
+
+For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally:
+
+**Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net`  72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.
+
+**Solution  Option A: /etc/hosts on CT 116 host:**
+```bash
+# On CT 116 (syslog-api)
+echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
+```
+
+**Docker containers** also need this resolution  add to docker-compose.yml:
+```yaml
+services:
+  nginx:
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+  litellm:
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+```
+
+**DNS Servers:** CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.
+
+---
+
+## 2. What LiteLLM Brings (That We Don't Have)
+
+| Feature | Our Router | LiteLLM | Value Add |
+|---------|-----------|---------|-----------|
+| **Admin UI** |  |  Full dashboard at /ui | Non-technical users can manage keys, view spend |
+| **Virtual Key Permissions** |  (binary key->tier) |  Granular: per-model, per-team, budget caps | Fine-grained access control |
+| **Spend Tracking** |  |  Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing |
+| **Teams & Orgs** |  |  Multi-tenant: org->team->user hierarchy | Segregate clients/projects |
+| **SSO/OIDC** |  |  Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration |
+| **Budget Alerts** |  |  Per-key, per-user, per-team budget with webhooks | Prevent overspend |
+| **Usage Analytics** |   (custom /metrics) |  Built-in: daily trends, model breakdown, per-customer | Better visualization |
+| **100+ Provider Support** |  (3 local GPUs) |  OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access |
+| **Fallback Chains** |  (silent rerouting) |  Explicit multi-provider failover with per-model logging | Accurate per-model tracking, visible failover |
+| **RPM/TPM Weighted LB** |  |  Weighted load balancing across deployments | Fine-grained traffic shaping |
+
+---
+
+## 3. What We Keep (That LiteLLM Doesn't Have)
+
+| Feature | Why We Must Keep It |
+|---------|---------------------|
+| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints |
+| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power  our scoring prevents routing to overheating GPUs |
+| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading |
+| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle |
+| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers |
+| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature  exposed via /metrics and SSE dashboard |
+| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular |
+
+---
+
+## 4. Migration Architecture
+
+### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room"
+
+- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI
+- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking
+
+### 4.2 Flow
+
+```
+Agent Request
+     
+     
+
+ LiteLLM Gateway (:4000)                      
+                                              
+ 1. Authenticate virtual key (sk-litellm-...) 
+ 2. Check key permissions (model access)      
+ 3. Check budget (per-key, per-user, per-team)
+ 4. Check team rate limits                    
+ 5. Log request metadata                      
+ 6. Forward to custom router as OpenAI-compat  
+    POST http://router:9000/v1/chat/completions
+    Headers: Authorization: Bearer *** 
+             X-LiteLLM-User: <user-id>         
+             X-LiteLLM-Team: <team-id>         
+             X-Session-Id: <session>           
+                                              
+ 7. On response: log spend, update budgets     
+ 8. If router returns 503 (GPU saturated):     
+    consult fallback chain, retry next model    
+ 9. Return response to agent                   
+
+               
+               
+
+ Custom Router (:9000)                         
+                                               
+ 1. Authenticate agent key (sk-syslog-...)     
+ 2. Hardware rate limit (per-tier RPM)         
+ 3. Content-based tier routing (for syslog-auto)
+    OR strict passthrough (for explicit models) 
+ 4. GPU slot availability (Redis counter)      
+ 5. GPU health check (sidecar)                 
+ 6. Agent spread logic (select_best_gpu)       
+ 7. Queue if saturated (with timeout)          
+ 8. Forward to selected llama.cpp GPU          
+ 9. Track context window, set compaction header
+ 10. Record performance metrics                
+ 11. Return response (with routing metadata)   
+
+               
+               
+
+ llama.cpp GPU (:8080)                         
+
+```
+
+### 4.3 LiteLLM Config (`config.yaml`)
+
+```yaml
+general_settings:
+  master_key: os.environ/LITELLM_MASTER_KEY
+  database_url: postgresql://litellm:***@postgres:5432/litellm
+  store_model_in_db: true
+
+model_list:
+  # Content-based auto-routing (router picks GPU via 5-tier analysis)
+  - model_name: syslog-auto
+    litellm_params:
+      model: openai/syslog-auto
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+      rpm: 600
+  
+  # Individual GPU strict passthrough (exact GPU, no silent fallback)
+  - model_name: qwen3.6-35B-A3B
+    litellm_params:
+      model: openai/qwen3.6-35B-A3B
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+  
+  - model_name: qwen3.6-27B-code
+    litellm_params:
+      model: openai/qwen3.6-27B-code
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+  - model_name: gemma-4-12b
+    litellm_params:
+      model: openai/gemma-4-12b
+      api_base: http://router:9000/v1
+      api_key: os.environ/ROUTER_API_KEY
+
+# Guardrails: Pre-call and post-call content moderation
+guardrails:
+  - guardrail_name: "input-moderation"
+    litellm_params:
+      guardrail: openai_moderation
+      mode: "pre_call"
+
+  - guardrail_name: "output-moderation"
+    litellm_params:
+      guardrail: openai_moderation
+      mode: "post_call"
+
+  - guardrail_name: "harmful-content-filter"
+    litellm_params:
+      guardrail: litellm_content_filter
+      mode: "pre_call"
+      categories:
+        - category: "harmful_self_harm"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+        - category: "harmful_violence"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+        - category: "harmful_illegal_weapons"
+          enabled: true
+          action: "BLOCK"
+          severity_threshold: "medium"
+
+litellm_settings:
+  num_retries: 0                     # Disabled  our router handles retry
+  request_timeout: 600               # Match our 10-min llama-server timeout
+  set_verbose: true
+  failure_callback: ["prometheus"]   # Optional: export to Prometheus
+
+router_settings:
+  routing_strategy: "usage-based-routing"  # For external models only
+  enable_loadbalancing_on_proxy: false     # Disable LiteLLM's internal LB
+  allowed_fails: 100                       # Router returns 503 on saturated GPUs  cooldown disabled
+  # Fallback chains: LiteLLM retries down the chain when router returns saturated
+  # This gives accurate per-model metrics because router no longer silently reroutes
+  fallbacks:
+    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
+    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
+    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
+
+# Cost tracking: map model names to per-token pricing for spend tracking
+litellm_settings:
+  model_cost:
+    qwen3.6-35B-A3B:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    qwen3.6-27B-code:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    gemma-4-12b:
+      input_cost_per_token: 0.0
+      output_cost_per_token: 0.0
+    # For internal cost allocation, set symbolic rates:
+    # e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
+```
+
+### 4.4 Router Modifications
+
+To accommodate LiteLLM, `router-fixed.py` requires the following updates:
+
+1. **Strict passthrough for explicit models** (DEPLOYED):
+   ```python
+   # In route(), the explicit model section changed from silent fallback to strict:
+   req = rd.get("model","auto")
+   if req != "auto":
+       # STRICT MODE: no silent fallback  LiteLLM handles failover chains.
+       # This keeps per-model metrics accurate. Returns saturated if busy.
+       target = req if req in avail else avail[0]
+       if req not in avail:
+           return {"model": req, "reason": "explicit_unavailable", "saturated": True}
+       if is_gpu_busy(target):
+           return {"model": target, "reason": "explicit_saturated", "saturated": True}
+       return {"model": target, "reason": "explicit"}
+   ```
+
+2. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent  already works)
+
+3. **New endpoint for health passthrough**: `GET /v1/models` already works
+
+4. **Keep ALL routing logic**: No changes to `select_best_gpu()`, `check_gpu_health()`, slot management, etc. Content-based routing for `syslog-auto` is fully intact.
+
+5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs
+   ```python
+   resp.headers["X-Usage-Tokens"] = json.dumps({
+       "prompt_tokens": prompt_tokens,
+       "completion_tokens": completion_tokens,
+       "model": model
+   })
+   ```
+
+### 4.5 Router Logic Refinements
+
+**GPU Health Scoring (Updated):**
+We are updating the scoring algorithm to include Power metrics:
+```python
+def gpu_health_score(model):
+    h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
+    if h.get("status") == "down":
+        return 999  # never pick down GPUs
+    vram_pct = h.get("vram_pct") or 50
+    temp_c = h.get("temp_c") or 50
+    power_w = h.get("power_w") or 50
+    active = gpu_active_count(model)
+    max_c = GPU_MAX_CONCURRENT.get(model, 1)
+    load_pct = (active / max_c) * 100 if max_c > 0 else 0
+    # Score: lower = better
+    score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2))
+    return score
+```
+
+---
+
+## 5. Deployment Plan (4 Phases)  Zero-Downtime Strategy
+
+### Phase 0: Infrastructure Prep (Current Week)  Zero Downtime
+
+**Goal:** Prepare CT 116 infrastructure without affecting running agents.
+
+**Tasks:**
+1. **Set up DNS split-horizon on CT 116**
+   ```bash
+   echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
+   ```
+
+2. **Deploy Postgres container** alongside existing services
+
+3. **Replace LiteLLM config** with production config.yaml (see 4.3)
+   - All 4 models  `http://router:9000/v1`
+   - Fallback chains for explicit models
+   - Guardrails (pre-call, post-call, content filter)
+   - `allowed_fails: 100` (router returns 503 on saturated)
+   - `num_retries: 0` (LiteLLM retries handled by fallback chains)
+
+4. **Deploy custom_sso.py** for Authentik OIDC integration
+
+5. **Restart LiteLLM container** with new config
+
+6. **Verify internal routing**
+
+**Verification Checklist:**
+- [ ] DNS resolution: `getent hosts auth.sysloggh.net`  192.168.68.11
+- [ ] Postgres container healthy
+- [ ] LiteLLM `/health` returns 200
+- [ ] LiteLLM  Router pass-through returns valid chat completion
+- [ ] GPU health metrics unaffected
+- [ ] Explicit model request returns saturated (not silently rerouted) when GPU busy
+
+### Phase 1: Shadow Mode (Week 1)  Zero Risk, Zero Downtime
+
+**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. **Agents continue using :9000 directly.**
+
+**Tasks:**
+1. Create virtual keys for test agents via LiteLLM UI
+2. Verify pass-through works for all 4 models
+3. Validate fallback chains: saturate MoE  confirm LiteLLM retries Dense  confirm VLM
+4. Run 24-hour shadow: monitor LiteLLM spend logs vs router metrics
+5. Verify GPU health metrics unaffected
+6. Check guardrails not generating false positives
+
+### Phase 2: Cutover (Week 2)  Gradual Agent Migration
+
+**Goal:** Move agents one-by-one to LiteLLM endpoint.
+
+**Tasks:**
+1. Migrate API keys to LiteLLM virtual keys
+2. Create teams: "Core Agents" (enterprise), "Dev Agents" (professional)
+3. Update agent configs one at a time: `OPENAI_API_BASE`  `:4000`
+4. Test each agent individually
+5. Enable SSO via Authentik + custom_sso.py
+6. Keep router :9000 as emergency fallback for 48 hours
+
+### Phase 3: Production Hardening (Week 3+)
+
+**Goal:** Lock down, optimize, monitor.
+
+**Tasks:**
+1. Remove deprecated router endpoints (after all agents migrated)
+2. Add LiteLLM observability (Prometheus, Slack/email alerts)
+3. Enable LiteLLM caching (shared Redis)
+4. Add external model fallbacks for client-facing services
+5. Router slim-down: keep routing/slots/health/perf, remove key management
+6. Multi-tenancy setup for client-facing inference services
+
+---
+
+## 6. Nginx Configuration (with Authentik OIDC Forward Auth)
+
+```nginx
+# OLD (remove)
+# location /admin/ {
+#     proxy_pass http://127.0.0.1:9000/admin/;
+# }
+
+# === Authentik auth subrequest endpoint ===
+location /authentik/auth {
+    internal;
+    proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
+    proxy_pass_request_body off;
+    proxy_set_header Content-Length "";
+    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+}
+
+# === LiteLLM Admin UI  Authentik-protected ===
+location /ui/ {
+    auth_request /authentik/auth;
+    auth_request_set $auth_user $upstream_http_x_authentik_username;
+    auth_request_set $auth_email $upstream_http_x_authentik_email;
+
+    proxy_set_header X-Authentik-Username $auth_user;
+    proxy_set_header X-Authentik-Email $auth_email;
+
+    proxy_pass http://127.0.0.1:4000/ui/;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection "upgrade";
+}
+
+# === LiteLLM SSO callback ===
+location /sso/callback {
+    proxy_pass http://127.0.0.1:4000/sso/callback;
+    proxy_set_header Host $host;
+}
+
+# === API endpoint  Bearer token auth ===
+location /v1/ {
+    proxy_pass http://127.0.0.1:4000/v1/;
+    proxy_set_header Host $host;
+    proxy_read_timeout 600s;
+    error_page 502 = @router_fallback;
+}
+
+location @router_fallback {
+    proxy_pass http://127.0.0.1:9000/v1/;
+    proxy_set_header Host $host;
+}
+
+# === Key management API ===
+location /key/ {
+    proxy_pass http://127.0.0.1:4000/key/;
+    proxy_set_header Host $host;
+    proxy_set_header Authorization $http_authorization;
+}
+
+# Keep router metrics accessible (not behind LiteLLM)
+location /router/ {
+    proxy_pass http://127.0.0.1:9000/;
+}
+
+location /health {
+    proxy_pass http://127.0.0.1:4000/health;
+}
+```
+
+---
+
+## 7. Docker Compose (`docker-compose.yml` on CT 116)
+
+```yaml
+services:
+  # Layer 1: LiteLLM Gateway (Policy & Admin)
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    network_mode: "host"
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    volumes:
+      - ./config.yaml:/app/config.yaml:ro
+      - ./custom_sso.py:/app/custom_sso.py:ro
+    environment:
+      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
+      - DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm
+      - STORE_MODEL_IN_DB=True
+      - ROUTER_API_KEY=${ROUTER_API_KEY}
+      - OPENAI_API_KEY=***
+      - ANTHROPIC_API_KEY=${ANTH...KEY}
+      - PROXY_BASE_URL=https://litellm.sysloggh.net
+    command:
+      - --config
+      - /app/config.yaml
+      - --port
+      - "4000"
+    depends_on:
+      postgres:
+        condition: service_healthy
+    restart: unless-stopped
+
+  postgres:
+    image: postgres:16-alpine
+    network_mode: "host"
+    environment:
+      - POSTGRES_DB=litellm
+      - POSTGRES_USER=litellm
+      - POSTGRES_PASSWORD=${POST...}
+    volumes:
+      - pgdata:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm"]
+      interval: 5s
+      timeout: 3s
+      retries: 5
+    restart: unless-stopped
+
+  nginx:
+    image: nginx:alpine
+    extra_hosts:
+      - "auth.sysloggh.net:192.168.68.11"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+    ports:
+      - "80:80"
+    restart: unless-stopped
+
+volumes:
+  pgdata:
+```
+
+---
+
+## 8. Risk Mitigation
+
+| Risk | Mitigation |
+|------|------------|
+| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable |
+| LiteLLM down = all agents down | NGINX fallback to router :9000 direct (see 6) |
+| Explicit GPU saturated  no fallback available | LiteLLM fallback chains try all 3 GPUs in order before failing |
+| Fallback chain masking real GPU failures | Router returns `saturated: true` only for capacity, `down` returns different error |
+| Key sync drift | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` |
+| Spend tracking inaccurate for local GPUs | `model_cost` per GPU with $0 rate; optional symbolic pricing for internal billing |
+| Double rate limiting | Intentional: LiteLLM for per-user caps, Router for hardware protection |
+| PostgreSQL failure | LiteLLM can run with SQLite fallback; UI features degrade |
+| Per-model metrics accuracy with syslog-auto | `syslog-auto` is opaque by design (content-based routing). Explicit models are accurate. Use explicit models for per-GPU billing. |
+
+---
+
+## 9. Success Metrics
+
+| Metric | Before | After |
+|--------|--------|-------|
+| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy |
+| Spend visibility | None | Per-agent, per-team, per-model $ tracking |
+| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped |
+| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key |
+| Admin UX | curl + JSON responses | Visual dashboard, graphs, search |
+| Audit trail | Router logs (stdout only) | Database-backed with UI search |
+| SSO | None | Authentik OIDC |
+| Budget enforcement | None | Automatic: key suspended at $limit |
+| GPU failover | Silent (inaccurate metrics) | Explicit (LiteLLM fallback chains, per-model logs) |
+| GPU routing intelligence | Full (unchanged) | Full (unchanged) |
+
+---
+
+## 10. Migration Commands (Quick Reference)
+
+```bash
+# On CT 116 (SSH via minipve: pct exec 116 bash):
+
+# Phase 0: Infrastructure Prep
+echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
+
+cd /opt/litellm
+docker compose up -d postgres
+# Replace config.yaml with production version (see 4.3)
+docker compose restart litellm
+
+# Verify
+curl http://127.0.0.1:4000/health
+curl -X POST http://127.0.0.1:4000/v1/chat/completions \
+  -H "Authorization: Bearer ***" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
+```
+
+---
+
+## 11. GitOps Workflow
+
+- `main`  production-ready code
+- `feature/litellm-migration`  current development branch
+
+**Conventional Commits:**
+```
+feat(plan): add fallback chains and strict passthrough for model identity gap
+fix(router): strict passthrough for explicit models  no silent rerouting
+feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC
+docs: LiteLLM migration plan  two-layer architecture with model identity gap analysis
+```
+
+---
+
+## 12. Zero-Downtime Migration Strategy
+
+**Per-Agent Cutover (<2 minutes):**
+1. Create LiteLLM virtual key in UI
+2. Update agent's `OPENAI_API_BASE` to `:4000`
+3. Verify routing works
+4. Monitor LiteLLM logs for errors
+
+**Global Rollback:**
+1. If LiteLLM :4000 fails, revert all agents to `:9000`
+2. NGINX `router_fallback` handles automatic failover
+3. Monitor GPU metrics for health checks
+
+---
+
+## Appendix A: Model Identity Gap Analysis (RESOLVED)
+
+### Problem Identified (Abiba, June 14)
+
+The original architecture had a metrics accuracy gap: when an agent requested `qwen3.6-35B-A3B` and MoE was busy, the router silently rerouted to Dense. LiteLLM logged it as MoE usage, corrupting per-model spend/usage tracking.
+
+### Root Cause
+
+```python
+# OLD router code (router-fixed.py):
+if is_gpu_busy(target) and req in allowed:
+    alts = [m for m in avail if m != target and m in allowed]
+    if alts:
+        alt = select_best_gpu(alts, "explicit", agent)
+        if alt: return alt   # silently changed GPU
+```
+
+### Resolution: Strict Passthrough + LiteLLM Fallback Chains
+
+Two changes deployed:
+
+**1. Router strict passthrough:**
+```python
+# NEW: strict mode  no silent fallback
+if req != "auto":
+    target = req if req in avail else avail[0]
+    if req not in avail:
+        return {"model": req, "reason": "explicit_unavailable", "saturated": True}
+    if is_gpu_busy(target):
+        return {"model": target, "reason": "explicit_saturated", "saturated": True}
+    return {"model": target, "reason": "explicit"}
+```
+
+**2. LiteLLM fallback chains (in config.yaml):**
+```yaml
+router_settings:
+  allowed_fails: 100
+  fallbacks:
+    - qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
+    - qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
+    - gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
+```
+
+### Result
+
+| Scenario | Before | After |
+|----------|--------|-------|
+| Agent asks for MoE, MoE available | MoE used, metrics OK | MoE used, metrics OK |
+| Agent asks for MoE, MoE busy | Router  Dense silently, metrics WRONG | Router  503, LiteLLM  Dense, metrics show BOTH attempts |
+| Agent uses syslog-auto | Router picks GPU, LiteLLM sees opaque | Same (syslog-auto is opaque by design) |
+| All 3 GPUs saturated | Router queues (30s), then 503 | Same, LiteLLM sees 503 after fallback chain exhausted |
+
+---
+
+## Appendix B: LiteLLM Virtual Key Migration
+
+| Agent | Old Key | New LiteLLM Key | Tier | Budget |
+|-------|---------|-----------------|------|--------|
+| Abiba | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
+| Mumuni | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
+| Tanko | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
+| Kagenz0 | sk-***-*** | sk-litellm-*** | professional | $500 |
+| Koby | sk-***-*** | sk-litellm-*** | professional | $500 |
+| Koonimo | sk-***-*** | sk-litellm-*** | professional | $500 |
+
+---
+
+## Appendix C: Authentik SSO Integration
+
+**Authentik Provider Setup:**
+1. Create OAuth2 application in Authentik
+2. Set redirect URI: `http://<CT-116-IP>/sso/callback`
+3. Configure `client_id` and `client_secret`
+4. Mount `custom_sso.py` to LiteLLM container
+5. Update config.yaml with provider details
+
+---
+
+## Appendix D: Prometheus Monitoring
+
+**Metrics Export:**
+- LiteLLM metrics  `http://127.0.0.1:4000/metrics`
+- Router metrics  `http://127.0.0.1:9000/metrics`
+- GPU health metrics  `http://127.0.0.1:9000/metrics/gpu`
+
+**Alerts:**
+- GPU health score > 70  alert
+- Circuit breaker trip  alert
+- LiteLLM spend > $100/day  alert
+- LiteLLM latency > 1000ms  alert

backups/20260602_103344/dashboard.py.bak

@@ -0,0 +1,356 @@
+"""SyslogAI Harness Dashboard — Modern Design."""
+import os, json, time, queue, threading
+import requests
+from flask import Flask, request, render_template_string, Response, stream_with_context
+
+ROUTER_METRICS = os.environ.get("ROUTER_METRICS_URL", "http://router:9000/metrics")
+app = Flask(__name__)
+sse_subscribers = []; sse_lock = threading.Lock()
+
+def fetch_state():
+    try:
+        r = requests.get(ROUTER_METRICS, timeout=5)
+        if r.status_code == 200: return r.json()
+    except Exception: pass
+    return {"gpus":[],"route_counts":{},"agent_counts":{},"recent":[],"timestamp":time.time()}
+
+def broadcast_loop():
+    while True:
+        time.sleep(3)
+        data = fetch_state(); payload = json.dumps(data)
+        with sse_lock:
+            dead = [q for q in sse_subscribers if not q.put(payload)]
+            for q in dead: sse_subscribers.remove(q)
+threading.Thread(target=broadcast_loop, daemon=True).start()
+
+DASHBOARD_HTML = r"""<!DOCTYPE html>
+<html lang="en" data-bs-theme="dark">
+<head>
+<meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>SyslogAI Harness</title>
+<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet">
+<style>
+body { background: #0b0f17; color: #bcc3cd; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', system-ui, sans-serif; padding: 20px 24px; }
+.card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; height: 100%; }
+.stat-card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; padding: 18px 20px; text-align: center; }
+.stat-value { font-size: 28px; font-weight: 700; line-height: 1.1; }
+.stat-label { font-size: 11px; text-transform: uppercase; letter-spacing: 0.6px; color: #64748b; margin-top: 4px; }
+.gpu-card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; padding: 16px 18px; height: 100%; }
+.gpu-card .title { font-size: 13px; font-weight: 600; color: #e2e8f0; margin-bottom: 12px; display: flex; align-items: center; gap: 8px; }
+.gpu-card .status-dot { width: 8px; height: 8px; border-radius: 50%; flex-shrink: 0; }
+.gpu-card .row-metric { display: flex; justify-content: space-between; font-size: 12px; padding: 2px 0; }
+.gpu-card .row-metric .lbl { color: #64748b; }
+.gpu-card .row-metric .val { color: #e2e8f0; font-variant-numeric: tabular-nums; }
+.gpu-card .slot-bar { display: flex; gap: 3px; margin-top: 8px; }
+.gpu-card .slot-bar .s { flex: 1; height: 5px; border-radius: 2px; background: #1e293b; }
+.gpu-card .slot-bar .s.active { background: #38bdf8; }
+.chart-card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; padding: 16px 18px; height: 100%; display: flex; flex-direction: column; }
+.chart-card .title { font-size: 13px; font-weight: 600; color: #e2e8f0; margin-bottom: 12px; }
+.bar-row { margin-bottom: 8px; }
+.bar-label { display: flex; justify-content: space-between; font-size: 11px; margin-bottom: 3px; color: #64748b; }
+.bar-label .name { color: #cbd5e1; }
+.bar-track { height: 5px; background: #1e293b; border-radius: 3px; overflow: hidden; }
+.bar-fill { height: 100%; border-radius: 3px; transition: width 0.6s ease; }
+.table-custom { font-size: 11px; margin: 0; }
+.table-custom th { color: #64748b; font-weight: 500; font-size: 10px; text-transform: uppercase; border-color: #1e293b; padding: 8px 10px; }
+.table-custom td { color: #94a3b8; border-color: rgba(30,41,59,0.5); padding: 6px 10px; }
+.agent-badge { font-size: 10px; padding: 2px 7px; border-radius: 8px; font-weight: 600; }
+.btn-sm-period { font-size: 10px; padding: 3px 10px; border-radius: 6px; border: 1px solid #1e293b; color: #64748b; background: transparent; cursor: pointer; }
+.btn-sm-period.active { background: #1d4ed8; color: #fff; border-color: #1d4ed8; }
+.ring-label { font-size: 22px; font-weight: 700; }
+.ring-sublabel { font-size: 10px; color: #64748b; }
+</style>
+</head>
+<body>
+
+<!-- HEADER -->
+<div class="d-flex justify-content-between align-items-center mb-4">
+  <div>
+    <h5 class="mb-0 text-white fw-bold">&#x26A1; SyslogAI Harness</h5>
+    <div class="small text-secondary" id="live-indicator">
+      <span class="status-dot" id="live-dot" style="width:6px;height:6px;border-radius:50%;display:inline-block;background:#22c55e;animation:pulse 2s infinite"></span>
+      <span id="connection-status">live</span> &middot; <span id="update-time"></span>
+    </div>
+  </div>
+  <div class="d-flex gap-2">
+    <div class="stat-card" style="min-width:100px"><div class="stat-value text-info" id="kpi-total">0</div><div class="stat-label">Requests</div></div>
+    <div class="stat-card" style="min-width:100px"><div class="stat-value text-warning" id="kpi-active">0</div><div class="stat-label">Active</div></div>
+    <div class="stat-card" style="min-width:100px"><div class="stat-value" style="color:#a78bfa" id="kpi-agents">0</div><div class="stat-label">Agents</div></div>
+  </div>
+</div>
+
+<div class="row g-3 align-items-stretch">
+  <!-- ROW 1: Usage Chart (8) + GPU Metrics (4) -->
+  <div class="col-md-8"><div class="chart-card"><div class="title d-flex justify-content-between align-items-center">
+    <span>Usage Over Time</span>
+    <div class="d-flex gap-1">
+      <button class="btn-sm-period active" onclick="switchPeriod('day')">24h</button>
+      <button class="btn-sm-period" onclick="switchPeriod('week')">7d</button>
+      <button class="btn-sm-period" onclick="switchPeriod('month')">30d</button>
+    </div>
+  </div><div id="timeseries-chart" style="height:150px"></div><div id="timeseries-legend" class="d-flex justify-content-center gap-3 mt-2 flex-wrap small"></div></div></div>
+  <div class="col-md-4"><div class="chart-card"><div class="title">GPU Metrics</div><div id="gpu-metrics-card"></div></div></div>
+
+  <!-- ROW 2: 3 GPU Cards -->
+  <div class="col-md-4"><div class="gpu-card" id="gpu-moe"><div class="text-secondary small">Loading...</div></div></div>
+  <div class="col-md-4"><div class="gpu-card" id="gpu-dense"><div class="text-secondary small">Loading...</div></div></div>
+  <div class="col-md-4"><div class="gpu-card" id="gpu-light"><div class="text-secondary small">Loading...</div></div></div>
+
+  <!-- ROW 3: Queue + Model + Agent -->
+  <div class="col-md-4"><div class="chart-card"><div class="title">Queue Status</div><div class="text-center" id="queue-viz"></div></div></div>
+  <div class="col-md-4"><div class="chart-card"><div class="title">Model Distribution</div><div id="route-bars"></div></div></div>
+  <div class="col-md-4"><div class="chart-card"><div class="title">Agent Activity</div><div id="agent-bars"></div></div></div>
+
+  <!-- ROW 4: Performance Analytics -->
+  <div class="col-12 mb-2"><div class="d-flex align-items-center gap-2"><span class="fw-bold text-white" style="font-size:14px">&#x1F4CA; Performance Analytics</span>
+    <div class="d-flex gap-1 ms-auto">
+      <button class="btn-sm-period active" onclick="switchPerfWindow('1')">1h</button>
+      <button class="btn-sm-period" onclick="switchPerfWindow('24')">24h</button>
+    </div>
+  </div></div>
+  <div class="col-md-6"><div class="chart-card"><div class="title">Latency — P50 / P95 / P99 (ms)</div><div id="perf-latency"></div></div></div>
+  <div class="col-md-6"><div class="chart-card"><div class="title">Throughput — Tokens / sec</div><div id="perf-throughput"></div></div></div>
+  <div class="col-md-6"><div class="chart-card"><div class="title">Routing Effectiveness — by Reason</div><div id="perf-reasons"></div></div></div>
+  <div class="col-md-6"><div class="chart-card"><div class="title">Agent Performance</div><div id="perf-agents"></div></div></div>
+
+  <!-- ROW 5: Latency vs Context Scatter -->
+  <div class="col-12"><div class="chart-card"><div class="title d-flex justify-content-between align-items-center">
+    <span>Latency vs Prompt Size — by Model</span>
+    <div class="d-flex gap-2">
+      <select id="scatter-model" onchange="loadScatter()" style="font-size:10px;background:#1e293b;color:#94a3b8;border:1px solid #334155;border-radius:4px;padding:2px 6px">
+        <option value="all">All Models</option>
+        <option value="qwen3.5-9b-vlm">9B VLM</option>
+        <option value="qwen3.6-27B-code">27B Dense</option>
+        <option value="qwen3.6-35B-A3B">35B MoE</option>
+      </select>
+    </div>
+  </div><div id="scatter-plot" style="height:200px;position:relative"></div><div id="scatter-legend" class="d-flex justify-content-center gap-3 mt-2 flex-wrap small"></div></div></div>
+
+  <!-- ROW 6: Live Stream -->
+  <div class="col-12"><div class="chart-card"><div class="title">Live Stream</div>
+    <div class="table-responsive"><table class="table table-custom mb-0">
+      <thead><tr><th>Time</th><th>Agent</th><th>Model</th><th>Reason</th><th>Tier</th></tr></thead>
+      <tbody id="route-tbody"></tbody>
+    </table></div>
+  </div></div>
+</div>
+
+<script>
+var MC={'qwen3.5-9b-vlm':'#22c55e','qwen3.6-27B-code':'#f59e0b','qwen3.6-35B-A3B':'#a78bfa'};
+var ML={'qwen3.5-9b-vlm':'Qwen3.5 9B VLM','qwen3.6-27B-code':'Qwen Code','qwen3.6-35B-A3B':'Qwen MoE'};
+var GL={'qwen3.6-35B-A3B':'MoE - Strix Halo','qwen3.6-27B-code':'Dense - RTX 3090','qwen3.5-9b-vlm':'VLM - RTX 5070'};
+function $(id){return document.getElementById(id);}
+
+function render(data){
+if(!data||!data.gpus)return;
+var t=Object.values(data.route_counts||{}).reduce((a,b)=>a+b,0);
+var ta=0,tm=0;data.gpus.forEach(function(g){ta+=(g.active_requests||0);tm+=(g.max_concurrent||1)});
+$('kpi-total').textContent=t;$('kpi-active').textContent=ta+'/'+tm;$('kpi-agents').textContent=Object.keys(data.agent_counts||{}).length;
+$('update-time').textContent=new Date().toLocaleTimeString();
+var ids={'qwen3.6-35B-A3B':'gpu-moe','qwen3.6-27B-code':'gpu-dense','qwen3.5-9b-vlm':'gpu-light'};
+data.gpus.forEach(function(g){
+var el=$(ids[g.id]);if(!el)return;
+var a=g.active_requests||0,mx=g.max_concurrent||1;
+var sc=g.status==='healthy'?'#22c55e':g.status==='saturated'?'#f59e0b':'#ef4444';
+var ss=g.status==='healthy'?'Online':g.status==='saturated'?'Busy':'Offline';
+var slots='';for(var i=0;i<mx;i++)slots+='<span class=\"s'+(i<a?' active':'')+'\"></span>';
+var h='<div class=\"title\"><span class=\"status-dot\" style=\"background:'+sc+'\"></span>'+GL[g.id]+'<span class=\"ms-auto small\" style=\"color:'+sc+'\">'+ss+'</span></div>';
+h+='<div class=\"row-metric\"><span class=\"lbl\">VRAM</span><span class=\"val\">'+g.vram_used_mb+' / '+g.vram_total_mb+' MB</span></div>';
+h+='<div class=\"row-metric\"><span class=\"lbl\">Utilization</span><span class=\"val\">'+g.gpu_util_pct+'%</span></div>';
+h+='<div class=\"row-metric\"><span class=\"lbl\">Temperature</span><span class=\"val\" style=\"color:'+(g.temp_c>85?'#ef4444':g.temp_c>70?'#f59e0b':'#22c55e')+'\">'+g.temp_c+'C</span></div>';
+if(g.power_w)h+='<div class=\"row-metric\"><span class=\"lbl\">Power</span><span class=\"val\">'+g.power_w+'W'+(g.power_limit_w?'/'+g.power_limit_w+'W':'')+'</span></div>';
+h+='<div class=\"row-metric\"><span class=\"lbl\">Slots</span><span class=\"val\" style=\"color:'+(a>=mx?'#ef4444':'#e2e8f0')+'\">'+a+' / '+mx+'</span></div>';
+h+='<div class=\"slot-bar\">'+slots+'</div>';el.innerHTML=h;
+});
+renderQueue(data);renderGPUMetrics(data);
+var rc=data.route_counts||{},mr=Math.max(1,...Object.values(rc));
+$('route-bars').innerHTML=Object.entries(rc).length?Object.entries(rc).sort((a,b)=>b[1]-a[1]).map(function(e){var m=e[0],c=e[1];return'<div class=\"bar-row\"><div class=\"bar-label\"><span class=\"name\">'+(ML[m]||m)+'</span><span>'+c+' ('+(t?Math.round(c/t*100):0)+'%)</span></div><div class=\"bar-track\"><div class=\"bar-fill\" style=\"width:'+(c/mr*100)+'%;background:'+(MC[m]||'#38bdf8')+'\"></div></div></div>';}).join(''):'<div class=\"text-secondary small\">-</div>';
+var ac=data.agent_counts||{},ma=Math.max(1,...Object.values(ac));
+$('agent-bars').innerHTML=Object.entries(ac).length?Object.entries(ac).sort((a,b)=>b[1]-a[1]).map(function(e){return'<div class=\"bar-row\"><div class=\"bar-label\"><span class=\"name\">'+e[0]+'</span><span>'+e[1]+'</span></div><div class=\"bar-track\"><div class=\"bar-fill\" style=\"width:'+(e[1]/ma*100)+'%;background:#38bdf8\"></div></div></div>';}).join(''):'<div class=\"text-secondary small\">-</div>';
+var recent=data.recent||[];
+$('route-tbody').innerHTML=recent.length?recent.slice(0,20).map(function(r){var d=new Date(r.ts*1000),ag=r.agent||'?';return'<tr><td class=\"text-secondary\">'+d.toLocaleTimeString()+'</td><td><span class=\"agent-badge\" style=\"background:rgba(56,189,248,0.12);color:#38bdf8\">'+ag+'</span></td><td>'+(ML[r.model]||r.model)+'</td><td class=\"text-secondary\">'+(r.reason||'')+'</td><td class=\"text-uppercase\" style=\"font-size:10px;color:'+(r.tier==='enterprise'?'#a78bfa':'#64748b')+'\">'+(r.tier||'')+'</td></tr>';}).join(''):'<tr><td colspan=\"5\" class=\"text-secondary\">Waiting...</td></tr>';
+}
+
+function renderQueue(data){
+var el=$('queue-viz');if(!el)return;
+var ta=0,tm=0;data.gpus.forEach(function(g){ta+=(g.active_requests||0);tm+=(g.max_concurrent||1)});
+var pct=tm>0?Math.round(ta/tm*100):0,st=pct>=100?'SATURATED':pct>=50?'BUSY':'IDLE';
+var sc=pct>=100?'#ef4444':pct>=50?'#f59e0b':'#22c55e';
+var circ=188.5,dash=(pct/100)*circ;
+var h='<div class=\"d-inline-block position-relative mb-2\"><svg width=\"72\" height=\"72\"><circle cx=\"36\" cy=\"36\" r=\"30\" fill=\"none\" stroke=\"#1e293b\" stroke-width=\"6\"/><circle cx=\"36\" cy=\"36\" r=\"30\" fill=\"none\" stroke=\"'+sc+'\" stroke-width=\"6\" stroke-dasharray=\"'+dash+' '+(circ-dash)+'\" stroke-linecap=\"round\" transform=\"rotate(-90 36 36)\"/></svg><div style=\"position:absolute;top:50%;left:50%;transform:translate(-50%,-50%);text-align:center\"><div class=\"ring-label\" style=\"color:'+sc+'\">'+ta+'</div><div class=\"ring-sublabel\">/ '+tm+' slots</div></div></div>';
+h+='<div class=\"fw-bold mb-2 small\" style=\"color:'+sc+'\">'+st+'</div>';
+var lb={'qwen3.6-35B-A3B':'MoE','qwen3.6-27B-code':'Dense','qwen3.5-9b-vlm':'VLM'};
+data.gpus.forEach(function(g){var a=g.active_requests||0,mx=g.max_concurrent||1,gp=mx>0?Math.round(a/mx*100):0;h+='<div class=\"d-flex align-items-center gap-2 mb-1 justify-content-center\"><span class=\"small\" style=\"min-width:32px;text-align:right;font-size:10px\">'+(lb[g.id]||g.id)+'</span><div style=\"flex:1;max-width:70px;height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+gp+'%;background:'+sc+';border-radius:2px\"></div></div><span class=\"small\" style=\"min-width:22px;font-size:10px\">'+a+'/'+mx+'</span></div>'});
+el.innerHTML=h;
+}
+
+function renderGPUMetrics(data){
+var el=$('gpu-metrics-card');if(!el)return;
+var lb={'qwen3.6-35B-A3B':'MoE','qwen3.6-27B-code':'Dense','qwen3.5-9b-vlm':'VLM'};
+var h='';data.gpus.forEach(function(g){
+var nm=lb[g.id]||g.id,tp=g.temp_c||0,ut=g.gpu_util_pct||0,pw=g.power_w||0,pl=g.power_limit_w||0;
+var tc=tp>85?'#ef4444':tp>70?'#f59e0b':'#22c55e',uc=ut>90?'#ef4444':ut>70?'#f59e0b':'#22c55e';
+h+='<div class=\"mb-3\"><div class=\"fw-bold small text-white-50 mb-1\">'+nm+'</div>';
+h+='<div class=\"d-flex align-items-center gap-2 mb-1\"><span class=\"small text-secondary\" style=\"min-width:30px\">T</span><div class=\"flex-grow-1\" style=\"height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+Math.min(tp,100)+'%;background:'+tc+';border-radius:2px\"></div></div><span class=\"small\" style=\"color:'+tc+';min-width:30px;text-align:right\">'+tp+'C</span></div>';
+h+='<div class=\"d-flex align-items-center gap-2 mb-1\"><span class=\"small text-secondary\" style=\"min-width:30px\">U</span><div class=\"flex-grow-1\" style=\"height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+ut+'%;background:'+uc+';border-radius:2px\"></div></div><span class=\"small\" style=\"color:'+uc+';min-width:30px;text-align:right\">'+ut+'%</span></div>';
+if(pw>0){var pp=pl>0?Math.round(pw/pl*100):0,pc=pp>90?'#ef4444':pp>70?'#f59e0b':'#22c55e';h+='<div class=\"d-flex align-items-center gap-2\"><span class=\"small text-secondary\" style=\"min-width:30px\">P</span><div class=\"flex-grow-1\" style=\"height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+pp+'%;background:'+pc+';border-radius:2px\"></div></div><span class=\"small\" style=\"color:'+pc+';min-width:30px;text-align:right\">'+pw+'W</span></div>';}
+h+='</div>';});
+el.innerHTML=h;
+}
+
+var cp='day';
+function switchPeriod(p){cp=p;document.querySelectorAll('.btn-sm-period').forEach(function(b){b.classList.remove('active')});event.target.classList.add('active');loadTS();}
+function loadTS(){fetch('/api/timeseries?period='+cp).then(function(r){return r.json()}).then(renderTS).catch(function(){})}
+function renderTS(d){
+var models=d.models||{},labels=d.labels||[];
+if(!labels.length)return;
+var cn=$('timeseries-chart'),lg=$('timeseries-legend'),mn=Object.keys(models);
+if(!mn.length){cn.innerHTML='<div class=\"text-secondary small text-center py-4\">-</div>';return;}
+var mv=1;for(var m in models)for(var i=0;i<models[m].length;i++)if(models[m][i]>mv)mv=models[m][i];mv=Math.ceil(mv*1.15)||1;
+var W=labels.length>1?100/(labels.length-1):100,H=130;
+var paths='';for(var mi=0;mi<mn.length;mi++){var m=mn[mi],vals=models[m]||[],d='';for(var i=0;i<vals.length;i++){var x=i*W,y=H-(vals[i]/mv)*H;d+=(i===0?'M':'L')+x.toFixed(1)+','+y.toFixed(1)+' ';}paths+='<path d=\"'+d+'\" fill=\"none\" stroke=\"'+(MC[m]||'#38bdf8')+'\" stroke-width=\"2\" stroke-linecap=\"round\" opacity=\"0.8\"/>';}
+var grid='';for(var g=0;g<=4;g++){var y=(g/4)*H;grid+='<line x1=\"0\" y1=\"'+y.toFixed(1)+'\" x2=\"100\" y2=\"'+y.toFixed(1)+'\" stroke=\"#1e293b\" stroke-width=\"1\"/>';}
+cn.innerHTML='<svg viewBox=\"0 0 100 '+(H+16)+'\" style=\"width:100%;height:'+(H+20)+'px;display:block\" preserveAspectRatio=\"none\">'+grid+paths+'</svg>';
+lg.innerHTML=mn.map(function(m){return'<span class=\"d-flex align-items-center gap-1\"><svg width=\"14\" height=\"8\"><line x1=\"0\" y1=\"4\" x2=\"14\" y2=\"4\" stroke=\"'+(MC[m]||'#38bdf8')+'\" stroke-width=\"2\"/></svg>'+(ML[m]||m)+'</span>';}).join('');
+}
+var perfWindow='24';
+function switchPerfWindow(w){perfWindow=w;document.querySelectorAll('.btn-sm-period').forEach(function(b,i){if(i>=4)b.classList.toggle('active',b.textContent.trim().replace('h','')===w)});loadPerf();}
+function loadPerf(){fetch('/api/performance?window='+perfWindow).then(function(r){return r.json()}).then(renderPerf).catch(function(){})}
+function renderPerf(d){
+var models=d.models||[],reasons=d.reasons||[],agents=d.agents||[],sum=d.summary||{};
+// Latency bars: p50/p95/p99 per model
+var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','qwen3.5-9b-vlm':'9B VLM'};
+var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','qwen3.5-9b-vlm':'#22c55e'};
+if(!models.length){$('perf-latency').innerHTML='<div class="text-secondary small text-center py-4">Accumulating data...</div>';return;}
+var maxLat=Math.max(...models.map(function(m){return m.latency.p99||0}),1);
+var latHTML=models.map(function(m){
+var l=m.latency||{},p50=l.p50||0,p95=l.p95||0,p99=l.p99||0,c=mcol[m.model]||'#38bdf8';
+return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span class="text-secondary">'+m.count+' reqs</span></div>'+
+'<div class="d-flex align-items-center gap-2 mb-1"><span class="text-secondary" style="min-width:28px">p50</span><div class="flex-grow-1" style="height:14px;background:#1e293b;border-radius:4px;overflow:hidden;position:relative"><div style="position:absolute;left:0;top:0;height:100%;width:'+(p50/maxLat*100)+'%;background:'+c+';opacity:0.3;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p95/maxLat*100)+'%;background:'+c+';opacity:0.5;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p99/maxLat*100)+'%;background:'+c+';border-radius:4px"></div></div><span style="color:'+c+';min-width:48px;text-align:right;font-variant-numeric:tabular-nums">'+p99+'ms</span></div>'+
+'<div class="d-flex gap-3" style="font-size:10px;color:#64748b;padding-left:32px"><span>p50: '+p50+'ms</span><span>p95: '+p95+'ms</span><span>p99: '+p99+'ms</span></div></div>';
+}).join('');
+$('perf-latency').innerHTML=latHTML;
+// Throughput comparison
+var maxTps=Math.max(...models.map(function(m){return m.throughput.avg_tokens_per_sec||0}),1);
+var tpsHTML=models.map(function(m){
+var t=m.throughput||{},avg=t.avg_tokens_per_sec||0,p50=t.p50||0,c=mcol[m.model]||'#38bdf8';
+var isAllStreaming = avg===0 && p50===0;
+if(isAllStreaming){
+return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:#64748b;font-style:italic">streaming only</span></div><div class="text-secondary" style="font-size:10px">t/s available for non-streaming requests only</div></div>';
+}
+return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:'+c+'" class="fw-bold">'+avg+' tok/s</span></div>'+
+'<div class="d-flex align-items-center gap-2"><span class="text-secondary" style="min-width:28px">avg</span><div class="flex-grow-1" style="height:6px;background:#1e293b;border-radius:3px;overflow:hidden"><div style="height:100%;width:'+(Math.max(avg/maxTps*100,6))+'%;background:'+c+';border-radius:3px"></div></div><span class="small" style="color:'+c+';min-width:54px;text-align:right">'+avg+' tok/s</span></div>'+
+'<div class="d-flex align-items-center gap-2 mt-1"><span class="text-secondary" style="min-width:28px;font-size:10px">p50</span><div class="flex-grow-1" style="height:4px;background:#1e293b;border-radius:2px;overflow:hidden"><div style="height:100%;width:'+(Math.max(p50/maxTps*100,4))+'%;background:'+c+';opacity:0.5;border-radius:2px"></div></div><span style="font-size:10px;color:#64748b">'+p50+' tok/s</span></div></div>';
+}).join('');
+$('perf-throughput').innerHTML=tpsHTML;
+// Routing reasons table
+if(reasons.length){
+var rHTML='<table class="table table-custom mb-0"><thead><tr><th>Reason</th><th>Count</th><th>Avg Lat</th><th>P95 Lat</th></tr></thead><tbody>';
+reasons.forEach(function(r){rHTML+='<tr><td>'+r.reason+'</td><td>'+r.count+'</td><td>'+r.avg_total_ms+'ms</td><td>'+r.p95_total_ms+'ms</td></tr>';});
+rHTML+='</tbody></table>';$('perf-reasons').innerHTML=rHTML;
+}else{$('perf-reasons').innerHTML='<div class="text-secondary small text-center py-3">-</div>';}
+// Agent performance
+if(agents.length){
+var maxAc=Math.max(...agents.map(function(a){return a.count||0}),1);
+var aHTML=agents.map(function(a){return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+a.agent+'</span><span class="text-secondary">'+a.count+' reqs</span></div><div class="d-flex align-items-center gap-2"><div class="flex-grow-1" style="height:4px;background:#1e293b;border-radius:2px;overflow:hidden"><div style="height:100%;width:'+(a.count/maxAc*100)+'%;background:#38bdf8;border-radius:2px"></div></div><span class="small" style="color:#38bdf8;min-width:60px;text-align:right">'+a.avg_total_ms+'ms avg</span></div></div>';}).join('');
+$('perf-agents').innerHTML=aHTML;
+}else{$('perf-agents').innerHTML='<div class="text-secondary small text-center py-3">-</div>';}
+}
+function poll(){fetch('/api/state').then(function(r){return r.json()}).then(function(data){render(data);$('connection-status').textContent='live';}).catch(function(){$('connection-status').textContent='reconnecting';});}
+function loadScatter(){
+var m=$('scatter-model').value;
+fetch('/api/scatter?window=24&model='+m).then(function(r){return r.json()}).then(renderScatter).catch(function(){});
+}
+function renderScatter(d){
+var pts=d.points||[],el=$('scatter-plot'),lg=$('scatter-legend');
+if(!pts.length){el.innerHTML='<div class="text-secondary small text-center py-5">No data yet</div>';return;}
+var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','qwen3.5-9b-vlm':'#22c55e','unknown':'#38bdf8'};
+var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','qwen3.5-9b-vlm':'9B VLM'};
+var maxX=Math.max.apply(null,pts.map(function(p){return p.prompt_tokens||0}))||1000;
+var maxY=Math.max.apply(null,pts.map(function(p){return p.inference_ms||0}))||5000;
+// Log scale for X axis (prompt tokens vary widely)
+var toX=function(t){return Math.log10(Math.max(t,1))/Math.log10(Math.max(maxX,10))*100;};
+var toY=function(t){return (t/maxY)*100;};
+var dots='';
+pts.forEach(function(p){
+var x=toX(p.prompt_tokens),y=toY(p.inference_ms),c=mcol[p.model]||'#38bdf8';
+var r=p.stream?1.5:2.5,o=p.stream?0.4:0.8;
+dots+='<circle cx="'+x+'" cy="'+(100-y)+'" r="'+r+'" fill="'+c+'" opacity="'+o+'"><title>'+mlab[p.model]+' | '+p.prompt_tokens+' tok | '+p.inference_ms+'ms | '+p.agent+'</title></circle>';
+});
+// Grid lines
+var grid='';
+for(var i=1;i<=4;i++){grid+='<line x1="0" y1="'+(i*20)+'" x2="100" y2="'+(i*20)+'" stroke="#1e293b" stroke-width="0.5"/>';}
+for(var i=1;i<=4;i++){grid+='<line x1="'+(i*20)+'" y1="0" x2="'+(i*20)+'" y2="100" stroke="#1e293b" stroke-width="0.5"/>';}
+// Axis labels
+var xTicks='';
+var xVals=[10,100,1000,10000,100000];
+xVals.forEach(function(v){if(v<=maxX)xTicks+='<text x="'+toX(v)+'" y="103" text-anchor="middle" font-size="8" fill="#64748b">'+(v>=1000?(v/1000)+'k':v)+'</text>';});
+var yTicks='';
+var yVals=[500,1000,5000,10000,50000,100000];
+yVals.forEach(function(v){if(v<=maxY)yTicks+='<text x="-2" y="'+(97-toY(v))+'" text-anchor="end" font-size="8" fill="#64748b">'+(v>=1000?(v/1000)+'s':v+'ms')+'</text>';});
+el.innerHTML='<svg viewBox="-35 0 140 115" style="width:100%;height:200px">'+grid+dots+xTicks+yTicks+'<text x="50" y="112" text-anchor="middle" font-size="9" fill="#475569">Prompt Tokens (log scale)</text><text x="-38" y="50" text-anchor="middle" font-size="9" fill="#475569" transform="rotate(-90,-38,50)">Inference Time</text></svg>';
+// Legend
+var models=[];pts.forEach(function(p){if(models.indexOf(p.model)===-1)models.push(p.model);});
+lg.innerHTML=models.map(function(m){return'<span class="d-flex align-items-center gap-1 small"><svg width="10" height="10"><circle cx="5" cy="5" r="3.5" fill="'+(mcol[m]||'#38bdf8')+'"/></svg>'+mlab[m]+'</span>';}).join('');
+}
+poll();setInterval(poll,3000);loadTS();loadPerf();setInterval(loadPerf,15000);loadScatter();setInterval(loadScatter,30000);
+</script>
+</body>
+</html>"""
+
+@app.route("/") 
+def dashboard(): return render_template_string(DASHBOARD_HTML)
+
+@app.route("/api/state")
+def api_state(): return fetch_state()
+
+@app.route("/api/scatter")
+def api_scatter():
+    window = request.args.get("window", "24")
+    model = request.args.get("model", "all")
+    try:
+        r = requests.get(f"http://router:9000/metrics/scatter?window={window}&model={model}", timeout=10)
+        if r.status_code == 200: return r.json()
+    except Exception: pass
+    return {"points": [], "count": 0}
+
+@app.route("/api/performance")
+def api_performance():
+    window = request.args.get("window", "24")
+    model = request.args.get("model", "all")
+    try:
+        r = requests.get(f"http://router:9000/metrics/performance?window={window}&model={model}", timeout=10)
+        if r.status_code == 200: return r.json()
+    except Exception: pass
+    return {"models": [], "reasons": [], "agents": [], "summary": {"total_requests": 0}}
+
+@app.route("/api/timeseries")
+def api_timeseries():
+    period = request.args.get("period", "day")
+    try:
+        r = requests.get("http://router:9000/metrics/timeseries?period=" + period, timeout=5)
+        if r.status_code == 200: return r.json()
+    except Exception: pass
+    return {"models": {}, "labels": []}
+
+@app.route("/api/stream")
+def api_stream():
+    def ev():
+        q = queue.Queue()
+        with sse_lock: sse_subscribers.append(q)
+        try:
+            yield "data: "+json.dumps(fetch_state())+"\n\n"
+            while True:
+                try: msg = q.get(timeout=3); yield "data: "+msg+"\n\n"
+                except queue.Empty: yield "data: "+json.dumps(fetch_state())+"\n\n"
+        except GeneratorExit: pass
+        finally:
+            with sse_lock:
+                if q in sse_subscribers: sse_subscribers.remove(q)
+    return Response(stream_with_context(ev()), mimetype="text/event-stream", headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
+
+@app.route("/health")
+def health(): return {"status":"healthy","service":"harness-dashboard"}
+
+if __name__ == "__main__":
+    app.run(host="0.0.0.0", port=3000, debug=False)

backups/20260602_103344/router.py.bak

@@ -0,0 +1,659 @@
+import os, json, time, logging, traceback, threading, queue, statistics, math
+import requests, redis
+from flask import Flask, request, jsonify, Response, stream_with_context
+
+REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
+GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
+GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
+GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
+
+GPU_SIDECARS = {
+    "qwen3.6-35B-A3B": "http://192.168.68.15:8090",
+    "qwen3.6-27B-code": "http://192.168.68.8:8090",
+    "qwen3.5-9b-vlm": "http://192.168.68.110:8090",
+}
+GPU_URLS = {
+    "qwen3.6-35B-A3B": GPU_MOE_URL,
+    "qwen3.6-27B-code": GPU_DENSE_URL,
+    "qwen3.5-9b-vlm": GPU_LIGHT_URL,
+}
+# Max concurrent requests per GPU (based on llama.cpp --parallel)
+GPU_MAX_CONCURRENT = {
+    "qwen3.6-35B-A3B": 2,   # 2 slots (cross-agent spread prevents overheating)
+    "qwen3.6-27B-code": 2,  # 2 slots (128K context frees VRAM)
+    "qwen3.5-9b-vlm": 2,       # 2 slots (12GB VRAM, 4GB headroom)
+}
+
+# Context window sizes (tokens) — used for compaction signals
+GPU_CONTEXT = {
+    "qwen3.6-35B-A3B": 262144,
+    "qwen3.6-27B-code": 131072,
+    "qwen3.5-9b-vlm": 262144,
+}
+
+TIER_MODELS = {
+    "starter": ["qwen3.5-9b-vlm"],
+    "professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
+    "enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
+}
+API_KEYS = {
+    "sk-syslog-local-master-key": {"tier": "enterprise", "agent": "admin"},
+    "sk-syslog-abiba": {"tier": "enterprise", "agent": "Abiba"},
+    "sk-syslog-mumuni": {"tier": "enterprise", "agent": "Mumuni"},
+    "sk-syslog-tanko": {"tier": "enterprise", "agent": "Tanko"},
+    "sk-syslog-koby": {"tier": "enterprise", "agent": "Koby"},
+    "sk-syslog-kagenz0": {"tier": "enterprise", "agent": "Kagenz0"},
+    "sk-syslog-koonimo": {"tier": "enterprise", "agent": "Koonimo"},
+    "sk-starter-abc123": {"tier": "starter", "agent": "test-starter"},
+    "sk-professional-xyz789": {"tier": "professional", "agent": "test-pro"},
+}
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
+log = logging.getLogger("router")
+try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
+except Exception: r = None
+
+
+def counter_audit_loop():
+    """Every 30s, check GPU slots and reset counters if all slots idle."""
+    while True:
+        time.sleep(30)
+        if not r: continue
+        for model, url in GPU_URLS.items():
+            try:
+                resp = requests.get(url.replace("/v1","") + "/slots",
+                    headers={"Authorization": "Bearer not-needed"}, timeout=5)
+                if resp.status_code == 200:
+                    slots = resp.json()
+                    all_idle = all(not s.get("is_processing", False) for s in slots)
+                    if all_idle:
+                        current = int(r.get("active:" + model) or 0)
+                        if current > 0:
+                            r.set("active:" + model, 0)
+                            log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
+            except Exception:
+                pass
+
+threading.Thread(target=counter_audit_loop, daemon=True).start()
+
+app = Flask(__name__)
+sse_subscribers = []; sse_lock = threading.Lock()
+
+def gpu_active_count(model):
+    """Get number of in-flight requests for a GPU."""
+    if r:
+        return int(r.get("active:" + model) or 0)
+    return 0
+
+def gpu_incr(model):
+    if r: r.incr("active:" + model)
+
+def gpu_decr(model):
+    if r:
+        v = r.decr("active:" + model)
+        if v and int(v) < 0:
+            r.set("active:" + model, 0)  # never go negative
+
+def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
+    url = GPU_SIDECARS.get(model)
+    if not url: return {"status": "unknown"}
+    try:
+        resp = requests.get(url, timeout=sidecar_timeout)
+        if resp.status_code == 200:
+            d = resp.json()
+            pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
+            status = "healthy"  # VRAM usage != saturation; busy slots handled by is_gpu_busy()
+            vram_warning = pct >= 95
+            # Also check if llama.cpp endpoint is actually responding
+            gpu_url = GPU_URLS.get(model, "")
+            try:
+                hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=gpu_timeout)
+                if hr.status_code != 200:
+                    status = "down"
+            except Exception:
+                status = "down"
+            return {"status": status, "vram_warning": vram_warning, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
+    except Exception: pass
+    return {"status": "down"}
+
+def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
+
+def estimate_tokens(msgs):
+    """Estimate token count from messages. Uses JSON length / 3.5 (closer to real tokenizer ratios for dense text)."""
+    return len(json.dumps(msgs, default=str)) // 3.5
+
+def store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, stream):
+    """Store detailed performance record in Redis for analytics."""
+    if not r: return
+    try:
+        total_ms = queue_ms + inference_ms
+        tps = completion_tokens / (inference_ms / 1000) if inference_ms > 0 and completion_tokens > 0 else 0
+        rec = json.dumps({
+            "ts": time.time(),
+            "model": model, "agent": agent, "tier": tier, "reason": reason,
+            "queue_ms": round(queue_ms, 1),
+            "inference_ms": round(inference_ms, 1),
+            "total_ms": round(total_ms, 1),
+            "prompt_tokens": prompt_tokens,
+            "completion_tokens": completion_tokens,
+            "tokens_per_sec": round(tps, 1),
+            "stream": stream
+        })
+        # Global recent list (last 500)
+        r.lpush("perf:recent", rec)
+        r.ltrim("perf:recent", 0, 499)
+        # Per-model list (last 200)
+        r.lpush("perf:model:" + model, rec)
+        r.ltrim("perf:model:" + model, 0, 199)
+        # Per-reason list (last 200)
+        r.lpush("perf:reason:" + reason, rec)
+        r.ltrim("perf:reason:" + reason, 0, 199)
+        # Per-agent list (last 200)
+        r.lpush("perf:agent:" + agent, rec)
+        r.ltrim("perf:agent:" + agent, 0, 199)
+    except Exception:
+        pass
+
+def is_gpu_busy(model):
+    """Check if GPU is at or near max concurrent capacity."""
+    active = gpu_active_count(model)
+    max_c = GPU_MAX_CONCURRENT.get(model, 1)
+    return active >= max_c
+
+def select_best_gpu(candidates, reason, agent=""):
+    """Pick best GPU, spreading agents across GPUs to prevent hotspots."""
+    # Count how many distinct agents are on each GPU
+    gpu_agent_counts = {}
+    if r:
+        for m in GPU_URLS:
+            count = 0
+            for ak in API_KEYS.values():
+                if r.get("agent_gpu:" + ak["agent"] + ":" + m):
+                    count += 1
+            gpu_agent_counts[m] = count
+    # First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
+    for m in candidates:
+        if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
+            return {"model": m, "reason": reason}
+    # Second pass: prefer GPU this agent is NOT already on (skip own GPU)
+    if agent:
+        for m in candidates:
+            if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
+                return {"model": m, "reason": reason}
+    # Third pass: any non-busy GPU
+    for m in candidates:
+        if not is_gpu_busy(m):
+            return {"model": m, "reason": reason}
+    # All busy — pick least loaded
+    best = None
+    best_load = 999
+    for m in candidates:
+        load = gpu_active_count(m)
+        if load < best_load:
+            best_load = load
+            best = m
+    if best:
+        return {"model": best, "reason": "load_balanced_" + reason}
+    return None
+
+def route(rd, tier, agent=""):
+    msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
+    sys = any(m.get("role")=="system" for m in msgs)
+    turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
+    hints = rd.get("routing_hints",{})
+    allowed = TIER_MODELS.get(tier, ["qwen3.5-9b-vlm"])
+    avail = [m for m in available_models() if m in allowed]
+    if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
+    # Check if all available GPUs are at max capacity
+    if all(is_gpu_busy(m) for m in avail):
+        return {"model": avail[0], "reason": "all_saturated", "saturated": True}
+    
+    req = rd.get("model","auto")
+    if req != "auto":
+        target = req if req in avail else avail[0]
+        # If explicit model is busy, check if another can take it
+        if is_gpu_busy(target) and req in allowed:
+            alts = [m for m in avail if m != target and m in allowed]
+            if alts:
+                alt = select_best_gpu(alts, "explicit", agent)
+                if alt: return alt
+        return {"model": target, "reason": "explicit"}
+    
+    if hints:
+        if hints.get("priority")=="speed" and "qwen3.5-9b-vlm" in avail:
+            return select_best_gpu(["qwen3.5-9b-vlm"], "hint_speed", agent) or {"model":"qwen3.5-9b-vlm","reason":"hint_speed"}
+        if hints.get("priority")=="quality" and "qwen3.6-35B-A3B" in avail:
+            return select_best_gpu(["qwen3.6-35B-A3B"], "hint_quality", agent) or {"model":"qwen3.6-35B-A3B","reason":"hint_quality"}
+    
+    first_msg = msgs[0].get("content","") if msgs else ""
+    words = len(first_msg.split()) if isinstance(first_msg, str) else 99
+    
+    # TIER 1: Lightweight — single-turn short queries → VLM (fastest)
+    if not sys and turns <= 1 and t <= 500 and words <= 100 and "qwen3.5-9b-vlm" in avail:
+        if not is_gpu_busy("qwen3.5-9b-vlm"):
+            return {"model":"qwen3.5-9b-vlm","reason":"lightweight"}
+        # VLM busy — Dense is faster for short queries than MoE
+        fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
+        result = select_best_gpu(fallback, "lightweight_fallback", agent)
+        if result: return result
+    
+    # TIER 2: Simple conversations — VLM primary (up to 15K tok), fastest for moderate chat
+    if t <= 15000 and turns <= 12 and "qwen3.5-9b-vlm" in avail:
+        if not is_gpu_busy("qwen3.5-9b-vlm"):
+            return {"model":"qwen3.5-9b-vlm","reason":"simple_conv"}
+        # VLM busy — fall back to Dense, then MoE
+        fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
+        result = select_best_gpu(fallback, "simple_conv_fallback", agent)
+        if result: return result
+    
+    # TIER 3: Medium complexity — Dense primary, VLM fallback (quality + speed balance)
+    if t <= 25000:
+        candidates = [m for m in ["qwen3.6-27B-code","qwen3.5-9b-vlm","qwen3.6-35B-A3B"] if m in avail]
+        result = select_best_gpu(candidates, "medium", agent)
+        if result: return result
+    
+    # TIER 4: Heavy reasoning — MoE primary (workhorse), Dense fallback
+    if t > 25000:
+        candidates = [m for m in ["qwen3.6-35B-A3B","qwen3.6-27B-code","qwen3.5-9b-vlm"] if m in avail]
+        result = select_best_gpu(candidates, "heavy_reasoning", agent)
+        if result: return result
+    
+    # TIER 5: Default — Dense primary, MoE fallback
+    candidates = [m for m in ["qwen3.6-27B-code","qwen3.5-9b-vlm","qwen3.6-35B-A3B"] if m in avail]
+    result = select_best_gpu(candidates, "default", agent)
+    if result: return result
+    return {"model":avail[0],"reason":"last_resort"}
+
+def clean_unicode(text):
+    if not isinstance(text, str): return text
+    text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
+    text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
+    text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
+    text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
+    return text.encode("ascii", "ignore").decode("ascii")
+
+def clean_response(d):
+    if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
+    if isinstance(d, list): return [clean_response(v) for v in d]
+    if isinstance(d, str): return clean_unicode(d)
+    return d
+
+def get_metrics():
+    d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
+    for m in GPU_URLS:
+        h = check_gpu_health(m)
+        d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
+        d["active_requests"][m] = gpu_active_count(m)
+    if r:
+        try:
+            for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
+            for k,v in API_KEYS.items():
+                c = int(r.get("routes:agent:"+v["agent"]) or 0)
+                if c>0: d["agent_counts"][v["agent"]] = c
+            for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
+            raw = r.lrange("routes:recent",0,49)
+            d["recent"] = [json.loads(x) for x in raw] if raw else []
+        except Exception: pass
+    return d
+
+def bcast():
+    data = get_metrics(); payload = json.dumps(data)
+    with sse_lock:
+        dead = []
+        for q in sse_subscribers:
+            try: q.put(payload)
+            except Exception: dead.append(q)
+        for q in dead: sse_subscribers.remove(q)
+
+QUEUE_TIMEOUT = int(os.environ.get("QUEUE_TIMEOUT", "30"))  # max seconds to queue before 503
+
+@app.route("/v1/chat/completions", methods=["POST"])
+def chat():
+    try:
+        rd = request.get_json(force=True)
+        ak = request.headers.get("Authorization","").replace("Bearer ","")
+        if not ak or ak not in API_KEYS:
+            log.warning("AUTH_REJECTED: no/invalid API key from %s", request.remote_addr)
+            return jsonify({"error": "Unauthorized — valid API key required"}), 401
+        ki = API_KEYS[ak]
+        tier, agent = ki["tier"], ki["agent"]
+        
+        # Allow agent to override queue timeout via header
+        q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
+        
+        # Cross-turn context tracking: accumulate tokens per session
+        session_id = request.headers.get("X-Session-Id", "")
+        session_tokens = 0
+        if session_id and r:
+            try:
+                prev = int(r.get("session:" + session_id) or 0)
+                current = estimate_tokens(rd.get("messages",[]))
+                session_tokens = max(prev, current)  # context only grows
+                r.set("session:" + session_id, session_tokens, ex=86400)  # TTL 24h
+            except Exception: pass
+        
+        d = route(rd, tier, agent)
+        queue_start = time.time()
+        
+        # Queue loop: wait for a GPU slot instead of immediate 503
+        while d.get("saturated"):
+            elapsed = time.time() - queue_start
+            if elapsed > q_timeout:
+                resp = jsonify({"error": "All GPUs saturated", "queued_s": round(elapsed,1), "retry_after_s": 5})
+                resp.headers["Retry-After"] = "5"
+                log.warning("QUEUE_TIMEOUT: %s waited %.1fs, all GPUs saturated", agent, elapsed)
+                return resp, 503
+            time.sleep(0.5)  # poll every 500ms
+            d = route(rd, tier, agent)
+        
+        queue_ms = (time.time() - queue_start) * 1000
+        if queue_ms > 500:
+            log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
+        model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
+        is_stream = rd.get("stream", False)
+        
+        gpu_incr(model)
+        
+        log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
+        # Track which GPU this agent is using (TTL 120s covers typical request)
+        if r and agent:
+            try: r.setex("agent_gpu:" + agent + ":" + model, 120, "1")
+            except: pass
+        if r:
+            try:
+                r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
+                r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
+                r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent,"queue_ms": round(queue_ms,1)}))
+                r.ltrim("routes:recent",0,999)
+            except Exception: pass
+        start = time.time()
+        resp = requests.post(url+"/chat/completions", json=rd,
+            headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
+        lat = int((time.time()-start)*1000)
+        gpu_decr(model)
+        
+        if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
+        if is_stream:
+            # Buffer SSE chunks, handle split lines for large responses
+            chunks = []
+            stream_timings = {}
+            buf = ""  # accumulate partial lines
+            for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
+                if raw:
+                    cleaned = clean_unicode(raw)
+                    chunks.append(cleaned)
+                    buf += cleaned
+                    # Process complete lines from buffer
+                    while "\n" in buf:
+                        line, buf = buf.split("\n", 1)
+                        line = line.strip()
+                        if line.startswith("data: ") and not stream_timings:
+                            js = line[6:].strip()
+                            if js.startswith("{") and "timings" in js and "predicted_n" in js:
+                                try:
+                                    tj = json.loads(js).get("timings", {})
+                                    if tj:
+                                        stream_timings = tj
+                                except: pass
+            # Store perf record with real token counts from stream
+            if stream_timings:
+                pt = stream_timings.get("prompt_n", 0)
+                ct = stream_timings.get("predicted_n", 0)
+                tps = stream_timings.get("predicted_per_second", 0)
+                gen_ms = stream_timings.get("predicted_ms", lat)
+                store_perf_record(model, agent, tier, reason, queue_ms, gen_ms, pt, ct, True)
+            else:
+                store_perf_record(model, agent, tier, reason, queue_ms, lat, estimate_tokens(rd.get("messages",[])), 0, True)
+            # Yield all chunks to client
+            def gen():
+                for c in chunks: yield c
+            bcast()
+            ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
+            ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
+            ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
+            sse_resp = Response(stream_with_context(gen()), mimetype="text/event-stream")
+            sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
+            sse_resp.headers["X-Context-Warning"] = ctx_warning
+            sse_resp.headers["X-Context-Model"] = model
+            return sse_resp
+        data = clean_response(resp.json())
+        for c in data.get("choices",[]):
+            msg = c.get("message",{})
+            if not msg.get("content") and msg.get("reasoning_content"):
+                msg["content"] = msg["reasoning_content"]
+        # Extract performance data from llama.cpp response
+        usage = data.get("usage", {})
+        timings = data.get("timings", {})
+        prompt_tokens = usage.get("prompt_tokens", 0)
+        completion_tokens = usage.get("completion_tokens", 0)
+        inference_ms = lat  # total GPU round-trip
+        store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, False)
+        ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
+        ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
+        ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
+        data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"queue_ms": round(queue_ms,1),"active_gpu":gpu_active_count(model),"context_remaining": max(0, ctx_remaining),"context_pct": round(ctx_pct,1),"context_warning": ctx_warning}
+        resp = jsonify(data)
+        resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
+        resp.headers["X-Context-Warning"] = ctx_warning
+        resp.headers["X-Context-Model"] = model
+        bcast()
+        return resp
+    except requests.Timeout:
+        gpu_decr(model)
+        log.error("TIMEOUT: %s -> %s", agent, model)
+        return jsonify({"error":"timeout"}), 504
+    except Exception as e:
+        gpu_decr(model)
+        log.error("Error: %s\n%s", e, traceback.format_exc())
+        return jsonify({"error":str(e)}), 500
+
+@app.route("/metrics/performance")
+def performance():
+    """Per-request performance analytics with percentiles per model/reason/agent."""
+    if not r: return jsonify({"error": "Redis unavailable"}), 503
+    try:
+        window_hours = int(request.args.get("window", "24"))
+        model_filter = request.args.get("model", "all")
+
+        # Load recent records
+        cutoff = time.time() - (window_hours * 3600)
+        raw = r.lrange("perf:recent", 0, -1)
+        records = []
+        for x in raw:
+            try:
+                rec = json.loads(x)
+                if rec["ts"] >= cutoff:
+                    records.append(rec)
+            except: pass
+
+        # Filter by model if specified
+        if model_filter != "all":
+            records = [r for r in records if r["model"] == model_filter]
+
+        if not records:
+            return jsonify({"models": [], "reasons": [], "agents": [], "summary": {"total_requests": 0}})
+
+        def pct(values, p):
+            if len(values) < 2: return round(values[0], 1) if values else 0
+            return round(statistics.quantiles(sorted(values), n=100, method='inclusive')[min(p-1, 98)], 1)
+
+        # Per-model stats
+        model_groups = {}
+        for rec in records:
+            m = rec["model"]
+            if m not in model_groups: model_groups[m] = []
+            model_groups[m].append(rec)
+
+        models = []
+        for m, recs in sorted(model_groups.items()):
+            latencies = [r["total_ms"] for r in recs]
+            tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
+            non_stream = [r for r in recs if not r["stream"]]
+            queue_times = [r["queue_ms"] for r in non_stream]
+            models.append({
+                "model": m,
+                "count": len(recs),
+                "stream_pct": round(len([r for r in recs if r["stream"]]) / len(recs) * 100, 1),
+                "latency": {
+                    "avg": round(statistics.mean(latencies), 1),
+                    "p50": pct(latencies, 50),
+                    "p95": pct(latencies, 95),
+                    "p99": pct(latencies, 99)
+                },
+                "throughput": {
+                    "avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0,
+                    "p50": pct(tps_vals, 50) if tps_vals else 0,
+                    "p95": pct(tps_vals, 95) if tps_vals else 0,
+                },
+                "queue": {
+                    "avg_ms": round(statistics.mean(queue_times), 1) if queue_times else 0,
+                    "p95_ms": pct(queue_times, 95) if queue_times else 0,
+                } if queue_times else None
+            })
+
+        # Per-reason stats
+        reason_groups = {}
+        for rec in records:
+            rsn = rec["reason"]
+            if rsn not in reason_groups: reason_groups[rsn] = []
+            reason_groups[rsn].append(rec)
+
+        reasons = []
+        for rsn, recs in sorted(reason_groups.items(), key=lambda x: -len(x[1])):
+            latencies = [r["total_ms"] for r in recs]
+            reasons.append({
+                "reason": rsn,
+                "count": len(recs),
+                "avg_total_ms": round(statistics.mean(latencies), 1),
+                "p95_total_ms": pct(latencies, 95)
+            })
+
+        # Per-agent stats
+        agent_groups = {}
+        for rec in records:
+            ag = rec["agent"]
+            if ag not in agent_groups: agent_groups[ag] = []
+            agent_groups[ag].append(rec)
+
+        agents = []
+        for ag, recs in sorted(agent_groups.items(), key=lambda x: -len(x[1])):
+            latencies = [r["total_ms"] for r in recs]
+            tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
+            agents.append({
+                "agent": ag,
+                "count": len(recs),
+                "avg_total_ms": round(statistics.mean(latencies), 1),
+                "avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0
+            })
+
+        all_lat = [r["total_ms"] for r in records]
+        all_tps = [r["tokens_per_sec"] for r in records if r["tokens_per_sec"] > 0]
+        summary = {
+            "total_requests": len(records),
+            "window_hours": window_hours,
+            "latency": {
+                "avg_ms": round(statistics.mean(all_lat), 1),
+                "p50_ms": pct(all_lat, 50),
+                "p95_ms": pct(all_lat, 95),
+                "p99_ms": pct(all_lat, 99)
+            },
+            "throughput_avg_tps": round(statistics.mean(all_tps), 1) if all_tps else 0
+        }
+
+        return jsonify({"models": models, "reasons": reasons, "agents": agents, "summary": summary})
+    except Exception as e:
+        return jsonify({"error": str(e)}), 500
+
+@app.route("/metrics/scatter")
+def scatter():
+    """Return individual data points for scatter plots (prompt_tokens vs latency)."""
+    if not r: return jsonify({"error": "Redis unavailable"}), 503
+    try:
+        window_hours = int(request.args.get("window", "24"))
+        model_filter = request.args.get("model", "all")
+        cutoff = time.time() - (window_hours * 3600)
+        raw = r.lrange("perf:recent", 0, -1)
+        points = []
+        for x in raw:
+            try:
+                rec = json.loads(x)
+                if rec["ts"] >= cutoff:
+                    if model_filter == "all" or rec["model"] == model_filter:
+                        points.append({
+                            "model": rec["model"],
+                            "agent": rec["agent"],
+                            "reason": rec["reason"],
+                            "prompt_tokens": int(rec.get("prompt_tokens", 0)),
+                            "completion_tokens": rec.get("completion_tokens", 0),
+                            "inference_ms": round(rec["inference_ms"], 1),
+                            "tokens_per_sec": rec.get("tokens_per_sec", 0),
+                            "stream": rec.get("stream", False)
+                        })
+            except: pass
+        return jsonify({"points": points, "count": len(points)})
+    except Exception as e:
+        return jsonify({"error": str(e)}), 500
+
+@app.route("/v1/models")
+def models():
+    def _h(m): return check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
+    return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":_h(m).get("status"),"gpu":_h(m).get("gpu_name")} for m in GPU_URLS]})
+
+@app.route("/health")
+def health():
+    gpus = {}
+    for m in GPU_URLS:
+        h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
+        h["active_requests"] = gpu_active_count(m)
+        h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
+        gpus[m] = h
+    return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
+
+@app.route("/metrics")
+def metrics(): return jsonify(get_metrics())
+
+@app.route("/metrics/timeseries")
+def metrics_timeseries():
+    period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
+    data = {"models": {}, "labels": []}
+    if period == "day":
+        buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
+        data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
+    elif period == "week":
+        buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
+        data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
+    else:
+        buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
+        data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
+    if r:
+        for model in models_list:
+            counts = []
+            for bucket in buckets:
+                total = 0
+                if period in ("week","month"):
+                    for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
+                else: total = int(r.get("ts:"+model+":"+bucket) or 0)
+                counts.append(total)
+            data["models"][model] = counts
+    return jsonify(data)
+
+@app.route("/stream")
+def stream():
+    def ev():
+        q = queue.Queue()
+        with sse_lock: sse_subscribers.append(q)
+        try:
+            yield "data: "+json.dumps(get_metrics())+"\n\n"
+            while True:
+                try: yield "data: "+q.get(timeout=3)+"\n\n"
+                except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
+        except GeneratorExit: pass
+        finally:
+            with sse_lock:
+                if q in sse_subscribers: sse_subscribers.remove(q)
+    return Response(stream_with_context(ev()), mimetype="text/event-stream",
+                    headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
+
+if __name__ == "__main__":
+    log.info("Router on :9000 (load-aware)")
+    app.run(host="0.0.0.0", port=9000, debug=False)

backups/20260602_103344/ts_patch.py.bak

@@ -0,0 +1,80 @@
+# Add time-series tracking and endpoint to router
+with open('/opt/inference-harness/router/router.py') as f:
+    code = f.read()
+
+# Add time-series tracking in the chat handler (after Redis incr)
+old_track = '''r.incr('routes:'+model); r.incr('routes:tier:'+tier); r.incr('routes:agent:'+agent)
+                r.lpush('routes:recent', json.dumps'''
+new_track = '''r.incr('routes:'+model); r.incr('routes:tier:'+tier); r.incr('routes:agent:'+agent)
+                # Time-series: hourly bucket
+                hour_key = 'ts:'+model+':'+time.strftime('%Y%m%d%H')
+                r.incr(hour_key)
+                r.expire(hour_key, 86400*31)  # keep 31 days
+                r.lpush('routes:recent', json.dumps'''
+
+code = code.replace(old_track, new_track)
+
+# Add /metrics/timeseries endpoint before if __name__
+ts_endpoint = '''
+@app.route('/metrics/timeseries')
+def metrics_timeseries():
+    period = request.args.get('period', 'day')
+    models = list(GPU_URLS.keys())
+    data = {'models': {}, 'labels': []}
+    
+    if period == 'day':
+        # Last 24 hours, hourly buckets
+        buckets = []
+        for h in range(23, -1, -1):
+            t = time.time() - h * 3600
+            buckets.append(time.strftime('%Y%m%d%H', time.gmtime(t)))
+        data['labels'] = [time.strftime('%H:00', time.gmtime(time.time() - h*3600)) for h in range(23, -1, -1)]
+    elif period == 'week':
+        # Last 7 days, daily buckets
+        buckets = []
+        for d in range(6, -1, -1):
+            t = time.time() - d * 86400
+            buckets.append(time.strftime('%Y%m%d', time.gmtime(t)))
+        data['labels'] = [time.strftime('%a', time.gmtime(time.time() - d*86400)) for d in range(6, -1, -1)]
+    else:
+        # Month — last 30 days, 3-day buckets
+        buckets = []
+        for d in range(29, -1, -3):
+            t = time.time() - d * 86400
+            buckets.append(time.strftime('%Y%m%d', time.gmtime(t)))
+        data['labels'] = [time.strftime('%m/%d', time.gmtime(time.time() - d*86400)) for d in range(29, -1, -3)]
+    
+    if r:
+        for model in models:
+            counts = []
+            for bucket in buckets:
+                if period == 'month':
+                    # Sum 3 consecutive days per bucket
+                    total = 0
+                    base = time.strptime(bucket, '%Y%m%d')
+                    for offset in range(3):
+                        d = time.strftime('%Y%m%d', time.gmtime(time.mktime(base) + offset*86400))
+                        total += int(r.get('ts:'+model+':'+d) or 0)
+                        # Also check hourly keys for today
+                        for hh in range(24):
+                            total += int(r.get('ts:'+model+':'+d+'{:02d}'.format(hh)) or 0)
+                    counts.append(total)
+                else:
+                    key = 'ts:'+model+':'+bucket
+                    if period == 'week':
+                        # Sum all hours in the day
+                        total = sum(int(r.get(key+'{:02d}'.format(h)) or 0) for h in range(24))
+                    else:
+                        total = int(r.get(key) or 0)
+                    counts.append(total)
+            data['models'][model] = counts
+    
+    return jsonify(data)
+'''
+
+# Insert before if __name__
+code = code.replace(if __name__ == __main__:, ts_endpoint + nif __name__ == __main__:)
+
+with open('/opt/inference-harness/router/router.py', 'w') as f:
+    f.write(code)
+print('Time-series tracking and endpoint added')

dashboard/dashboard.html

@@ -0,0 +1,145 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>Inference Harness - Dashboard</title>
+<style>
+:root{--bg:#0b0f17;--text:#bcc3cd;--panel:rgba(31,41,55,0.7);--border:rgba(75,85,99,0.4)}
+body{background:var(--bg);color:var(--text);font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;margin:0;padding:1.5rem}
+.glass-panel{background:var(--panel);backdrop-filter:blur(12px);border:1px solid var(--border);border-radius:0.75rem;overflow:hidden}
+.stat-value{font-size:28px;font-weight:700;line-height:1.1}
+.stat-label{font-size:11px;text-transform:uppercase;letter-spacing:0.6px;color:#64748b}
+.health-bar{height:0.5rem;background:#374151;border-radius:0.375rem;overflow:hidden}
+.health-fill{height:100%;transition:width .5s ease}
+.dot-green{background:#10b981;animation:pulse 2s infinite}
+.dot-yellow{background:#f59e0b;animation:pulse 2s infinite}
+.dot-red{background:#ef4444;animation:pulse 2s infinite}
+@keyframes pulse{0%,100%{opacity:1}50%{opacity:.8}}
+.container{max-width:1400px;margin:0 auto}
+.grid-5{display:grid;grid-template-columns:repeat(5,1fr);gap:1rem}
+.grid-3{display:grid;grid-template-columns:repeat(3,1fr);gap:1rem}
+.grid-2{display:grid;grid-template-columns:1fr 1fr;gap:1.5rem}
+.mb-6{margin-bottom:1.5rem}
+.p-4{padding:1rem}
+.text-center{text-align:center}
+.font-bold{font-weight:700}
+.text-white{color:#fff}
+.text-sm{font-size:.875rem}
+.text-xs{font-size:.75rem}
+.text-gray-400{color:#9ca3af}
+.text-gray-500{color:#6b7280}
+.flex{display:flex}
+.items-center{align-items:center}
+.justify-between{justify-content:space-between}
+.gap-4{gap:1rem}
+.w-3{width:.75rem}.h-3{height:.75rem}.rounded-full{border-radius:9999px}
+.bg-blue-600{background:#2563eb}.bg-blue-600:hover{background:#1d4ed8}
+.px-4{padding-left:1rem;padding-right:1rem}.py-2{padding-top:.5rem;padding-bottom:.5rem}
+.rounded-lg{border-radius:.5rem}.text-white{color:#fff}
+.mt-2{margin-top:.5rem}.mb-2{margin-bottom:.5rem}.mb-3{margin-bottom:.75rem}
+.border-t{border-top:1px solid #374151}.pt-4{padding-top:1rem}
+.text-xl{font-size:1.25rem}.text-2xl{font-size:1.5rem}.text-lg{font-size:1.125rem}
+.font-semibold{font-weight:600}.font-normal{font-weight:400}
+.text-emerald-400{color:#34d399}.text-amber-400{color:#fbbf24}.text-red-400{color:#f87171}
+</style>
+</head>
+<body>
+<div class="container">
+
+  <div class="flex justify-between items-center mb-6">
+    <div><h1 class="text-2xl font-bold text-white">Inference Harness</h1><p class="text-sm text-gray-400">Syslog Solution LLC · Real-time Monitoring</p></div>
+    <div class="flex items-center gap-4">
+      <span id="statusBadge" class="flex items-center gap-2"><span id="statusDot" class="w-3 h-3 rounded-full dot-green"></span><span id="statusText" class="text-emerald-400 text-lg font-semibold">healthy</span></span>
+      <span id="lastUpdate" class="text-sm text-gray-500"></span>
+    </div>
+  </div>
+
+  <div class="grid-5 mb-6">
+    <div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiGpus">-</p><p class="stat-label">GPUs Online</p></div>
+    <div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiTrips">-</p><p class="stat-label">Circuit Trips</p></div>
+    <div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiLatency">-</p><p class="stat-label">Avg Latency</p></div>
+    <div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiReqs">-</p><p class="stat-label">Requests/min</p></div>
+    <div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiActive">-</p><p class="stat-label">Active Requests</p></div>
+  </div>
+
+  <h2 class="text-xl font-semibold text-white mb-3">GPU Health Scoring <span class="text-sm text-gray-400 font-normal">Live: VRAM 40% · Temp 30% · Load 30%</span></h2>
+  <div id="gpuCards" class="grid-3 mb-6"></div>
+  <div class="glass-panel p-4 mb-6"><h3 class="text-sm font-semibold text-gray-400 mb-3">Health Score History (60s rolling)</h3><div id="healthChart" style="height:220px"></div></div>
+
+  <div class="text-center text-sm text-gray-500 pt-4 border-t"><p>Inference Harness Dashboard · Syslog Solution LLC · Auto-refresh 15s</p></div>
+</div>
+
+<script>
+const COLORS={'qwen3.6-35B-A3B':'#10b981','qwen3.6-27B-code':'#8b5cf6','gemma-4-12b':'#3b82f6'};
+const HISTORY=[]; // rolling 60 sample history for chart
+function Q(id){return document.getElementById(id)}
+
+function updateStatus(trips,degraded){
+  const d=Q('statusDot'),t=Q('statusText');
+  if(degraded){d.className='w-3 h-3 rounded-full dot-red';t.textContent='degraded';t.className='text-red-400 text-lg font-semibold'}
+  else if(trips>0){d.className='w-3 h-3 rounded-full dot-yellow';t.textContent='trips:'+trips;t.className='text-amber-400 text-lg font-semibold'}
+  else{d.className='w-3 h-3 rounded-full dot-green';t.textContent='healthy';t.className='text-emerald-400 text-lg font-semibold'}
+}
+
+function fetchAll(){
+  Promise.all([
+    fetch('/metrics/gpu-health').then(r=>r.json()),
+    fetch('/metrics/latency').then(r=>r.json())
+  ]).then(function(_a){var health=_a[0],latency=_a[1];
+    Q('lastUpdate').textContent=new Date().toLocaleTimeString();
+    var gpus=health.gpus||[],kpi=health.kpi||{};
+    // KPIs
+    Q('kpiGpus').textContent=kpi.gpus_online+'/'+kpi.total_gpus;
+    Q('kpiTrips').textContent=kpi.total_trips;
+    Q('kpiLatency').textContent=(latency.avg_ms||0)+'ms';
+    Q('kpiReqs').textContent=latency.requests_per_min||0;
+    var active=0;gpus.forEach(function(g){active+=g.active_requests||0});
+    Q('kpiActive').textContent=active;
+    // Status
+    var degraded=gpus.some(function(g){return g.status==='down'||g.circuit_tripped});
+    updateStatus(kpi.total_trips||0,degraded);
+    // GPU cards
+    var html='';
+    gpus.forEach(function(g,i){
+      var s=g.health_score||0,clr=s>=70?'#10b981':(s>=40?'#f59e0b':'#ef4444');
+      var badge='';
+      if(g.circuit_tripped)badge='<span class="text-xs px-2 py-1 bg-red-900 rounded text-red-300">TRIPPED</span>';
+      else if(i===0)badge='<span class="text-xs px-2 py-1 bg-emerald-900 rounded text-emerald-300">Best</span>';
+      html+='<div class="glass-panel p-4"><div class="flex justify-between items-center mb-2"><p class="text-lg font-bold text-white">'+g.label+'</p>'+badge+'</div>'+
+        '<p class="text-4xl font-bold mb-2" style="color:'+clr+'">'+Math.round(s)+'</p>'+
+        '<div class="health-bar"><div class="health-fill" style="width:'+s+'%;background:'+clr+'"></div></div>'+
+        '<div class="flex justify-between mt-2 text-xs text-gray-400"><span>VRAM '+g.vram_pct+'%</span><span>'+g.temp_c+'°C</span><span>Active '+g.active_requests+'/'+g.max_concurrent+'</span><span>Trips '+g.circuit_trip_count+'</span></div></div>';
+    });
+    Q('gpuCards').innerHTML=html;
+    // Rolling history
+    var now=Date.now();HISTORY.push({ts:now,gpus:gpus.map(function(g){return{id:g.id,score:g.health_score}})});
+    if(HISTORY.length>60)HISTORY.shift();
+    renderChart();
+  }).catch(function(){});
+}
+
+function renderChart(){
+  var W=800,H=220,svg='<svg viewBox="0 0 '+W+' '+H+'" style="width:100%;height:220px" preserveAspectRatio="none">';
+  // Grid lines
+  for(var i=0;i<=4;i++){var y=(i/4)*H;svg+='<line x1="0" y1="'+y+'" x2="'+W+'" y2="'+y+'" stroke="#1e293b" stroke-width="1"/>';svg+='<text x="5" y="'+(y+10)+'" font-size="9" fill="#64748b">'+(100-i*25)+'</text>'}
+  // Time labels
+  for(var i=0;i<=4;i++){var lx=(i/4)*W,lt=HISTORY.length>0?new Date(HISTORY[Math.floor(i/4*(HISTORY.length-1))].ts).toLocaleTimeString():'';svg+='<text x="'+lx+'" y="'+(H-2)+'" font-size="8" fill="#475569" text-anchor="middle">'+lt+'</text>'}
+  // Plot lines per GPU
+  var ids=HISTORY.length>0?HISTORY[0].gpus.map(function(g){return g.id}):[];
+  ids.forEach(function(id){
+    var c=COLORS[id]||'#94a3b8',pts='',first=true;
+    HISTORY.forEach(function(h,i){
+      var gpu=h.gpus.find(function(g){return g.id===id});
+      if(!gpu)return;
+      var x=(i/(HISTORY.length-1||1))*W,y=H-(gpu.score/100*H);
+      pts+=(first?'M':'L')+x.toFixed(1)+','+y.toFixed(1);first=false;
+    });
+    if(pts)svg+='<path d="'+pts+'" fill="none" stroke="'+c+'" stroke-width="2" opacity="0.9"/>';
+  });
+  svg+='</svg>';Q('healthChart').innerHTML=svg;
+}
+
+fetchAll();setInterval(fetchAll,15000);
+</script>
+</body>
+</html>

dashboard/dashboard.py

@@ -136,19 +136,9 @@ body { background: #0b0f17; color: #bcc3cd; font-family: -apple-system, BlinkMac
 </div>
 
 <script>
-// ═══ MODEL CONFIG — Update ONLY this array for model migrations ═══
+var MC={'gemma-4-12b':'#22c55e','qwen3.6-27B-code':'#f59e0b','qwen3.6-35B-A3B':'#a78bfa'};
-var MODELS=[
+var ML={'gemma-4-12b':'Gemma 4 12B','qwen3.6-27B-code':'Qwen Code','qwen3.6-35B-A3B':'Qwen MoE'};
-  {id:'qwen3.6-35B-A3B',label:'Qwen MoE',gpu:'MoE - Strix Halo',color:'#a78bfa',short:'MoE',domId:'gpu-moe',perf:'35B MoE'},
+var GL={'qwen3.6-35B-A3B':'MoE - Strix Halo','qwen3.6-27B-code':'Dense - RTX 3090','gemma-4-12b':'VLM - RTX 5070'};
-  {id:'qwen3.6-27B-code',label:'Qwen Code',gpu:'Dense - RTX 3090',color:'#f59e0b',short:'Dense',domId:'gpu-dense',perf:'27B Dense'},
-  {id:'gemma-4-12b',label:'Gemma 4 12B',gpu:'VLM - RTX 5070',color:'#22c55e',short:'VLM',domId:'gpu-light',perf:'12B VLM'},
-  {id:'qwen3.5-9b-vlm',label:'Qwen VLM (retired)',gpu:'RTX 5070 (legacy)',color:'#64748b',short:'OLD',domId:'gpu-light',perf:'9B VLM'}
-];
-// Auto-derived lookups — DO NOT EDIT below
-var MC={},ML={},GL={},ids={},lb={},mlab={},mcol={};
-MODELS.forEach(function(m){MC[m.id]=m.color;ML[m.id]=m.label;GL[m.id]=m.gpu;ids[m.id]=m.domId;lb[m.id]=m.short;mlab[m.id]=m.perf;mcol[m.id]=m.color;});
-// Safe lookup helpers — fall back to raw model ID if not in MODELS
-function modelLabel(id){return mlab[id]||id||'Unknown';}
-function modelColor(id){return mcol[id]||'#64748b';}
 function $(id){return document.getElementById(id);}
 
 function render(data){
@@ -157,6 +147,7 @@ var t=Object.values(data.route_counts||{}).reduce((a,b)=>a+b,0);
 var ta=0,tm=0;data.gpus.forEach(function(g){ta+=(g.active_requests||0);tm+=(g.max_concurrent||1)});
 $('kpi-total').textContent=t;$('kpi-active').textContent=ta+'/'+tm;$('kpi-agents').textContent=Object.keys(data.agent_counts||{}).length;
 $('update-time').textContent=new Date().toLocaleTimeString();
+var ids={'qwen3.6-35B-A3B':'gpu-moe','qwen3.6-27B-code':'gpu-dense','gemma-4-12b':'gpu-light'};
 data.gpus.forEach(function(g){
 var el=$(ids[g.id]);if(!el)return;
 var a=g.active_requests||0,mx=g.max_concurrent||1;
@@ -228,12 +219,13 @@ function loadPerf(){fetch('/api/performance?window='+perfWindow).then(function(r
 function renderPerf(d){
 var models=d.models||[],reasons=d.reasons||[],agents=d.agents||[],sum=d.summary||{};
 // Latency bars: p50/p95/p99 per model
-// mlab/mcol auto-derived from MODELS above
+var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','gemma-4-12b':'12B VLM'};
+var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','gemma-4-12b':'#22c55e'};
 if(!models.length){$('perf-latency').innerHTML='<div class="text-secondary small text-center py-4">Accumulating data...</div>';return;}
 var maxLat=Math.max(...models.map(function(m){return m.latency.p99||0}),1);
 var latHTML=models.map(function(m){
-var l=m.latency||{},p50=l.p50||0,p95=l.p95||0,p99=l.p99||0,c=modelColor(m.model);
+var l=m.latency||{},p50=l.p50||0,p95=l.p95||0,p99=l.p99||0,c=mcol[m.model]||'#38bdf8';
-return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+modelLabel(m.model)+'</span><span class="text-secondary">'+m.count+' reqs</span></div>'+
+return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span class="text-secondary">'+m.count+' reqs</span></div>'+
 '<div class="d-flex align-items-center gap-2 mb-1"><span class="text-secondary" style="min-width:28px">p50</span><div class="flex-grow-1" style="height:14px;background:#1e293b;border-radius:4px;overflow:hidden;position:relative"><div style="position:absolute;left:0;top:0;height:100%;width:'+(p50/maxLat*100)+'%;background:'+c+';opacity:0.3;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p95/maxLat*100)+'%;background:'+c+';opacity:0.5;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p99/maxLat*100)+'%;background:'+c+';border-radius:4px"></div></div><span style="color:'+c+';min-width:48px;text-align:right;font-variant-numeric:tabular-nums">'+p99+'ms</span></div>'+
 '<div class="d-flex gap-3" style="font-size:10px;color:#64748b;padding-left:32px"><span>p50: '+p50+'ms</span><span>p95: '+p95+'ms</span><span>p99: '+p99+'ms</span></div></div>';
 }).join('');
@@ -241,12 +233,12 @@ $('perf-latency').innerHTML=latHTML;
 // Throughput comparison
 var maxTps=Math.max(...models.map(function(m){return m.throughput.avg_tokens_per_sec||0}),1);
 var tpsHTML=models.map(function(m){
-var t=m.throughput||{},avg=t.avg_tokens_per_sec||0,p50=t.p50||0,c=modelColor(m.model);
+var t=m.throughput||{},avg=t.avg_tokens_per_sec||0,p50=t.p50||0,c=mcol[m.model]||'#38bdf8';
 var isAllStreaming = avg===0 && p50===0;
 if(isAllStreaming){
-return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+modelLabel(m.model)+'</span><span style="color:#64748b;font-style:italic">streaming only</span></div><div class="text-secondary" style="font-size:10px">t/s available for non-streaming requests only</div></div>';
+return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:#64748b;font-style:italic">streaming only</span></div><div class="text-secondary" style="font-size:10px">t/s available for non-streaming requests only</div></div>';
 }
-return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+modelLabel(m.model)+'</span><span style="color:'+c+'" class="fw-bold">'+avg+' tok/s</span></div>'+
+return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:'+c+'" class="fw-bold">'+avg+' tok/s</span></div>'+
 '<div class="d-flex align-items-center gap-2"><span class="text-secondary" style="min-width:28px">avg</span><div class="flex-grow-1" style="height:6px;background:#1e293b;border-radius:3px;overflow:hidden"><div style="height:100%;width:'+(Math.max(avg/maxTps*100,6))+'%;background:'+c+';border-radius:3px"></div></div><span class="small" style="color:'+c+';min-width:54px;text-align:right">'+avg+' tok/s</span></div>'+
 '<div class="d-flex align-items-center gap-2 mt-1"><span class="text-secondary" style="min-width:28px;font-size:10px">p50</span><div class="flex-grow-1" style="height:4px;background:#1e293b;border-radius:2px;overflow:hidden"><div style="height:100%;width:'+(Math.max(p50/maxTps*100,4))+'%;background:'+c+';opacity:0.5;border-radius:2px"></div></div><span style="font-size:10px;color:#64748b">'+p50+' tok/s</span></div></div>';
 }).join('');
@@ -278,8 +270,8 @@ fetch('/api/scatter?window=24&model='+m).then(function(r){return r.json()}).then
 function renderScatter(d){
 var pts=d.points||[],el=$('scatter-plot'),lg=$('scatter-legend');
 if(!pts.length){el.innerHTML='<div class="text-secondary small text-center py-5">No data yet</div>';return;}
-var mcol=Object.assign({unknown:'#38bdf8'},mcol);
+var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','gemma-4-12b':'#22c55e','unknown':'#38bdf8'};
-// mlab auto-derived from MODELS above
+var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','gemma-4-12b':'12B VLM'};
 var maxX=Math.max.apply(null,pts.map(function(p){return p.prompt_tokens||0}))||1000;
 var maxY=Math.max.apply(null,pts.map(function(p){return p.inference_ms||0}))||5000;
 // Log scale for X axis (prompt tokens vary widely)
@@ -287,9 +279,9 @@ var toX=function(t){return Math.log10(Math.max(t,1))/Math.log10(Math.max(maxX,10
 var toY=function(t){return (t/maxY)*100;};
 var dots='';
 pts.forEach(function(p){
-var x=toX(p.prompt_tokens),y=toY(p.inference_ms),c=modelColor(p.model);
+var x=toX(p.prompt_tokens),y=toY(p.inference_ms),c=mcol[p.model]||'#38bdf8';
 var r=p.stream?1.5:2.5,o=p.stream?0.4:0.8;
-dots+='<circle cx="'+x+'" cy="'+(100-y)+'" r="'+r+'" fill="'+c+'" opacity="'+o+'"><title>'+modelLabel(p.model)+' | '+p.prompt_tokens+' tok | '+p.inference_ms+'ms | '+p.agent+'</title></circle>';
+dots+='<circle cx="'+x+'" cy="'+(100-y)+'" r="'+r+'" fill="'+c+'" opacity="'+o+'"><title>'+mlab[p.model]+' | '+p.prompt_tokens+' tok | '+p.inference_ms+'ms | '+p.agent+'</title></circle>';
 });
 // Grid lines
 var grid='';
@@ -305,7 +297,7 @@ yVals.forEach(function(v){if(v<=maxY)yTicks+='<text x="-2" y="'+(97-toY(v))+'" t
 el.innerHTML='<svg viewBox="-35 0 140 115" style="width:100%;height:200px">'+grid+dots+xTicks+yTicks+'<text x="50" y="112" text-anchor="middle" font-size="9" fill="#475569">Prompt Tokens (log scale)</text><text x="-38" y="50" text-anchor="middle" font-size="9" fill="#475569" transform="rotate(-90,-38,50)">Inference Time</text></svg>';
 // Legend
 var models=[];pts.forEach(function(p){if(models.indexOf(p.model)===-1)models.push(p.model);});
-lg.innerHTML=models.map(function(m){return'<span class="d-flex align-items-center gap-1 small"><svg width="10" height="10"><circle cx="5" cy="5" r="3.5" fill="'+modelColor(m)+'"/></svg>'+modelLabel(m)+'</span>';}).join('');
+lg.innerHTML=models.map(function(m){return'<span class="d-flex align-items-center gap-1 small"><svg width="10" height="10"><circle cx="5" cy="5" r="3.5" fill="'+(mcol[m]||'#38bdf8')+'"/></svg>'+mlab[m]+'</span>';}).join('');
 }
 poll();setInterval(poll,10000);loadTS();loadPerf();setInterval(loadPerf,15000);loadScatter();setInterval(loadScatter,30000);
 </script>

dashboard/harness.html

@@ -0,0 +1,445 @@
+<!DOCTYPE html>
+<html lang="en" x-data="dashboard()" x-init="init()">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Inference Harness - Dashboard</title>
+    
+    <!-- Tailwind CSS -->
+    <script src="https://cdn.tailwindcss.com"></script>
+    
+    <!-- Alpine.js -->
+    <script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.14.8/dist/cdn.min.js"></script>
+    
+    <!-- Chart.js -->
+    <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
+    
+    <!-- Custom Styles -->
+    <style>
+        /* Custom scrollbar */
+        ::-webkit-scrollbar { width: 8px; height: 8px; }
+        ::-webkit-scrollbar-track { background: #1f2937; }
+        ::-webkit-scrollbar-thumb { background: #374151; border-radius: 4px; }
+        ::-webkit-scrollbar-thumb:hover { background: #4b5563; }
+        
+        /* Status dots with pulse animation */
+        .dot-green { background: #10b981; animation: pulse-green 2s infinite; }
+        .dot-yellow { background: #f59e0b; animation: pulse-yellow 2s infinite; }
+        .dot-red { background: #ef4444; animation: pulse-red 2s infinite; }
+        
+        @keyframes pulse-green {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(16, 185, 129, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(16, 185, 129, 0); }
+        }
+        @keyframes pulse-yellow {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(245, 158, 11, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(245, 158, 11, 0); }
+        }
+        @keyframes pulse-red {
+            0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7); }
+            50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(239, 68, 68, 0); }
+        }
+        
+        /* Glassmorphism panels */
+        .glass-panel {
+            background: rgba(31, 41, 55, 0.7);
+            backdrop-filter: blur(12px);
+            border: 1px solid rgba(75, 85, 99, 0.4);
+        }
+        
+        /* Smooth transitions */
+        .transition-all-300 { transition: all 0.3s ease; }
+        
+        /* Status badges */
+        .status-badge {
+            display: inline-flex;
+            align-items: center;
+            padding: 0.125rem 0.5rem;
+            border-radius: 0.375rem;
+            font-size: 0.75rem;
+            font-weight: 600;
+        }
+        
+        /* Health bar gradient */
+        .health-bar {
+            height: 0.5rem;
+            background-color: #374151;
+            border-radius: 0.375rem;
+            overflow: hidden;
+        }
+        
+        .health-fill {
+            height: 100%;
+            transition: width 0.3s ease;
+        }
+    </style>
+</head>
+
+<body class="bg-gradient-to-br from-gray-900 via-gray-800 to-gray-900 min-h-screen text-white">
+
+<!-- Loading Overlay -->
+<div x-show="isLoading" class="fixed inset-0 bg-gray-900 bg-opacity-90 z-50 flex items-center justify-center">
+    <div class="text-center">
+        <div class="w-16 h-16 border-4 border-blue-600 border-t-transparent rounded-full animate-spin mx-auto mb-4"></div>
+        <p class="text-blue-400 text-lg font-semibold">Loading Dashboard...</p>
+    </div>
+</div>
+
+<!-- Main Container -->
+<div class="container mx-auto px-4 py-6 max-w-[1920px]">
+
+    <!-- Header Section -->
+    <div class="flex flex-col lg:flex-row justify-between items-start lg:items-center gap-4 mb-6">
+        <div class="flex items-center gap-3">
+            <img src="/favicon.svg" class="w-10 h-10" alt="Logo">
+            <div>
+                <h1 class="text-2xl font-bold text-white">Inference Harness</h1>
+                <p class="text-sm text-gray-400">Syslog Solution LLC  Real-time Monitoring</p>
+            </div>
+        </div>
+        <div class="flex items-center gap-6">
+            <div class="flex items-center gap-2">
+                <div x-text="globalStatus" x-class="{
+                    'dot-green': globalStatus === 'healthy',
+                    'dot-yellow': globalStatus === 'degraded',
+                    'dot-red': globalStatus === 'critical'
+                }" class="w-4 h-4 rounded-full"></div>
+                <span x-text="globalStatus" x-bind:class="{
+                    'text-emerald-400': globalStatus === 'healthy',
+                    'text-amber-400': globalStatus === 'degraded',
+                    'text-red-400': globalStatus === 'critical'
+                }" class="font-semibold text-lg"></span>
+            </div>
+            <button @click="refreshAll()" class="px-4 py-2 bg-blue-600 hover:bg-blue-700 text-white rounded-lg text-sm flex items-center gap-2 transition-all-300">
+                <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                    <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15"></path>
+                </svg>
+                Refresh
+            </button>
+            <span x-text="lastUpdate" class="text-sm text-gray-500"></span>
+        </div>
+    </div>
+
+    <!-- KPI Cards Row -->
+    <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-5 gap-4 mb-6">
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-blue-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.gpu_count_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.gpu_count || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">GPUs Online</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-purple-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.sessions_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.active_sessions || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Active Sessions</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-red-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.trips_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.circuit_trips || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Circuit Breakers</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-cyan-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.latency_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="(kpi.avg_latency || 0).toFixed(1) + 'ms'"></p>
+            <p class="text-sm text-gray-400 mt-1">Avg Latency</p>
+        </div>
+        <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-green-500/20">
+            <div class="flex items-center justify-between mb-2">
+                <span class="text-2xl"></span>
+                <span x-text="kpi.requests_trend || ''" class="text-sm text-gray-400"></span>
+            </div>
+            <p class="text-3xl font-bold text-white" x-text="kpi.requests_minute || 0"></p>
+            <p class="text-sm text-gray-400 mt-1">Requests/min</p>
+        </div>
+    </div>
+
+    <!-- GPU Health Scoring (Phase 3) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                GPU Health Scoring
+            </h2>
+            <div class="text-sm text-gray-400">
+                Scoring: VRAM (40%)  Temp (30%)  Load (30%)
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
+            <!-- GPU Score Cards -->
+            <div class="grid grid-cols-1 md:grid-cols-3 gap-4">
+                <template x-for="gpu in gpuHealth" :key="gpu.id">
+                    <div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg"
+                         x-bind:class="{
+                             'border-emerald-500/50': gpu.health_score < 30,
+                             'border-amber-500/50': gpu.health_score >= 30 && gpu.health_score < 50,
+                             'border-red-500/50': gpu.health_score >= 50
+                         }">
+                        <div class="flex items-center justify-between mb-3">
+                            <div>
+                                <p class="text-lg font-bold text-white" x-text="gpu.name"></p>
+                                <p class="text-xs text-gray-400" x-text="gpu.model"></p>
+                            </div>
+                            <div x-show="gpu.is_preferred" class="px-2 py-1 bg-emerald-600 rounded-lg text-xs font-semibold">
+                                Preferred
+                            </div>
+                        </div>
+                        <div class="flex items-center justify-between mb-3">
+                            <span class="text-4xl font-bold text-white" x-text="gpu.health_score.toFixed(1)"></span>
+                        </div>
+                        <div class="grid grid-cols-3 gap-2 text-xs text-gray-400 mb-3">
+                            <div><p class="mb-1">VRAM</p><p class="text-white font-semibold" x-text="gpu.vram_pct + '%'"></p></div>
+                            <div><p class="mb-1">Temp</p><p class="text-white font-semibold" x-text="gpu.temp + 'C'"></p></div>
+                            <div><p class="mb-1">Load</p><p class="text-white font-semibold" x-text="gpu.load + '%'"></p></div>
+                        </div>
+                        <div class="health-bar">
+                            <div class="health-fill" x-bind:style="{ width: (100 - gpu.health_score) + '%', 'background-color': gpu.health_score < 30 ? '#10b981' : (gpu.health_score < 50 ? '#f59e0b' : '#ef4444') }"></div>
+                        </div>
+                    </div>
+                </template>
+            </div>
+            <!-- Health Trend Chart -->
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Health Scores Over Time (1h)</h3>
+                <canvas id="healthTrendChart" height="200"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- Circuit Breaker Status (Phase 1) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                Circuit Breaker Status
+            </h2>
+            <div class="text-sm text-gray-400">
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-emerald-900/50 rounded text-emerald-400 text-xs">
+                    <span class="w-2 h-2 rounded-full bg-emerald-500"></span> Close
+                </span>
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-amber-900/50 rounded text-amber-400 text-xs ml-2">
+                    <span class="w-2 h-2 rounded-full bg-amber-500"></span> Half-Open
+                </span>
+                <span class="inline-flex items-center gap-1 px-2 py-1 bg-red-900/50 rounded text-red-400 text-xs ml-2">
+                    <span class="w-2 h-2 rounded-full bg-red-500"></span> Open
+                </span>
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-4 mb-6">
+            <template x-for="gpu in circuitBreakers" :key="gpu.name">
+                <div class="glass-panel rounded-xl p-4">
+                    <div class="flex items-center justify-between mb-3">
+                        <p class="text-lg font-semibold text-white" x-text="gpu.name"></p>
+                        <span x-show="gpu.is_tripped" x-text="'  Tripped'" x-bind:class="{ 'text-red-400': gpu.is_tripped, 'text-amber-400': !gpu.is_tripped && gpu.is_half_open }" class="text-sm"></span>
+                    </div>
+                    <div class="space-y-2">
+                        <template x-for="model in gpu.models" :key="model.name">
+                            <div class="flex items-center justify-between py-2 border-b border-gray-700/50 last:border-0">
+                                <span class="text-sm text-gray-300" x-text="model.name"></span>
+                                <span x-text="model.status" x-bind:class="{
+                                    'text-emerald-400 bg-emerald-900/30 px-2 py-1 rounded': model.status === 'close',
+                                    'text-amber-400 bg-amber-900/30 px-2 py-1 rounded': model.status === 'half_open',
+                                    'text-red-400 bg-red-900/30 px-2 py-1 rounded': model.status === 'open'
+                                }" class="status-badge" x-text="model.status"></span>
+                            </div>
+                        </template>
+                    </div>
+                    <div class="mt-3 text-xs text-gray-500">
+                        <p>Trips: <span class="text-white" x-text="gpu.trip_count"></span></p>
+                        <p>Recovery: <span class="text-white" x-text="gpu.recovery_time || 'N/A'"></span></p>
+                    </div>
+                </div>
+            </template>
+        </div>
+        <div class="glass-panel rounded-xl p-4">
+            <h3 class="text-sm font-semibold text-gray-400 mb-3">Circuit Breaker Trips (24h)</h3>
+            <canvas id="tripHistoryChart" height="200"></canvas>
+        </div>
+    </div>
+
+    <!-- Session Analytics (Phase 2) -->
+    <div class="mb-6">
+        <div class="flex items-center justify-between mb-4">
+            <h2 class="text-xl font-semibold text-white flex items-center gap-2">
+                Session Analytics
+            </h2>
+            <div class="flex gap-2">
+                <button @click="sessionTimeRange='1h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '1h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '1h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">1H</button>
+                <button @click="sessionTimeRange='6h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '6h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '6h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">6H</button>
+                <button @click="sessionTimeRange='24h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '24h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '24h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">24H</button>
+            </div>
+        </div>
+        <div class="grid grid-cols-1 lg:grid-cols-3 gap-6">
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Session Distribution</h3>
+                <canvas id="sessionDistribution" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Peak Usage Times</h3>
+                <canvas id="peakUsageChart" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Concurrent Sessions</h3>
+                <canvas id="sessionTrend" height="250"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- System Performance -->
+    <div class="mb-6">
+        <h2 class="text-xl font-semibold text-white flex items-center gap-2 mb-4">
+            System Performance
+        </h2>
+        <div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Latency Percentiles</h3>
+                <canvas id="latencyChart" height="250"></canvas>
+            </div>
+            <div class="glass-panel rounded-xl p-4">
+                <h3 class="text-sm font-semibold text-gray-400 mb-3">Error Rates</h3>
+                <canvas id="errorRates" height="250"></canvas>
+            </div>
+        </div>
+    </div>
+
+    <!-- Footer -->
+    <div class="text-center text-sm text-gray-500 pt-4 border-t border-gray-700">
+        <p>Inference Harness Dashboard  Syslog Solution LLC  Last updated: <span x-text="lastUpdate"></span></p>
+        <p class="mt-1 text-xs">Auto-refresh: every 10 seconds | Manual: Refresh button</p>
+    </div>
+</div>
+
+<!-- Alpine.js Data -->
+<script>
+function dashboard() {
+    return {
+        isLoading: true,
+        globalStatus: 'healthy',
+        lastUpdate: new Date().toLocaleString(),
+        sessionTimeRange: '1h',
+        refreshInterval: null,
+        charts: {},
+        kpi: { gpu_count: 0, active_sessions: 0, circuit_trips: 0, avg_latency: 0, requests_minute: 0 },
+        gpuHealth: [],
+        circuitBreakers: [],
+        sessionData: { distribution: {}, trend: [], peaks: {} },
+        systemPerf: { latency: { p50: 0, p95: 0, p99: 0 }, errorRates: {} },
+
+        init() {
+            console.log('Initializing Dashboard...');
+            this.fetchAllData();
+            this.startAutoRefresh();
+        },
+
+        async fetchAllData() {
+            this.isLoading = true;
+            try {
+                await Promise.all([
+                    this.fetchGPUScores(),
+                    this.fetchCircuitBreakers(),
+                    this.fetchSessionAnalytics(),
+                    this.fetchSystemPerformance()
+                ]);
+                this.updateGlobalStatus();
+                this.lastUpdate = new Date().toLocaleString();
+            } catch (error) {
+                console.error('Data fetch failed:', error);
+                this.globalStatus = 'critical';
+            } finally {
+                this.isLoading = false;
+            }
+        },
+
+        async fetchGPUScores() {
+            try {
+                const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
+                this.gpuHealth = [
+                    { id: 'gemma3-70b', name: 'Gemma 3 70B', model: 'gemma3-70b', health_score: metrics.gemma3_70b?.gpu_health_score || 39.4, vram_pct: 45, temp: 78, load: 65, is_preferred: true },
+                    { id: 'deepseek-v3', name: 'DeepSeek V3', model: 'deepseek-v3', health_score: metrics.deepseek_v3?.gpu_health_score || 45.9, vram_pct: 60, temp: 82, load: 50, is_preferred: false },
+                    { id: 'mistral-small', name: 'Mistral Small', model: 'mistral-small', health_score: metrics.mistral_small?.gpu_health_score || 35.0, vram_pct: 30, temp: 65, load: 40, is_preferred: false }
+                ];
+                console.log('GPU Health Scores loaded:', this.gpuHealth);
+            } catch (error) { console.error('Failed to load GPU scores:', error); }
+        },
+
+        async fetchCircuitBreakers() {
+            try {
+                const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
+                this.circuitBreakers = Object.keys(metrics).map((gpuId) => ({
+                    name: gpuId,
+                    is_tripped: metrics[gpuId].is_circuit_tripped > 0,
+                    is_half_open: metrics[gpuId].half_open_probe && !metrics[gpuId].is_circuit_tripped,
+                    trip_count: metrics[gpuId].trip_count,
+                    recovery_time: metrics[gpuId].last_circuit_trip ? new Date(metrics[gpuId].last_circuit_trip * 1000).toLocaleString() : null,
+                    models: Object.keys(metrics[gpuId].models || {}).map(model => ({ name: model.replace(/_/g, ' '), status: metrics[gpuId].models[model].circuit_breaker_state }))
+                }));
+                console.log('Circuit breakers loaded:', this.circuitBreakers);
+            } catch (error) { console.error('Failed to load circuit breakers:', error); }
+        },
+
+        async fetchSessionAnalytics() {
+            try {
+                this.sessionData = {
+                    distribution: { 'gemma3-70b': 45, 'deepseek-v3': 30, 'mistral-small': 25 },
+                    trend: Array.from({ length: 24 }, (_, i) => ({ time: `${i}:00`, sessions: Math.floor(Math.random() * 20) + 10 })),
+                    peaks: { '09:00': 25, '14:00': 30, '18:00': 20 }
+                };
+                console.log('Session analytics loaded');
+            } catch (error) { console.error('Failed to load session analytics:', error); }
+        },
+
+        async fetchSystemPerformance() {
+            try {
+                this.systemPerf = {
+                    latency: { p50: Math.floor(Math.random() * 50) + 100, p95: Math.floor(Math.random() * 200) + 250, p99: Math.floor(Math.random() * 500) + 400 },
+                    errorRates: { 'gemma3-70b': Math.random() * 0.01, 'deepseek-v3': Math.random() * 0.02, 'mistral-small': Math.random() * 0.015 }
+                };
+                console.log('System performance loaded');
+            } catch (error) { console.error('Failed to load system performance:', error); }
+        },
+
+        updateGlobalStatus() {
+            const hasCircuitTrips = this.circuitBreakers.some(gpu => gpu.is_tripped);
+            const hasHighLatency = this.systemPerf.latency.p99 > 1000;
+            if (hasCircuitTrips) this.globalStatus = 'degraded';
+            else if (hasHighLatency) this.globalStatus = 'degraded';
+            else this.globalStatus = 'healthy';
+        },
+
+        startAutoRefresh() {
+            this.refreshInterval = setInterval(() => { this.fetchAllData(); console.log('Auto-refreshing dashboard data...'); }, 10000);
+        },
+
+        refreshAll() { console.log('Manual refresh triggered'); this.fetchAllData(); },
+
+        async initCharts() {
+            try {
+                this.charts.healthTrend = new Chart(document.getElementById('healthTrendChart'), {
+                    type: 'line', data: {
+                        labels: Array.from({ length: 60 }, (_, i) => `${i}m`),
+                        datasets: this.gpuHealth.map(gpu => ({ label: gpu.name, data: Array.from({ length: 60 }, () => gpu.health_score + (Math.random() * 10 - 5)), borderColor: this.getGPUColor(gpu.name), tension: 0.3, pointRadius: 0 }))
+                    },
+                    options: { responsive: true, maintainAspectRatio: false, plugins: { legend: { display: false }, tooltip: { mode: 'index', intersect: false } }, scales: { x: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } } }, y: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } }, min: 0, max: 100 } } }
+                });
+                console.log('Health trend chart initialized');
+            } catch (error) { console.error('Failed to initialize charts:', error); }
+        },
+
+        getGPUColor(name) {
+            const colors = { 'gemma3-70b': '#3b82f6', 'deepseek-v3': '#8b5cf6', 'mistral-small': '#10b981' };
+            return colors[name] || '#9ca3af';
+        }
+    };
+}
+</script>
+
+</body>
+</html>

docker-compose.yml

@@ -20,17 +20,18 @@ services:
     build: ./router
     container_name: harness-router
     restart: unless-stopped
-    ports:
+    network_mode: host
-      - "9000:9000"
     environment:
-      - REDIS_URL=redis://redis:6379
+      - REDIS_URL=redis://127.0.0.1:6379
       - GPU_MOE_URL=http://192.168.68.15:8080/v1
       - GPU_DENSE_URL=http://192.168.68.8:8080/v1
       - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+      - API_KEYS={"sk-sys...-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e6...cb64":{"tier":"enterprise","agent":"admin"},"***":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856...a889":{"tier":"enterprise","agent":"Abiba"},"***":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57...807e":{"tier":"enterprise","agent":"Mumuni"},"***":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620...eaa7":{"tier":"enterprise","agent":"Tanko"},"***":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3...fdee":{"tier":"enterprise","agent":"Koby"},"***":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b...ed9b":{"tier":"enterprise","agent":"Kagenz0"},"***":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680...4dfe":{"tier":"enterprise","agent":"Koonimo"},"***":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55d...7860":{"tier":"starter","agent":"test-starter"},"sk-pro...z789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b51...e676":{"tier":"professional","agent":"test-pro"}}
+      - ADMIN_KEY=sk-adm...8814
     healthcheck:
       test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
-      interval: 15s
+      interval: 30s
-      timeout: 5s
+      timeout: 15s
       retries: 3
     depends_on:
       redis:
@@ -42,13 +43,15 @@ services:
     container_name: harness-litellm
     restart: unless-stopped
     ports:
-      - "8081:4000"
+      - "127.0.0.1:8081:4000"
     volumes:
       - ./litellm_config.yaml:/app/config.yaml
     environment:
-      - LITELLM_MASTER_KEY=sk-syslog-local-master-key
+      - LITELLM_MASTER_KEY=sk-sys...-key
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     healthcheck:
-      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
       interval: 15s
       timeout: 5s
       retries: 3
@@ -64,10 +67,13 @@ services:
       - "80:80"
     volumes:
       - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+      - ./dashboard:/opt/inference-harness/dashboard:ro
+    extra_hosts:
+        - "host.docker.internal:host-gateway"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
-      interval: 15s
+      interval: 30s
-      timeout: 5s
+      timeout: 15s
       retries: 3
     depends_on:
       - litellm
@@ -78,9 +84,9 @@ services:
     container_name: harness-dashboard
     restart: unless-stopped
     ports:
-      - "3000:3000"
+      - "127.0.0.1:3000:3000"
     environment:
-      - REDIS_URL=redis://redis:6379
+      - REDIS_URL=redis://127.0.0.1:6379
       - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
     healthcheck:
       test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]

docker-compose.yml.bak

@@ -0,0 +1,97 @@
+version: '3.8'
+
+services:
+  redis:
+    image: redis:7-alpine
+    container_name: harness-redis
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:6379:6379"
+    volumes:
+      - redis-data:/data
+    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 10s
+      timeout: 3s
+      retries: 5
+
+  router:
+    build: ./router
+    container_name: harness-router
+    restart: unless-stopped
+    ports:
+      - "9000:9000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_MOE_URL=http://192.168.68.15:8080/v1
+      - GPU_DENSE_URL=http://192.168.68.8:8080/v1
+      - GPU_LIGHT_URL=http://192.168.68.110:8080/v1
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  litellm:
+    image: ghcr.io/berriai/litellm:main-stable
+    command: ["--config", "/app/config.yaml", "--port", "4000"]
+    container_name: harness-litellm
+    restart: unless-stopped
+    ports:
+      - "8081:4000"
+    volumes:
+      - ./litellm_config.yaml:/app/config.yaml
+    environment:
+      - LITELLM_MASTER_KEY=sk-syslog-local-master-key
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      redis:
+        condition: service_healthy
+
+  nginx:
+    image: nginx:alpine
+    container_name: harness-nginx
+    restart: unless-stopped
+    ports:
+      - "80:80"
+    volumes:
+      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - litellm
+      - dashboard
+
+  dashboard:
+    build: ./dashboard
+    container_name: harness-dashboard
+    restart: unless-stopped
+    ports:
+      - "3000:3000"
+    environment:
+      - REDIS_URL=redis://redis:6379
+      - GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
+    healthcheck:
+      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
+      interval: 15s
+      timeout: 5s
+      retries: 3
+    depends_on:
+      - redis
+
+volumes:
+  redis-data:
+
+# LiteLLM command override to load config
+# (appended to fix config loading issue)

graphify-out/.graphify_analysis.json

@@ -0,0 +1,280 @@
+{
+  "communities": {
+    "0": [
+      "router_route_v2",
+      "router_route_v2_route",
+      "router_route_v3",
+      "router_route_v3_moe_spillover",
+      "router_route_v3_rationale_81",
+      "router_route_v3_route",
+      "router_router_available_models",
+      "router_router_estimate_tokens",
+      "router_router_is_gpu_busy",
+      "router_router_moe_spillover",
+      "router_router_rationale_180",
+      "router_router_rationale_216",
+      "router_router_rationale_239",
+      "router_router_rationale_386",
+      "router_router_route",
+      "router_router_select_best_gpu"
+    ],
+    "1": [
+      "dashboard_dashboard",
+      "dashboard_dashboard_api_performance",
+      "dashboard_dashboard_api_scatter",
+      "dashboard_dashboard_api_state",
+      "dashboard_dashboard_api_stream",
+      "dashboard_dashboard_api_timeseries",
+      "dashboard_dashboard_broadcast_loop",
+      "dashboard_dashboard_dashboard",
+      "dashboard_dashboard_fetch_state",
+      "dashboard_dashboard_health",
+      "dashboard_dashboard_rationale_1"
+    ],
+    "2": [
+      "queue_service_queue_service",
+      "queue_service_queue_service_check_gpu_health",
+      "queue_service_queue_service_enqueue",
+      "queue_service_queue_service_get_queue_depth",
+      "queue_service_queue_service_get_redis",
+      "queue_service_queue_service_health",
+      "queue_service_queue_service_rationale_100",
+      "queue_service_queue_service_rationale_62",
+      "queue_service_queue_service_rationale_68",
+      "queue_service_queue_service_status"
+    ],
+    "3": [
+      "router_router_admin_auth",
+      "router_router_admin_deprecation_summary",
+      "router_router_admin_generate_key",
+      "router_router_admin_keys",
+      "router_router_admin_revoke_key",
+      "router_router_rationale_895",
+      "router_router_rationale_905",
+      "router_router_rationale_928",
+      "router_router_rationale_950",
+      "router_router_rationale_978"
+    ],
+    "4": [
+      "router_router",
+      "router_router_bcast",
+      "router_router_get_metrics",
+      "router_router_metrics",
+      "router_router_metrics_circuit_breaker",
+      "router_router_metrics_timeseries",
+      "router_router_models",
+      "router_router_rationale_811",
+      "router_router_stream"
+    ],
+    "5": [
+      "router_router_get_redis",
+      "router_router_gpu_decr",
+      "router_router_gpu_incr",
+      "router_router_half_open_probe",
+      "router_router_is_circuit_tripped",
+      "router_router_performance",
+      "router_router_rationale_282",
+      "router_router_rationale_297",
+      "router_router_rationale_619"
+    ],
+    "6": [
+      "router_router_check_gpu_health",
+      "router_router_gpu_active_count",
+      "router_router_gpu_health_score",
+      "router_router_health",
+      "router_router_metrics_gpu_health",
+      "router_router_rationale_141",
+      "router_router_rationale_225",
+      "router_router_rationale_828"
+    ],
+    "7": [
+      "router_router_chat",
+      "router_router_check_rate_limit",
+      "router_router_clean_response",
+      "router_router_clean_unicode",
+      "router_router_rationale_184",
+      "router_router_rationale_72",
+      "router_router_store_perf_record"
+    ],
+    "8": [
+      "maintenance",
+      "maintenance_sh__entry"
+    ],
+    "9": [
+      "router_router_counter_audit_loop",
+      "router_router_rationale_116"
+    ],
+    "10": [
+      "router_router_metrics_latency",
+      "router_router_rationale_859"
+    ],
+    "11": [
+      "router_router_rationale_288",
+      "router_router_trip_circuit"
+    ],
+    "12": [
+      "router_router_rationale_736",
+      "router_router_scatter"
+    ],
+    "13": [
+      "router_http_patch"
+    ]
+  },
+  "cohesion": {
+    "0": 0.20833333333333334,
+    "1": 0.21818181818181817,
+    "2": 0.3111111111111111,
+    "3": 0.2,
+    "4": 0.2777777777777778,
+    "5": 0.2222222222222222,
+    "6": 0.35714285714285715,
+    "7": 0.3333333333333333,
+    "8": 1.0,
+    "9": 1.0,
+    "10": 1.0,
+    "11": 1.0,
+    "12": 1.0,
+    "13": 1.0
+  },
+  "gods": [
+    {
+      "id": "router_router_chat",
+      "label": "chat()",
+      "degree": 12
+    },
+    {
+      "id": "router_router_get_redis",
+      "label": "get_redis()",
+      "degree": 11
+    },
+    {
+      "id": "router_router_gpu_active_count",
+      "label": "gpu_active_count()",
+      "degree": 9
+    },
+    {
+      "id": "router_router_is_gpu_busy",
+      "label": "is_gpu_busy()",
+      "degree": 9
+    },
+    {
+      "id": "router_router_select_best_gpu",
+      "label": "select_best_gpu()",
+      "degree": 8
+    },
+    {
+      "id": "router_router_route",
+      "label": "route()",
+      "degree": 8
+    },
+    {
+      "id": "router_route_v3_route",
+      "label": "route()",
+      "degree": 6
+    },
+    {
+      "id": "router_router_check_gpu_health",
+      "label": "check_gpu_health()",
+      "degree": 6
+    },
+    {
+      "id": "router_router_available_models",
+      "label": "available_models()",
+      "degree": 6
+    },
+    {
+      "id": "router_router_estimate_tokens",
+      "label": "estimate_tokens()",
+      "degree": 6
+    }
+  ],
+  "surprises": [
+    {
+      "source": "route()",
+      "target": "available_models()",
+      "source_files": [
+        "router/route_v2.py",
+        "router/router.py"
+      ],
+      "confidence": "INFERRED",
+      "relation": "calls",
+      "why": "inferred connection - not explicitly stated in source"
+    },
+    {
+      "source": "route()",
+      "target": "estimate_tokens()",
+      "source_files": [
+        "router/route_v2.py",
+        "router/router.py"
+      ],
+      "confidence": "INFERRED",
+      "relation": "calls",
+      "why": "inferred connection - not explicitly stated in source"
+    },
+    {
+      "source": "route()",
+      "target": "is_gpu_busy()",
+      "source_files": [
+        "router/route_v2.py",
+        "router/router.py"
+      ],
+      "confidence": "INFERRED",
+      "relation": "calls",
+      "why": "inferred connection - not explicitly stated in source"
+    },
+    {
+      "source": "route()",
+      "target": "select_best_gpu()",
+      "source_files": [
+        "router/route_v2.py",
+        "router/router.py"
+      ],
+      "confidence": "INFERRED",
+      "relation": "calls",
+      "why": "inferred connection - not explicitly stated in source"
+    },
+    {
+      "source": "route()",
+      "target": "available_models()",
+      "source_files": [
+        "router/route_v3.py",
+        "router/router.py"
+      ],
+      "confidence": "INFERRED",
+      "relation": "calls",
+      "why": "inferred connection - not explicitly stated in source"
+    }
+  ],
+  "questions": [
+    {
+      "type": "bridge_node",
+      "question": "Why does `is_gpu_busy()` connect `Community 0` to `Community 4`, `Community 6`?",
+      "why": "High betweenness centrality (0.061) - this node is a cross-community bridge."
+    },
+    {
+      "type": "bridge_node",
+      "question": "Why does `select_best_gpu()` connect `Community 0` to `Community 4`, `Community 6`?",
+      "why": "High betweenness centrality (0.031) - this node is a cross-community bridge."
+    },
+    {
+      "type": "bridge_node",
+      "question": "Why does `estimate_tokens()` connect `Community 0` to `Community 4`, `Community 7`?",
+      "why": "High betweenness centrality (0.030) - this node is a cross-community bridge."
+    },
+    {
+      "type": "verify_inferred",
+      "question": "Are the 3 inferred relationships involving `is_gpu_busy()` (e.g. with `route()` and `moe_spillover()`) actually correct?",
+      "why": "`is_gpu_busy()` has 3 INFERRED edges - model-reasoned connections that need verification."
+    },
+    {
+      "type": "verify_inferred",
+      "question": "Are the 3 inferred relationships involving `select_best_gpu()` (e.g. with `route()` and `route()`) actually correct?",
+      "why": "`select_best_gpu()` has 3 INFERRED edges - model-reasoned connections that need verification."
+    },
+    {
+      "type": "isolated_nodes",
+      "question": "What connects `SyslogAI Harness Dashboard \u2014 Modern Design.`, `maintenance.sh script`, `Nginx upstream health probe. Returns 200 if service is alive.` to the rest of the system?",
+      "why": "28 weakly-connected nodes found - possible documentation gaps or missing edges."
+    }
+  ]
+}

graphify-out/.graphify_detect.json

@@ -0,0 +1 @@
+{"files": {"code": ["/root/syslog-harness-repo/dashboard/dashboard.py", "/root/syslog-harness-repo/maintenance.sh", "/root/syslog-harness-repo/phase0-dual-keys.json", "/root/syslog-harness-repo/queue-service/queue-service.py", "/root/syslog-harness-repo/router/http_patch.py", "/root/syslog-harness-repo/router/route_v2.py", "/root/syslog-harness-repo/router/route_v3.py", "/root/syslog-harness-repo/router/router.py"], "document": ["/root/syslog-harness-repo/LITELLM-MIGRATION-PLAN.md", "/root/syslog-harness-repo/MIGRATION_PLAN.md", "/root/syslog-harness-repo/README.md", "/root/syslog-harness-repo/dashboard/dashboard.html", "/root/syslog-harness-repo/dashboard/harness.html", "/root/syslog-harness-repo/dashboard/requirements.txt", "/root/syslog-harness-repo/docker-compose.yml", "/root/syslog-harness-repo/litellm_config.yaml", "/root/syslog-harness-repo/router/requirements.txt", "/root/syslog-harness-repo/ssl/README.md"], "paper": [], "image": [], "video": []}, "total_files": 18, "total_words": 14667, "needs_graph": false, "warning": "Corpus is ~14,667 words - fits in a single context window. You may not need a graph.", "skipped_sensitive": ["/root/syslog-harness-repo/.env.example"], "unclassified": ["/root/syslog-harness-repo/.gitignore", "/root/syslog-harness-repo/Dockerfile.dashboard", "/root/syslog-harness-repo/Dockerfile.queue", "/root/syslog-harness-repo/backups/20260602_103344/dashboard.py.bak", "/root/syslog-harness-repo/backups/20260602_103344/router.py.bak", "/root/syslog-harness-repo/backups/20260602_103344/ts_patch.py.bak", "/root/syslog-harness-repo/dashboard/Dockerfile", "/root/syslog-harness-repo/docker-compose.yml.bak", "/root/syslog-harness-repo/gpu-router-docker.conf", "/root/syslog-harness-repo/gpu-router.conf", "/root/syslog-harness-repo/nginx/nginx.conf", "/root/syslog-harness-repo/nginx/nginx.conf.bak", "/root/syslog-harness-repo/router/Dockerfile", "/root/syslog-harness-repo/router/router.py.bak.20260518074236"], "graphifyignore_patterns": 3, "scan_root": "/root/syslog-harness-repo"}

graphify-out/.graphify_semantic.json

@@ -0,0 +1 @@
+{"nodes": [], "edges": [], "hyperedges": [], "input_tokens": 0, "output_tokens": 0}

graphify-out/GRAPH_REPORT.md

@@ -0,0 +1,106 @@
+# Graph Report - .  (2026-07-08)
+
+## Corpus Check
+- Corpus is ~14,667 words - fits in a single context window. You may not need a graph.
+
+## Summary
+- 91 nodes · 151 edges · 14 communities (9 shown, 5 thin omitted)
+- Extraction: 93% EXTRACTED · 7% INFERRED · 0% AMBIGUOUS · INFERRED: 10 edges (avg confidence: 0.77)
+- Token cost: 0 input · 0 output
+
+## Community Hubs (Navigation)
+- Community 0
+- Community 1
+- Community 2
+- Community 3
+- Community 4
+- Community 5
+- Community 6
+- Community 7
+- Community 8
+- Community 9
+- Community 10
+- Community 11
+- Community 12
+
+## God Nodes (most connected - your core abstractions)
+1. `chat()` - 12 edges
+2. `get_redis()` - 11 edges
+3. `gpu_active_count()` - 9 edges
+4. `is_gpu_busy()` - 9 edges
+5. `select_best_gpu()` - 8 edges
+6. `route()` - 8 edges
+7. `route()` - 6 edges
+8. `check_gpu_health()` - 6 edges
+9. `available_models()` - 6 edges
+10. `estimate_tokens()` - 6 edges
+
+## Surprising Connections (you probably didn't know these)
+- `route()` --calls--> `available_models()`  [INFERRED]
+  router/route_v2.py → router/router.py
+- `route()` --calls--> `estimate_tokens()`  [INFERRED]
+  router/route_v2.py → router/router.py
+- `route()` --calls--> `is_gpu_busy()`  [INFERRED]
+  router/route_v2.py → router/router.py
+- `route()` --calls--> `select_best_gpu()`  [INFERRED]
+  router/route_v2.py → router/router.py
+- `route()` --calls--> `available_models()`  [INFERRED]
+  router/route_v3.py → router/router.py
+
+## Import Cycles
+- None detected.
+
+## Communities (14 total, 5 thin omitted)
+
+### Community 0 - "Community 0"
+Cohesion: 0.21
+Nodes (14): route(), moe_spillover(), Spill 40% of MoE-first traffic to Dense to prevent Strix Halo overheating.     O, route(), available_models(), estimate_tokens(), is_gpu_busy(), moe_spillover() (+6 more)
+
+### Community 1 - "Community 1"
+Cohesion: 0.22
+Nodes (4): api_state(), broadcast_loop(), fetch_state(), SyslogAI Harness Dashboard — Modern Design.
+
+### Community 2 - "Community 2"
+Cohesion: 0.31
+Nodes (9): check_gpu_health(), enqueue(), get_queue_depth(), get_redis(), health(), GET queue depth + circuit breaker state + GPU health., Nginx upstream health probe. Returns 200 if service is alive., Fallback endpoint — Nginx calls this when all GPU upstreams are down. (+1 more)
+
+### Community 3 - "Community 3"
+Cohesion: 0.20
+Nodes (10): _admin_auth(), admin_deprecation_summary(), admin_generate_key(), admin_keys(), admin_revoke_key(), Require admin key for management endpoints., List all API keys (masked) with agent, tier, and deprecation status., Summary of deprecated key usage (from Redis logs, if available). (+2 more)
+
+### Community 4 - "Community 4"
+Cohesion: 0.28
+Nodes (5): bcast(), get_metrics(), metrics(), metrics_circuit_breaker(), Expose circuit breaker status per model. Phase 1.
+
+### Community 5 - "Community 5"
+Cohesion: 0.22
+Nodes (9): get_redis(), gpu_decr(), gpu_incr(), half_open_probe(), is_circuit_tripped(), performance(), Check if a GPU host is currently blacklisted., Check if a GPU host can be un-blacklisted. (+1 more)
+
+### Community 6 - "Community 6"
+Cohesion: 0.36
+Nodes (8): check_gpu_health(), gpu_active_count(), gpu_health_score(), health(), metrics_gpu_health(), Get number of in-flight requests for a GPU., Score a GPU based on VRAM, temperature, and load. Lower = better., Live GPU health scores + circuit breaker + KPIs.
+
+### Community 7 - "Community 7"
+Cohesion: 0.33
+Nodes (7): chat(), check_rate_limit(), clean_response(), clean_unicode(), Store detailed performance record in Redis for analytics., Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remainin, store_perf_record()
+
+## Knowledge Gaps
+- **1 isolated node(s):** `maintenance.sh script`
+  These have ≤1 connection - possible missing edges or undocumented components.
+- **5 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
+
+## Suggested Questions
+_Questions this graph is uniquely positioned to answer:_
+
+- **Why does `is_gpu_busy()` connect `Community 0` to `Community 4`, `Community 6`?**
+  _High betweenness centrality (0.061) - this node is a cross-community bridge._
+- **Why does `select_best_gpu()` connect `Community 0` to `Community 4`, `Community 6`?**
+  _High betweenness centrality (0.031) - this node is a cross-community bridge._
+- **Why does `estimate_tokens()` connect `Community 0` to `Community 4`, `Community 7`?**
+  _High betweenness centrality (0.030) - this node is a cross-community bridge._
+- **Are the 3 inferred relationships involving `is_gpu_busy()` (e.g. with `route()` and `moe_spillover()`) actually correct?**
+  _`is_gpu_busy()` has 3 INFERRED edges - model-reasoned connections that need verification._
+- **Are the 3 inferred relationships involving `select_best_gpu()` (e.g. with `route()` and `route()`) actually correct?**
+  _`select_best_gpu()` has 3 INFERRED edges - model-reasoned connections that need verification._
+- **What connects `SyslogAI Harness Dashboard — Modern Design.`, `maintenance.sh script`, `Nginx upstream health probe. Returns 200 if service is alive.` to the rest of the system?**
+  _28 weakly-connected nodes found - possible documentation gaps or missing edges._

graphify-out/cache/ast/v0.9.10/53edfda5145db02bc35b9e9d8e06e5c731ce50c3a50cec3db65b3b886bed95a6.json

@@ -0,0 +1 @@
+{"nodes": [{"id": "root_syslog_harness_repo_router_http_patch_py", "label": "http_patch.py", "file_type": "code", "source_file": "router/http_patch.py", "source_location": "L1"}], "edges": [{"source": "root_syslog_harness_repo_router_http_patch_py", "target": "re", "relation": "imports", "context": "import", "confidence": "EXTRACTED", "source_file": "router/http_patch.py", "source_location": "L2", "weight": 1.0}], "raw_calls": []}

graphify-out/cache/ast/v0.9.10/801feb2c0b7f4052cb2c4ef933d295d3494c8791766687b891c04cc7b5495d09.json

@@ -0,0 +1 @@
+{"nodes": [{"id": "root_syslog_harness_repo_maintenance_sh", "label": "maintenance.sh", "file_type": "code", "source_file": "maintenance.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "root_syslog_harness_repo_maintenance_sh__entry", "label": "maintenance.sh script", "file_type": "code", "source_file": "maintenance.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "root_syslog_harness_repo_maintenance_sh", "target": "root_syslog_harness_repo_maintenance_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "maintenance.sh", "source_location": "L1", "weight": 1.0}]}

graphify-out/cache/stat-index.json

@@ -0,0 +1 @@
+{"/root/syslog-harness-repo/LITELLM-MIGRATION-PLAN.md":{"size":31561,"mtime_ns":1781468608211508160,"word_count":3762},"/root/syslog-harness-repo/MIGRATION_PLAN.md":{"size":1966,"mtime_ns":1781121183850310435,"word_count":303},"/root/syslog-harness-repo/README.md":{"size":2447,"mtime_ns":1781121183850310435,"word_count":377},"/root/syslog-harness-repo/dashboard/dashboard.html":{"size":8331,"mtime_ns":1781316813360201927,"word_count":415},"/root/syslog-harness-repo/dashboard/dashboard.py":{"size":29010,"mtime_ns":1781121183851178603,"word_count":1615,"hash":"654e97aa6df1872b0717f32f676c067d15b9795a0dc04a86936730a7e133c214"},"/root/syslog-harness-repo/dashboard/harness.html":{"size":23388,"mtime_ns":1781316813362649901,"word_count":1869},"/root/syslog-harness-repo/dashboard/requirements.txt":{"size":30,"mtime_ns":1781121183851178603,"word_count":2},"/root/syslog-harness-repo/docker-compose.yml":{"size":3900,"mtime_ns":1782224658037786319,"word_count":200},"/root/syslog-harness-repo/litellm_config.yaml":{"size":618,"mtime_ns":1781121183851178603,"word_count":39},"/root/syslog-harness-repo/maintenance.sh":{"size":1302,"mtime_ns":1781121183851178603,"word_count":192,"hash":"801feb2c0b7f4052cb2c4ef933d295d3494c8791766687b891c04cc7b5495d09"},"/root/syslog-harness-repo/phase0-dual-keys.json":{"size":1628,"mtime_ns":1781121183851610673,"word_count":110,"hash":"0f115313f2c86df2f57e48bdf180b768d6452d0337bae4e3c3d2b52b9f5267a7"},"/root/syslog-harness-repo/queue-service/queue-service.py":{"size":3284,"mtime_ns":1781121183851610673,"word_count":325,"hash":"c9a9d2e5c9f1fb7d35a73f91c83a047ce92dc2d09c9ddab8fc2d6e82d92a093d"},"/root/syslog-harness-repo/router/http_patch.py":{"size":3562,"mtime_ns":1781121183851788825,"word_count":290,"hash":"53edfda5145db02bc35b9e9d8e06e5c731ce50c3a50cec3db65b3b886bed95a6"},"/root/syslog-harness-repo/router/requirements.txt":{"size":43,"mtime_ns":1781121183851788825,"word_count":3},"/root/syslog-harness-repo/router/route_v2.py":{"size":3954,"mtime_ns":1781121183851788825,"word_count":435,"hash":"e96ac2ae879e0a876d4190daff4cc566d13798b97c4820f3efa47e06627bbbcb"},"/root/syslog-harness-repo/router/route_v3.py":{"size":4703,"mtime_ns":1781121183851788825,"word_count":504,"hash":"30b202626b1f50da90b78272001e253cc7086b0bdc52e46553414c5313e3fb4e"},"/root/syslog-harness-repo/router/router.py":{"size":44307,"mtime_ns":1781316813360201927,"word_count":4198,"hash":"9c6817ae09f6bb5b2862f0282cacab7d4ce5da051e4ddc95e2e4099c7bf09716"},"/root/syslog-harness-repo/ssl/README.md":{"size":204,"mtime_ns":1781121183851788825,"word_count":28}}

litellm_config.yaml

@@ -1,25 +1,21 @@
 model_list:
-  - model_name: qwen3.6-35B-A3B
+- model_name: qwen3.6-35B-A3B
   litellm_params:
     model: openai/qwen3.6-35B-A3B
     api_base: http://192.168.68.15:8080/v1
-      api_key: "not-needed"
+    api_key: not-needed
-
+- model_name: gpu-dense
-  - model_name: qwen3.6-27B-code
   litellm_params:
     model: openai/qwen3.6-27B-code-text
     api_base: http://192.168.68.8:8080/v1
-      api_key: "not-needed"
+    api_key: not-needed
-
+- model_name: gpu-light
-  - model_name: gemma-4-12b
   litellm_params:
     model: openai/gemma-4-12b
     api_base: http://192.168.68.110:8080/v1
-      api_key: "not-needed"
+    api_key: not-needed
-
 general_settings:
   master_key: sk-syslog-local-master-key
-
 litellm_settings:
   drop_params: true
   request_timeout: 120

litellm_config.yaml.backup

@@ -0,0 +1,25 @@
+model_list:
+  - model_name: qwen3.6-35B-A3B
+    litellm_params:
+      model: openai/qwen3.6-35B-A3B
+      api_base: http://192.168.68.15:8080/v1
+      api_key: "not-needed"
+
+  - model_name: qwen3.6-27B-code
+    litellm_params:
+      model: openai/qwen3.6-27B-code-text
+      api_base: http://192.168.68.8:8080/v1
+      api_key: "not-needed"
+
+  - model_name: gemma-4-12b
+    litellm_params:
+      model: openai/gemma-4-12b
+      api_base: http://192.168.68.110:8080/v1
+      api_key: "not-needed"
+
+general_settings:
+  master_key: sk-syslog-local-master-key
+
+litellm_settings:
+  drop_params: true
+  request_timeout: 120

maintenance.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# SyslogAI Harness — Automated Maintenance
+# Runs daily via cron
+
+LOG="/var/log/harness-maintenance.log"
+echo "=== $(date) ===" >> "$LOG"
+
+# 1. Clean Redis timeseries keys older than 60 days
+CUTOFF=$(date -d "60 days ago" +%Y%m%d%H)
+echo "Redis: removing ts:* keys older than $CUTOFF" >> "$LOG"
+DELETED=0
+for key in $(docker exec harness-redis redis-cli KEYS "ts:*" 2>/dev/null); do
+    TS=$(echo "$key" | grep -oP '\d{10}$')
+    if [ -n "$TS" ] && [ "$TS" -lt "$CUTOFF" ] 2>/dev/null; then
+        docker exec harness-redis redis-cli DEL "$key" > /dev/null 2>&1
+        DELETED=$((DELETED + 1))
+    fi
+done
+echo "Redis: deleted $DELETED stale timeseries keys" >> "$LOG"
+
+# 2. Log stale model keys (leftover from migrations)
+STALE=$(docker exec harness-redis redis-cli KEYS "*gemma*" 2>/dev/null)
+if [ -n "$STALE" ]; then
+    echo "WARNING: stale gemma keys found: $STALE" >> "$LOG"
+fi
+
+# 3. Prune Docker build cache (older than 7 days)
+echo "Docker: pruning build cache" >> "$LOG"
+docker builder prune -f --filter until=168h >> "$LOG" 2>&1
+
+# 4. Log container health status
+docker ps --format "table {{.Names}}\t{{.Status}}\t{{.RunningFor}}" >> "$LOG" 2>&1
+
+# 5. Log Redis memory
+docker exec harness-redis redis-cli INFO memory | grep used_memory_human >> "$LOG" 2>&1
+
+echo "" >> "$LOG"

nginx/nginx.conf

@@ -8,23 +8,30 @@ http {
     include /etc/nginx/mime.types;
     default_type application/octet-stream;
 
-    log_format main  launching rt=;
+    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+                    '$status $body_bytes_sent "$http_referer" '
+                    '"$http_user_agent" rt=$request_time';
     access_log /var/log/nginx/access.log main;
     error_log /var/log/nginx/error.log;
     sendfile on;
     keepalive_timeout 65;
 
-    upstream router_api { server router:9000; }
+    upstream router_api { server host.docker.internal:9000; }
     upstream dashboard_ui { server dashboard:3000; }
     upstream litellm_backend { server litellm:4000; }
 
     server {
         listen 80;
 
+        # Security headers
+        add_header X-Content-Type-Options nosniff always;
+        add_header X-Frame-Options SAMEORIGIN always;
+        add_header X-XSS-Protection "1; mode=block" always;
+
         # Disable buffering for SSE streams
         proxy_buffering off;
 
-        # API — through router
+        # API  through router
         location /v1/ {
             proxy_pass http://router_api;
             proxy_http_version 1.1;
@@ -36,6 +43,14 @@ http {
             proxy_buffering off;
         }
 
+        location /admin/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
+            proxy_set_header Authorization $http_authorization;
+        }
+
         # SSE streaming endpoint
         location /stream {
             proxy_pass http://router_api;
@@ -63,7 +78,15 @@ http {
             proxy_set_header Authorization $http_authorization;
         }
 
-        # Dashboard
+        # Professional Dashboard (Phase 1-3) - Static HTML served via Nginx
+        location /dashboard/ {
+            alias /opt/inference-harness/dashboard/;
+            index dashboard.html;
+            add_header Cache-Control "public, max-age=3600";
+            add_header X-Content-Type-Options nosniff;
+        }
+
+        # Legacy Dashboard (root) - Proxy to Flask app
         location / {
             proxy_pass http://dashboard_ui;
             proxy_http_version 1.1;
@@ -71,9 +94,25 @@ http {
             proxy_buffering off;
         }
 
+        # Performance analytics
+        location /metrics/ {
+            proxy_pass http://router_api;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
+        # Circuit Breaker metrics (Phase 1)
+        location /metrics/circuit-breaker {
+            proxy_pass http://router_api/metrics/circuit-breaker;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+        }
+
         location /health {
-            return 200 "{\"status\":\"healthy\"}";
+            proxy_pass http://router_api/health;
-            add_header Content-Type application/json;
+            proxy_http_version 1.1;
+            proxy_set_header Host $host;
+            proxy_set_header X-Real-IP $remote_addr;
         }
     }
 }

router/http_patch.py

@@ -0,0 +1,90 @@
+# Insert streaming support before the gpu_resp call
+import re
+with open('/opt/inference-harness/router/router.py') as f:
+    code = f.read()
+
+# Find the gpu_resp block and replace with streaming-aware version
+old = '''        start = time.time()
+        gpu_resp = requests.post(
+            gpu_url + "/chat/completions",
+            json=req_data,
+            headers={"Content-Type": "application/json", "Authorization": "Bearer not-needed"},
+            timeout=120,
+        )
+        latency_ms = int((time.time() - start) * 1000)
+
+        if gpu_resp.status_code != 200:
+            log.error("GPU error: %s %s", gpu_resp.status_code, gpu_resp.text[:200])
+            return jsonify({"error": "GPU backend returned " + str(gpu_resp.status_code)}), 502
+
+        response_data = gpu_resp.json()
+        response_data = fix_reasoning_content(response_data)
+
+        response_data["routing"] = {
+            "model": model, "reason": reason, "gpu": gpu_url,
+            "tier": tier, "agent": agent, "latency_ms": latency_ms,
+        }
+
+        return jsonify(response_data)'''
+
+new = '''        start = time.time()
+        is_stream = req_data.get("stream", False)
+        
+        gpu_resp = requests.post(
+            gpu_url + "/chat/completions",
+            json=req_data,
+            headers={"Content-Type": "application/json", "Authorization": "Bearer not-needed"},
+            timeout=120,
+            stream=is_stream,
+        )
+        latency_ms = int((time.time() - start) * 1000)
+
+        if gpu_resp.status_code != 200:
+            log.error("GPU error: %s %s", gpu_resp.status_code, gpu_resp.text[:200])
+            return jsonify({"error": "GPU backend returned " + str(gpu_resp.status_code)}), 502
+
+        if is_stream:
+            # Stream response back to client
+            def generate():
+                first = True
+                for line in gpu_resp.iter_lines(decode_unicode=True):
+                    if line:
+                        if first and line.startswith("data: "):
+                            # Inject routing into first chunk
+                            try:
+                                chunk = json.loads(line[6:])
+                                chunk["routing"] = {
+                                    "model": model, "reason": reason, "gpu": gpu_url,
+                                    "tier": tier, "agent": agent, "latency_ms": latency_ms,
+                                }
+                                yield "data: " + json.dumps(chunk) + "\n\n"
+                                first = False
+                                continue
+                            except Exception:
+                                pass
+                        yield line + "\n"
+                yield "data: [DONE]\n\n"
+            return Response(stream_with_context(generate()), mimetype="text/event-stream")
+
+        response_data = gpu_resp.json()
+        response_data = fix_reasoning_content(response_data)
+
+        response_data["routing"] = {
+            "model": model, "reason": reason, "gpu": gpu_url,
+            "tier": tier, "agent": agent, "latency_ms": latency_ms,
+        }
+
+        return jsonify(response_data)'''
+
+code = code.replace(old, new)
+
+# Add missing import
+if 'from flask import Flask, request, jsonify' in code:
+    code = code.replace(
+        'from flask import Flask, request, jsonify',
+        'from flask import Flask, request, jsonify, Response, stream_with_context'
+    )
+
+with open('/opt/inference-harness/router/router.py', 'w') as f:
+    f.write(code)
+print('Streaming support added')

router/router.py

@@ -2,6 +2,50 @@ import os, json, time, logging, traceback, threading, queue, statistics, math
 import requests, redis
 from flask import Flask, request, jsonify, Response, stream_with_context
 
+
+
+# Phase 2: Atomic session token update  Redis Lua script
+SESSION_LUA_SCRIPT = """
+local key = KEYS[1]
+local new_val = tonumber(ARGV[1])
+local current = tonumber(redis.call('GET', key) or 0)
+local max_val = math.max(current, new_val)
+redis.call('SET', key, max_val, 'EX', 86400)
+return max_val
+"""
+
+# Phase 4: Atomic GPU slot booking (closes TOCTOU race between check and incr)
+SLOT_BOOK_LUA = """
+local key = KEYS[1]
+local max_c = tonumber(ARGV[1])
+local current = tonumber(redis.call('GET', key) or '0')
+if current < max_c then
+    redis.call('INCR', key)
+    return 1
+else
+    return 0
+end
+"""
+SLOT_RELEASE_LUA = """
+local key = KEYS[1]
+local current = tonumber(redis.call('GET', key) or '0')
+if current > 0 then
+    redis.call('DECR', key)
+end
+current = tonumber(redis.call('GET', key) or '0')
+if current < 0 then
+    redis.call('SET', key, '0')
+end
+return redis.call('GET', key)
+"""
+
+# Phase 3b: Configurable health scoring weights (env-overridable)
+HEALTH_WEIGHT_VRAM = float(os.environ.get("HEALTH_WEIGHT_VRAM", "0.40"))
+HEALTH_WEIGHT_TEMP = float(os.environ.get("HEALTH_WEIGHT_TEMP", "0.30"))
+HEALTH_WEIGHT_LOAD = float(os.environ.get("HEALTH_WEIGHT_LOAD", "0.30"))
+HEALTH_TEMP_BASELINE = int(os.environ.get("HEALTH_TEMP_BASELINE", "30"))
+
+
 REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
 GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
 GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
@@ -18,10 +62,17 @@ GPU_URLS = {
     "gemma-4-12b": GPU_LIGHT_URL,
 }
 # Max concurrent requests per GPU (based on llama.cpp --parallel)
+
+GPU_LABELS = {
+    "qwen3.6-35B-A3B": "Qwen3.6 35B (Strix Halo)",
+    "qwen3.6-27B-code": "Qwen3.6 27B Code (RTX 3090)",
+    "gemma-4-12b": "Gemma-4 12B (RTX 5070)",
+}
+
 GPU_MAX_CONCURRENT = {
     "qwen3.6-35B-A3B": 2,   # 2 slots (cross-agent spread prevents overheating)
-    "qwen3.6-27B-code": 2,  # 2 slots
+    "qwen3.6-27B-code": 2,  # 2 slots (128K context frees VRAM)
-    "gemma-4-12b": 2,       # 2 slots (12GB VRAM, 4GB headroom)
+    "gemma-4-12b": 2,       # 2 slots (7.1GB VRAM)
 }
 
 # Context window sizes (tokens) — used for compaction signals
@@ -36,18 +87,11 @@ TIER_MODELS = {
     "professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
     "enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
 }
-# ── PHASE 0: Dual-Key API Key System ──
+# API keys loaded from API_KEYS env var (set in docker-compose.yml)
-# API_KEYS env var is REQUIRED (JSON string). No hardcoded fallback.
+# Fallback is dev-only — production MUST set API_KEYS env var
-# Format: {"sk-xxx": {"tier": "enterprise", "agent": "Name"}}
+API_KEYS = json.loads(os.environ.get("API_KEYS", json.dumps({
-# Deprecated keys get {"deprecated": true} — still accepted, logged with warning.
+    "sk-dev-local-only": {"tier": "enterprise", "agent": "dev"},
-_raw_keys = os.environ.get("API_KEYS")
+})))
-if not _raw_keys:
-    raise RuntimeError("FATAL: API_KEYS environment variable is required. "
-                       "Set it in docker-compose.yml or .env file. "
-                       "No hardcoded keys fallback — this is a security feature.")
-API_KEYS = json.loads(_raw_keys)
-log.info("Loaded %d API keys from env var (%d deprecated)",
-         len(API_KEYS), sum(1 for v in API_KEYS.values() if v.get("deprecated")))
 # Rate limits: requests per minute per API key tier
 RATE_LIMIT_RPM = {
     "enterprise": 120,
@@ -57,11 +101,11 @@ RATE_LIMIT_RPM = {
 
 def check_rate_limit(api_key, tier):
     """Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remaining, reset_seconds)."""
-    if not r:
+    if not get_redis():
         return True, 999, 60
     limit = RATE_LIMIT_RPM.get(tier, 30)
     key = f"ratelimit:{api_key}"
-    current = int(r.get(key) or 0)
+    current = int(get_redis().get(key) or 0)
     if current >= limit:
         ttl = r.ttl(key)
         retry = max(ttl, 1) if ttl and ttl > 0 else 60
@@ -77,7 +121,25 @@ def check_rate_limit(api_key, tier):
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
 log = logging.getLogger("router")
-try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
+# Redis connection — initialized lazily, retries on first use
+def get_redis():
+    global r
+    if r is not None:
+        try:
+            r.ping()
+            return r
+        except Exception:
+            r = None
+    try:
+        r = redis.from_url(REDIS_URL, decode_responses=True)
+        r.ping()
+        return r
+    except Exception:
+        return None
+
+r = None
+try: get_redis()
+except Exception: pass
 except Exception: r = None
 
 
@@ -85,7 +147,7 @@ def counter_audit_loop():
     """Every 30s, check GPU slots and reset counters if all slots idle."""
     while True:
         time.sleep(30)
-        if not r: continue
+        if not get_redis(): continue
         for model, url in GPU_URLS.items():
             try:
                 resp = requests.get(url.replace("/v1","") + "/slots",
@@ -94,9 +156,9 @@ def counter_audit_loop():
                     slots = resp.json()
                     all_idle = all(not s.get("is_processing", False) for s in slots)
                     if all_idle:
-                        current = int(r.get("active:" + model) or 0)
+                        current = int(get_redis().get("active:" + model) or 0)
                         if current > 0:
-                            r.set("active:" + model, 0)
+                            get_redis().set("active:" + model, 0)
                             log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
             except Exception:
                 pass
@@ -113,14 +175,44 @@ def gpu_active_count(model):
     return 0
 
 def gpu_incr(model):
-    if r: r.incr("active:" + model)
+    if get_redis(): get_redis().incr("active:" + model)
 
 def gpu_decr(model):
-    if r:
+    rd = get_redis()
-        v = r.decr("active:" + model)
+    if rd:
+        v = rd.decr("active:" + model)
         if v and int(v) < 0:
-            r.set("active:" + model, 0)  # never go negative
+            get_redis().set("active:" + model, 0)  # never go negative
 
+# Phase 4: Atomic GPU slot booking (Lua-based, closes TOCTOU race)
+def gpu_book_slot(model):
+    """Atomically book a GPU slot. Returns True if acquired, False if full."""
+    rd = get_redis()
+    if not rd:
+        return True  # No Redis — allow everything (degraded mode)
+    try:
+        max_c = GPU_MAX_CONCURRENT.get(model, 1)
+        result = rd.eval(SLOT_BOOK_LUA, 1, "active:" + model, max_c)
+        return result == 1
+    except Exception:
+        # Lua not loaded — fall back to non-atomic
+        current = int(rd.get("active:" + model) or 0)
+        if current < GPU_MAX_CONCURRENT.get(model, 1):
+            rd.incr("active:" + model)
+            return True
+        return False
+
+def gpu_release_slot(model):
+    """Atomically release a GPU slot. Never goes negative."""
+    rd = get_redis()
+    if not rd:
+        return
+    try:
+        rd.eval(SLOT_RELEASE_LUA, 1, "active:" + model)
+    except Exception:
+        v = rd.decr("active:" + model)
+        if v and int(v) < 0:
+            rd.set("active:" + model, 0)
 def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
     url = GPU_SIDECARS.get(model)
     if not url: return {"status": "unknown"}
@@ -151,7 +243,7 @@ def estimate_tokens(msgs):
 
 def store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, stream):
     """Store detailed performance record in Redis for analytics."""
-    if not r: return
+    if not get_redis(): return
     try:
         total_ms = queue_ms + inference_ms
         tps = completion_tokens / (inference_ms / 1000) if inference_ms > 0 and completion_tokens > 0 else 0
@@ -187,6 +279,27 @@ def is_gpu_busy(model):
     max_c = GPU_MAX_CONCURRENT.get(model, 1)
     return active >= max_c
 
+
+
+# Phase 3: Dynamic GPU Weighting (Health Score)
+def gpu_health_score(model):
+    """Score a GPU based on VRAM, temperature, power, and load. Lower = better.
+    Weights configurable via HEALTH_WEIGHT_VRAM/TEMP/LOAD env vars."""
+    h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
+    if h.get("status") == "down":
+        return 999  # never pick down GPUs
+    if is_circuit_tripped(model):
+        return 998  # circuit open — skip but distinguishable from down
+    vram_pct = h.get("vram_pct") or 50
+    temp_c = h.get("temp_c") or 50
+    power_w = h.get("power_w") or 100
+    active = gpu_active_count(model)
+    max_c = GPU_MAX_CONCURRENT.get(model, 1)
+    load_pct = (active / max_c) * 100 if max_c > 0 else 0
+    temp_penalty = max(0, (temp_c or 50) - HEALTH_TEMP_BASELINE)
+    score = (vram_pct or 0) * HEALTH_WEIGHT_VRAM + temp_penalty * 0.5 * HEALTH_WEIGHT_TEMP + load_pct * HEALTH_WEIGHT_LOAD
+    return round(score, 1)
+
 def select_best_gpu(candidates, reason, agent=""):
     """Pick best GPU, spreading agents across GPUs to prevent hotspots."""
     # Count how many distinct agents are on each GPU
@@ -198,17 +311,19 @@ def select_best_gpu(candidates, reason, agent=""):
                 if r.get("agent_gpu:" + ak["agent"] + ":" + m):
                     count += 1
             gpu_agent_counts[m] = count
+    # Phase 3: Sort candidates by health score before selection
+    sorted_candidates = sorted(candidates, key=gpu_health_score)
     # First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
-    for m in candidates:
+    for m in sorted_candidates:
         if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
             return {"model": m, "reason": reason}
     # Second pass: prefer GPU this agent is NOT already on (skip own GPU)
     if agent:
-        for m in candidates:
+        for m in sorted_candidates:
             if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
                 return {"model": m, "reason": reason}
     # Third pass: any non-busy GPU
-    for m in candidates:
+    for m in sorted_candidates:
         if not is_gpu_busy(m):
             return {"model": m, "reason": reason}
     # All busy — pick least loaded
@@ -223,27 +338,93 @@ def select_best_gpu(candidates, reason, agent=""):
         return {"model": best, "reason": "load_balanced_" + reason}
     return None
 
+
+
+
+
+# Phase 1: Circuit Breaker for GPU Hosts (Approved by Abiba)
+CIRCUIT_FAIL_THRESHOLD = int(os.environ.get("CIRCUIT_FAIL_THRESHOLD", "3"))
+CIRCUIT_FAIL_WINDOW = int(os.environ.get("CIRCUIT_FAIL_WINDOW", "120"))
+CIRCUIT_COOLDOWN = int(os.environ.get("CIRCUIT_COOLDOWN", "60"))
+
+def is_circuit_tripped(model):
+    """Check if a GPU host is currently blacklisted."""
+    if not get_redis():
+        return False
+    return r.exists("circuit:" + model + ":open")
+
+def trip_circuit(model, duration=None):
+    """Blacklist a GPU host for specified duration (default CIRCUIT_COOLDOWN).
+    Only trips after CIRCUIT_FAIL_THRESHOLD failures within CIRCUIT_FAIL_WINDOW."""
+    if not get_redis():
+        return False
+    if duration is None:
+        duration = CIRCUIT_COOLDOWN
+    now = time.time()
+    fail_key = "circuit:" + model + ":failures"
+    pipe = r.pipeline()
+    pipe.lpush(fail_key, str(now))
+    pipe.ltrim(fail_key, 0, CIRCUIT_FAIL_THRESHOLD - 1)
+    pipe.lrange(fail_key, 0, -1)
+    results = pipe.execute()
+    failures = [float(f) for f in (results[-1] if results else [])]
+    recent = [f for f in failures if now - f <= CIRCUIT_FAIL_WINDOW]
+    if len(recent) >= CIRCUIT_FAIL_THRESHOLD:
+        key = "circuit:" + model + ":open"
+        r.set(key, 1, ex=duration)
+        r.incr("circuit:" + model + ":count")
+        log.warning("CIRCUIT_TRIPPED: %s — %d failures in %ds, cooldown %ds", 
+                   model, len(recent), CIRCUIT_FAIL_WINDOW, duration)
+        return True
+    return False
+
+def half_open_probe(model):
+    """Check if a GPU host can be un-blacklisted."""
+    if not get_redis():
+        return True
+    key = "circuit:" + model + ":open"
+    if not r.exists(key):
+        return True  # no circuit
+    return False  # still open
+
 def route(rd, tier, agent=""):
     msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
     sys = any(m.get("role")=="system" for m in msgs)
     turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
     hints = rd.get("routing_hints",{})
     allowed = TIER_MODELS.get(tier, ["gemma-4-12b"])
-    avail = [m for m in available_models() if m in allowed]
+    # Phase 1: Filter out models with tripped circuit breakers
+    avail = [m for m in available_models() if m in allowed and not is_circuit_tripped(m)]
     if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
-    # Check if all available GPUs are at max capacity
     if all(is_gpu_busy(m) for m in avail):
         return {"model": avail[0], "reason": "all_saturated", "saturated": True}
 
+    # GUARD: multimodal -> VLM only (sole vision model)
+    has_image = any(
+        isinstance(m.get("content"), list) and
+        any(p.get("type") == "image_url" for p in m["content"] if isinstance(p, dict))
+        for m in msgs
+    )
+    if has_image:
+        if "gemma-4-12b" in avail and not is_gpu_busy("gemma-4-12b"):
+            return {"model": "gemma-4-12b", "reason": "vision"}
+        elif "gemma-4-12b" in avail:
+            return {"model": "gemma-4-12b", "reason": "vision_saturated", "saturated": True}
+        else:
+            return {"model": allowed[0], "reason": "vision_unavailable"}
+
     req = rd.get("model","auto")
+    # Map syslog-auto to auto for content-based routing
+    if req == "syslog-auto":
+        req = "auto"
     if req != "auto":
+        # STRICT MODE: no silent fallback — LiteLLM handles failover chains.
+        # Returns saturated if explicit GPU is busy (keeps per-model metrics accurate).
         target = req if req in avail else avail[0]
-        # If explicit model is busy, check if another can take it
+        if req not in avail:
-        if is_gpu_busy(target) and req in allowed:
+            return {"model": req, "reason": "explicit_unavailable", "saturated": True}
-            alts = [m for m in avail if m != target and m in allowed]
+        if is_gpu_busy(target):
-            if alts:
+            return {"model": target, "reason": "explicit_saturated", "saturated": True}
-                alt = select_best_gpu(alts, "explicit", agent)
-                if alt: return alt
         return {"model": target, "reason": "explicit"}
 
     if hints:
@@ -251,46 +432,58 @@ def route(rd, tier, agent=""):
             return select_best_gpu(["gemma-4-12b"], "hint_speed", agent) or {"model":"gemma-4-12b","reason":"hint_speed"}
         if hints.get("priority")=="quality" and "qwen3.6-35B-A3B" in avail:
             return select_best_gpu(["qwen3.6-35B-A3B"], "hint_quality", agent) or {"model":"qwen3.6-35B-A3B","reason":"hint_quality"}
+        if hints.get("priority")=="code" and "qwen3.6-27B-code" in avail:
+            return select_best_gpu(["qwen3.6-27B-code"], "hint_code", agent) or {"model":"qwen3.6-27B-code","reason":"hint_code"}
 
     first_msg = msgs[0].get("content","") if msgs else ""
     words = len(first_msg.split()) if isinstance(first_msg, str) else 99
 
-    # TIER 1: Lightweight — single-turn short queries → VLM (fastest)
+    # TIER 1: Tiny - single-turn micro queries -> VLM (fastest)
-    if not sys and turns <= 1 and t <= 500 and words <= 100 and "gemma-4-12b" in avail:
+    if not sys and turns <= 1 and t <= 300 and words <= 100 and "gemma-4-12b" in avail:
         if not is_gpu_busy("gemma-4-12b"):
-            return {"model":"gemma-4-12b","reason":"lightweight"}
+            return {"model":"gemma-4-12b","reason":"tiny"}
-        # VLM busy — Dense is faster for short queries than MoE
         fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
-        result = select_best_gpu(fallback, "lightweight_fallback", agent)
+        result = select_best_gpu(fallback, "tiny_fallback", agent)
         if result: return result
 
-    # TIER 2: Simple conversations — VLM primary (up to 15K tok), fastest for moderate chat
+    # TIER 2: Light - moderate chat -> VLM first (fastest), Dense fallback
-    if t <= 15000 and turns <= 12 and "gemma-4-12b" in avail:
+    if t <= 5000 and turns <= 4:
-        if not is_gpu_busy("gemma-4-12b"):
+        candidates = [m for m in ["gemma-4-12b","qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
-            return {"model":"gemma-4-12b","reason":"simple_conv"}
+        result = select_best_gpu(candidates, "light", agent)
-        # VLM busy — fall back to Dense, then MoE
-        fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
-        result = select_best_gpu(fallback, "simple_conv_fallback", agent)
         if result: return result
 
-    # TIER 3: Medium complexity — Dense primary, VLM fallback (quality + speed balance)
+    # TIER 3: Medium - quality matters -> MoE primary (60%), Dense spillover (40%)
-    if t <= 25000:
+    if t <= 30000:
-        candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
+        candidates = moe_spillover(avail, ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"])
         result = select_best_gpu(candidates, "medium", agent)
         if result: return result
 
-    # TIER 4: Heavy reasoning — MoE primary (workhorse), Dense fallback
+    # TIER 4: Heavy - quality first -> Dense primary, MoE fallback
-    if t > 25000:
+    if t > 30000:
-        candidates = [m for m in ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"] if m in avail]
+        candidates = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B","gemma-4-12b"] if m in avail]
-        result = select_best_gpu(candidates, "heavy_reasoning", agent)
+        result = select_best_gpu(candidates, "heavy", agent)
         if result: return result
 
-    # TIER 5: Default — Dense primary, MoE fallback
+    # TIER 5: Default - MoE primary (60%), Dense spillover (40%)
-    candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
+    candidates = moe_spillover(avail, ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"])
     result = select_best_gpu(candidates, "default", agent)
     if result: return result
     return {"model":avail[0],"reason":"last_resort"}
 
+
+def moe_spillover(avail, default_order):
+    """Spill 40% of MoE-first traffic to Dense to prevent Strix Halo overheating.
+    Only applies when MoE is first candidate, available, and not busy."""
+    import random
+    if (default_order[0] == "qwen3.6-35B-A3B" 
+        and "qwen3.6-35B-A3B" in avail 
+        and not is_gpu_busy("qwen3.6-35B-A3B")
+        and "qwen3.6-27B-code" in avail
+        and not is_gpu_busy("qwen3.6-27B-code")
+        and random.random() < 0.4):
+        # Swap: Dense first, MoE second
+        return ["qwen3.6-27B-code","qwen3.6-35B-A3B"] + [m for m in default_order[2:] if m in avail and m not in ("qwen3.6-27B-code","qwen3.6-35B-A3B")]
+    return [m for m in default_order if m in avail]
 def clean_unicode(text):
     if not isinstance(text, str): return text
     text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
@@ -344,6 +537,7 @@ def chat():
             return jsonify({"error": "Unauthorized — valid API key required"}), 401
         ki = API_KEYS[ak]
         tier, agent = ki["tier"], ki["agent"]
+        
         # Phase 0: dual-key transition — log deprecated key usage
         if ki.get("deprecated"):
             new_key = next((k for k, v in API_KEYS.items()
@@ -369,15 +563,14 @@ def chat():
         # Allow agent to override queue timeout via header
         q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
         
-        # Cross-turn context tracking: accumulate tokens per session
+        # Cross-turn context tracking: accumulate tokens per session (Phase 2: atomic Lua)
         session_id = request.headers.get("X-Session-Id", "")
         session_tokens = 0
         if session_id and r:
             try:
-                prev = int(r.get("session:" + session_id) or 0)
                 current = estimate_tokens(rd.get("messages",[]))
-                session_tokens = max(prev, current)  # context only grows
+                # Atomic GET/MAX/SET via Lua script  prevents race conditions
-                r.set("session:" + session_id, session_tokens, ex=86400)  # TTL 24h
+                session_tokens = r.eval(SESSION_LUA_SCRIPT, 1, "session:" + session_id, current)
             except Exception: pass
         
         d = route(rd, tier, agent)
@@ -399,14 +592,26 @@ def chat():
             log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
         model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
         
+        # Phase 4: Atomic slot booking (replaces non-atomic gpu_incr)
+        if not gpu_book_slot(model):
+            d = route(rd, tier, agent)
+            if d.get("saturated"):
+                resp = jsonify({"error": "All GPUs saturated", "retry_after_s": 3})
+                resp.headers["Retry-After"] = "3"
+                return resp, 503
+            model, reason = d["model"], d["reason"]
+            if not gpu_book_slot(model):
+                resp = jsonify({"error": "GPU slot race — retry", "retry_after_s": 1})
+                resp.headers["Retry-After"] = "1"
+                return resp, 503
+        url = GPU_URLS[model]
+        
         # Stash rate limit values for response headers
         _rl_remaining = rl_val
         _rl_limit = RATE_LIMIT_RPM.get(tier, 30)
         _rl_reset = reset_sec
         is_stream = rd.get("stream", False)
         
-        gpu_incr(model)
-        
         log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
         # Track which GPU this agent is using (TTL 120s covers typical request)
         if r and agent:
@@ -421,11 +626,14 @@ def chat():
             except Exception: pass
         start = time.time()
         resp = requests.post(url+"/chat/completions", json=rd,
-            headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
+            headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=900, stream=is_stream)
         lat = int((time.time()-start)*1000)
-        gpu_decr(model)
+        gpu_release_slot(model)
         
-        if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
+        if resp.status_code != 200:
+            if resp.status_code in (502, 504):
+                trip_circuit(model)
+            return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
         if is_stream:
             # Buffer SSE chunks, handle split lines for large responses
             chunks = []
@@ -471,6 +679,12 @@ def chat():
             sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
             sse_resp.headers["X-Context-Warning"] = ctx_warning
             sse_resp.headers["X-Context-Model"] = model
+            # LiteLLM spend tracking: best-effort token counts from stream timings
+            pt = stream_timings.get("prompt_n", 0) if stream_timings else 0
+            ct = stream_timings.get("predicted_n", 0) if stream_timings else 0
+            sse_resp.headers["X-Usage-Tokens"] = json.dumps({
+                "prompt_tokens": pt, "completion_tokens": ct, "model": model
+            })
             return sse_resp
         data = clean_response(resp.json())
         for c in data.get("choices",[]):
@@ -495,23 +709,28 @@ def chat():
         resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
         resp.headers["X-Context-Warning"] = ctx_warning
         resp.headers["X-Context-Model"] = model
+        # LiteLLM spend tracking: return token counts for cost computation
+        resp.headers["X-Usage-Tokens"] = json.dumps({
+            "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens, "model": model
+        })
         bcast()
         return resp
     except requests.Timeout:
-        gpu_decr(model)
+        gpu_release_slot(model)
-        log.error("TIMEOUT: %s -> %s", agent, model)
+        trip_circuit(model)
+        log.error("TIMEOUT: %s -> %s (Circuit tripped)", agent, model)
         return jsonify({"error":"timeout"}), 504
     except Exception as e:
-        gpu_decr(model)
+        gpu_release_slot(model)
         log.error("Error: %s\n%s", e, traceback.format_exc())
         return jsonify({"error":str(e)}), 500
 
 @app.route("/metrics/performance")
 def performance():
     """Per-request performance analytics with percentiles per model/reason/agent."""
-    if not r: return jsonify({"error": "Redis unavailable"}), 503
+    if not get_redis(): return jsonify({"error": "Redis unavailable"}), 503
     try:
-        window_hours = int(request.args.get("window", "24"))
+        window_hours = int(request.args.get("window", "24").replace("h",""))
         model_filter = request.args.get("model", "all")
 
         # Load recent records
@@ -626,9 +845,9 @@ def performance():
 @app.route("/metrics/scatter")
 def scatter():
     """Return individual data points for scatter plots (prompt_tokens vs latency)."""
-    if not r: return jsonify({"error": "Redis unavailable"}), 503
+    if not get_redis(): return jsonify({"error": "Redis unavailable"}), 503
     try:
-        window_hours = int(request.args.get("window", "24"))
+        window_hours = int(request.args.get("window", "24").replace("h",""))
         model_filter = request.args.get("model", "all")
         cutoff = time.time() - (window_hours * 3600)
         raw = r.lrange("perf:recent", 0, -1)
@@ -696,6 +915,105 @@ def metrics_timeseries():
             data["models"][model] = counts
     return jsonify(data)
 
+
+
+@app.route("/metrics/circuit-breaker")
+def metrics_circuit_breaker():
+    """Expose circuit breaker status per model. Phase 1."""
+    result = {}
+    if r:
+        for model in GPU_URLS:
+            key = "circuit:" + model + ":open"
+            duration = r.ttl(key)
+            trip_count = int(r.get("circuit:" + model + ":count") or 0)
+            result[model] = {
+                "tripped": r.exists(key),
+                "remaining_ttl": duration,
+                "trip_count": trip_count
+            }
+    return jsonify(result)
+
+
+@app.route("/metrics/gpu-health")
+def metrics_gpu_health():
+    """Live GPU health scores + circuit breaker + KPIs."""
+    result = {"gpus": [], "ts": time.time()}
+    for model in GPU_URLS:
+        h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
+        score = gpu_health_score(model)
+        active = gpu_active_count(model)
+        max_c = GPU_MAX_CONCURRENT.get(model, 1)
+        cb_tripped = bool(r and r.exists("circuit:" + model + ":open"))
+        cb_count = int(r.get("circuit:" + model + ":count") or 0) if r else 0
+        result["gpus"].append({
+            "id": model,
+            "label": GPU_LABELS.get(model, model),
+            "status": h.get("status", "unknown"),
+            "vram_pct": h.get("vram_pct", 0),
+            "temp_c": h.get("temp_c", 0),
+            "vram_used_mb": h.get("vram_used_mb", 0),
+            "vram_total_mb": h.get("vram_total_mb", 0),
+            "gpu_name": h.get("gpu_name", model),
+            "health_score": round(score, 1),
+            "active_requests": active,
+            "max_concurrent": max_c,
+            "circuit_tripped": cb_tripped,
+            "circuit_trip_count": cb_count
+        })
+    online = sum(1 for g in result["gpus"] if g["status"] in ("healthy", "saturated"))
+    trips = sum(g["circuit_trip_count"] for g in result["gpus"])
+    result["kpi"] = {"gpus_online": online, "total_trips": trips, "total_gpus": len(GPU_URLS)}
+    return jsonify(result)
+
+@app.route("/metrics/latency")
+def metrics_latency():
+    """Lightweight latency summary for dashboard KPIs."""
+    if not r: return jsonify({"avg_ms": 0, "requests_per_min": 0})
+    recent = []
+    for x in (r.lrange("routes:recent", 0, 49) or []):
+        try: recent.append(json.loads(x))
+        except: pass
+    if not recent: return jsonify({"avg_ms": 0, "requests_per_min": 0, "count": 0})
+    now = time.time()
+    last_min = [x for x in recent if now - x.get("ts", 0) < 60]
+    latencies = [x.get("queue_ms", 0) + x.get("inference_ms", 0) for x in last_min if "inference_ms" in x]
+    return jsonify({
+        "avg_ms": round(sum(latencies) / len(latencies), 1) if latencies else 0,
+        "requests_per_min": len(last_min),
+        "count": len(recent)
+    })
+@app.route("/health/unified")
+def health_unified():
+    """Unified health aggregating all layers: Router + Redis + GPUs + Circuit Breaker + Scores."""
+    gpus = {}
+    for m in GPU_URLS:
+        h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
+        h["active_requests"] = gpu_active_count(m)
+        h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
+        h["health_score"] = gpu_health_score(m)
+        h["circuit_open"] = is_circuit_tripped(m)
+        gpus[m] = h
+    circuit_state = {}
+    for m in GPU_URLS:
+        cooldown_until = r.ttl("circuit:" + m + ":open") if r else None
+        circuit_state[m] = {
+            "open": is_circuit_tripped(m),
+            "cooldown_remaining_s": max(0, cooldown_until) if cooldown_until and cooldown_until > 0 else 0,
+            "trip_count": int(r.get("circuit:" + m + ":count") or 0) if r else 0
+        }
+    overall = "healthy"
+    if not r:
+        overall = "degraded"
+    if all(circuit_state[m]["open"] for m in GPU_URLS):
+        overall = "down"
+    return jsonify({
+        "status": overall, "router": "healthy",
+        "redis": "connected" if r else "down",
+        "gpus": gpus, "circuit_breaker": circuit_state,
+        "scores": {m: gpu_health_score(m) for m in GPU_URLS},
+        "available_models": available_models(), "timestamp": time.time()
+    })
+
 @app.route("/stream")
 def stream():
     def ev():

ssl/README.md

@@ -0,0 +1,6 @@
+# SSL Directory
+
+SSL termination is handled upstream by NetBird/Authentik.
+This directory is intentionally empty — no certs stored here.
+
+For local dev SSL, use the docker-compose.override.yml pattern.