fix: LiteLLM contracts — align with 2026-07-08 architecture changes

- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
  GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
  Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
  (router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
  removed qwen3.6-35B-A3B from model list (never deployed)

disk-gc-threat-response.prose.md

@@ -3,10 +3,11 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
   Recurring disk health scan, garbage collection, and threat response
-  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
   2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
   Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB via `docker system prune -a --force`.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
+  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
 
 ## Scope
 
-All CTs in the Proxmox cluster, with special attention to Docker hosts:
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
+Docker hosts get special attention:
 
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
+| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All others | — | LOW — no Docker | apt clean, log rotate |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
+| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
+> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 
 ## Threat Levels
 
@@ -123,7 +130,7 @@ call summary-reporter
 
 ## GC Strategies by Host Type
 
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -290,3 +297,10 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
 | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+
+### KVM VM (via direct SSH)
+| Host | IP | Role | Status |
+|------|-----|------|--------|
+| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.

infrastructure-control.prose.md

@@ -144,7 +144,7 @@ description: >
 
 ### Ecosystem A: docker-vm (192.168.68.7)
 
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
+| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
+
+**Last verified:** 2026-07-09 — 16/16 containers running healthy.
 
 ### Home Stack Services
 

infrastructure-monitoring.prose.md

@@ -7,10 +7,14 @@ description: >
   from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
   via existing /metrics Prometheus endpoint.
   
-  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  DEPLOYMENT STATUS (2026-07-09):
-  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  This contract defines the target state; proxmox-monitor is the as-built.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
+     NVIDIA sidecar exporters (.8/.110:9400) never installed.
+     Router falls back to direct GPU /health probes.
+  ⚠️ This contract is target-state aspirational — not as-built.
+     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
 

litellm-api-keys.prose.md

@@ -8,6 +8,10 @@ description: >
   Ensures agents never use the master key directly. Rotation is event-driven,
   not calendar-driven — rotate only on compromise, personnel change, or
   periodic security hygiene (quarterly/annually).
+  
+  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
+  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
+  on CT 116. Last verified: 2026-07-09.
 ---
 
 ## Parameters
@@ -38,7 +42,8 @@ description: >
    - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
    - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
    - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
+   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
    - Return the new key
 5. **If action == "rotate"**:
    - Generate new key with same alias (LiteLLM replaces the old key)

litellm-health.prose.md

@@ -2,34 +2,41 @@
 kind: function
 name: litellm-health
 description: >
-  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
-  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
-  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Router (harness-router :9000) is DEPRECATED — container still runs but
-  Prometheus/Grafana and fleet dashboard.
+  is not in the request path. GPU monitoring via Prometheus/Grafana and
+  fleet dashboard (gpu-monitor :9100).
   Designed as a reusable contract for any Syslog agent.
+  
+  Source of truth: gpu-fleet.prose.md
 ---
 
-## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 
 ```
-Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
-                        │                    │
+                        │
-                  Key validation      Model routing
+                  Key validation
-                  Fallback chains     Slot booking
+                  Fallback chains
-                  Complexity router   Circuit breakers
+                  Budget tracking
-                  Budget tracking     Redis-backed
+                        │
-                        │                    │
+                  Prometheus ← metrics
-                  Prometheus ← metrics  ←───┘
                         │
                     Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
 ```
 
-**What changed (v3.1.0 → v3.2.0)**:
+**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
-- Router is BACK — runs internally on :9000 behind LiteLLM
+- Router REMOVED from request path — LiteLLM proxies directly to GPU
-- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- All GPUs at parallel 2 (was parallel 1)
-- Router :9000 is 127.0.0.1-only, not publicly accessible
+- NVIDIA context reduced 256K→128K to free VRAM
-- GPU fleet dashboard: http://192.168.68.24:9100
+- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
-- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
 
 ## Parameters
 
@@ -55,27 +62,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se
 
 ## GPU Fleet Topology
 
-| Host | IP | Hardware | Models Served | Engine |
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
-|------|-----|----------|---------------|--------|
+|------|-----|----------|---------------|--------|---------|----------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
-| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 
 ## Model Fallback Chains (LiteLLM)
 
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
-| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
-| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | — | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
 
 ## Containers on CT 116
 
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |

litellm-self-heal.prose.md

@@ -6,6 +6,9 @@ note: >
   Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
   This contract is now manual-only — triggers require explicit user request.
   Consider reimplementing as a standalone cron job or prose contract.
+  
+  Source of truth for GPU topology: gpu-fleet.prose.md
+  Last verified: 2026-07-09
 description: >
   Standing responsibility that monitors LiteLLM health and proactively
   fixes common issues. Reports every action via Zulip DM and RA-H OS
@@ -73,11 +76,11 @@ message is sent to the responsible agent with full context.
 
 ## GPU Fleet
 
-| Host | IP | Hardware | Models | Engine |
+| Host | IP | Hardware | Models | Engine | Context |
-|------|-----|----------|--------|--------|
+|------|-----|----------|--------|--------|---------|
-| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
+| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b | llama-server :8080 (Vulkan) | 256K |
-| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
+| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 (systemd) | 128K |
-| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
+| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 (systemd) | 128K |
 
 ## Remediation Rules
 
@@ -105,12 +108,10 @@ Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts
 
-### Rule 7: Router Returns 503 (All GPUs Saturated)
+### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
-Detect → router logs show QUEUE_TIMEOUT with GPUs idle
+Router is no longer in the request path (2026-07-08). LiteLLM proxies
-Root cause → roster not loaded at startup (empty TIER_MODELS)
+directly to GPU. This rule is retained for reference but is inactive.
-Fix → POST /admin/roster/reload (if endpoint available) or restart router
+If 503 errors occur, check LiteLLM timeouts and GPU health directly.
-Permanent fix → add load_roster() to router startup code
-Escalate → if reload doesn't resolve within 2 min
 
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors

scripts/pct-run.sh

@@ -30,6 +30,12 @@ declare -A CT_NODES=(
     # acerpve (192.168.68.9)
     [102]=acerpve   # adguard
     # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
+    #
+    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
+    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
+    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
+    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
+    #   118 jitsi     → stopped, not in service
 )
 
 # Each node must be root-accessible via SSH hostname