feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60)

- Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract
fix: remaining template YAML and topology table corrections
2026-07-12 23:38:21 +00:00 · 2026-07-12 22:55:24 +00:00 · 2026-07-12 22:54:30 +00:00 · 2026-07-12 22:49:39 +00:00 · 2026-07-12 22:27:15 +00:00 · 2026-07-09 06:28:04 +00:00
22 changed files with 1591 additions and 318 deletions
@@ -115,10 +115,11 @@ Run on trigger or schedule. Maintain persistent world-model state across runs.
 | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
 | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
 | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
 | `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
 ### Instantiable Templates (`kind: pattern` — run with `prose run`)
@@ -146,8 +147,9 @@ Called on-demand as single-render tools.
 | Contract | Description |
 |---|---|
-| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
+| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
-| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
+| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
 | `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
@@ -158,8 +160,8 @@ Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
-| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 ## Contract Structure
@@ -3,10 +3,11 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
  Recurring disk health scan, garbage collection, and threat response
-  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
  Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB via `docker system prune -a --force`.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
 ## Scope
-All CTs in the Proxmox cluster, with special attention to Docker hosts:
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
 Docker hosts get special attention:
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
 | amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All others | — | LOW — no Docker | apt clean, log rotate |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
 | GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
 > **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
 > **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 ## Threat Levels
@@ -123,7 +130,7 @@ call summary-reporter
 ## GC Strategies by Host Type
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -224,32 +231,76 @@ dangling images and orphaned build cache. No automated GC was in place.
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
 - amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```
-Done. Here's what we have:
+## Incident Log: 2026-07-09 — amdpve docker bloat
-## Storage Summary
+### Discovery
 Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
 amdpve (.15) flagged at 78% (AMBER threshold: 75%).
-| System | Used | Free | Health |
+### Diagnosis
-|--------|------|------|--------|
+- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
-| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
+- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
-| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
+- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
-| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
+- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
-| **/root/go** (abiba) | 773M | — | Fine |
+  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
 - Not a running service — zero containers, zero active volumes
-## New Contract: `disk-gc-threat-response.prose.md`
+### Resolution
 ```
 docker system prune -a --force
 → Reclaimed 11.56GB
 → Post: 55G / 94G (62%), 35G free
 → 1 image removed (llama-strix-hip:latest, 15.5GB)
 → 8 build cache layers removed
 → amdpve now GREEN
 ```
-Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+### Root Cause
 Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
 artifacts. Docker on amdpve serves no running purpose — it's only used for
 one-off GPU builds. No automated post-migration cleanup was in place.
-### Three-in-One
+### Preventive Measures
 - amdpve added to Docker GC scan list
 - Post-migration cleanup step added: after any GPU backend migration, prune
  the old backend's Docker images within 24 hours
 - Contract now scans GPU bare-metal hosts alongside CTs
 - Access via `pct-run` script for all CTs (no hardcoded IPs)
-1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+## Access Matrix (documented 2026-07-09)
-2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+### CT Access (via pct-run)
 | CT | Name | Node | Status |
 |----|------|------|--------|
 | 100 | abiba | amdpve | local |
 | 102 | adguard | acerpve | ✅ reachable |
 | 104 | authentik | minipve | ✅ reachable |
 | 105 | kagentz | amdpve | ✅ reachable |
 | 106 | ra-h-os | storepve | ✅ reachable |
 | 107 | pbs | storepve | ✅ reachable |
 | 108 | media | storepve | ✅ reachable |
 | 110 | gitea | minipve | ✅ reachable |
 | 111 | tdunna | amdpve | ✅ reachable |
 | 112 | tanko | amdpve | ✅ reachable |
 | 113 | baggy | amdpve | ✅ reachable |
 | 114 | mumuni | minipve | ✅ reachable |
 | 115 | scottdenya | amdpve | ✅ reachable |
 | 116 | syslog-api | minipve | ✅ reachable |
 | 117 | zulip | storepve | ✅ reachable |
-3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
+### GPU Bare Metal (via direct SSH)
 | Host | IP | GPU | Status |
 |------|-----|-----|--------|
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
 | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
-### The incident root cause
+### KVM VM (via direct SSH)
-Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
+| Host | IP | Role | Status |
 |------|-----|------|--------|
 | docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
-Want me to push this to the prose-contracts repo?
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
@@ -5,9 +5,13 @@ description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
-  Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2
+  UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
-  on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully
+  to llama-server on each GPU host (no router in inference path). Router
-  deprecated — nginx routes /v1 → LiteLLM directly.
+  (port 9000) is running but NOT in request path. All GPUs standardized on
  api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
  caused cascading 401→timeout→401 fallback loops — fixed.
  Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
  Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
 agent: abiba
 triggers:
  - on model add/remove
@@ -30,7 +34,7 @@ triggers:
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
 - port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding
-## Fleet Topology (Current — June 2026)
+## Fleet Topology (Current — July 2026)
 ```
 ┌──────────────────────────────────────────────────────────────────┐
@@ -47,9 +51,9 @@ triggers:
 │  Containers:                                                     │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
 │  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
-│  │ :4000    │─▶│ :9000    │  │ :3000    │  │ :3000     │        │
+│  │ :4000    │  │ :9000    │  │ :3000    │  │ :3000     │        │
-│  │ keys+sync│  │internal  │  │ harness  │  │ Prometheus│        │
+│  │ keys+sync│  │deprecated│  │ harness  │  │ Prometheus│        │
-│  │ fallback │  │only!     │  │ UI       │  │ data src  │        │
+│  │ fallback │  │not in    │  │ UI       │  │ data src  │        │
 │  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
 │                    │                                             │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
@@ -64,7 +68,7 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 128K ctx │  │ 128K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│ 256K ctx │  │ 131K ctx    │  │ 256K ctx  │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
 │ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
 │ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
@@ -72,13 +76,42 @@ triggers:
 └──────────┘
 ```
-## Current Model Assignments (2026-07-08)
+## Current Model Assignments (2026-07-12)
 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy |
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
 ## Routing Configuration (LiteLLM — July 2026)
 ### syslog-auto Weighted Pool
 | Model | GPU | Weight | RPM Cap | Purpose |
 |-------|-----|--------|---------|---------|
 | qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context |
 | ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling |
 | gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision |
 ### Direct Model Endpoints
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
 | ornith-1.0-35b | 40 | Tight cap — prevents Strix overload |
 | qwen3.6-27B-code | 500 | High cap — primary workhorse |
 | gemma-4-12b | 500 | High cap — fast 12B |
 ### Fallback Chains
 - gemma → qwen
 - qwen → gemma
 - ornith → qwen → gemma
 - syslog-auto → qwen → gemma → ornith
 ### Why ornith RPM Is Capped
 - Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks
 - Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
 - Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
 ## Operations
@@ -114,10 +147,10 @@ triggers:
 ### sync-keys
 1. List all agent keys in LiteLLM DB via `GET /key/list`
-2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
+2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
 3. Generate missing keys via `POST /key/generate` with unlimited budget
-4. Update agent configs — `/etc/environment` LITELLM_API_KEY
+4. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production`
-5. Send Zulip DM to agents that can't be reached via SSH
+5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
 6. Verify each key with test request through full chain
 7. Document keys in knowledge graph
@@ -136,19 +169,25 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 7. Check port conflicts: verify only one llama-server on :8080 per host
 8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
-## Agent Keys (LiteLLM DB — Current 2026-06-30)
+## Agent Keys (LiteLLM DB — Current 2026-07-11)
-| Agent | CT | IP | Key | Access |
+Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY).
-|-------|-----|-----|-----|--------|
+Agent gateways inject keys at runtime via `infisical run --` wrapper.
-| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
+Plaintext keys removed from this contract post-vault-migration.
 | Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
 | Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
 | Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
 | Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
 | Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
-**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
+| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
-If no SSH access, send Zulip DM via abiba-bot.
+|-------|-----|-----|---------------|------------|--------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
 | Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
 | Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
 | Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
 | Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
 | Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | no SSH |
 > **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.
 **Key update procedure**: Update Infisical vault → `infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production` → restart agent gateway. Agent picks up new key via `infisical run --` wrapper at startup.
 If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 ## Configuration Files
@@ -159,10 +198,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
@@ -170,7 +209,7 @@ If no SSH access, send Zulip DM via abiba-bot.
 | Component | URL | Details |
 |-----------|-----|---------|
-| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
+| Grafana | `http://192.168.68.116:3001/` | admin / vault (`GRAFANA_ADMIN_PASSWORD`) |
 | GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
 | Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
 | GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
@@ -181,10 +220,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110).
+- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline).
- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput.
+- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2.
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`.
 - **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
@@ -210,12 +249,12 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.
 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
-## Agent Config Implications (2026-07-08)
+## Agent Config Implications (2026-07-12)
-With NVIDIA GPUs at 128K context:
+With RTX 3090 at 256K context (verified July 2026):
- Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow
+- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it
- Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule
+- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified)
- Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072`
+- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072`
- Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx)
+- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo
 - All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
 - Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
@@ -19,7 +19,7 @@ agent: abiba
 ```
 ┌─────────────────────────────────────────────────────────┐
-│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  GPU Monitor Server (:9100) — 192.168.68.24              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
   │          │          │
@@ -35,7 +35,7 @@ agent: abiba
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
-Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
 poll .15:8080 directly; must go through router on .116.
 ```
@@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
 | File | Host | Purpose |
 |------|------|---------|
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
 ## Execution
@@ -0,0 +1,287 @@
 ---
 kind: responsibility
 name: gpu-self-heal
 description: >
  GPU fleet self-healing — detects anomalies, applies remediation, tracks
  benchmarks, and predicts failures before they happen. Extends gpu-monitor
  (v2.1.0) with active remediation rules, Prometheus metrics consumption,
  VRAM trend analysis, and predictive alerting.
 agent: abiba
 depends_on:
  - gpu-monitor.prose.md (live data source on .24:9100)
  - gpu-fleet.prose.md (source of truth for topology)
 ---
 ## Maintains
 - gpu-health: { status: "healthy"|"degraded"|"down", issues: array, actions: array }
 - gpu-self-heal-log: array of { timestamp, gpu, issue, action, result } — audit trail
 - benchmark-regression: { gpu, baseline_tok_sec, current_tok_sec, trend, alerts }
 - vram-trend: { gpu, current_mb, rate_mb_per_hour, projected_full_in_hours }
 - circuit-breaker-status: { gpu, open, auto_reset_attempted, last_reset }
 ## Requires
 - gpu-monitor:function — Live fleet data from .24:9100/gpu-data
 - Prometheus exporters on all 3 GPUs (:9400/metrics)
 - SSH access to GPU hosts for restart operations
 ## Continuity
 - Self-driven: check every 60 seconds against GPU monitor data
 - Also wakes on gpu-fleet health degradation
 - On fix: verify with benchmark inference test before declaring resolved
 - Escalate: after 3 failed remediation attempts → Zulip #agent-hub alert
 ---
 ## Remediation Rules
 ### Rule 1: GPU Temperature Critical (>85°C for >2 min)
 - **Detect**: Any GPU temp >85°C sustained for 2+ consecutive polls
 - **Fix**: 
  1. Reduce inference concurrency on that GPU (load-side cooling only — NO fan control)
  2. Redirect new requests to cooler GPUs via LiteLLM fallback chains
  3. If all GPUs hot, alert about cooling infrastructure
 - **Verify**: Temp drops below 80°C within 5 minutes
 - **Escalate after**: 3 verification failures → Zulip alert
 ### Rule 2: VRAM Leak Detection (tiered by GPU capacity)
 - **Detect**: VRAM growing at sustained rate over 6+ hour window
  - RTX 3090 (24GB): ≥100MB/hour
  - RTX 5070 (12GB): ≥50MB/hour
  - Strix Halo (64GB UMA): ≥200MB/hour
 - **Fix**: 
  1. Log VRAM snapshot with process list (nvidia-smi/rocm-smi + ps aux)
  2. If llama-server is the growth source → restart with memory cap flag
  3. If unknown process → kill and alert
 - **Verify**: VRAM growth rate drops below threshold
 - **Escalate after**: persistent leak after restart → hardware investigation
 ### Rule 3: Model Inference Timeout / GPU Stuck
 - **Detect**: >50% failure rate over 60s window + 30s grace period (not just single stuck request)
 - **Fix**: 
  1. Restart llama-server on affected GPU host
  2. Wait 15s for model to reload
  3. Run benchmark inference test
 - **Verify**: Model returns 200 with <30s response, failure rate drops to 0%
 - **Escalate after**: 3 restarts in 1 hour → GPU hardware check
 ### Rule 4: Benchmark Regression (>20% drop)
 - **Detect**: gen_tok_per_sec drops >20% below baseline over 3+ benchmarks
 - **Fix**:
  1. Check GPU utilization — if >90%, other process is competing
  2. Check power limit — if throttled, restore to max
  3. Check thermal — if hot, apply Rule 1
 - **Verify**: Benchmark returns to within 10% of baseline
 - **Escalate after**: persistent regression → possible hardware degradation
 ### Rule 5: Circuit Breaker Stuck Open
 - **Detect**: Circuit breaker open >10 minutes with GPU reporting healthy
 - **Fix**: 
  1. Verify GPU /health returns 200
  2. If GPU healthy, send 1 test inference
  3. If test succeeds → reset circuit breaker via router API
  4. 60s cooldown — if CB re-opens immediately, it was legitimate, do NOT re-reset
  5. Max 1 auto-reset per GPU per hour
 - **Verify**: CB closes, inference succeeds, CB stays closed for 60s+
 - **Escalate after**: CB won't close after reset → router issue
 ### Rule 6: Strix Halo Unreachable
 - **Detect**: Strix not responding — probe .15:8080 directly (firewall opened .24→.15)
 - **Fix**: 
  1. SSH to .15 → check llama-server process
  2. Restart llama-server if not running
  3. Verify through both direct probe AND router
 - **Verify**: Direct health probe returns 200, router reports Strix healthy
 - **Escalate**: If host .15 itself is unreachable → infrastructure alert
 ### Rule 7: Prometheus Exporter Down
 - **Detect**: Any GPU :9400/metrics unreachable for >2 polls
 - **Fix**:
  1. SSH to GPU host → check prometheus-exporter process
  2. Restart exporter if dead
  3. While exporter is down, fall back to nvidia-smi/rocm-smi direct probes
  4. If exporter is running but unreachable → check firewall/host networking
 - **Verify**: :9400/metrics returns 200
 - **Escalate after**: 3 failed restarts → networking issue
 ### Rule 8: Predictive Thermal Warning (two-tier)
 - **Detect**: 
  - Tier 1 (warning): temp >70°C AND rising >2°C/min → reduce concurrency, no alert
  - Tier 2 (critical): temp >80°C AND still rising → full alert + aggressive load shedding
 - **Fix**: 
  - Tier 1: silently reduce parallel requests to that GPU by 50%
  - Tier 2: redirect all new requests away, alert #agent-hub, apply Rule 1 logic
 - **Verify**: Temp rise rate drops below 1°C/min (Tier 1) or temp drops below 80°C (Tier 2)
 - **Escalate**: If Tier 2 triggers and temp still rising after 5 min → possible hardware failure
 ### Rule 9: Context Window Optimization
 - **Detect**: Benchmark tok/s vs baseline for each GPU at current context
  - RTX 3090 (256K ctx, qwen3.6-27B-code): target 75+ tok/s — currently at baseline
  - RTX 5070 (131K ctx, gemma-4-12b): target 76+ tok/s — optimal for vision/web role
  - Strix Halo (256K ctx, ornith-1.0-35b): target 70+ tok/s — currently above baseline
 - **Fix**:
  - If tok/s > baseline → context has headroom, consider increasing
  - If tok/s < 90% baseline → reduce context by 25% and retest
  - If tok/s within 10% of baseline → optimal, no change
 - **Verify**: Re-benchmark after context change, confirm within 10% of target
 - **Escalate**: If context can't be adjusted without significant perf loss
 ### Rule 10: Workload Distribution Optimization
 - **Detect**: GPU roles misaligned with hardware capabilities
 - **Target distribution**:
  - RTX 3090 (24GB, 256K, 75 tok/s) → Heavy reasoning, code gen, long conversations
  - RTX 5070 (12GB, 131K, 76 tok/s) → Vision/image, web search, quick lightweight tasks
  - Strix Halo (64GB, 256K, 72 tok/s) → Context compression, summarization, long docs
 - **Fix**:
  - Alert if any GPU is handling workload outside its designated role
  - Recommend Hermes agent profile updates to match workload to GPU
  - Track per-GPU request distribution via LiteLLM spend logs
 - **Verify**: Each GPU's request pattern matches its designated role within 24h
 - **Escalate**: If role mismatch persists >48h → agent profile audit needed
 ---
 ## Execution
 ```prose
 -- Phase 1: Fetch live GPU data
 let fleet = call gpu-monitor
  endpoint: "http://192.168.68.24:9100/gpu-data"
 -- Phase 2: Evaluate each GPU against remediation rules
 let actions = []
 for gpu in fleet.gpus:
  -- Rule 1: Thermal critical
  if gpu.temp_c > 85 and sustained_for(gpu, 120):
    push actions apply-thermal-fix(gpu)
  -- Rule 2: VRAM leak
  let vram_rate = calculate-vram-trend(gpu, hours=6)
  if vram_rate > 50:
    push actions apply-vram-fix(gpu, vram_rate)
  -- Rule 4: Benchmark regression
  let bench = fleet.benchmarks[gpu.hostname]
  if bench.current_tok_sec < bench.baseline_tok_sec * 0.8:
    push actions apply-benchmark-fix(gpu, bench)
 -- Rule 3: Model stuck
 for model in fleet.router.available_models:
  if model.consecutive_timeouts >= 3:
    push actions apply-model-restart(model)
 -- Rule 5: Circuit breaker
 for cb in fleet.router.circuit_breaker:
  if cb.open and cb.open_duration > 600 and gpu_is_healthy(cb.gpu):
    push actions apply-cb-reset(cb)
 -- Rule 6: Strix Halo
 if not fleet.strix.running and pingable("192.168.68.15"):
  push actions apply-strix-restart()
 -- Rule 7: Prometheus exporters
 for gpu in fleet.gpus:
  if not prometheus_reachable(gpu.hostname, 9400):
    push actions apply-exporter-restart(gpu)
 -- Rule 8: Predictive thermal
 for gpu in fleet.gpus:
  let rise_rate = calculate-temp-rise(gpu, minutes=5)
  if rise_rate > 2.0 and gpu.temp_c < 80:
    push actions apply-proactive-cooling(gpu)
 -- Phase 3: Execute actions, verify, log
 for action in actions:
  let result = execute-with-verify(action)
  log-to-kg(action, result)
  if result.failed:
    escalate-if-needed(action)
 -- Phase 4: Update health state
 call update-gpu-health
  gpus: fleet.gpus
  actions: actions
  status: derive-overall-status(fleet, actions)
 -- Wait 60s and repeat
 ```
 ## Audit Trail Format
 ```json
 {
  "run_id": "gpu-self-heal-20260712-001",
  "timestamp": "2026-07-12T16:00:00Z",
  "gpu": "ct8-rtx3090",
  "issue": "thermal-critical",
  "detected": { "temp_c": 87, "duration_s": 180 },
  "action": "set-fan-100pct",
  "result": "resolved",
  "verification": { "temp_c": 76, "after_s": 300 },
  "escalated": false
 }
 ```
 ---
 ## Reporting
 ### 1. Knowledge Graph
 Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
 ### 2. Zulip Alerts (#agent-hub → alerts-gpu)
 - `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
 - `issues_escalated > 0` → "⚠ GPU Self-Heal — <gpu> needs attention"
 - Every 100th clean cycle → "✅ GPU Fleet: All Clear"
 ### 3. Prometheus/Grafana Integration
 - GPU self-heal actions exposed as Prometheus counter metrics
 - Dashboard panel: "GPU Interventions (24h)" showing count/type/result
 ### 4. Weekly Benchmark Report
 - Per-GPU tok/s trend over 7 days
 - Regression alerts if any GPU degrades >10% week-over-week
 ---
 ## Design Decisions (Grilled & Confirmed — 2026-07-12)
 1. **Fan control**: ❌ NO auto fan control. Load-side cooling only (reduce concurrency, redirect).
 2. **Model restart**: ✅ Only if >50% failure rate over 60s + 30s grace period. Not on single stuck request.
 3. **Strix direct access**: ✅ Open firewall .15:8080 → .24 for direct health probe + restart.
 4. **VRAM thresholds**: Tiered — 100MB/h (RTX 3090), 50MB/h (RTX 5070), 200MB/h (Strix).
 5. **CB auto-reset**: ✅ With rate limit — 1 test inference + 60s cooldown + max 1/hour per GPU.
 6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Original baseline kept in Grafana.
 7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
 8. **Prometheus**: Primary source. Fall back to nvidia-smi/rocm-smi direct probes if exporter down.
 ## Lessons Learned (2026-07-12)
 ### L1: API Key Standardization Is Critical
 - All GPU llama-servers MUST use the same api-key as the LiteLLM config.
 - RTX 5070 had `--api-key sk-loc...5678` while LiteLLM sent `not-needed`.
  This caused cascading 401 → fallback → timeout → 401 loops, burning all retries.
 - **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config.
 ### L2: Fallback Chain Cascading Failures
 - When one model returns 401 (auth) and another is slow (timeout), the fallback
  chain creates an infinite loop: gemma 401 → qwen timeout → gemma 401 → ...
 - **Rule**: If a model returns 401 (auth error), do NOT fall back to it again.
  Mark it as permanently failed for this request.
 ### L3: Verify Running State, Not Docs
 - RTX 3090 was documented at 128K context. Actually running at 256K.
 - Parallel count wrong (docs said 2, actual is 1 on RTX 3090).
 - **Rule**: Before making decisions, check `/proc/PID/cmdline` on GPU hosts.
 ### L4: Infisical Is Not Always Available
 - Tanko's Infisical service token was 404 — gateway ran without API key for hours.
 - **Rule**: Always keep a local `.env` fallback for `LITELLM_API_KEY`.
 - Contract hermes-config-template Rule 3 updated.
 ### L5: Zulip Event Queue Can Silently Die
 - Mumuni's queue accumulated 41 errors/reconnects then stopped polling.
  Gateway was running but ignoring all messages.
 - **Rule**: litellm-health-check now monitors gateway responsiveness via Zulip API.
@@ -22,30 +22,38 @@ done
 ## Agent Map
-| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
+| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
-|-------|-----|------|-----|-------------|---------------|----------|
+|-------|-----|------|-----|---------------|------------|----------|
-| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
+| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
-| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
+| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
-| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
+| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
-| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |
+| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
 | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
 > **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
 Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
 Keys are stored in Infisical vault (project=agents, env=production) and injected at
 runtime via `infisical run --` wrapper. Plaintext keys removed from this baseline.
 ## Key Architecture
 ```
 Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
                       ↓
 Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
                                     └── Key DB (Postgres)
 ```
- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
+- **Master key**: stored in Infisical vault (project=infrastructure, secret=LITELLM_MASTER_KEY) — ADMIN ONLY
 - **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
+- **Key injection**: `infisical run --project=agents --env=production -- hermes gateway run` injects `LITELLM_API_KEY` at runtime
- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
+- **Key source**: Infisical vault → runtime env var. /etc/environment is CLEAN (stripped, tagged `# [INFISICAL]`)
 - **Legacy override** (pre-migration): `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` — should be REMOVED
 ## Config Pattern — Mandatory Fields
-### For Hermes Agents (Tanko, Mumuni, Baggy)
+### For Hermes Agents (Tanko, Mumuni, Koonimo)
 Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
@@ -76,7 +84,7 @@ auxiliary:
    model: gemma-4-12b              # or syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
-    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30
 ```
@@ -91,7 +99,7 @@ auxiliary:
    model: syslog-auto               # or gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
-    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 120
 ```
@@ -110,8 +118,7 @@ LiteLLM/harness will fail with:
 401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
 ```
-**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
+**Workaround**: Set `api_key` directly (copy the value from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=agents --env=production`) alongside `api_key_env` in every auxiliary task config that uses the harness provider.
 `api_key_env` in every auxiliary task config that uses the harness provider.
 **Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
 `api_key` is empty:
@@ -129,7 +136,11 @@ if not cfg_api_key:
 ```bash
 for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
-    pct-run $ct grep LITELLM_API_KEY /etc/environment
+    # Verify /etc/environment is CLEAN (no LITELLM_API_KEY)
    pct-run $ct "grep -c LITELLM_API_KEY /etc/environment 2>/dev/null || echo '0 (clean)'"
    # Verify gateway uses infisical run wrapper
    pct-run $ct "ps aux | grep 'infisical run' | grep -v grep"
    # Check for hardcoded harness keys
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
 done
@@ -137,15 +148,17 @@ done
 ### Master Key Leak Check
 ```bash
-# On every agent:
+# On every agent — must return empty:
-pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
+pct-run <CT> grep -rl "sk-litellm" /root/ /etc/ 2>/dev/null
-# Must return empty
+# Vault is the only place the master key should exist
 ```
 ### Verify Key Works
 ```bash
 # Retrieve key from vault and test:
 KEY=$(infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
 curl -s http://192.168.68.116:80/v1/models \
-  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
+  -H "Authorization: Bearer $KEY" | grep syslog-auto
 # Must return model list
 ```
@@ -156,19 +169,27 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```
-### For pi Agents (Tdunna)
+### For Koby (CT 111 / tdunna)
-Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
+Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
 Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
 **LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
 ### For pi Agents (Abiba)
 Abiba (CT100) runs pi via PM2 with the Zulip extension.
 Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
-**models.json** — Must only list models authorized for the agent's LiteLLM key:
+**models.json** — Must only list models authorized for the agent's LiteLLM key.
 Key is injected via `infisical run --` wrapper at PM2 startup:
 ```json
 {
  "providers": {
    "syslog-harness": {
      "baseUrl": "http://192.168.68.116/v1",
      "api": "openai-completions",
-      "apiKey": "sk-...",
+      "apiKey": "${LITELLM_API_KEY}",
      "models": [
        { "id": "syslog-auto" },
        { "id": "ornith-1.0-35b" },
@@ -257,6 +278,6 @@ If they differ → ghost detected → kill ghost → start fresh.
 | Date | Change |
 |------|--------|
-| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
+| 2026-07-08 | Koby: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added config section. Key rotated and stored in vault. Added Failure Mode #11 to zulip-adapter-lessons. |
 | 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
 | 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
@@ -5,35 +5,40 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110),
+  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (ornith-1.0-35b).
-  parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added.
+  RTX 3090 context verified at 256K (was incorrectly documented as 128K).
  RTX 5070 stays at 131K for vision/web. Infisical .env fallback required (Rule 3).
  Parallel counts corrected (RTX 3090=1, RTX 5070=2, Strix=2).
 ---
 ## Maintains
 - template_version: "2.1.0"
 - last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "tdunna", "baggy", "kagenz0"]
+- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
 - agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array
-## Agent Keys (LiteLLM — Current 2026-07-04)
+## Agent Keys (LiteLLM — Current 2026-07-11)
 Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM
-PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB, not in
+PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB.
-config files. The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent
+The env var `LITELLM_API_KEY` is injected at runtime via `infisical run --` wrapper
-host AND in `~/.hermes/.env` for gateway env propagation.
+(project=agents, env=production). /etc/environment and ~/.hermes/.env are NO LONGER
 used for agent keys — stripped and tagged `# [INFISICAL]` post-migration.
 Sub-agent profiles inherit auth from the main config — no separate keys needed.
 | Agent | Key Alias | Host | SSH | Sub-Agents |
 |-------|-----------|------|-----|-----------|
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
 | Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
-| Abiba | `abiba-*` | 192.168.68.24 | local | — |
+| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
-| Tdunna | `tdunna-*` | ? | Zulip | — |
+| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
-| Baggy | `baggy-*` | ? | Zulip | — |
+| Koonimo | `koonimo` | CT 113 (baggy) | Zulip | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |
 > CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
 ✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
  syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`
@@ -51,11 +56,12 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 ## API Key Rules
 - `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred)
  Key is injected at runtime via `infisical run --` wrapper — never in /etc/environment
 - `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider
 - `api_key: sk-...` — Hardcoded key only as fallback when env var not possible
- Set `LITELLM_API_KEY` in `/etc/environment` on each host
+- Store `LITELLM_API_KEY` in Infisical vault (project=agents, env=production)
 - Sub-agents NEVER get their own key — they share the host agent's key
- Restart Hermes after updating `/etc/environment`
+- Restart Hermes gateway after updating vault secret (key auto-injected via wrapper)
 ### Sub-Agent Profiles (Mumuni pattern)
@@ -79,7 +85,7 @@ Sub-agent profile rules:
 5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty
 6. **Never hardcode a key** in sub-agent profiles
-This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`.
+This ensures all 6 sub-agents use the same LiteLLM key injected via `infisical run --` wrapper.
 When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
 work immediately after restart.
@@ -91,10 +97,10 @@ model:
  default: <agent_model>          # e.g., ornith-1.0-35b, qwen3.6-27B-code
  provider: harness
  base_url: http://192.168.68.116/v1
-  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
+  api_key_env: LITELLM_API_KEY    # Injected via infisical run -- wrapper
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
-  context_length: 262144          # For syslog-auto (ornith route supports 256K).
+  context_length: 262144          # For syslog-auto (all GPUs support 256K).
-                                  # Set 131072 if using qwen3.6-27B-code or gemma-4-12b directly.
+                                  # Set 131072 if using gemma-4-12b directly (12GB VRAM constraint).
 fallback_providers:
  provider: deepseek
@@ -123,10 +129,10 @@ mcp_servers:
 # ─── Compression ───
 compression:
  enabled: true
-  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
+  model: ornith-1.0-35b           # ⚠️ Must match auxiliary.compression.model
  provider: harness
-  max_context_window: 262144      # For syslog-auto (ornith supports 256K).
+  max_context_window: 262144      # For Strix Halo compression (256K ctx).
-                                  # Set 131072 if using qwen or gemma directly.
+                                  # Set 131072 if using gemma directly.
  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
  target_ratio: 0.30
  protect_last_n: 40
@@ -158,7 +164,7 @@ auxiliary:
    timeout: 30
  compression:
    provider: harness
-    model: gemma-4-12b
+    model: ornith-1.0-35b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    timeout: 60
@@ -176,7 +182,7 @@ custom_providers:
 When LiteLLM keys are regenerated (e.g., after infrastructure changes):
-1. **If SSH available**: `ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"`
+1. **If SSH available**: Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-<NEW> --project=agents --env=production`, then `ssh <host> "systemctl restart hermes-gateway"`
 2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command
 3. **After update**: Restart Hermes on the agent host
 4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
@@ -198,7 +204,14 @@ The following MUST be identical across ALL profiles:
 ### Rule 3: API Keys via Environment
 - Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
 - Hardcoded keys in config.yaml become stale after key rotation
- `/etc/environment` persists across config updates
+- Infisical vault secrets persist across config updates / reinstalls
 - **NEW (July 2026): Always keep a local `.env` fallback.** Infisical service tokens
  can expire/404 (tanko incident: token not found, gateway ran without key for hours).
  The `.env` file should have the key uncommented as a fallback:
  ```
  LITELLM_API_KEY=sk-...
  # [INFISICAL] Also sourced from vault.sysloggh.net
  ```
 - Restart Hermes after env var updates
 ### Rule 4: Sub-Agent Profiles Inherit Auth
@@ -208,7 +221,7 @@ The following MUST be identical across ALL profiles:
  - Auxiliary tasks: `api_key: ''`, `provider: harness`
 - Never hardcode a key in sub-agent profiles
 - When main config uses `api_key_env`, sub-agents automatically use it
- This means key rotation only touches ONE file (`/etc/environment`)
+- This means key rotation only touches ONE vault secret (`LITELLM_API_KEY`)
 ### Rule 5: Main Config Base URL
 |- Use direct IP: `http://192.168.68.116/v1`
@@ -223,23 +236,42 @@ The following MUST be identical across ALL profiles:
 - Apply to BOTH main config AND all sub-agent profiles
 - For agents needing longer outputs: raise to 8192, but never omit
-### Rule 7: Auxiliary Model Consistency
+### Rule 7: Auxiliary Model Consistency (UPDATED July 2026)
- All auxiliary services (vision, web_extract, compression) MUST use the same model:
+- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
-  - `model: gemma-4-12b`
+- Compression uses `ornith-1.0-35b` (Strix Halo — 64GB, 256K ctx, compression-optimized)
 - All auxiliary services MUST use identical routing:
  - `base_url: http://192.168.68.116/v1`
  - `api_key_env: LITELLM_API_KEY`
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes to the primary 35B reasoning GPU
+- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
- gemma-4-12b is a lightweight 12B model on the RTX 5070, keeping the Strix Halo free for reasoning
+- **Compression moved to Strix Halo (July 2026)**: The ornith-1.0-35b model on Strix Halo
- The `compression:` block's `model` MUST match `auxiliary: compression: model` — they are two different configs for the same service
+  (64GB UMA, 256K context, 72.4 tok/s) is the designated compression GPU. This frees the
  RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
 - The `compression:` block's `model` MUST match `auxiliary: compression: model`
 - The `compression: max_context_window: 262144` MUST match Strix Halo's actual capacity
-### Rule 8: Compression Threshold for 256K Models
+### Rule 8: GPU Workload Distribution (July 2026)
 - **RTX 3090 (24GB, 256K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
 - **RTX 5070 (12GB, 131K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract
 - **Strix Halo (64GB, 256K ctx, ornith-1.0-35b)**: Context compression, summarization, long docs
 - Agent profiles MUST route auxiliary tasks to the correct GPU:
  - `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
  - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
  - `auxiliary.compression.model: ornith-1.0-35b` (Strix Halo)
 - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
 - For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
 - Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
 - `max_context_window: 262144` MUST match the model's actual capacity
 - See `devops-hermes-compression` skill for full reference
-### Rule 9: Default Model Must Be `syslog-auto` (All Agents)
+### Rule 9: Compression Threshold for 256K Models
 - For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
 - Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
 - `max_context_window: 262144` MUST match the model's actual capacity (Strix Halo = 256K)
 - See `devops-hermes-compression` skill for full reference
 ### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
 - **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
 - **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
 - `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b
@@ -250,7 +282,7 @@ The following MUST be identical across ALL profiles:
 - **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
  for specialized tasks, but MUST validate those models exist in the key's authorized list
-### Rule 10: Validate Model IDs Before Deployment (pi Agents)
+### Rule 11: Validate Model IDs Before Deployment (pi Agents)
 - After configuring a pi agent's `models.json`, verify every model ID:
  ```bash
  curl -s http://192.168.68.116:4000/v1/models \
@@ -5,8 +5,10 @@ version: 1.0.0
 description: >
  Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
  providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
-  Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each
+  Anthropic) may use hardcoded keys. Single source of truth: Infisical vault
-  agent host. Designed to make key rotation a one-step operation.
+  (project=agents, env=production) — injected at runtime via `infisical run --` wrapper.
  /etc/environment is DEPRECATED for agent keys post-migration. Designed to make key
  rotation a one-step vault operation.
 author: Abiba (pi agent)
 ---
@@ -14,7 +16,7 @@ author: Abiba (pi agent)
 ## Rule (One Sentence)
-**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.**
+**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
 ## Scope
@@ -26,6 +28,39 @@ Applies to all Hermes agent configs across all hosts. Covers these config sectio
 - `compression.api_key` (when `provider` is `harness` or contains `litellm`)
 - `fallback_providers[].api_key` (when provider is harness)
 ## Architecture (2026-07-10)
 Syslog is migrating away from **unauthenticated direct access** to the shared inference harness.
 | Path | Auth | Status |
 |------|------|--------|
 | `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
 | `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
 All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
 Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
 ### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
 When `api_mode: responses` is set, Hermes **appends `/v1/responses`** to `base_url`.
 If `base_url` already includes `/litellm/v1/responses`, the result is:
 ```
 http://192.168.68.116/litellm/v1/responses/v1/responses → 404
 ```
 **The `base_url` must end at `/v1` — never include `/responses`:**
 ```yaml
 # ✅ CORRECT — Hermes appends /v1/responses for api_mode: responses
 base_url: http://192.168.68.116/litellm/v1
 # ❌ WRONG — produces double path
 base_url: http://192.168.68.116/litellm/v1/responses
 ```
 This applies to ALL sections using the harness provider: `custom_providers`, `delegation`, `auxiliary.*`.
 ## Exemptions
 External providers are **explicitly exempt** and may use hardcoded keys:
@@ -38,35 +73,42 @@ External providers are **explicitly exempt** and may use hardcoded keys:
 ## Standard Pattern
 ```yaml
-# ✅ CORRECT — all harness/litellm providers
+# ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
 model:
-  provider: harness                    # or custom:litellm.sysloggh.net
+  provider: harness
-  base_url: http://192.168.68.116/v1
+  base_url: http://192.168.68.116/litellm/v1           # ← Hermes appends /v1/responses
-  api_key_env: LITELLM_API_KEY        # ← indirection
+  api_key_env: LITELLM_API_KEY
 custom_providers:
  - name: harness
-    base_url: http://192.168.68.116/v1
+    api_mode: responses
-    api_key_env: LITELLM_API_KEY      # ← indirection
+    base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
    api_key_env: LITELLM_API_KEY
 auxiliary:
  compression:
    provider: harness
-    api_key_env: LITELLM_API_KEY      # ← indirection
+    base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
    api_key_env: LITELLM_API_KEY
 # ✅ ALSO CORRECT — external providers
 fallback_providers:
  - provider: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-b7d9...               # ← hardcoded OK (external)
-    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in /etc/environment
+    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```
 ```yaml
-# ❌ FORBIDDEN — hardcoded harness/litellm key
+# ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
 model:
  provider: harness
-  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION
+  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
 model:
  provider: harness
  base_url: http://192.168.68.116/v1   # ← RULE VIOLATION: unauthenticated path
  api_key_env: LITELLM_API_KEY
 ```
 ## Detection Query
@@ -75,13 +117,18 @@ Run on any Hermes host to detect violations:
 ```bash
 # 1. Check config.yaml for hardcoded harness keys
-grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
+grep -rn 'api_key: sk-' /root/.hermes/ \
  --include='config.yaml' \
  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
 # 1b. Check for double-path bug: base_url ending with /responses
 # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
 grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
 # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
-grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
@@ -92,26 +139,32 @@ If any output from step 2 — **critical violation** (master key leaked). Fix im
 ## Rotation Procedure
-With this standard enforced, key rotation is one step:
+With this standard enforced, key rotation is one vault update:
 ```bash
-# 1. Generate new key in LiteLLM
+# 1. Generate new key in LiteLLM: POST /key/generate with agent alias
-# 2. Update /etc/environment on agent host
+# 2. Update Infisical vault secret
-ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment"
+infisical secrets set LITELLM_API_KEY=sk-NEW_KEY \
-# 3. Restart agent gateway
+  --project=agents --env=production
-ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &"
+# 3. Restart agent gateway (key auto-injected via infisical run -- wrapper)
 ssh root@<host> "systemctl restart hermes-gateway"
 # 4. Verify
-curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
+curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/litellm/v1/models
 ```
-**Done.** No config file changes needed. The agent picks up the new key on restart.
+**Done.** No config file changes needed. No /etc/environment edits needed.
 The agent picks up the new key via `infisical run --` at gateway startup.
 > **Post-migration note**: /etc/environment is NO LONGER the key source.
 > Strip all `LITELLM_API_KEY` lines from /etc/environment (comment out with `# [INFISICAL]`)
 > and let the `infisical run --` wrapper inject the key at runtime.
 ## Key Longevity Policy (2026-07-04)
 **Keys are permanent and use bare agent name aliases.**
 - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
 - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
 - **Max budget**: $100 per key (config default).
@@ -128,36 +181,54 @@ litellm_settings:
 ## Verified Agents (2026-07-05 update)
-| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
+| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
-|-------|-----|-----|---------------|-----|--------|----------------|---------------|
+|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
-| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
+| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
-| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
+| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
-| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
+| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
-| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
+| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
-| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
 | Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
-### Systemd Service Pattern (2026-07-04 fix)
+> **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
 > LiteLLM key aliases use agent identity, not CT hostname.
-All Hermes agents use systemd to manage their gateway. Two issues were fixed:
+### Migration Status: Authenticated Path
-1. **Drop-in override** — `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` (or user equivalent) had hardcoded `LITELLM_API_KEY` that bypassed `/etc/environment`.
+| Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
-2. **Missing EnvironmentFile** — Services did not source `/etc/environment`.
+|-------|--------------------------|--------------------|--------|
 | Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
 | Tanko | ⚠️ No SSH access | — | Needs check |
 | Koby | ⚠️ No route to host | — | Needs check |
 | Koonimo | ⚠️ Connection timed out | — | Needs check |
-**Correct pattern:**
+### Systemd Service Pattern (2026-07-11 — vault migration)
 All Hermes agents use systemd to manage their gateway. The gateway service is wrapped
 with `infisical run --` to inject secrets at runtime.
 **Correct pattern (post-migration):**
 ```ini
-# In service file:
+# Service file wraps gateway with Infisical:
-EnvironmentFile=/etc/environment
+[Service]
 ExecStart=/usr/bin/infisical run --project=agents --env=production -- \
  /usr/bin/hermes gateway run
-# Drop-in only for overrides, NOT primary key storage.
+# /etc/environment is CLEAN — no LITELLM_API_KEY present
-# If a drop-in exists, it must match /etc/environment.
+# (strip it and tag with # [INFISICAL] if present)
 ```
-**Rotation procedure** (one step with this standard):
+**Legacy pattern (deprecated — pre-migration only):**
 ```ini
 # DO NOT USE post-migration:
 EnvironmentFile=/etc/environment
 # This pattern was replaced by infisical run -- wrapper
 ```
 **Rotation procedure** (one vault operation with this standard):
 1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
-2. Update `/etc/environment`: `sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment`
+2. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-NEW --project=agents --env=production`
-3. Update drop-in (if exists): same sed on `litellm-key.conf`
+3. Restart: `systemctl restart hermes-gateway` (key auto-injected via wrapper)
 4. Restart: `systemctl [--user] restart hermes-gateway`
 ## Violation Response
@@ -167,7 +238,7 @@ EnvironmentFile=/etc/environment
 4. **Restart** — gateway must restart to pick up env var
 5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
 6. **Update** — bump the verified table above
-7. **Use safe-mutate** — if the fix requires changing `/etc/environment` or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
+7. **Use safe-mutate** — if the fix requires updating vault secrets or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
 ## Related Contracts
@@ -206,13 +277,13 @@ task config:
 ```yaml
 auxiliary:
  vision:
-    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
+    api_key: sk-<agent-key-from-vault>           # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
  compression:
-    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
+    api_key: sk-<agent-key-from-vault>           # ← workaround (same as above)
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
@@ -0,0 +1,238 @@
 ---
 kind: function
 name: hermes-zulip-plugin
 description: >
  Installs or updates the Zulip platform plugin for any Hermes agent from the
  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
  recovery). Verifies the installation, restarts the gateway, and sends a relay
  success signal.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Plugin — Install & Repair
 Single-shot function that pulls the latest zulip-platform plugin from the
 canonical git repo, installs it to the correct Hermes bundled plugin path,
 verifies the installation, and signals completion.
 Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
 layer and a targeted gateway restart. It does NOT verify env credentials or
 run a live Zulip connection test. Use `hermes-zulip-restore` for full
 connectivity recovery including end-to-end DM validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 ## Maintains
 - plugin_installed: bool — Whether all three adapter files exist at the bundled path
 - plugin_version: string — Git commit SHA of the installed version
 - strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
 - signal_sent: bool — Whether relay success message was dispatched
 ### Postconditions
 - All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
 - `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
 - Installed version matches HEAD of the requested branch
 - Relay success signal sent to Hermes agent's inbox
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
 | Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
 ## Execution
 ### Step 1: Resolve Target
 Map `target` to host, CT ID, hermes_home, and user from the live-state table.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Pull Latest Plugin Source
 On the target host:
 ```bash
 # Ensure deploy scratch space
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 # Clone or pull
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins
  git fetch origin
  git checkout {{branch}}
  git pull origin {{branch}}
 else
  git clone --branch {{branch}} \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
  cd zulip-platform-plugins
 fi
 # Capture installed version
 INSTALLED_SHA=$(git rev-parse HEAD)
 echo "Installed SHA: $INSTALLED_SHA"
 # Verify we're at HEAD
 HEAD_SHA=$(git rev-parse origin/{{branch}})
 if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
  echo "At latest commit on {{branch}}"
 else
  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
 fi
 ```
 ### Step 3: Install Plugin Files
 ```bash
 # Ensure target directory exists
 mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp plugins/platforms/zulip/adapter.py \
   plugins/platforms/zulip/__init__.py \
   plugins/platforms/zulip/plugin.yaml \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only — runs as jerome user)
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 echo "Plugin files installed"
 ```
 ### Step 4: Verify Installation
 ```bash
 # Check all three files exist
 for f in __init__.py adapter.py plugin.yaml; do
  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
    echo "✅ $f present"
  else
    echo "❌ $f MISSING"
    exit 1
  fi
 done
 # Verify _strip_html fix
 grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
  && echo "✅ _strip_html fix present" \
  || echo "❌ _strip_html MISSING — plugin may be stale"
 # Show installed plugin.yaml version
 grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
 ```
 ### Step 5: Restart Gateway
 Plugin changes require a gateway restart to take effect:
 ```bash
 cd {{hermes_home}}/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart 2>&1 || \
  venv/bin/python -m hermes_cli.main gateway restart 2>&1
 ```
 Wait for restart to complete (up to 45s), then confirm:
 ```bash
 grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
 ```
 Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
 Quick smoke check — confirm zulip platform loaded:
 ```bash
 grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
 ```
 If gateway fails to restart, check logs for the crash cause before proceeding.
 ### Step 6: Send Relay Success Signal
 On the Abiba host (local), dispatch a relay message to the target agent:
 ```
 ra-h-os-createRelayNode:
  title: "Zulip plugin updated — {{target}}"
  source: |
    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
    _strip_html fix: PRESENT
    Timestamp: {{timestamp}}
  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
 ```
 ### Step 7: Report
 Compile results into a single status block:
 | Field | Value |
 |-------|-------|
 | Target | `{{target}}` |
 | Branch | `{{branch}}` |
 | Commit SHA | `{{INSTALLED_SHA}}` |
 | Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
 | `_strip_html` | `{{present|missing}}` |
 | Gateway restarted | `{{yes|no}}` |
 | Signal sent | `{{yes|no}}` |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
 | Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
 | `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
 | Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
 | Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
 ## Edge Differences from hermes-zulip-restore
 | Concern | hermes-zulip-restore | hermes-zulip-plugin |
 |---------|---------------------|---------------------|
 | Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
 | Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
 | Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
 | Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
 | Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
 | Signal dispatch | ❌ None | ✅ Relay message to target |
 For full connectivity recovery after a plugin install, chain this contract
 with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
 ---
 **Last updated**: 2026-07-08 — Switched default branch to `master`; added
 gateway restart step. If outstanding unmerged feature branches exist, address
 them in a follow-up merge after this contract completes.
@@ -0,0 +1,189 @@
 ---
 kind: function
 name: hermes-zulip-restore
 description: >
  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
  Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
  path, verifies env credentials, restarts the gateway, and confirms Zulip
  connects. Run this whenever a Hermes agent stops responding on Zulip or after
  a fresh agent deployment.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Restore — Bring Any Agent Back to Good State
 Single-shot function that restores full Zulip connectivity for a Hermes agent.
 Covers adapter deployment, HTML stripping (slash command fix), env verification,
 gateway restart, and connection validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
 ## Maintains
 - adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
 - zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
 - env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
 - gateway_running: bool — Whether gateway process is running
 ### Postconditions
 - `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
 - All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
 - Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
 - Gateway restarted and zulip platform reports state `connected`
 - HTML stripping enabled for `/approve` and `/deny` slash command support
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 - Zulip server accessible at `https://chat.sysloggh.net`
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Zulip server | https://chat.sysloggh.net | ✅ Verified |
 | Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
 ## Execution
 ### Step 1: Locate Target
 Map `target` to connectivity parameters from the live-state table above.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Deploy Zulip Adapter
 On the target host:
 ```bash
 # Clone or update the plugin repo
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
 else
  git clone --branch feat/zulip-streaming \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
 fi
 # Ensure bundled plugin directory exists
 mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
 # Clean up
 rm -rf /tmp/zulip-deploy
 ```
 ### Step 3: Verify _strip_html is Present
 ```bash
 grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
 ```
 Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
 ### Step 4: Verify Env Credentials
 ```bash
 grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
 ```
 Expected: all three variables set with non-empty values. If any missing:
 - ZULIP_SITE: `https://chat.sysloggh.net`
 - ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
 - ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
 ### Step 5: Restart Gateway
 ```bash
 cd <HERMES_HOME>/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
 ```
 Wait for the restart to complete (up to 45s). Check:
 ```bash
 grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
 ```
 Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
 ### Step 6: Validate Zulip Connection
 ```bash
 python3 -c "
 import json
 d = json.load(open('$HERMES_HOME/gateway_state.json'))
 print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
 "
 ```
 Expected: `zulip: connected`. If not connected, check gateway log:
 ```bash
 grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
 ```
 ### Step 7: Report
 Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
 | State | Action |
 |-------|--------|
 | All true | ✅ Agent restored — relay success to user |
 | `adapter_deployed: false` | Re-run Step 2 |
 | `env_valid: false` | Prompt for missing credentials |
 | `zulip_connected: false` | Check Zulip server reachability, verify API key |
 | `gateway_running: false` | Check process logs for crash cause |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
 | Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
 | No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
 | `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
 | Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
 | Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
 ## Git Branch Reference
 The `_strip_html` fix lives on `feat/zulip-streaming` branch:
 ```
 https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
 ```
 Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
 Pull request #33 is the primary integration branch.
 ---
 **Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
@@ -62,19 +62,19 @@ description: >
 | Resource | Auth Method | Credential Source | Status |
 |----------|------------|-------------------|--------|
-| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ |
+| Proxmox Cluster | PVE API Token | Infisical vault (`PROXMOX_API_TOKEN`) | ✅ |
-| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
+| Proxmox Root | Password via API ticket | Infisical vault (`PROXMOX_ROOT_PASSWORD`) | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
 | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
 | Baggy CT (113) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
-| Gitea | API token | abiba-bot token | ✅ |
+| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
-| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
+| Zulip | Bot API key | Infisical vault (`ZULIP_BOT_KEY`) | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
-| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ VERIFY-BEFORE-USE |
+| LiteLLM Admin | master key | Infisical vault (`LITELLM_MASTER_KEY`) | ✅ VERIFY-BEFORE-USE |
-| Grafana | admin password | `syslog-grafana-2026` | ✅ VERIFY-BEFORE-USE |
+| Grafana | admin password | Infisical vault (`GRAFANA_ADMIN_PASSWORD`) | ✅ VERIFY-BEFORE-USE |
 > **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
 > contract are live-state fields. Test them against the live system before
@@ -144,7 +144,7 @@ description: >
 ### Ecosystem A: docker-vm (192.168.68.7)
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
 | **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
 **Last verified:** 2026-07-09 — 16/16 containers running healthy.
 ### Home Stack Services
@@ -547,13 +550,13 @@ curl -s http://192.168.68.116/health/unified | jq .status
 curl -s http://192.168.68.24:9100/gpu-data | jq .summary
 # Grafana status
-curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health
+curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health
 # Prometheus targets
 curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
 # LiteLLM key check
-curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
+curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
 # Storage check
@@ -583,7 +586,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 108 | media | storepve | — | Media | ❌ |
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | — | Git | ❌ |
-| 111 | tdunna | amdpve | — | ? | ❌ |
+| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
 | 113 | baggy | amdpve | ? | Hermes agent | ✅ |
 | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
@@ -7,10 +7,14 @@ description: >
  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
  via existing /metrics Prometheus endpoint.
-  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  DEPLOYMENT STATUS (2026-07-09):
-  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  This contract defines the target state; proxmox-monitor is the as-built.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
     NVIDIA sidecar exporters (.8/.110:9400) never installed.
     Router falls back to direct GPU /health probes.
  ⚠️ This contract is target-state aspirational — not as-built.
     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
@@ -11,7 +11,7 @@ triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
-version: 1.1.0
+version: 1.2.0
 ---
 ## Maintains
@@ -24,13 +24,15 @@ version: 1.1.0
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
-4. ✅ LiteLLM health check passing
+4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
-5. ✅ Zulip server reachable
+5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
-6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
+6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
-7. ✅ Disk >20% free on all nodes
+7. ✅ Zulip server reachable
-8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
+8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 9. ✅ Disk >20% free on all nodes
 10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
 ## Wave 1: Storage & Infra Nodes (lowest impact)
@@ -64,6 +66,7 @@ Before ANY update wave:
 **Verify after Wave 2:**
 - All VMs/CTs running: check via Proxmox API
 - LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
 - LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools
 - GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`
@@ -82,6 +85,7 @@ Before ANY update wave:
 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
 - End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
 - MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
 - Zulip test: send test message to #agent-hub
 - Dashboard loading: `curl localhost:3001/` (via CT 116)
 - Firecrawl test: `curl :3002/`
@@ -111,8 +115,8 @@ If ANY verification fails:
 Before Wave 1, snapshot these files:
 ```
-/opt/inference-harness/docker-compose.yml               (CT 116 .116)
+/opt/inference-harness/docker-compose.yml               (CT 116 .116) ⚡ contains MCP_SERVER env vars
-/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
+/opt/inference-harness/litellm_config.yaml              (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
 /opt/monitoring/prometheus.yml                          (CT 116 .116)
 /etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
 /opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
@@ -122,9 +126,51 @@ Before Wave 1, snapshot these files:
 /root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
 /etc/systemd/system/ornith-server.service               (amdpve .15)
 /etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
 # Hermes agent configs (key enforcement — 2026-07-10)
 /root/.hermes/config.yaml                               (Mumuni CT 114, Tanko CT 112, etc.)
 /root/.config/systemd/user/hermes-gateway.service       (Mumuni CT 114 — EnvironmentFile fixed)
 /etc/environment                                        (Mumuni CT 114 — LITELLM_API_KEY)
 ```
-Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
+## MCP Gateway (2026-07-10)
 LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.
 ### Configuration
 **litellm_config.yaml** (`/opt/inference-harness/litellm_config.yaml`):
 ```yaml
 mcp_servers:
  ra_h_os:
    url: "http://192.168.68.65:3100/mcp"
    transport: "http"
    auth_type: "none"
 ```
 **docker-compose.yml** env vars:
 ```yaml
 - MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
 - MCP_SERVER_RAHOS_TRANSPORT=http
 ```
 ### Access
 | Key | MCP Access |
 |-----|-----------|
 | Master key | ✅ Full — 90 tools (vault-injected) |
 | Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |
 ### Known Limitations
 - Per-key MCP server grants not functional — only master key has access
 - Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
 - HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
 - `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
 ### Migration Path
 When LiteLLM is upgraded to a version supporting per-key MCP grants:
 1. Grant agent keys `mcp_servers: ["ra_h_os"]`
 2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
 3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
 ## Security-Specific Updates
@@ -143,6 +189,7 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
 - [ ] LiteLLM MCP gateway healthy (90 tools via master key)
 - [ ] Zero security CVEs remaining
 - [ ] <10 min total downtime per service
@@ -8,6 +8,15 @@ description: >
  Ensures agents never use the master key directly. Rotation is event-driven,
  not calendar-driven — rotate only on compromise, personnel change, or
  periodic security hygiene (quarterly/annually).
  UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
  BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
  expire/404. The .env fallback prevents agents from running without keys.
  Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
  on CT 116. Last verified: 2026-07-12.
 ---
 ## Parameters
@@ -15,7 +24,10 @@ description: >
 - agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
 - action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
 - litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
- master_key: string — LiteLLM master key (default from environment)
+- master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
 - vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
 - vault_project: string — Infisical project slug (default: "infrastructure")
 - vault_env: string — Infisical environment (default: "production")
 - agent_host: string — Agent's IP for SSH (default: resolved from infra)
 - agent_user: string — SSH user (default: "jerome")
@@ -26,27 +38,32 @@ description: >
 - key_prefix: string — First 10 chars of the new key (for identification)
 - previous_key_alias: string | null — Previous key alias if rotating
 - litellm_response: object — Raw response from LiteLLM /key/generate
- agent_config_updated: boolean — Whether /etc/environment was updated
+- vault_updated: boolean — Whether Infisical vault secret was updated
 - agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
 - verification: { status: string, detail: string } — Final health check
 ## Execution
-1. **Authenticate** — Verify master_key works against LiteLLM /key/list
+1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
 2. **Check existing keys** — List all keys, find any with agent_name alias
 3. **If action == "list"**: Return all keys with their aliases and spend
 4. **If action == "create"**:
   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
   - Return the new key
 5. **If action == "rotate"**:
   - Generate new key with same alias (LiteLLM replaces the old key)
-   - SSH to agent_host, update /etc/environment LITELLM_API_KEY
+   - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
-   - Restart agent gateway (hermes gateway restart for Hermes agents)
+   - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
     The gateway automatically picks up the new key via `infisical run --` wrapper
   - Verify: curl test against /v1/models with new key
   - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
   - Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
 6. **If action == "verify"**:
-   - SSH to agent, read /etc/environment
+   - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
   - Test the key against LiteLLM /v1/models
   - Confirm key alias matches agent_name in LiteLLM key list
   - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
@@ -1,35 +1,50 @@
 ---
 kind: function
 name: litellm-health
 status: deprecated
 deprecated_on: 2026-07-09
 replaced_by: litellm-self-heal.prose.md
 note: >
  Consolidated into litellm-self-heal.prose.md to eliminate duplication
  of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within litellm-self-heal.
  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
-  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
-  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Router (harness-router :9000) is DEPRECATED — container still runs but
-  Prometheus/Grafana and fleet dashboard.
+  is not in the request path. GPU monitoring via Prometheus/Grafana and
  fleet dashboard (gpu-monitor :9100).
  Designed as a reusable contract for any Syslog agent.
  Source of truth: gpu-fleet.prose.md
 ---
-## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 ```
-Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
-                        │                    │
+                        │
-                  Key validation      Model routing
+                  Key validation
-                  Fallback chains     Slot booking
+                  Fallback chains
-                  Complexity router   Circuit breakers
+                  Budget tracking
-                  Budget tracking     Redis-backed
+                        │
-                        │                    │
+                  Prometheus ← metrics
                  Prometheus ← metrics  ←───┘
                        │
                    Grafana :3001
  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
 ```
-**What changed (v3.1.0 → v3.2.0)**:
+**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
- Router is BACK — runs internally on :9000 behind LiteLLM
+- Router REMOVED from request path — LiteLLM proxies directly to GPU
- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- All GPUs at parallel 2 (was parallel 1)
- Router :9000 is 127.0.0.1-only, not publicly accessible
+- NVIDIA context reduced 256K→128K to free VRAM
- GPU fleet dashboard: http://192.168.68.24:9100
+- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
 ## Parameters
@@ -55,27 +70,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se
 ## GPU Fleet Topology
-| Host | IP | Hardware | Models Served | Engine |
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
-|------|-----|----------|---------------|--------|
+|------|-----|----------|---------------|--------|---------|----------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
-| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 ## Model Fallback Chains (LiteLLM)
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
-| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
-| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | — | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
 > Global: request_timeout=300s, nginx proxy_read_timeout=600s
 ## Containers on CT 116
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
@@ -1,27 +1,99 @@
 ---
 kind: responsibility
 name: litellm-self-heal
-status: manual-only
+status: deployed
 note: >
-  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
+  DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
-  This contract is now manual-only — triggers require explicit user request.
+  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
-  Consider reimplementing as a standalone cron job or prose contract.
+  now reimplemented as `litellm-health-check.sh` on CT 116.
  Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
  GPU monitoring integrated from gpu-monitor on .24:9100.
  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
  duplication of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within this contract.
  Source of truth for GPU topology and keys: gpu-fleet.prose.md
  Last verified: 2026-07-12
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
-  knowledge graph for full audit trail.
+  inference, and agent keys. Applies remediation rules for common failures.
  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 # LiteLLM Operations — Health Check + Self-Heal
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 ```
 Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001
  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
 ```
 ## Parameters
 - public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
 - backend_host: string — Internal CT host (default: "192.168.68.116")
 - auth_host: string — Authentik OIDC server (default: "192.168.68.11")
 - gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
 - gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
 - grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
 ## GPU Fleet Topology
 | Host | IP | Hardware | Models Served | Engine | Context | Parallel |
 |------|-----|----------|---------------|--------|---------|----------|
 | llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 1 |
 | ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 131K | 2 |
 | amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 ## Model Fallback Chains (LiteLLM)
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
 | qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
 | gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
 | syslog-auto (balanced) | 90s | qwen → gemma | — |
 > Global: request_timeout=300s, nginx proxy_read_timeout=600s
 ## Containers on CT 116
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
 | harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
 | harness-dashboard | inference-harness-dashboard | :3000 | /health |
 | harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
 | harness-prometheus | prom/prometheus | :9090 | /-/healthy |
 | harness-docker-stats | python:3.12-alpine | — | container stats exporter |
 | harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
 ## Maintains
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
-## Requires
+- litellm-agent-keys: { count: int, valid: int }
 - litellm-health:function
 ## Continuity
@@ -29,55 +101,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
-## Reporting
+---
-Every remediation cycle produces a structured report in three channels:
+## Health Check
-### 1. RA-H OS Knowledge Graph Node
+Run this first on every cycle. Results feed into remediation rules below.
 Created as `[LEARN] litellm-self-heal: <run_id>` with:
 - metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
 - source: Full JSON report of the cycle
 - description: Summary of what was found, fixed, and escalated
-### 2. Zulip DM to Owner
+### 1. Check public endpoints
-Sent immediately for:
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
 - `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
 - Every 10th clean cycle — "✅ All Clear (10 checks passed)"
-### 3. Daily Digest (end of day)
+### 2. Check LiteLLM health (no-auth)
-A summary of the last 24 hours sent as a single Zulip DM:
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
 ```
 📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
-Total cycles: 288 (every 5 min)
+### 3. Check backend container health
-Issues found: 3
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+- Critical: harness-litellm, harness-nginx, harness-postgres
-  └─ Escalated: 0
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
 - Deprecated but running: harness-router (not in path, reference only)
-Top actions:
+### 4. Check GPU fleet health (via fleet dashboard)
-  • harness-nginx restarted — 2026-06-26 02:15
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
-  • DOCS_URL corrected — 2026-06-26 07:30  
+- Verify GPUs reporting status "healthy"
-  • harness-nginx restarted — 2026-06-26 14:46
+- Check alerts array for active warnings/critical
-Uptime: 23h 47m — All healthy now ✅
+### 5. Check model inference via LiteLLM — test each model
-```
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
 - POST /v1/chat/completions model=ornith-1.0-35b → expect 200
 - Use master key for auth
-### 4. Weekly Digest (every Monday)
+### 6. Check agent keys
-Same format as daily but covering 7 days. Sent to both Zulip DM and
+- GET /key/list with master key → verify all 6 agents have keys
 logged as a `[REPORT]` knowledge graph node for long-term trending.
-### 5. Relay Message (if cross-agent)
+### 7. Check Grafana
-If a fix required another agent's help (e.g., Authentik restart), a relay
+- GET {{grafana_url}}/api/health → expect 200
 message is sent to the responsible agent with full context.
-## GPU Fleet
+### 8. Compile overall status
 Determine overall_status from individual check results:
 - "healthy" — all checks pass
 - "degraded" — 1-2 non-critical checks fail
 - "down" — critical checks fail
-| Host | IP | Hardware | Models | Engine |
+---
 |------|-----|----------|--------|--------|
 | amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
 | llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
 | ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
 ## Remediation Rules
@@ -105,39 +171,89 @@ Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts
-### Rule 7: Router Returns 503 (All GPUs Saturated)
+### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
-Detect → router logs show QUEUE_TIMEOUT with GPUs idle
+Router is no longer in the request path (2026-07-08). LiteLLM proxies
-Root cause → roster not loaded at startup (empty TIER_MODELS)
+directly to GPU. This rule is retained for reference but is inactive.
-Fix → POST /admin/roster/reload (if endpoint available) or restart router
+If 503 errors occur, check LiteLLM timeouts and GPU health directly.
 Permanent fix → add load_roster() to router startup code
 Escalate → if reload doesn't resolve within 2 min
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
-Root cause → keys in router env var, not in LiteLLM DB
+Root cause → keys not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM
-### Rule 9: Stale Active Counter in Redis
+### Rule 9: Stale Active Counter in Redis — DEPRECATED
-Detect → active:<model> > 0 while GPU idle
+Router no longer in path so Redis active counters are unused. Rule retained
-Fix → `redis-cli DEL active:<model>` → restart router
+for reference but inactive. If Redis issues occur, check harness-redis container.
-Prevent → add TTL to active keys in router code
+
 ---
 ## Reporting
 Every remediation cycle produces a structured report:
 ### 1. RA-H OS Knowledge Graph Node
 Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
 ### 2. Zulip DM to Owner
 - `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
 - `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
 - Every 10th clean cycle — "✅ All Clear (10 checks passed)"
 ### 3. Daily Digest (end of day)
 Summary of last 24 hours: total cycles, issues found/fixed/escalated,
 top actions, uptime.
 ### 4. Relay Message (if cross-agent)
 If a fix requires another agent (e.g., Authentik restart), relay sent
 to responsible agent with full context.
 ---
 ## Execution
-1. Run health check
+```prose
-2. Apply remediation for each failure
+-- Phase 1: Health Check
-3. **Generate report** with all actions taken
+let health = call health-check
-4. **Log to knowledge graph** — create `[LEARN]` node
+  public_url: public_url
-5. **Notify user** via Zulip DM if anything changed
+  backend_host: backend_host
-6. Wait 300s and repeat
+  gpu_dashboard_url: gpu_dashboard_url
  grafana_url: grafana_url
 -- Phase 2: Apply remediation for each failure
 let actions = []
 for check in health.failed:
  let fix = apply-remediation-rule
    rule: lookup-rule(check.name)
    target: check.target
  push actions fix
 -- Phase 3: Generate report
 call report-generator
  health: health
  actions: actions
 -- Phase 4: Log to knowledge graph
 call kg-logger
  run_id: run_id
  health: health
  actions: actions
 -- Phase 5: Notify if anything changed
 if actions.length > 0:
  call zulip-notifier
    actions: actions
    health: health
 -- Wait 300s and repeat
 ```
 ## Audit Trail Format
 ```json
 {
-  "run_id": "self-heal-20260626-001",
+  "run_id": "self-heal-20260709-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "timestamp": "2026-07-09T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
@@ -151,7 +267,7 @@ Prevent → add TTL to active keys in router code
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
@@ -1,7 +1,7 @@
 ---
 name: memory-audit-maintenance
 kind: responsibility
-description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---
@@ -15,13 +15,13 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,
 ### Scope
-This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
 **Agent Roster:**
 - Mumuni
 - Tanko
- Tdunna
+- Koby (CT 111 / tdunna)
- Baggy
+- Koonimo (CT 113 / baggy)
 **Isolation Principle:** Each agent has its own:
 - `MEMORY.md` and `USER.md`
@@ -12,6 +12,7 @@ description: >
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
 - gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
@@ -41,7 +41,7 @@ agent: abiba
 - User: `monitoring@pve` (cluster-replicated)
 - Role: `PVEAuditor` on `/` (read-only, whole cluster)
- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
+- Token: `monitoring@pve!prometheus` — stored in Infisical vault (`PROXMOX_MONITOR_TOKEN`)
 - `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
 ## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
@@ -62,7 +62,7 @@ agent: abiba
 - **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
 - **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
- **Credentials**: admin / syslog-grafana-2026
+- **Credentials**: admin / password stored in Infisical vault (`GRAFANA_ADMIN_PASSWORD`)
 - Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
 - grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
@@ -19,13 +19,53 @@ from datetime import datetime
 LITELLM = "http://192.168.68.116:80"
 def _get_agent_key(agent_name):
    """Retrieve agent key from Infisical vault."""
    try:
        result = subprocess.run(
            ["infisical", "secrets", "get", "LITELLM_API_KEY",
             "--project=agents", "--env=production", "--plain"],
            capture_output=True, text=True, timeout=10
        )
        if result.returncode == 0:
            return result.stdout.strip()
    except Exception:
        pass
    # Fallback: try exporting all secrets
    try:
        result = subprocess.run(
            ["infisical", "export", "--project=agents", "--env=production",
             "--format=dotenv"],
            capture_output=True, text=True, timeout=10
        )
        if result.returncode == 0:
            for line in result.stdout.splitlines():
                if line.startswith(f"LITELLM_API_KEY_{agent_name.upper()}") or \
                   (line.startswith("LITELLM_API_KEY=") and agent_name == os.uname().nodename):
                    return line.split("=", 1)[1].strip().strip('"').strip("'")
    except Exception:
        pass
    return None
 # Agent keys are pulled from Infisical vault at runtime.
 # The 'key' field is populated dynamically below.
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome"},
-    "mumuni":  {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"},
+    "mumuni":  {"ct": 114, "host": "192.168.68.123", "user": "root"},
-    "tdunna":  {"ct": 111, "host": None,              "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None},
+    "koby":    {"ct": 111, "host": None,              "user": None},
-    "baggy":   {"ct": 113, "host": None,              "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None},
+    "koonimo": {"ct": 113, "host": None,              "user": None},
 }
 # Inject keys from vault
 for agent_name in AGENTS:
    key = _get_agent_key(agent_name)
    if key:
        AGENTS[agent_name]["key"] = key
    else:
        AGENTS[agent_name]["key"] = None
 GPU_HOSTS = {
    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
@@ -30,6 +30,12 @@ declare -A CT_NODES=(
    # acerpve (192.168.68.9)
    [102]=acerpve   # adguard
    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
    #
    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
    #   118 jitsi     → stopped, not in service
 )
 # Each node must be root-accessible via SSH hostname
@@ -0,0 +1,92 @@
 ---
 kind: responsibility
 name: zulip-oidc-redirect-fix
 status: active
 description: >
  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
  uses the internal IP (192.168.68.19) instead of the public domain
  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
  Survives container restarts through compose.override.yaml.
 agent: abiba
 triggers:
  - Zulip OIDC login returns "Redirect URI Error" from Authentik
  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
  - After Zulip server restart, Authentik SSO login broken
 ---
 ## Maintains
 - zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
 - authentik-acceptance: { status: "accepted" | "rejected" }
 - patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
 ## Detection
 ### Rule 1: Wrong redirect_uri host
 - **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
 - **Status**: CRITICAL — Authentik will reject
 - **Trigger fix** → Execute self-heal
 ### Rule 2: OIDC flow broken
 - **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
 - **Diagnose**: Check redirect_uri in the 302 Location header chain
 ## Fix (Self-Heal)
 Two layers applied:
 ### Layer 1: Live patch (inside container, immediate)
 ```bash
 # Add to /home/zulip/deployments/current/zproject/computed_settings.py:
 SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
 import urllib.parse
 from social_core.strategy import BaseStrategy
 _original_absolute_uri = BaseStrategy.absolute_uri
 def _patched_absolute_uri(self, path=None):
    from django.conf import settings
    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
    if path is not None:
        return urllib.parse.urljoin(root, path)
    return root
 BaseStrategy.absolute_uri = _patched_absolute_uri
 # Restart Django
 supervisorctl restart zulip-django
 ```
 ### Layer 2: Persistent fix (compose.override.yaml)
 The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
 `/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
 ### Verification
 ```bash
 STEP1=$(curl -sk -w "%{redirect_url}" \
  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
 curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
 # Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
 # Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
 ```
 ### Rollback
 Remove the patch block from `compose.override.yaml` and restart the container:
 ```bash
 docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
 ```
 ## Root Cause
 After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
 Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
 Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
 `Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
 Django falls back to the server's IP.
 The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
 `ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
 Host header.
 ## Related Contracts
 - `zulip-health.prose.md` — General Zulip health monitoring
 - `zulip-self-heal.prose.md` — RETIRED (pi extension removed)
Author	SHA1	Message	Date
root	0c298eb9d9	feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60) PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s Details - Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract	2026-07-12 23:38:21 +00:00
root	af41f8f57a	fix: remaining template YAML and topology table corrections - hermes-config-template: Template YAML compression model gemma→ornith. Context guidance updated (qwen=256K, only gemma limited to 131K). - litellm-self-heal: GPU topology table fixed (RTX 3090: 128K→256K, parallel 2→1)	2026-07-12 22:55:24 +00:00
root	be02b0e843	fix: stale dates, context values, and architecture references - gpu-fleet: Updated all context references (128K→256K for RTX 3090). Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram (router labeled deprecated/not in path). Corrected VRAM values. Updated dates: June→July 2026. - hermes-config-template: Fixed description (removed incorrect '128K on NVIDIA' claim). Updated to reflect verified 256K on RTX 3090. - litellm-self-heal: Updated last-verified date. - litellm-api-keys: Updated last-verified date.	2026-07-12 22:54:30 +00:00
root	79d4a73895	docs: lessons learned from 2026-07-12 session - gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement	2026-07-12 22:49:39 +00:00
root	19b6db9891	feat: GPU workload redistribution — compression → Strix Halo - Move compression model from gemma-4-12b (RTX 5070) to ornith-1.0-35b (Strix Halo) - Add Rule 8: GPU Workload Distribution — per-GPU role assignment - Add Rule 9: Compression Threshold for 256K models - Update Rule 7: Auxiliary Model Consistency with new compression routing - Add gpu-self-heal.prose.md contract with 10 remediation rules - Strix Halo (64GB, 256K, 72.4 tok/s) → compression specialist - RTX 5070 (12GB) → vision/web search specialist - RTX 3090 (24GB, 256K) → heavy reasoning specialist - All rules grilled and confirmed with Kwame 2026-07-12	2026-07-12 22:27:15 +00:00
abiba-bot	0134ad8be6	Merge pull request 'fix: cron alignment + VM→bare metal drift' (#11 ) from fix/cron-alignment-jul2026 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details	2026-07-09 06:28:04 +00:00
root	546ca79849	fix: cron alignment + VM→bare metal drift PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s Details - pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract) - pm2-self-heal: added gpu-monitor to tracked PM2 processes - infra-monitor+control: removed daytime-only restriction (now 24×7) - infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110 - Added @reboot pm2 resurrect for persistence across reboots - README: zulip-monitor.sh → agent-health-check.py - Cleaned up commented-out zulip-monitor cron line Live actions: gpu-monitor-server added to PM2 (persistent across reboots)	2026-07-09 06:27:48 +00:00
abiba-bot	c31dbad95f	Merge pull request 'fix: IP drift and container count from live testing' (#10 ) from fix/testing-drift-jul2026 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details	2026-07-09 06:24:07 +00:00
root	d829f595b4	fix: IP drift and container count from live testing PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 13m1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details Testing the consolidated litellm-self-heal contract against live infra found: - IP drift: pi host is .24, not .65 (confused with ra-h-os bridge) - Fixed gpu-monitor.prose.md: 192.168.68.65 → 192.168.68.24 (3 locations) - Fixed gpu-fleet.prose.md: .65 → .24 in config files + agent keys tables - Container count: 10 running on CT 116, not 8 as contract stated - Added harness-docker-stats + harness-pve-exporter to litellm-self-heal - Updated container health check list Verified: GPU monitor restarted on .24:9100, ornith model inference working, all 10 containers healthy.	2026-07-09 06:23:44 +00:00
abiba-bot	4164ee5ed0	Merge pull request 'feat: consolidate litellm-health into litellm-self-heal' (#9 ) from feat/consolidate-litellm-contracts into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 2s Details	2026-07-09 06:02:47 +00:00
root	053d30ce7c	feat: consolidate litellm-health into litellm-self-heal PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m53s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details - litellm-self-heal: merged health check as §Health Check section, includes full architecture diagram (v4.0.0), GPU topology, timeout tables, container list, and execution flow. Eliminates duplication between the two contracts. - litellm-health: marked DEPRECATED, retained for reference - README.md: updated with consolidated contract, deprecation notice, disk-gc-threat-response, and infrastructure-monitoring status	2026-07-09 06:01:48 +00:00
abiba-bot	e4ab43f1c6	Merge pull request 'fix: LiteLLM contracts — align with 2026-07-08 architecture changes' (#8 ) from fix/litellm-contracts-jul2026 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-07-09 05:37:48 +00:00
root	a2c5a14c74	fix: LiteLLM contracts — align with 2026-07-08 architecture changes PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m29s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details - litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly), GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s), Strix Halo: CPU→Vulkan, context sizes + parallel slots added - litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED (router not in path), GPU topology synced with gpu-fleet - litellm-api-keys: reference gpu-fleet as source of truth for keys, removed qwen3.6-35B-A3B from model list (never deployed)	2026-07-09 05:37:16 +00:00
abiba-bot	a7fedf71c6	Merge pull request 'fix: contract improvements from 2026-07-09 run log review' (#7 ) from fix/contract-improvements-jul2026 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Failing after 13m28s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details	2026-07-09 05:33:25 +00:00
root	97977f320c	fix: contract improvements from 2026-07-09 run log review PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m43s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details - pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs - disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope, remove jitsi (decommissioned), update access matrix with KVM VM section - infrastructure-control: docker-vm container count 11→16, add Trove agents stack - infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)	2026-07-09 05:32:00 +00:00
abiba-bot	a3786ab289	Merge pull request 'disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB)' (#6 ) from feat/disk-gc-amdpve-2026-07-09 into master	2026-07-09 05:23:30 +00:00
root	15e8998289	disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details	2026-07-09 05:20:42 +00:00
root	25c6c76fb3	disk-gc: 2026-07-09 run — amdpve Docker bloat resolved (11.56GB reclaimed), access matrix documented, pct-run verified for all 15 CTs PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details	2026-07-09 05:17:05 +00:00
root	4819247e39	Add zulip-oidc-redirect-fix prose contract PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details Fixes Zulip OIDC authentication where redirect_uri used internal IP (192.168.68.19) instead of public domain (chat.sysloggh.net) after server restart. Monkey-patches social_core.strategy.BaseStrategy to use ROOT_DOMAIN_URI for all redirect URI construction.	2026-07-09 00:02:18 +00:00
root	1dc4155d9a	fix(hermes-zulip-plugin): default to master branch, add gateway restart PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details - Switched default branch from feat/zulip-streaming to master - Added Step 5: gateway restart with smoke check after plugin install - Plugin changes require restart to take effect for a valid test - Outstanding feature branch merges should be addressed post-install	2026-07-08 17:51:19 +00:00
root	09efbdf4e5	feat: add hermes-zulip-plugin (install & repair) and relocate hermes-zulip-restore to repo PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details - hermes-zulip-plugin.prose.md: installs latest zulip-platform plugin from canonical repo, verifies _strip_html fix, sends relay success signal. Distinct from restore — plugin-layer only, no env/gateway/connection checks. - hermes-zulip-restore.prose.md: relocated from /root/ into prose-contracts repo. - Removed duplicate zulip-platform-verification from /root/vault/contracts/.	2026-07-08 17:47:23 +00:00
jerome	4058010e57	Merge pull request 'feat: hermes-zulip-restore contract' (#5 ) from feat/hermes-zulip-restore into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details Reviewed-on: #5	2026-07-08 17:38:16 +00:00
jerome	c0191c9edc	feat: hermes-zulip-restore contract — restore Zulip on any Hermes agent PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details Deploys zulip-platform adapter to bundled plugin path, verifies env credentials, restarts gateway, and validates Zulip connection. Covers Mumuni (CT114), Tanko (CT112), Koby (CT111). Includes _strip_html slash-command fix, known failure modes, and owner-aware restart paths.	2026-07-08 03:19:01 -04:00