Revert "infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths"

This reverts commit 09ffbf10aa.
Revert "infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks"
2026-07-06 18:46:36 -04:00 · 2026-07-06 18:46:36 -04:00 · 2026-07-06 18:46:36 -04:00
25 changed files with 341 additions and 1761 deletions
@@ -21,7 +21,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -34,7 +34,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -64,7 +64,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -77,7 +77,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -115,11 +115,10 @@ Run on trigger or schedule. Maintain persistent world-model state across runs.
 | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
 | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
 | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
+| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
-| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |

 ### Instantiable Templates (`kind: pattern` — run with `prose run`)
@@ -147,9 +146,8 @@ Called on-demand as single-render tools.

 | Contract | Description |
 |---|---|
-| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
-| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
-| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
+| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
+| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |

@@ -160,8 +158,8 @@ Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
-| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
+| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |

 ## Contract Structure

@@ -21,11 +21,9 @@ description: >
  - connected == true — Establishes and maintains Zulip event queue
  - reconnect_on_failure == true — Reconnects after queue expiry
  - recovers_from_network_loss == true — Handles temporary network drops
-  - stream_subscription_check == true — Bot is subscribed to primary streams (Gen 5+)
  
  *Message Flow*
  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
-  - dm_response_time_offline_verifiable == true — simulate_dm_latency() passes without live Zulip (Gen 6+)
  - stream_mention_detection == true — @mentions in streams are detected and routed
  - all_bots_detection == true — @all-bots mentions are detected
  - echo_loop_prevention == true — Never replies to its own messages
@@ -40,14 +38,6 @@ description: >
  - graceful_disconnect == true — shutdown doesn't crash
  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
  - typing_indicators_work == true — Sends typing notifications
-  - known_issues_tracked == true — add_known_issue() persists problems across generations (Gen 6+)
-
-## Status
-
-**Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
-for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
-Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
-plugin system, which is unaffected.

 ## Parameters

@@ -100,28 +90,13 @@ Each generation stores:

 ## Execution

-1. **Read state** — Check previous generations from knowledge graph node `build-zulip-plugin:state`
+1. **Read state** — Check previous generations from knowledge graph
 2. **Generate or improve** — Based on generation count:
   - Generation 1: Scaffold full plugin from scratch
   - Generation N: Read previous code, apply improvements
 3. **Validate** — Syntax check, structure check, dependency check
-4. **Run offline tests** — Always run `simulate_dm_latency()` and selftest() if no live Zulip
-5. **Log** — Save generation result to knowledge graph as [LEARN] node. Update the
-   `build-zulip-plugin:state` node with current generation_count, plugin_version,
-   and known_issues.
-6. **Report** — Output what was generated, what changed, what needs review
-
-## State Persistence (Gen 6+)
-
-All generation state is stored in a knowledge graph node titled `build-zulip-plugin:state`:
- `generation_count` — incremented on each verified run
- `plugin_version` — current version string
- `last_generation` — ISO timestamp of most recent run
- `known_issues` — array of unresolved issues carried forward
- `last_postcondition_results` — pass/fail map from most recent verification
-
-Agents executing this contract MUST read this node on startup and update it after
-validation. If the node does not exist, create it with generation_count=1.
+4. **Log** — Save generation result to knowledge graph as [LEARN]
+5. **Report** — Output what was generated, what changed, what needs review

 ## Example Output

@@ -3,11 +3,10 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
  Recurring disk health scan, garbage collection, and threat response
-  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
+  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
  Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
-  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
+  Recovered 35.67GB via `docker system prune -a --force`.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -22,21 +21,15 @@ logged within 5 minutes of discovery.

 ## Scope

-All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
-Docker hosts get special attention:
+All CTs in the Proxmox cluster, with special attention to Docker hosts:

 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
-| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
+| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All other CTs | — | LOW — no Docker | apt clean, log rotate |
-| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
-
-> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
-> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
+| All others | — | LOW — no Docker | apt clean, log rotate |

 ## Threat Levels

@@ -130,7 +123,7 @@ call summary-reporter

 ## GC Strategies by Host Type

-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)

 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -231,76 +224,32 @@ dangling images and orphaned build cache. No automated GC was in place.
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```

-## Incident Log: 2026-07-09 — amdpve docker bloat
+Done. Here's what we have:

-### Discovery
-Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
-amdpve (.15) flagged at 78% (AMBER threshold: 75%).
+## Storage Summary

-### Diagnosis
- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
-  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
- Not a running service — zero containers, zero active volumes
+| System | Used | Free | Health |
+|--------|------|------|--------|
+| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
+| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
+| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
+| **/root/go** (abiba) | 773M | — | Fine |

-### Resolution
-```
-docker system prune -a --force
-→ Reclaimed 11.56GB
-→ Post: 55G / 94G (62%), 35G free
-→ 1 image removed (llama-strix-hip:latest, 15.5GB)
-→ 8 build cache layers removed
-→ amdpve now GREEN
-```
+## New Contract: `disk-gc-threat-response.prose.md`

-### Root Cause
-Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
-artifacts. Docker on amdpve serves no running purpose — it's only used for
-one-off GPU builds. No automated post-migration cleanup was in place.
+Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:

-### Preventive Measures
- amdpve added to Docker GC scan list
- Post-migration cleanup step added: after any GPU backend migration, prune
-  the old backend's Docker images within 24 hours
- Contract now scans GPU bare-metal hosts alongside CTs
- Access via `pct-run` script for all CTs (no hardcoded IPs)
+### Three-in-One

-## Access Matrix (documented 2026-07-09)
+1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.

-### CT Access (via pct-run)
-| CT | Name | Node | Status |
-|----|------|------|--------|
-| 100 | abiba | amdpve | local |
-| 102 | adguard | acerpve | ✅ reachable |
-| 104 | authentik | minipve | ✅ reachable |
-| 105 | kagentz | amdpve | ✅ reachable |
-| 106 | ra-h-os | storepve | ✅ reachable |
-| 107 | pbs | storepve | ✅ reachable |
-| 108 | media | storepve | ✅ reachable |
-| 110 | gitea | minipve | ✅ reachable |
-| 111 | tdunna | amdpve | ✅ reachable |
-| 112 | tanko | amdpve | ✅ reachable |
-| 113 | baggy | amdpve | ✅ reachable |
-| 114 | mumuni | minipve | ✅ reachable |
-| 115 | scottdenya | amdpve | ✅ reachable |
-| 116 | syslog-api | minipve | ✅ reachable |
-| 117 | zulip | storepve | ✅ reachable |
+2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.

-### GPU Bare Metal (via direct SSH)
-| Host | IP | GPU | Status |
-|------|-----|-----|--------|
-| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
-| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
-| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.

-### KVM VM (via direct SSH)
-| Host | IP | Role | Status |
-|------|-----|------|--------|
-| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
+### The incident root cause
+Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.

-> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
+Want me to push this to the prose-contracts repo?
@@ -5,13 +5,8 @@ description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
-  UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
-  to llama-server on each GPU host (no router in inference path). Router
-  (port 9000) is running but NOT in request path. All GPUs standardized on
-  api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
-  caused cascading 401→timeout→401 fallback loops — fixed.
-  Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
-  Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
+  Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
+  router runs behind LiteLLM on :9000 (internal-only).
 agent: abiba
 triggers:
  - on model add/remove
@@ -34,7 +29,7 @@ triggers:
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
 - port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding

-## Fleet Topology (Current — July 2026)
+## Fleet Topology (Current — June 2026)

 ```
 ┌──────────────────────────────────────────────────────────────────┐
@@ -51,9 +46,9 @@ triggers:
 │  Containers:                                                     │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
 │  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
-│  │ :4000    │  │ :9000    │  │ :3000    │  │ :3000     │        │
-│  │ keys+sync│  │deprecated│  │ harness  │  │ Prometheus│        │
-│  │ fallback │  │not in    │  │ UI       │  │ data src  │        │
+│  │ :4000    │─▶│ :9000    │  │ :3000    │  │ :3000     │        │
+│  │ keys+sync│  │internal  │  │ harness  │  │ Prometheus│        │
+│  │ fallback │  │only!     │  │ UI       │  │ data src  │        │
 │  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
 │                    │                                             │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
@@ -68,50 +63,21 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 256K ctx │  │ 131K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│          │  │             │  │ llama-srv │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
-│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
-│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
+│ 27B-code │  │ :8090       │  │ :8080     │  │ exporter     │
+│ :8090    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
 └──────────┘
 ```

-## Current Model Assignments (2026-07-12)
+## Current Model Assignments

-| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
-|-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
-
-## Routing Configuration (LiteLLM — July 2026)
-
-### syslog-auto Weighted Pool
-
-| Model | GPU | Weight | RPM Cap | Purpose |
-|-------|-----|--------|---------|---------|
-| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context |
-| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling |
-| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision |
-
-### Direct Model Endpoints
-
-| Model | RPM Cap | Notes |
-|-------|---------|-------|
-| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload |
-| qwen3.6-27B-code | 500 | High cap — primary workhorse |
-| gemma-4-12b | 500 | High cap — fast 12B |
-
-### Fallback Chains
- gemma → qwen
- qwen → gemma
- ornith → qwen → gemma
- syslog-auto → qwen → gemma → ornith
-
-### Why ornith RPM Is Capped
- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
+| Model | GPU | Host | VRAM | Status |
+|-------|-----|------|------|--------|
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |

 ## Operations

@@ -147,47 +113,29 @@ triggers:

 ### sync-keys
 1. List all agent keys in LiteLLM DB via `GET /key/list`
-2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
+2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
 3. Generate missing keys via `POST /key/generate` with unlimited budget
-4. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production`
-5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
+4. Update agent configs — `/etc/environment` LITELLM_API_KEY
+5. Send Zulip DM to agents that can't be reached via SSH
 6. Verify each key with test request through full chain
 7. Document keys in knowledge graph

 ### list
-Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys
+Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys

-### health-check
-1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
-2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
-3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
-4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
-5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
-   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
-   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
-6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
-7. Check port conflicts: verify only one llama-server on :8080 per host
-8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
+## Agent Keys (LiteLLM DB — Current 2026-06-30)

-## Agent Keys (LiteLLM DB — Current 2026-07-11)
+| Agent | CT | IP | Key | Access |
+|-------|-----|-----|-----|--------|
+| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
+| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
+| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
+| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
+| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |

-Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY).
-Agent gateways inject keys at runtime via `infisical run --` wrapper.
-Plaintext keys removed from this contract post-vault-migration.
-
-| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
-|-------|-----|-----|---------------|------------|--------|
-| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
-| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
-| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
-| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
-| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
-| Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | no SSH |
-
-> **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.
-
-**Key update procedure**: Update Infisical vault → `infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production` → restart agent gateway. Agent picks up new key via `infisical run --` wrapper at startup.
-If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
+**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
+If no SSH access, send Zulip DM via abiba-bot.

 ## Configuration Files

@@ -198,10 +146,10 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
+| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |

@@ -209,7 +157,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.

 | Component | URL | Details |
 |-----------|-----|---------|
-| Grafana | `http://192.168.68.116:3001/` | admin / vault (`GRAFANA_ADMIN_PASSWORD`) |
+| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
 | GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
 | Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
 | GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
@@ -220,11 +168,9 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline).
- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2.
- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`.
- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
+- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
+- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
+- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
@@ -239,8 +185,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.

 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
-| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
+| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
+| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
 | Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |

 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
@@ -248,13 +194,3 @@ Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
 History stored at `/root/data/toks-history.json` with 7-day rolling window.

 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
-
-## Agent Config Implications (2026-07-12)
-
-With RTX 3090 at 256K context (verified July 2026):
- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it
- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified)
- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072`
- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo
- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
@@ -19,7 +19,7 @@ agent: abiba

 ```
 ┌─────────────────────────────────────────────────────────┐
-│  GPU Monitor Server (:9100) — 192.168.68.24              │
+│  GPU Monitor Server (:9100) — 192.168.68.65              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
   │          │          │
@@ -35,7 +35,7 @@ agent: abiba
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
-Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
+Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
 poll .15:8080 directly; must go through router on .116.
 ```

@@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).

 | File | Host | Purpose |
 |------|------|---------|
-| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
-| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
+| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |

 ## Execution
@@ -1,287 +0,0 @@
---
-kind: responsibility
-name: gpu-self-heal
-description: >
-  GPU fleet self-healing — detects anomalies, applies remediation, tracks
-  benchmarks, and predicts failures before they happen. Extends gpu-monitor
-  (v2.1.0) with active remediation rules, Prometheus metrics consumption,
-  VRAM trend analysis, and predictive alerting.
-agent: abiba
-depends_on:
-  - gpu-monitor.prose.md (live data source on .24:9100)
-  - gpu-fleet.prose.md (source of truth for topology)
---
-
-## Maintains
-
- gpu-health: { status: "healthy"|"degraded"|"down", issues: array, actions: array }
- gpu-self-heal-log: array of { timestamp, gpu, issue, action, result } — audit trail
- benchmark-regression: { gpu, baseline_tok_sec, current_tok_sec, trend, alerts }
- vram-trend: { gpu, current_mb, rate_mb_per_hour, projected_full_in_hours }
- circuit-breaker-status: { gpu, open, auto_reset_attempted, last_reset }
-
-## Requires
-
- gpu-monitor:function — Live fleet data from .24:9100/gpu-data
- Prometheus exporters on all 3 GPUs (:9400/metrics)
- SSH access to GPU hosts for restart operations
-
-## Continuity
-
- Self-driven: check every 60 seconds against GPU monitor data
- Also wakes on gpu-fleet health degradation
- On fix: verify with benchmark inference test before declaring resolved
- Escalate: after 3 failed remediation attempts → Zulip #agent-hub alert
-
---
-
-## Remediation Rules
-
-### Rule 1: GPU Temperature Critical (>85°C for >2 min)
- **Detect**: Any GPU temp >85°C sustained for 2+ consecutive polls
- **Fix**: 
-  1. Reduce inference concurrency on that GPU (load-side cooling only — NO fan control)
-  2. Redirect new requests to cooler GPUs via LiteLLM fallback chains
-  3. If all GPUs hot, alert about cooling infrastructure
- **Verify**: Temp drops below 80°C within 5 minutes
- **Escalate after**: 3 verification failures → Zulip alert
-
-### Rule 2: VRAM Leak Detection (tiered by GPU capacity)
- **Detect**: VRAM growing at sustained rate over 6+ hour window
-  - RTX 3090 (24GB): ≥100MB/hour
-  - RTX 5070 (12GB): ≥50MB/hour
-  - Strix Halo (64GB UMA): ≥200MB/hour
- **Fix**: 
-  1. Log VRAM snapshot with process list (nvidia-smi/rocm-smi + ps aux)
-  2. If llama-server is the growth source → restart with memory cap flag
-  3. If unknown process → kill and alert
- **Verify**: VRAM growth rate drops below threshold
- **Escalate after**: persistent leak after restart → hardware investigation
-
-### Rule 3: Model Inference Timeout / GPU Stuck
- **Detect**: >50% failure rate over 60s window + 30s grace period (not just single stuck request)
- **Fix**: 
-  1. Restart llama-server on affected GPU host
-  2. Wait 15s for model to reload
-  3. Run benchmark inference test
- **Verify**: Model returns 200 with <30s response, failure rate drops to 0%
- **Escalate after**: 3 restarts in 1 hour → GPU hardware check
-
-### Rule 4: Benchmark Regression (>20% drop)
- **Detect**: gen_tok_per_sec drops >20% below baseline over 3+ benchmarks
- **Fix**:
-  1. Check GPU utilization — if >90%, other process is competing
-  2. Check power limit — if throttled, restore to max
-  3. Check thermal — if hot, apply Rule 1
- **Verify**: Benchmark returns to within 10% of baseline
- **Escalate after**: persistent regression → possible hardware degradation
-
-### Rule 5: Circuit Breaker Stuck Open
- **Detect**: Circuit breaker open >10 minutes with GPU reporting healthy
- **Fix**: 
-  1. Verify GPU /health returns 200
-  2. If GPU healthy, send 1 test inference
-  3. If test succeeds → reset circuit breaker via router API
-  4. 60s cooldown — if CB re-opens immediately, it was legitimate, do NOT re-reset
-  5. Max 1 auto-reset per GPU per hour
- **Verify**: CB closes, inference succeeds, CB stays closed for 60s+
- **Escalate after**: CB won't close after reset → router issue
-
-### Rule 6: Strix Halo Unreachable
- **Detect**: Strix not responding — probe .15:8080 directly (firewall opened .24→.15)
- **Fix**: 
-  1. SSH to .15 → check llama-server process
-  2. Restart llama-server if not running
-  3. Verify through both direct probe AND router
- **Verify**: Direct health probe returns 200, router reports Strix healthy
- **Escalate**: If host .15 itself is unreachable → infrastructure alert
-
-### Rule 7: Prometheus Exporter Down
- **Detect**: Any GPU :9400/metrics unreachable for >2 polls
- **Fix**:
-  1. SSH to GPU host → check prometheus-exporter process
-  2. Restart exporter if dead
-  3. While exporter is down, fall back to nvidia-smi/rocm-smi direct probes
-  4. If exporter is running but unreachable → check firewall/host networking
- **Verify**: :9400/metrics returns 200
- **Escalate after**: 3 failed restarts → networking issue
-
-### Rule 8: Predictive Thermal Warning (two-tier)
- **Detect**: 
-  - Tier 1 (warning): temp >70°C AND rising >2°C/min → reduce concurrency, no alert
-  - Tier 2 (critical): temp >80°C AND still rising → full alert + aggressive load shedding
- **Fix**: 
-  - Tier 1: silently reduce parallel requests to that GPU by 50%
-  - Tier 2: redirect all new requests away, alert #agent-hub, apply Rule 1 logic
- **Verify**: Temp rise rate drops below 1°C/min (Tier 1) or temp drops below 80°C (Tier 2)
- **Escalate**: If Tier 2 triggers and temp still rising after 5 min → possible hardware failure
-
-### Rule 9: Context Window Optimization
- **Detect**: Benchmark tok/s vs baseline for each GPU at current context
-  - RTX 3090 (256K ctx, qwen3.6-27B-code): target 75+ tok/s — currently at baseline
-  - RTX 5070 (131K ctx, gemma-4-12b): target 76+ tok/s — optimal for vision/web role
-  - Strix Halo (256K ctx, ornith-1.0-35b): target 70+ tok/s — currently above baseline
- **Fix**:
-  - If tok/s > baseline → context has headroom, consider increasing
-  - If tok/s < 90% baseline → reduce context by 25% and retest
-  - If tok/s within 10% of baseline → optimal, no change
- **Verify**: Re-benchmark after context change, confirm within 10% of target
- **Escalate**: If context can't be adjusted without significant perf loss
-
-### Rule 10: Workload Distribution Optimization
- **Detect**: GPU roles misaligned with hardware capabilities
- **Target distribution**:
-  - RTX 3090 (24GB, 256K, 75 tok/s) → Heavy reasoning, code gen, long conversations
-  - RTX 5070 (12GB, 131K, 76 tok/s) → Vision/image, web search, quick lightweight tasks
-  - Strix Halo (64GB, 256K, 72 tok/s) → Context compression, summarization, long docs
- **Fix**:
-  - Alert if any GPU is handling workload outside its designated role
-  - Recommend Hermes agent profile updates to match workload to GPU
-  - Track per-GPU request distribution via LiteLLM spend logs
- **Verify**: Each GPU's request pattern matches its designated role within 24h
- **Escalate**: If role mismatch persists >48h → agent profile audit needed
-
---
-
-## Execution
-
-```prose
-- Phase 1: Fetch live GPU data
-let fleet = call gpu-monitor
-  endpoint: "http://192.168.68.24:9100/gpu-data"
-
-- Phase 2: Evaluate each GPU against remediation rules
-let actions = []
-for gpu in fleet.gpus:
-  -- Rule 1: Thermal critical
-  if gpu.temp_c > 85 and sustained_for(gpu, 120):
-    push actions apply-thermal-fix(gpu)
-  
-  -- Rule 2: VRAM leak
-  let vram_rate = calculate-vram-trend(gpu, hours=6)
-  if vram_rate > 50:
-    push actions apply-vram-fix(gpu, vram_rate)
-  
-  -- Rule 4: Benchmark regression
-  let bench = fleet.benchmarks[gpu.hostname]
-  if bench.current_tok_sec < bench.baseline_tok_sec * 0.8:
-    push actions apply-benchmark-fix(gpu, bench)
-
-- Rule 3: Model stuck
-for model in fleet.router.available_models:
-  if model.consecutive_timeouts >= 3:
-    push actions apply-model-restart(model)
-
-- Rule 5: Circuit breaker
-for cb in fleet.router.circuit_breaker:
-  if cb.open and cb.open_duration > 600 and gpu_is_healthy(cb.gpu):
-    push actions apply-cb-reset(cb)
-
-- Rule 6: Strix Halo
-if not fleet.strix.running and pingable("192.168.68.15"):
-  push actions apply-strix-restart()
-
-- Rule 7: Prometheus exporters
-for gpu in fleet.gpus:
-  if not prometheus_reachable(gpu.hostname, 9400):
-    push actions apply-exporter-restart(gpu)
-
-- Rule 8: Predictive thermal
-for gpu in fleet.gpus:
-  let rise_rate = calculate-temp-rise(gpu, minutes=5)
-  if rise_rate > 2.0 and gpu.temp_c < 80:
-    push actions apply-proactive-cooling(gpu)
-
-- Phase 3: Execute actions, verify, log
-for action in actions:
-  let result = execute-with-verify(action)
-  log-to-kg(action, result)
-  if result.failed:
-    escalate-if-needed(action)
-
-- Phase 4: Update health state
-call update-gpu-health
-  gpus: fleet.gpus
-  actions: actions
-  status: derive-overall-status(fleet, actions)
-
-- Wait 60s and repeat
-```
-
-## Audit Trail Format
-
-```json
-{
-  "run_id": "gpu-self-heal-20260712-001",
-  "timestamp": "2026-07-12T16:00:00Z",
-  "gpu": "ct8-rtx3090",
-  "issue": "thermal-critical",
-  "detected": { "temp_c": 87, "duration_s": 180 },
-  "action": "set-fan-100pct",
-  "result": "resolved",
-  "verification": { "temp_c": 76, "after_s": 300 },
-  "escalated": false
-}
-```
-
---
-
-## Reporting
-
-### 1. Knowledge Graph
-Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
-
-### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
- `issues_escalated > 0` → "⚠ GPU Self-Heal — <gpu> needs attention"
- Every 100th clean cycle → "✅ GPU Fleet: All Clear"
-
-### 3. Prometheus/Grafana Integration
- GPU self-heal actions exposed as Prometheus counter metrics
- Dashboard panel: "GPU Interventions (24h)" showing count/type/result
-
-### 4. Weekly Benchmark Report
- Per-GPU tok/s trend over 7 days
- Regression alerts if any GPU degrades >10% week-over-week
-
---
-
-## Design Decisions (Grilled & Confirmed — 2026-07-12)
-
-1. **Fan control**: ❌ NO auto fan control. Load-side cooling only (reduce concurrency, redirect).
-2. **Model restart**: ✅ Only if >50% failure rate over 60s + 30s grace period. Not on single stuck request.
-3. **Strix direct access**: ✅ Open firewall .15:8080 → .24 for direct health probe + restart.
-4. **VRAM thresholds**: Tiered — 100MB/h (RTX 3090), 50MB/h (RTX 5070), 200MB/h (Strix).
-5. **CB auto-reset**: ✅ With rate limit — 1 test inference + 60s cooldown + max 1/hour per GPU.
-6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Original baseline kept in Grafana.
-7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
-8. **Prometheus**: Primary source. Fall back to nvidia-smi/rocm-smi direct probes if exporter down.
-
-## Lessons Learned (2026-07-12)
-
-### L1: API Key Standardization Is Critical
- All GPU llama-servers MUST use the same api-key as the LiteLLM config.
- RTX 5070 had `--api-key sk-loc...5678` while LiteLLM sent `not-needed`.
-  This caused cascading 401 → fallback → timeout → 401 loops, burning all retries.
- **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config.
-
-### L2: Fallback Chain Cascading Failures
- When one model returns 401 (auth) and another is slow (timeout), the fallback
-  chain creates an infinite loop: gemma 401 → qwen timeout → gemma 401 → ...
- **Rule**: If a model returns 401 (auth error), do NOT fall back to it again.
-  Mark it as permanently failed for this request.
-
-### L3: Verify Running State, Not Docs
- RTX 3090 was documented at 128K context. Actually running at 256K.
- Parallel count wrong (docs said 2, actual is 1 on RTX 3090).
- **Rule**: Before making decisions, check `/proc/PID/cmdline` on GPU hosts.
-
-### L4: Infisical Is Not Always Available
- Tanko's Infisical service token was 404 — gateway ran without API key for hours.
- **Rule**: Always keep a local `.env` fallback for `LITELLM_API_KEY`.
- Contract hermes-config-template Rule 3 updated.
-
-### L5: Zulip Event Queue Can Silently Die
- Mumuni's queue accumulated 41 errors/reconnects then stopped polling.
-  Gateway was running but ignoring all messages.
- **Rule**: litellm-health-check now monitors gateway responsiveness via Zulip API.
@@ -1,11 +1,11 @@
 ---
-kind: template
+kind: reference
 name: hermes-agent-baseline
 version: 1.0.0
 description: >
  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
  configuration state, keys, workarounds, and audit procedure. When an agent's
-  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-05.
 author: Abiba (pi agent)
 ---

@@ -22,39 +22,29 @@ done

 ## Agent Map

-| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
-|-------|-----|------|-----|---------------|------------|----------|
-| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
-| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
-| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
-| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
-| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
-
-> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
+| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias |
+|-------|-----|------|-----|-------------|---------------|
+| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` |
+| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` |
+| Tdunna | 111 | amdpve | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | `tdunna` |
+| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` |

 Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
-Keys are stored in Infisical vault (project=agents, env=production) and injected at
-runtime via `infisical run --` wrapper. Plaintext keys removed from this baseline.

 ## Key Architecture

 ```
-Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
-                       ↓
 Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
-                                     └── Key DB (Postgres)
+                                                     └── Key DB (Postgres)
 ```

- **Master key**: stored in Infisical vault (project=infrastructure, secret=LITELLM_MASTER_KEY) — ADMIN ONLY
+- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
 - **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key injection**: `infisical run --project=agents --env=production -- hermes gateway run` injects `LITELLM_API_KEY` at runtime
- **Key source**: Infisical vault → runtime env var. /etc/environment is CLEAN (stripped, tagged `# [INFISICAL]`)
- **Legacy override** (pre-migration): `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` — should be REMOVED
+- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
+- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)

 ## Config Pattern — Mandatory Fields

-### For Hermes Agents (Tanko, Mumuni, Koonimo)
-
 Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:

 ### 1. Main Model
@@ -84,7 +74,7 @@ auxiliary:
    model: gemma-4-12b              # or syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
-    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30
 ```
@@ -99,7 +89,7 @@ auxiliary:
    model: syslog-auto               # or gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
-    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 120
 ```

@@ -118,7 +108,8 @@ LiteLLM/harness will fail with:
 401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
 ```

-**Workaround**: Set `api_key` directly (copy the value from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=agents --env=production`) alongside `api_key_env` in every auxiliary task config that uses the harness provider.
+**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
+`api_key_env` in every auxiliary task config that uses the harness provider.

 **Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
 `api_key` is empty:
@@ -136,11 +127,7 @@ if not cfg_api_key:
 ```bash
 for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
-    # Verify /etc/environment is CLEAN (no LITELLM_API_KEY)
-    pct-run $ct "grep -c LITELLM_API_KEY /etc/environment 2>/dev/null || echo '0 (clean)'"
-    # Verify gateway uses infisical run wrapper
-    pct-run $ct "ps aux | grep 'infisical run' | grep -v grep"
-    # Check for hardcoded harness keys
+    pct-run $ct grep LITELLM_API_KEY /etc/environment
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
 done
@@ -148,17 +135,15 @@ done

 ### Master Key Leak Check
 ```bash
-# On every agent — must return empty:
-pct-run <CT> grep -rl "sk-litellm" /root/ /etc/ 2>/dev/null
-# Vault is the only place the master key should exist
+# On every agent:
+pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
+# Must return empty
 ```

 ### Verify Key Works
 ```bash
-# Retrieve key from vault and test:
-KEY=$(infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
 curl -s http://192.168.68.116:80/v1/models \
-  -H "Authorization: Bearer $KEY" | grep syslog-auto
+  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
 # Must return model list
 ```

@@ -169,59 +154,6 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```

-### For Koby (CT 111 / tdunna)
-
-Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
-Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
-
-**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
-
-### For pi Agents (Abiba)
-
-Abiba (CT100) runs pi via PM2 with the Zulip extension.
-Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
-
-**models.json** — Must only list models authorized for the agent's LiteLLM key.
-Key is injected via `infisical run --` wrapper at PM2 startup:
-```json
-{
-  "providers": {
-    "syslog-harness": {
-      "baseUrl": "http://192.168.68.116/v1",
-      "api": "openai-completions",
-      "apiKey": "${LITELLM_API_KEY}",
-      "models": [
-        { "id": "syslog-auto" },
-        { "id": "ornith-1.0-35b" },
-        { "id": "qwen3.6-27B-code" },
-        { "id": "gemma-4-12b" }
-      ]
-    }
-  }
-}
-```
-
-**settings.json** — Always use `syslog-auto` as default:
-```json
-{
-  "defaultProvider": "syslog-harness",
-  "defaultModel": "syslog-auto"
-}
-```
-
-**Validation**: Verify models match LiteLLM's authorized list:
-```bash
-curl -s http://192.168.68.116:4000/v1/models \
-  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
-  | jq '.data[].id'
-```
-
-**Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
-a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
-stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.
-
-**Service**: `pm2 restart koby-zulip`, health at `:9201/health`.
-
 ## Systemd Pattern

 For agents where the gateway runs as a user service:
@@ -278,6 +210,5 @@ If they differ → ghost detected → kill ghost → start fresh.

 | Date | Change |
 |------|--------|
-| 2026-07-08 | Koby: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added config section. Key rotated and stored in vault. Added Failure Mode #11 to zulip-adapter-lessons. |
 | 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
 | 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
@@ -5,40 +5,34 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (ornith-1.0-35b).
-  RTX 3090 context verified at 256K (was incorrectly documented as 128K).
-  RTX 5070 stays at 131K for vision/web. Infisical .env fallback required (Rule 3).
-  Parallel counts corrected (RTX 3090=1, RTX 5070=2, Strix=2).
+  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
 ---

 ## Maintains

 - template_version: "2.1.0"
 - last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
+- agents_configured: ["tanko", "mumuni", "abiba", "tdunna", "baggy", "kagenz0"]
 - agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array

-## Agent Keys (LiteLLM — Current 2026-07-11)
+## Agent Keys (LiteLLM — Current 2026-07-04)

 Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM
-PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB.
-The env var `LITELLM_API_KEY` is injected at runtime via `infisical run --` wrapper
-(project=agents, env=production). /etc/environment and ~/.hermes/.env are NO LONGER
-used for agent keys — stripped and tagged `# [INFISICAL]` post-migration.
+PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB, not in
+config files. The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent
+host AND in `~/.hermes/.env` for gateway env propagation.
 Sub-agent profiles inherit auth from the main config — no separate keys needed.

 | Agent | Key Alias | Host | SSH | Sub-Agents |
 |-------|-----------|------|-----|-----------|
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
 | Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
-| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
-| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
-| Koonimo | `koonimo` | CT 113 (baggy) | Zulip | — |
+| Abiba | `abiba-*` | 192.168.68.24 | local | — |
+| Tdunna | `tdunna-*` | ? | Zulip | — |
+| Baggy | `baggy-*` | ? | Zulip | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |

-> CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
-
 ✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
  syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`

@@ -56,12 +50,11 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 ## API Key Rules

 - `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred)
-  Key is injected at runtime via `infisical run --` wrapper — never in /etc/environment
 - `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider
 - `api_key: sk-...` — Hardcoded key only as fallback when env var not possible
- Store `LITELLM_API_KEY` in Infisical vault (project=agents, env=production)
+- Set `LITELLM_API_KEY` in `/etc/environment` on each host
 - Sub-agents NEVER get their own key — they share the host agent's key
- Restart Hermes gateway after updating vault secret (key auto-injected via wrapper)
+- Restart Hermes after updating `/etc/environment`

 ### Sub-Agent Profiles (Mumuni pattern)

@@ -85,7 +78,7 @@ Sub-agent profile rules:
 5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty
 6. **Never hardcode a key** in sub-agent profiles

-This ensures all 6 sub-agents use the same LiteLLM key injected via `infisical run --` wrapper.
+This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`.
 When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
 work immediately after restart.

@@ -97,10 +90,9 @@ model:
  default: <agent_model>          # e.g., ornith-1.0-35b, qwen3.6-27B-code
  provider: harness
  base_url: http://192.168.68.116/v1
-  api_key_env: LITELLM_API_KEY    # Injected via infisical run -- wrapper
+  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
-  context_length: 262144          # For syslog-auto (all GPUs support 256K).
-                                  # Set 131072 if using gemma-4-12b directly (12GB VRAM constraint).
+  context_length: 262144          # Must match model's max context window

 fallback_providers:
  provider: deepseek
@@ -129,11 +121,10 @@ mcp_servers:
 # ─── Compression ───
 compression:
  enabled: true
-  model: ornith-1.0-35b           # ⚠️ Must match auxiliary.compression.model
+  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
  provider: harness
-  max_context_window: 262144      # For Strix Halo compression (256K ctx).
-                                  # Set 131072 if using gemma directly.
-  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
+  max_context_window: 262144      # Must match model's actual capacity
+  threshold: 0.65                 # Fires at ~170K for 262K window (not 0.25!)
  target_ratio: 0.30
  protect_last_n: 40
  hygiene_hard_message_limit: 350
@@ -164,7 +155,7 @@ auxiliary:
    timeout: 30
  compression:
    provider: harness
-    model: ornith-1.0-35b
+    model: gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    timeout: 60
@@ -182,7 +173,7 @@ custom_providers:

 When LiteLLM keys are regenerated (e.g., after infrastructure changes):

-1. **If SSH available**: Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-<NEW> --project=agents --env=production`, then `ssh <host> "systemctl restart hermes-gateway"`
+1. **If SSH available**: `ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"`
 2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command
 3. **After update**: Restart Hermes on the agent host
 4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
@@ -204,14 +195,7 @@ The following MUST be identical across ALL profiles:
 ### Rule 3: API Keys via Environment
 - Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
 - Hardcoded keys in config.yaml become stale after key rotation
- Infisical vault secrets persist across config updates / reinstalls
- **NEW (July 2026): Always keep a local `.env` fallback.** Infisical service tokens
-  can expire/404 (tanko incident: token not found, gateway ran without key for hours).
-  The `.env` file should have the key uncommented as a fallback:
-  ```
-  LITELLM_API_KEY=sk-...
-  # [INFISICAL] Also sourced from vault.sysloggh.net
-  ```
+- `/etc/environment` persists across config updates
 - Restart Hermes after env var updates

 ### Rule 4: Sub-Agent Profiles Inherit Auth
@@ -221,7 +205,7 @@ The following MUST be identical across ALL profiles:
  - Auxiliary tasks: `api_key: ''`, `provider: harness`
 - Never hardcode a key in sub-agent profiles
 - When main config uses `api_key_env`, sub-agents automatically use it
- This means key rotation only touches ONE vault secret (`LITELLM_API_KEY`)
+- This means key rotation only touches ONE file (`/etc/environment`)

 ### Rule 5: Main Config Base URL
 |- Use direct IP: `http://192.168.68.116/v1`
@@ -236,63 +220,22 @@ The following MUST be identical across ALL profiles:
 - Apply to BOTH main config AND all sub-agent profiles
 - For agents needing longer outputs: raise to 8192, but never omit

-### Rule 7: Auxiliary Model Consistency (UPDATED July 2026)
- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
- Compression uses `ornith-1.0-35b` (Strix Halo — 64GB, 256K ctx, compression-optimized)
- All auxiliary services MUST use identical routing:
+### Rule 7: Auxiliary Model Consistency
+- All auxiliary services (vision, web_extract, compression) MUST use the same model:
+  - `model: gemma-4-12b`
  - `base_url: http://192.168.68.116/v1`
  - `api_key_env: LITELLM_API_KEY`
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
- **Compression moved to Strix Halo (July 2026)**: The ornith-1.0-35b model on Strix Halo
-  (64GB UMA, 256K context, 72.4 tok/s) is the designated compression GPU. This frees the
-  RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
- The `compression:` block's `model` MUST match `auxiliary: compression: model`
- The `compression: max_context_window: 262144` MUST match Strix Halo's actual capacity
+- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes to the primary 35B reasoning GPU
+- gemma-4-12b is a lightweight 12B model on the RTX 5070, keeping the Strix Halo free for reasoning
+- The `compression:` block's `model` MUST match `auxiliary: compression: model` — they are two different configs for the same service

-### Rule 8: GPU Workload Distribution (July 2026)
- **RTX 3090 (24GB, 256K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
- **RTX 5070 (12GB, 131K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract
- **Strix Halo (64GB, 256K ctx, ornith-1.0-35b)**: Context compression, summarization, long docs
- Agent profiles MUST route auxiliary tasks to the correct GPU:
-  - `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
-  - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
-  - `auxiliary.compression.model: ornith-1.0-35b` (Strix Halo)
- Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
+### Rule 8: Compression Threshold for 256K Models
 - For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
 - Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
 - `max_context_window: 262144` MUST match the model's actual capacity
 - See `devops-hermes-compression` skill for full reference

-### Rule 9: Compression Threshold for 256K Models
- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
- `max_context_window: 262144` MUST match the model's actual capacity (Strix Halo = 256K)
- See `devops-hermes-compression` skill for full reference
-
-### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
- `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b
-  and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
-  - Model name typos that cause 403 errors and silent worker failures
-  - Single GPU downtime (routing falls back automatically)
-  - Key/model authorization mismatches
- **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
-  for specialized tasks, but MUST validate those models exist in the key's authorized list
-
-### Rule 11: Validate Model IDs Before Deployment (pi Agents)
- After configuring a pi agent's `models.json`, verify every model ID:
-  ```bash
-  curl -s http://192.168.68.116:4000/v1/models \
-    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
-  ```
- All model IDs in `models.json` MUST appear in the LiteLLM response
- The agent's API key may have a SUBSET of the full model catalog — check per-key
- A non-existent model ID causes 403 errors that silently break the pi RPC worker
-  (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
-
 ## Execution

 1. **Check current config** — Read the target agent's config.yaml
@@ -5,10 +5,8 @@ version: 1.0.0
 description: >
  Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
  providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
-  Anthropic) may use hardcoded keys. Single source of truth: Infisical vault
-  (project=agents, env=production) — injected at runtime via `infisical run --` wrapper.
-  /etc/environment is DEPRECATED for agent keys post-migration. Designed to make key
-  rotation a one-step vault operation.
+  Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each
+  agent host. Designed to make key rotation a one-step operation.
 author: Abiba (pi agent)
 ---

@@ -16,7 +14,7 @@ author: Abiba (pi agent)

 ## Rule (One Sentence)

-**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
+**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.**

 ## Scope

@@ -28,39 +26,6 @@ Applies to all Hermes agent configs across all hosts. Covers these config sectio
 - `compression.api_key` (when `provider` is `harness` or contains `litellm`)
 - `fallback_providers[].api_key` (when provider is harness)

-## Architecture (2026-07-10)
-
-Syslog is migrating away from **unauthenticated direct access** to the shared inference harness.
-
-| Path | Auth | Status |
-|------|------|--------|
-| `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
-| `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
-
-All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
-Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
-
-### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
-
-When `api_mode: responses` is set, Hermes **appends `/v1/responses`** to `base_url`.
-If `base_url` already includes `/litellm/v1/responses`, the result is:
-
-```
-http://192.168.68.116/litellm/v1/responses/v1/responses → 404
-```
-
-**The `base_url` must end at `/v1` — never include `/responses`:**
-
-```yaml
-# ✅ CORRECT — Hermes appends /v1/responses for api_mode: responses
-base_url: http://192.168.68.116/litellm/v1
-
-# ❌ WRONG — produces double path
-base_url: http://192.168.68.116/litellm/v1/responses
-```
-
-This applies to ALL sections using the harness provider: `custom_providers`, `delegation`, `auxiliary.*`.
-
 ## Exemptions

 External providers are **explicitly exempt** and may use hardcoded keys:
@@ -73,42 +38,35 @@ External providers are **explicitly exempt** and may use hardcoded keys:
 ## Standard Pattern

 ```yaml
-# ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
+# ✅ CORRECT — all harness/litellm providers
 model:
-  provider: harness
-  base_url: http://192.168.68.116/litellm/v1           # ← Hermes appends /v1/responses
-  api_key_env: LITELLM_API_KEY
+  provider: harness                    # or custom:litellm.sysloggh.net
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY        # ← indirection

 custom_providers:
  - name: harness
-    api_mode: responses
-    base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
-    api_key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY      # ← indirection

 auxiliary:
  compression:
    provider: harness
-    base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
-    api_key_env: LITELLM_API_KEY
+    api_key_env: LITELLM_API_KEY      # ← indirection

 # ✅ ALSO CORRECT — external providers
 fallback_providers:
  - provider: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-b7d9...               # ← hardcoded OK (external)
-    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
+    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in /etc/environment
 ```

 ```yaml
-# ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
+# ❌ FORBIDDEN — hardcoded harness/litellm key
 model:
  provider: harness
-  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
-
-model:
-  provider: harness
-  base_url: http://192.168.68.116/v1   # ← RULE VIOLATION: unauthenticated path
-  api_key_env: LITELLM_API_KEY
+  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION
 ```

 ## Detection Query
@@ -117,18 +75,13 @@ Run on any Hermes host to detect violations:

 ```bash
 # 1. Check config.yaml for hardcoded harness keys
-grep -rn 'api_key: sk-' /root/.hermes/ \
+grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
  --include='config.yaml' \
  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'

-# 1b. Check for double-path bug: base_url ending with /responses
-# (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
-grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
-# ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
-
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
-grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null

 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
@@ -139,32 +92,26 @@ If any output from step 2 — **critical violation** (master key leaked). Fix im

 ## Rotation Procedure

-With this standard enforced, key rotation is one vault update:
+With this standard enforced, key rotation is one step:

 ```bash
-# 1. Generate new key in LiteLLM: POST /key/generate with agent alias
-# 2. Update Infisical vault secret
-infisical secrets set LITELLM_API_KEY=sk-NEW_KEY \
-  --project=agents --env=production
-# 3. Restart agent gateway (key auto-injected via infisical run -- wrapper)
-ssh root@<host> "systemctl restart hermes-gateway"
+# 1. Generate new key in LiteLLM
+# 2. Update /etc/environment on agent host
+ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment"
+# 3. Restart agent gateway
+ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &"
 # 4. Verify
-curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/litellm/v1/models
+curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
 ```

-**Done.** No config file changes needed. No /etc/environment edits needed.
-The agent picks up the new key via `infisical run --` at gateway startup.
-
-> **Post-migration note**: /etc/environment is NO LONGER the key source.
-> Strip all `LITELLM_API_KEY` lines from /etc/environment (comment out with `# [INFISICAL]`)
-> and let the `infisical run --` wrapper inject the key at runtime.
+**Done.** No config file changes needed. The agent picks up the new key on restart.

 ## Key Longevity Policy (2026-07-04)

 **Keys are permanent and use bare agent name aliases.**

 - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
 - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
 - **Max budget**: $100 per key (config default).

@@ -181,54 +128,36 @@ litellm_settings:

 ## Verified Agents (2026-07-05 update)

-| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
-|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
-| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
-| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
-| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
-| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
-| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
+|-------|-----|-----|---------------|-----|--------|----------------|---------------|
+| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
+| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
+| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
+| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
+| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
 | Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |

-> **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
-> LiteLLM key aliases use agent identity, not CT hostname.
+### Systemd Service Pattern (2026-07-04 fix)

-### Migration Status: Authenticated Path
+All Hermes agents use systemd to manage their gateway. Two issues were fixed:

-| Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
-|-------|--------------------------|--------------------|--------|
-| Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
-| Tanko | ⚠️ No SSH access | — | Needs check |
-| Koby | ⚠️ No route to host | — | Needs check |
-| Koonimo | ⚠️ Connection timed out | — | Needs check |
+1. **Drop-in override** — `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` (or user equivalent) had hardcoded `LITELLM_API_KEY` that bypassed `/etc/environment`.
+2. **Missing EnvironmentFile** — Services did not source `/etc/environment`.

-### Systemd Service Pattern (2026-07-11 — vault migration)
-
-All Hermes agents use systemd to manage their gateway. The gateway service is wrapped
-with `infisical run --` to inject secrets at runtime.
-
-**Correct pattern (post-migration):**
+**Correct pattern:**
 ```ini
-# Service file wraps gateway with Infisical:
-[Service]
-ExecStart=/usr/bin/infisical run --project=agents --env=production -- \
-  /usr/bin/hermes gateway run
-
-# /etc/environment is CLEAN — no LITELLM_API_KEY present
-# (strip it and tag with # [INFISICAL] if present)
-```
-
-**Legacy pattern (deprecated — pre-migration only):**
-```ini
-# DO NOT USE post-migration:
+# In service file:
 EnvironmentFile=/etc/environment
-# This pattern was replaced by infisical run -- wrapper
+
+# Drop-in only for overrides, NOT primary key storage.
+# If a drop-in exists, it must match /etc/environment.
 ```

-**Rotation procedure** (one vault operation with this standard):
+**Rotation procedure** (one step with this standard):
 1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
-2. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-NEW --project=agents --env=production`
-3. Restart: `systemctl restart hermes-gateway` (key auto-injected via wrapper)
+2. Update `/etc/environment`: `sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment`
+3. Update drop-in (if exists): same sed on `litellm-key.conf`
+4. Restart: `systemctl [--user] restart hermes-gateway`

 ## Violation Response

@@ -238,7 +167,7 @@ EnvironmentFile=/etc/environment
 4. **Restart** — gateway must restart to pick up env var
 5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
 6. **Update** — bump the verified table above
-7. **Use safe-mutate** — if the fix requires updating vault secrets or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
+7. **Use safe-mutate** — if the fix requires changing `/etc/environment` or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.

 ## Related Contracts

@@ -277,13 +206,13 @@ task config:
 ```yaml
 auxiliary:
  vision:
-    api_key: sk-<agent-key-from-vault>           # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
+    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
  compression:
-    api_key: sk-<agent-key-from-vault>           # ← workaround (same as above)
+    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
@@ -1,238 +0,0 @@
---
-kind: function
-name: hermes-zulip-plugin
-description: >
-  Installs or updates the Zulip platform plugin for any Hermes agent from the
-  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
-  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
-  recovery). Verifies the installation, restarts the gateway, and sends a relay
-  success signal.
-agent: abiba
-version: 1.0.0
-status: active
-runtime_contract: 2
---
-
-# Hermes Zulip Plugin — Install & Repair
-
-Single-shot function that pulls the latest zulip-platform plugin from the
-canonical git repo, installs it to the correct Hermes bundled plugin path,
-verifies the installation, and signals completion.
-
-Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
-layer and a targeted gateway restart. It does NOT verify env credentials or
-run a live Zulip connection test. Use `hermes-zulip-restore` for full
-connectivity recovery including end-to-end DM validation.
-
-## Parameters
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
-| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
-
-## Maintains
-
- plugin_installed: bool — Whether all three adapter files exist at the bundled path
- plugin_version: string — Git commit SHA of the installed version
- strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
- signal_sent: bool — Whether relay success message was dispatched
-
-### Postconditions
-
- All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
- `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
- Installed version matches HEAD of the requested branch
- Relay success signal sent to Hermes agent's inbox
-
-## Requires
-
- SSH access to target host (direct or via amdpve for CTs)
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
- Python 3 with `httpx` installed on target
-
-## Live-State Fields
-
-| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
-|------|-----|---------|-------------|-------------|------|
-| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
-| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
-| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
-| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
-
-| Field | Value | Trust |
-|-------|-------|-------|
-| Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
-| Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
-| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
-| Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
-| Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
-
-## Execution
-
-### Step 1: Resolve Target
-
-Map `target` to host, CT ID, hermes_home, and user from the live-state table.
-For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
-
-### Step 2: Pull Latest Plugin Source
-
-On the target host:
-
-```bash
-# Ensure deploy scratch space
-mkdir -p /tmp/zulip-deploy
-cd /tmp/zulip-deploy
-
-# Clone or pull
-if [ -d zulip-platform-plugins ]; then
-  cd zulip-platform-plugins
-  git fetch origin
-  git checkout {{branch}}
-  git pull origin {{branch}}
-else
-  git clone --branch {{branch}} \
-    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
-  cd zulip-platform-plugins
-fi
-
-# Capture installed version
-INSTALLED_SHA=$(git rev-parse HEAD)
-echo "Installed SHA: $INSTALLED_SHA"
-
-# Verify we're at HEAD
-HEAD_SHA=$(git rev-parse origin/{{branch}})
-if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
-  echo "At latest commit on {{branch}}"
-else
-  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
-fi
-```
-
-### Step 3: Install Plugin Files
-
-```bash
-# Ensure target directory exists
-mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
-
-# Copy adapter files
-cp plugins/platforms/zulip/adapter.py \
-   plugins/platforms/zulip/__init__.py \
-   plugins/platforms/zulip/plugin.yaml \
-   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
-
-# Fix ownership (Tanko only — runs as jerome user)
-[ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
-  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
-
-echo "Plugin files installed"
-```
-
-### Step 4: Verify Installation
-
-```bash
-# Check all three files exist
-for f in __init__.py adapter.py plugin.yaml; do
-  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
-    echo "✅ $f present"
-  else
-    echo "❌ $f MISSING"
-    exit 1
-  fi
-done
-
-# Verify _strip_html fix
-grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
-  && echo "✅ _strip_html fix present" \
-  || echo "❌ _strip_html MISSING — plugin may be stale"
-
-# Show installed plugin.yaml version
-grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
-```
-
-### Step 5: Restart Gateway
-
-Plugin changes require a gateway restart to take effect:
-
-```bash
-cd {{hermes_home}}/hermes-agent
-# Use venv if available
-python3 -m hermes_cli.main gateway restart 2>&1 || \
-  venv/bin/python -m hermes_cli.main gateway restart 2>&1
-```
-
-Wait for restart to complete (up to 45s), then confirm:
-
-```bash
-grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
-```
-
-Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
-
-Quick smoke check — confirm zulip platform loaded:
-
-```bash
-grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
-```
-
-If gateway fails to restart, check logs for the crash cause before proceeding.
-
-### Step 6: Send Relay Success Signal
-
-On the Abiba host (local), dispatch a relay message to the target agent:
-
-```
-ra-h-os-createRelayNode:
-  title: "Zulip plugin updated — {{target}}"
-  source: |
-    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
-    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
-    _strip_html fix: PRESENT
-    Timestamp: {{timestamp}}
-
-  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
-```
-
-### Step 7: Report
-
-Compile results into a single status block:
-
-| Field | Value |
-|-------|-------|
-| Target | `{{target}}` |
-| Branch | `{{branch}}` |
-| Commit SHA | `{{INSTALLED_SHA}}` |
-| Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
-| `_strip_html` | `{{present|missing}}` |
-| Gateway restarted | `{{yes|no}}` |
-| Signal sent | `{{yes|no}}` |
-
-## Known Failure Modes
-
-| Symptom | Root Cause | Recovery |
-|---------|-----------|----------|
-| Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
-| Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
-| `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
-| Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
-| Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
-
-## Edge Differences from hermes-zulip-restore
-
-| Concern | hermes-zulip-restore | hermes-zulip-plugin |
-|---------|---------------------|---------------------|
-| Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
-| Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
-| Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
-| Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
-| Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
-| Signal dispatch | ❌ None | ✅ Relay message to target |
-
-For full connectivity recovery after a plugin install, chain this contract
-with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
-
---
-
-**Last updated**: 2026-07-08 — Switched default branch to `master`; added
-gateway restart step. If outstanding unmerged feature branches exist, address
-them in a follow-up merge after this contract completes.
@@ -1,189 +0,0 @@
---
-kind: function
-name: hermes-zulip-restore
-description: >
-  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
-  Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
-  path, verifies env credentials, restarts the gateway, and confirms Zulip
-  connects. Run this whenever a Hermes agent stops responding on Zulip or after
-  a fresh agent deployment.
-agent: abiba
-version: 1.0.0
-status: active
-runtime_contract: 2
---
-
-# Hermes Zulip Restore — Bring Any Agent Back to Good State
-
-Single-shot function that restores full Zulip connectivity for a Hermes agent.
-Covers adapter deployment, HTML stripping (slash command fix), env verification,
-gateway restart, and connection validation.
-
-## Parameters
-
-| Param | Type | Required | Default | Description |
-|-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
-
-## Maintains
-
- adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
- zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
- env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
- gateway_running: bool — Whether gateway process is running
-
-### Postconditions
-
- `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
- All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
- Gateway restarted and zulip platform reports state `connected`
- HTML stripping enabled for `/approve` and `/deny` slash command support
-
-## Requires
-
- SSH access to target host (direct or via amdpve for CTs)
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
- Python 3 with `httpx` installed on target
- Zulip server accessible at `https://chat.sysloggh.net`
-
-## Live-State Fields
-
-| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
-|------|-----|---------|-------------|-------------|------|
-| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
-| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
-| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
-| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
-
-| Field | Value | Trust |
-|-------|-------|-------|
-| Zulip server | https://chat.sysloggh.net | ✅ Verified |
-| Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
-| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
-| Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
-
-## Execution
-
-### Step 1: Locate Target
-
-Map `target` to connectivity parameters from the live-state table above.
-For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
-
-### Step 2: Deploy Zulip Adapter
-
-On the target host:
-
-```bash
-# Clone or update the plugin repo
-mkdir -p /tmp/zulip-deploy
-cd /tmp/zulip-deploy
-if [ -d zulip-platform-plugins ]; then
-  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
-else
-  git clone --branch feat/zulip-streaming \
-    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
-fi
-
-# Ensure bundled plugin directory exists
-mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
-
-# Copy adapter files
-cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
-   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
-   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
-   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
-
-# Fix ownership (Tanko only)
-chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
-
-# Clean up
-rm -rf /tmp/zulip-deploy
-```
-
-### Step 3: Verify _strip_html is Present
-
-```bash
-grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
-```
-Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
-
-### Step 4: Verify Env Credentials
-
-```bash
-grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
-```
-
-Expected: all three variables set with non-empty values. If any missing:
- ZULIP_SITE: `https://chat.sysloggh.net`
- ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
- ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
-
-### Step 5: Restart Gateway
-
-```bash
-cd <HERMES_HOME>/hermes-agent
-# Use venv if available
-python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
-```
-
-Wait for the restart to complete (up to 45s). Check:
-
-```bash
-grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
-```
-
-Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
-
-### Step 6: Validate Zulip Connection
-
-```bash
-python3 -c "
-import json
-d = json.load(open('$HERMES_HOME/gateway_state.json'))
-print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
-"
-```
-
-Expected: `zulip: connected`. If not connected, check gateway log:
-
-```bash
-grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
-```
-
-### Step 7: Report
-
-Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
-
-| State | Action |
-|-------|--------|
-| All true | ✅ Agent restored — relay success to user |
-| `adapter_deployed: false` | Re-run Step 2 |
-| `env_valid: false` | Prompt for missing credentials |
-| `zulip_connected: false` | Check Zulip server reachability, verify API key |
-| `gateway_running: false` | Check process logs for crash cause |
-
-## Known Failure Modes
-
-| Symptom | Root Cause | Recovery |
-|---------|-----------|----------|
-| Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
-| Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
-| No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
-| `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
-| Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
-| Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
-
-## Git Branch Reference
-
-The `_strip_html` fix lives on `feat/zulip-streaming` branch:
-```
-https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
-```
-
-Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
-Pull request #33 is the primary integration branch.
-
---
-
-**Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
@@ -62,19 +62,19 @@ description: >

 | Resource | Auth Method | Credential Source | Status |
 |----------|------------|-------------------|--------|
-| Proxmox Cluster | PVE API Token | Infisical vault (`PROXMOX_API_TOKEN`) | ✅ |
-| Proxmox Root | Password via API ticket | Infisical vault (`PROXMOX_ROOT_PASSWORD`) | ✅ |
+| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ |
+| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
 | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
 | Baggy CT (113) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
-| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
-| Zulip | Bot API key | Infisical vault (`ZULIP_BOT_KEY`) | ✅ |
+| Gitea | API token | abiba-bot token | ✅ |
+| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
-| LiteLLM Admin | master key | Infisical vault (`LITELLM_MASTER_KEY`) | ✅ VERIFY-BEFORE-USE |
-| Grafana | admin password | Infisical vault (`GRAFANA_ADMIN_PASSWORD`) | ✅ VERIFY-BEFORE-USE |
+| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ VERIFY-BEFORE-USE |
+| Grafana | admin password | `syslog-grafana-2026` | ✅ VERIFY-BEFORE-USE |

 > **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
 > contract are live-state fields. Test them against the live system before
@@ -144,7 +144,7 @@ description: >

 ### Ecosystem A: docker-vm (192.168.68.7)

-16 containers across 4 compose stacks + trove agents:
+11 containers across 4 compose stacks:

 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,9 +152,6 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
-| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
-
-**Last verified:** 2026-07-09 — 16/16 containers running healthy.

 ### Home Stack Services

@@ -550,13 +547,13 @@ curl -s http://192.168.68.116/health/unified | jq .status
 curl -s http://192.168.68.24:9100/gpu-data | jq .summary

 # Grafana status
-curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health
+curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health

 # Prometheus targets
 curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

 # LiteLLM key check
-curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
+curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'

 # Storage check
@@ -586,7 +583,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 108 | media | storepve | — | Media | ❌ |
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | — | Git | ❌ |
-| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
+| 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
 | 113 | baggy | amdpve | ? | Hermes agent | ✅ |
 | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
@@ -7,14 +7,10 @@ description: >
  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
  via existing /metrics Prometheus endpoint.
  
-  DEPLOYMENT STATUS (2026-07-09):
-  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
-     NVIDIA sidecar exporters (.8/.110:9400) never installed.
-     Router falls back to direct GPU /health probes.
-  ⚠️ This contract is target-state aspirational — not as-built.
-     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
+  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+  This contract defines the target state; proxmox-monitor is the as-built.
 version: 1.0.0
 ---

@@ -3,7 +3,7 @@ kind: responsibility
 name: infrastructure-update
 description: >
  Autonomous system-wide update contract covering all 5 Proxmox nodes,
-  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
+  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
 agent: abiba
@@ -11,7 +11,7 @@ triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
-version: 1.2.0
+version: 1.0.0
 ---

 ## Maintains
@@ -24,15 +24,13 @@ version: 1.2.0

 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
-4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
-5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
-6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
-7. ✅ Zulip server reachable
-8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
-9. ✅ Disk >20% free on all nodes
-10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
+2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
+3. ✅ Docker healthy on .7, .116
+4. ✅ LiteLLM health check passing
+5. ✅ Zulip server reachable
+6. ✅ GPU fleet healthy (all 3 GPUs)
+7. ✅ Disk >20% free on all nodes
+8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

 ## Wave 1: Storage & Infra Nodes (lowest impact)

@@ -40,7 +38,7 @@ Before ANY update wave:
 |--------|------|---------|---------|
 | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
-| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
+| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
 | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
 | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |

@@ -57,17 +55,16 @@ Before ANY update wave:
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
-| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
-| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
-| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
+| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
+| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |

 **Verify after Wave 2:**
- All VMs/CTs running: check via Proxmox API
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
- LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools
- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- All CTs running: check via Proxmox API
+- LiteLLM healthy: `curl :4001/health/liveliness`
+- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`

@@ -75,21 +72,20 @@ Before ANY update wave:

 | Host | Stack | Command |
 |------|-------|---------|
-| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
-| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
 | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |

 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
- MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
+- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
 - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116)
- Firecrawl test: `curl :3002/`
- SearXNG test: `curl :8888`
+- Dashboard loading: `curl :3001/`
+- Firecrawl test: `curl :3002/health`
+- SearXNG test: `curl storepve:8888`

 ## Wave 4: Proxmox Kernel Reboot (if needed)

@@ -115,62 +111,16 @@ If ANY verification fails:

 Before Wave 1, snapshot these files:
 ```
-/opt/inference-harness/docker-compose.yml               (CT 116 .116) ⚡ contains MCP_SERVER env vars
-/opt/inference-harness/litellm_config.yaml              (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
-/opt/monitoring/prometheus.yml                          (CT 116 .116)
-/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
-/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
-/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
-/opt/home_stack/docker-compose.yml                      (VM 109 .7)
-/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
-/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
-/etc/systemd/system/ornith-server.service               (amdpve .15)
-/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
-# Hermes agent configs (key enforcement — 2026-07-10)
-/root/.hermes/config.yaml                               (Mumuni CT 114, Tanko CT 112, etc.)
-/root/.config/systemd/user/hermes-gateway.service       (Mumuni CT 114 — EnvironmentFile fixed)
-/etc/environment                                        (Mumuni CT 114 — LITELLM_API_KEY)
+/opt/inference-harness/litellm_config.yaml        (.116)
+/opt/inference-harness/docker-compose.yml          (.116)
+/etc/nginx/nginx.conf                              (.116 container)
+/opt/search-stack/*/docker-compose.yaml            (.7)
+/root/.pi/agent/extensions/config.yaml             (.24)
+/etc/systemd/system/ornith-server.service          (.15)
+/etc/systemd/system/llama-server.service           (.8, .110)
 ```

-## MCP Gateway (2026-07-10)
-
-LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.
-
-### Configuration
-
-**litellm_config.yaml** (`/opt/inference-harness/litellm_config.yaml`):
-```yaml
-mcp_servers:
-  ra_h_os:
-    url: "http://192.168.68.65:3100/mcp"
-    transport: "http"
-    auth_type: "none"
-```
-
-**docker-compose.yml** env vars:
-```yaml
- MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
- MCP_SERVER_RAHOS_TRANSPORT=http
-```
-
-### Access
-
-| Key | MCP Access |
-|-----|-----------|
-| Master key | ✅ Full — 90 tools (vault-injected) |
-| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |
-
-### Known Limitations
- Per-key MCP server grants not functional — only master key has access
- Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
- HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
- `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
-
-### Migration Path
-When LiteLLM is upgraded to a version supporting per-key MCP grants:
-1. Grant agent keys `mcp_servers: ["ra_h_os"]`
-2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
-3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`

 ## Security-Specific Updates

@@ -184,12 +134,11 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants:
 ## Success Criteria

 - [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
+- [ ] All CTs running post-update
+- [ ] All 22 Docker containers healthy
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
- [ ] LiteLLM MCP gateway healthy (90 tools via master key)
 - [ ] Zero security CVEs remaining
 - [ ] <10 min total downtime per service

@@ -199,7 +148,7 @@ After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD

-Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
+Updated: 5 PVE nodes, 12 CTs, 22 containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>
@@ -8,15 +8,6 @@ description: >
  Ensures agents never use the master key directly. Rotation is event-driven,
  not calendar-driven — rotate only on compromise, personnel change, or
  periodic security hygiene (quarterly/annually).
-  
-  UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
-  BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
-  expire/404. The .env fallback prevents agents from running without keys.
-  Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
-  
-  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
-  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
-  on CT 116. Last verified: 2026-07-12.
 ---

 ## Parameters
@@ -24,10 +15,7 @@ description: >
 - agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
 - action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
 - litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
- master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
- vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
- vault_project: string — Infisical project slug (default: "infrastructure")
- vault_env: string — Infisical environment (default: "production")
+- master_key: string — LiteLLM master key (default from environment)
 - agent_host: string — Agent's IP for SSH (default: resolved from infra)
 - agent_user: string — SSH user (default: "jerome")

@@ -38,32 +26,27 @@ description: >
 - key_prefix: string — First 10 chars of the new key (for identification)
 - previous_key_alias: string | null — Previous key alias if rotating
 - litellm_response: object — Raw response from LiteLLM /key/generate
- vault_updated: boolean — Whether Infisical vault secret was updated
- agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
+- agent_config_updated: boolean — Whether /etc/environment was updated
 - verification: { status: string, detail: string } — Final health check

 ## Execution

-1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
+1. **Authenticate** — Verify master_key works against LiteLLM /key/list
 2. **Check existing keys** — List all keys, find any with agent_name alias
 3. **If action == "list"**: Return all keys with their aliases and spend
 4. **If action == "create"**:
   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
-   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
+   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
   - Return the new key
 5. **If action == "rotate"**:
   - Generate new key with same alias (LiteLLM replaces the old key)
-   - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
-   - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
-     The gateway automatically picks up the new key via `infisical run --` wrapper
+   - SSH to agent_host, update /etc/environment LITELLM_API_KEY
+   - Restart agent gateway (hermes gateway restart for Hermes agents)
   - Verify: curl test against /v1/models with new key
   - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
-   - Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
 6. **If action == "verify"**:
-   - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
+   - SSH to agent, read /etc/environment
   - Test the key against LiteLLM /v1/models
   - Confirm key alias matches agent_name in LiteLLM key list
-   - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
@@ -1,50 +1,35 @@
 ---
 kind: function
 name: litellm-health
-status: deprecated
-deprecated_on: 2026-07-09
-replaced_by: litellm-self-heal.prose.md
-note: >
-  Consolidated into litellm-self-heal.prose.md to eliminate duplication
-  of architecture diagrams, GPU topology, timeout tables, and container
-  lists. Health check is now § Health Check within litellm-self-heal.
-  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
-  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
-  Router (harness-router :9000) is DEPRECATED — container still runs but
-  is not in the request path. GPU monitoring via Prometheus/Grafana and
-  fleet dashboard (gpu-monitor :9100).
+  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Prometheus/Grafana and fleet dashboard.
  Designed as a reusable contract for any Syslog agent.
-  
-  Source of truth: gpu-fleet.prose.md
 ---

-## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)

 ```
-Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
-                        │
-                  Key validation
-                  Fallback chains
-                  Budget tracking
-                        │
-                  Prometheus ← metrics
+Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+                        │                    │
+                  Key validation      Model routing
+                  Fallback chains     Slot booking
+                  Complexity router   Circuit breakers
+                  Budget tracking     Redis-backed
+                        │                    │
+                  Prometheus ← metrics  ←───┘
                        │
                    Grafana :3001
-
-  harness-router :9000 — DEPRECATED, container still runs but
-  NOT in request path. nginx routes /v1 → LiteLLM directly.
-  Router slot booking + circuit breakers replaced by
-  LiteLLM native fallbacks + timeouts.
 ```

-**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
- Router REMOVED from request path — LiteLLM proxies directly to GPU
- All GPUs at parallel 2 (was parallel 1)
- NVIDIA context reduced 256K→128K to free VRAM
- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
+**What changed (v3.1.0 → v3.2.0)**:
+- Router is BACK — runs internally on :9000 behind LiteLLM
+- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- Router :9000 is 127.0.0.1-only, not publicly accessible
+- GPU fleet dashboard: http://192.168.68.24:9100
+- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)

 ## Parameters

@@ -70,29 +55,27 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)

 ## GPU Fleet Topology

-| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
-|------|-----|----------|---------------|--------|---------|----------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
-| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
+| Host | IP | Hardware | Models Served | Engine |
+|------|-----|----------|---------------|--------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |

 ## Model Fallback Chains (LiteLLM)

 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
-| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
-| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
+| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | 90s | qwen → gemma | — |
-
-> Global: request_timeout=300s, nginx proxy_read_timeout=600s
+| syslog-auto (balanced) | — | qwen → gemma | — |

 ## Containers on CT 116

 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
@@ -1,99 +1,27 @@
 ---
 kind: responsibility
 name: litellm-self-heal
-status: deployed
+status: manual-only
 note: >
-  DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
-  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
-  now reimplemented as `litellm-health-check.sh` on CT 116.
-  Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
-  GPU monitoring integrated from gpu-monitor on .24:9100.
-  
-  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
-  duplication of architecture diagrams, GPU topology, timeout tables, and container
-  lists. Health check is now § Health Check within this contract.
-  
-  Source of truth for GPU topology and keys: gpu-fleet.prose.md
-  Last verified: 2026-07-12
+  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
+  This contract is now manual-only — triggers require explicit user request.
+  Consider reimplementing as a standalone cron job or prose contract.
 description: >
-  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
-  inference, and agent keys. Applies remediation rules for common failures.
-  Reports every action via Zulip DM and RA-H OS knowledge graph.
+  Standing responsibility that monitors LiteLLM health and proactively
+  fixes common issues. Reports every action via Zulip DM and RA-H OS
+  knowledge graph for full audit trail.
 ---

-# LiteLLM Operations — Health Check + Self-Heal
-
-## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
-
-```
-Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
-                        │
-                  Key validation
-                  Fallback chains
-                  Budget tracking
-                        │
-                  Prometheus ← metrics
-                        │
-                    Grafana :3001
-
-  harness-router :9000 — DEPRECATED, container still runs but
-  NOT in request path. nginx routes /v1 → LiteLLM directly.
-  Router slot booking + circuit breakers replaced by
-  LiteLLM native fallbacks + timeouts.
-```
-
-## Parameters
-
- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host (default: "192.168.68.116")
- auth_host: string — Authentik OIDC server (default: "192.168.68.11")
- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
-
-## GPU Fleet Topology
-
-| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
-|------|-----|----------|---------------|--------|---------|----------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 1 |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 131K | 2 |
-| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
-
-## Model Fallback Chains (LiteLLM)
-
-| Primary | Timeout | Fallback | Timeout |
-|---------|---------|----------|---------|
-| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
-| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
-| ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | 90s | qwen → gemma | — |
-
-> Global: request_timeout=300s, nginx proxy_read_timeout=600s
-
-## Containers on CT 116
-
-| Container | Image | Port | Health Check |
-|-----------|-------|------|-------------|
-| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
-| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
-| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
-| harness-redis | redis:7-alpine | :6379 | PING |
-| harness-dashboard | inference-harness-dashboard | :3000 | /health |
-| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
-| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
-| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
-| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
-
 ## Maintains

 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
- litellm-agent-keys: { count: int, valid: int }
+
+## Requires
+
+- litellm-health:function

 ## Continuity

@@ -101,49 +29,55 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures

---
+## Reporting

-## Health Check
+Every remediation cycle produces a structured report in three channels:

-Run this first on every cycle. Results feed into remediation rules below.
+### 1. RA-H OS Knowledge Graph Node
+Created as `[LEARN] litellm-self-heal: <run_id>` with:
+- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
+- source: Full JSON report of the cycle
+- description: Summary of what was found, fixed, and escalated

-### 1. Check public endpoints
- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
+### 2. Zulip DM to Owner
+Sent immediately for:
+- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
+- Every 10th clean cycle — "✅ All Clear (10 checks passed)"

-### 2. Check LiteLLM health (no-auth)
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
+### 3. Daily Digest (end of day)
+A summary of the last 24 hours sent as a single Zulip DM:
+```
+📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)

-### 3. Check backend container health
- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
- Critical: harness-litellm, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
- Deprecated but running: harness-router (not in path, reference only)
+Total cycles: 288 (every 5 min)
+Issues found: 3
+  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+  └─ Escalated: 0

-### 4. Check GPU fleet health (via fleet dashboard)
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
- Verify GPUs reporting status "healthy"
- Check alerts array for active warnings/critical
+Top actions:
+  • harness-nginx restarted — 2026-06-26 02:15
+  • DOCS_URL corrected — 2026-06-26 07:30  
+  • harness-nginx restarted — 2026-06-26 14:46

-### 5. Check model inference via LiteLLM — test each model
- POST /v1/chat/completions model=gemma-4-12b → expect 200
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
- Use master key for auth
+Uptime: 23h 47m — All healthy now ✅
+```

-### 6. Check agent keys
- GET /key/list with master key → verify all 6 agents have keys
+### 4. Weekly Digest (every Monday)
+Same format as daily but covering 7 days. Sent to both Zulip DM and
+logged as a `[REPORT]` knowledge graph node for long-term trending.

-### 7. Check Grafana
- GET {{grafana_url}}/api/health → expect 200
+### 5. Relay Message (if cross-agent)
+If a fix required another agent's help (e.g., Authentik restart), a relay
+message is sent to the responsible agent with full context.

-### 8. Compile overall status
-Determine overall_status from individual check results:
- "healthy" — all checks pass
- "degraded" — 1-2 non-critical checks fail
- "down" — critical checks fail
+## GPU Fleet

---
+| Host | IP | Hardware | Models | Engine |
+|------|-----|----------|--------|--------|
+| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
+| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
+| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |

 ## Remediation Rules

@@ -171,89 +105,39 @@ Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts

-### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
-Router is no longer in the request path (2026-07-08). LiteLLM proxies
-directly to GPU. This rule is retained for reference but is inactive.
-If 503 errors occur, check LiteLLM timeouts and GPU health directly.
+### Rule 7: Router Returns 503 (All GPUs Saturated)
+Detect → router logs show QUEUE_TIMEOUT with GPUs idle
+Root cause → roster not loaded at startup (empty TIER_MODELS)
+Fix → POST /admin/roster/reload (if endpoint available) or restart router
+Permanent fix → add load_roster() to router startup code
+Escalate → if reload doesn't resolve within 2 min

 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
-Root cause → keys not in LiteLLM DB
+Root cause → keys in router env var, not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM

-### Rule 9: Stale Active Counter in Redis — DEPRECATED
-Router no longer in path so Redis active counters are unused. Rule retained
-for reference but inactive. If Redis issues occur, check harness-redis container.
-
---
-
-## Reporting
-
-Every remediation cycle produces a structured report:
-
-### 1. RA-H OS Knowledge Graph Node
-Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
-
-### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
-
-### 3. Daily Digest (end of day)
-Summary of last 24 hours: total cycles, issues found/fixed/escalated,
-top actions, uptime.
-
-### 4. Relay Message (if cross-agent)
-If a fix requires another agent (e.g., Authentik restart), relay sent
-to responsible agent with full context.
-
---
+### Rule 9: Stale Active Counter in Redis
+Detect → active:<model> > 0 while GPU idle
+Fix → `redis-cli DEL active:<model>` → restart router
+Prevent → add TTL to active keys in router code

 ## Execution

-```prose
-- Phase 1: Health Check
-let health = call health-check
-  public_url: public_url
-  backend_host: backend_host
-  gpu_dashboard_url: gpu_dashboard_url
-  grafana_url: grafana_url
-
-- Phase 2: Apply remediation for each failure
-let actions = []
-for check in health.failed:
-  let fix = apply-remediation-rule
-    rule: lookup-rule(check.name)
-    target: check.target
-  push actions fix
-
-- Phase 3: Generate report
-call report-generator
-  health: health
-  actions: actions
-
-- Phase 4: Log to knowledge graph
-call kg-logger
-  run_id: run_id
-  health: health
-  actions: actions
-
-- Phase 5: Notify if anything changed
-if actions.length > 0:
-  call zulip-notifier
-    actions: actions
-    health: health
-
-- Wait 300s and repeat
-```
+1. Run health check
+2. Apply remediation for each failure
+3. **Generate report** with all actions taken
+4. **Log to knowledge graph** — create `[LEARN]` node
+5. **Notify user** via Zulip DM if anything changed
+6. Wait 300s and repeat

 ## Audit Trail Format

 ```json
 {
-  "run_id": "self-heal-20260709-001",
-  "timestamp": "2026-07-09T14:00:00Z",
+  "run_id": "self-heal-20260626-001",
+  "timestamp": "2026-06-26T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
@@ -267,7 +151,7 @@ if actions.length > 0:
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260709-002",
+  "run_id": "self-heal-20260626-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
@@ -1,7 +1,7 @@
 ---
 name: memory-audit-maintenance
 kind: responsibility
-description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---

@@ -15,13 +15,13 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,

 ### Scope

-This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.

 **Agent Roster:**
 - Mumuni
 - Tanko
- Koby (CT 111 / tdunna)
- Koonimo (CT 113 / baggy)
+- Tdunna
+- Baggy

 **Isolation Principle:** Each agent has its own:
 - `MEMORY.md` and `USER.md`
@@ -12,7 +12,6 @@ description: >

 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
- gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp

@@ -41,7 +41,7 @@ agent: abiba

 - User: `monitoring@pve` (cluster-replicated)
 - Role: `PVEAuditor` on `/` (read-only, whole cluster)
- Token: `monitoring@pve!prometheus` — stored in Infisical vault (`PROXMOX_MONITOR_TOKEN`)
+- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
 - `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)

 ## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
@@ -62,7 +62,7 @@ agent: abiba

 - **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
 - **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
- **Credentials**: admin / password stored in Infisical vault (`GRAFANA_ADMIN_PASSWORD`)
+- **Credentials**: admin / syslog-grafana-2026
 - Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
 - grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)

@@ -19,53 +19,13 @@ from datetime import datetime

 LITELLM = "http://192.168.68.116:80"

-def _get_agent_key(agent_name):
-    """Retrieve agent key from Infisical vault."""
-    try:
-        result = subprocess.run(
-            ["infisical", "secrets", "get", "LITELLM_API_KEY",
-             "--project=agents", "--env=production", "--plain"],
-            capture_output=True, text=True, timeout=10
-        )
-        if result.returncode == 0:
-            return result.stdout.strip()
-    except Exception:
-        pass
-
-    # Fallback: try exporting all secrets
-    try:
-        result = subprocess.run(
-            ["infisical", "export", "--project=agents", "--env=production",
-             "--format=dotenv"],
-            capture_output=True, text=True, timeout=10
-        )
-        if result.returncode == 0:
-            for line in result.stdout.splitlines():
-                if line.startswith(f"LITELLM_API_KEY_{agent_name.upper()}") or \
-                   (line.startswith("LITELLM_API_KEY=") and agent_name == os.uname().nodename):
-                    return line.split("=", 1)[1].strip().strip('"').strip("'")
-    except Exception:
-        pass
-
-    return None
-
-# Agent keys are pulled from Infisical vault at runtime.
-# The 'key' field is populated dynamically below.
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome"},
-    "mumuni":  {"ct": 114, "host": "192.168.68.123", "user": "root"},
-    "koby":    {"ct": 111, "host": None,              "user": None},
-    "koonimo": {"ct": 113, "host": None,              "user": None},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"},
+    "mumuni":  {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"},
+    "tdunna":  {"ct": 111, "host": None,              "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None},
+    "baggy":   {"ct": 113, "host": None,              "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None},
 }

-# Inject keys from vault
-for agent_name in AGENTS:
-    key = _get_agent_key(agent_name)
-    if key:
-        AGENTS[agent_name]["key"] = key
-    else:
-        AGENTS[agent_name]["key"] = None
-
 GPU_HOSTS = {
    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
@@ -30,12 +30,6 @@ declare -A CT_NODES=(
    # acerpve (192.168.68.9)
    [102]=acerpve   # adguard
    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
-    #
-    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
-    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
-    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
-    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
-    #   118 jitsi     → stopped, not in service
 )

 # Each node must be root-accessible via SSH hostname
@@ -73,25 +73,6 @@ description: >
 - **Fix**: If edit fails, send the response as a new message instead
 - **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)

-### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
- **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
-  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
-  `messages_processed` stalls.
- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
-  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
-  configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session
-  workers emit 403 on first prompt, then never recover because the error doesn't trigger
-  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
-  `workers=[<id>:busy:N]` with growing N in extension logs.
- **Fix**: (1) Update `models.json` to only include models from the authorized list.
-  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
-  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
-  routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should
-  only be used when explicitly requested. Validate model IDs at agent setup time.
- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
-
 ## Deployment Checklist

 When deploying a new Zulip adapter, verify:
@@ -105,4 +86,3 @@ When deploying a new Zulip adapter, verify:
 - [ ] `@all-bots` detected via configurable user_id
 - [ ] Poll uses long-poll (not `dont_block=true` polling)
 - [ ] Stuck/idle detection accounts for quiet periods
- [ ] Model IDs in config validated against `GET /v1/models` with actual API key
@@ -1,92 +0,0 @@
---
-kind: responsibility
-name: zulip-oidc-redirect-fix
-status: active
-description: >
-  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
-  uses the internal IP (192.168.68.19) instead of the public domain
-  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
-  Survives container restarts through compose.override.yaml.
-agent: abiba
-triggers:
-  - Zulip OIDC login returns "Redirect URI Error" from Authentik
-  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
-  - After Zulip server restart, Authentik SSO login broken
---
-
-## Maintains
-
- zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
- authentik-acceptance: { status: "accepted" | "rejected" }
- patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
-
-## Detection
-
-### Rule 1: Wrong redirect_uri host
- **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
- **Status**: CRITICAL — Authentik will reject
- **Trigger fix** → Execute self-heal
-
-### Rule 2: OIDC flow broken
- **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
- **Diagnose**: Check redirect_uri in the 302 Location header chain
-
-## Fix (Self-Heal)
-
-Two layers applied:
-
-### Layer 1: Live patch (inside container, immediate)
-```bash
-# Add to /home/zulip/deployments/current/zproject/computed_settings.py:
-SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
-
-import urllib.parse
-from social_core.strategy import BaseStrategy
-_original_absolute_uri = BaseStrategy.absolute_uri
-def _patched_absolute_uri(self, path=None):
-    from django.conf import settings
-    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
-    if path is not None:
-        return urllib.parse.urljoin(root, path)
-    return root
-BaseStrategy.absolute_uri = _patched_absolute_uri
-
-# Restart Django
-supervisorctl restart zulip-django
-```
-
-### Layer 2: Persistent fix (compose.override.yaml)
-The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
-`/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
-
-### Verification
-```bash
-STEP1=$(curl -sk -w "%{redirect_url}" \
-  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
-curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
-# Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
-# Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
-```
-
-### Rollback
-Remove the patch block from `compose.override.yaml` and restart the container:
-```bash
-docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
-```
-
-## Root Cause
-
-After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
-Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
-Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
-`Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
-Django falls back to the server's IP.
-
-The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
-`ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
-Host header.
-
-## Related Contracts
-
- `zulip-health.prose.md` — General Zulip health monitoring
- `zulip-self-heal.prose.md` — RETIRED (pi extension removed)
Author	SHA1	Message	Date
Agent Zero	b3db21b5e0	Revert "infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths" PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details This reverts commit `09ffbf10aa`.	2026-07-06 18:46:36 -04:00
Agent Zero	7dd8adf35b	Revert "infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks" This reverts commit `f14b2fc04e`.	2026-07-06 18:46:36 -04:00
Agent Zero	22ee074d5b	Revert "fix: Firecrawl health endpoint is / not /health" This reverts commit `f1a031b682`.	2026-07-06 18:46:36 -04:00