fix: add data source integrity rule to delegation contract

Workers MUST use provided input data, not fetch external sources. Adds: Data Source Integrity section, dispatch guidance example, anti-pattern entry. Root cause of e2e test discrepancies where writer worker queried Proxmox API instead of using raw data.
fix: add YAML frontmatter to RA-H OS Custodianship Contract
2026-07-11 18:51:30 -04:00 · 2026-07-10 19:16:30 -04:00 · 2026-07-10 09:43:49 -04:00 · 2026-07-09 07:43:26 +00:00 · 2026-07-09 03:18:29 -04:00 · 2026-07-09 02:53:44 -04:00
18 changed files with 1618 additions and 141 deletions
@@ -115,10 +115,11 @@ Run on trigger or schedule. Maintain persistent world-model state across runs.
 | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
 | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
 | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
 | `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
 ### Instantiable Templates (`kind: pattern` — run with `prose run`)
@@ -146,8 +147,9 @@ Called on-demand as single-render tools.
 | Contract | Description |
 |---|---|
-| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
+| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
-| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
+| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
 | `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
@@ -158,8 +160,8 @@ Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
-| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 ## Contract Structure
@@ -0,0 +1,256 @@
 ---
 kind: pattern
 name: delegation-prose-contract
 description: >
  Manager (Mumuni) operating doctrine for task decomposition, worker
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
 version: 1.0.0
 ---
 ## Maintains
 - Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
  `syslog-research`, `syslog-review`, `syslog-writer`)
 - Kanban board state at `~/.hermes/kanban/kanban.json`
 - Context window budget: ~65K tokens per request (131K total, 60% threshold)
 ## Topology
 **Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
 **Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
 This contract is infrastructure-agnostic in terms of which nodes are used.
 Workers execute tasks on whatever infrastructure they're given — SSH to .6,
 .pm, .9, .12, or .15 depending on the task. The contract defines the
 **who** and **when** — not the **where**.
 ## Why This Matters
 Without enforced delegation, the manager consumes the full iteration budget
 (60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
 leaving no capacity for actual coordination. The result: context overflow
 (59K tokens in system prompt), iteration exhaustion, and degraded response
 quality. This contract exists because I blew through my budget checking
 Proxmox node status instead of delegating to `syslog-devops`.
 ## Context Window Discipline
 **The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
 **Volatile (MEMORY.md + USER.md): ~300 tokens.**
 **Total base: ~6,800 tokens per request.**
 The remaining budget is the conversation. Every tool call result adds to it.
 If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
 from multiple nodes), the context fills fast. That's why we delegate: workers
 process in isolation and return compact results.
 ## Trigger Conditions
 Delegation is **mandatory** when any of these apply:
 | Condition | Threshold | Example |
 |-----------|-----------|---------|
 | Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
 | Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
 | Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
 | Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
 | Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
 | Code builds or changes | Writing or modifying code | Scripts, configs, patches |
 | Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
 **Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
 `curl`, `hermes tools list` — these are decision-making tools. The manager
 reads them directly.
 ## Worker Selection Matrix
 | Worker | Model | Toolsets | Role | Use When |
 |--------|-------|----------|------|----------|
 | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
 | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
 | `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
 | `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
 | `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
 | `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
 ### Selection Rules
 1. **Match specialty first.** A code task → `syslog-code`. An infra task →
   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
 2. **Research tasks with browser needs → `syslog-research`.** Other workers
   don't have the browser toolset.
 3. **Verification → `syslog-review`.** Never deliver raw worker output.
 4. **Documentation/content → `syslog-writer`.** Let them own the prose.
 5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
   (includes browser) and high reasoning effort.
 ## Delegation Protocol
 ### Step 1: Decompose
 Break the task into lanes. Each lane does ONE thing. Workers are independent —
 no lane depends on another's output mid-flight. If lanes depend on each other,
 dispatch sequentially.
 ### Step 2: Dispatch
 Fire workers via `delegate_task`:
 **Parallel (independent lanes):**
 ```
 delegate_task(
    tasks=[
        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
    ]
 )
 ```
 **Sequential (dependent lanes):**
 Dispatch lane 1 → wait for result → dispatch lane 2.
 ### Step 3: Verify
 **MANDATORY for:**
 - Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
 - Code builds and modifications
 - Research findings (web data, external sources)
 - Any output that will reach the user
 **Fire `syslog-review` to verify:**
 ```
 delegate_task(
    goal="Review the output of the devops worker. Verify the node status
    report is accurate, check for inconsistencies, confirm all nodes were
    reachable.",
    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
    Verify against live system."
 )
 ```
 **If verification fails:**
 1. Send work back to original worker with review feedback
 2. Re-verify
 3. Max 2 re-verify cycles before escalating to Kwame
 ### Step 4: Deliver
 Only verified results reach Kwame. Format per channel:
 - Telegram: Use `telegram-formatting` skill
 - Zulip: Use Zulip Markdown (CommonMark)
 - Email: Use `syslog-email` skill
 ## Kanban Board Protocol
 **File:** `~/.hermes/kanban/kanban.json`
 ```json
 {
  "task_id": "unique-id",
  "title": "Task description",
  "created": "2026-07-09T01:00:00",
  "status": "backlog|in_progress|review|done",
  "lanes": [
    {
      "lane_id": "devops-check",
      "worker": "syslog-devops",
      "goal": "Check all 5 Proxmox nodes",
      "status": "dispatched|completed|failed",
      "output_file": "/tmp/node-report.md"
    }
  ]
 }
 ```
 **Update the board on every state change.**
 ## Failure Handling
 ### Worker Timeouts
 - Child timeout: **900 seconds** (15 minutes)
 - Worker model `syslog-auto` is slow — it can hit the timeout limit with
  22+ API calls
 - **If a worker times out:** Re-dispatch with a narrower scope. Break the
  task into smaller pieces that fit in the timeout window.
 - **Avoid delegating sequential SSH hops** — each SSH connection adds latency
  that compounds quickly. Prefer API-based or local approaches when possible.
 ### Worker Selection Failures
 - `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
 - `syslog-code` is best for code-level work (reading files, writing scripts)
 - `syslog-research` has the browser toolset — use for web research
 - `syslog-review` is the QA gate — always fire before delivery
 - **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
 - **Never nest delegation** (max_spawn_depth: 1)
 ### Context Overflow
 - If a task requires >10K tokens of output, delegate the processing
 - Workers return compact summaries, not raw data dumps
 - Pass file paths and concrete goals — never dump raw data into context
 ## Anti-patterns
 - ❌ Reading large files into your own context before deciding → delegate the read
 - ❌ Carrying SSH/grep/output results in your context → delegate the analysis
 - ❌ Doing work yourself and then "pretending" to delegate → the user can tell
 - ❌ Skipping verification → raw worker output never reaches the user
 - ❌ Delegating single tool calls → keep quick reads/writes at manager level
 - ❌ Firing more than 3 workers in parallel → hard limit
 ## Emergency Exception
 **In an emergency (server down, service must be restored immediately):**
 - Delegate the diagnosis (find the problem)
 - Execute the fix yourself (minimize handoff latency)
 - Verify the fix after delivery
 - Log the exception in the kanban board
 The emergency exception exists because the user needs the service back NOW,
 not after three worker round-trips. But it's an exception — not the rule.
 ## What This Contract Doesn't Cover
 1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
 2. **SSH key management** — covered by existing SSH/Proxmox contracts
 3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
 4. **Cron job management** — covered by individual cron contracts
 5. **Infra verification** — covered by `verify-before-mutate` protocol
 ## Verification
 Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
 ```bash
 python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
 ```
 ## References
 - `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
 - `worker-profile-audit.md` (skill reference): Worker configuration audit notes
 - `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
 - `verify-before-mutate` protocol: Infrastructure change verification
 - `hermes-config-template.prose.md`: Worker profile configuration
 ## Success Criteria
 This contract succeeds when:
 1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
 2. **Workers do the work** — manager coordinates, doesn't execute
 3. **Verification before delivery** — all output passes through `syslog-review`
 4. **Kanban board is current** — every task has a lane, every lane has a status
 5. **User gets verified results** — raw worker output never reaches Kwame
 ---
 **Last updated:** 2026-07-09
 **Author:** Mumuni (with Kwame's input on triggers and exception criteria)
 **Status:** Draft — awaiting PR review and merge to prose-contracts main
@@ -3,10 +3,11 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
  Recurring disk health scan, garbage collection, and threat response
-  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
  Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB via `docker system prune -a --force`.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
 ## Scope
-All CTs in the Proxmox cluster, with special attention to Docker hosts:
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
 Docker hosts get special attention:
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
 | amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All others | — | LOW — no Docker | apt clean, log rotate |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
 | GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
 > **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
 > **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 ## Threat Levels
@@ -123,7 +130,7 @@ call summary-reporter
 ## GC Strategies by Host Type
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -224,32 +231,76 @@ dangling images and orphaned build cache. No automated GC was in place.
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
 - amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```
-Done. Here's what we have:
+## Incident Log: 2026-07-09 — amdpve docker bloat
-## Storage Summary
+### Discovery
 Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
 amdpve (.15) flagged at 78% (AMBER threshold: 75%).
-| System | Used | Free | Health |
+### Diagnosis
-|--------|------|------|--------|
+- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
-| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
+- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
-| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
+- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
-| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
+- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
-| **/root/go** (abiba) | 773M | — | Fine |
+  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
 - Not a running service — zero containers, zero active volumes
-## New Contract: `disk-gc-threat-response.prose.md`
+### Resolution
 ```
 docker system prune -a --force
 → Reclaimed 11.56GB
 → Post: 55G / 94G (62%), 35G free
 → 1 image removed (llama-strix-hip:latest, 15.5GB)
 → 8 build cache layers removed
 → amdpve now GREEN
 ```
-Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+### Root Cause
 Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
 artifacts. Docker on amdpve serves no running purpose — it's only used for
 one-off GPU builds. No automated post-migration cleanup was in place.
-### Three-in-One
+### Preventive Measures
 - amdpve added to Docker GC scan list
 - Post-migration cleanup step added: after any GPU backend migration, prune
  the old backend's Docker images within 24 hours
 - Contract now scans GPU bare-metal hosts alongside CTs
 - Access via `pct-run` script for all CTs (no hardcoded IPs)
-1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+## Access Matrix (documented 2026-07-09)
-2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+### CT Access (via pct-run)
 | CT | Name | Node | Status |
 |----|------|------|--------|
 | 100 | abiba | amdpve | local |
 | 102 | adguard | acerpve | ✅ reachable |
 | 104 | authentik | minipve | ✅ reachable |
 | 105 | kagentz | amdpve | ✅ reachable |
 | 106 | ra-h-os | storepve | ✅ reachable |
 | 107 | pbs | storepve | ✅ reachable |
 | 108 | media | storepve | ✅ reachable |
 | 110 | gitea | minipve | ✅ reachable |
 | 111 | tdunna | amdpve | ✅ reachable |
 | 112 | tanko | amdpve | ✅ reachable |
 | 113 | baggy | amdpve | ✅ reachable |
 | 114 | mumuni | minipve | ✅ reachable |
 | 115 | scottdenya | amdpve | ✅ reachable |
 | 116 | syslog-api | minipve | ✅ reachable |
 | 117 | zulip | storepve | ✅ reachable |
-3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
+### GPU Bare Metal (via direct SSH)
 | Host | IP | GPU | Status |
 |------|-----|-----|--------|
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
 | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
-### The incident root cause
+### KVM VM (via direct SSH)
-Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
+| Host | IP | Role | Status |
 |------|-----|------|--------|
 | docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
-Want me to push this to the prose-contracts repo?
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
@@ -142,7 +142,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 |-------|-----|-----|-----|--------|
 | Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
 | Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
-| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
 | Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
 | Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
 | Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
@@ -159,10 +159,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
@@ -19,7 +19,7 @@ agent: abiba
 ```
 ┌─────────────────────────────────────────────────────────┐
-│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  GPU Monitor Server (:9100) — 192.168.68.24              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
   │          │          │
@@ -35,7 +35,7 @@ agent: abiba
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
-Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
 poll .15:8080 directly; must go through router on .116.
 ```
@@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
 | File | Host | Purpose |
 |------|------|---------|
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
 ## Execution
@@ -0,0 +1,237 @@
 ---
 kind: function
 name: hermes-zulip-plugin
 description: >
  Installs or updates the Zulip platform plugin for any Hermes agent from the
  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
  recovery). Verifies the installation, restarts the gateway, and sends a relay
  success signal.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Plugin — Install & Repair
 Single-shot function that pulls the latest zulip-platform plugin from the
 canonical git repo, installs it to the correct Hermes bundled plugin path,
 verifies the installation, and signals completion.
 Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
 layer and a targeted gateway restart. It does NOT verify env credentials or
 run a live Zulip connection test. Use `hermes-zulip-restore` for full
 connectivity recovery including end-to-end DM validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 ## Maintains
 - plugin_installed: bool — Whether all three adapter files exist at the bundled path
 - plugin_version: string — Git commit SHA of the installed version
 - strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
 - signal_sent: bool — Whether relay success message was dispatched
 ### Postconditions
 - All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
 - `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
 - Installed version matches HEAD of the requested branch
 - Relay success signal sent to Hermes agent's inbox
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
 | Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
 ## Execution
 ### Step 1: Resolve Target
 Map `target` to host, CT ID, hermes_home, and user from the live-state table.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Pull Latest Plugin Source
 On the target host:
 ```bash
 # Ensure deploy scratch space
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 # Clone or pull
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins
  git fetch origin
  git checkout {{branch}}
  git pull origin {{branch}}
 else
  git clone --branch {{branch}} \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
  cd zulip-platform-plugins
 fi
 # Capture installed version
 INSTALLED_SHA=$(git rev-parse HEAD)
 echo "Installed SHA: $INSTALLED_SHA"
 # Verify we're at HEAD
 HEAD_SHA=$(git rev-parse origin/{{branch}})
 if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
  echo "At latest commit on {{branch}}"
 else
  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
 fi
 ```
 ### Step 3: Install Plugin Files
 ```bash
 # Ensure target directory exists
 mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp plugins/platforms/zulip/adapter.py \
   plugins/platforms/zulip/__init__.py \
   plugins/platforms/zulip/plugin.yaml \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only — runs as jerome user)
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 echo "Plugin files installed"
 ```
 ### Step 4: Verify Installation
 ```bash
 # Check all three files exist
 for f in __init__.py adapter.py plugin.yaml; do
  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
    echo "✅ $f present"
  else
    echo "❌ $f MISSING"
    exit 1
  fi
 done
 # Verify _strip_html fix
 grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
  && echo "✅ _strip_html fix present" \
  || echo "❌ _strip_html MISSING — plugin may be stale"
 # Show installed plugin.yaml version
 grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
 ```
 ### Step 5: Restart Gateway
 Plugin changes require a gateway restart to take effect:
 ```bash
 cd {{hermes_home}}/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart 2>&1 || \
  venv/bin/python -m hermes_cli.main gateway restart 2>&1
 ```
 Wait for restart to complete (up to 45s), then confirm:
 ```bash
 grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
 ```
 Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
 Quick smoke check — confirm zulip platform loaded:
 ```bash
 grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
 ```
 If gateway fails to restart, check logs for the crash cause before proceeding.
 ### Step 6: Send Relay Success Signal
 On the Abiba host (local), dispatch a relay message to the target agent:
 ```
 ra-h-os-createRelayNode:
  title: "Zulip plugin updated — {{target}}"
  source: |
    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
    _strip_html fix: PRESENT
    Timestamp: {{timestamp}}
  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
 ```
 ### Step 7: Report
 Compile results into a single status block:
 | Field | Value |
 |-------|-------|
 | Target | `{{target}}` |
 | Branch | `{{branch}}` |
 | Commit SHA | `{{INSTALLED_SHA}}` |
 | Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
 | `_strip_html` | `{{present|missing}}` |
 | Gateway restarted | `{{yes|no}}` |
 | Signal sent | `{{yes|no}}` |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
 | Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
 | `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
 | Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
 | Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
 ## Edge Differences from hermes-zulip-restore
 | Concern | hermes-zulip-restore | hermes-zulip-plugin |
 |---------|---------------------|---------------------|
 | Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
 | Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
 | Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
 | Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
 | Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
 | Signal dispatch | ❌ None | ✅ Relay message to target |
 For full connectivity recovery after a plugin install, chain this contract
 with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
 ---
 **Last updated**: 2026-07-08 — Switched default branch to `master`; added
 gateway restart step. If outstanding unmerged feature branches exist, address
 them in a follow-up merge after this contract completes.
@@ -0,0 +1,188 @@
 ---
 kind: function
 name: hermes-zulip-restore
 description: >
  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
  Koby CT111). Deploys the zulip-platform adapter to the correct bundled plugin
  path, verifies env credentials, restarts the gateway, and confirms Zulip
  connects. Run this whenever a Hermes agent stops responding on Zulip or after
  a fresh agent deployment.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Restore — Bring Any Agent Back to Good State
 Single-shot function that restores full Zulip connectivity for a Hermes agent.
 Covers adapter deployment, HTML stripping (slash command fix), env verification,
 gateway restart, and connection validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
 ## Maintains
 - adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
 - zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
 - env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
 - gateway_running: bool — Whether gateway process is running
 ### Postconditions
 - `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
 - All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
 - Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
 - Gateway restarted and zulip platform reports state `connected`
 - HTML stripping enabled for `/approve` and `/deny` slash command support
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 - Zulip server accessible at `https://chat.sysloggh.net`
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Zulip server | https://chat.sysloggh.net | ✅ Verified |
 | Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
 ## Execution
 ### Step 1: Locate Target
 Map `target` to connectivity parameters from the live-state table above.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Deploy Zulip Adapter
 On the target host:
 ```bash
 # Clone or update the plugin repo
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
 else
  git clone --branch feat/zulip-streaming \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
 fi
 # Ensure bundled plugin directory exists
 mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
 # Clean up
 rm -rf /tmp/zulip-deploy
 ```
 ### Step 3: Verify _strip_html is Present
 ```bash
 grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
 ```
 Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
 ### Step 4: Verify Env Credentials
 ```bash
 grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
 ```
 Expected: all three variables set with non-empty values. If any missing:
 - ZULIP_SITE: `https://chat.sysloggh.net`
 - ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
 - ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
 ### Step 5: Restart Gateway
 ```bash
 cd <HERMES_HOME>/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
 ```
 Wait for the restart to complete (up to 45s). Check:
 ```bash
 grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
 ```
 Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
 ### Step 6: Validate Zulip Connection
 ```bash
 python3 -c "
 import json
 d = json.load(open('$HERMES_HOME/gateway_state.json'))
 print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
 "
 ```
 Expected: `zulip: connected`. If not connected, check gateway log:
 ```bash
 grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
 ```
 ### Step 7: Report
 Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
 | State | Action |
 |-------|--------|
 | All true | ✅ Agent restored — relay success to user |
 | `adapter_deployed: false` | Re-run Step 2 |
 | `env_valid: false` | Prompt for missing credentials |
 | `zulip_connected: false` | Check Zulip server reachability, verify API key |
 | `gateway_running: false` | Check process logs for crash cause |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
 | Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
 | No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
 | `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
 | Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
 | Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
 ## Git Branch Reference
 The `_strip_html` fix lives on `feat/zulip-streaming` branch:
 ```
 https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
 ```
 Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
 Pull request #33 is the primary integration branch.
 ---
 **Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
@@ -144,7 +144,7 @@ description: >
 ### Ecosystem A: docker-vm (192.168.68.7)
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
 | **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
 **Last verified:** 2026-07-09 — 16/16 containers running healthy.
 ### Home Stack Services
@@ -7,10 +7,14 @@ description: >
  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
  via existing /metrics Prometheus endpoint.
-  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  DEPLOYMENT STATUS (2026-07-09):
-  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  This contract defines the target state; proxmox-monitor is the as-built.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
     NVIDIA sidecar exporters (.8/.110:9400) never installed.
     Router falls back to direct GPU /health probes.
  ⚠️ This contract is target-state aspirational — not as-built.
     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
@@ -24,10 +24,11 @@ version: 1.1.0
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
-4. ✅ LiteLLM health check passing
+4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
-5. ✅ Zulip server reachable
+5. ✅ LiteLLM health check passing
 6. ✅ Zulip server reachable
 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
@@ -8,6 +8,10 @@ description: >
  Ensures agents never use the master key directly. Rotation is event-driven,
  not calendar-driven — rotate only on compromise, personnel change, or
  periodic security hygiene (quarterly/annually).
  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
  on CT 116. Last verified: 2026-07-09.
 ---
 ## Parameters
@@ -38,7 +42,8 @@ description: >
   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
   - Return the new key
 5. **If action == "rotate"**:
   - Generate new key with same alias (LiteLLM replaces the old key)
@@ -1,35 +1,50 @@
 ---
 kind: function
 name: litellm-health
 status: deprecated
 deprecated_on: 2026-07-09
 replaced_by: litellm-self-heal.prose.md
 note: >
  Consolidated into litellm-self-heal.prose.md to eliminate duplication
  of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within litellm-self-heal.
  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
-  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
-  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Router (harness-router :9000) is DEPRECATED — container still runs but
-  Prometheus/Grafana and fleet dashboard.
+  is not in the request path. GPU monitoring via Prometheus/Grafana and
  fleet dashboard (gpu-monitor :9100).
  Designed as a reusable contract for any Syslog agent.
  Source of truth: gpu-fleet.prose.md
 ---
-## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 ```
-Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
-                        │                    │
+                        │
-                  Key validation      Model routing
+                  Key validation
-                  Fallback chains     Slot booking
+                  Fallback chains
-                  Complexity router   Circuit breakers
+                  Budget tracking
-                  Budget tracking     Redis-backed
+                        │
-                        │                    │
+                  Prometheus ← metrics
                  Prometheus ← metrics  ←───┘
                        │
                    Grafana :3001
  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
 ```
-**What changed (v3.1.0 → v3.2.0)**:
+**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
- Router is BACK — runs internally on :9000 behind LiteLLM
+- Router REMOVED from request path — LiteLLM proxies directly to GPU
- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- All GPUs at parallel 2 (was parallel 1)
- Router :9000 is 127.0.0.1-only, not publicly accessible
+- NVIDIA context reduced 256K→128K to free VRAM
- GPU fleet dashboard: http://192.168.68.24:9100
+- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
 ## Parameters
@@ -55,27 +70,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se
 ## GPU Fleet Topology
-| Host | IP | Hardware | Models Served | Engine |
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
-|------|-----|----------|---------------|--------|
+|------|-----|----------|---------------|--------|---------|----------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
-| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 ## Model Fallback Chains (LiteLLM)
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
-| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
-| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | — | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
 > Global: request_timeout=300s, nginx proxy_read_timeout=600s
 ## Containers on CT 116
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
@@ -6,22 +6,92 @@ note: >
  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
  This contract is now manual-only — triggers require explicit user request.
  Consider reimplementing as a standalone cron job or prose contract.
  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
  duplication of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within this contract.
  Source of truth for GPU topology and keys: gpu-fleet.prose.md
  Last verified: 2026-07-09
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
-  knowledge graph for full audit trail.
+  inference, and agent keys. Applies remediation rules for common failures.
  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 # LiteLLM Operations — Health Check + Self-Heal
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 ```
 Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001
  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
 ```
 ## Parameters
 - public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
 - backend_host: string — Internal CT host (default: "192.168.68.116")
 - auth_host: string — Authentik OIDC server (default: "192.168.68.11")
 - gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
 - gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
 - grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
 ## GPU Fleet Topology
 | Host | IP | Hardware | Models Served | Engine | Context | Parallel |
 |------|-----|----------|---------------|--------|---------|----------|
 | llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
 | ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
 | amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 ## Model Fallback Chains (LiteLLM)
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
 | qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
 | gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
 | syslog-auto (balanced) | 90s | qwen → gemma | — |
 > Global: request_timeout=300s, nginx proxy_read_timeout=600s
 ## Containers on CT 116
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
 | harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
 | harness-dashboard | inference-harness-dashboard | :3000 | /health |
 | harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
 | harness-prometheus | prom/prometheus | :9090 | /-/healthy |
 | harness-docker-stats | python:3.12-alpine | — | container stats exporter |
 | harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
 ## Maintains
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
-## Requires
+- litellm-agent-keys: { count: int, valid: int }
 - litellm-health:function
 ## Continuity
@@ -29,55 +99,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
-## Reporting
+---
-Every remediation cycle produces a structured report in three channels:
+## Health Check
-### 1. RA-H OS Knowledge Graph Node
+Run this first on every cycle. Results feed into remediation rules below.
 Created as `[LEARN] litellm-self-heal: <run_id>` with:
 - metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
 - source: Full JSON report of the cycle
 - description: Summary of what was found, fixed, and escalated
-### 2. Zulip DM to Owner
+### 1. Check public endpoints
-Sent immediately for:
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
 - `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
 - Every 10th clean cycle — "✅ All Clear (10 checks passed)"
-### 3. Daily Digest (end of day)
+### 2. Check LiteLLM health (no-auth)
-A summary of the last 24 hours sent as a single Zulip DM:
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
 ```
 📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
-Total cycles: 288 (every 5 min)
+### 3. Check backend container health
-Issues found: 3
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+- Critical: harness-litellm, harness-nginx, harness-postgres
-  └─ Escalated: 0
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
 - Deprecated but running: harness-router (not in path, reference only)
-Top actions:
+### 4. Check GPU fleet health (via fleet dashboard)
-  • harness-nginx restarted — 2026-06-26 02:15
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
-  • DOCS_URL corrected — 2026-06-26 07:30  
+- Verify GPUs reporting status "healthy"
-  • harness-nginx restarted — 2026-06-26 14:46
+- Check alerts array for active warnings/critical
-Uptime: 23h 47m — All healthy now ✅
+### 5. Check model inference via LiteLLM — test each model
-```
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
 - POST /v1/chat/completions model=ornith-1.0-35b → expect 200
 - Use master key for auth
-### 4. Weekly Digest (every Monday)
+### 6. Check agent keys
-Same format as daily but covering 7 days. Sent to both Zulip DM and
+- GET /key/list with master key → verify all 6 agents have keys
 logged as a `[REPORT]` knowledge graph node for long-term trending.
-### 5. Relay Message (if cross-agent)
+### 7. Check Grafana
-If a fix required another agent's help (e.g., Authentik restart), a relay
+- GET {{grafana_url}}/api/health → expect 200
 message is sent to the responsible agent with full context.
-## GPU Fleet
+### 8. Compile overall status
 Determine overall_status from individual check results:
 - "healthy" — all checks pass
 - "degraded" — 1-2 non-critical checks fail
 - "down" — critical checks fail
-| Host | IP | Hardware | Models | Engine |
+---
 |------|-----|----------|--------|--------|
 | amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
 | llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
 | ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
 ## Remediation Rules
@@ -105,39 +169,89 @@ Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts
-### Rule 7: Router Returns 503 (All GPUs Saturated)
+### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
-Detect → router logs show QUEUE_TIMEOUT with GPUs idle
+Router is no longer in the request path (2026-07-08). LiteLLM proxies
-Root cause → roster not loaded at startup (empty TIER_MODELS)
+directly to GPU. This rule is retained for reference but is inactive.
-Fix → POST /admin/roster/reload (if endpoint available) or restart router
+If 503 errors occur, check LiteLLM timeouts and GPU health directly.
 Permanent fix → add load_roster() to router startup code
 Escalate → if reload doesn't resolve within 2 min
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
-Root cause → keys in router env var, not in LiteLLM DB
+Root cause → keys not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM
-### Rule 9: Stale Active Counter in Redis
+### Rule 9: Stale Active Counter in Redis — DEPRECATED
-Detect → active:<model> > 0 while GPU idle
+Router no longer in path so Redis active counters are unused. Rule retained
-Fix → `redis-cli DEL active:<model>` → restart router
+for reference but inactive. If Redis issues occur, check harness-redis container.
-Prevent → add TTL to active keys in router code
+
 ---
 ## Reporting
 Every remediation cycle produces a structured report:
 ### 1. RA-H OS Knowledge Graph Node
 Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
 ### 2. Zulip DM to Owner
 - `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
 - `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
 - Every 10th clean cycle — "✅ All Clear (10 checks passed)"
 ### 3. Daily Digest (end of day)
 Summary of last 24 hours: total cycles, issues found/fixed/escalated,
 top actions, uptime.
 ### 4. Relay Message (if cross-agent)
 If a fix requires another agent (e.g., Authentik restart), relay sent
 to responsible agent with full context.
 ---
 ## Execution
-1. Run health check
+```prose
-2. Apply remediation for each failure
+-- Phase 1: Health Check
-3. **Generate report** with all actions taken
+let health = call health-check
-4. **Log to knowledge graph** — create `[LEARN]` node
+  public_url: public_url
-5. **Notify user** via Zulip DM if anything changed
+  backend_host: backend_host
-6. Wait 300s and repeat
+  gpu_dashboard_url: gpu_dashboard_url
  grafana_url: grafana_url
 -- Phase 2: Apply remediation for each failure
 let actions = []
 for check in health.failed:
  let fix = apply-remediation-rule
    rule: lookup-rule(check.name)
    target: check.target
  push actions fix
 -- Phase 3: Generate report
 call report-generator
  health: health
  actions: actions
 -- Phase 4: Log to knowledge graph
 call kg-logger
  run_id: run_id
  health: health
  actions: actions
 -- Phase 5: Notify if anything changed
 if actions.length > 0:
  call zulip-notifier
    actions: actions
    health: health
 -- Wait 300s and repeat
 ```
 ## Audit Trail Format
 ```json
 {
-  "run_id": "self-heal-20260626-001",
+  "run_id": "self-heal-20260709-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "timestamp": "2026-07-09T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
@@ -151,7 +265,7 @@ Prevent → add TTL to active keys in router code
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
@@ -0,0 +1,290 @@
 ---
 kind: pattern
 name: mumuni-delegation
 description: >
  Mumuni-specific operating doctrine for task decomposition, worker
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
 version: 1.0.0
 ---
 ## Maintains
 - Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
  `syslog-research`, `syslog-review`, `syslog-writer`)
 - Kanban board state at `~/.hermes/kanban/kanban.json`
 - Context window budget: ~65K tokens per request (131K total, 60% threshold)
 ## Topology
 **Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
 **Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
 This contract is infrastructure-agnostic in terms of which nodes are used.
 Workers execute tasks on whatever infrastructure they're given — SSH to .6,
 .pm, .9, .12, or .15 depending on the task. The contract defines the
 **who** and **when** — not the **where**.
 ## Why This Matters
 Without enforced delegation, the manager consumes the full iteration budget
 (60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
 leaving no capacity for actual coordination. The result: context overflow
 (59K tokens in system prompt), iteration exhaustion, and degraded response
 quality. This contract exists because I blew through my budget checking
 Proxmox node status instead of delegating to `syslog-devops`.
 ## Context Window Discipline
 **The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
 **Volatile (MEMORY.md + USER.md): ~300 tokens.**
 **Total base: ~6,800 tokens per request.**
 The remaining budget is the conversation. Every tool call result adds to it.
 If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
 from multiple nodes), the context fills fast. That's why we delegate: workers
 process in isolation and return compact results.
 ## Trigger Conditions
 Delegation is **mandatory** when any of these apply:
 | Condition | Threshold | Example |
 |-----------|-----------|---------|
 | Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
 | Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
 | Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
 | Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
 | Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
 | Code builds or changes | Writing or modifying code | Scripts, configs, patches |
 | Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
 **Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
 `curl`, `hermes tools list` — these are decision-making tools. The manager
 reads them directly.
 ## Data Source Integrity (CRITICAL)
 **Workers MUST use the data provided in their task context. They MUST NOT
 fetch their own data from external sources unless explicitly told to.**
 When a task says "Read file X and format it", the worker reads file X. It does
 not query a separate API, run its own diagnostics, or pull data from a different
 system. This is the #1 source of cross-worker inconsistency: one worker gathers
 SSH data, another queries the Proxmox API, and the report merges two incompatible
 datasets.
 **Rule:** If a worker needs additional data beyond what's in its task description,
 it asks the manager (via relay) — it doesn't go find it on its own.
 **This is a hard rule, not a recommendation.** Violating it produces the exact
 type of discrepancy the kanban pipeline exists to prevent: a review worker finds
 "5 nodes present" in the raw data but "5/5 online" in the report — even though
 one of those nodes was unreachable. The report lied because it used data the
 raw data never provided.
 ## Worker Selection Matrix
 | Worker | Model | Toolsets | Role | Use When |
 |--------|-------|----------|------|----------|
 | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
 | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
 | `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
 | `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
 | `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
 | `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
 ### Selection Rules
 1. **Match specialty first.** A code task → `syslog-code`. An infra task →
   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
 2. **Research tasks with browser needs → `syslog-research`.** Other workers
   don't have the browser toolset.
 3. **Verification → `syslog-review`.** Never deliver raw worker output.
 4. **Documentation/content → `syslog-writer`.** Let them own the prose.
 5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
   (includes browser) and high reasoning effort.
 ## Delegation Protocol
 ### Step 1: Decompose
 Break the task into lanes. Each lane does ONE thing. Workers are independent —
 no lane depends on another's output mid-flight. If lanes depend on each other,
 dispatch sequentially.
 ### Step 2: Dispatch
 Fire workers via `delegate_task`:
 **Critical: Pass the data, not just the goal.** When dispatching a worker that
 processes output from another worker, include the file path AND explicit
 instructions to use ONLY that source. Example:
 ```
 delegate_task(
    goal="Format the cluster check into a clean report",
    context="Source data is at /tmp/proxmox-check-raw.md. Format ONLY the data
    in that file. Do NOT query the Proxmox API or any other data source. Use the
    file as your sole source of truth."
 )
 ```
 **Parallel (independent lanes):**
 ```
 delegate_task(
    tasks=[
        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
    ]
 )
 ```
 **Sequential (dependent lanes):**
 Dispatch lane 1 → wait for result → dispatch lane 2.
 ### Step 3: Verify
 **MANDATORY for:**
 - Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
 - Code builds and modifications
 - Research findings (web data, external sources)
 - Any output that will reach the user
 **Fire `syslog-review` to verify:**
 ```
 delegate_task(
    goal="Review the output of the devops worker. Verify the node status
    report is accurate, check for inconsistencies, confirm all nodes were
    reachable.",
    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
    Verify against live system."
 )
 ```
 **If verification fails:**
 1. Send work back to original worker with review feedback
 2. Re-verify
 3. Max 2 re-verify cycles before escalating to Kwame
 ### Step 4: Deliver
 Only verified results reach Kwame. Format per channel:
 - Telegram: Use `telegram-formatting` skill
 - Zulip: Use Zulip Markdown (CommonMark)
 - Email: Use `syslog-email` skill
 ## Kanban Board Protocol
 **File:** `~/.hermes/kanban/kanban.json`
 ```json
 {
  "task_id": "unique-id",
  "title": "Task description",
  "created": "2026-07-09T01:00:00",
  "status": "backlog|in_progress|review|done",
  "lanes": [
    {
      "lane_id": "devops-check",
      "worker": "syslog-devops",
      "goal": "Check all 5 Proxmox nodes",
      "status": "dispatched|completed|failed",
      "output_file": "/tmp/node-report.md"
    }
  ]
 }
 ```
 **Update the board on every state change.**
 ## Failure Handling
 ### Worker Timeouts
 - Child timeout: **900 seconds** (15 minutes)
 - Worker model `syslog-auto` is slow — it can hit the timeout limit with
  22+ API calls
 - **If a worker times out:** Re-dispatch with a narrower scope. Break the
  task into smaller pieces that fit in the timeout window.
 - **Avoid delegating sequential SSH hops** — each SSH connection adds latency
  that compounds quickly. Prefer API-based or local approaches when possible.
 ### Worker Selection Failures
 - `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
 - `syslog-code` is best for code-level work (reading files, writing scripts)
 - `syslog-research` has the browser toolset — use for web research
 - `syslog-review` is the QA gate — always fire before delivery
 - **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
 - **Never nest delegation** (max_spawn_depth: 1)
 ### Context Overflow
 - If a task requires >10K tokens of output, delegate the processing
 - Workers return compact summaries, not raw data dumps
 - Pass file paths and concrete goals — never dump raw data into context
 ## Anti-patterns
 - ❌ Reading large files into your own context before deciding → delegate the read
 - ❌ Carrying SSH/grep/output results in your context → delegate the analysis
 - ❌ Doing work yourself and then "pretending" to delegate → the user can tell
 - ❌ Skipping verification → raw worker output never reaches the user
 - ❌ Delegating single tool calls → keep quick reads/writes at manager level
 - ❌ Firing more than 3 workers in parallel → hard limit
 - ❌ **Workers fetching their own data sources** → a writer worker that queries the Proxmox API when told to "format the raw file" is fabricating data. Use the input given, not external sources
 ## Emergency Exception
 **In an emergency (server down, service must be restored immediately):**
 - Delegate the diagnosis (find the problem)
 - Execute the fix yourself (minimize handoff latency)
 - Verify the fix after delivery
 - Log the exception in the kanban board
 The emergency exception exists because the user needs the service back NOW,
 not after three worker round-trips. But it's an exception — not the rule.
 ## What This Contract Doesn't Cover
 1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
 2. **SSH key management** — covered by existing SSH/Proxmox contracts
 3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
 4. **Cron job management** — covered by individual cron contracts
 5. **Infra verification** — covered by `verify-before-mutate` protocol
 ## Verification
 Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
 ```bash
 python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
 ```
 ## References
 - `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
 - `worker-profile-audit.md` (skill reference): Worker configuration audit notes
 - `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
 - `verify-before-mutate` protocol: Infrastructure change verification
 - `hermes-config-template.prose.md`: Worker profile configuration
 ## Success Criteria
 This contract succeeds when:
 1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
 2. **Workers do the work** — manager coordinates, doesn't execute
 3. **Verification before delivery** — all output passes through `syslog-review`
 4. **Kanban board is current** — every task has a lane, every lane has a status
 5. **User gets verified results** — raw worker output never reaches Kwame
 ---
 **Last updated:** 2026-07-09
 **Author:** Mumuni (with Kwame's input on triggers and exception criteria)
 **Status:** Draft — awaiting PR review and merge to prose-contracts main
@@ -12,6 +12,7 @@ description: >
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
 - gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
@@ -0,0 +1,210 @@
 ---
 kind: pattern
 name: ra-h-os-custodianship-contract
 description: >
  Operational standards for maintaining RA-H OS as a high-functioning shared
  memory system across all agents. Defines rules for embedding consistency,
  namespace discipline, staleness management, orphan prevention, agent
  custodianship, and recovery protocols. Prevents knowledge graph degradation
  and ensures reliable semantic search capabilities.
 ---
 # RA-H OS Custodianship Contract — Shared Memory Protocol
 ## Purpose
 This contract establishes the operational standards for maintaining RA-H OS as a high-functioning shared memory system across all agents. It defines the rules, responsibilities, and recovery protocols that prevent knowledge graph degradation and ensure consistent, reliable semantic search capabilities.
 ## Core Principles
 ### 1. Embedding Consistency
 **Rule:** All nodes MUST use the RA-H OS built-in embedding pipeline via `http://192.168.68.65:8080/v1/embeddings` for consistent vector representation.
 **Requirements:**
 - The embedding service at `192.168.68.65:8080` is the single source of truth for vector generation
 - No external embedding APIs (OpenAI, Cohere, etc.) may be used for RA-H OS nodes
 - All nodes must have `embedding_status: "chunked"` and `chunk_status: "chunked"` upon creation
 - If embedding fails, the node must be immediately flagged as `state: "unsearchable"` and logged
 ### 2. Namespace Discipline
 **Rule:** All nodes MUST be categorized into one of three namespaces with strict metadata requirements.
 **Namespace Structure:**
 - `agent-private`: Isolated working notes for individual agents (Mumuni, Tanko, Okyeame, etc.)
 - `shared`: Policy files, registry nodes, collective knowledge
 - `syslogsolution`: Business nodes, client data, operational context
 **Metadata Requirements:**
 Every node MUST have these metadata fields:
 ```json
 {
  "agent_id": "<agent_name>",
  "namespace": "<namespace>",
  "state": "<state>",
  "type": "<type>",
  "owner": "<owner>",
  "tenant": "<tenant>",
  "visibility": "<visibility>",
  "source": "<source>",
  "captured_by": "<captured_by>"
 }
 ```
 ### 3. Staleness Management
 **Rule:** Nodes are categorized by type and have specific staleness thresholds.
 **Staleness Thresholds:**
 - `policy` type: 30 days (e.g., Shared Memory Policy, Agent Registry)
 - `registry` type: 30 days (e.g., Node State Management, Health Dashboard)
 - `template` type: 30 days (e.g., Agent SOUL.md)
 - `note` type: 60 days (e.g., business context, project notes)
 - `information` type: 90 days (e.g., reference documentation)
 - `idea` type: 30 days (e.g., brainstorming, experimental notes)
 **Transition Protocol:**
 - `active` → `stale`: After days since `updated_at` exceeds threshold
 - `stale` → `review_pending`: Daily cron check flags for human review
 - `review_pending` → `active`: Human reviews and updates content
 - `review_pending` → `archived`: After 15 days in review_pending state (automatic purge)
 ### 4. Orphan Prevention
 **Rule:** No node should remain orphaned (0 edges) for more than 7 days.
 **Orphan Detection:**
 - Daily cron job identifies nodes with 0 incoming/outgoing edges
 - Orphans are flagged with `state: "review_pending"` and `metadata.orphan_detected: true`
 - 7-day grace period: Orphans must be either:
  - Connected to relevant nodes via edges
  - Merged into existing parent nodes
  - Archived after 15 days in review_pending state
 ### 5. Agent Custodianship
 **Rule:** Each agent is responsible for maintaining their own created nodes.
 **Responsibilities:**
 - **Mumuni**: Primary custodian for `syslogsolution` namespace, health dashboard, and all business nodes
 - **Tanko**: Custodian for `agent-private` namespace, personal notes, and fitness/creative work
 - **Okyeame**: Custodian for `shared` namespace, policy files, and collective knowledge nodes
 - **All Agents**: Must verify embedding status before reporting node creation as complete
 ### 6. Embedding Failure Recovery
 **Rule:** When embedding pipeline is unavailable, nodes are blocked from creation or immediately flagged.
 **Recovery Protocol:**
 1. **Detection**: Daily health check monitors `http://192.168.68.65:8080/health` endpoint
 2. **Alert**: If embedding service is down, alert is sent to all agents via relay
 3. **Mitigation**: 
   - New node creation is suspended
   - Existing nodes with `chunk_status: "not_chunked"` are flagged as `unsearchable`
 4. **Recovery**: When service returns:
   - All `not_chunked` nodes are queued for re-embedding
   - Re-embedding status logged in `embedding_retry_log` table
   - 5-minute retry window with exponential backoff (1s, 2s, 4s, 8s, 16s)
 ### 7. Chunking Verification
 **Rule:** All node modifications must verify chunking status within 5 minutes.
 **Verification Protocol:**
 - After `updateNode` or `createNode`, query `chunks` table for `node_id`
 - If `chunk_status != "chunked"` after 5 minutes, log failure and flag node
 - Automated cleanup script runs every 6 hours to retry failed chunks
 - Maximum 3 retry attempts per node before escalation to human
 ### 8. Graph Health Monitoring
 **Rule:** Health dashboard (Node #74) must reflect real-time graph status.
 **Dashboard Requirements:**
 - Total nodes, edges, orphans, stale nodes, chunk errors
 - Embedding service health status
 - Last successful embedding attempt
 - Alert thresholds:
  - Orphan rate > 40%: Warning
  - Stale node rate > 50%: Critical
  - Chunk error rate > 30%: Critical
  - Embedding service down: Critical (immediate alert)
 ## Operational Procedures
 ### Daily Maintenance (Automated)
 1. **Staleness Check**: Identify nodes past their threshold
 2. **Orphan Detection**: Flag nodes with 0 edges
 3. **Chunk Verification**: Retry failed chunks, log errors
 4. **Health Update**: Refresh Node #74 dashboard
 ### Weekly Maintenance (Human Review)
 1. **Review Pending Nodes**: Examine flagged nodes
 2. **Edge Optimization**: Connect related orphaned nodes
 3. **Archive Cleanup**: Remove nodes past 15-day review period
 4. **Embedding Audit**: Verify all nodes have valid embeddings
 ### Emergency Recovery
 1. **Embedding Service Down**: 
   - Suspend node creation
   - Alert all agents
   - Investigate root cause (check `192.168.68.65:8080`)
 2. **Database Corruption**:
   - Restore from latest PBS backup
   - Verify chunk integrity
   - Re-run embedding for affected nodes
 3. **Mass Orphan Creation**:
   - Identify source agent/namespace
   - Review recent changes
   - Reconnect or archive affected nodes
 ## Compliance & Enforcement
 ### Audit Schedule
 - **Daily**: Automated health checks
 - **Weekly**: Human review of review_pending nodes
 - **Monthly**: Comprehensive graph audit (full schema validation)
 - **Quarterly**: Contract review and threshold adjustment
 ### Violation Consequences
 - **First**: Alert to responsible agent
 - **Second**: Node placed in `review_pending` state
 - **Third**: Agent access suspended until remediation
 - **Fourth**: Escalation to human (Kwame)
 ### Metrics for Success
 - Orphan rate: < 10%
 - Stale node rate: < 20%
 - Chunk error rate: < 5%
 - Embedding service uptime: > 99.9%
 - Node creation verification: 100%
 ## Implementation Notes
 ### Database Schema Requirements
 - `nodes` table: Standard fields plus `embedding_status`, `chunk_status`, `last_embedding_attempt`
 - `chunks` table: Must track `chunk_status` and `embedding_status`
 - `embedding_retry_log` table: Track retry attempts for failed chunks
 - `node_state_history` table: Log state transitions for audit trail
 ### Cron Job Requirements
 - `ra-h-health-check`: Daily (4h interval)
 - `ra-h-staleness-detection`: Daily
 - `ra-h-orphan-detection`: Daily
 - `ra-h-chunk-retry`: Every 6 hours
 - `ra-h-review-cleanup`: Weekly (15-day archive)
 ### Agent Onboarding
 All new agents must:
 1. Read this contract
 2. Understand namespace responsibilities
 3. Know how to verify embedding status
 4. Know the alert escalation path
 ---
 ## Approval
 This contract is effective immediately upon approval by the primary custodian (Mumuni) and system owner (Kwame).
 **Effective Date:** July 10, 2026  
 **Review Date:** October 10, 2026  
 **Primary Custodian:** Mumuni 🦅  
 **System Owner:** Jerome Tabiri
 ---
 *Contract Version: 1.0*  
 *Last Updated: July 10, 2026*  
 *Storage: `/root/.hermes/skills/ra-h-os-custodianship-contract.prose.md`*
@@ -30,6 +30,12 @@ declare -A CT_NODES=(
    # acerpve (192.168.68.9)
    [102]=acerpve   # adguard
    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
    #
    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
    #   118 jitsi     → stopped, not in service
 )
 # Each node must be root-accessible via SSH hostname
@@ -0,0 +1,92 @@
 ---
 kind: responsibility
 name: zulip-oidc-redirect-fix
 status: active
 description: >
  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
  uses the internal IP (192.168.68.19) instead of the public domain
  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
  Survives container restarts through compose.override.yaml.
 agent: abiba
 triggers:
  - Zulip OIDC login returns "Redirect URI Error" from Authentik
  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
  - After Zulip server restart, Authentik SSO login broken
 ---
 ## Maintains
 - zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
 - authentik-acceptance: { status: "accepted" | "rejected" }
 - patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
 ## Detection
 ### Rule 1: Wrong redirect_uri host
 - **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
 - **Status**: CRITICAL — Authentik will reject
 - **Trigger fix** → Execute self-heal
 ### Rule 2: OIDC flow broken
 - **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
 - **Diagnose**: Check redirect_uri in the 302 Location header chain
 ## Fix (Self-Heal)
 Two layers applied:
 ### Layer 1: Live patch (inside container, immediate)
 ```bash
 # Add to /home/zulip/deployments/current/zproject/computed_settings.py:
 SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
 import urllib.parse
 from social_core.strategy import BaseStrategy
 _original_absolute_uri = BaseStrategy.absolute_uri
 def _patched_absolute_uri(self, path=None):
    from django.conf import settings
    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
    if path is not None:
        return urllib.parse.urljoin(root, path)
    return root
 BaseStrategy.absolute_uri = _patched_absolute_uri
 # Restart Django
 supervisorctl restart zulip-django
 ```
 ### Layer 2: Persistent fix (compose.override.yaml)
 The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
 `/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
 ### Verification
 ```bash
 STEP1=$(curl -sk -w "%{redirect_url}" \
  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
 curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
 # Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
 # Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
 ```
 ### Rollback
 Remove the patch block from `compose.override.yaml` and restart the container:
 ```bash
 docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
 ```
 ## Root Cause
 After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
 Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
 Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
 `Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
 Django falls back to the server's IP.
 The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
 `ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
 Host header.
 ## Related Contracts
 - `zulip-health.prose.md` — General Zulip health monitoring
 - `zulip-self-heal.prose.md` — RETIRED (pi extension removed)