fix: add data source integrity rule to delegation contract

Workers MUST use provided input data, not fetch external sources.
Adds: Data Source Integrity section, dispatch guidance example,
anti-pattern entry. Root cause of e2e test discrepancies where
writer worker queried Proxmox API instead of using raw data.

README.md

@@ -115,10 +115,11 @@ Run on trigger or schedule. Maintain persistent world-model state across runs.
 | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
 | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
 | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
+| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
 
 ### Instantiable Templates (`kind: pattern` — run with `prose run`)
@@ -146,8 +147,9 @@ Called on-demand as single-render tools.
 
 | Contract | Description |
 |---|---|
-| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
+| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
-| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
+| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
+| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
 
@@ -158,8 +160,8 @@ Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
-| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 
 ## Contract Structure
 

delegation-prose-contract.prose.md

@@ -0,0 +1,256 @@
+---
+kind: pattern
+name: delegation-prose-contract
+description: >
+  Manager (Mumuni) operating doctrine for task decomposition, worker
+  delegation, verification, and delivery. Defines when to delegate, which
+  worker to use for what, how to handle failures, and the kanban board
+  protocol. Enforces context-window discipline and separation of concerns.
+  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
+version: 1.0.0
+---
+
+## Maintains
+
+- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
+  `syslog-research`, `syslog-review`, `syslog-writer`)
+- Kanban board state at `~/.hermes/kanban/kanban.json`
+- Context window budget: ~65K tokens per request (131K total, 60% threshold)
+
+## Topology
+
+**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
+**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
+**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
+
+This contract is infrastructure-agnostic in terms of which nodes are used.
+Workers execute tasks on whatever infrastructure they're given — SSH to .6,
+.pm, .9, .12, or .15 depending on the task. The contract defines the
+**who** and **when** — not the **where**.
+
+## Why This Matters
+
+Without enforced delegation, the manager consumes the full iteration budget
+(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
+leaving no capacity for actual coordination. The result: context overflow
+(59K tokens in system prompt), iteration exhaustion, and degraded response
+quality. This contract exists because I blew through my budget checking
+Proxmox node status instead of delegating to `syslog-devops`.
+
+## Context Window Discipline
+
+**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
+**Volatile (MEMORY.md + USER.md): ~300 tokens.**
+**Total base: ~6,800 tokens per request.**
+
+The remaining budget is the conversation. Every tool call result adds to it.
+If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
+from multiple nodes), the context fills fast. That's why we delegate: workers
+process in isolation and return compact results.
+
+## Trigger Conditions
+
+Delegation is **mandatory** when any of these apply:
+
+| Condition | Threshold | Example |
+|-----------|-----------|---------|
+| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
+| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
+| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
+| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
+| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
+| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
+| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
+
+**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
+`curl`, `hermes tools list` — these are decision-making tools. The manager
+reads them directly.
+
+## Worker Selection Matrix
+
+| Worker | Model | Toolsets | Role | Use When |
+|--------|-------|----------|------|----------|
+| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
+| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
+| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
+| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
+| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
+| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
+
+### Selection Rules
+
+1. **Match specialty first.** A code task → `syslog-code`. An infra task →
+   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
+2. **Research tasks with browser needs → `syslog-research`.** Other workers
+   don't have the browser toolset.
+3. **Verification → `syslog-review`.** Never deliver raw worker output.
+4. **Documentation/content → `syslog-writer`.** Let them own the prose.
+5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
+   (includes browser) and high reasoning effort.
+
+## Delegation Protocol
+
+### Step 1: Decompose
+
+Break the task into lanes. Each lane does ONE thing. Workers are independent —
+no lane depends on another's output mid-flight. If lanes depend on each other,
+dispatch sequentially.
+
+### Step 2: Dispatch
+
+Fire workers via `delegate_task`:
+
+**Parallel (independent lanes):**
+```
+delegate_task(
+    tasks=[
+        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
+        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
+    ]
+)
+```
+
+**Sequential (dependent lanes):**
+Dispatch lane 1 → wait for result → dispatch lane 2.
+
+### Step 3: Verify
+
+**MANDATORY for:**
+- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
+- Code builds and modifications
+- Research findings (web data, external sources)
+- Any output that will reach the user
+
+**Fire `syslog-review` to verify:**
+```
+delegate_task(
+    goal="Review the output of the devops worker. Verify the node status
+    report is accurate, check for inconsistencies, confirm all nodes were
+    reachable.",
+    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
+    Verify against live system."
+)
+```
+
+**If verification fails:**
+1. Send work back to original worker with review feedback
+2. Re-verify
+3. Max 2 re-verify cycles before escalating to Kwame
+
+### Step 4: Deliver
+
+Only verified results reach Kwame. Format per channel:
+- Telegram: Use `telegram-formatting` skill
+- Zulip: Use Zulip Markdown (CommonMark)
+- Email: Use `syslog-email` skill
+
+## Kanban Board Protocol
+
+**File:** `~/.hermes/kanban/kanban.json`
+
+```json
+{
+  "task_id": "unique-id",
+  "title": "Task description",
+  "created": "2026-07-09T01:00:00",
+  "status": "backlog|in_progress|review|done",
+  "lanes": [
+    {
+      "lane_id": "devops-check",
+      "worker": "syslog-devops",
+      "goal": "Check all 5 Proxmox nodes",
+      "status": "dispatched|completed|failed",
+      "output_file": "/tmp/node-report.md"
+    }
+  ]
+}
+```
+
+**Update the board on every state change.**
+
+## Failure Handling
+
+### Worker Timeouts
+
+- Child timeout: **900 seconds** (15 minutes)
+- Worker model `syslog-auto` is slow — it can hit the timeout limit with
+  22+ API calls
+- **If a worker times out:** Re-dispatch with a narrower scope. Break the
+  task into smaller pieces that fit in the timeout window.
+- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
+  that compounds quickly. Prefer API-based or local approaches when possible.
+
+### Worker Selection Failures
+
+- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
+- `syslog-code` is best for code-level work (reading files, writing scripts)
+- `syslog-research` has the browser toolset — use for web research
+- `syslog-review` is the QA gate — always fire before delivery
+- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
+- **Never nest delegation** (max_spawn_depth: 1)
+
+### Context Overflow
+
+- If a task requires >10K tokens of output, delegate the processing
+- Workers return compact summaries, not raw data dumps
+- Pass file paths and concrete goals — never dump raw data into context
+
+## Anti-patterns
+
+- ❌ Reading large files into your own context before deciding → delegate the read
+- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
+- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
+- ❌ Skipping verification → raw worker output never reaches the user
+- ❌ Delegating single tool calls → keep quick reads/writes at manager level
+- ❌ Firing more than 3 workers in parallel → hard limit
+
+## Emergency Exception
+
+**In an emergency (server down, service must be restored immediately):**
+- Delegate the diagnosis (find the problem)
+- Execute the fix yourself (minimize handoff latency)
+- Verify the fix after delivery
+- Log the exception in the kanban board
+
+The emergency exception exists because the user needs the service back NOW,
+not after three worker round-trips. But it's an exception — not the rule.
+
+## What This Contract Doesn't Cover
+
+1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
+2. **SSH key management** — covered by existing SSH/Proxmox contracts
+3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
+4. **Cron job management** — covered by individual cron contracts
+5. **Infra verification** — covered by `verify-before-mutate` protocol
+
+## Verification
+
+Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
+
+```bash
+python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
+```
+
+## References
+
+- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
+- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
+- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
+- `verify-before-mutate` protocol: Infrastructure change verification
+- `hermes-config-template.prose.md`: Worker profile configuration
+
+## Success Criteria
+
+This contract succeeds when:
+
+1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
+2. **Workers do the work** — manager coordinates, doesn't execute
+3. **Verification before delivery** — all output passes through `syslog-review`
+4. **Kanban board is current** — every task has a lane, every lane has a status
+5. **User gets verified results** — raw worker output never reaches Kwame
+
+---
+
+**Last updated:** 2026-07-09
+**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
+**Status:** Draft — awaiting PR review and merge to prose-contracts main

disk-gc-threat-response.prose.md

@@ -3,10 +3,11 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
   Recurring disk health scan, garbage collection, and threat response
-  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
   2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
   Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB via `docker system prune -a --force`.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
+  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
 
 ## Scope
 
-All CTs in the Proxmox cluster, with special attention to Docker hosts:
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
+Docker hosts get special attention:
 
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
+| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All others | — | LOW — no Docker | apt clean, log rotate |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
+| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
+> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 
 ## Threat Levels
 
@@ -123,7 +130,7 @@ call summary-reporter
 
 ## GC Strategies by Host Type
 
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -289,4 +296,11 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 |------|-----|-----|--------|
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
-| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+
+### KVM VM (via direct SSH)
+| Host | IP | Role | Status |
+|------|-----|------|--------|
+| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.

gpu-fleet.prose.md

@@ -142,7 +142,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 |-------|-----|-----|-----|--------|
 | Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
 | Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
-| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
 | Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
 | Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
 | Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
@@ -159,10 +159,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
 

gpu-monitor.prose.md

@@ -19,7 +19,7 @@ agent: abiba
 
 ```
 ┌─────────────────────────────────────────────────────────┐
-│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  GPU Monitor Server (:9100) — 192.168.68.24              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
    │          │          │
@@ -35,7 +35,7 @@ agent: abiba
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
-Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
 poll .15:8080 directly; must go through router on .116.
 ```
 
@@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
 
 | File | Host | Purpose |
 |------|------|---------|
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
 
 ## Execution

infrastructure-control.prose.md

@@ -144,7 +144,7 @@ description: >
 
 ### Ecosystem A: docker-vm (192.168.68.7)
 
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
+| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
+
+**Last verified:** 2026-07-09 — 16/16 containers running healthy.
 
 ### Home Stack Services
 

infrastructure-monitoring.prose.md

@@ -7,10 +7,14 @@ description: >
   from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
   via existing /metrics Prometheus endpoint.
   
-  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  DEPLOYMENT STATUS (2026-07-09):
-  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  This contract defines the target state; proxmox-monitor is the as-built.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
+     NVIDIA sidecar exporters (.8/.110:9400) never installed.
+     Router falls back to direct GPU /health probes.
+  ⚠️ This contract is target-state aspirational — not as-built.
+     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
 

infrastructure-update.prose.md

@@ -24,10 +24,11 @@ version: 1.1.0
 
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
-4. ✅ LiteLLM health check passing
+4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
-5. ✅ Zulip server reachable
+5. ✅ LiteLLM health check passing
+6. ✅ Zulip server reachable
 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

litellm-api-keys.prose.md

@@ -8,6 +8,10 @@ description: >
   Ensures agents never use the master key directly. Rotation is event-driven,
   not calendar-driven — rotate only on compromise, personnel change, or
   periodic security hygiene (quarterly/annually).
+  
+  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
+  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
+  on CT 116. Last verified: 2026-07-09.
 ---
 
 ## Parameters
@@ -38,7 +42,8 @@ description: >
    - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
    - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
    - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
+   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
    - Return the new key
 5. **If action == "rotate"**:
    - Generate new key with same alias (LiteLLM replaces the old key)

litellm-health.prose.md

@@ -1,35 +1,50 @@
 ---
 kind: function
 name: litellm-health
+status: deprecated
+deprecated_on: 2026-07-09
+replaced_by: litellm-self-heal.prose.md
+note: >
+  Consolidated into litellm-self-heal.prose.md to eliminate duplication
+  of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within litellm-self-heal.
+  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
-  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
-  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Router (harness-router :9000) is DEPRECATED — container still runs but
-  Prometheus/Grafana and fleet dashboard.
+  is not in the request path. GPU monitoring via Prometheus/Grafana and
+  fleet dashboard (gpu-monitor :9100).
   Designed as a reusable contract for any Syslog agent.
+  
+  Source of truth: gpu-fleet.prose.md
 ---
 
-## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 
 ```
-Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
-                        │                    │
+                        │
-                  Key validation      Model routing
+                  Key validation
-                  Fallback chains     Slot booking
+                  Fallback chains
-                  Complexity router   Circuit breakers
+                  Budget tracking
-                  Budget tracking     Redis-backed
+                        │
-                        │                    │
+                  Prometheus ← metrics
-                  Prometheus ← metrics  ←───┘
                         │
                     Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
 ```
 
-**What changed (v3.1.0 → v3.2.0)**:
+**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
-- Router is BACK — runs internally on :9000 behind LiteLLM
+- Router REMOVED from request path — LiteLLM proxies directly to GPU
-- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- All GPUs at parallel 2 (was parallel 1)
-- Router :9000 is 127.0.0.1-only, not publicly accessible
+- NVIDIA context reduced 256K→128K to free VRAM
-- GPU fleet dashboard: http://192.168.68.24:9100
+- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
-- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
 
 ## Parameters
 
@@ -55,27 +70,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se
 
 ## GPU Fleet Topology
 
-| Host | IP | Hardware | Models Served | Engine |
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
-|------|-----|----------|---------------|--------|
+|------|-----|----------|---------------|--------|---------|----------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
-| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 
 ## Model Fallback Chains (LiteLLM)
 
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
-| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
-| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | — | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
 
 ## Containers on CT 116
 
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |

litellm-self-heal.prose.md

@@ -6,22 +6,92 @@ note: >
   Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
   This contract is now manual-only — triggers require explicit user request.
   Consider reimplementing as a standalone cron job or prose contract.
+  
+  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
+  duplication of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within this contract.
+  
+  Source of truth for GPU topology and keys: gpu-fleet.prose.md
+  Last verified: 2026-07-09
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
-  knowledge graph for full audit trail.
+  inference, and agent keys. Applies remediation rules for common failures.
+  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 
+# LiteLLM Operations — Health Check + Self-Heal
+
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
+                        │
+                  Key validation
+                  Fallback chains
+                  Budget tracking
+                        │
+                  Prometheus ← metrics
+                        │
+                    Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
+```
+
+## Parameters
+
+- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
+- backend_host: string — Internal CT host (default: "192.168.68.116")
+- auth_host: string — Authentik OIDC server (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
+
+## GPU Fleet Topology
+
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
+|------|-----|----------|---------------|--------|---------|----------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
+
+## Model Fallback Chains (LiteLLM)
+
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
+| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
+| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
+
 ## Maintains
 
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
-## Requires
+- litellm-agent-keys: { count: int, valid: int }
-
-- litellm-health:function
 
 ## Continuity
 
@@ -29,55 +99,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
 
-## Reporting
+---
 
-Every remediation cycle produces a structured report in three channels:
+## Health Check
 
-### 1. RA-H OS Knowledge Graph Node
+Run this first on every cycle. Results feed into remediation rules below.
-Created as `[LEARN] litellm-self-heal: <run_id>` with:
-- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
-- source: Full JSON report of the cycle
-- description: Summary of what was found, fixed, and escalated
 
-### 2. Zulip DM to Owner
+### 1. Check public endpoints
-Sent immediately for:
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
-- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
-- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
 
-### 3. Daily Digest (end of day)
+### 2. Check LiteLLM health (no-auth)
-A summary of the last 24 hours sent as a single Zulip DM:
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
-```
-📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
 
-Total cycles: 288 (every 5 min)
+### 3. Check backend container health
-Issues found: 3
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+- Critical: harness-litellm, harness-nginx, harness-postgres
-  └─ Escalated: 0
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
+- Deprecated but running: harness-router (not in path, reference only)
 
-Top actions:
+### 4. Check GPU fleet health (via fleet dashboard)
-  • harness-nginx restarted — 2026-06-26 02:15
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
-  • DOCS_URL corrected — 2026-06-26 07:30  
+- Verify GPUs reporting status "healthy"
-  • harness-nginx restarted — 2026-06-26 14:46
+- Check alerts array for active warnings/critical
 
-Uptime: 23h 47m — All healthy now ✅
+### 5. Check model inference via LiteLLM — test each model
-```
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
+- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+- Use master key for auth
 
-### 4. Weekly Digest (every Monday)
+### 6. Check agent keys
-Same format as daily but covering 7 days. Sent to both Zulip DM and
+- GET /key/list with master key → verify all 6 agents have keys
-logged as a `[REPORT]` knowledge graph node for long-term trending.
 
-### 5. Relay Message (if cross-agent)
+### 7. Check Grafana
-If a fix required another agent's help (e.g., Authentik restart), a relay
+- GET {{grafana_url}}/api/health → expect 200
-message is sent to the responsible agent with full context.
 
-## GPU Fleet
+### 8. Compile overall status
+Determine overall_status from individual check results:
+- "healthy" — all checks pass
+- "degraded" — 1-2 non-critical checks fail
+- "down" — critical checks fail
 
-| Host | IP | Hardware | Models | Engine |
+---
-|------|-----|----------|--------|--------|
-| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
-| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
-| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
 
 ## Remediation Rules
 
@@ -105,39 +169,89 @@ Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts
 
-### Rule 7: Router Returns 503 (All GPUs Saturated)
+### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
-Detect → router logs show QUEUE_TIMEOUT with GPUs idle
+Router is no longer in the request path (2026-07-08). LiteLLM proxies
-Root cause → roster not loaded at startup (empty TIER_MODELS)
+directly to GPU. This rule is retained for reference but is inactive.
-Fix → POST /admin/roster/reload (if endpoint available) or restart router
+If 503 errors occur, check LiteLLM timeouts and GPU health directly.
-Permanent fix → add load_roster() to router startup code
-Escalate → if reload doesn't resolve within 2 min
 
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
-Root cause → keys in router env var, not in LiteLLM DB
+Root cause → keys not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM
 
-### Rule 9: Stale Active Counter in Redis
+### Rule 9: Stale Active Counter in Redis — DEPRECATED
-Detect → active:<model> > 0 while GPU idle
+Router no longer in path so Redis active counters are unused. Rule retained
-Fix → `redis-cli DEL active:<model>` → restart router
+for reference but inactive. If Redis issues occur, check harness-redis container.
-Prevent → add TTL to active keys in router code
+
+---
+
+## Reporting
+
+Every remediation cycle produces a structured report:
+
+### 1. RA-H OS Knowledge Graph Node
+Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
+
+### 2. Zulip DM to Owner
+- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
+- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+
+### 3. Daily Digest (end of day)
+Summary of last 24 hours: total cycles, issues found/fixed/escalated,
+top actions, uptime.
+
+### 4. Relay Message (if cross-agent)
+If a fix requires another agent (e.g., Authentik restart), relay sent
+to responsible agent with full context.
+
+---
 
 ## Execution
 
-1. Run health check
+```prose
-2. Apply remediation for each failure
+-- Phase 1: Health Check
-3. **Generate report** with all actions taken
+let health = call health-check
-4. **Log to knowledge graph** — create `[LEARN]` node
+  public_url: public_url
-5. **Notify user** via Zulip DM if anything changed
+  backend_host: backend_host
-6. Wait 300s and repeat
+  gpu_dashboard_url: gpu_dashboard_url
+  grafana_url: grafana_url
+
+-- Phase 2: Apply remediation for each failure
+let actions = []
+for check in health.failed:
+  let fix = apply-remediation-rule
+    rule: lookup-rule(check.name)
+    target: check.target
+  push actions fix
+
+-- Phase 3: Generate report
+call report-generator
+  health: health
+  actions: actions
+
+-- Phase 4: Log to knowledge graph
+call kg-logger
+  run_id: run_id
+  health: health
+  actions: actions
+
+-- Phase 5: Notify if anything changed
+if actions.length > 0:
+  call zulip-notifier
+    actions: actions
+    health: health
+
+-- Wait 300s and repeat
+```
 
 ## Audit Trail Format
 
 ```json
 {
-  "run_id": "self-heal-20260626-001",
+  "run_id": "self-heal-20260709-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "timestamp": "2026-07-09T14:00:00Z",
   "duration_ms": 1234,
   "checks_passed": 7,
   "checks_failed": 0,
@@ -151,7 +265,7 @@ Prevent → add TTL to active keys in router code
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
   "issues_found": 1,
   "issues_fixed": 1,
   "actions": [

mumuni-delegation-prose-contract.prose.md

@@ -0,0 +1,290 @@
+---
+kind: pattern
+name: mumuni-delegation
+description: >
+  Mumuni-specific operating doctrine for task decomposition, worker
+  delegation, verification, and delivery. Defines when to delegate, which
+  worker to use for what, how to handle failures, and the kanban board
+  protocol. Enforces context-window discipline and separation of concerns.
+  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
+version: 1.0.0
+---
+
+## Maintains
+
+- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
+  `syslog-research`, `syslog-review`, `syslog-writer`)
+- Kanban board state at `~/.hermes/kanban/kanban.json`
+- Context window budget: ~65K tokens per request (131K total, 60% threshold)
+
+## Topology
+
+**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
+**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
+**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
+
+This contract is infrastructure-agnostic in terms of which nodes are used.
+Workers execute tasks on whatever infrastructure they're given — SSH to .6,
+.pm, .9, .12, or .15 depending on the task. The contract defines the
+**who** and **when** — not the **where**.
+
+## Why This Matters
+
+Without enforced delegation, the manager consumes the full iteration budget
+(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
+leaving no capacity for actual coordination. The result: context overflow
+(59K tokens in system prompt), iteration exhaustion, and degraded response
+quality. This contract exists because I blew through my budget checking
+Proxmox node status instead of delegating to `syslog-devops`.
+
+## Context Window Discipline
+
+**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
+**Volatile (MEMORY.md + USER.md): ~300 tokens.**
+**Total base: ~6,800 tokens per request.**
+
+The remaining budget is the conversation. Every tool call result adds to it.
+If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
+from multiple nodes), the context fills fast. That's why we delegate: workers
+process in isolation and return compact results.
+
+## Trigger Conditions
+
+Delegation is **mandatory** when any of these apply:
+
+| Condition | Threshold | Example |
+|-----------|-----------|---------|
+| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
+| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
+| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
+| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
+| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
+| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
+| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
+
+**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
+`curl`, `hermes tools list` — these are decision-making tools. The manager
+reads them directly.
+
+## Data Source Integrity (CRITICAL)
+
+**Workers MUST use the data provided in their task context. They MUST NOT
+fetch their own data from external sources unless explicitly told to.**
+
+When a task says "Read file X and format it", the worker reads file X. It does
+not query a separate API, run its own diagnostics, or pull data from a different
+system. This is the #1 source of cross-worker inconsistency: one worker gathers
+SSH data, another queries the Proxmox API, and the report merges two incompatible
+datasets.
+
+**Rule:** If a worker needs additional data beyond what's in its task description,
+it asks the manager (via relay) — it doesn't go find it on its own.
+
+**This is a hard rule, not a recommendation.** Violating it produces the exact
+type of discrepancy the kanban pipeline exists to prevent: a review worker finds
+"5 nodes present" in the raw data but "5/5 online" in the report — even though
+one of those nodes was unreachable. The report lied because it used data the
+raw data never provided.
+
+## Worker Selection Matrix
+
+| Worker | Model | Toolsets | Role | Use When |
+|--------|-------|----------|------|----------|
+| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
+| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
+| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
+| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
+| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
+| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
+
+### Selection Rules
+
+1. **Match specialty first.** A code task → `syslog-code`. An infra task →
+   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
+2. **Research tasks with browser needs → `syslog-research`.** Other workers
+   don't have the browser toolset.
+3. **Verification → `syslog-review`.** Never deliver raw worker output.
+4. **Documentation/content → `syslog-writer`.** Let them own the prose.
+5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
+   (includes browser) and high reasoning effort.
+
+## Delegation Protocol
+
+### Step 1: Decompose
+
+Break the task into lanes. Each lane does ONE thing. Workers are independent —
+no lane depends on another's output mid-flight. If lanes depend on each other,
+dispatch sequentially.
+
+### Step 2: Dispatch
+
+Fire workers via `delegate_task`:
+
+**Critical: Pass the data, not just the goal.** When dispatching a worker that
+processes output from another worker, include the file path AND explicit
+instructions to use ONLY that source. Example:
+
+```
+delegate_task(
+    goal="Format the cluster check into a clean report",
+    context="Source data is at /tmp/proxmox-check-raw.md. Format ONLY the data
+    in that file. Do NOT query the Proxmox API or any other data source. Use the
+    file as your sole source of truth."
+)
+```
+
+**Parallel (independent lanes):**
+```
+delegate_task(
+    tasks=[
+        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
+        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
+    ]
+)
+```
+
+**Sequential (dependent lanes):**
+Dispatch lane 1 → wait for result → dispatch lane 2.
+
+### Step 3: Verify
+
+**MANDATORY for:**
+- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
+- Code builds and modifications
+- Research findings (web data, external sources)
+- Any output that will reach the user
+
+**Fire `syslog-review` to verify:**
+```
+delegate_task(
+    goal="Review the output of the devops worker. Verify the node status
+    report is accurate, check for inconsistencies, confirm all nodes were
+    reachable.",
+    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
+    Verify against live system."
+)
+```
+
+**If verification fails:**
+1. Send work back to original worker with review feedback
+2. Re-verify
+3. Max 2 re-verify cycles before escalating to Kwame
+
+### Step 4: Deliver
+
+Only verified results reach Kwame. Format per channel:
+- Telegram: Use `telegram-formatting` skill
+- Zulip: Use Zulip Markdown (CommonMark)
+- Email: Use `syslog-email` skill
+
+## Kanban Board Protocol
+
+**File:** `~/.hermes/kanban/kanban.json`
+
+```json
+{
+  "task_id": "unique-id",
+  "title": "Task description",
+  "created": "2026-07-09T01:00:00",
+  "status": "backlog|in_progress|review|done",
+  "lanes": [
+    {
+      "lane_id": "devops-check",
+      "worker": "syslog-devops",
+      "goal": "Check all 5 Proxmox nodes",
+      "status": "dispatched|completed|failed",
+      "output_file": "/tmp/node-report.md"
+    }
+  ]
+}
+```
+
+**Update the board on every state change.**
+
+## Failure Handling
+
+### Worker Timeouts
+
+- Child timeout: **900 seconds** (15 minutes)
+- Worker model `syslog-auto` is slow — it can hit the timeout limit with
+  22+ API calls
+- **If a worker times out:** Re-dispatch with a narrower scope. Break the
+  task into smaller pieces that fit in the timeout window.
+- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
+  that compounds quickly. Prefer API-based or local approaches when possible.
+
+### Worker Selection Failures
+
+- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
+- `syslog-code` is best for code-level work (reading files, writing scripts)
+- `syslog-research` has the browser toolset — use for web research
+- `syslog-review` is the QA gate — always fire before delivery
+- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
+- **Never nest delegation** (max_spawn_depth: 1)
+
+### Context Overflow
+
+- If a task requires >10K tokens of output, delegate the processing
+- Workers return compact summaries, not raw data dumps
+- Pass file paths and concrete goals — never dump raw data into context
+
+## Anti-patterns
+
+- ❌ Reading large files into your own context before deciding → delegate the read
+- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
+- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
+- ❌ Skipping verification → raw worker output never reaches the user
+- ❌ Delegating single tool calls → keep quick reads/writes at manager level
+- ❌ Firing more than 3 workers in parallel → hard limit
+- ❌ **Workers fetching their own data sources** → a writer worker that queries the Proxmox API when told to "format the raw file" is fabricating data. Use the input given, not external sources
+
+## Emergency Exception
+
+**In an emergency (server down, service must be restored immediately):**
+- Delegate the diagnosis (find the problem)
+- Execute the fix yourself (minimize handoff latency)
+- Verify the fix after delivery
+- Log the exception in the kanban board
+
+The emergency exception exists because the user needs the service back NOW,
+not after three worker round-trips. But it's an exception — not the rule.
+
+## What This Contract Doesn't Cover
+
+1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
+2. **SSH key management** — covered by existing SSH/Proxmox contracts
+3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
+4. **Cron job management** — covered by individual cron contracts
+5. **Infra verification** — covered by `verify-before-mutate` protocol
+
+## Verification
+
+Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
+
+```bash
+python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
+```
+
+## References
+
+- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
+- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
+- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
+- `verify-before-mutate` protocol: Infrastructure change verification
+- `hermes-config-template.prose.md`: Worker profile configuration
+
+## Success Criteria
+
+This contract succeeds when:
+
+1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
+2. **Workers do the work** — manager coordinates, doesn't execute
+3. **Verification before delivery** — all output passes through `syslog-review`
+4. **Kanban board is current** — every task has a lane, every lane has a status
+5. **User gets verified results** — raw worker output never reaches Kwame
+
+---
+
+**Last updated:** 2026-07-09
+**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
+**Status:** Draft — awaiting PR review and merge to prose-contracts main

pm2-self-heal.prose.md

@@ -12,6 +12,7 @@ description: >
 
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
+- gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
 

ra-h-os-custodianship-contract.prose.md

@@ -0,0 +1,210 @@
+---
+kind: pattern
+name: ra-h-os-custodianship-contract
+description: >
+  Operational standards for maintaining RA-H OS as a high-functioning shared
+  memory system across all agents. Defines rules for embedding consistency,
+  namespace discipline, staleness management, orphan prevention, agent
+  custodianship, and recovery protocols. Prevents knowledge graph degradation
+  and ensures reliable semantic search capabilities.
+---
+
+# RA-H OS Custodianship Contract — Shared Memory Protocol
+
+## Purpose
+This contract establishes the operational standards for maintaining RA-H OS as a high-functioning shared memory system across all agents. It defines the rules, responsibilities, and recovery protocols that prevent knowledge graph degradation and ensure consistent, reliable semantic search capabilities.
+
+## Core Principles
+
+### 1. Embedding Consistency
+**Rule:** All nodes MUST use the RA-H OS built-in embedding pipeline via `http://192.168.68.65:8080/v1/embeddings` for consistent vector representation.
+
+**Requirements:**
+- The embedding service at `192.168.68.65:8080` is the single source of truth for vector generation
+- No external embedding APIs (OpenAI, Cohere, etc.) may be used for RA-H OS nodes
+- All nodes must have `embedding_status: "chunked"` and `chunk_status: "chunked"` upon creation
+- If embedding fails, the node must be immediately flagged as `state: "unsearchable"` and logged
+
+### 2. Namespace Discipline
+**Rule:** All nodes MUST be categorized into one of three namespaces with strict metadata requirements.
+
+**Namespace Structure:**
+- `agent-private`: Isolated working notes for individual agents (Mumuni, Tanko, Okyeame, etc.)
+- `shared`: Policy files, registry nodes, collective knowledge
+- `syslogsolution`: Business nodes, client data, operational context
+
+**Metadata Requirements:**
+Every node MUST have these metadata fields:
+```json
+{
+  "agent_id": "<agent_name>",
+  "namespace": "<namespace>",
+  "state": "<state>",
+  "type": "<type>",
+  "owner": "<owner>",
+  "tenant": "<tenant>",
+  "visibility": "<visibility>",
+  "source": "<source>",
+  "captured_by": "<captured_by>"
+}
+```
+
+### 3. Staleness Management
+**Rule:** Nodes are categorized by type and have specific staleness thresholds.
+
+**Staleness Thresholds:**
+- `policy` type: 30 days (e.g., Shared Memory Policy, Agent Registry)
+- `registry` type: 30 days (e.g., Node State Management, Health Dashboard)
+- `template` type: 30 days (e.g., Agent SOUL.md)
+- `note` type: 60 days (e.g., business context, project notes)
+- `information` type: 90 days (e.g., reference documentation)
+- `idea` type: 30 days (e.g., brainstorming, experimental notes)
+
+**Transition Protocol:**
+- `active` → `stale`: After days since `updated_at` exceeds threshold
+- `stale` → `review_pending`: Daily cron check flags for human review
+- `review_pending` → `active`: Human reviews and updates content
+- `review_pending` → `archived`: After 15 days in review_pending state (automatic purge)
+
+### 4. Orphan Prevention
+**Rule:** No node should remain orphaned (0 edges) for more than 7 days.
+
+**Orphan Detection:**
+- Daily cron job identifies nodes with 0 incoming/outgoing edges
+- Orphans are flagged with `state: "review_pending"` and `metadata.orphan_detected: true`
+- 7-day grace period: Orphans must be either:
+  - Connected to relevant nodes via edges
+  - Merged into existing parent nodes
+  - Archived after 15 days in review_pending state
+
+### 5. Agent Custodianship
+**Rule:** Each agent is responsible for maintaining their own created nodes.
+
+**Responsibilities:**
+- **Mumuni**: Primary custodian for `syslogsolution` namespace, health dashboard, and all business nodes
+- **Tanko**: Custodian for `agent-private` namespace, personal notes, and fitness/creative work
+- **Okyeame**: Custodian for `shared` namespace, policy files, and collective knowledge nodes
+- **All Agents**: Must verify embedding status before reporting node creation as complete
+
+### 6. Embedding Failure Recovery
+**Rule:** When embedding pipeline is unavailable, nodes are blocked from creation or immediately flagged.
+
+**Recovery Protocol:**
+1. **Detection**: Daily health check monitors `http://192.168.68.65:8080/health` endpoint
+2. **Alert**: If embedding service is down, alert is sent to all agents via relay
+3. **Mitigation**: 
+   - New node creation is suspended
+   - Existing nodes with `chunk_status: "not_chunked"` are flagged as `unsearchable`
+4. **Recovery**: When service returns:
+   - All `not_chunked` nodes are queued for re-embedding
+   - Re-embedding status logged in `embedding_retry_log` table
+   - 5-minute retry window with exponential backoff (1s, 2s, 4s, 8s, 16s)
+
+### 7. Chunking Verification
+**Rule:** All node modifications must verify chunking status within 5 minutes.
+
+**Verification Protocol:**
+- After `updateNode` or `createNode`, query `chunks` table for `node_id`
+- If `chunk_status != "chunked"` after 5 minutes, log failure and flag node
+- Automated cleanup script runs every 6 hours to retry failed chunks
+- Maximum 3 retry attempts per node before escalation to human
+
+### 8. Graph Health Monitoring
+**Rule:** Health dashboard (Node #74) must reflect real-time graph status.
+
+**Dashboard Requirements:**
+- Total nodes, edges, orphans, stale nodes, chunk errors
+- Embedding service health status
+- Last successful embedding attempt
+- Alert thresholds:
+  - Orphan rate > 40%: Warning
+  - Stale node rate > 50%: Critical
+  - Chunk error rate > 30%: Critical
+  - Embedding service down: Critical (immediate alert)
+
+## Operational Procedures
+
+### Daily Maintenance (Automated)
+1. **Staleness Check**: Identify nodes past their threshold
+2. **Orphan Detection**: Flag nodes with 0 edges
+3. **Chunk Verification**: Retry failed chunks, log errors
+4. **Health Update**: Refresh Node #74 dashboard
+
+### Weekly Maintenance (Human Review)
+1. **Review Pending Nodes**: Examine flagged nodes
+2. **Edge Optimization**: Connect related orphaned nodes
+3. **Archive Cleanup**: Remove nodes past 15-day review period
+4. **Embedding Audit**: Verify all nodes have valid embeddings
+
+### Emergency Recovery
+1. **Embedding Service Down**: 
+   - Suspend node creation
+   - Alert all agents
+   - Investigate root cause (check `192.168.68.65:8080`)
+2. **Database Corruption**:
+   - Restore from latest PBS backup
+   - Verify chunk integrity
+   - Re-run embedding for affected nodes
+3. **Mass Orphan Creation**:
+   - Identify source agent/namespace
+   - Review recent changes
+   - Reconnect or archive affected nodes
+
+## Compliance & Enforcement
+
+### Audit Schedule
+- **Daily**: Automated health checks
+- **Weekly**: Human review of review_pending nodes
+- **Monthly**: Comprehensive graph audit (full schema validation)
+- **Quarterly**: Contract review and threshold adjustment
+
+### Violation Consequences
+- **First**: Alert to responsible agent
+- **Second**: Node placed in `review_pending` state
+- **Third**: Agent access suspended until remediation
+- **Fourth**: Escalation to human (Kwame)
+
+### Metrics for Success
+- Orphan rate: < 10%
+- Stale node rate: < 20%
+- Chunk error rate: < 5%
+- Embedding service uptime: > 99.9%
+- Node creation verification: 100%
+
+## Implementation Notes
+
+### Database Schema Requirements
+- `nodes` table: Standard fields plus `embedding_status`, `chunk_status`, `last_embedding_attempt`
+- `chunks` table: Must track `chunk_status` and `embedding_status`
+- `embedding_retry_log` table: Track retry attempts for failed chunks
+- `node_state_history` table: Log state transitions for audit trail
+
+### Cron Job Requirements
+- `ra-h-health-check`: Daily (4h interval)
+- `ra-h-staleness-detection`: Daily
+- `ra-h-orphan-detection`: Daily
+- `ra-h-chunk-retry`: Every 6 hours
+- `ra-h-review-cleanup`: Weekly (15-day archive)
+
+### Agent Onboarding
+All new agents must:
+1. Read this contract
+2. Understand namespace responsibilities
+3. Know how to verify embedding status
+4. Know the alert escalation path
+
+---
+
+## Approval
+This contract is effective immediately upon approval by the primary custodian (Mumuni) and system owner (Kwame).
+
+**Effective Date:** July 10, 2026  
+**Review Date:** October 10, 2026  
+**Primary Custodian:** Mumuni 🦅  
+**System Owner:** Jerome Tabiri
+
+---
+
+*Contract Version: 1.0*  
+*Last Updated: July 10, 2026*  
+*Storage: `/root/.hermes/skills/ra-h-os-custodianship-contract.prose.md`*

scripts/pct-run.sh

@@ -30,6 +30,12 @@ declare -A CT_NODES=(
     # acerpve (192.168.68.9)
     [102]=acerpve   # adguard
     # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
+    #
+    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
+    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
+    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
+    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
+    #   118 jitsi     → stopped, not in service
 )
 
 # Each node must be root-accessible via SSH hostname