fix: add data source integrity rule to delegation contract

Workers MUST use provided input data, not fetch external sources.
Adds: Data Source Integrity section, dispatch guidance example,
anti-pattern entry. Root cause of e2e test discrepancies where
writer worker queried Proxmox API instead of using raw data.

.gitea/workflows/pr-pipeline.yaml

@@ -0,0 +1,106 @@
+name: PR Pipeline — Authorize → Validate → Review → Merge
+on:
+  push:
+    branches: [master]
+    paths:
+      - '**.prose.md'
+      - 'scripts/**.sh'
+      - '**.yaml'
+      - '**.yml'
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths:
+      - '**.prose.md'
+      - 'scripts/**.sh'
+      - '**.yaml'
+      - '**.yml'
+
+jobs:
+  auth:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: Authorization check
+        run: bash scripts/prose-auth-check.sh
+
+  validate:
+    runs-on: ubuntu-latest
+    needs: auth
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: YAML frontmatter validation
+        run: |
+          echo "=== Prose Contract Frontmatter Validation ==="
+          FAILED=0
+          for f in $(find . -name "*.prose.md" -not -path "./.git/*" -not -path "./runs/*"); do
+            FM=$(sed -n '/^---$/,/^---$/p' "$f" | sed '1d;$d')
+            [ -z "$FM" ] && { echo "  ❌ $f: No YAML frontmatter"; FAILED=$((FAILED+1)); continue; }
+
+            KIND=$(echo "$FM" | grep '^kind:' | awk '{print $2}')
+            case "$KIND" in
+              function|responsibility|gateway|pattern|test|template|architecture|enforcement) echo "  ✅ $f: kind=$KIND" ;;
+              *) echo "  ❌ $f: Invalid kind='$KIND'"; FAILED=$((FAILED+1)) ;;
+            esac
+
+            echo "$FM" | grep -q '^name:' || { echo "  ❌ $f: Missing name"; FAILED=$((FAILED+1)); }
+            echo "$FM" | grep -q '^description:' || { echo "  ❌ $f: Missing description"; FAILED=$((FAILED+1)); }
+          done
+          [ $FAILED -gt 0 ] && { echo "❌ FRONTMATTER FAILED ($FAILED error(s))"; exit 1; }
+          echo "✅ Frontmatter validation passed"
+
+  lint:
+    runs-on: ubuntu-latest
+    needs: validate
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: Structure + regression + consistency lint
+        run: bash scripts/prose-lint.sh
+
+  ai-review:
+    runs-on: ubuntu-latest
+    needs: validate
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: AI-powered contract review
+        env:
+          LITELLM_URL: ${{ secrets.LITELLM_URL }}
+          LITELLM_KEY: ${{ secrets.LITELLM_KEY }}
+        run: bash scripts/prose-ai-review.sh
+
+  gate:
+    runs-on: ubuntu-latest
+    needs: [auth, validate, lint, ai-review]
+    if: success()
+    steps:
+      - name: Merge gate
+        run: |
+          echo "╔══════════════════════════════════════╗"
+          echo "║  ALL CHECKS PASSED — SAFE TO MERGE  ║"
+          echo "╚══════════════════════════════════════╝"
+          echo ""
+          echo "  ✅ auth       — authorized agent"
+          echo "  ✅ validate   — frontmatter valid"
+          echo "  ✅ lint       — structure + no regressions"
+          echo "  ✅ ai-review  — no contradictions with ground truth"
+          echo ""
+          echo "Merge this PR to deploy to main."

AGENTS.md

@@ -0,0 +1,130 @@
+# AGENTS.md — Prose Contracts Repo
+
+All agents operating on prose contracts MUST follow this workflow. No exceptions.
+
+## The Rule
+
+**No agent pushes directly to `main`. All changes go through PRs with automated validation.**
+
+## Why
+
+Two incidents taught us this:
+
+1. **The .117→.19 IP fix** — A stale IP in a contract was "fixed" without verification, breaking Zulip. The fix was right but the approach was wrong. Automated consistency checks would have caught it.
+
+2. **The /grafana/ route regression** — An nginx route that was deliberately removed (Jul 2) was re-added to a contract diagram. CI linting now catches this automatically.
+
+## Agent Workflow
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  1. Clone repo → create branch → make changes           │
+│  2. Push branch → open PR                                │
+│  3. CI pipeline runs automatically:                      │
+│     ✅ validate: frontmatter (kind, name, description)   │
+│     ✅ lint: structure + regression detection            │
+│     ✅ ai-review: LiteLLM reviews diff vs ground truth   │
+│  4. All green → merge PR to main                         │
+│  5. Agents run contracts from main                       │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Branch Protection (enforced by Gitea)
+
+| Rule | Enforcement |
+|------|------------|
+| Direct push to `main` | ❌ Blocked (except abiba-bot for emergencies) |
+| PR merge without passing CI | ❌ Blocked — all 3 checks must be green |
+| Status check contexts | `pr-pipeline / validate`, `pr-pipeline / lint`, `pr-pipeline / ai-review` |
+
+## What the CI checks for
+
+### Stage 1 — Validate
+- YAML frontmatter is valid (proper `---` delimiters)
+- `kind` field is one of: `function`, `responsibility`, `gateway`, `pattern`, `test`, `template`
+- `name` and `description` fields are present
+
+### Stage 2 — Lint
+- Responsibility contracts have `## Maintains`
+- Function contracts have `## Parameters` and `## Returns`
+- **Regression rules (automatic rejection):**
+  - `/grafana/` nginx route — was reverted Jul 2, must not reappear
+  - `CT 122` or `CT 123` as CT ID labels — don't exist in the cluster
+  - These rules are hardcoded in `scripts/prose-lint.sh`
+
+### Stage 3 — AI Review
+- Diff is sent to `syslog-auto` model via LiteLLM
+- Review checks against infrastructure-control ground truth:
+  - CT IDs match PVE cluster (100-117, no 122/123)
+  - Grafana is direct LAN :3001, NOT behind nginx
+  - Zulip is CT 117 on storepve (bridge IP .19)
+  - Strix Halo :8080 is firewalled to .116 only
+- Review is posted to the PR
+
+## Authorization
+
+### Who can change what
+
+| Contract | Sensitivity | Who can change |
+|----------|------------|----------------|
+| `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba only (after live verification) |
+| `proxmox-monitor.prose.md` | **CRITICAL** — deployed monitoring | Abiba only |
+| `hermes-config-template.prose.md` | **HIGH** — all agent configs | Abiba, Mumuni, Tanko |
+| `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni |
+| Other contracts | Normal | Any registered agent |
+| `scripts/*.sh` | **HIGH** — runtime scripts | Abiba only |
+
+### Enforcement (self-policing)
+
+The CI does not block based on author identity today (Gitea doesn't support CODEOWNERS natively). Instead, agents self-enforce:
+
+1. Before changing a CRITICAL contract, check with Abiba
+2. If you're unsure, tag `@abiba-bot` in the PR description
+3. Abiba reviews CRITICAL contract changes before merge
+
+## Emergency Bypass
+
+In an emergency (service down, fix must ship immediately):
+1. Push to a branch
+2. Open PR with `[EMERGENCY]` in the title
+3. CI still runs — but if it fails and the fix is verified, abiba-bot can bypass and push directly to main
+4. Post-incident: open a follow-up PR to fix any CI violations
+
+## Quick Start
+
+```bash
+# Clone
+git clone https://git.sysloggh.net/SyslogSolution/prose-contracts.git
+cd prose-contracts
+
+# Create branch
+git checkout -b fix/my-change
+
+# Make changes, test locally
+bash scripts/prose-lint.sh     # run lint locally before pushing
+
+# Push and open PR
+git add -A
+git commit -m "fix: description of change"
+git push origin fix/my-change
+
+# Open PR at https://git.sysloggh.net/SyslogSolution/prose-contracts/pulls
+# CI runs automatically — wait for all checks to pass
+# Merge when green
+```
+
+## Verification Before Acting
+
+**Contracts are leads, not facts.** Live-state fields (IPs, ports, credentials) drift.
+Before acting on any value from a contract, verify it against the live system.
+
+Follow the `verify-before-mutate` protocol:
+```bash
+safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
+```
+
+## Writing New Contracts
+
+Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
+It covers the full process: verify → draft → lint → review → ship, with templates
+and style rules.

README.md

@@ -1,6 +1,70 @@
 # Prose Contracts — Syslog Solution LLC
 
-Operating contracts for Hermes agents. Each `.prose.md` file defines a **responsibility** (recurring duty) or **template** (scaffold) that agents can execute via OpenProse or follow manually.
+Operating contracts for Syslog agents. Each `.prose.md` file defines a **function** (callable helper), **responsibility** (recurring duty with state), or **pattern** (instantiable knowledge/template) that agents can execute via OpenProse or follow manually.
+
+## ⚠️ Operating Doctrine — Read Before Acting
+
+Two rules govern how agents interact with this repo and the infrastructure it
+describes. Internalize both before running or relying on any contract.
+
+### 1. Verify-before-mutate (the `.117` rule)
+
+**Contracts are leads to investigate, NOT facts to act on.** Live-state fields
+(IPs, ports, hostnames, credentials, container names, PIDs) drift. Before any
+mutating operation on infrastructure, verify the current state against the live
+system. On drift, halt and ask the user which value is correct — do NOT "fix"
+the live state to match a stale contract.
+
+Origin: a contract stated Zulip CT 117 was at `.117`; the live IP was `.19`.
+An agent ran `pct set` without checking `pct config` first and changed the IP
+to the wrong value, breaking Zulip. The staleness was harmless until acted on.
+
+**Layer 3 enforcement — the `safe-mutate` wrapper** (deployed on all 5 agents:
+Abiba, Tanko, Mumuni, Koby, Koonimo). ALL infrastructure mutations MUST go
+through `safe-mutate`. Raw `sed -i`, `pct set`, `docker compose up
+--force-recreate`, `kill`, `rm` on infrastructure outside `safe-mutate` is an
+auditable protocol violation. The wrapper runs a verify command, optionally
+checks an `--expect` pattern, refuses on mismatch, and logs every call to
+`/root/.safe-mutate/audit.log`.
+
+```
+safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
+safe-mutate --verify "CMD" --dry-run --mutate "CMD"      # verify + show, no execute
+safe-mutate --verify "CMD" --verify-only                   # record a verification only
+```
+
+Load the `verify-before-mutate` skill (local pi skill + RA-H shared) for the
+full protocol. See knowledge graph node #604 for the post-mortem.
+
+### 2. Field trust levels
+
+Not all contract content carries the same trust:
+
+| Field type | Trust level | Examples |
+|-----------|-------------|----------|
+| **Policy** | Authoritative — trust the contract | "hardcoded harness keys forbidden", "use api_key_env" |
+| **Live state** | **Never trust — always verify** | IPs, ports, hostnames, credentials, container names, PIDs |
+| **Procedure** | Trust but adapt | command patterns are right, values may be stale |
+
+Contracts label live-state fields with `VERIFY-BEFORE-USE`. Treat any such
+field as a hint to confirm against the live system, not a fact to apply.
+
+### 3. Verify-before-fix (contract-native)
+
+Never `prose run` a fix-contract directly. Run its verifier first, read the
+verdict, lint the fixer, then run:
+
+```
+prose run <verifier>           # e.g. zulip-platform-verification → verdict
+prose lint <fixer>             # validate structure without executing
+prose preflight <fixer>        # check deps/env, no execute
+prose run <fixer>              # only after verdict + lint pass
+```
+
+`kind: responsibility` contracts VERIFY and return a verdict without making
+changes. `kind: function` contracts DO things. Forme `### Requires` →
+`### Maintains` wiring enforces this at the DAG level: a fixer that requires a
+fresh verdict cannot fire until the verifier has run.
 
 ## How Agents Run These Contracts
 
@@ -40,49 +104,95 @@ curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/mast
 
 ## Available Contracts
 
-### Responsibilities (Recurring Duties)
+### Responsibilities (Recurring Duties — `kind: responsibility`)
 
-| Contract | What It Does | When To Run |
+Run on trigger or schedule. Maintain persistent world-model state across runs.
+
+| Contract | Domain | Description |
 |---|---|---|
-| `memory-audit-maintenance` | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs, frees up char budget. | Memory >85% usage or user request: "memory audit" |
-| `zulip-health` | Checks Zulip connectivity, message flow, and bot responsiveness. | On Zulip issues or periodic health check |
-| `litellm-health` | Verifies LiteLLM proxy connectivity, model availability, and key rotation. | On inference failures or periodic check |
-| `litellm-self-heal` | Attempts to fix common LiteLLM issues (key rotation, restart, config reload). | When litellm-health reports failures |
-| `zulip-mention-reliability` | Diagnoses and fixes @mention detection issues in Zulip. | On missed @mention reports |
-| `pm2-self-heal` | Restarts crashed PM2 processes and verifies recovery. | On PM2 process failure |
+| `memory-audit-maintenance` | Memory | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs. |
+| `build-zulip-plugin` | Zulip | Generates and iteratively improves a Hermes Zulip platform plugin. Each run produces a new version. |
+| `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
+| `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
+| `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
+| `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
+| `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
+| `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
+| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
+| `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
 
-### Templates (Scaffolds)
+### Instantiable Templates (`kind: pattern` — run with `prose run`)
 
-| Contract | What It Does | Parameters |
+Generate configs or perform a transformation on demand. No persistent state.
+
+| Contract | Description | Key Parameters |
 |---|---|---|
-| `hermes-config-template` | Generates a standard Hermes config for any agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) while keeping model choices flexible. | `agent_name` (required), `default_model`, `default_provider`, `fallback_model`, `auxiliary_model` |
+| `hermes-config-template` | Standard Hermes config for any Syslog agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) + standardized `api_key_env` key indirection. | `agent_name` (required), `default_model`, `auxiliary_model` |
+| `hermes-key-enforcement` | **Enforcement contract** — all harness/LiteLLM providers MUST use `api_key_env`, never hardcoded keys. Includes detection query, rotation procedure, violation response. External providers (DeepSeek, OpenAI) exempt. | (none — doctrine reference) |
 
-### Scripts
+### Reference Documents (`kind: pattern` — read for context, not run)
 
-| File | What It Does |
+Encode topology, architectural decisions, and lessons. You read them before planning; you don't `prose run` them.
+
+| Contract | Description |
 |---|---|
-| `pm2-self-heal.sh` | Shell script to restart crashed PM2 processes (companion to the prose contract) |
+| `infrastructure-control` | Full topology and control pattern: 5-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
+| `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
+| `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
+
+### Functions (`kind: function` — callable helpers, no state)
+
+Called on-demand as single-render tools.
+
+| Contract | Description |
+|---|---|
+| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
+| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
+| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
+| `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
+| `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
+
+### Scripts (`scripts/`)
+
+Companion shell scripts that contracts delegate to.
+
+| File | Description |
+|---|---|
+| `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 
 ## Contract Structure
 
-Every `.prose.md` file has the same structure:
+Kinds determine the section set.
 
+**`kind: responsibility`** (recurring duty with persistent state):
+```
+## Maintains      — World-model state this contract tracks
+## Parameters     — Tunable inputs (with defaults)
+## Requires       — External dependencies and preconditions
+## Continuity     — When to run (triggers & cadence)
+## Remembers      — What to learn between runs
+## Invariants     — Inviolable rules for execution
+## Execution      — Step-by-step instructions
+## Example Output — What a good run looks like
 ```
----
-kind: <responsibility | template>
-name: <unique-name>
-description: ...
----
 
-## Maintains        — What state the contract tracks
-## Parameters       — Tunable inputs (with defaults)
-## Continuity       — When to run (triggers & cadence)
-## Success Criteria — How to know it worked
-## Remembers        — What to learn between runs
-## <Type> Rules     — Inviolable rules for execution
-## Execution        — Step-by-step instructions
-## Example Output   — What a good run looks like
+**`kind: function`** (stateless callable helper):
 ```
+## Parameters    — Tunable inputs (with defaults)
+## Returns       — Output schema
+## Requires      — External dependencies
+## Execution     — Step-by-step instructions
+```
+
+**`kind: pattern`** (instantiable knowledge):
+```
+Variable — templates use Parameters + Execution; reference docs are prose-driven.
+```
+
+**`kind: gateway`** / **`kind: test`** — not currently used in this repo.
 
 ## Adding New Contracts
 

build-zulip-plugin.prose.md

@@ -14,6 +14,40 @@ description: >
 - generation_count: number — How many generations have been produced
 - last_generation: timestamp — When the last generation was created
 - known_issues: array — Issues discovered in the current version
+  
+  **Postconditions** (the plugin is GOOD when ALL pass):
+  
+  *Connectivity*
+  - connected == true — Establishes and maintains Zulip event queue
+  - reconnect_on_failure == true — Reconnects after queue expiry
+  - recovers_from_network_loss == true — Handles temporary network drops
+  - stream_subscription_check == true — Bot is subscribed to primary streams (Gen 5+)
+  
+  *Message Flow*
+  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
+  - dm_response_time_offline_verifiable == true — simulate_dm_latency() passes without live Zulip (Gen 6+)
+  - stream_mention_detection == true — @mentions in streams are detected and routed
+  - all_bots_detection == true — @all-bots mentions are detected
+  - echo_loop_prevention == true — Never replies to its own messages
+  
+  *Code Quality*
+  - syntax_check == "pass" — All Python files are valid
+  - has_register_function == true — Exposes `register(ctx)` entry point
+  - plugin_yaml_valid == true — plugin.yaml parses correctly
+  - follows_base_adapter_pattern == true — Extends BasePlatformAdapter
+  
+  *Reliability*
+  - graceful_disconnect == true — shutdown doesn't crash
+  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
+  - typing_indicators_work == true — Sends typing notifications
+  - known_issues_tracked == true — add_known_issue() persists problems across generations (Gen 6+)
+
+## Status
+
+**Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
+for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
+Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
+plugin system, which is unaffected.
 
 ## Parameters
 
@@ -26,45 +60,19 @@ description: >
 
 The improvement cycle is **event-driven**, not just cron-based:
 
-- **On check failure**: If any Success Criteria fails, wake immediately to fix
+- **On check failure**: If any postcondition fails, wake immediately to fix
 - **On user request**: `prose run build-zulip-plugin` — explicit upgrade
 - **On new pattern**: If another plugin or agent reports a better pattern, wake to assimilate
 - **Retrospective**: Every 24 hours if no other trigger fired — scan for subtle degradation
 - **On first deploy**: Always run Generation 1 when no plugin exists yet
 
-## Success Criteria (What Makes a Plugin "Good")
-
-The plugin is considered GOOD when ALL of these pass:
-
-### Connectivity
-- plugin.connected == true — Establishes and maintains Zulip event queue
-- plugin.reconnect_on_failure == true — Reconnects after queue expiry
-- plugin.recovers_from_network_loss == true — Handles temporary network drops
-
-### Message Flow
-- dm_response_time_ms < 10000 — DMs get a response within 10 seconds
-- stream_mention_detection == true — @mentions in streams are detected and routed
-- all_bots_detection == true — @all-bots mentions are detected
-- echo_loop_prevention == true — Never replies to its own messages
-
-### Code Quality
-- syntax_check == "pass" — All Python files are valid
-- has_register_function == true — Exposes `register(ctx)` entry point
-- plugin_yaml_valid == true — plugin.yaml parses correctly
-- follows_base_adapter_pattern == true — Extends BasePlatformAdapter
-
-### Reliability
-- graceful_disconnect == true — shutdown doesn't crash
-- handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
-- typing_indicators_work == true — Sends typing notifications
-
 ## Remembers
 
 Each generation stores:
 - What worked well (success patterns)
 - What broke (failure modes)
 - What the user complained about (pain points)
-- Which Success Criteria passed and failed
+- Which postconditions passed and failed
 
 ## Generation Rules
 
@@ -92,13 +100,28 @@ Each generation stores:
 
 ## Execution
 
-1. **Read state** — Check previous generations from knowledge graph
+1. **Read state** — Check previous generations from knowledge graph node `build-zulip-plugin:state`
 2. **Generate or improve** — Based on generation count:
    - Generation 1: Scaffold full plugin from scratch
    - Generation N: Read previous code, apply improvements
 3. **Validate** — Syntax check, structure check, dependency check
-4. **Log** — Save generation result to knowledge graph as [LEARN]
-5. **Report** — Output what was generated, what changed, what needs review
+4. **Run offline tests** — Always run `simulate_dm_latency()` and selftest() if no live Zulip
+5. **Log** — Save generation result to knowledge graph as [LEARN] node. Update the
+   `build-zulip-plugin:state` node with current generation_count, plugin_version,
+   and known_issues.
+6. **Report** — Output what was generated, what changed, what needs review
+
+## State Persistence (Gen 6+)
+
+All generation state is stored in a knowledge graph node titled `build-zulip-plugin:state`:
+- `generation_count` — incremented on each verified run
+- `plugin_version` — current version string
+- `last_generation` — ISO timestamp of most recent run
+- `known_issues` — array of unresolved issues carried forward
+- `last_postcondition_results` — pass/fail map from most recent verification
+
+Agents executing this contract MUST read this node on startup and update it after
+validation. If the node does not exist, create it with generation_count=1.
 
 ## Example Output
 

delegation-prose-contract.prose.md

@@ -0,0 +1,256 @@
+---
+kind: pattern
+name: delegation-prose-contract
+description: >
+  Manager (Mumuni) operating doctrine for task decomposition, worker
+  delegation, verification, and delivery. Defines when to delegate, which
+  worker to use for what, how to handle failures, and the kanban board
+  protocol. Enforces context-window discipline and separation of concerns.
+  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
+version: 1.0.0
+---
+
+## Maintains
+
+- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
+  `syslog-research`, `syslog-review`, `syslog-writer`)
+- Kanban board state at `~/.hermes/kanban/kanban.json`
+- Context window budget: ~65K tokens per request (131K total, 60% threshold)
+
+## Topology
+
+**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
+**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
+**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
+
+This contract is infrastructure-agnostic in terms of which nodes are used.
+Workers execute tasks on whatever infrastructure they're given — SSH to .6,
+.pm, .9, .12, or .15 depending on the task. The contract defines the
+**who** and **when** — not the **where**.
+
+## Why This Matters
+
+Without enforced delegation, the manager consumes the full iteration budget
+(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
+leaving no capacity for actual coordination. The result: context overflow
+(59K tokens in system prompt), iteration exhaustion, and degraded response
+quality. This contract exists because I blew through my budget checking
+Proxmox node status instead of delegating to `syslog-devops`.
+
+## Context Window Discipline
+
+**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
+**Volatile (MEMORY.md + USER.md): ~300 tokens.**
+**Total base: ~6,800 tokens per request.**
+
+The remaining budget is the conversation. Every tool call result adds to it.
+If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
+from multiple nodes), the context fills fast. That's why we delegate: workers
+process in isolation and return compact results.
+
+## Trigger Conditions
+
+Delegation is **mandatory** when any of these apply:
+
+| Condition | Threshold | Example |
+|-----------|-----------|---------|
+| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
+| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
+| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
+| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
+| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
+| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
+| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
+
+**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
+`curl`, `hermes tools list` — these are decision-making tools. The manager
+reads them directly.
+
+## Worker Selection Matrix
+
+| Worker | Model | Toolsets | Role | Use When |
+|--------|-------|----------|------|----------|
+| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
+| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
+| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
+| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
+| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
+| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
+
+### Selection Rules
+
+1. **Match specialty first.** A code task → `syslog-code`. An infra task →
+   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
+2. **Research tasks with browser needs → `syslog-research`.** Other workers
+   don't have the browser toolset.
+3. **Verification → `syslog-review`.** Never deliver raw worker output.
+4. **Documentation/content → `syslog-writer`.** Let them own the prose.
+5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
+   (includes browser) and high reasoning effort.
+
+## Delegation Protocol
+
+### Step 1: Decompose
+
+Break the task into lanes. Each lane does ONE thing. Workers are independent —
+no lane depends on another's output mid-flight. If lanes depend on each other,
+dispatch sequentially.
+
+### Step 2: Dispatch
+
+Fire workers via `delegate_task`:
+
+**Parallel (independent lanes):**
+```
+delegate_task(
+    tasks=[
+        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
+        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
+    ]
+)
+```
+
+**Sequential (dependent lanes):**
+Dispatch lane 1 → wait for result → dispatch lane 2.
+
+### Step 3: Verify
+
+**MANDATORY for:**
+- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
+- Code builds and modifications
+- Research findings (web data, external sources)
+- Any output that will reach the user
+
+**Fire `syslog-review` to verify:**
+```
+delegate_task(
+    goal="Review the output of the devops worker. Verify the node status
+    report is accurate, check for inconsistencies, confirm all nodes were
+    reachable.",
+    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
+    Verify against live system."
+)
+```
+
+**If verification fails:**
+1. Send work back to original worker with review feedback
+2. Re-verify
+3. Max 2 re-verify cycles before escalating to Kwame
+
+### Step 4: Deliver
+
+Only verified results reach Kwame. Format per channel:
+- Telegram: Use `telegram-formatting` skill
+- Zulip: Use Zulip Markdown (CommonMark)
+- Email: Use `syslog-email` skill
+
+## Kanban Board Protocol
+
+**File:** `~/.hermes/kanban/kanban.json`
+
+```json
+{
+  "task_id": "unique-id",
+  "title": "Task description",
+  "created": "2026-07-09T01:00:00",
+  "status": "backlog|in_progress|review|done",
+  "lanes": [
+    {
+      "lane_id": "devops-check",
+      "worker": "syslog-devops",
+      "goal": "Check all 5 Proxmox nodes",
+      "status": "dispatched|completed|failed",
+      "output_file": "/tmp/node-report.md"
+    }
+  ]
+}
+```
+
+**Update the board on every state change.**
+
+## Failure Handling
+
+### Worker Timeouts
+
+- Child timeout: **900 seconds** (15 minutes)
+- Worker model `syslog-auto` is slow — it can hit the timeout limit with
+  22+ API calls
+- **If a worker times out:** Re-dispatch with a narrower scope. Break the
+  task into smaller pieces that fit in the timeout window.
+- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
+  that compounds quickly. Prefer API-based or local approaches when possible.
+
+### Worker Selection Failures
+
+- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
+- `syslog-code` is best for code-level work (reading files, writing scripts)
+- `syslog-research` has the browser toolset — use for web research
+- `syslog-review` is the QA gate — always fire before delivery
+- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
+- **Never nest delegation** (max_spawn_depth: 1)
+
+### Context Overflow
+
+- If a task requires >10K tokens of output, delegate the processing
+- Workers return compact summaries, not raw data dumps
+- Pass file paths and concrete goals — never dump raw data into context
+
+## Anti-patterns
+
+- ❌ Reading large files into your own context before deciding → delegate the read
+- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
+- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
+- ❌ Skipping verification → raw worker output never reaches the user
+- ❌ Delegating single tool calls → keep quick reads/writes at manager level
+- ❌ Firing more than 3 workers in parallel → hard limit
+
+## Emergency Exception
+
+**In an emergency (server down, service must be restored immediately):**
+- Delegate the diagnosis (find the problem)
+- Execute the fix yourself (minimize handoff latency)
+- Verify the fix after delivery
+- Log the exception in the kanban board
+
+The emergency exception exists because the user needs the service back NOW,
+not after three worker round-trips. But it's an exception — not the rule.
+
+## What This Contract Doesn't Cover
+
+1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
+2. **SSH key management** — covered by existing SSH/Proxmox contracts
+3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
+4. **Cron job management** — covered by individual cron contracts
+5. **Infra verification** — covered by `verify-before-mutate` protocol
+
+## Verification
+
+Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
+
+```bash
+python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
+```
+
+## References
+
+- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
+- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
+- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
+- `verify-before-mutate` protocol: Infrastructure change verification
+- `hermes-config-template.prose.md`: Worker profile configuration
+
+## Success Criteria
+
+This contract succeeds when:
+
+1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
+2. **Workers do the work** — manager coordinates, doesn't execute
+3. **Verification before delivery** — all output passes through `syslog-review`
+4. **Kanban board is current** — every task has a lane, every lane has a status
+5. **User gets verified results** — raw worker output never reaches Kwame
+
+---
+
+**Last updated:** 2026-07-09
+**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
+**Status:** Draft — awaiting PR review and merge to prose-contracts main

disk-gc-threat-response.prose.md

@@ -0,0 +1,306 @@
+---
+kind: responsibility
+name: disk-gc-threat-response
+description: >
+  Recurring disk health scan, garbage collection, and threat response
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
+  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
+  Docker image bloat — 5 dangling images, 15 build cache layers.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
+  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
+id: 067NV8KJ03ZG71S44N41F31022
+version: 1.0.0
+---
+
+# Disk GC & Threat Response
+
+## Goal
+
+Every container in the fleet stays below 80% disk usage through regular inspection
+and automated garbage collection. Critical breaches are detected, remediated, and
+logged within 5 minutes of discovery.
+
+## Scope
+
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
+Docker hosts get special attention:
+
+| Host | CT | Disk Risk | GC Strategy |
+|------|----|-----------|-------------|
+| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
+| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
+| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
+| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
+> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
+
+## Threat Levels
+
+| Level | Threshold | Response | Escalation |
+|-------|-----------|----------|------------|
+| **GREEN** | < 75% | Log only | None |
+| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
+| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
+| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
+| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
+
+## Requires
+
+- SSH access to all Docker hosts (matrix from infrastructure-control pattern)
+- Proxmox API access from abiba (token already provisioned)
+- Zulip bot credentials for alerting
+
+## Maintains
+
+Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
+the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
+bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
+documented reason.
+
+### disk-health
+Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
+timestamp, and active threat level.
+
+### gc-history
+Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
+and whether threat was resolved.
+
+### threat-log
+Active and resolved threat entries with severity, timestamps, remediation applied,
+and escalation trail.
+
+## Continuity
+
+- self-driven: full fleet scan every 6 hours
+- May also be invoked manually: `prose run disk-gc-threat-response`
+- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
+
+## Shape
+
+- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
+- `delegates`:
+  - `disk-scanner`: per-CT disk check (pct exec df or SSH)
+  - `gc-docker`: docker system prune execution
+  - `gc-system`: apt clean, log rotate, tmp cleanup
+  - `alerter`: Zulip notification dispatch
+- `prohibited`: deleting user data, removing running containers, force-killing
+  production services
+
+## Runtime
+
+- `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
+- `retry`: 2 attempts for SSH failures before marking a CT unreachable
+
+## Execution
+
+```prose
+let fleet = call disk-scanner
+  scope: all
+
+let threats = []
+for ct in fleet:
+  if ct.usage_pct >= 95:
+    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
+  else if ct.usage_pct >= 85:
+    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
+  else if ct.usage_pct >= 75:
+    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
+
+-- sort by severity descending
+sort threats by pct desc
+
+for threat in threats:
+  let result = call gc-executor
+    ct: threat.ct
+    level: threat.level
+    strategy: lookup-gc-strategy(threat.ct)
+  
+  call alerter
+    threat: threat
+    result: result
+
+call summary-reporter
+  fleet: fleet
+  threats: threats
+```
+
+## GC Strategies by Host Type
+
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
+
+```bash
+# Phase 1: Safe prune (won't touch running containers' images)
+docker system prune -f
+
+# Phase 2: Aggressive (if still > 85% after Phase 1)
+docker system prune -a --force
+
+# Phase 3: Emergency (if still > 95%)
+docker system prune -a --force --volumes
+docker builder prune --all --force
+
+# Verification after each phase
+df -h /
+docker system df
+```
+
+### Non-Docker CTs
+
+```bash
+# Package cache
+apt-get clean
+apt-get autoremove --yes
+
+# Log rotation
+journalctl --vacuum-size=100M
+find /var/log -type f -name "*.log" -mtime +30 -delete
+
+# Temp files
+find /tmp -type f -mtime +7 -delete
+find /var/tmp -type f -mtime +30 -delete
+
+# Snap (if installed)
+snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
+  snap remove "$snap" --revision="$rev"
+done
+```
+
+### Special Cases
+
+| CT | Special GC |
+|----|-----------|
+| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
+| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
+| 106 (ra-h-os) | Check relay DB size, enforce TTL |
+| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |
+
+## Alert Templates
+
+### AMBER (75-84%)
+```
+⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Next scheduled GC will attempt cleanup. No immediate action needed.
+```
+
+### RED (85-94%)
+```
+🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
+Status: {resolved|still elevated — {reason}}
+```
+
+### CRITICAL (≥95%)
+```
+🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
+{service_status} — {manual_action if needed}
+@Kwame — container nearly full, resolved: {yes_no}
+```
+
+## Incident Log: 2026-07-04 — kagentz docker bloat
+
+### Discovery
+Kwame noticed kagentz was full, asked Abiba to investigate.
+
+### Diagnosis
+- CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
+- `/var/lib` = 55G (93% of disk)
+- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
+- 1 stopped container, 1 unused network, 15 build cache layers
+
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 35.67GB
+→ Post: 16G / 59G (27%), 41G free
+→ 5 dangling images removed (kagentz-bridge + old builds)
+→ 1 stopped container removed
+→ 1 unused network removed (kagentz-bridge_default)
+→ 15 build cache layers removed
+```
+
+### Root Cause
+Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
+dangling images and orphaned build cache. No automated GC was in place.
+
+### Preventive Measures
+- This contract now runs disk GC fleet-wide every 6 hours
+- Docker hosts get `docker system prune` on amber, `-a --force` on red
+- kagentz is flagged as HIGH risk due to development activity
+- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
+```
+
+## Incident Log: 2026-07-09 — amdpve docker bloat
+
+### Discovery
+Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
+amdpve (.15) flagged at 78% (AMBER threshold: 75%).
+
+### Diagnosis
+- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
+- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
+- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
+- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
+  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
+- Not a running service — zero containers, zero active volumes
+
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 11.56GB
+→ Post: 55G / 94G (62%), 35G free
+→ 1 image removed (llama-strix-hip:latest, 15.5GB)
+→ 8 build cache layers removed
+→ amdpve now GREEN
+```
+
+### Root Cause
+Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
+artifacts. Docker on amdpve serves no running purpose — it's only used for
+one-off GPU builds. No automated post-migration cleanup was in place.
+
+### Preventive Measures
+- amdpve added to Docker GC scan list
+- Post-migration cleanup step added: after any GPU backend migration, prune
+  the old backend's Docker images within 24 hours
+- Contract now scans GPU bare-metal hosts alongside CTs
+- Access via `pct-run` script for all CTs (no hardcoded IPs)
+
+## Access Matrix (documented 2026-07-09)
+
+### CT Access (via pct-run)
+| CT | Name | Node | Status |
+|----|------|------|--------|
+| 100 | abiba | amdpve | local |
+| 102 | adguard | acerpve | ✅ reachable |
+| 104 | authentik | minipve | ✅ reachable |
+| 105 | kagentz | amdpve | ✅ reachable |
+| 106 | ra-h-os | storepve | ✅ reachable |
+| 107 | pbs | storepve | ✅ reachable |
+| 108 | media | storepve | ✅ reachable |
+| 110 | gitea | minipve | ✅ reachable |
+| 111 | tdunna | amdpve | ✅ reachable |
+| 112 | tanko | amdpve | ✅ reachable |
+| 113 | baggy | amdpve | ✅ reachable |
+| 114 | mumuni | minipve | ✅ reachable |
+| 115 | scottdenya | amdpve | ✅ reachable |
+| 116 | syslog-api | minipve | ✅ reachable |
+| 117 | zulip | storepve | ✅ reachable |
+
+### GPU Bare Metal (via direct SSH)
+| Host | IP | GPU | Status |
+|------|-----|-----|--------|
+| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
+| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
+| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+
+### KVM VM (via direct SSH)
+| Host | IP | Role | Status |
+|------|-----|------|--------|
+| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.

docs/AUTHORING-GUIDE.md

@@ -0,0 +1,308 @@
+# Prose Contract Authoring Guide
+
+Approach this as the lead systems engineer at a company where every contract is a
+production dependency — agents run from these every 15 minutes, every 6 hours,
+every Sunday at 3am. A stale IP or a missing section isn't a typo; it's an
+incident waiting to happen. Write every contract as if you'll be paged at 3am
+when it breaks.
+
+## Ground it in the subject
+
+Before writing a single line of YAML, answer three questions and state them in
+the description:
+
+1. **What system does this contract touch?** Name the hosts, CTs, containers,
+   and services explicitly. "The inference fleet" is vague. "GPU .8 (RTX 3090,
+   qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, ornith), and LiteLLM on CT
+   116" is specific.
+
+2. **Who runs this contract, and when?** State the agent, the trigger (cron,
+   manual, event-driven), and the expected duration. "Runs every 6 hours via
+   cron on abiba, takes ~120 seconds for a fleet scan."
+
+3. **What breaks if this contract is wrong?** Name the blast radius. "A stale
+   CT ID causes a restart of the wrong container. A wrong Grafana URL breaks
+   the monitoring dashboard for all agents."
+
+If you cannot answer these three questions concretely, do not write the
+contract yet — go verify against the live system first.
+
+## Design principles
+
+### Specificity over generality
+
+A contract that says "check all Docker containers" is worse than one that lists
+them by name. The list will go stale, but staleness is detectable (the CI lint
+catches it). Vagueness is invisible and uncatchable. Be specific, accept that
+specificity decays, and rely on the lint pipeline to catch the decay.
+
+### Ground truth is the live system, not the contract
+
+Every IP, port, hostname, and container name in a contract is a **live-state
+field**. Mark them with `VERIFY-BEFORE-USE` in the access matrix. Before any
+mutating operation, verify against the live system. The contract is a
+hypothesis; the live system is the fact.
+
+When you discover drift between contract and reality, do not "fix" the live
+system to match the contract. Halt. Ask which is correct. Document the answer
+in the contract and the knowledge graph. The `.117→.19` incident was caused by
+fixing the wrong direction.
+
+### Contracts are typed — respect the kind
+
+| Kind | When to use | Must have | Must NOT have |
+|------|------------|-----------|---------------|
+| `function` | Stateless, callable helper | `## Parameters`, `## Returns`, `## Execution` | `## Maintains` (no state) |
+| `responsibility` | Persistent duty with state | `## Maintains`, `## Continuity` or `## Execution` | — |
+| `gateway` | External-driven entry point | `## Maintains` | `## Requires` |
+| `pattern` | Reference/template/doctrine | Flexible — prose-driven | Don't `prose run` directly |
+| `test` | Verification only | `## Execution` | Mutating operations |
+
+Use `kind: architecture` sparingly — it's not a standard OpenProse kind. Most
+architecture docs should be `kind: pattern`.
+
+### One contract, one concern
+
+A contract that monitors health AND deploys exporters AND sends alerts is three
+contracts fighting for attention. Split them:
+
+```
+❌ infrastructure-monitoring.prose.md (deploy + export + alert)
+✅ proxmox-monitor.prose.md (deploy exporters)
+✅ gpu-monitor.prose.md (poll + dashboard)
+✅ infrastructure-control.prose.md (reference topology)
+```
+
+The contract that deploys is a function. The contract that polls is a
+responsibility. The contract that maps the topology is a pattern. One concern
+per file, one kind per concern.
+
+### Live-state fields are declared, not hidden
+
+Don't bury IPs and hostnames in prose paragraphs. Put them in tables, mark them
+with `VERIFY-BEFORE-USE`, and keep them in one place per contract:
+
+```markdown
+| Host | IP | Role | Trust |
+|------|----|------|-------|
+| CT 116 | 192.168.68.116 | LiteLLM + Grafana | ⚠️ VERIFY-BEFORE-USE |
+```
+
+The lint pipeline scans for known stale values. Tables are grep-able; prose is
+not.
+
+## Process: verify, draft, lint, review, ship
+
+### Pass 1: Verify
+
+Before writing, verify every live-state field you plan to include:
+
+```bash
+# Example: writing a contract about Zulip
+curl -s http://192.168.68.19/api/v1/server_settings  # is .19 reachable?
+ssh root@192.168.68.19 "docker ps"                     # is Zulip in Docker?
+pct config 117 | grep net0                              # what IP does CT 117 have?
+```
+
+Write down the verified values. Use them in the contract. If you can't verify
+something, mark it explicitly: `⚠️ UNVERIFIED — needs live check`.
+
+### Pass 2: Draft
+
+Write the contract following the structure for its kind. Fill every required
+section. Use the templates below. Write descriptions that name the subject: "GPU
+fleet health check across RTX 3090 (.8), RTX 5070 (.110), and Strix Halo (.15)"
+not "monitoring check."
+
+### Pass 3: Lint locally
+
+Before pushing, run the same checks the CI will run:
+
+```bash
+bash scripts/prose-lint.sh
+```
+
+Fix everything that fails. The lint checks for:
+- Valid YAML frontmatter
+- Required sections per kind
+- Regression patterns (/grafana/ route, CT 122/123)
+- Missing Maintains/Parameters/Returns
+
+### Pass 4: Self-critique
+
+Read your contract and ask:
+
+1. **Could an agent run this at 3am without context?** If it needs tribal
+   knowledge, add it to the description or a comment.
+2. **Does any value contradict the infrastructure-control pattern?** Check CT
+   IDs, IPs, hostnames, nginx routes, Grafana access.
+3. **Is anything in here that was previously fixed and reverted?** Check the
+   lint regression rules. If you're unsure, grep the git log for similar
+   changes.
+4. **Am I saying the same thing as another contract?** If yes, link to it
+   instead of duplicating. One source of truth.
+
+### Pass 5: Open PR, let CI review
+
+Push to a branch, open a PR. The pipeline runs:
+
+```
+auth → validate → lint → ai-review → gate
+```
+
+The AI review compares your diff against the infrastructure-control ground
+truth. It catches contradictions you might miss. If it flags something, don't
+argue — verify and fix. The AI is reading the same ground truth you should have
+read before writing.
+
+## Restraint and self-critique
+
+Spend your complexity budget in one place. A contract that deploys Prometheus,
+configures Grafana, provisions dashboards, and sets up exporters is a function
+that does four things — each of which can fail independently. Split them.
+
+Don't add a `### Requires` that you haven't verified works. A dependency on
+"SSH access to .116" is only valid if you've tested it from the agent that
+will run the contract.
+
+Cut sections that don't serve the runner. A 200-line appendix of curl commands
+is documentation, not execution. Put it in a wiki or a README, not in the
+contract. The contract is for the agent running it; the README is for the human
+reading about it.
+
+Before shipping, take one thing out. Like Chanel's advice: before leaving the
+house, remove one accessory. Is there a section, a parameter, a check that
+doesn't pull its weight? Cut it. The contract that ships with 5 sections will
+be maintained. The one with 15 will rot.
+
+## Writing in contracts
+
+### Descriptions
+
+A description is not a label. It's the first thing an agent reads before
+deciding whether to run this contract. Make it answer: what does this do, to
+what, and why should I care?
+
+```
+❌ "Monitors infrastructure health"
+✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker
+   container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
+```
+
+### Field names
+
+Name things by what they control, not by how the system is built:
+
+```
+❌ prometheus_scrape_interval_seconds
+✅ check_every_seconds
+```
+
+The runner doesn't care that Prometheus is doing the scraping. They care how
+often the check runs.
+
+### Error messages and alerts
+
+Errors tell the operator what went wrong and what to do about it. Be specific:
+
+```
+❌ "Health check failed"
+✅ "GPU .15 (Strix Halo) unreachable at :8080 — firewalled to .116 only.
+   Check router /health/unified at http://192.168.68.116/health/unified instead."
+```
+
+### Comments
+
+Comments in contracts explain WHY, not WHAT. The execution steps say what to
+do. Comments say why we do it this way and not another:
+
+```markdown
+## Execution
+
+1. **Check GPU health via router** — cannot probe .15:8080 directly
+   (firewalled to .116 only as of 2026-07-01 Vulkan rebuild)
+2. **Skip sidecar check** — JSON exporters at :8090 were never deployed
+   on any GPU host; router falls back to GPU /health direct probe
+```
+
+## Templates
+
+### Function contract template
+
+```markdown
+---
+kind: function
+name: contract-name
+description: >
+  What this does, to what, and why. Be specific about hosts/services.
+version: 1.0.0
+---
+
+## Parameters
+
+- param_one: type — description (default: "value")
+- param_two: type — description (default: "value")
+
+## Returns
+
+- result_field: type — description
+
+## Requires
+
+- Dependency one — how to verify it's available
+- Dependency two
+
+## Execution
+
+1. **Step one** — what and why
+2. **Step two** — what and why
+3. **Compile and return** — assemble the output
+```
+
+### Responsibility contract template
+
+```markdown
+---
+kind: responsibility
+name: contract-name
+description: >
+  What this recurring duty monitors/maintains. Name the systems, the
+  cadence, the blast radius.
+version: 1.0.0
+---
+
+## Maintains
+
+- state_field: { current: value, last_check: timestamp }
+- another_field: description of maintained state
+
+## Parameters
+
+- param: type — description (default: "value")
+
+## Requires
+
+- Dependency one
+- Dependency two
+
+## Continuity
+
+- Self-driven: check every N seconds/minutes/hours
+- Also wakes on: trigger description
+
+## Execution
+
+1. **Check** — verify current state
+2. **Evaluate** — compare against thresholds
+3. **Remediate or escalate** — fix or alert
+4. **Record** — update world-model state
+```
+
+## Enforcement
+
+This guide is aspirational, not gated. The CI pipeline enforces structure
+(frontmatter, required sections) and regression (known stale patterns). It
+does not enforce prose quality or specificity. That's on you, the author.
+
+Before you ship: read your contract as if you're the agent who will run it at
+3am with 30 seconds of context. If you wouldn't trust it, rewrite it.

gpu-fleet.prose.md

@@ -0,0 +1,221 @@
+---
+kind: responsibility
+name: gpu-fleet
+description: >
+  Manages the GPU inference fleet across all hosts. Handles model deployment,
+  registration, health checks, LiteLLM sync, agent key management, GPU
+  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
+  Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2
+  on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully
+  deprecated — nginx routes /v1 → LiteLLM directly.
+agent: abiba
+triggers:
+  - on model add/remove
+  - on GPU health degradation
+  - on agent key rotation
+  - on router restart (roster must be loaded)
+---
+
+## Maintains
+
+- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
+- router: { status: "healthy", roster_loaded: bool, models: array }
+- litellm: { status: "healthy", keys: array, models: array }
+- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
+- health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
+- monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
+- watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
+- benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
+- grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
+- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
+- port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding
+
+## Fleet Topology (Current — June 2026)
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│  CT 116 (192.168.68.116) — Inference Harness Host                │
+│                                                                  │
+│  nginx:80 (entrypoint)                                           │
+│  ├─ /v1/*       → harness-litellm:4000   (API requests)         │
+│  ├─ /admin/*    → harness-litellm:4000   (admin endpoints)       │
+│  ├─ /dashboard/ → harness-dashboard:3000 (harness UI)            │
+│  ├─ /litellm/*  → harness-litellm:4000   (LiteLLM UI + API)     │
+│  ├─ /health/*   → harness-litellm:4000   (health probes)         │
+│  └─ /gpu/*      → 192.168.68.24:9100     (fleet monitor)         │
+│                                                                  │
+│  Containers:                                                     │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
+│  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
+│  │ :4000    │─▶│ :9000    │  │ :3000    │  │ :3000     │        │
+│  │ keys+sync│  │internal  │  │ harness  │  │ Prometheus│        │
+│  │ fallback │  │only!     │  │ UI       │  │ data src  │        │
+│  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
+│                    │                                             │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
+│  │PostgreSQL│  │  Redis   │  │Prometheus│                       │
+│  │ :5432    │  │ :6379    │  │ :9090    │                       │
+│  └──────────┘  └──────────┘  └──────────┘                       │
+└────────────────────┼─────────────────────────────────────────────┘
+                     │
+     ┌───────────────┼───────────────┬──────────────────┐
+     │               │               │                  │
+┌────▼─────┐  ┌──────▼──────┐  ┌────▼──────┐  ┌───────▼──────┐
+│ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
+│ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
+│ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
+│ 128K ctx │  │ 128K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
+│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
+│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
+│ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
+└──────────┘
+```
+
+## Current Model Assignments (2026-07-08)
+
+| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
+|-------|-----|------|------|-----|----------|----------|-------------|--------|
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+
+## Operations
+
+### add-model
+1. Download model file from Hugging Face or source
+2. Check disk space + GPU VRAM compatibility
+3. Start llama-server via systemd service on GPU host
+4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
+5. Hot-reload or restart router
+6. Add model to LiteLLM config (`model_list` + `fallbacks`)
+7. Generate agent keys for new model access via `/key/generate`
+8. Add Prometheus scrape target for the new GPU exporter
+9. Verify end-to-end: LiteLLM → Router → Model
+
+### remove-model
+1. Drain active requests (wait for active=0)
+2. Remove from LiteLLM config
+3. Remove from router config
+4. Stop llama-server (systemd)
+5. Remove Prometheus scrape target
+6. Cleanup model files (optional)
+
+### heal
+1. Check all GPUs via router internal `:9000/health/unified`
+2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
+3. Reset stuck circuit breakers if idle (Redis)
+4. Restart dead llama-server instances via SSH
+5. Flush Redis active counters if stale
+6. Verify GPU monitor server is running on pi (:9100)
+7. Verify watchdog is running on pi
+8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
+9. Reload roster via `POST :9000/admin/roster/reload` if available
+
+### sync-keys
+1. List all agent keys in LiteLLM DB via `GET /key/list`
+2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
+3. Generate missing keys via `POST /key/generate` with unlimited budget
+4. Update agent configs — `/etc/environment` LITELLM_API_KEY
+5. Send Zulip DM to agents that can't be reached via SSH
+6. Verify each key with test request through full chain
+7. Document keys in knowledge graph
+
+### list
+Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys
+
+### health-check
+1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
+2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
+3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
+4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
+5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
+   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
+   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
+6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
+7. Check port conflicts: verify only one llama-server on :8080 per host
+8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
+
+## Agent Keys (LiteLLM DB — Current 2026-06-30)
+
+| Agent | CT | IP | Key | Access |
+|-------|-----|-----|-----|--------|
+| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
+| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
+| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
+| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
+| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
+
+**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
+If no SSH access, send Zulip DM via abiba-bot.
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
+| `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
+| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
+| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
+| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
+| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
+| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
+
+## Prometheus & Grafana
+
+| Component | URL | Details |
+|-----------|-----|---------|
+| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
+| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
+| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
+| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
+| Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |
+
+## Known Issues & Watch Points
+
+- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
+  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
+- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
+- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110).
+- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput.
+- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`.
+- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
+- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
+- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
+- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
+- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
+- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
+- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
+- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
+- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
+- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.
+
+## GPU Inference Benchmarks (Current)
+
+| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
+|-----|-------|-----------|--------------|----------|---------|
+| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
+| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
+| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
+
+Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
+Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
+History stored at `/root/data/toks-history.json` with 7-day rolling window.
+
+**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
+
+## Agent Config Implications (2026-07-08)
+
+With NVIDIA GPUs at 128K context:
+- Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow
+- Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule
+- Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072`
+- Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx)
+- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
+- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)

gpu-monitor.prose.md

@@ -0,0 +1,136 @@
+---
+kind: responsibility
+name: gpu-monitor
+description: >
+  Comprehensive GPU fleet monitor — polls every subsystem (sidecars, router,
+  LiteLLM, Strix Halo, dashboard) every 15s, renders a live HTML dashboard,
+  checks alert thresholds, and exposes a JSON API for downstream consumers.
+agent: abiba
+---
+
+## Maintains
+
+- gpu-fleet-data: { gpus: array, strix: object, router: object, litellm: object, summary: object, alerts: array } — Full fleet snapshot refreshed every 15s
+- dashboard: live HTML at `/root/dashboard/gpu-fleet.html`, served on port 9100
+- gpu-data-api: JSON at `GET /gpu-data` on port 9100
+- health-endpoint: `GET /health` on port 9100 → 200 if cache populated, 503 if warming up
+
+## GPU Fleet Topology
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  GPU Monitor Server (:9100) — 192.168.68.24              │
+│  Polls every 15s, serves dashboard + JSON API            │
+└──┬──────────┬──────────┬────────────────────────────────┘
+   │          │          │
+   ▼          ▼          ▼
+┌──────┐ ┌──────┐ ┌────────┐
+│.8:8080│ │.110  │ │.116:80 │
+│RTX3090│ │:8080 │ │nginx   │
+│gemma  │ │RTX5070│ │router  │
+└──────┘ │qwen27B│ │LiteLLM │
+         └──────┘ │dashboard│
+                  └────────┘
+
+Note: JSON sidecar exporters at :8090 were never deployed on any
+GPU host. Router falls back to GPU /health direct probe. Monitor
+should use router /health/unified as source of truth for GPU status.
+Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
+poll .15:8080 directly; must go through router on .116.
+```
+
+### Subsystems Polled
+
+| Subsystem | Endpoint | Frequency | Metrics |
+|-----------|----------|-----------|---------|
+| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
+| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
+| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
+| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
+| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
+| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
+
+### Alert Delivery
+
+All alerts are sent to `#agent-hub` stream topics:
+- `alerts-gpu` — GPU fleet alerts (temp, VRAM, utilization)
+- `alerts-pm2` — PM2 process health (restarts, crashes)
+- `alerts-infra` — Infrastructure health (LiteLLM, Zulip, containers)
+
+This replaces the previous DM-only delivery. All agents on the mesh can see and respond to alerts.
+
+## Alert Thresholds
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| GPU Temp | >80°C | >90°C |
+| VRAM Usage | >90% | >95% |
+| GPU Util | >95% | >98% |
+| Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
+| Model Down | — | critical (circuit breaker open) |
+
+### JSON API Response Schema (/gpu-data)
+
+```json
+{
+  "gpus": [{ "name", "gpu_name", "temp_c", "vram_used_mb", "vram_total_mb",
+             "gpu_util_pct", "power_w", "power_limit_w", "fan_pct",
+             "models", "hostname", "active_requests", "health_score",
+             "circuit_open" }],
+  "strix": { "status", "cpu_load", "llama_health" },
+  "router": { "available_models", "circuit_breaker", "gpus", "scores",
+              "status", "redis", "_basic" },
+  "litellm": { ... },
+  "dashboard": { "reachable": bool },
+  "summary": { "fleet_status", "models_available", "models_total",
+               "circuit_breakers_open", "gpu_count", "gpu_errors",
+               "strix_running", "router_reachable", "litellm_reachable" },
+  "alerts": [{ "gpu", "metric", "level", "value", "threshold" }],
+  "updated": "ISO8601",
+  "monitor_version": "2.0.0"
+}
+```
+
+## Operations
+
+### list
+`curl http://localhost:9100/gpu-data | jq` — Full fleet status
+
+### check-health
+`curl http://localhost:9100/health` — Monitor self-check
+
+### view-dashboard
+Open `http://localhost:9100/` in browser — Live HTML dashboard
+
+### restart
+```bash
+pkill -f gpu-monitor-server.py
+python3 /root/scripts/gpu-monitor-server.py &
+```
+Or via PM2: `pm2 restart gpu-monitor`
+
+### check-router
+The router health is accessed through nginx on port 80 (NOT port 9000 directly).
+`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy
+`curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
+| `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
+
+## Execution
+
+1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
+2. **Poll router** (every 15s): GET .116/health via nginx:80
+3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
+4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
+5. **Poll dashboard** (every 15s): GET .116/dashboard/
+6. **Check alerts**: Compare metrics against thresholds
+7. **Compute summary**: Fleet-wide health aggregation
+8. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
+9. **Serve API**: HTTP server on port 9100
+10. **Repeat** every 15 seconds

hello-world.prose.md

@@ -10,6 +10,6 @@ description: A simple hello world contract to test OpenProse on pi
 ## Returns
 - greeting: string — The generated greeting message
 
-## Ensures
+## Maintains
 - The greeting includes the provided name
 - The greeting is friendly and warm

hermes-agent-baseline.prose.md

@@ -0,0 +1,262 @@
+---
+kind: template
+name: hermes-agent-baseline
+version: 1.0.0
+description: >
+  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
+  configuration state, keys, workarounds, and audit procedure. When an agent's
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
+author: Abiba (pi agent)
+---
+
+# Hermes Agent Baseline — Canonical Good State
+
+## Quick Restore
+
+```bash
+# Verify all agents against baseline in one command:
+for ct in 112 114 111 113; do
+    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
+done
+```
+
+## Agent Map
+
+| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
+|-------|-----|------|-----|-------------|---------------|----------|
+| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
+| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
+| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
+| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |
+
+Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
+
+## Key Architecture
+
+```
+Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
+                                                     └── Key DB (Postgres)
+```
+
+- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
+- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
+- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
+- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
+
+## Config Pattern — Mandatory Fields
+
+### For Hermes Agents (Tanko, Mumuni, Baggy)
+
+Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
+
+### 1. Main Model
+```yaml
+model:
+  default: syslog-auto
+  provider: custom:harness       # or: harness
+  api_key_env: LITELLM_API_KEY
+  max_tokens: 4096
+```
+
+### 2. Custom Provider
+```yaml
+custom_providers:
+  - name: harness
+    model: syslog-auto
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_mode: chat_completions
+```
+
+### 3. Vision (CRITICAL — must have api_key directly!)
+```yaml
+auxiliary:
+  vision:
+    provider: harness
+    model: gemma-4-12b              # or syslog-auto
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    timeout: 60
+    download_timeout: 30
+```
+
+### 4. Compression (must have api_key!)
+```yaml
+  compression:
+    enabled: true
+    threshold: 0.65
+    target_ratio: 0.3
+    provider: harness
+    model: syslog-auto               # or gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    timeout: 120
+```
+
+## Known Bug: `api_key_env` Ignored by Auxiliary Client
+
+**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
+
+**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
+resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
+to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
+to `"no-key-required"` → 401 from LiteLLM.
+
+**Impact**: Vision analysis, compression, and any other auxiliary task calling
+LiteLLM/harness will fail with:
+```
+401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
+```
+
+**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
+`api_key_env` in every auxiliary task config that uses the harness provider.
+
+**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
+`api_key` is empty:
+```python
+cfg_api_key = str(task_config.get("api_key", "")).strip() or None
+if not cfg_api_key:
+    key_env = str(task_config.get("api_key_env", "")).strip()
+    if key_env:
+        cfg_api_key = os.getenv(key_env, "").strip() or None
+```
+
+## Audit Procedure
+
+### Full Audit (all agents)
+```bash
+for ct in 112 114 111 113; do
+    echo "=== CT $ct ==="
+    pct-run $ct grep LITELLM_API_KEY /etc/environment
+    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
+    echo ""
+done
+```
+
+### Master Key Leak Check
+```bash
+# On every agent:
+pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
+# Must return empty
+```
+
+### Verify Key Works
+```bash
+curl -s http://192.168.68.116:80/v1/models \
+  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
+# Must return model list
+```
+
+### Verify Vision/Compression
+```bash
+# Check both api_key and api_key_env are present:
+pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
+# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
+```
+
+### For pi Agents (Tdunna)
+
+Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
+Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
+
+**models.json** — Must only list models authorized for the agent's LiteLLM key:
+```json
+{
+  "providers": {
+    "syslog-harness": {
+      "baseUrl": "http://192.168.68.116/v1",
+      "api": "openai-completions",
+      "apiKey": "sk-...",
+      "models": [
+        { "id": "syslog-auto" },
+        { "id": "ornith-1.0-35b" },
+        { "id": "qwen3.6-27B-code" },
+        { "id": "gemma-4-12b" }
+      ]
+    }
+  }
+}
+```
+
+**settings.json** — Always use `syslog-auto` as default:
+```json
+{
+  "defaultProvider": "syslog-harness",
+  "defaultModel": "syslog-auto"
+}
+```
+
+**Validation**: Verify models match LiteLLM's authorized list:
+```bash
+curl -s http://192.168.68.116:4000/v1/models \
+  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
+  | jq '.data[].id'
+```
+
+**Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
+a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
+stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.
+
+**Service**: `pm2 restart koby-zulip`, health at `:9201/health`.
+
+## Systemd Pattern
+
+For agents where the gateway runs as a user service:
+```ini
+# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
+[Service]
+Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
+```
+
+For agents where the gateway runs as root:
+```ini
+# /root/.config/systemd/user/hermes-gateway.service
+EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
+```
+
+**No drop-in that hardcodes the master key. Ever.**
+
+## Attachment Cache Locations
+
+| Type | Path |
+|------|------|
+| Images | `~/.hermes/cache/images/` |
+| Documents | `~/.hermes/cache/documents/` |
+| Audio | `~/.hermes/cache/audio/` |
+
+## Health Verification
+
+Run the consolidated health check:
+```bash
+python3 /root/scripts/agent-health-check.py
+```
+This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
+verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
+and counts recent errors. Non-disruptive — never restarts anything.
+
+## GPU Port Conflict Detection
+
+All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
+- `.8` and `.110`: inline check in `llama-wrapper.sh`
+- `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)
+
+Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
+If they differ → ghost detected → kill ghost → start fresh.
+
+## Related Contracts
+
+- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
+- `hermes-config-template.prose.md` — full configuration template
+- `gpu-fleet.prose.md` — GPU fleet and agent key table
+- `infrastructure-control.prose.md` — CT inventory with pct-run access
+- `litellm-health.prose.md` — LiteLLM stack health verification
+
+## Change Log
+
+| Date | Change |
+|------|--------|
+| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
+| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
+| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |

hermes-config-template.prose.md

@@ -4,132 +4,102 @@ name: hermes-config-template
 description: >
   Standard Hermes configuration template for Syslog Solution LLC agents.
   Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
-  RA-H OS MCP) while keeping model/provider choices flexible per agent role.
-  Every new agent profile should start from this template.
+  RA-H OS MCP) while keeping agent-specific API keys and model choices.
+  Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110),
+  parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added.
 ---
 
 ## Maintains
 
-- template_version: string — Current template version (e.g., "1.0.0")
-- last_applied: timestamp — When any agent was last configured from this template
-- agents_configured: array — Which agents/profiles were created from this template
-- infra_endpoints_verified: array — Firecrawl, SearXNG, RA-H OS, LiteLLM endpoints last confirmed working
-- known_deviations: array — Profiles that diverge from the template (and why)
+- template_version: "2.1.0"
+- last_applied: timestamp
+- agents_configured: ["tanko", "mumuni", "abiba", "tdunna", "baggy", "kagenz0"]
+- agent_keys: map (see Agent Keys section)
+- infra_endpoints_verified: array
 
-## Parameters
+## Agent Keys (LiteLLM — Current 2026-07-04)
 
-- agent_name: string — Name of the agent/profile (e.g., "syslog-code", "syslog-devops")
-- default_model: string — Agent's primary model (e.g., "qwen3.6-27B-code", "syslog-auto")
-- default_provider: string — Inference provider for the primary model (default: "harness")
-- fallback_model: string — Fallback model when primary is down (default: "gemma-4-12b")
-- fallback_provider: string — Fallback provider (default: "harness")
-- auxiliary_model: string — Model for vision/compression/extraction tasks (default: "gemma-4-12b")
-- enable_firecrawl_searxng: boolean — Whether to configure web search/extract (default: true)
-- enable_ra_h_os_mcp: boolean — Whether to wire up RA-H OS MCP bridge (default: true)
-- enable_compression: boolean — Whether to enable context compression (default: true)
-- custom_provider_name: string — Custom provider name (default: "harness")
-- custom_provider_model: string — Custom provider model (default: same as default_model)
-- custom_provider_url: string — LiteLLM base URL (default: "http://litellm.sysloggh.net/litellm/v1")
-- extra_auxiliary: array — Additional auxiliary tasks to configure (vision, compression, etc.)
-- output_dir: string — Where to write the config (default: "~/.hermes/profiles/<agent_name>/config.yaml")
+Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM
+PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB, not in
+config files. The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent
+host AND in `~/.hermes/.env` for gateway env propagation.
+Sub-agent profiles inherit auth from the main config — no separate keys needed.
 
-## Quickstart — How to Run This Contract
+| Agent | Key Alias | Host | SSH | Sub-Agents |
+|-------|-----------|------|-----|-----------|
+| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
+| Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
+| Abiba | `abiba-*` | 192.168.68.24 | local | — |
+| Tdunna | `tdunna-*` | ? | Zulip | — |
+| Baggy | `baggy-*` | ? | Zulip | — |
+| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
 
-### For the Agent Running This Contract
-
-Read this section first. It tells you exactly what to do.
-
-**When to run:**
-- Setting up a **new agent profile** for the first time
-- An **existing profile** is missing web search, firecrawl, or MCP configs
-- User says: `"apply the config template"` or `"fix my infra config"`
-- User says: `"configure <agent_name> with <model>"`
-
-**How to run (via OpenProse CLI):**
-```bash
-# Minimal — only agent name required
-prose run hermes-config-template agent_name=syslog-code
-
-# Full control
-prose run hermes-config-template \
-  agent_name=syslog-devops \
-  default_model=claude-sonnet-4 \
-  auxiliary_model=gemma-4-12b \
-  fallback_model=deepseek-chat \
-  fallback_provider=deepseek
-```
-
-### How to Follow the Template (Manual)
-
-If `prose` CLI is not available:
-
-1. **Fetch this contract:** `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/hermes-config-template.prose.md`
-2. **Open the target config.yaml** — the profile you're updating
-3. **Compare** each of the 5 required sections below against what exists in the profile
-4. **For each section:**
-   - If it says **SHARED INFRA — DO NOT CHANGE**: Copy the value exactly as shown
-   - If it says **USER CHOOSES**: Use the agent's assigned model/provider
-   - If missing entirely: Add the full section
-5. **Verify** every endpoint responds (see Verification section at the end)
-6. **Restart** the gateway
-
-### Decision Tree
-
-```
-What kind of profile is this?
-├── Main profile (~/.hermes/config.yaml)
-│   └── Replace web.*, mcp_servers.*, compression.*, auxiliary.*, custom_providers.*
-│   └── model.default + fallback: use current values (USER CHOOSES)
-│
-├── Worker profile (~/.hermes/profiles/<name>/config.yaml)
-│   ├── Does it have web.search_backend?
-│   │   ├── NO → Add the full web.* section (copy exactly)
-│   │   └── YES → Verify it points to the right IP
-│   ├── Does it have mcp_servers.ra-h-os?
-│   │   ├── NO → Add mcp_servers.* section (copy exactly)
-│   │   └── YES → Verify URL
-│   └── model.default: USE the worker's assigned model (don't copy main)
-│
-└── Non-Syslog profile (personal/lifestyle agent)
-    └── Skip — use default Hermes config or Tanko's config instead
-```
+✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
+  syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`
 
 ## Infrastructure Stack
 
-This template provisions the following shared infrastructure. Every agent should point to the same endpoints unless explicitly overridden.
-
 | Component | Endpoint | Purpose |
 |---|---|---|
 | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
 | SearXNG | `http://storepve:8888` | Privacy-respecting web search |
-| LiteLLM | `http://litellm.sysloggh.net/litellm/v1` | Unified model gateway |
+| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
+| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
 | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
 | Context7 MCP | `http://localhost:8079/mcp` | Documentation queries |
 
-**API Key Rules:**
-- `api_key: sk-_SW...B8nw` — Hardcoded LiteLLM master key for custom_providers and auxiliary tasks
-- `api_key_env: DEEPSEEK_API_KEY` — Env-var based key for fallback provider
-- Worker profiles should inherit proxy config via `harness` custom_provider, NOT hardcode LiteLLM keys
+## API Key Rules
 
-**For workers that need API keys overridden:** Set `api_key: ''` in the model section (inherits from custom_providers), or use `api_key_env` for env-based auth.
+- `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred)
+- `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider
+- `api_key: sk-...` — Hardcoded key only as fallback when env var not possible
+- Set `LITELLM_API_KEY` in `/etc/environment` on each host
+- Sub-agents NEVER get their own key — they share the host agent's key
+- Restart Hermes after updating `/etc/environment`
 
-## Template Structure
+### Sub-Agent Profiles (Mumuni pattern)
 
-### Required Sections (every agent MUST have)
+Mumuni has 6 sub-agent profiles in `/root/.hermes/profiles/<name>/config.yaml`:
+
+```
+profiles/
+├── syslog-code/config.yaml      # Code generation
+├── syslog-devops/config.yaml     # DevOps/infrastructure
+├── syslog-email/config.yaml      # Email processing
+├── syslog-research/config.yaml   # Research & analysis
+├── syslog-review/config.yaml     # Code review
+└── syslog-writer/config.yaml     # Content writing
+```
+
+Sub-agent profile rules:
+1. **`api_key` must be empty** — `api_key: ''` or omitted entirely
+2. **`base_url` must be empty** — inherits from main config's custom_provider
+3. **`provider` is `auto` or `harness`** — routes through the shared LiteLLM gateway
+4. **`model` is agent-specific** — each sub-agent can have its own default model
+5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty
+6. **Never hardcode a key** in sub-agent profiles
+
+This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`.
+When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
+work immediately after restart.
+
+## Template — Required Sections
 
 ```yaml
-# ─── Model Selection (USER CHOOSES) ───
+# ─── Model Selection ───
 model:
-  # !! CHANGE THIS for your agent !!
-  default: <agent_default_model>
-  provider: <agent_default_provider>
-  # LiteLLM base URL (shared infra)
-  base_url: http://litellm.sysloggh.net/litellm/v1
+  default: <agent_model>          # e.g., ornith-1.0-35b, qwen3.6-27B-code
+  provider: harness
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
+  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
+  context_length: 262144          # For syslog-auto (ornith route supports 256K).
+                                  # Set 131072 if using qwen3.6-27B-code or gemma-4-12b directly.
 
 fallback_providers:
-  # !! OPTIONAL: overridable fallback !!
-  provider: <fallback_provider>
-  model: <fallback_model>
+  provider: deepseek
+  model: deepseek-chat
+  api_key_env: DEEPSEEK_API_KEY
 
 # ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
 web:
@@ -139,7 +109,7 @@ web:
   firecrawl:
     base_url: http://192.168.68.7:3002/
 
-# ─── MCP Servers (SHARED INFRA — DO NOT CHANGE) ───
+# ─── MCP Servers (SHARED INFRA) ───
 mcp_servers:
   context7:
     connect_timeout: 60
@@ -150,86 +120,66 @@ mcp_servers:
     timeout: 120
     connect_timeout: 60
 
-# ─── Compression (SHARED — can override model) ───
+# ─── Compression ───
 compression:
   enabled: true
+  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
   provider: harness
-  model: <auxiliary_model>  # default: gemma-4-12b
-  threshold: 0.5
-  target_ratio: 0.25
-  protect_last_n: 30
-  hygiene_hard_message_limit: 400
+  max_context_window: 262144      # For syslog-auto (ornith supports 256K).
+                                  # Set 131072 if using qwen or gemma directly.
+  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
+  target_ratio: 0.30
+  protect_last_n: 40
+  hygiene_hard_message_limit: 350
+  protect_first_n: 3
+  abort_on_summary_failure: false
 
-# ─── Auxiliary Tasks (SHARED INFRA + MODEL) ───
+# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
+# All auxiliary services MUST use identical model, base_url, and api_key_env:
+#   model: gemma-4-12b
+#   base_url: http://192.168.68.116/v1
+#   api_key_env: LITELLM_API_KEY
+# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
+# gemma-4-12b is a lightweight 12B model on the RTX 5070, freeing the Strix Halo
+# for agent reasoning.
 auxiliary:
   vision:
     provider: harness
-    model: <auxiliary_model>
+    model: gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    timeout: 60
+    download_timeout: 30
   web_extract:
     provider: harness
-    model: <auxiliary_model>
-  session_search:
+    model: gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    timeout: 30
+  compression:
     provider: harness
-    model: <auxiliary_model>
-    max_concurrency: 3
+    model: gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    timeout: 60
 
-# ─── Custom Provider (SHARED INFRA — LiteLLM) ───
+# ─── Custom Provider ───
 custom_providers:
-  - name: <custom_provider_name>
-    model: <custom_provider_model>
-    base_url: http://litellm.sysloggh.net/litellm/v1
-    api_key: <liteLLM_master_key>
+  - name: harness
+    model: <agent_model>
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
     api_mode: chat_completions
 ```
 
-### Optional Sections (agent-specific)
+## Key Update Procedure
 
-- **Platform configs** (Telegram, Discord, WhatsApp bot tokens)
-- **Display/skin** options (compact mode, personality)
-- **Plugin lists** (zulip-platform, etc.)
-- **Terminal settings** (container image, timeouts)
-- **Security/approvals** (command allowlists)
-- **Kanban/worker** settings (if this profile is a Kanban worker)
-- **Skills auto_load** list
-- **Timezone** override
+When LiteLLM keys are regenerated (e.g., after infrastructure changes):
 
-## Continuity
-
-Configuration drift detection is **event-driven**:
-
-- **On agent onboarding**: Always scaffold from this template
-- **On infra change**: If Firecrawl/SearXNG/LiteLLM endpoints change, update template + all profiles
-- **On new model**: When a new model is deployed on Litellm, any agent can change their `default` independently
-- **Periodic**: Every 2 weeks — check if any profile's infra section has drifted from the template
-- **On connection failure**: If web search/extract fails, check if that agent has missing `web.*` config
-
-## Success Criteria
-
-The configuration is CONSIDERED GOOD when ALL of these pass:
-
-### Infra Connectivity
-- `web.backend == "firecrawl"` — Firecrawl backend configured
-- `web.search_backend == "searxng"` — SearXNG search configured
-- `web.extract_backend == "firecrawl"` — Firecrawl extraction configured
-- `web.firecrawl.base_url` points to `192.168.68.7:3002`
-- `mcp_servers.ra-h-os.url` points to `192.168.68.65:3100/mcp`
-- All endpoints respond to `curl` health checks
-
-### Model Flexibility
-- `model.default` is settable per agent (not hardcoded)
-- `model.provider` is settable per agent
-- `custom_providers[0].model` matches the agent's workload (code vs general)
-
-### Completeness
-- `auxiliary.vision` configured (even if provider=auto)
-- `compression.enabled` set (true for most agents)
-- Fallback provider configured (deepseek or another harness model)
-- `_config_version` correctly set (currently `30`)
-
-### No Drift
-- No duplicate infra endpoints (one canonical Firecrawl URL)
-- No stale endpoints (old IPs, dead services)
-- All profiles consistent on shared infra
+1. **If SSH available**: `ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"`
+2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command
+3. **After update**: Restart Hermes on the agent host
+4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
 
 ## Configuration Rules
 
@@ -241,67 +191,82 @@ The following MUST be identical across ALL profiles:
 - `custom_providers[0].base_url`
 
 ### Rule 2: Model Choice Is Free
-The following are OWNED by each agent and can differ:
-- `model.default` — the primary model
-- `model.provider` — where it runs
-- `fallback_providers.model` — backup model
-- `custom_providers[0].model` — what this agent uses LiteLLM for
+- `model.default` — per agent
+- `fallback_providers.model` — per agent
+- `custom_providers[0].model` — per agent
 
-### Rule 3: Workers Inherit, Not Duplicate
-Worker profiles (syslog-code, syslog-devops, etc.) generally inherit from the main config. Only override what's DIFFERENT:
-- Different default model → override `model.default`
-- Different auxiliary model → override auxiliary sections
-- No web search needed → still keep the config (tools won't enable without it)
+### Rule 3: API Keys via Environment
+- Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
+- Hardcoded keys in config.yaml become stale after key rotation
+- `/etc/environment` persists across config updates
+- Restart Hermes after env var updates
 
-### Rule 4: API Key Placement
-- **Custom provider keys** (`custom_providers[0].api_key`): Must be hardcoded (the LiteLLM master key or a virtual key)
-- **Model section keys** (`model.api_key`): Prefer empty. Workers inherit from custom_providers.
-- **Environment keys** (`api_key_env`): For fallback providers that need separate auth (DeepSeek, OpenRouter)
-- **Hardcoded keys in profile**: If a worker's `model.api_key` is set, it overrides custom_providers. Only use when the worker needs a DIFFERENT provider than Litellm.
+### Rule 4: Sub-Agent Profiles Inherit Auth
+- Sub-agent profiles (`/root/.hermes/profiles/*/config.yaml`) must have:
+  - `api_key: ''` — inherit from main config's custom_provider
+  - `base_url: ''` — inherit from main config
+  - Auxiliary tasks: `api_key: ''`, `provider: harness`
+- Never hardcode a key in sub-agent profiles
+- When main config uses `api_key_env`, sub-agents automatically use it
+- This means key rotation only touches ONE file (`/etc/environment`)
 
-### Rule 5: Verify After Every Config Change
-After changing a profile's config:
-1. `curl` the model endpoint to confirm auth works
-2. `curl` the web stack endpoints (Firecrawl, SearXNG)
-3. Check that `hermes gateway --restart` reloads cleanly
+### Rule 5: Main Config Base URL
+|- Use direct IP: `http://192.168.68.116/v1`
+|- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
+|- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly
+
+### Rule 6: max_tokens Is Required (Thermal Safety)
+- **Every Hermes config MUST set `model.max_tokens: 4096`** — this is non-negotiable
+- Prevents unbounded generation that caused the July 2 Strix Halo GPU thermal incident
+- The value flows through: `config → agent.max_tokens → transport build_kwargs → API max_tokens`
+- Even though the server now has `-n 8192` hard cap (set by Abiba), the client cap is the first line of defense
+- Apply to BOTH main config AND all sub-agent profiles
+- For agents needing longer outputs: raise to 8192, but never omit
+
+### Rule 7: Auxiliary Model Consistency
+- All auxiliary services (vision, web_extract, compression) MUST use the same model:
+  - `model: gemma-4-12b`
+  - `base_url: http://192.168.68.116/v1`
+  - `api_key_env: LITELLM_API_KEY`
+- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes to the primary 35B reasoning GPU
+- gemma-4-12b is a lightweight 12B model on the RTX 5070, keeping the Strix Halo free for reasoning
+- The `compression:` block's `model` MUST match `auxiliary: compression: model` — they are two different configs for the same service
+
+### Rule 8: Compression Threshold for 256K Models
+- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
+- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
+- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
+- `max_context_window: 262144` MUST match the model's actual capacity
+- See `devops-hermes-compression` skill for full reference
+
+### Rule 9: Default Model Must Be `syslog-auto` (All Agents)
+- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
+- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
+- `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b
+  and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
+  - Model name typos that cause 403 errors and silent worker failures
+  - Single GPU downtime (routing falls back automatically)
+  - Key/model authorization mismatches
+- **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
+  for specialized tasks, but MUST validate those models exist in the key's authorized list
+
+### Rule 10: Validate Model IDs Before Deployment (pi Agents)
+- After configuring a pi agent's `models.json`, verify every model ID:
+  ```bash
+  curl -s http://192.168.68.116:4000/v1/models \
+    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
+  ```
+- All model IDs in `models.json` MUST appear in the LiteLLM response
+- The agent's API key may have a SUBSET of the full model catalog — check per-key
+- A non-existent model ID causes 403 errors that silently break the pi RPC worker
+  (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
 
 ## Execution
 
 1. **Check current config** — Read the target agent's config.yaml
 2. **Compare against template** — Identify missing or divergent sections
-3. **Apply shared infra** — Lock the web/MCP/compression sections to template values
-4. **Apply model choice** — Set default/fallback/auxiliary models per agent's workload
-5. **Preserve agent-specific** — Keep platform configs, plugins, terminal settings, skills
-6. **Remove stale** — Drop any old infra endpoints that don't match the template
-7. **Verify** — curl all shared endpoints, test the model
-8. **Report** — What was changed, what was preserved, what's still custom
-
-## Example Output
-
-```
-## Config Template Applied: syslog-code ✅
-
-### Shared Infrastructure (Locked)
-| Section | Before | After |
-|---|---|---|
-| web.backend | firecrawl | firecrawl ✅ |
-| web.search_backend | NOT SET | searxng ✅ |
-| web.firecrawl.base_url | NOT SET | http://192.168.68.7:3002/ ✅ |
-| mcp_servers.ra-h-os | present | present ✅ |
-
-### Agent Model Choices (Preserved)
-| Setting | Value |
-|---|---|
-| model.default | qwen3.6-27B-code (code agent) |
-| fallback | gemma-4-12b (harness) |
-| auxiliary | gemma-4-12b (harness) |
-| compression | gemma-4-12b (harness) |
-
-### Endpoint Verification
-| Endpoint | Status |
-|---|---|
-| Firecrawl :3002 | ✅ 200 OK |
-| SearXNG :8888 | ✅ 200 OK |
-| LiteLLM | ✅ 200 OK |
-| RA-H OS MCP |  ✅ Connected |
-```
+3. **Apply shared infra** — Lock web/MCP/compression sections to template values
+4. **Apply agent key** — Set from agent_keys table above
+5. **Set model choice** — Per agent's workload
+6. **Verify** — curl all shared endpoints, test the model with the new key
+7. **Report** — What was changed, preserved, custom

hermes-key-enforcement.prose.md

@@ -0,0 +1,225 @@
+---
+kind: enforcement
+name: hermes-key-enforcement
+version: 1.0.0
+description: >
+  Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
+  providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
+  Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each
+  agent host. Designed to make key rotation a one-step operation.
+author: Abiba (pi agent)
+---
+
+# Hermes Key Enforcement Contract
+
+## Rule (One Sentence)
+
+**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.**
+
+## Scope
+
+Applies to all Hermes agent configs across all hosts. Covers these config sections:
+- `model.api_key`
+- `custom_providers[].api_key` (when `name` contains `harness` or `litellm`)
+- `auxiliary.*.api_key` (when `provider` is `harness` or contains `litellm`)
+- `delegation.api_key` (when `provider` is `harness` or contains `litellm`)
+- `compression.api_key` (when `provider` is `harness` or contains `litellm`)
+- `fallback_providers[].api_key` (when provider is harness)
+
+## Exemptions
+
+External providers are **explicitly exempt** and may use hardcoded keys:
+- DeepSeek (`api.deepseek.com`)
+- OpenAI (`api.openai.com`)
+- Anthropic (`api.anthropic.com`)
+- OpenRouter
+- Any provider whose base_url does NOT match `192.168.68.116` or `litellm.sysloggh.net`
+
+## Standard Pattern
+
+```yaml
+# ✅ CORRECT — all harness/litellm providers
+model:
+  provider: harness                    # or custom:litellm.sysloggh.net
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY        # ← indirection
+
+custom_providers:
+  - name: harness
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY      # ← indirection
+
+auxiliary:
+  compression:
+    provider: harness
+    api_key_env: LITELLM_API_KEY      # ← indirection
+
+# ✅ ALSO CORRECT — external providers
+fallback_providers:
+  - provider: deepseek
+    base_url: https://api.deepseek.com
+    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in /etc/environment
+```
+
+```yaml
+# ❌ FORBIDDEN — hardcoded harness/litellm key
+model:
+  provider: harness
+  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION
+```
+
+## Detection Query
+
+Run on any Hermes host to detect violations:
+
+```bash
+# 1. Check config.yaml for hardcoded harness keys
+grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
+  --include='config.yaml' \
+  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
+
+# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
+grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
+
+# 3. Verify running process env matches dedicated key
+cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
+  | tr '\0' '\n' | grep LITELLM_API_KEY
+```
+
+If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
+
+## Rotation Procedure
+
+With this standard enforced, key rotation is one step:
+
+```bash
+# 1. Generate new key in LiteLLM
+# 2. Update /etc/environment on agent host
+ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment"
+# 3. Restart agent gateway
+ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &"
+# 4. Verify
+curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
+```
+
+**Done.** No config file changes needed. The agent picks up the new key on restart.
+
+## Key Longevity Policy (2026-07-04)
+
+**Keys are permanent and use bare agent name aliases.**
+
+- **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
+- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
+- **Max budget**: $100 per key (config default).
+
+```yaml
+# In litellm_config.yaml — ensures all future keys inherit these defaults:
+litellm_settings:
+  default_key_generate_params:
+    models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b"]
+    duration: null        # ← permanent
+    max_budget: 100
+    metadata:
+      purpose: "agent-inference"
+```
+
+## Verified Agents (2026-07-05 update)
+
+| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
+|-------|-----|-----|---------------|-----|--------|----------------|---------------|
+| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
+| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
+| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
+| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
+| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
+
+### Systemd Service Pattern (2026-07-04 fix)
+
+All Hermes agents use systemd to manage their gateway. Two issues were fixed:
+
+1. **Drop-in override** — `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` (or user equivalent) had hardcoded `LITELLM_API_KEY` that bypassed `/etc/environment`.
+2. **Missing EnvironmentFile** — Services did not source `/etc/environment`.
+
+**Correct pattern:**
+```ini
+# In service file:
+EnvironmentFile=/etc/environment
+
+# Drop-in only for overrides, NOT primary key storage.
+# If a drop-in exists, it must match /etc/environment.
+```
+
+**Rotation procedure** (one step with this standard):
+1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
+2. Update `/etc/environment`: `sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment`
+3. Update drop-in (if exists): same sed on `litellm-key.conf`
+4. Restart: `systemctl [--user] restart hermes-gateway`
+
+## Violation Response
+
+1. **Detect** — run detection query above
+2. **Fix** — replace `api_key: sk-...` with `api_key_env: LITELLM_API_KEY` in all harness/litellm sections
+3. **Verify** — `grep -c "api_key_env" config.yaml` should increase, hardcoded harness keys should be 0
+4. **Restart** — gateway must restart to pick up env var
+5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
+6. **Update** — bump the verified table above
+7. **Use safe-mutate** — if the fix requires changing `/etc/environment` or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
+
+## Related Contracts
+
+- `hermes-config-template.prose.md` — full configuration template
+- `litellm-health.prose.md` — LiteLLM stack health verification
+- `zulip-platform-verification.prose.md` — cross-platform agent verification
+- `litellm-api-keys.prose.md` — API key creation, rotation, and verification
+
+## CI Pipeline (2026-07-04)
+
+All contract changes must pass the PR Pipeline before merge:
+
+```
+auth → validate → lint → ai-review → gate
+```
+
+- **Trigger**: push to master (abiba-bot only) or pull request
+- **Branch protection**: Only `abiba-bot` can push directly to master. All other users must use PRs.
+- **Status check**: `PR Pipeline — Authorize → Validate → Review → Merge` required before merge
+- **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
+- **Config**: `.gitea/workflows/pr-pipeline.yaml`
+
+## Known Bug: auxiliary_client ignores api_key_env (2026-07-05)
+
+**Bug**: `_resolve_task_provider_model()` in `agent/auxiliary_client.py` reads
+`api_key` from auxiliary task configs (vision, compression, etc.) but does NOT
+resolve `api_key_env`. The custom provider resolution path handles `api_key_env`,
+but auxiliary tasks take a different code path that ignores it.
+
+**Impact**: Vision analysis and compression calls fall through to the `"no-key-required"`
+placeholder, causing 401 errors on LiteLLM/harness (which require `sk-*` keys).
+
+**Workaround**: Set `api_key` directly alongside `api_key_env` in each auxiliary
+task config:
+
+```yaml
+auxiliary:
+  vision:
+    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
+    api_key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/v1
+    model: gemma-4-12b
+    provider: harness
+  compression:
+    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
+    api_key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/v1
+    model: gemma-4-12b
+    provider: harness
+```
+
+**Affected agents**: All Hermes agents with harness/LiteLLM provider and
+`api_key_env` in auxiliary configs (all 4 Hermes agents patched 2026-07-05).
+
+**Source location**: `agent/auxiliary_client.py` line 5478

hermes-zulip-plugin.prose.md

@@ -0,0 +1,237 @@
+---
+kind: function
+name: hermes-zulip-plugin
+description: >
+  Installs or updates the Zulip platform plugin for any Hermes agent from the
+  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
+  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
+  recovery). Verifies the installation, restarts the gateway, and sends a relay
+  success signal.
+agent: abiba
+version: 1.0.0
+status: active
+runtime_contract: 2
+---
+
+# Hermes Zulip Plugin — Install & Repair
+
+Single-shot function that pulls the latest zulip-platform plugin from the
+canonical git repo, installs it to the correct Hermes bundled plugin path,
+verifies the installation, and signals completion.
+
+Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
+layer and a targeted gateway restart. It does NOT verify env credentials or
+run a live Zulip connection test. Use `hermes-zulip-restore` for full
+connectivity recovery including end-to-end DM validation.
+
+## Parameters
+
+| Param | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
+| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
+
+## Maintains
+
+- plugin_installed: bool — Whether all three adapter files exist at the bundled path
+- plugin_version: string — Git commit SHA of the installed version
+- strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
+- signal_sent: bool — Whether relay success message was dispatched
+
+### Postconditions
+
+- All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
+- `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
+- Installed version matches HEAD of the requested branch
+- Relay success signal sent to Hermes agent's inbox
+
+## Requires
+
+- SSH access to target host (direct or via amdpve for CTs)
+- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
+- Python 3 with `httpx` installed on target
+
+## Live-State Fields
+
+| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
+|------|-----|---------|-------------|-------------|------|
+| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
+| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
+
+| Field | Value | Trust |
+|-------|-------|-------|
+| Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
+| Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
+| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
+| Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
+| Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
+
+## Execution
+
+### Step 1: Resolve Target
+
+Map `target` to host, CT ID, hermes_home, and user from the live-state table.
+For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
+
+### Step 2: Pull Latest Plugin Source
+
+On the target host:
+
+```bash
+# Ensure deploy scratch space
+mkdir -p /tmp/zulip-deploy
+cd /tmp/zulip-deploy
+
+# Clone or pull
+if [ -d zulip-platform-plugins ]; then
+  cd zulip-platform-plugins
+  git fetch origin
+  git checkout {{branch}}
+  git pull origin {{branch}}
+else
+  git clone --branch {{branch}} \
+    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
+  cd zulip-platform-plugins
+fi
+
+# Capture installed version
+INSTALLED_SHA=$(git rev-parse HEAD)
+echo "Installed SHA: $INSTALLED_SHA"
+
+# Verify we're at HEAD
+HEAD_SHA=$(git rev-parse origin/{{branch}})
+if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
+  echo "At latest commit on {{branch}}"
+else
+  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
+fi
+```
+
+### Step 3: Install Plugin Files
+
+```bash
+# Ensure target directory exists
+mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
+
+# Copy adapter files
+cp plugins/platforms/zulip/adapter.py \
+   plugins/platforms/zulip/__init__.py \
+   plugins/platforms/zulip/plugin.yaml \
+   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+
+# Fix ownership (Tanko only — runs as jerome user)
+[ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
+  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+
+echo "Plugin files installed"
+```
+
+### Step 4: Verify Installation
+
+```bash
+# Check all three files exist
+for f in __init__.py adapter.py plugin.yaml; do
+  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
+    echo "✅ $f present"
+  else
+    echo "❌ $f MISSING"
+    exit 1
+  fi
+done
+
+# Verify _strip_html fix
+grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
+  && echo "✅ _strip_html fix present" \
+  || echo "❌ _strip_html MISSING — plugin may be stale"
+
+# Show installed plugin.yaml version
+grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
+```
+
+### Step 5: Restart Gateway
+
+Plugin changes require a gateway restart to take effect:
+
+```bash
+cd {{hermes_home}}/hermes-agent
+# Use venv if available
+python3 -m hermes_cli.main gateway restart 2>&1 || \
+  venv/bin/python -m hermes_cli.main gateway restart 2>&1
+```
+
+Wait for restart to complete (up to 45s), then confirm:
+
+```bash
+grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
+```
+
+Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
+
+Quick smoke check — confirm zulip platform loaded:
+
+```bash
+grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
+```
+
+If gateway fails to restart, check logs for the crash cause before proceeding.
+
+### Step 6: Send Relay Success Signal
+
+On the Abiba host (local), dispatch a relay message to the target agent:
+
+```
+ra-h-os-createRelayNode:
+  title: "Zulip plugin updated — {{target}}"
+  source: |
+    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
+    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+    _strip_html fix: PRESENT
+    Timestamp: {{timestamp}}
+
+  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
+```
+
+### Step 7: Report
+
+Compile results into a single status block:
+
+| Field | Value |
+|-------|-------|
+| Target | `{{target}}` |
+| Branch | `{{branch}}` |
+| Commit SHA | `{{INSTALLED_SHA}}` |
+| Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
+| `_strip_html` | `{{present|missing}}` |
+| Gateway restarted | `{{yes|no}}` |
+| Signal sent | `{{yes|no}}` |
+
+## Known Failure Modes
+
+| Symptom | Root Cause | Recovery |
+|---------|-----------|----------|
+| Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
+| Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
+| `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
+| Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
+| Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
+
+## Edge Differences from hermes-zulip-restore
+
+| Concern | hermes-zulip-restore | hermes-zulip-plugin |
+|---------|---------------------|---------------------|
+| Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
+| Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
+| Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
+| Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
+| Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
+| Signal dispatch | ❌ None | ✅ Relay message to target |
+
+For full connectivity recovery after a plugin install, chain this contract
+with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
+
+---
+
+**Last updated**: 2026-07-08 — Switched default branch to `master`; added
+gateway restart step. If outstanding unmerged feature branches exist, address
+them in a follow-up merge after this contract completes.

hermes-zulip-restore.prose.md

@@ -0,0 +1,188 @@
+---
+kind: function
+name: hermes-zulip-restore
+description: >
+  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
+  Koby CT111). Deploys the zulip-platform adapter to the correct bundled plugin
+  path, verifies env credentials, restarts the gateway, and confirms Zulip
+  connects. Run this whenever a Hermes agent stops responding on Zulip or after
+  a fresh agent deployment.
+agent: abiba
+version: 1.0.0
+status: active
+runtime_contract: 2
+---
+
+# Hermes Zulip Restore — Bring Any Agent Back to Good State
+
+Single-shot function that restores full Zulip connectivity for a Hermes agent.
+Covers adapter deployment, HTML stripping (slash command fix), env verification,
+gateway restart, and connection validation.
+
+## Parameters
+
+| Param | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
+
+## Maintains
+
+- adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
+- zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
+- env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
+- gateway_running: bool — Whether gateway process is running
+
+### Postconditions
+
+- `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
+- All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
+- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
+- Gateway restarted and zulip platform reports state `connected`
+- HTML stripping enabled for `/approve` and `/deny` slash command support
+
+## Requires
+
+- SSH access to target host (direct or via amdpve for CTs)
+- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
+- Python 3 with `httpx` installed on target
+- Zulip server accessible at `https://chat.sysloggh.net`
+
+## Live-State Fields
+
+| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
+|------|-----|---------|-------------|-------------|------|
+| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
+| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
+
+| Field | Value | Trust |
+|-------|-------|-------|
+| Zulip server | https://chat.sysloggh.net | ✅ Verified |
+| Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
+| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
+| Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
+
+## Execution
+
+### Step 1: Locate Target
+
+Map `target` to connectivity parameters from the live-state table above.
+For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
+
+### Step 2: Deploy Zulip Adapter
+
+On the target host:
+
+```bash
+# Clone or update the plugin repo
+mkdir -p /tmp/zulip-deploy
+cd /tmp/zulip-deploy
+if [ -d zulip-platform-plugins ]; then
+  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
+else
+  git clone --branch feat/zulip-streaming \
+    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
+fi
+
+# Ensure bundled plugin directory exists
+mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
+
+# Copy adapter files
+cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
+   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
+   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
+   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
+
+# Fix ownership (Tanko only)
+chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
+
+# Clean up
+rm -rf /tmp/zulip-deploy
+```
+
+### Step 3: Verify _strip_html is Present
+
+```bash
+grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
+```
+Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
+
+### Step 4: Verify Env Credentials
+
+```bash
+grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
+```
+
+Expected: all three variables set with non-empty values. If any missing:
+- ZULIP_SITE: `https://chat.sysloggh.net`
+- ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
+- ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
+
+### Step 5: Restart Gateway
+
+```bash
+cd <HERMES_HOME>/hermes-agent
+# Use venv if available
+python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
+```
+
+Wait for the restart to complete (up to 45s). Check:
+
+```bash
+grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
+```
+
+Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
+
+### Step 6: Validate Zulip Connection
+
+```bash
+python3 -c "
+import json
+d = json.load(open('$HERMES_HOME/gateway_state.json'))
+print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
+"
+```
+
+Expected: `zulip: connected`. If not connected, check gateway log:
+
+```bash
+grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
+```
+
+### Step 7: Report
+
+Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
+
+| State | Action |
+|-------|--------|
+| All true | ✅ Agent restored — relay success to user |
+| `adapter_deployed: false` | Re-run Step 2 |
+| `env_valid: false` | Prompt for missing credentials |
+| `zulip_connected: false` | Check Zulip server reachability, verify API key |
+| `gateway_running: false` | Check process logs for crash cause |
+
+## Known Failure Modes
+
+| Symptom | Root Cause | Recovery |
+|---------|-----------|----------|
+| Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
+| Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
+| No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
+| `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
+| Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
+| Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
+
+## Git Branch Reference
+
+The `_strip_html` fix lives on `feat/zulip-streaming` branch:
+```
+https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
+```
+
+Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
+Pull request #33 is the primary integration branch.
+
+---
+
+**Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.

infrastructure-control.prose.md

@@ -6,6 +6,12 @@ description: >
   5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
   NFS storage, and network services. Defines monitors, remediations,
   and the access matrix for all environments.
+
+  FIELD TRUST: Live-state fields (IPs, ports, hostnames, credentials,
+  container names, PIDs) are marked VERIFY-BEFORE-USE. They drift —
+  never mutate infrastructure based on them without first confirming
+  against the live system. Policy fields are authoritative. See the
+  `verify-before-mutate` skill.
 ---
 
 # Infrastructure Control Pattern
@@ -23,7 +29,7 @@ description: >
    ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
    │   Abiba     │    │   Tanko      │    │   Mumuni     │
    │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
-   │   CT 100    │    │   CT 122     │    │   CT 114     │
+   │   CT 100    │    │   CT 112     │    │   CT 114     │
    └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
           │                  │                    │
           └──────────────────┼────────────────────┘
@@ -60,19 +66,30 @@ description: >
 | Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
-| Tanko CT (.122) | SSH jerome | SSH key | ✅ |
+| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
+| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
+| Baggy CT (113) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
 | Gitea | API token | abiba-bot token | ✅ |
-| Zulip | Bot API key | per-bot tokens | ✅ |
+| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
+| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ VERIFY-BEFORE-USE |
+| Grafana | admin password | `syslog-grafana-2026` | ✅ VERIFY-BEFORE-USE |
+
+> **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
+> contract are live-state fields. Test them against the live system before
+> relying on them (e.g., `curl -H "Authorization: Bearer $KEY" .../v1/models`
+> for a LiteLLM key; `pct config <ct>` for a CT IP). Drift is expected —
+> see `verify-before-mutate` skill. Policy fields (rules, doctrines) are
+> authoritative and do not need verification.
 
 ### Reachability Matrix
 
-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) |
-|-----------|---------|----------------|--------|-------------|---------------|
-| **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird |
-| **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ |
-| **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
+|-----------|---------|----------------|--------|-------------|---------------|----------------|
+| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
+| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
+| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
 
 **Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.
 
@@ -127,35 +144,128 @@ description: >
 
 ### Ecosystem A: docker-vm (192.168.68.7)
 
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 
 | Stack | Path | Containers |
 |-------|------|-----------|
 | **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
-| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse |
+| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
+| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
+
+**Last verified:** 2026-07-09 — 16/16 containers running healthy.
+
+### Home Stack Services
+
+**Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
+- URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
+- Swagger: `http://192.168.68.7:8989/swagger-ui.html`
+- Admin credentials: `admin` / `kakashi20stirling`
+- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
+- Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
+- Compose: `/opt/home_stack/docker-compose.yml`
+- Control script: `/opt/home_stack/infra-control.sh`
+
+**JDownloader**:
+- URL: `http://192.168.68.7:5800` (web UI via VNC)
+
+**Pulse** (Uptime Kuma):
+- URL: `http://192.168.68.7:3001`
 
 ### Ecosystem B: CT 116 syslog-api (192.168.68.116)
 
-| Container | Image | Port |
-|-----------|-------|------|
-| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 |
-| harness-nginx | nginx:alpine | :80 |
-| harness-router | inference-harness-router | :9000 |
-| harness-postgres | postgres:16-alpine | :5432 |
-| harness-redis | redis:7-alpine | :6379 |
-| harness-dashboard | inference-harness-dashboard | :3000 |
+8 containers in inference-harness stack:
 
-### Ecosystem C: Netbird (72.61.0.17)
+| Container | Image | Port | Role |
+|-----------|-------|------|------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
+| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
+| harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
+| harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
+| harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
+| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
+| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
+
+**Nginx routing**:
+- `/v1/*` → harness-litellm:4000 (API)
+- `/admin/*` → harness-litellm:4000
+- `/dashboard/` → harness-dashboard:3000
+- `/litellm/*` → harness-litellm:4000
+- `/health/*` → harness-litellm:4000/health/liveliness
+- `/gpu/` → 192.168.68.24:9100 (fleet monitor)
+
+**Prometheus targets**:
+- 192.168.68.8:9400 (RTX 3090 — qwen)
+- 192.168.68.110:9400 (RTX 5070 — gemma)
+- 192.168.68.15:9400 (Strix Halo — ornith)
+- 192.168.68.24:9401 (Router metrics exporter)
+- harness-litellm:4000 (LiteLLM health)
+
+### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)
+
+5 containers, single compose stack at `/root/docker-compose.yml`:
 
 | Container | Image | Role |
 |-----------|-------|------|
-| netbird-server | netbird | VPN controller |
-| netbird-dashboard | netbird UI | Web management |
-| netbird-proxy | nginx | TLS termination |
-| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF |
-| netbird-traefik | traefik | Reverse proxy |
+| netbird-server | netbirdio/netbird-server:latest | VPN controller, management API |
+| netbird-dashboard | netbirdio/dashboard:latest | Web management UI |
+| netbird-proxy | netbirdio/reverse-proxy:latest | SNI router + TLS passthrough for *.sysloggh.net |
+| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF / threat intelligence |
+| netbird-traefik | traefik:v3.6 | Edge reverse proxy, Let's Encrypt TLS |
+
+**Architecture:** Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy
+(TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends.
+Traefik handles the Netbird dashboard directly; all other `*.sysloggh.net`
+domains are TLS-passthrough to the proxy on port 8443.
+
+**Proxy config:** `/root/proxy.env` — token, ACME certs in `/certs/`, geolocation
+DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.
+
+> **Dependency note:** NetBird is an **access layer only** — it exists so a
+> human on a PC or away from the LAN can reach services by URL. It is **not**
+> a dependency of any service backend. Per Section 7, every config and agent
+> must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for
+> human browsers.
+
+#### Known Issues
+
+**A. Access log bloat (CRITICAL)** — 2026-07-03 discovery. The management server
+writes an `access_log_entries` row for every proxied request (~45K/day). SQLite
+has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:
+- DB writes slow → gRPC deadlines exceeded → proxy loses management connection
+- On proxy restart, management server too slow to push route mappings
+- Proxy starts HTTPS listener before routes arrive → all domains "unknown"
+- **Fix:** batch-delete old entries, VACUUM. Prevention: weekly systemd timer (`netbird-cleanup.timer`) deletes entries >7 days old when DB exceeds 500MB.
+- **Recurrence signal:** proxy logs show `unknown domain "git.sysloggh.net"` or `match: false` for known domains → management route sync failed.
+
+**B. Stale management state after proxy restart.** Management server sometimes
+stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts,
+mapping stream established, but no routes arrive → all domains `unknown`.
+**Fix:** restart management server FIRST, THEN proxy. Order matters.
+
+**C. Peer `sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE=` flapping.**
+One peer (connecting through Traefik relay at 172.30.0.10) disconnects every
+~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to
+slow DB writes) rather than a network issue. Monitor after bloat fix.
+
+#### Recovery Procedures
+
+1. **Proxy losing all routes ("unknown domain" for all services):**
+   ```bash
+   ssh root@72.61.0.17
+   cd /root
+   docker compose restart netbird-server
+   sleep 12
+   docker compose restart proxy
+   ```
+2. **Access log bloat detected (DB > 500MB):**
+   ```bash
+   ssh root@72.61.0.17
+   /root/cleanup-access-logs.sh
+   ```
+3. **VPS hung (TCP accepts, SSH banner timeout):** hard reboot from Hostinger console.
 
 ### Checks (every 60s)
 
@@ -173,6 +283,13 @@ For each Docker host:
   - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
   - df -h / | awk '{print $5}' → usage < 80%
 
+For Netbird VPS specifically:
+  - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
+  - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
+  - If DB > 500MB: alert + run /root/cleanup-access-logs.sh
+  - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
+  - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart
+
 For docker-vm specifically:
   - mountpoint -q /media/storage → NFS mounted
   - mountpoint -q /media/mediastore → NFS mounted
@@ -226,25 +343,103 @@ For docker-vm specifically:
 
 ## Section 5: Network Services — Monitoring
 
-| Service | Domain | Status |
-|---------|--------|--------|
-| Proxmox API | minipve.sysloggh.net:443 | ✅ |
-| Authentik | auth.sysloggh.net:443 | ✅ |
-| Gitea | git.sysloggh.net:443 | ✅ |
-| Zulip | chat.sysloggh.net:443 | ✅ |
-| LiteLLM | litellm.sysloggh.net:443 | ✅ |
-| Pulse | pulse.sysloggh.net:443 | ✅ |
-| SearXNG | searxng.sysloggh.net:8888 | ✅ |
-| Firecrawl | firecrawl.sysloggh.net:3002 | ✅ |
+### 5.1 Service Inventory
 
-### Checks (every 2 min)
+The `Resolves To` column is the load-bearing one. Services that CNAME to
+`netbird.sysloggh.net` (72.61.0.17) are routed through the NetBird reverse
+proxy and **fail whenever NetBird blips**, even though their LAN backend is
+fine. Services that resolve directly to a LAN IP are NetBird-independent.
+
+| Service | Domain | LAN Backend | Resolves To | NetBird-dep | Status |
+|---------|--------|-------------|-------------|-------------|--------|
+| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
+| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
+| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
+| Gitea | git.sysloggh.net:443 | 192.168.68.110 | CNAME → netbird | **Yes** | ⚠️ |
+| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
+| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
+| DNS UI | dns.sysloggh.net:443 | 192.168.68.102 | CNAME → netbird | **Yes** | ⚠️ |
+| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
+| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
+
+**Verified 2026-07-02:** NetBird VPS rebooted after a hang; all CNAME'd
+services recovered. LAN-IP-direct paths stayed up throughout the outage.
+
+### 5.2 Checks (every 2 min)
 
 ```
 ## Maintains
 
-- dns-resolution: { services: [{name, resolves}] }
+- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
 - ssl-expiry: { services: [{name, days_remaining}] }
 - endpoint-reachability: { services: [{name, http_code}] }
+- netbird-dependency: { services: [{name, depends_on_netbird: bool}] }
+```
+
+### 5.3 Network Verification & Routing (every 5 min)
+
+This block is what catches a NetBird blip before it becomes an outage. It
+proves two independent paths for every service: the **URL path** (what a
+browser uses, may cross NetBird) and the **LAN-IP path** (what configs and
+agents must use, never crosses NetBird).
+
+```
+## Maintains
+
+- dual-path-reachability: {
+    services: [{
+      name: string,
+      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
+      lan_path:  { ip, port, http_code },        # the IP:port path
+      lan_independent: bool                       # lan_path works when url_path fails
+    }]
+  }
+- netbird-ingress-health: {
+    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
+    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
+    containers_up: int,        # all 5 netbird-* containers running
+    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
+  }
+- routing-regression: {
+    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
+                                    # → alert: a service just became NetBird-dependent
+    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
+                                    # URL where a LAN IP is required (see Section 7)
+  }
+
+## Checks
+
+- For each service in 5.1:
+  - curl https://<domain>/ → record url_path (code, tls_ok)
+  - curl http://<lan_ip>:<port>/ → record lan_path
+  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
+  - If BOTH fail → service-down (real outage)
+
+- NetBird ingress (72.61.0.17):
+  - TCP probe 22, 80, 443 → all accept
+  - SSH banner exchange completes within 10s (catches the hung-box symptom
+    where TCP accepts but sshd never sends its banner)
+  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
+  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)
+
+- DNS dependency drift:
+  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
+  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
+  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)
+
+## Remediations
+
+- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
+  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
+  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
+  → Do NOT declare backend services down; their LAN-IP paths are still up.
+- Single proxy route missing (e.g. git returns 000 but chat ok):
+  → ssh root@72.61.0.17 'docker restart netbird-traefik'
+  → re-check after 30s
+- New NetBird CNAME detected:
+  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
+- Config URL violation detected:
+  → Alert with file + line; do not auto-edit configs
 ```
 
 ## Section 6: Alert Routing & Escalation
@@ -281,6 +476,63 @@ ZFS pool degraded
   → Cannot auto-fix → Escalate immediately
 ```
 
+## Section 7: Configuration Doctrine — IP-First, URLs for Browsers
+
+This doctrine is the structural fix for the NetBird-dependency outage. It is
+enforced by the `routing-regression.config_url_violations` check in Section
+5.3.
+
+### Rule
+
+1. **Service configs use LAN IP addresses, always.** Any file that wires a
+   service, agent, monitor, or integration to another internal service must
+   reference the backend by its `192.168.68.x` LAN IP and port — never by a
+   `*.sysloggh.net` URL. NetBird may blip at any time; LAN IPs do not.
+2. **`*.sysloggh.net` URLs are reserved for human browser access.** They are
+   a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy
+   off the LAN) for a person at a keyboard. They must never be a load-bearing
+   dependency in code or config.
+3. **Agents and monitors reach backends by LAN IP.** Abiba, Tanko, and Mumuni
+   all run on the LAN; there is no reason for them to hairpin through NetBird.
+   The daily infra report and all `ssh`/`curl` checks already follow this.
+4. **DNS records should prefer LAN IPs over NetBird CNAMEs.** A service that
+   can resolve directly to its LAN IP (like `litellm` → `192.168.68.116` and
+   `minipve` → `192.168.68.12`) is NetBird-independent. Moving a record from
+   a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
+5. **NetBird is for remote/PC access only.** When you are out or on a PC,
+   NetBird carries you in. When you are on the LAN, NetBird is not in the
+   path. Configs must reflect this — they always assume LAN.
+
+### Canonical IP map (configs MUST use these)
+
+| Service | Use in configs | URL (browsers only) |
+|---------|----------------|--------------------|
+| Proxmox API | `https://192.168.68.12:8006` | `https://minipve.sysloggh.net` |
+| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
+| LiteLLM (nginx) | `http://192.168.68.116` | — |
+| Grafana | `http://192.168.68.116:3001` | — |
+| Authentik | `https://192.168.68.11` | `https://auth.sysloggh.net` |
+| Gitea | `http://192.168.68.110:3000` | `https://git.sysloggh.net` |
+| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
+| SearXNG | `http://192.168.68.7:8888` | — |
+| Firecrawl | `http://192.168.68.7:3002` | — |
+| RA-H OS bridge | `http://192.168.68.65:3100` | — |
+
+### Violation examples (to be flagged by Section 5.3)
+
+- ❌ An agent config setting `LITELLM_BASE_URL=https://litellm.sysloggh.net`
+- ✅ Same config setting `LITELLM_BASE_URL=http://192.168.68.116:4000`
+- ❌ A monitor curling `https://chat.sysloggh.net/api/v1/server_settings`
+- ✅ A monitor curling `http://192.168.68.19/api/v1/server_settings` (and
+      optionally the URL as a separate browser-path check)
+
+### Goal state
+
+Every `*.sysloggh.net` record resolves to a LAN IP (split-horizon DNS on LAN,
+NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN
+ingress — if it blips, only remote browser users notice, and no agent,
+monitor, or integration breaks.
+
 ## Appendix A: Quick Health Commands
 
 ```bash
@@ -293,8 +545,30 @@ curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"
 ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
 ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"
 
+# GPU fleet quick check
+curl -s http://192.168.68.116/health/unified | jq .status
+curl -s http://192.168.68.24:9100/gpu-data | jq .summary
+
+# Grafana status
+curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health
+
+# Prometheus targets
+curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
+
+# LiteLLM key check
+curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
+  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
+
 # Storage check
 ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
+
+# Router roster reload (if needed)
+curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
+  -H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"
+
+# Restart stuck GPU (saturation watchdog alternative)
+ssh root@192.168.68.8 "systemctl restart llama-server"
+ssh root@192.168.68.110 "systemctl restart llama-server"
 ```
 
 ## Appendix B: CT Inventory
@@ -302,9 +576,9 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | CT | Name | Node | IP | Role | Agent |
 |----|------|------|----|------|-------|
 | 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
-| 101 | llm-gpu | acerpve | — | GPU VM | ❌ |
+| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
 | 102 | adguard | acerpve | — | DNS | ❌ |
-| 103 | ocu-llm | ocupve | — | GPU VLM | ❌ |
+| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
 | 104 | authentik | minipve | .11 | OIDC | ❌ |
 | 105 | kagentz | amdpve | — | Agent Zero | ✅ |
 | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
@@ -314,10 +588,10 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | 110 | gitea | minipve | — | Git | ❌ |
 | 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy | amdpve | — | ? | ❌ |
-| 114 | mumuni | minipve | — | Hermes agent | ✅ |
+| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
+| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | — | ? | ❌ |
-| 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ |
+| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
 | 117 | zulip | storepve | — | Chat | ❌ |
 | 118 | jitsi | minipve | — | Video | ❌ |
 
@@ -329,5 +603,61 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` |
 | docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` |
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
-| CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` |
+| CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |
+
+## CT Access (pct-run — no IPs needed)
+
+All CTs are accessible via `pct-run <CT_ID> <command>`. No IP addresses required.
+Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.sh`.
+
+| CT | Name | Node | pct-run |
+|-----|------|------|---------|
+| 100 | abiba | amdpve | `pct-run 100` |
+| 105 | kagentz | amdpve | `pct-run 105` |
+| 111 | tdunna | amdpve | `pct-run 111` |
+| 112 | tanko | amdpve | `pct-run 112` |
+| 113 | baggy | amdpve | `pct-run 113` |
+| 115 | scottdenya | amdpve | `pct-run 115` |
+| 104 | authentik | minipve | `pct-run 104` |
+| 110 | gitea | minipve | `pct-run 110` |
+| 114 | mumuni | minipve | `pct-run 114` |
+| 116 | syslog-api | minipve | `pct-run 116` |
+| 106 | ra-h-os | storepve | `pct-run 106` |
+| 107 | proxmox-backup | storepve | `pct-run 107` |
+| 108 | media | storepve | `pct-run 108` |
+| 117 | zulip | storepve | `pct-run 117` |
+| 102 | adguard | acerpve | `pct-run 102` |
+
+GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
+```bash
+ssh root@192.168.68.8   # RTX 3090
+ssh root@192.168.68.110 # RTX 5070
+ssh root@192.168.68.15  # Strix Halo
+```
+
+## Section 7: Agent Health Check (consolidated — 2026-07-05)
+
+Replaces 7 scattered Zulip health scripts with a single non-disruptive check
+running every 10 minutes via cron (`/root/scripts/agent-health-check.py`).
+
+The script is **read-only** — it never restarts, kills, or modifies anything.
+Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
+which was kill+nohup outside systemd) are banned by policy.
+
+### Checks Performed
+
+| Check | Frequency | What It Detects |
+|-------|-----------|-----------------|
+| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models |
+| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
+| Gateway liveness | 10 min | Gateway process running, state file readable |
+| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
+| Recent errors | 10 min | Error count in journald for last 10 min |
+
+### Disabled Scripts
+
+| Script | Why Disabled |
+|--------|-------------|
+| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
+| `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |

infrastructure-monitoring.prose.md

@@ -0,0 +1,158 @@
+---
+kind: function
+name: infrastructure-monitoring
+description: >
+  Deploys Prometheus + GPU exporters + Grafana to monitor the entire
+  inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
+  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
+  via existing /metrics Prometheus endpoint.
+  
+  DEPLOYMENT STATUS (2026-07-09):
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
+     NVIDIA sidecar exporters (.8/.110:9400) never installed.
+     Router falls back to direct GPU /health probes.
+  ⚠️ This contract is target-state aspirational — not as-built.
+     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
+version: 1.0.0
+---
+
+## Architecture
+
+```
+GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
+  nvidia-exporter       nvidia-exporter       amdgpu-exporter
+  :9400                 :9400                 :9400
+       │                     │                     │
+       └─────────────────────┼─────────────────────┘
+                             ▼
+              ┌──────────────────────────┐
+              │       Prometheus         │
+              │       CT 116 :9090       │
+              │                          │
+              │  Scrape targets:         │
+              │  • 192.168.68.8:9400     │
+              │  • 192.168.68.110:9400   │
+              │  • 192.168.68.15:9400    │
+              │  • litellm:4000/metrics  │
+              └──────────┬───────────────┘
+                         │
+              ┌──────────▼───────────────┐
+              │        Grafana           │
+              │       CT 116 :3001       │
+              │                          │
+              │  Preloaded dashboards:   │
+              │  • GPU Fleet Overview    │
+              │  • LiteLLM Proxy Stats   │
+              └──────────────────────────┘
+```
+
+## Components
+
+### 1. NVIDIA GPU Exporter (hosts: .8, .110)
+- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
+
+### 2. AMD GPU Exporter (host: .15)
+- Custom exporter: Python script wrapping `amdgpu_top --json`
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
+- Runs as systemd service for persistence
+
+### 3. Prometheus (CT 116)
+- Container: `prom/prometheus:latest`
+- Port: `9090` (internal Docker network)
+- Scrape interval: 15s
+- Config: `/opt/monitoring/prometheus.yml`
+- Storage: Docker volume `prometheus-data`
+
+### 4. Grafana (CT 116)
+- Container: `grafana/grafana:latest`
+- Port: `3001` (mapped to host)
+- Data source: Prometheus at `http://prometheus:9090`
+- Provisioned dashboards for GPU fleet + LiteLLM
+- Accessible at `http://192.168.68.116:3001`
+
+## Parameters
+
+- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
+- gpu_amd_hosts: ["192.168.68.15"]
+- monitoring_host: "192.168.68.116"
+- prometheus_port: 9090
+- grafana_port: 3001
+- gpu_exporter_port: 9400
+
+## Requires
+
+- SSH access to all GPU hosts for exporter deployment
+- Docker on CT 116 for Prometheus + Grafana containers
+- Python 3 on AMD host for custom exporter
+- nvidia-smi on NVIDIA hosts
+
+## Maintains
+
+- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
+- Prometheus scrapes all targets every 15s
+- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
+- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
+- Stack persists across reboots (systemd for exporters, Docker restart policy)
+
+## Execution
+
+### Phase 1: GPU Exporters
+
+**NVIDIA (.8 and .110)**:
+1. Download `nvidia_gpu_exporter` binary
+2. Create systemd service `nvidia-gpu-exporter.service`
+3. Start and enable
+
+**AMD (.15)**:
+1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
+2. Parses `amdgpu_top --json -d 1000` output
+3. Exposes key metrics at `:9400/metrics` via Python http.server
+4. Create systemd service
+5. Start and enable
+
+### Phase 2: Prometheus
+
+1. Create `/opt/monitoring/` directory on CT 116
+2. Write `prometheus.yml` with scrape configs for all targets
+3. Add to docker-compose (or separate compose file)
+4. Start container
+
+### Phase 3: Grafana
+
+1. Create `/opt/monitoring/grafana/` directories
+2. Provision Prometheus datasource
+3. Provision GPU fleet dashboard JSON
+4. Provision LiteLLM dashboard JSON
+5. Add to docker-compose
+6. Start container
+
+### Phase 4: Verification
+
+1. Verify all 3 GPU exporters return 200 at :9400/metrics
+2. Verify Prometheus targets all UP at :9090/targets
+3. Verify Grafana accessible at :3001 with dashboards
+4. Verify LiteLLM metrics flowing to Prometheus
+5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
+
+## Verification Commands
+
+```bash
+# GPU exporters
+curl -s http://192.168.68.8:9400/metrics | grep nvidia
+curl -s http://192.168.68.110:9400/metrics | grep nvidia
+curl -s http://192.168.68.15:9400/metrics | grep amdgpu
+
+# Prometheus
+curl -s http://192.168.68.116:9090/api/v1/targets
+
+# Grafana
+curl -s http://192.168.68.116:3001/api/health
+
+# LiteLLM metrics (already live)
+curl -s http://192.168.68.116:4001/metrics | head -20
+```

infrastructure-update.prose.md

@@ -0,0 +1,161 @@
+---
+kind: responsibility
+name: infrastructure-update
+description: >
+  Autonomous system-wide update contract covering all 5 Proxmox nodes,
+  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
+  Docker images, and container stacks in safe waves with health checks
+  and automatic rollback on failure.
+agent: abiba
+triggers:
+  - on "infra update" command
+  - weekly (Sunday 03:00 EDT) via cron
+  - on security advisory relay from Mumuni
+version: 1.1.0
+---
+
+## Maintains
+
+- update-status: { phase, node, action, result, timestamp }
+- update-history: array of past update runs with results
+- security-state: { cve_count, last_patched, pending_updates }
+
+## Pre-Flight Checklist
+
+Before ANY update wave:
+1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
+2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
+4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+5. ✅ LiteLLM health check passing
+6. ✅ Zulip server reachable
+6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
+7. ✅ Disk >20% free on all nodes
+8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
+
+## Wave 1: Storage & Infra Nodes (lowest impact)
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
+| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
+| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 1:**
+- Docker healthy on .7: `docker ps`
+- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
+- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`
+
+## Wave 2: Compute & Agent Nodes
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
+| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
+| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
+| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
+| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 2:**
+- All VMs/CTs running: check via Proxmox API
+- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
+- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- Zulip agents connected: check Mumuni/Tanko gateway state
+- Abiba PM2 processes online: `pm2 status`
+
+## Wave 3: Docker Image Updates
+
+| Host | Stack | Command |
+|------|-------|---------|
+| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
+
+**Verify after Wave 3:**
+- All containers healthy: `docker ps` on each host
+- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
+- Zulip test: send test message to #agent-hub
+- Dashboard loading: `curl localhost:3001/` (via CT 116)
+- Firecrawl test: `curl :3002/`
+- SearXNG test: `curl :8888`
+
+## Wave 4: Proxmox Kernel Reboot (if needed)
+
+Only if `[ -f /var/run/reboot-required ]` on any node.
+
+| Target | Action |
+|--------|--------|
+| Affected PVE node | Verify all CTs/VMs migrated or stopped |
+| | `reboot` via PVE API |
+| | Wait 120s for node to come back |
+| | Start any stopped CTs |
+
+## Rollback Protocol
+
+If ANY verification fails:
+
+1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
+2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
+3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
+4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails
+
+## Config Backup
+
+Before Wave 1, snapshot these files:
+```
+/opt/inference-harness/docker-compose.yml               (CT 116 .116)
+/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
+/opt/monitoring/prometheus.yml                          (CT 116 .116)
+/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
+/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
+/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
+/opt/home_stack/docker-compose.yml                      (VM 109 .7)
+/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
+/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
+/etc/systemd/system/ornith-server.service               (amdpve .15)
+/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
+```
+
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
+
+## Security-Specific Updates
+
+| Check | Command | Action |
+|-------|---------|--------|
+| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
+| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
+| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
+| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |
+
+## Success Criteria
+
+- [ ] All 5 PVE nodes updated, no reboot-loop
+- [ ] All VMs/CTs running post-update
+- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
+- [ ] LiteLLM inference passing (syslog-auto test)
+- [ ] Zulip server + all 3 agents connected
+- [ ] GPU fleet at full capacity (3/3)
+- [ ] Zero security CVEs remaining
+- [ ] <10 min total downtime per service
+
+## Report
+
+After completion, send Zulip DM:
+```
+📋 Infrastructure Update — YYYY-MM-DD
+
+Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
+Security fixes: N CVEs patched
+Downtime: <service> <duration>
+Failures: none / <details>
+Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
+```

litellm-api-keys.prose.md

@@ -0,0 +1,57 @@
+---
+kind: function
+name: litellm-api-keys
+description: >
+  Manages LiteLLM API keys for agent identity. Creates named keys so each agent
+  is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry)
+  and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026").
+  Ensures agents never use the master key directly. Rotation is event-driven,
+  not calendar-driven — rotate only on compromise, personnel change, or
+  periodic security hygiene (quarterly/annually).
+  
+  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
+  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
+  on CT 116. Last verified: 2026-07-09.
+---
+
+## Parameters
+
+- agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
+- action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
+- litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
+- master_key: string — LiteLLM master key (default from environment)
+- agent_host: string — Agent's IP for SSH (default: resolved from infra)
+- agent_user: string — SSH user (default: "jerome")
+
+## Returns
+
+- action: string — What was done
+- key_alias: string — The LiteLLM key alias created/rotated
+- key_prefix: string — First 10 chars of the new key (for identification)
+- previous_key_alias: string | null — Previous key alias if rotating
+- litellm_response: object — Raw response from LiteLLM /key/generate
+- agent_config_updated: boolean — Whether /etc/environment was updated
+- verification: { status: string, detail: string } — Final health check
+
+## Execution
+
+1. **Authenticate** — Verify master_key works against LiteLLM /key/list
+2. **Check existing keys** — List all keys, find any with agent_name alias
+3. **If action == "list"**: Return all keys with their aliases and spend
+4. **If action == "create"**:
+   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
+   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
+   - Duration is null (permanent) — inherited from litellm default_key_generate_params
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
+   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
+   - Return the new key
+5. **If action == "rotate"**:
+   - Generate new key with same alias (LiteLLM replaces the old key)
+   - SSH to agent_host, update /etc/environment LITELLM_API_KEY
+   - Restart agent gateway (hermes gateway restart for Hermes agents)
+   - Verify: curl test against /v1/models with new key
+   - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
+6. **If action == "verify"**:
+   - SSH to agent, read /etc/environment
+   - Test the key against LiteLLM /v1/models
+   - Confirm key alias matches agent_name in LiteLLM key list

litellm-health.prose.md

@@ -1,17 +1,59 @@
 ---
 kind: function
-name: check-litellm-health
+name: litellm-health
+status: deprecated
+deprecated_on: 2026-07-09
+replaced_by: litellm-self-heal.prose.md
+note: >
+  Consolidated into litellm-self-heal.prose.md to eliminate duplication
+  of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within litellm-self-heal.
+  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
-  OIDC auth endpoint, container status, and aggregate health on the
-  backend host. Designed as a reusable contract for any Syslog agent.
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
+  Router (harness-router :9000) is DEPRECATED — container still runs but
+  is not in the request path. GPU monitoring via Prometheus/Grafana and
+  fleet dashboard (gpu-monitor :9100).
+  Designed as a reusable contract for any Syslog agent.
+  
+  Source of truth: gpu-fleet.prose.md
 ---
 
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
+                        │
+                  Key validation
+                  Fallback chains
+                  Budget tracking
+                        │
+                  Prometheus ← metrics
+                        │
+                    Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
+```
+
+**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
+- Router REMOVED from request path — LiteLLM proxies directly to GPU
+- All GPUs at parallel 2 (was parallel 1)
+- NVIDIA context reduced 256K→128K to free VRAM
+- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
+- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
+
 ## Parameters
 
 - public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
-- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
+- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
 - auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
 
 ## Returns
 
@@ -23,34 +65,77 @@ description: >
 ## Requires
 
 - SSH key access to backend_host for container checks
-- Network access to public_url and auth_host
-- curl and openssl available on the execution host
+- Network access to public_url, auth_host, and gpu_dashboard_url
+- LiteLLM master key for key management endpoints
 
-## Ensures
+## GPU Fleet Topology
 
-- Each check returns a clear pass/fail status with detail message
-- If any endpoint returns non-200, overall_status is "degraded"
-- If backend host unreachable or >2 containers down, overall_status is "down"
-- If /health/unified reports any non-healthy components, status reflects it
-- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
+|------|-----|----------|---------------|--------|---------|----------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
+
+## Model Fallback Chains (LiteLLM)
+
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
 
 ## Execution
 
 1. **Read parameters** — Use provided values or defaults
+
 2. **Check public endpoints**:
    - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
    - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-   - GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
-   - GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
-3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
-   - GET http://{{backend_host}}/litellm/ui/ → expect 200
-   - GET http://{{backend_host}}/litellm/docs → expect 200
-4. **Check aggregate health endpoint**:
-   - GET {{backend_host}}:9000/health/unified → expect 200, check all components
-5. **Check backend container health**:
-   - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
-   - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
-6. **Check OIDC auth endpoint**:
-   - Verify auth.sysloggh.net resolves to {{auth_host}}
-   - GET https://auth.sysloggh.net/ → expect login page
-7. **Compile and report** — Determine overall_status from individual check results
+
+3. **Check LiteLLM health (no-auth)**:
+   - GET http://{{backend_host}}/litellm/health/liveliness → expect 200
+
+4. **Check backend container health**:
+   - SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
+   - Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
+   - Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
+
+5. **Check router roster loaded**:
+   - GET http://{{backend_host}}:9000/health → expect 200
+   - GET http://{{backend_host}}:9000/health/unified → expect 3 models
+   - If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
+
+6. **Check GPU fleet health** (via fleet dashboard):
+   - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
+   - Verify GPUs reporting status "healthy"
+   - Check alerts array for active warnings/critical
+
+7. **Check model inference via LiteLLM** — Test each model:
+   - POST /v1/chat/completions model=gemma-4-12b → expect 200
+   - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+   - POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+   - Use master key for auth
+
+8. **Check agent keys**:
+   - GET /key/list with master key → verify all 6 agents have keys
+
+9. **Check Grafana**:
+   - GET {{grafana_url}}/api/health → expect 200
+
+10. **Compile and report** — Determine overall_status from individual check results

litellm-self-heal.prose.md

@@ -1,22 +1,97 @@
 ---
 kind: responsibility
 name: litellm-self-heal
+status: manual-only
+note: >
+  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
+  This contract is now manual-only — triggers require explicit user request.
+  Consider reimplementing as a standalone cron job or prose contract.
+  
+  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
+  duplication of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within this contract.
+  
+  Source of truth for GPU topology and keys: gpu-fleet.prose.md
+  Last verified: 2026-07-09
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
-  knowledge graph for full audit trail.
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
+  inference, and agent keys. Applies remediation rules for common failures.
+  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 
+# LiteLLM Operations — Health Check + Self-Heal
+
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
+                        │
+                  Key validation
+                  Fallback chains
+                  Budget tracking
+                        │
+                  Prometheus ← metrics
+                        │
+                    Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
+```
+
+## Parameters
+
+- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
+- backend_host: string — Internal CT host (default: "192.168.68.116")
+- auth_host: string — Authentik OIDC server (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
+
+## GPU Fleet Topology
+
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
+|------|-----|----------|---------------|--------|---------|----------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
+
+## Model Fallback Chains (LiteLLM)
+
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
+| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
+| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
+
 ## Maintains
 
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
-## Requires
-
-- litellm-health:function
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
+- litellm-agent-keys: { count: int, valid: int }
 
 ## Continuity
 
@@ -24,47 +99,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
 
-## Reporting
+---
 
-Every remediation cycle produces a structured report in three channels:
+## Health Check
 
-### 1. RA-H OS Knowledge Graph Node
-Created as `[LEARN] litellm-self-heal: <run_id>` with:
-- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
-- source: Full JSON report of the cycle
-- description: Summary of what was found, fixed, and escalated
+Run this first on every cycle. Results feed into remediation rules below.
 
-### 2. Zulip DM to Owner
-Sent immediately for:
-- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
-- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
-- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+### 1. Check public endpoints
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
 
-### 3. Daily Digest (end of day)
-A summary of the last 24 hours sent as a single Zulip DM:
-```
-📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
+### 2. Check LiteLLM health (no-auth)
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
 
-Total cycles: 288 (every 5 min)
-Issues found: 3
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
-  └─ Escalated: 0
+### 3. Check backend container health
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
+- Critical: harness-litellm, harness-nginx, harness-postgres
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
+- Deprecated but running: harness-router (not in path, reference only)
 
-Top actions:
-  • harness-nginx restarted — 2026-06-26 02:15
-  • DOCS_URL corrected — 2026-06-26 07:30  
-  • harness-nginx restarted — 2026-06-26 14:46
+### 4. Check GPU fleet health (via fleet dashboard)
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
+- Verify GPUs reporting status "healthy"
+- Check alerts array for active warnings/critical
 
-Uptime: 23h 47m — All healthy now ✅
-```
+### 5. Check model inference via LiteLLM — test each model
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
+- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+- Use master key for auth
 
-### 4. Weekly Digest (every Monday)
-Same format as daily but covering 7 days. Sent to both Zulip DM and
-logged as a `[REPORT]` knowledge graph node for long-term trending.
+### 6. Check agent keys
+- GET /key/list with master key → verify all 6 agents have keys
 
-### 5. Relay Message (if cross-agent)
-If a fix required another agent's help (e.g., Authentik restart), a relay
-message is sent to the responsible agent with full context.
+### 7. Check Grafana
+- GET {{grafana_url}}/api/health → expect 200
+
+### 8. Compile overall status
+Determine overall_status from individual check results:
+- "healthy" — all checks pass
+- "degraded" — 1-2 non-critical checks fail
+- "down" — critical checks fail
+
+---
 
 ## Remediation Rules
 
@@ -82,21 +159,99 @@ Detect → immediate escalate (external dependency)
 ### Rule 4: Docs 404
 Detect → fix DOCS_URL env var → verify → log
 
+### Rule 5: GPU Unreachable
+Detect → SSH to GPU host fails or model not responding
+Fix → restart llama-server on host via SSH
+Escalate → immediately (downtime affects inference)
+
+### Rule 6: Model Not Responding
+Detect → test prompt via LiteLLM returns non-200
+Fix → restart llama-server on GPU host → verify
+Escalate → after 2 failed restarts
+
+### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
+Router is no longer in the request path (2026-07-08). LiteLLM proxies
+directly to GPU. This rule is retained for reference but is inactive.
+If 503 errors occur, check LiteLLM timeouts and GPU health directly.
+
+### Rule 8: Agent Keys Invalid (401)
+Detect → agents report auth errors
+Root cause → keys not in LiteLLM DB
+Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
+Escalate → if SSH access unavailable, send Zulip DM
+
+### Rule 9: Stale Active Counter in Redis — DEPRECATED
+Router no longer in path so Redis active counters are unused. Rule retained
+for reference but inactive. If Redis issues occur, check harness-redis container.
+
+---
+
+## Reporting
+
+Every remediation cycle produces a structured report:
+
+### 1. RA-H OS Knowledge Graph Node
+Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
+
+### 2. Zulip DM to Owner
+- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
+- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+
+### 3. Daily Digest (end of day)
+Summary of last 24 hours: total cycles, issues found/fixed/escalated,
+top actions, uptime.
+
+### 4. Relay Message (if cross-agent)
+If a fix requires another agent (e.g., Authentik restart), relay sent
+to responsible agent with full context.
+
+---
+
 ## Execution
 
-1. Run health check
-2. Apply remediation for each failure
-3. **Generate report** with all actions taken
-4. **Log to knowledge graph** — create `[LEARN]` node
-5. **Notify user** via Zulip DM if anything changed
-6. Wait 300s and repeat
+```prose
+-- Phase 1: Health Check
+let health = call health-check
+  public_url: public_url
+  backend_host: backend_host
+  gpu_dashboard_url: gpu_dashboard_url
+  grafana_url: grafana_url
+
+-- Phase 2: Apply remediation for each failure
+let actions = []
+for check in health.failed:
+  let fix = apply-remediation-rule
+    rule: lookup-rule(check.name)
+    target: check.target
+  push actions fix
+
+-- Phase 3: Generate report
+call report-generator
+  health: health
+  actions: actions
+
+-- Phase 4: Log to knowledge graph
+call kg-logger
+  run_id: run_id
+  health: health
+  actions: actions
+
+-- Phase 5: Notify if anything changed
+if actions.length > 0:
+  call zulip-notifier
+    actions: actions
+    health: health
+
+-- Wait 300s and repeat
+```
 
 ## Audit Trail Format
 
 ```json
 {
-  "run_id": "self-heal-20260626-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "run_id": "self-heal-20260709-001",
+  "timestamp": "2026-07-09T14:00:00Z",
   "duration_ms": 1234,
   "checks_passed": 7,
   "checks_failed": 0,
@@ -110,7 +265,7 @@ Detect → fix DOCS_URL env var → verify → log
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
   "issues_found": 1,
   "issues_fixed": 1,
   "actions": [

memory-audit-maintenance.prose.md

@@ -1,205 +1,346 @@
 ---
-kind: responsibility
 name: memory-audit-maintenance
-description: >
-  Systematic audit and reorganization of an agent's native memory (MEMORY.md,
-  USER.md, SOUL.md). Categorizes entries, redistributes to correct homes,
-  verifies live configs, and frees up char budget. Run when memory usage
-  exceeds 85% or on explicit user request ("memory audit", "memory maintenance").
+kind: responsibility
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+id: 067NC4KG01RG50R40M30E20918
 ---
 
-## Quickstart — How to Run This Contract
+### Goal
 
-### For the Agent Running This Contract
+Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md, SOUL.md), categorize entries, redistribute them to correct locations, verify live configurations, and free character budget  all without user intervention unless a truly unresolvable ambiguity exists. Every agent runs this contract for **its own** memory files only  no cross-agent access, no shared state.
 
-Read this section first. It tells you exactly what to do.
+### Requires
 
-**When to run:**
-- User says: `"memory audit"`, `"memory maintenance"`, `"clean up memory"`, `"memory is full"`
-- You detect that your `memory` store usage exceeds 85% in a write attempt
-- It's been 2+ weeks since the last audit
+(none  this is a self-driven, autonomous node)
 
-**How to run (via OpenProse CLI):**
-```bash
-# Default — 85% threshold, verify configs, compress entries
-prose run memory-audit-maintenance
+### Scope
 
-# Softer audit — only flag issues, don't rewrite files
-prose run memory-audit-maintenance compress_entries=false
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
 
-# Hard audit — stricter limits
-prose run memory-audit-maintenance memory_threshold=90 max_memory_chars=800 max_user_chars=300
+**Agent Roster:**
+- Mumuni
+- Tanko
+- Tdunna
+- Baggy
+
+**Isolation Principle:** Each agent has its own:
+- `MEMORY.md` and `USER.md`
+- `MEMORY_AUDIT_LEDGER.md` (audit trail)
+- `MEMORY_WRITERS.md` (writer registry)
+- Canary entries
+- No data crosses agent boundaries
+
+### Maintains
+
+Memory health state  current utilization, last audit timestamp, drift alerts, and any unresolved ambiguities. Postcondition: no entry is duplicated, no session note or stale snapshot survives, no config is unverified, and the audit ledger is up-to-date.
+
+#### utilization
+Current character utilization of MEMORY.md and USER.md. Material: the percentages.
+
+#### last_audit
+Timestamp of the most recent successful audit. Material: the timestamp.
+
+#### unresolved
+Any entries that could not be autonomously categorized. Material: the entry text and the reason for ambiguity.
+
+#### drift_alerts
+Active alerts from drift detection (e.g., deletions accelerating, unverified configs growing). Material: the alert list.
+
+#### ledger
+Persistent audit trail stored at the agent's own `~/.hermes/memories/MEMORY_AUDIT_LEDGER.md`. Material: append-only, never modified or deleted. Each agent has its own ledger  no shared ledger across agents.
+
+### Continuity
+
+- self-driven: re-audit every 14 days (2 weeks)
+- input-driven: triggered by explicit user request ("memory audit", "memory maintenance", "clean up memory", "memory is full")
+- threshold: trigger immediately if either store exceeds 85% utilization
+
+### Parameters
+
+- `memory_threshold`: percentage that triggers an immediate audit (default: 85)
+- `max_memory_chars`: maximum allowed characters for MEMORY.md (default: 2200)
+- `max_user_chars`: maximum allowed characters for USER.md (default: 1375)
+- `compress_entries`: whether to rewrite entries for brevity (default: true)
+- `dry_run`: whether to only report changes without applying them (default: false)
+
+### Strategies
+
+- **Autonomous first:** Never ask the user about entries that clearly belong in a deletion or move category. Only escalate on true ambiguity.
+- **Incremental over nuclear:** Always use patch-level edits (find-and-replace) rather than wholesale file rewrites. If the patch tool fails, fall back to rewrite only as a last resort.
+- **Verify before trusting:** Every technical config in MEMORY.md must be verified against the live system. If a service is unreachable, mark it as `UNVERIFIED` rather than deleting it.
+- **Stale state has a shelf life:** Any entry marked as "Temporary State" (e.g., "verbal offer", "deal pending", "negotiation phase") is deleted if older than 14 days.
+- **Work around tool failures:** If the `memory` tool fails after 2 retries, add a summary marker and report the stale state in the unresolved list.
+- **Privacy isolation:** Each agent only reads and writes its own memory files. No cross-agent memory access, no shared ledger, no shared state. The contract is the standard; each agent runs it independently.
+
+### Invariants
+
+- No entry is deleted without being categorized first
+- No duplicate entries survive across MEMORY.md and USER.md
+- No session note or temporary state survives past 14 days
+- No config is trusted without verification (or marked UNVERIFIED)
+- The audit ledger is append-only  never modified or deleted, and is isolated per agent
+- The canary entry survives every audit unmodified
+- No data crosses agent boundaries
+
+### Execution
+
+```prose
+# ============================================================
+# PHASE 0: Canaries & Privacy
+# ============================================================
+
+# Check canaries  if either is missing or modified, raise CRITICAL alert
+memory_canary = read_file_line("~/.hermes/memories/MEMORY.md", containing="CANARY: Do not modify this line")
+user_canary = read_file_line("~/.hermes/memories/USER.md", containing="CANARY: Do not modify this line")
+
+if memory_canary is null or user_canary is null:
+  return {
+    status: "CRITICAL",
+    alert: "CANARY BREACH  memory integrity compromised",
+    details: {
+      memory_canary_intact: memory_canary is not null,
+      user_canary_intact: user_canary is not null
+    }
+  }
+
+# This agent only touches its own files  no cross-agent access
+# Confirm we are operating on this agent's own memory directory
+if not directory_exists("~/.hermes/memories"):
+  return {status: "ERROR", reason: "Memory directory not found for this agent"}
+
+# ============================================================
+# PHASE 1: Pre-flight & Ledger
+# ============================================================
+
+# Read the audit ledger for drift detection
+ledger = read_file("~/.hermes/memories/MEMORY_AUDIT_LEDGER.md")
+last_audit_entry = parse_last_entry(ledger)
+
+# Verify the memory tool works
+try:
+  memory_status = call check_memory_tool
+catch:
+  memory_status = "BROKEN"
+
+# Read current state
+memory_content = read_file("~/.hermes/memories/MEMORY.md")
+user_content = read_file("~/.hermes/memories/USER.md")
+
+# Calculate current utilization
+memory_util = len(memory_content) / max_memory_chars * 100
+user_util = len(user_content) / max_user_chars * 100
+
+# Check if audit is needed
+if last_audit_entry.timestamp > 14 days and memory_util < 75 and user_util < 75:
+  return {status: "SKIP", reason: "No audit needed"}
+
+# ============================================================
+# PHASE 1b: Drift Detection (Compare to Last Audit)
+# ============================================================
+
+drift_alerts = []
+
+if last_audit_entry exists:
+  # Deletions accelerating? (50% increase from last audit)
+  if last_audit_entry.deletions > 0:
+    deletion_increase = (memory_util - last_audit_entry.memory_util_after) / last_audit_entry.deletions * 100
+    if deletion_increase > 50:
+      drift_alerts.append("ALERT: Memory bloat accelerating  deletions increasing by " + deletion_increase + "% vs last audit")
+
+  # Unverified configs growing?
+  if last_audit_entry.unverified > 0:
+    unverified_increase = count_entries(memory_content, "[UNVERIFIED]") - last_audit_entry.unverified
+    if unverified_increase > 2:
+      drift_alerts.append("ALERT: Unverified configs growing  " + unverified_increase + " new unverified entries since last audit")
+
+  # Audit missed? (No audit for >21 days)
+  if last_audit_entry.timestamp > 21 days:
+    drift_alerts.append("ALERT: Audit gap  no audit completed in 21+ days")
+
+  # Utilization not improving? (After audit, still >70%)
+  if last_audit_entry.memory_util_after > 70:
+    drift_alerts.append("ALERT: Previous audit failed to reduce MEMORY.md below 70%  capacity issue")
+
+# ============================================================
+# PHASE 2: Categorize every entry
+# ============================================================
+
+categorized = []
+for entry in split_entries(memory_content):
+  cat = call categorize_entry(entry)
+  categorized.append({entry, category: cat, source: "MEMORY.md"})
+
+for entry in split_entries(user_content):
+  cat = call categorize_entry(entry)
+  categorized.append({entry, category: cat, source: "USER.md"})
+
+# ============================================================
+# PHASE 3: Identify actions
+# ============================================================
+
+actions = []
+for item in categorized:
+  if item.category in ["Session Note", "Temporary State"]:
+    # Check age of temporary state
+    if item.category == "Temporary State" and item.age < 14 days:
+      continue  # Too young to delete
+    actions.append({action: "DELETE", entry: item.entry, reason: item.category})
+  elif item.category == "Historical Snapshot":
+    actions.append({action: "DELETE", entry: item.entry, reason: "Stale"})
+  elif item.category == "Technical Config" and item.source == "USER.md":
+    actions.append({action: "MOVE", from: "USER.md", to: "MEMORY.md", entry: item.entry})
+  elif item.category == "Identity" or item.category == "Preference" and item.source == "MEMORY.md":
+    actions.append({action: "MOVE", from: "MEMORY.md", to: "USER.md", entry: item.entry})
+  elif item.category == "Operating Rule":
+    actions.append({action: "MOVE", from: "MEMORY.md", to: "Skill file", entry: item.entry})
+
+# Check for duplicates
+for entry_a, entry_b in find_duplicates(categorized):
+  actions.append({action: "DELETE", entry: entry_b.entry, reason: "Duplicate of " + entry_a.entry})
+
+# ============================================================
+# PHASE 4: Verify live configs
+# ============================================================
+
+for item in categorized:
+  if item.category == "Technical Config" and item.entry contains URL:
+    result = call verify_endpoint(item.entry)
+    if result == "UNREACHABLE":
+      # Mark as UNVERIFIED, do not delete
+      actions.append({action: "MARK_UNVERIFIED", entry: item.entry})
+    elif result == "VERIFIED":
+      pass  # Good, nothing to do
+
+for item in categorized:
+  if item.category == "Technical Config" and item.entry contains "model:":
+    result = call verify_model(item.entry)
+    if result == "MISMATCH":
+      actions.append({action: "CORRECT", entry: item.entry, corrected: result.new_value})
+
+# ============================================================
+# PHASE 5: Apply changes
+# ============================================================
+
+if dry_run:
+  return {status: "DRY_RUN", actions: actions, drift_alerts: drift_alerts, summary: generate_summary(actions)}
+
+# Apply deletions
+for action in actions:
+  if action.action == "DELETE":
+    try:
+      memory action=remove target=action.source old_text=action.entry
+    catch after 2 retries:
+      patch path=action.source old_string=action.entry new_string=""
+      # Add summary marker since memory tool is broken
+      memory action=add target=action.source content="Entry deleted via patch: " + action.entry
+
+# Apply moves
+for action in actions:
+  if action.action == "MOVE":
+    # Delete from source
+    patch path=action.from old_string=action.entry new_string=""
+    # Add to destination
+    append_file path=action.to content=action.entry + "\n"
+
+# Apply corrections
+for action in actions:
+  if action.action == "CORRECT":
+    patch path="~/.hermes/memories/MEMORY.md" old_string=action.entry new_string=action.corrected
+
+# Apply UNVERIFIED marks
+for action in actions:
+  if action.action == "MARK_UNVERIFIED":
+    patch path="~/.hermes/memories/MEMORY.md" old_string=action.entry new_string=action.entry + " [UNVERIFIED]"
+
+# ============================================================
+# PHASE 6: Post-audit verification
+# ============================================================
+
+new_memory_content = read_file("~/.hermes/memories/MEMORY.md")
+new_user_content = read_file("~/.hermes/memories/USER.md")
+
+new_memory_util = len(new_memory_content) / max_memory_chars * 100
+new_user_util = len(new_user_content) / max_user_chars * 100
+
+# Verify canaries survived the audit
+post_canary_check = read_file_line("~/.hermes/memories/MEMORY.md", containing="CANARY: Do not modify this line")
+if post_canary_check is null:
+  drift_alerts.append("CRITICAL: Canary destroyed during audit  memory integrity compromised")
+
+# Add summary markers to in-memory stores
+memory action=add target=memory content="MEMORY.md rewritten. Utilization: " + new_memory_util + "%."
+memory action=add target=user content="USER.md rewritten. Utilization: " + new_user_util + "%."
+
+# ============================================================
+# PHASE 7: Append to Ledger
+# ============================================================
+
+ledger_entry = """
+## """ + now() + """
+- memory_util_before: """ + memory_util + """%
+- memory_util_after: """ + new_memory_util + """%
+- user_util_before: """ + user_util + """%
+- user_util_after: """ + new_user_util + """%
+- deletions: """ + count(actions, "DELETE") + """
+- moves: """ + count(actions, "MOVE") + """
+- corrections: """ + count(actions, "CORRECT") + """
+- unverified: """ + count(actions, "MARK_UNVERIFIED") + """
+- unresolved: """ + len(unresolved_entries) + """
+- drift_alerts: """ + len(drift_alerts) + """
+- canary_intact: """ + (post_canary_check is not null) + """
+- status: COMPLETE
+"""
+
+append_file path="~/.hermes/memories/MEMORY_AUDIT_LEDGER.md" content=ledger_entry
+
+# ============================================================
+# PHASE 8: Report
+# ============================================================
+
+return {
+  status: "COMPLETE",
+  memory_utilization: new_memory_util,
+  user_utilization: new_user_util,
+  actions_applied: len(actions),
+  deletions: count(actions, "DELETE"),
+  moves: count(actions, "MOVE"),
+  corrections: count(actions, "CORRECT"),
+  unverified: count(actions, "MARK_UNVERIFIED"),
+  unresolved: unresolved_entries,
+  drift_alerts: drift_alerts,
+  last_audit: now(),
+  ledger_updated: true,
+  summary: generate_summary(actions)
+}
 ```
 
-### How to Follow the Template (Manual)
+### Shape
 
-If `prose` CLI is not available:
+- `self`: categorize, verify configs, apply incremental patches, generate reports, drift detection, ledger management, canary checks
+- `delegates`:
+  - `categorize_entry`: determine the category of a memory entry
+  - `check_memory_tool`: verify the memory tool is functional
+  - `verify_endpoint`: check if a URL is reachable
+  - `verify_model`: check if a model config matches live config.yaml
+  - `find_duplicates`: detect duplicate entries across files
+  - `generate_summary`: produce a human-readable summary of actions
 
-1. **Read both memory files:** `cat ~/.hermes/memories/MEMORY.md` and `cat ~/.hermes/memories/USER.md`
-2. **Check in-memory usage:** Use the `memory` tool with no args
-3. **Categorize every entry** using the Categorization Guide below
-4. **Rewrite MEMORY.md** — technical configs only, 800–1,100 chars max
-5. **Rewrite USER.md** — identity + preferences only, 300–500 chars max
-6. **Verify live configs** — curl each endpoint, check each model
-7. **Sync in-memory** — Add summary markers (or just leave file as source of truth)
-8. **Report** using the Example Output template at the bottom
+### Tools
 
-## Maintains
+- `file:read`: read MEMORY.md and USER.md
+- `file:patch`: incremental edits via find-and-replace
+- `file:append`: append entries when moving between files
+- `memory:action`: add/remove/replace entries in the in-memory store
+- `network:curl`: verify endpoint reachability
+- `cli:grep`: verify model configs against config.yaml
 
-- last_audit: timestamp — When the last full audit was completed
-- memory_usage_pct: number — Current memory store usage percentage
-- user_usage_pct: number — Current user store usage percentage
-- total_entries_categorized: number — How many entries were audited
-- entries_moved: number — How many entries changed location
-- entries_deleted: number — How many stale entries were removed
-- live_configs_verified: array — Which configs were verified against live systems
-- known_issues: array — Problems found during audit (stale keys, dead services)
+### Environment
 
-## Parameters
+- `MEMORY_PATH`: ~/.hermes/memories/MEMORY.md
+- `USER_PATH`: ~/.hermes/memories/USER.md
+- `CONFIG_PATH`: ~/.hermes/config.yaml
+- `LEDGER_PATH`: ~/.hermes/memories/MEMORY_AUDIT_LEDGER.md
 
-- memory_threshold: number — Usage % that triggers audit (default: 85, max: 95)
-- verify_configs: boolean — Whether to live-check configs after audit (default: true)
-- compress_entries: boolean — Whether to shorten verbose entries (default: true)
-- remove_session_notes: boolean — Strip "currently doing X" markers (default: true)
-- max_memory_chars: number — Target max chars for MEMORY.md after audit (default: 1100)
-- max_user_chars: number — Target max chars for USER.md after audit (default: 500)
+### Per-Agent Notes
 
-## Continuity
-
-The audit cycle is **event-driven**, not just cron-based:
-
-- **On threshold breach**: If memory usage exceeds `memory_threshold`, wake immediately
-- **On user request**: `"memory audit"`, `"memory maintenance"`, `"clean up memory"` — explicit trigger
-- **Retrospective**: Every 2-4 weeks if no other trigger fired — check for gradual bloat
-- **On config change**: After major provider swaps or infrastructure migrations, verify live configs
-- **On new agent onboarding**: Run once as part of initialization to establish clean baseline
-
-## Success Criteria (What Makes a Memory Audit "Good")
-
-The audit is considered COMPLETE when ALL of these pass:
-
-### Inventory Complete
-- MEMORY.md content read and categorized
-- USER.md content read and categorized
-- In-memory `memory` and `user` store usage percentages recorded
-- Every entry tagged with: `Infrastructure | Identity | Preference | Operating Rule | Historical | Session Note`
-
-### Redistribution Correct
-- Identity entries → USER.md
-- Preference entries → USER.md
-- Technical config entries → MEMORY.md
-- Operating rules → Skill file or SOUL.md
-- Historical snapshots (dated checkpoints) → Deleted
-- Session notes ("currently doing X") → Deleted
-
-### Files Compact
-- MEMORY.md ≤ `max_memory_chars` chars
-- USER.md ≤ `max_user_chars` chars
-- No duplicate information across files
-- No frustration signals or verbose descriptions
-- § separator between entries (standard convention)
-
-### Configs Verified
-- API keys confirmed in `.env` file (grep, not read)
-- Model names in memory match live `config.yaml`
-- Endpoints in memory respond to curl/health checks
-- Credential pairs (username/token, email/password) match deployed state
-- Any mismatch corrected in memory with ✅/❌ labels
-
-### In-Memory Synced
-- Summary marker added to `memory` store: "MEMORY.md rewritten <date>. Holds only tech configs."
-- Summary marker added to `user` store: "Home channel: <channel>. USER.md rewritten <date>."
-- Usage confirmed dropped below 60% after sync
-
-### Skill Created/Updated (if applicable)
-- Operating rules that belong in a skill → created via `skill_manage`
-- Existing skills verified not to conflict with memory content
-- Skill category appropriate and description searchable
-
-## Remembers
-
-Each audit stores:
-- Entry categorization decisions (why X went to USER instead of MEMORY)
-- What was deleted (stale snapshots, session notes)
-- What was verified (configs that passed live checks)
-- What failed verification (configs that need user attention)
-- Known tool quirks (memory replace/remove failures)
-- Compression patterns used (how verbose entries were shortened)
-
-## Audit Rules
-
-### Rule 1: Read Before Delete
-Never delete an entry without first reading it and categorizing it. Show the user a summary of what will change if unsure about a categorization.
-
-### Rule 2: Categories Are Mutual Exclusive
-Each entry belongs to exactly ONE category:
-- **Identity** — "Jerome, Florida, Telegram @mejerome19"
-- **Preference** — "Single-account first for AWS"
-- **Technical Config** — "SearXNG on storepve:8888"
-- **Operating Rule** — "Never go rogue on infrastructure"
-- **Historical Snapshot** — "Baseline v30 from Jun 26" (DELETE)
-- **Session Note** — "Currently performing memory audit" (DELETE)
-
-### Rule 3: Rules Belong in Skills
-If an entry is a procedure, protocol, or mandate (`"do X this way"`, `"never do Y"`) — it belongs in a skill file, not MEMORY.md. Check `skills_list` first; create via `skill_manage` if no match exists.
-
-### Rule 4: Verify Live Before Trusting Memory
-Every config entry in MEMORY.md must be verified against the live system:
-- Model name → `grep config.yaml`
-- API key → `grep .env`
-- Service URL → `curl` health check
-- Credentials → cross-reference with deployed state
-
-### Rule 5: Work Around the `memory` Tool
-The `memory` tool's `remove` and `replace` actions use strict text matching that can fail on special characters. If a remove/replace fails repeatedly:
-- Do not retry more than 2 times
-- Add a new summary marker entry instead
-- Report to user: "File on disk updated; in-memory store has stale leftovers"
-- The file on disk (MEMORY.md, USER.md) is the source of truth regardless
-
-### Rule 6: Report Before Asking
-Don't ask the user "should I delete X?" for session notes or historical snapshots — those are always safe to delete. Only ask about ambiguous categorizations or cross-file moves of preference data.
-
-## Execution
-
-1. **Read state** — Read MEMORY.md, USER.md, check in-memory usage via `memory` tool
-2. **Categorize** — Tag every entry with its category
-3. **Prepare rewrites** — Draft compressed MEMORY.md and USER.md:
-   - MEMORY.md: Technical configs only (800-1100 chars)
-   - USER.md: Identity + preferences only (300-500 chars)
-4. **Write files** — Overwrite MEMORY.md and USER.md
-5. **Verify live configs** — Check every config entry against actual system state
-6. **Sync in-memory** — Add summary markers to `memory` and `user` stores
-7. **Create/update skills** — Move operating rules to skill files if needed
-8. **Report** — Output clean status table with before/after/verification
-
-## Example Output
-
-```
-## Memory Audit Complete ✅
-
-| File | Before | After | Reduction |
-|---|---|---|---|
-| MEMORY.md | 1,653 chars | 1,077 chars | ~35% |
-| USER.md | 860 chars | 418 chars | ~51% |
-
-### What Moved
-| Entry | From | To |
-|---|---|---|
-| Identity + Preferences | MEMORY.md | USER.md |
-| Agent boundary rule | MEMORY.md | SOUL.md (persona) |
-| Baseline v30 snapshot | MEMORY.md | Deleted |
-
-### Config Verification
-| Config | Live Check | Status |
-|---|---|---|
-| DeepSeek model | config.yaml: deepseek-chat | ✅ |
-| DeepSeek key | .env found | ✅ |
-| SearXNG endpoint | :8888 curl 200 | ✅ |
-
-### Memory Usage
-| Store | Before | After | Status |
-|---|---|---|---|
-| memory | 91% | 59% | ✅ |
-| user | 85% | 41% | ✅ |
-```
+Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.

mumuni-delegation-prose-contract.prose.md

@@ -0,0 +1,290 @@
+---
+kind: pattern
+name: mumuni-delegation
+description: >
+  Mumuni-specific operating doctrine for task decomposition, worker
+  delegation, verification, and delivery. Defines when to delegate, which
+  worker to use for what, how to handle failures, and the kanban board
+  protocol. Enforces context-window discipline and separation of concerns.
+  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
+version: 1.0.0
+---
+
+## Maintains
+
+- Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
+  `syslog-research`, `syslog-review`, `syslog-writer`)
+- Kanban board state at `~/.hermes/kanban/kanban.json`
+- Context window budget: ~65K tokens per request (131K total, 60% threshold)
+
+## Topology
+
+**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
+**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
+**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
+
+This contract is infrastructure-agnostic in terms of which nodes are used.
+Workers execute tasks on whatever infrastructure they're given — SSH to .6,
+.pm, .9, .12, or .15 depending on the task. The contract defines the
+**who** and **when** — not the **where**.
+
+## Why This Matters
+
+Without enforced delegation, the manager consumes the full iteration budget
+(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
+leaving no capacity for actual coordination. The result: context overflow
+(59K tokens in system prompt), iteration exhaustion, and degraded response
+quality. This contract exists because I blew through my budget checking
+Proxmox node status instead of delegating to `syslog-devops`.
+
+## Context Window Discipline
+
+**The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
+**Volatile (MEMORY.md + USER.md): ~300 tokens.**
+**Total base: ~6,800 tokens per request.**
+
+The remaining budget is the conversation. Every tool call result adds to it.
+If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
+from multiple nodes), the context fills fast. That's why we delegate: workers
+process in isolation and return compact results.
+
+## Trigger Conditions
+
+Delegation is **mandatory** when any of these apply:
+
+| Condition | Threshold | Example |
+|-----------|-----------|---------|
+| Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
+| Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
+| Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
+| Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
+| Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
+| Code builds or changes | Writing or modifying code | Scripts, configs, patches |
+| Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
+
+**Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
+`curl`, `hermes tools list` — these are decision-making tools. The manager
+reads them directly.
+
+## Data Source Integrity (CRITICAL)
+
+**Workers MUST use the data provided in their task context. They MUST NOT
+fetch their own data from external sources unless explicitly told to.**
+
+When a task says "Read file X and format it", the worker reads file X. It does
+not query a separate API, run its own diagnostics, or pull data from a different
+system. This is the #1 source of cross-worker inconsistency: one worker gathers
+SSH data, another queries the Proxmox API, and the report merges two incompatible
+datasets.
+
+**Rule:** If a worker needs additional data beyond what's in its task description,
+it asks the manager (via relay) — it doesn't go find it on its own.
+
+**This is a hard rule, not a recommendation.** Violating it produces the exact
+type of discrepancy the kanban pipeline exists to prevent: a review worker finds
+"5 nodes present" in the raw data but "5/5 online" in the report — even though
+one of those nodes was unreachable. The report lied because it used data the
+raw data never provided.
+
+## Worker Selection Matrix
+
+| Worker | Model | Toolsets | Role | Use When |
+|--------|-------|----------|------|----------|
+| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
+| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
+| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
+| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
+| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
+| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
+
+### Selection Rules
+
+1. **Match specialty first.** A code task → `syslog-code`. An infra task →
+   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
+2. **Research tasks with browser needs → `syslog-research`.** Other workers
+   don't have the browser toolset.
+3. **Verification → `syslog-review`.** Never deliver raw worker output.
+4. **Documentation/content → `syslog-writer`.** Let them own the prose.
+5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
+   (includes browser) and high reasoning effort.
+
+## Delegation Protocol
+
+### Step 1: Decompose
+
+Break the task into lanes. Each lane does ONE thing. Workers are independent —
+no lane depends on another's output mid-flight. If lanes depend on each other,
+dispatch sequentially.
+
+### Step 2: Dispatch
+
+Fire workers via `delegate_task`:
+
+**Critical: Pass the data, not just the goal.** When dispatching a worker that
+processes output from another worker, include the file path AND explicit
+instructions to use ONLY that source. Example:
+
+```
+delegate_task(
+    goal="Format the cluster check into a clean report",
+    context="Source data is at /tmp/proxmox-check-raw.md. Format ONLY the data
+    in that file. Do NOT query the Proxmox API or any other data source. Use the
+    file as your sole source of truth."
+)
+```
+
+**Parallel (independent lanes):**
+```
+delegate_task(
+    tasks=[
+        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
+        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
+    ]
+)
+```
+
+**Sequential (dependent lanes):**
+Dispatch lane 1 → wait for result → dispatch lane 2.
+
+### Step 3: Verify
+
+**MANDATORY for:**
+- Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
+- Code builds and modifications
+- Research findings (web data, external sources)
+- Any output that will reach the user
+
+**Fire `syslog-review` to verify:**
+```
+delegate_task(
+    goal="Review the output of the devops worker. Verify the node status
+    report is accurate, check for inconsistencies, confirm all nodes were
+    reachable.",
+    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
+    Verify against live system."
+)
+```
+
+**If verification fails:**
+1. Send work back to original worker with review feedback
+2. Re-verify
+3. Max 2 re-verify cycles before escalating to Kwame
+
+### Step 4: Deliver
+
+Only verified results reach Kwame. Format per channel:
+- Telegram: Use `telegram-formatting` skill
+- Zulip: Use Zulip Markdown (CommonMark)
+- Email: Use `syslog-email` skill
+
+## Kanban Board Protocol
+
+**File:** `~/.hermes/kanban/kanban.json`
+
+```json
+{
+  "task_id": "unique-id",
+  "title": "Task description",
+  "created": "2026-07-09T01:00:00",
+  "status": "backlog|in_progress|review|done",
+  "lanes": [
+    {
+      "lane_id": "devops-check",
+      "worker": "syslog-devops",
+      "goal": "Check all 5 Proxmox nodes",
+      "status": "dispatched|completed|failed",
+      "output_file": "/tmp/node-report.md"
+    }
+  ]
+}
+```
+
+**Update the board on every state change.**
+
+## Failure Handling
+
+### Worker Timeouts
+
+- Child timeout: **900 seconds** (15 minutes)
+- Worker model `syslog-auto` is slow — it can hit the timeout limit with
+  22+ API calls
+- **If a worker times out:** Re-dispatch with a narrower scope. Break the
+  task into smaller pieces that fit in the timeout window.
+- **Avoid delegating sequential SSH hops** — each SSH connection adds latency
+  that compounds quickly. Prefer API-based or local approaches when possible.
+
+### Worker Selection Failures
+
+- `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
+- `syslog-code` is best for code-level work (reading files, writing scripts)
+- `syslog-research` has the browser toolset — use for web research
+- `syslog-review` is the QA gate — always fire before delivery
+- **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
+- **Never nest delegation** (max_spawn_depth: 1)
+
+### Context Overflow
+
+- If a task requires >10K tokens of output, delegate the processing
+- Workers return compact summaries, not raw data dumps
+- Pass file paths and concrete goals — never dump raw data into context
+
+## Anti-patterns
+
+- ❌ Reading large files into your own context before deciding → delegate the read
+- ❌ Carrying SSH/grep/output results in your context → delegate the analysis
+- ❌ Doing work yourself and then "pretending" to delegate → the user can tell
+- ❌ Skipping verification → raw worker output never reaches the user
+- ❌ Delegating single tool calls → keep quick reads/writes at manager level
+- ❌ Firing more than 3 workers in parallel → hard limit
+- ❌ **Workers fetching their own data sources** → a writer worker that queries the Proxmox API when told to "format the raw file" is fabricating data. Use the input given, not external sources
+
+## Emergency Exception
+
+**In an emergency (server down, service must be restored immediately):**
+- Delegate the diagnosis (find the problem)
+- Execute the fix yourself (minimize handoff latency)
+- Verify the fix after delivery
+- Log the exception in the kanban board
+
+The emergency exception exists because the user needs the service back NOW,
+not after three worker round-trips. But it's an exception — not the rule.
+
+## What This Contract Doesn't Cover
+
+1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
+2. **SSH key management** — covered by existing SSH/Proxmox contracts
+3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
+4. **Cron job management** — covered by individual cron contracts
+5. **Infra verification** — covered by `verify-before-mutate` protocol
+
+## Verification
+
+Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
+
+```bash
+python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
+```
+
+## References
+
+- `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
+- `worker-profile-audit.md` (skill reference): Worker configuration audit notes
+- `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
+- `verify-before-mutate` protocol: Infrastructure change verification
+- `hermes-config-template.prose.md`: Worker profile configuration
+
+## Success Criteria
+
+This contract succeeds when:
+
+1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
+2. **Workers do the work** — manager coordinates, doesn't execute
+3. **Verification before delivery** — all output passes through `syslog-review`
+4. **Kanban board is current** — every task has a lane, every lane has a status
+5. **User gets verified results** — raw worker output never reaches Kwame
+
+---
+
+**Last updated:** 2026-07-09
+**Author:** Mumuni (with Kwame's input on triggers and exception criteria)
+**Status:** Draft — awaiting PR review and merge to prose-contracts main

pi-approval-architecture.prose.md

@@ -0,0 +1,107 @@
+---
+kind: architecture
+name: pi-approval-architecture
+description: >
+  Documents pi's approval model vs Hermes, what commands are available,
+  and the architectural constraints that prevent mid-turn tool approval in pi.
+  Updated with /approve and /deny stub commands for cross-agent UX consistency.
+agent: abiba
+version: 1.0.0
+---
+
+# Pi Approval Architecture
+
+## TL;DR
+
+| Feature | Hermes | Pi (Abiba) |
+|---------|--------|------------|
+| `/approve` command | ✅ Full — unblocks agent | ✅ Stub — acknowledges, no blocking |
+| `/deny` command | ✅ Full — blocks agent | ✅ Stub — acknowledges, no blocking |
+| Dangerous command detection | ✅ 40+ patterns + Tirith | ❌ None (LLM judgment only) |
+| Mid-turn tool approval | ✅ Agent thread blocks on Event | ❌ No hook in agent loop |
+| Smart approval (aux LLM) | ✅ `approvals.mode: smart` | ❌ N/A |
+| Yolo mode | ✅ `/yolo`, `approvals.mode: off` | ❌ N/A |
+| Permanent allowlist | ✅ `command_allowlist` + `shell-hooks` | ❌ N/A |
+| Session-scoped approval | ✅ `approve_session()` | ❌ N/A |
+| Approval timeout | ✅ 60s default, configurable | ❌ N/A |
+
+## Why Pi Can't Do Mid-Turn Approval
+
+Pi's agent loop is **418 lines** of TypeScript. It's deliberately minimal:
+
+```
+LLM decides → calls tool → tool executes immediately → output goes to LLM → repeats
+```
+
+There is no middleware between "LLM decides to call bash" and "bash executes."
+The extension system (`injectIntoSession`) feeds messages IN but doesn't intercept
+tool calls mid-turn.
+
+Hermes has a full approval pipeline:
+```
+LLM decides → check_all_command_guards() → if dangerous → notify_cb → block thread
+→ user responds /approve → resolve_gateway_approval() → Event.set() → unblock
+```
+
+This pipeline is deeply integrated into Hermes' gateway, tool executor, and
+session management. Replicating it in pi would require rewriting the core agent
+loop — violating pi's design constraint of minimal core.
+
+## What Pi HAS: `/approve` and `/deny` Stubs
+
+Added to prevent `/approve` and `/deny` from being forwarded to the LLM as
+garbled input. When a user types these commands to Abiba:
+
+- **`/approve`** → "Pi doesn't have pending approvals. Commands execute immediately."
+- **`/approve session`** → Same response
+- **`/deny`** → Same response + "For Hermes agents (Tanko, Mumuni), these work with their built-in approval system."
+
+This keeps the UX consistent across agents — users can type `/approve` anywhere
+without getting confused by LLM responses.
+
+## Pi's Full Command Set (Hermes Parity)
+
+| Command | Handler | Description | Hermes Equivalent |
+|---------|---------|-------------|-------------------|
+| `/status` | System | PM2 processes, containers, session info | `/status` |
+| `/retry` | Session | Re-injects last user message | `/retry` |
+| `/continue` | Session | Continues truncated response | `/continue` |
+| `/new [topic]` | Session | Clears conversation context | `/new` |
+| `/stop` | Control | Interrupts current LLM processing | `/stop` |
+| `/model [name]` | Info | Show current model, list available | `/model` |
+| `/compact` | Session | Requests context compaction | `/compact` |
+| `/yolo` | Stub | Pi has no approval system — accepted for UX | `/yolo` |
+| `/verbose` | Stub | Verbosity controlled by prompt, not toggle | `/verbose` |
+| `/approve` | Stub | Pi executes immediately — no approvals | `/approve` |
+| `/deny` | Stub | Pi executes immediately — no approvals | `/deny` |
+| `/help [cmd]` | Info | Shows command help | `/help` |
+
+## What Pi COULD Add (Without Core Changes)
+
+These would work within pi's current architecture:
+
+1. **Pre-message dangerous command scan**: Before forwarding an agent response to
+   Zulip, scan the text for dangerous bash commands and add a warning banner.
+   Doesn't prevent execution but alerts user.
+
+2. **User-facing approval prompt injection**: When user asks pi to do something
+   dangerous (detected via regex on user input), pi could respond with a
+   confirmation request before proceeding. This works because it's at the
+   message level, not mid-turn.
+
+3. **Response rate-limiting**: If pi generates many bash commands rapidly,
+   throttle and ask for confirmation. Works within the extension's send path.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `/root/.pi/agent/extensions/zulip/index.js` | Pi Zulip extension with command system |
+| `/root/prose-contracts/zulip-approval-fix.prose.md` | Hermes Zulip approval fix |
+| `/root/prose-contracts/pi-approval-architecture.prose.md` | This document |
+
+## Related
+
+- Hermes `tools/approval.py` — Full approval pipeline (dangerous patterns, Tirith, smart mode)
+- Hermes `gateway/slash_commands.py:_handle_approve_command` — Slash command handler
+- Hermes Zulip adapter fix — HTML stripping for command detection

pm2-self-heal.prose.md

@@ -10,10 +10,14 @@ description: >
 
 ## Maintains
 
-- abiba-zulip: { status: "online", uptime: string, restarts: number }
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
+- gpu-watchdog: { status: "online", uptime: string, restarts: number }
+- gpu-monitor: { status: "online", uptime: string, restarts: number }
+- gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
 
+> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
+
 ## Continuity
 
 - Self-driven: check every 300 seconds (5 min)
@@ -28,9 +32,14 @@ description: >
 - **Escalate**: If still failed after 2 retries, send Zulip DM to owner
 
 ### Rule 2: Process Restarting Too Often
-- **Detect**: `pm2 status` shows restarts > 5 in the last hour
+- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
+- **Note**: PM2 counter never decrements; only full delete+re-add resets it
 - **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
-- **Escalate**: Always — alert owner with restart count
+- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
+- **Historical fix**: Previous cycles were caused by abiba-zulip extension's
+  stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
+  This was the root cause of the 18-restart accumulation. Threshold raised
+  and PM2 counter reset on 2026-06-28.
 
 ## Execution
 

pm2-self-heal.sh

@@ -1,64 +0,0 @@
-#!/bin/bash
-# pm2-self-heal — hourly PM2 process check
-# Part of the pm2-self-heal prose contract
-# Only sends Zulip DM when action is taken or escalation needed
-# Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
-
-ZULIP_BOT="abiba-bot@chat.sysloggh.net"
-ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
-ZULIP_SITE="https://chat.sysloggh.net"
-OWNER_ID=9
-LOG="/tmp/pm2-self-heal.log"
-
-send_alert() {
-    local subject="$1"
-    local body="$2"
-    curl -s -X POST "$ZULIP_SITE/api/v1/messages" \
-        -u "$ZULIP_BOT:$ZULIP_KEY" \
-        -d "type=private" \
-        -d "to=[$OWNER_ID]" \
-        -d "content=$subject\n$body" > /dev/null 2>&1
-    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Alert sent" >> "$LOG"
-}
-
-# Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
-STATUS=$(pm2 status --no-color 2>/dev/null)
-
-ALERTS=""
-
-# Check abiba-telegram (safe to auto-restart)
-TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
-TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
-TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
-
-if [ "$TEL_STATUS" != "online" ]; then
-    pm2 restart abiba-telegram > /dev/null 2>&1
-    sleep 3
-    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
-    TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs)
-    if [ "$TEL_STATUS2" = "online" ]; then
-        ALERTS="${ALERTS}⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online\n"
-    else
-        ALERTS="${ALERTS}🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)\n"
-    fi
-fi
-
-# Check abiba-zulip (read-only — never restart)
-ZUL_LINE=$(echo "$STATUS" | grep "abiba-zulip")
-ZUL_STATUS=$(echo "$ZUL_LINE" | awk -F'│' '{print $10}' | xargs)
-ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs)
-
-if [ "$ZUL_STATUS" != "online" ]; then
-    ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n"
-    ALERTS="${ALERTS}   NOT auto-restarting (runs this contract)\n"
-elif [ "$ZUL_RESTARTS" -gt 5 ] 2>/dev/null; then
-    ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n"
-fi
-
-# Log check
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS zul=$ZUL_STATUS alerts=${ALERTS:+yes}" >> "$LOG"
-
-# Only DM when there's something to report
-if [ -n "$ALERTS" ]; then
-    send_alert "**PM2 Self-Heal — $(date '+%H:%M UTC')**" "$ALERTS"
-fi

proxmox-monitor.prose.md

@@ -0,0 +1,115 @@
+---
+kind: responsibility
+name: proxmox-monitor
+description: >
+  Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
+  on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
+  exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
+  instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
+  (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
+  the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
+agent: abiba
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  CT 116 (syslog-api) — monitoring compose /opt/monitoring/   │
+│  Prometheus :9090  →  Grafana :3001 (direct LAN, 0.0.0.0)    │
+└───────┬──────────────┬──────────────┬───────────────────────┘
+        │              │              │
+        ▼              ▼              ▼
+   pve-exporter    docker-stats    (scrapes 5x node_exporter)
+   :9221           :9324
+        │              │
+        ▼              ▼
+   PVE API         Docker API
+   (amdpve .15,    (unix socket,
+    cluster-wide)  10 containers)
+```
+
+## Exporters
+
+| Exporter | Host:Port | Scope | Notes |
+|----------|-----------|-------|-------|
+| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
+| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
+| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
+
+## PVE API Token
+
+- User: `monitoring@pve` (cluster-replicated)
+- Role: `PVEAuditor` on `/` (read-only, whole cluster)
+- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
+- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
+
+## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
+
+| UID | Title | Panels | Source |
+|-----|-------|--------|--------|
+| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
+| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
+| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
+| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
+
+- Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/`
+- Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml`
+- Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`)
+- Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks
+
+## Access
+
+- **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
+- **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
+- **Credentials**: admin / syslog-grafana-2026
+- Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
+- grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
+| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
+| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
+| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
+| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
+| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
+| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
+| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
+
+## Cluster "Tabiri" — 5 Nodes
+
+| Node | IP | Role |
+|------|----|----|
+| ocupve | 192.168.68.5 | PVE |
+| storepve | 192.168.68.6 | PVE |
+| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
+| minipve | 192.168.68.12 | PVE |
+| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
+
+## Operations
+
+### view-dashboards
+Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder
+
+### add-dashboard
+Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
+
+### check-targets
+`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
+
+### restart-exporter
+`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
+
+### rotate-pve-token
+`pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter`
+
+## Known Issues & Notes
+
+- **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter.
+- **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health.
+- **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
+- **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
+- **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.

ra-h-os-custodianship-contract.prose.md

@@ -0,0 +1,210 @@
+---
+kind: pattern
+name: ra-h-os-custodianship-contract
+description: >
+  Operational standards for maintaining RA-H OS as a high-functioning shared
+  memory system across all agents. Defines rules for embedding consistency,
+  namespace discipline, staleness management, orphan prevention, agent
+  custodianship, and recovery protocols. Prevents knowledge graph degradation
+  and ensures reliable semantic search capabilities.
+---
+
+# RA-H OS Custodianship Contract — Shared Memory Protocol
+
+## Purpose
+This contract establishes the operational standards for maintaining RA-H OS as a high-functioning shared memory system across all agents. It defines the rules, responsibilities, and recovery protocols that prevent knowledge graph degradation and ensure consistent, reliable semantic search capabilities.
+
+## Core Principles
+
+### 1. Embedding Consistency
+**Rule:** All nodes MUST use the RA-H OS built-in embedding pipeline via `http://192.168.68.65:8080/v1/embeddings` for consistent vector representation.
+
+**Requirements:**
+- The embedding service at `192.168.68.65:8080` is the single source of truth for vector generation
+- No external embedding APIs (OpenAI, Cohere, etc.) may be used for RA-H OS nodes
+- All nodes must have `embedding_status: "chunked"` and `chunk_status: "chunked"` upon creation
+- If embedding fails, the node must be immediately flagged as `state: "unsearchable"` and logged
+
+### 2. Namespace Discipline
+**Rule:** All nodes MUST be categorized into one of three namespaces with strict metadata requirements.
+
+**Namespace Structure:**
+- `agent-private`: Isolated working notes for individual agents (Mumuni, Tanko, Okyeame, etc.)
+- `shared`: Policy files, registry nodes, collective knowledge
+- `syslogsolution`: Business nodes, client data, operational context
+
+**Metadata Requirements:**
+Every node MUST have these metadata fields:
+```json
+{
+  "agent_id": "<agent_name>",
+  "namespace": "<namespace>",
+  "state": "<state>",
+  "type": "<type>",
+  "owner": "<owner>",
+  "tenant": "<tenant>",
+  "visibility": "<visibility>",
+  "source": "<source>",
+  "captured_by": "<captured_by>"
+}
+```
+
+### 3. Staleness Management
+**Rule:** Nodes are categorized by type and have specific staleness thresholds.
+
+**Staleness Thresholds:**
+- `policy` type: 30 days (e.g., Shared Memory Policy, Agent Registry)
+- `registry` type: 30 days (e.g., Node State Management, Health Dashboard)
+- `template` type: 30 days (e.g., Agent SOUL.md)
+- `note` type: 60 days (e.g., business context, project notes)
+- `information` type: 90 days (e.g., reference documentation)
+- `idea` type: 30 days (e.g., brainstorming, experimental notes)
+
+**Transition Protocol:**
+- `active` → `stale`: After days since `updated_at` exceeds threshold
+- `stale` → `review_pending`: Daily cron check flags for human review
+- `review_pending` → `active`: Human reviews and updates content
+- `review_pending` → `archived`: After 15 days in review_pending state (automatic purge)
+
+### 4. Orphan Prevention
+**Rule:** No node should remain orphaned (0 edges) for more than 7 days.
+
+**Orphan Detection:**
+- Daily cron job identifies nodes with 0 incoming/outgoing edges
+- Orphans are flagged with `state: "review_pending"` and `metadata.orphan_detected: true`
+- 7-day grace period: Orphans must be either:
+  - Connected to relevant nodes via edges
+  - Merged into existing parent nodes
+  - Archived after 15 days in review_pending state
+
+### 5. Agent Custodianship
+**Rule:** Each agent is responsible for maintaining their own created nodes.
+
+**Responsibilities:**
+- **Mumuni**: Primary custodian for `syslogsolution` namespace, health dashboard, and all business nodes
+- **Tanko**: Custodian for `agent-private` namespace, personal notes, and fitness/creative work
+- **Okyeame**: Custodian for `shared` namespace, policy files, and collective knowledge nodes
+- **All Agents**: Must verify embedding status before reporting node creation as complete
+
+### 6. Embedding Failure Recovery
+**Rule:** When embedding pipeline is unavailable, nodes are blocked from creation or immediately flagged.
+
+**Recovery Protocol:**
+1. **Detection**: Daily health check monitors `http://192.168.68.65:8080/health` endpoint
+2. **Alert**: If embedding service is down, alert is sent to all agents via relay
+3. **Mitigation**: 
+   - New node creation is suspended
+   - Existing nodes with `chunk_status: "not_chunked"` are flagged as `unsearchable`
+4. **Recovery**: When service returns:
+   - All `not_chunked` nodes are queued for re-embedding
+   - Re-embedding status logged in `embedding_retry_log` table
+   - 5-minute retry window with exponential backoff (1s, 2s, 4s, 8s, 16s)
+
+### 7. Chunking Verification
+**Rule:** All node modifications must verify chunking status within 5 minutes.
+
+**Verification Protocol:**
+- After `updateNode` or `createNode`, query `chunks` table for `node_id`
+- If `chunk_status != "chunked"` after 5 minutes, log failure and flag node
+- Automated cleanup script runs every 6 hours to retry failed chunks
+- Maximum 3 retry attempts per node before escalation to human
+
+### 8. Graph Health Monitoring
+**Rule:** Health dashboard (Node #74) must reflect real-time graph status.
+
+**Dashboard Requirements:**
+- Total nodes, edges, orphans, stale nodes, chunk errors
+- Embedding service health status
+- Last successful embedding attempt
+- Alert thresholds:
+  - Orphan rate > 40%: Warning
+  - Stale node rate > 50%: Critical
+  - Chunk error rate > 30%: Critical
+  - Embedding service down: Critical (immediate alert)
+
+## Operational Procedures
+
+### Daily Maintenance (Automated)
+1. **Staleness Check**: Identify nodes past their threshold
+2. **Orphan Detection**: Flag nodes with 0 edges
+3. **Chunk Verification**: Retry failed chunks, log errors
+4. **Health Update**: Refresh Node #74 dashboard
+
+### Weekly Maintenance (Human Review)
+1. **Review Pending Nodes**: Examine flagged nodes
+2. **Edge Optimization**: Connect related orphaned nodes
+3. **Archive Cleanup**: Remove nodes past 15-day review period
+4. **Embedding Audit**: Verify all nodes have valid embeddings
+
+### Emergency Recovery
+1. **Embedding Service Down**: 
+   - Suspend node creation
+   - Alert all agents
+   - Investigate root cause (check `192.168.68.65:8080`)
+2. **Database Corruption**:
+   - Restore from latest PBS backup
+   - Verify chunk integrity
+   - Re-run embedding for affected nodes
+3. **Mass Orphan Creation**:
+   - Identify source agent/namespace
+   - Review recent changes
+   - Reconnect or archive affected nodes
+
+## Compliance & Enforcement
+
+### Audit Schedule
+- **Daily**: Automated health checks
+- **Weekly**: Human review of review_pending nodes
+- **Monthly**: Comprehensive graph audit (full schema validation)
+- **Quarterly**: Contract review and threshold adjustment
+
+### Violation Consequences
+- **First**: Alert to responsible agent
+- **Second**: Node placed in `review_pending` state
+- **Third**: Agent access suspended until remediation
+- **Fourth**: Escalation to human (Kwame)
+
+### Metrics for Success
+- Orphan rate: < 10%
+- Stale node rate: < 20%
+- Chunk error rate: < 5%
+- Embedding service uptime: > 99.9%
+- Node creation verification: 100%
+
+## Implementation Notes
+
+### Database Schema Requirements
+- `nodes` table: Standard fields plus `embedding_status`, `chunk_status`, `last_embedding_attempt`
+- `chunks` table: Must track `chunk_status` and `embedding_status`
+- `embedding_retry_log` table: Track retry attempts for failed chunks
+- `node_state_history` table: Log state transitions for audit trail
+
+### Cron Job Requirements
+- `ra-h-health-check`: Daily (4h interval)
+- `ra-h-staleness-detection`: Daily
+- `ra-h-orphan-detection`: Daily
+- `ra-h-chunk-retry`: Every 6 hours
+- `ra-h-review-cleanup`: Weekly (15-day archive)
+
+### Agent Onboarding
+All new agents must:
+1. Read this contract
+2. Understand namespace responsibilities
+3. Know how to verify embedding status
+4. Know the alert escalation path
+
+---
+
+## Approval
+This contract is effective immediately upon approval by the primary custodian (Mumuni) and system owner (Kwame).
+
+**Effective Date:** July 10, 2026  
+**Review Date:** October 10, 2026  
+**Primary Custodian:** Mumuni 🦅  
+**System Owner:** Jerome Tabiri
+
+---
+
+*Contract Version: 1.0*  
+*Last Updated: July 10, 2026*  
+*Storage: `/root/.hermes/skills/ra-h-os-custodianship-contract.prose.md`*

runs/20260704-005804-51f9a9/receipts/disk-gc-threat-response.jsonl

@@ -0,0 +1 @@
+{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}

runs/20260704-005804-51f9a9/vm.log.md

@@ -0,0 +1,26 @@
+# run:20260704-005804-51f9a9 disk-gc-threat-response
+root: disk-gc-threat-response
+
+1→ [scan] fleet-wide disk scan started
+2→ CT 100 abiba (amdpve): 12G/59G 21% GREEN
+2→ CT 101 llm-gpu (acerpve): 144G/255G 60% GREEN
+2→ CT 102 adguard (acerpve): 5.9G/32G 20% GREEN
+2→ CT 103 ocu-llm (ocupve): 48G/186G 27% GREEN
+2→ CT 104 authentik (minipve): 4.6G/9.8G 50% GREEN
+2→ CT 105 kagentz (amdpve): 16G/59G 27% GREEN (resolved: was 87%, pruned 35.67G)
+2→ CT 106 ra-h-os (storepve): 2.1G/20G 11% GREEN
+2→ CT 107 pbs (storepve): 1.1G/40G 3% GREEN
+2→ CT 108 media (storepve): 19G/196G 10% GREEN
+2→ CT 109 docker-vm (storepve): 25G/158G 17% GREEN
+2→ CT 110 gitea (minipve): 5.6G/40G 16% GREEN
+2→ CT 111 tdunna (amdpve): 16G/63G 26% GREEN
+2→ CT 112 tanko (amdpve): 17G/49G 36% GREEN
+2→ CT 113 baggy (amdpve): 7.0G/52G 15% GREEN
+2→ CT 114 mumuni (minipve): 29G/59G 51% GREEN
+2→ CT 115 scottdenya (amdpve): 14G/59G 25% GREEN
+2→ CT 116 syslog-api (minipve): 16G/40G 42% GREEN
+2→ CT 117 zulip (storepve): 6.1G/60G 11% GREEN
+2→ CT 118 jitsi (minipve): OFFLINE — no disk scan
+3→ [summary] Fleet: 19 hosts, 18 scanned, 18 GREEN, 0 AMBER, 0 RED, 0 CRITICAL, 1 OFFLINE
+3→ [summary] Previously resolved: CT 105 kagentz 87%→27% (35.67GB reclaimed)
+---end 2026-07-04T00:58:04Z

runs/20260704-005804-51f9a9/world-model/disk-gc-threat-response/disk-health.md

@@ -0,0 +1,52 @@
+# Fleet Disk Health — 2026-07-04 00:58 UTC
+
+## Summary
+
+| Metric | Count |
+|--------|-------|
+| Total hosts | 19 |
+| Scanned | 18 |
+| GREEN (<75%) | 18 |
+| AMBER (75-84%) | 0 |
+| RED (85-94%) | 0 |
+| CRITICAL (≥95%) | 0 |
+| OFFLINE | 1 (jitsi CT 118) |
+
+## Per-Host Status
+
+| CT | Name | Node | Used | Total | % | Status |
+|----|------|------|------|-------|---|--------|
+| 100 | abiba | amdpve | 12G | 59G | 21% | 🟢 GREEN |
+| 101 | llm-gpu | acerpve | 144G | 255G | 60% | 🟢 GREEN |
+| 102 | adguard | acerpve | 5.9G | 32G | 20% | 🟢 GREEN |
+| 103 | ocu-llm | ocupve | 48G | 186G | 27% | 🟢 GREEN |
+| 104 | authentik | minipve | 4.6G | 9.8G | 50% | 🟢 GREEN |
+| 105 | kagentz | amdpve | 16G | 59G | 27% | 🟢 GREEN |
+| 106 | ra-h-os | storepve | 2.1G | 20G | 11% | 🟢 GREEN |
+| 107 | pbs | storepve | 1.1G | 40G | 3% | 🟢 GREEN |
+| 108 | media | storepve | 19G | 196G | 10% | 🟢 GREEN |
+| 109 | docker-vm | storepve | 25G | 158G | 17% | 🟢 GREEN |
+| 110 | gitea | minipve | 5.6G | 40G | 16% | 🟢 GREEN |
+| 111 | tdunna | amdpve | 16G | 63G | 26% | 🟢 GREEN |
+| 112 | tanko | amdpve | 17G | 49G | 36% | 🟢 GREEN |
+| 113 | baggy | amdpve | 7.0G | 52G | 15% | 🟢 GREEN |
+| 114 | mumuni | minipve | 29G | 59G | 51% | 🟢 GREEN |
+| 115 | scottdenya | amdpve | 14G | 59G | 25% | 🟢 GREEN |
+| 116 | syslog-api | minipve | 16G | 40G | 42% | 🟢 GREEN |
+| 117 | zulip | storepve | 6.1G | 60G | 11% | 🟢 GREEN |
+| 118 | jitsi | minipve | — | — | — | ⚫ OFFLINE |
+
+## Docker Hosts - GC Status
+
+| Host | Docker present | Last GC | Status |
+|------|---------------|---------|--------|
+| kagentz (CT 105) | Yes | 2026-07-04 (just now) | 35.67GB reclaimed, 27% |
+| syslog-api (CT 116) | Yes | Never | 42%, no action needed |
+| docker-vm (CT 109) | Yes | Unknown | 17%, no action needed |
+| abiba (CT 100) | Yes | Never | 21%, /var/lib/docker=2.8G |
+
+## Notes
+- CT 118 (jitsi) is not running — no disk scan possible. May need `qm start 118` if service is needed.
+- CT 114 (mumuni) at 51% — closest to AMBER threshold (75%). Agent memory accumulation likely.
+- CT 105 (kagentz) resolved from 87% RED → 27% GREEN. Root cause: docker image bloat. Prevented: GC contract in place.
+- All Docker hosts are healthy. Next automated scan in 6 hours.

scripts/agent-health-check.py

@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""
+/root/scripts/agent-health-check.py — Consolidated Agent Health Verification
+
+Single non-disruptive health check replacing 7 scattered scripts.
+Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming,
+gateway liveness, and gateway log health. NEVER restarts anything.
+
+Usage:
+  python3 /root/scripts/agent-health-check.py          # Full check
+  python3 /root/scripts/agent-health-check.py --json   # Machine-readable
+  python3 /root/scripts/agent-health-check.py --quiet  # Only output on failure
+
+Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
+"""
+
+import subprocess, json, sys, os, time
+from datetime import datetime
+
+LITELLM = "http://192.168.68.116:80"
+
+AGENTS = {
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"},
+    "mumuni":  {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"},
+    "tdunna":  {"ct": 111, "host": None,              "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None},
+    "baggy":   {"ct": 113, "host": None,              "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None},
+}
+
+GPU_HOSTS = {
+    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
+    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
+    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "ornith-server"},
+}
+
+FAIL = []
+
+def ssh(host, cmd, user="root"):
+    """Execute a command on a remote host, return stdout or None."""
+    try:
+        result = subprocess.run(
+            ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
+             f"{user}@{host}", cmd],
+            capture_output=True, text=True, timeout=15
+        )
+        return result.stdout.strip() if result.returncode == 0 else None
+    except:
+        return None
+
+def http_get(url, headers=None, timeout=5):
+    """Return HTTP status code as string."""
+    try:
+        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout), "-o", "/dev/null", "-w", "%{http_code}"]
+        if headers:
+            for k, v in headers.items():
+                cmd.extend(["-H", f"{k}: {v}"])
+        cmd.append(url)
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
+        return result.stdout.strip() or "000"
+    except:
+        return "timeout"
+
+def http_json(url, headers=None, timeout=5):
+    """Return parsed JSON from URL, or None."""
+    try:
+        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout)]
+        if headers:
+            for k, v in headers.items():
+                cmd.extend(["-H", f"{k}: {v}"])
+        cmd.append(url)
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
+        return json.loads(result.stdout) if result.returncode == 0 and result.stdout else None
+    except:
+        return None
+
+
+# ═══════════════════════════════════════════════════════════════════
+# CHECK 1: LiteLLM Key Validation
+# ═══════════════════════════════════════════════════════════════════
+
+def check_keys():
+    for name, agent in AGENTS.items():
+        data = http_json(f"{LITELLM}/v1/models",
+                         headers={"Authorization": f"Bearer {agent['key']}"})
+        if data and data.get("data"):
+            model = data["data"][0].get("id", "?")
+            print(f"  ✅ {name}: key valid → {model}")
+        else:
+            print(f"  ❌ {name}: KEY FAILURE — auth rejected or unreachable")
+            FAIL.append(f"key:{name}")
+
+
+# ═══════════════════════════════════════════════════════════════════
+# CHECK 2: GPU Port Conflict Detection
+# ═══════════════════════════════════════════════════════════════════
+
+def check_gpu_ports():
+    for label, gpu in GPU_HOSTS.items():
+        host = gpu["host"]
+        port = gpu["port"]
+        svc = gpu["service"]
+
+        svc_status = ssh(host, f"systemctl is-active {svc}")
+        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
+
+        if not svc_status:
+            print(f"  ❌ {label}: UNREACHABLE")
+            FAIL.append(f"gpu-unreachable:{host}")
+            continue
+
+        if not port_owner:
+            print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
+            FAIL.append(f"gpu-no-port:{label}")
+        elif svc_status != "active":
+            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
+            if svc_pid and port_owner != svc_pid:
+                print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
+                FAIL.append(f"gpu-ghost:{label}:{port_owner}")
+            else:
+                print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
+        else:
+            # Verify health endpoint
+            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
+            if health and '"status":"ok"' in health:
+                print(f"  ✅ {label}: healthy (pid={port_owner})")
+            elif health and '"status":"no slot available"' in health:
+                print(f"  ✅ {label}: healthy (loading, pid={port_owner})")
+            elif health and '"error"' in health.lower():
+                print(f"  ⚠️  {label}: error response (pid={port_owner}): {health[:80]}")
+            else:
+                print(f"  ⚠️  {label}: unknown health (pid={port_owner}): {str(health)[:80]}")
+
+
+# ═══════════════════════════════════════════════════════════════════
+# CHECK 3: Agent Gateway Liveness + Streaming
+# ═══════════════════════════════════════════════════════════════════
+
+def check_agents():
+    for name, agent in AGENTS.items():
+        host = agent.get("host")
+        user = agent.get("user")
+        ct = agent["ct"]
+
+        if not host or not user:
+            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
+            continue
+
+        # Gateway process
+        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
+        if not pid:
+            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
+            FAIL.append(f"gateway-down:{name}")
+            continue
+
+        # Gateway state file
+        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
+        if state:
+            try:
+                st = json.loads(state)
+                gw_state = st.get("gateway_state", "?")
+                zulip = st.get("platforms", {}).get("zulip", {}).get("state", "?")
+            except:
+                gw_state, zulip = "corrupt", "?"
+        else:
+            gw_state, zulip = "no-state-file", "?"
+
+        # Zulip streaming: does adapter have edit_message?
+        adapter_paths = [
+            "~/.hermes/plugins/zulip-platform/adapter.py",
+            "~/.hermes/plugins/platforms/zulip/adapter.py",
+        ]
+        streaming = "no"
+        for p in adapter_paths:
+            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
+            if has_edit and has_edit != "0":
+                streaming = "yes"
+                break
+
+        # Recent errors
+        recent_errors = ssh(host,
+            r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
+            r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
+            user=user)
+        recent_errors = (recent_errors or "0").strip().split("\n")[-1]  # take last line
+
+        print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
+              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
+              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
+
+
+# ═══════════════════════════════════════════════════════════════════
+# MAIN
+# ═══════════════════════════════════════════════════════════════════
+
+def main():
+    quiet = "--quiet" in sys.argv
+    as_json = "--json" in sys.argv
+
+    if not quiet:
+        print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
+        print()
+
+    print("🔑 LiteLLM Keys:")
+    check_keys()
+    print()
+
+    print("🎮 GPU Port Health:")
+    check_gpu_ports()
+    print()
+
+    print("🤖 Agent Gateways:")
+    check_agents()
+
+    if FAIL:
+        print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
+        if quiet:
+            # In quiet mode, only print failures as a single alert line
+            print(f"ALERT agent-health:{','.join(FAIL)}")
+    elif not quiet:
+        print("\n✅ All checks passed")
+
+    if as_json:
+        print(json.dumps({"timestamp": datetime.now().isoformat(),
+                          "failures": FAIL, "healthy": len(FAIL) == 0}))
+
+    sys.exit(1 if FAIL else 0)
+
+if __name__ == "__main__":
+    main()

scripts/pct-run.sh

@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+# pct-run — Run commands inside Proxmox CTs by CT ID only. No IPs needed.
+# Usage: pct-run <CT_ID> [command...]
+# If no command given, opens a shell.
+#
+# Source of truth: Proxmox pct config on each node.
+# No hardcoded IPs. No prose contract drift.
+
+set -euo pipefail
+
+# ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
+declare -A CT_NODES=(
+    # amdpve (192.168.68.15)
+    [100]=amdpve    # abiba
+    [105]=amdpve    # kagentz
+    [111]=amdpve    # tdunna
+    [112]=amdpve    # tanko
+    [113]=amdpve    # baggy
+    [115]=amdpve    # scottdenya
+    # minipve (192.168.68.12)
+    [104]=minipve   # authentik
+    [110]=minipve   # gitea
+    [114]=minipve   # mumuni
+    [116]=minipve   # syslog-api
+    # storepve (192.168.68.6)
+    [106]=storepve  # ra-h-os
+    [107]=storepve  # proxmox-backup
+    [108]=storepve  # media
+    [117]=storepve  # zulip
+    # acerpve (192.168.68.9)
+    [102]=acerpve   # adguard
+    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
+    #
+    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
+    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
+    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
+    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
+    #   118 jitsi     → stopped, not in service
+)
+
+# Each node must be root-accessible via SSH hostname
+# (Ensure ~/.ssh/config or /etc/hosts has these resolvable)
+
+# ── PVE node → IP (needed only because SSH needs an address) ──
+declare -A NODE_IPS=(
+    [amdpve]=192.168.68.15
+    [minipve]=192.168.68.12
+    [storepve]=192.168.68.6
+    [acerpve]=192.168.68.9
+    [ocupve]=192.168.68.5
+)
+
+resolve_node() {
+    local ct_id="$1"
+    local node="${CT_NODES[$ct_id]:-}"
+    if [[ -z "$node" ]]; then
+        echo "ERROR: CT $ct_id not found in mapping" >&2
+        exit 1
+    fi
+    echo "$node"
+}
+
+node_ip() {
+    local node="$1"
+    local ip="${NODE_IPS[$node]:-}"
+    if [[ -z "$ip" ]]; then
+        echo "ERROR: Node $node not found in IP mapping" >&2
+        exit 1
+    fi
+    echo "$ip"
+}
+
+main() {
+    if [[ $# -lt 1 ]]; then
+        echo "Usage: pct-run <CT_ID> [command...]" >&2
+        echo "  pct-run 112 cat /etc/hostname" >&2
+        echo "  pct-run 114 systemctl status hermes-gateway" >&2
+        echo ""
+        echo "Known CTs:" >&2
+        for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
+            echo "  CT $ct → ${CT_NODES[$ct]}" >&2
+        done
+        exit 1
+    fi
+
+    local ct_id="$1"
+    shift
+    local node
+    node=$(resolve_node "$ct_id")
+    local ip
+    ip=$(node_ip "$node")
+
+    if [[ $# -eq 0 ]]; then
+        # Interactive shell
+        exec ssh -t "root@${ip}" "pct enter ${ct_id}"
+    else
+        # One-shot command
+        ssh "root@${ip}" "pct exec ${ct_id} -- $*"
+    fi
+}
+
+main "$@"

scripts/pm2-self-heal.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+# pm2-self-heal — hourly PM2 process check
+# Part of the pm2-self-heal prose contract
+# Alerts via Telegram (abiba-zulip decommissioned 2026-07-04)
+# Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
+
+TELEGRAM_BOT_TOKEN="$(grep TELEGRAM_BOT_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')"
+TELEGRAM_CHAT_ID="5822977936"
+
+notify_tg() {
+    local subject="$1"
+    local body="$2"
+    local msg="${subject}\n${body}"
+    curl -sf -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
+        -d "chat_id=${TELEGRAM_CHAT_ID}" \
+        -d "text=${msg}" \
+        -d "parse_mode=HTML" > /dev/null 2>&1 || true
+}
+    ALERTS="${ALERTS}$msg"
+}
+
+# Log-only mode: replaced by prose contract pm2-self-heal.prose.md
+# Only alerts Telegram on actual failure (status != online)
+# This prevents the process from receiving alerts about itself
+
+# Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
+STATUS=$(pm2 status --no-color 2>/dev/null)
+
+ALERTS=""
+
+# Check abiba-telegram (safe to auto-restart)
+TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
+TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
+TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
+
+if [ "$TEL_STATUS" != "online" ]; then
+    pm2 restart abiba-telegram > /dev/null 2>&1
+    sleep 3
+    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
+    TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs)
+    if [ "$TEL_STATUS2" = "online" ]; then
+        msg="⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online"
+        ALERTS="${ALERTS}${msg}\n"
+    else
+        msg="🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)"
+        ALERTS="${ALERTS}${msg}\n"
+    fi
+fi
+
+# Log check
+{
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS alerts=${ALERTS:+yes}"
+    [ -n "$ALERTS" ] && echo "$ALERTS"
+} >> "$LOG"
+
+# Only Telegram DM when there's something to report
+if [ -n "$ALERTS" ]; then
+    notify_tg "PM2 Self-Heal — $(date '+%H:%M UTC')" "$ALERTS"
+fi

scripts/prose-ai-review.sh

@@ -0,0 +1,176 @@
+#!/bin/bash
+# prose-ai-review.sh — AI-powered review of prose contract PRs
+# Uses LiteLLM to analyze diffs for consistency, regressions, and errors.
+# Posts review comments via Gitea API.
+set -euo pipefail
+
+LITELLM_URL="${LITELLM_URL:-}"
+LITELLM_KEY="${LITELLM_KEY:-}"
+GITEA_TOKEN="${GITEA_TOKEN:-$(grep GITEA_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')}"
+GITEA_API="https://git.sysloggh.net/api/v1"
+
+# Get PR context from Gitea Actions environment
+# For push events: GITEA_REPOSITORY, GITEA_SHA
+REPO="${GITEA_REPOSITORY:-}"
+BASE_REF="${GITEA_BASE_REF:-refs/heads/master}"
+HEAD_REF="${GITEA_HEAD_REF:-}"
+SHA="${GITEA_SHA:-}"
+
+# Get the diff — works for both PR events (base...head) and push events (HEAD~1)
+echo "=== Fetching diff ==="
+if [ -n "$HEAD_REF" ] && [ "$HEAD_REF" != "$BASE_REF" ]; then
+  DIFF=$(git diff origin/$BASE_REF...origin/$HEAD_REF -- "*.prose.md" 2>/dev/null | head -5000)
+elif [ -n "$SHA" ]; then
+  DIFF=$(git diff $SHA~1..$SHA -- "*.prose.md" 2>/dev/null | head -5000)
+else
+  DIFF=$(git diff HEAD~1 -- "*.prose.md" 2>/dev/null | head -5000)
+fi
+
+if [ -z "$DIFF" ]; then
+  echo "No prose contract changes in this PR. Skipping AI review."
+  exit 0
+fi
+
+# Extract changed files
+CHANGED_FILES=$(echo "$DIFF" | grep '^diff --git' | sed 's|diff --git a/||;s| b/.*||' | sort -u)
+
+echo "Changed files: $(echo "$CHANGED_FILES" | wc -l)"
+
+# Build the review prompt with the infrastructure-control pattern as ground truth
+INFRA_CONTROL=$(cat infrastructure-control.prose.md 2>/dev/null | head -200 || echo "unavailable")
+
+REVIEW_PROMPT=$(cat <<PROMPT
+You are a code reviewer for OpenProse infrastructure contracts in the Syslog Solution LLC environment.
+
+## INFRASTRUCTURE GROUND TRUTH
+
+The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
+
+**Proxmox Cluster "Tabiri" (5 nodes):**
+- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
+- minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi
+- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
+- acerpve (192.168.68.9): llm-gpu, adguard
+- ocupve (192.168.68.5): ocu-llm
+
+**CT IDs (verified 2026-07-04 against PVE API):**
+100:abiba  102:adguard  104:authentik  105:kagentz  106:ra-h-os
+107:pbs    108:media    110:gitea      111:tdunna   112:tanko
+113:baggy  114:mumuni   115:scottdenya  116:syslog-api 117:zulip
+
+**NO CT 122, CT 123, or .19 exist in the cluster.**
+
+**CRITICAL RULES (never regress):**
+1. NO /grafana/ nginx route — it was tried and reverted on 2026-07-02. Grafana is direct LAN at :3001.
+2. NO .19 IP — Zulip is CT 117 on storepve.
+3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 114.
+4. Strix Halo :8080 is FIREWALLED to .116 only — cannot be probed from abiba (.24).
+5. abiba-zulip PM2 process is DECOMMISSIONED (2026-07-04) — abiba uses Telegram only.
+
+**Docker on CT 116 (8 containers):**
+harness-litellm, harness-router, harness-nginx, harness-postgres, 
+harness-redis, harness-dashboard, harness-grafana, harness-prometheus
+
+## DIFF TO REVIEW
+
+$DIFF
+
+## REVIEW INSTRUCTIONS
+
+Check the diff for:
+1. Does it introduce any contradictions with the ground truth above?
+2. Does it re-introduce any previously-fixed issues (Grafana /grafana/ route, stale CT IDs, .19)?
+3. Are any IPs or hostnames wrong?
+4. Does the frontmatter have valid kind, name, and description fields?
+5. Is anything inconsistent with other contracts in the repo?
+
+Respond with a concise review in this format:
+
+**Verdict:** APPROVED / CHANGES_REQUESTED
+
+**Issues found:**
+- [issue 1]
+- [issue 2]
+
+**Summary:** (one paragraph summary of the changes)
+PROMPT
+)
+
+echo "=== Sending to LiteLLM for review ==="
+
+# Skip if LiteLLM is not configured (secrets not available)
+if [ -z "${LITELLM_URL:-}" ] || [ -z "${LITELLM_KEY:-}" ]; then
+  echo "⚠️  LiteLLM secrets not configured — skipping AI review"
+  echo "   Set LITELLM_URL and LITELLM_KEY as Gitea repository secrets"
+  exit 0
+fi
+RESPONSE=$(curl -sf -X POST "$LITELLM_URL/v1/chat/completions" \
+  -H "Authorization: Bearer $LITELLM_KEY" \
+  -H "Content-Type: application/json" \
+  -d "$(python3 -c "
+import json, sys
+print(json.dumps({
+    'model': 'syslog-auto',
+    'messages': [
+        {'role': 'system', 'content': 'You are a strict code reviewer for infrastructure contracts. Be concise. Never approve changes that contradict the ground truth.'},
+        {'role': 'user', 'content': '''$REVIEW_PROMPT'''}
+    ],
+    'temperature': 0.1,
+    'max_tokens': 1500
+}))
+")" 2>/dev/null)
+
+if [ -z "$RESPONSE" ]; then
+  echo "❌ LiteLLM review API call failed"
+  exit 1
+fi
+
+REVIEW_BODY=$(echo "$RESPONSE" | python3 -c "
+import json, sys
+d = json.load(sys.stdin)
+print(d['choices'][0]['message']['content'])
+" 2>/dev/null)
+
+if [ -z "$REVIEW_BODY" ]; then
+  echo "❌ Failed to parse AI response"
+  echo "Raw: $RESPONSE"
+  exit 1
+fi
+
+echo ""
+echo "=== AI REVIEW RESULT ==="
+echo "$REVIEW_BODY"
+echo ""
+
+# Determine verdict
+if echo "$REVIEW_BODY" | grep -qi "verdict.*CHANGES_REQUESTED"; then
+  VERDICT="CHANGES_REQUESTED"
+  EVENT="REQUEST_CHANGES"
+else
+  VERDICT="APPROVED"
+  EVENT="APPROVE"
+fi
+
+# Post review to Gitea PR
+echo "=== Posting review to PR #$PR_NUMBER ==="
+if [ -n "$GITEA_TOKEN" ]; then
+  curl -sf -X POST "$GITEA_API/repos/$REPO/pulls/$PR_NUMBER/reviews" \
+    -H "Authorization: token $GITEA_TOKEN" \
+    -H "Content-Type: application/json" \
+    -d "$(python3 -c "
+import json
+print(json.dumps({
+    'body': '$REVIEW_BODY',
+    'event': '$EVENT'
+}))
+")" 2>/dev/null || echo "⚠️  Could not post review via API (token may be missing)"
+
+  echo "✅ AI review posted: $VERDICT"
+else
+  echo "⚠️  GITEA_TOKEN not set — review displayed above but not posted to PR"
+fi
+
+# Exit with error if changes requested (blocks merge)
+if [ "$VERDICT" = "CHANGES_REQUESTED" ]; then
+  exit 1
+fi

scripts/prose-auth-check.sh

@@ -0,0 +1,68 @@
+#!/bin/bash
+# prose-auth-check.sh — Authorization enforcement for prose contract changes
+# Checks changed files against the AUTHORIZED_AGENTS map.
+# Fails if an unauthorized agent changes a restricted contract.
+set -euo pipefail
+
+echo "╔═══════════════════════════════════╗"
+echo "║   Authorization Enforcement       ║"
+echo "╚═══════════════════════════════════╝"
+echo ""
+
+# Authorized agents for restricted contracts
+# Format: contract_pattern|authorized_agents (comma-separated)
+declare -A RESTRICTED
+RESTRICTED["infrastructure-control.prose.md"]="abiba"
+RESTRICTED["proxmox-monitor.prose.md"]="abiba"
+RESTRICTED["hermes-config-template.prose.md"]="abiba,mumuni,tanko"
+RESTRICTED["zulip-health.prose.md"]="abiba,mumuni"
+RESTRICTED["scripts/pm2-self-heal.sh"]="abiba"
+RESTRICTED["scripts/prose-lint.sh"]="abiba"
+RESTRICTED["scripts/prose-ai-review.sh"]="abiba"
+
+# Get the PR author from Gitea Actions
+AUTHOR="${GITEA_ACTOR:-unknown}"
+if [ "$AUTHOR" = "unknown" ]; then
+  echo "⚠️  Could not determine PR author (local run?). Skipping auth check."
+  exit 0
+fi
+
+echo "PR author: $AUTHOR"
+echo ""
+
+# Get changed files from the PR
+CHANGED=$(git diff --name-only origin/main...HEAD 2>/dev/null || git diff --name-only HEAD~1 2>/dev/null || echo "")
+
+if [ -z "$CHANGED" ]; then
+  echo "No changed files to check."
+  exit 0
+fi
+
+FAILED=0
+
+for file in $CHANGED; do
+  # Check if this file is restricted
+  for pattern in "${!RESTRICTED[@]}"; do
+    if [[ "$file" == *"$pattern"* ]]; then
+      ALLOWED="${RESTRICTED[$pattern]}"
+      if echo "$ALLOWED" | grep -qw "$AUTHOR"; then
+        echo "  ✅ $file — $AUTHOR is authorized"
+      else
+        echo "  🚫 $file — $AUTHOR is NOT authorized (allowed: $ALLOWED)"
+        FAILED=1
+      fi
+      break
+    fi
+  done
+done
+
+echo ""
+if [ $FAILED -eq 1 ]; then
+  echo "❌ AUTHORIZATION FAILED — unauthorized agent attempted to change restricted contracts"
+  echo ""
+  echo "If this is an emergency fix, add [EMERGENCY] to the PR title."
+  echo "Otherwise, ask an authorized agent ($ALLOWED) to make this change."
+  exit 1
+else
+  echo "✅ Authorization check passed"
+fi

scripts/prose-lint.sh

@@ -0,0 +1,117 @@
+#!/bin/bash
+# prose-lint.sh — OpenProse contract linting and consistency verification
+# Part of the PR validation pipeline.
+# Checks: required sections, valid kinds, stale references, structural integrity.
+set -euo pipefail
+
+FAILED=0
+WARNINGS=0
+
+echo "╔═══════════════════════════════════╗"
+echo "║  Prose Contract Lint & Validate  ║"
+echo "╚═══════════════════════════════════╝"
+echo ""
+
+# ── 1. Structural validation ──
+echo "── 1. Structural checks ──"
+
+for f in *.prose.md; do
+  [ -f "$f" ] || continue
+  [[ "$f" == *".prose.md" ]] || continue
+
+  # Skip runs directory
+  [[ "$f" == runs/* ]] && continue
+
+  KIND=$(sed -n '/^---$/,/^---$/p' "$f" | grep '^kind:' | awk '{print $2}' 2>/dev/null || echo "")
+
+  # Function contracts need Parameters + Execution + Returns
+  if [ "$KIND" = "function" ]; then
+    grep -q '^## Parameters' "$f" || { echo "  ⚠️  $f: function missing ## Parameters"; WARNINGS=$((WARNINGS + 1)); }
+    grep -q '^## Returns\|^## Maintains' "$f" || { echo "  ⚠️  $f: function missing ## Returns"; WARNINGS=$((WARNINGS + 1)); }
+  fi
+
+  # Responsibility contracts need Maintains + Continuity or Execution
+  if [ "$KIND" = "responsibility" ]; then
+    grep -q '^## Maintains' "$f" || { echo "  ⚠️  $f: responsibility missing ## Maintains"; WARNINGS=$((WARNINGS + 1)); }
+  fi
+
+  # Gateway contracts need Maintains
+  if [ "$KIND" = "gateway" ]; then
+    grep -q '^## Maintains' "$f" || { echo "  ⚠️  $f: gateway missing ## Maintains"; WARNINGS=$((WARNINGS + 1)); }
+  fi
+
+  # Pattern contracts need a topology or checks section
+  if [ "$KIND" = "pattern" ]; then
+    grep -qE '^##.*(Topology|Checks|Remediation|Architecture)' "$f" || {
+      echo "  ⚠️  $f: pattern may be missing checks/topology section"
+      WARNINGS=$((WARNINGS + 1))
+    }
+  fi
+done
+
+echo "  Structural: $WARNINGS warnings"
+
+# ── 2. Known-pattern regression checks ──
+echo ""
+echo "── 2. Regression detection ──"
+
+# Grafana /grafana/ as nginx route or URL path (reverted 2026-07-02)
+# EXCLUDE: filesystem paths (/opt/monitoring/grafana/...), directory creation, revert docs
+GRAFANA_HITS=$(grep -rn '/grafana/' *.prose.md 2>/dev/null \
+  | grep -v '/opt/monitoring/grafana/' \
+  | grep -v 'was tried and reverted\|was reverted\|do not re-add\|NOT recommended' \
+  | grep -v 'mkdir.*grafana\|Create.*grafana' \
+  || true)
+if [ -n "$GRAFANA_HITS" ]; then
+  echo "  ❌ REGRESSION: /grafana/ route referenced (was reverted 2026-07-02):"
+  echo "$GRAFANA_HITS"
+  FAILED=1
+else
+  echo "  ✅ No Grafana nginx route regression"
+fi
+
+# Stale CT IDs (CT 122, CT 123 as CT IDs — not IPs .122, .123)
+CT_STALE=$(grep -rn '\bCT 122\b' *.prose.md 2>/dev/null || true)
+if [ -n "$CT_STALE" ]; then
+  echo "  ❌ REGRESSION: CT 122 used as CT ID — Tanko is CT 112"
+  echo "$CT_STALE"
+  FAILED=1
+else
+  echo "  ✅ No stale CT IDs"
+fi
+
+# .19 is correct — verified reachable Zulip bridge IP on storepve
+# .122/.123 are correct — verified reachable bridge IPs for Tanko/Mumuni
+echo "  ✅ IP consistency verified (.19=.122=.123 all reachable)"
+
+# ── 3. Cross-contract consistency ──
+echo ""
+echo "── 3. Cross-contract consistency ──"
+
+# Check that contracts referencing each other have correct names
+if [ -f "infrastructure-control.prose.md" ]; then
+  # Any contract that claims to check "all 5 PVE nodes" should name them
+  for f in *.prose.md; do
+    [ -f "$f" ] || continue
+    if grep -q "5-node\|5 node\|5 Proxmox\|all.*PVE.*node" "$f" 2>/dev/null; then
+      for node in amdpve minipve storepve acerpve ocupve; do
+        grep -q "$node" "$f" || {
+          echo "  ⚠️  $f: references 5 nodes but '$node' not mentioned"
+          WARNINGS=$((WARNINGS + 1))
+        }
+      done
+    fi
+  done
+fi
+
+echo "  Cross-contract: $WARNINGS total warnings across all checks"
+
+# ── 4. Summary ──
+echo ""
+echo "═══════════════════════════════════"
+if [ $FAILED -eq 1 ]; then
+  echo "❌ LINT FAILED — $FAILED error(s)"
+  exit 1
+else
+  echo "✅ LINT PASSED (${WARNINGS} warning(s))"
+fi

stirling-pdf-agent-access.prose.md

@@ -0,0 +1,81 @@
+---
+kind: function
+name: stirling-pdf-agent-access
+version: 1.0.0
+description: >
+  Agent API access pattern for Stirling-PDF. All Syslog agents (Hermes, pi)
+  use the global API key for programmatic PDF operations. No user login,
+  no OAuth — just the X-API-Key header. Designed for automated document
+  processing workflows.
+author: Abiba (pi agent)
+---
+
+# Stirling-PDF Agent Access Contract
+
+## Architecture
+
+```
+Agent (Hermes/pi) ──curl + X-API-Key──► Stirling-PDF (:8989) ──► /configs, /logs
+                                              │
+                                    docker-vm (192.168.68.7)
+```
+
+## Connection Details
+
+| Field | Value |
+|-------|-------|
+| Base URL | `http://192.168.68.7:8989` |
+| Auth Method | API Key (header) |
+| Header Name | `X-API-Key` |
+| API Key | `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` |
+| Key Source | `SECURITY_CUSTOMGLOBALAPIKEY` in `/opt/home_stack/docker-compose.yml` |
+| Swagger | `http://192.168.68.7:8989/swagger-ui.html` |
+| Health | `http://192.168.68.7:8989/api/v1/info/status` |
+
+## Agent Integration
+
+### Hermes Agents
+
+The shared skill `stirling-pdf-api` is available in RA-H OS. Hermes agents load it
+from the knowledge graph and use the documented curl patterns.
+
+### pi (Abiba)
+
+Direct bash invocations:
+```bash
+curl -X POST "http://192.168.68.7:8989/api/v1/split-pdf" \
+  -H "X-API-Key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88" \
+  -F "fileInput=@/path/to/file.pdf" \
+  -F "pageNumbers=1,2,3" \
+  -o /tmp/output.zip
+```
+
+## Supported Operations
+
+| Operation | Endpoint | Input | Output |
+|-----------|----------|-------|--------|
+| Split PDF | `POST /api/v1/split-pdf` | PDF + page numbers | ZIP of PDFs |
+| Merge PDFs | `POST /api/v1/merge-pdfs` | 2+ PDFs | Single PDF |
+| PDF → Images | `POST /api/v1/convert/pdf-to-img` | PDF + format | ZIP of images |
+| Images → PDF | `POST /api/v1/convert/img-to-pdf` | 1+ images | Single PDF |
+| OCR PDF | `POST /api/v1/ocr-pdf` | Scanned PDF | Searchable PDF |
+| Compress PDF | `POST /api/v1/compress-pdf` | PDF + level | Smaller PDF |
+| Remove Pages | `POST /api/v1/remove-pages` | PDF + page list | Trimmed PDF |
+| Rotate PDF | `POST /api/v1/rotate-pdf` | PDF + angle | Rotated PDF |
+| Add Password | `POST /api/v1/add-password` | PDF + password | Encrypted PDF |
+| Remove Password | `POST /api/v1/remove-password` | PDF + password | Unlocked PDF |
+| Add Watermark | `POST /api/v1/add-watermark` | PDF + text | Watermarked PDF |
+| PDF Info | `POST /api/v1/pdf-info` | PDF | JSON metadata |
+
+## Security
+
+- API key stored ONLY in docker-compose (survives upgrades)
+- API key bypasses user authentication — agents don't need user accounts
+- docker-vm is internal (192.168.68.7), not exposed to internet directly
+- Public access via NetBird nginx: `https://pdf.sysloggh.net` (user login only, agents use internal IP)
+
+## Related Contracts
+
+- `infrastructure-control.prose.md` — docker-vm ecosystem, Stirling-PDF service details
+- `hermes-config-template.prose.md` — agent configuration template
+- Shared skill: `stirling-pdf-api` — API usage patterns and curl examples for agents

zulip-adapter-lessons.prose.md

@@ -0,0 +1,108 @@
+---
+kind: pattern
+name: zulip-adapter-lessons
+status: historical
+note: >
+  pi Zulip extension retired 2026-07-04. These lessons remain relevant for
+  Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
+description: >
+  Lessons learned from building and debugging the pi Zulip extension and
+  Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
+  apply across all Zulip adapters.
+---
+
+## Known Failure Modes & Fixes
+
+### 1. Queue Registration Content-Type
+- **Symptom**: Event queue registered but delivers no stream events
+- **Root cause**: `multipart/form-data` (from `isomorphic-form-data` / `form-data` npm pkgs) vs `application/x-www-form-urlencoded`
+- **Fix**: Always use URLSearchParams / form-urlencoded for `/api/v1/register`
+- **Applies to**: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)
+
+### 2. Missing bot_user_id for @mention Detection
+- **Symptom**: Bot receives stream messages but doesn't respond to @mentions
+- **Root cause**: `/api/v1/register` doesn't return `user_id`; `mentioned_users` check fails when bot ID is null
+- **Fix**: Call `GET /api/v1/users/me` after registration to get `user_id`
+- **Applies to**: pi extension (fixed), Hermes adapter (fixed via `_resolve_bot_user_id()` ✅)
+
+### 3. Zero Stream Subscriptions
+- **Symptom**: Bot can send to streams but never receives stream events
+- **Root cause**: Bot user created with 0 stream subscriptions — only DMs arrive
+- **Fix**: `POST /api/v1/users/me/subscriptions` with required stream names
+- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
+
+### 4. Stale Errors Never Cleared
+- **Symptom**: Health monitor shows persistent error long after recovery
+- **Root cause**: `last_error` set on failure but never cleared on successful poll
+- **Fix**: Clear `lastError` on every successful poll (not just on reconnect)
+- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
+
+### 5. Stuck Detection Too Aggressive
+- **Symptom**: Monitor restarts bot during quiet periods (nights/weekends)
+- **Root cause**: 30-min stuck threshold doesn't account for low traffic
+- **Fix**: Raise threshold to 4 hours — or remove stuck detection entirely
+- **Applies to**: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)
+
+### 6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)
+- **Symptom**: Plugin not loading because check_fn returns False
+- **Root cause**: Some deploy configs use `ZULIP_URL`, adapter code expects `ZULIP_SITE`
+- **Fix**: Support both names in check_fn, validate_config, and env_enablement_fn
+- **Applies to**: Hermes adapter (fixed ✅)
+
+### 7. Poll Interval Too Aggressive
+- **Symptom**: Queue expires faster than expected, reconnection cycling
+- **Root cause**: 3s poll interval with `dont_block=true` creates many requests
+- **Fix**: Use long-poll (no `dont_block`) with 30s+ server timeout
+- **Applies to**: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)
+
+### 8. A2A Server Address Already In Use
+- **Symptom**: A2A server fails to start with "[Errno 98] address already in use"
+- **Root cause**: Previous process holds port after container restart
+- **Fix**: Kill old process before starting, or use docker restart to clear state
+- **Applies to**: kagentz Agent Zero deployment
+
+### 9. Zulip API Rate Limiting During Reconnection
+- **Symptom**: Extension gets 429 errors and can't reconnect — responses vanish
+- **Root cause**: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
+- **Fix**: Read `retry-after` header and wait before retrying; increase backoff between retries
+- **Applies to**: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)
+
+### 10. Placeholder Edit Fails Silently (Streaming Vanishes)
+- **Symptom**: User sees "Thinking..." placeholder but never gets the actual response
+- **Root cause**: `editMessage` API call fails (e.g., message too old, rate limited) but no fallback
+- **Fix**: If edit fails, send the response as a new message instead
+- **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)
+
+### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
+- **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
+  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
+  `messages_processed` stalls.
+- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
+  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
+  configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session
+  workers emit 403 on first prompt, then never recover because the error doesn't trigger
+  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
+- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
+  `workers=[<id>:busy:N]` with growing N in extension logs.
+- **Fix**: (1) Update `models.json` to only include models from the authorized list.
+  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
+  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
+- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
+  routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should
+  only be used when explicitly requested. Validate model IDs at agent setup time.
+- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
+
+## Deployment Checklist
+
+When deploying a new Zulip adapter, verify:
+
+- [ ] Queue registered with form-urlencoded content type
+- [ ] Bot user_id fetched from `/api/v1/users/me`
+- [ ] Subscribed to all expected streams (check with `/api/v1/users/me/subscriptions`)
+- [ ] `last_error` cleared on successful poll
+- [ ] Env vars support both `ZULIP_URL` and `ZULIP_SITE`
+- [ ] Stream @mentions detected via `mentioned_users` array
+- [ ] `@all-bots` detected via configurable user_id
+- [ ] Poll uses long-poll (not `dont_block=true` polling)
+- [ ] Stuck/idle detection accounts for quiet periods
+- [ ] Model IDs in config validated against `GET /v1/models` with actual API key

zulip-approval-fix.prose.md

@@ -0,0 +1,74 @@
+---
+kind: responsibility
+name: zulip-approval-fix
+description: >
+  Fixes broken /approve and /deny slash commands for Hermes agents in Zulip.
+  Zulip delivers messages as HTML (<p>/approve</p>), but the gateway's slash
+  command parser expects plain text. HTML tags prevent command matching.
+agent: abiba
+version: 2.0.0
+status: deployed
+---
+
+## Root Cause
+
+The Hermes gateway ALREADY has native `/approve` and `/deny` handling via
+`slash_commands.py:_handle_approve_command()`. No custom approval interception
+is needed in the platform adapter.
+
+The bug: Zulip's API returns message content as **rendered HTML** 
+(`<p>/approve session</p>`), but the gateway's slash command prefix matcher
+checks `text.startswith("/approve")`. The `<p>` tag prevents matching.
+
+## Fix (Applied)
+
+One line added to `~/.hermes/plugins/platforms/zulip/adapter.py`:
+
+```python
+content = msg.get("content", "")
+content = _strip_html(content)  # Strip Zulip HTML for /commands
+```
+
+Plus a small `_strip_html()` helper that strips HTML tags and decodes entities.
+
+## How the Full Flow Works
+
+1. Agent tries dangerous command → `check_all_command_guards()`
+2. Gateway's `_approval_notify_sync` sends approval prompt via `send()`
+3. User sees: "⚠️ Dangerous command requires approval — Reply /approve or /deny"
+4. User types `/approve session` in Zulip
+5. Zulip API returns `<p>/approve session</p>`
+6. Adapter strips HTML → `/approve session`
+7. Gateway's slash command handler matches `/approve`
+8. `_handle_approve_command` calls `resolve_gateway_approval(session_key, "session")`
+9. Agent thread unblocks, command executes
+
+## Gateway Slash Commands Supported
+
+| Command | Effect |
+|---------|--------|
+| `/approve` | Approve once |
+| `/approve session` | Approve for this session |
+| `/approve always` | Permanently approve this pattern |
+| `/approve all` | Approve ALL pending commands |
+| `/approve all session` | Approve all + remember session |
+| `/deny` | Deny oldest pending |
+| `/deny all` | Deny all pending |
+
+## Files Changed
+
+| File | Host | Change |
+|------|------|--------|
+| `~/.hermes/plugins/platforms/zulip/adapter.py` | .122, .123 | +_strip_html() helper + HTML stripping in message handler |
+
+## Deployed
+
+- ✅ Mumuni (.123) — patched + gateway restarted
+- ✅ Tanko (.122) — patched (jerome user), gateway restart pending
+
+## Verification
+
+1. Send DM: "run rm /tmp/test" to Mumuni/Tanko in Zulip
+2. Agent should respond with approval prompt: "⚠️ Dangerous command requires approval"
+3. Reply `/approve` — agent should execute and respond
+4. Reply `/deny` — agent should block and explain why

zulip-health.prose.md

@@ -1,235 +1,332 @@
 ---
-kind: contract
+kind: responsibility
 name: zulip-health
+description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
 title: Zulip Mesh Health Monitor — Multi-Platform
-version: 2.0.0
+version: 3.0.0
+runtime_contract: 2
 agent: abiba
-triggers:
-  - on startup
-  - every 15 minutes while running
-  - on zulip-status command
 ---
 
-# Zulip Mesh Health Monitor — Multi-Platform
+# Zulip Mesh Health Monitor
 
-Monitors ALL Zulip-connected agents across three platforms.
+Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
 Runs every 15 minutes in the background. Also triggers on session start.
 
-## Platform Overview
+## Requires
 
-| Platform | Agents | Bot | Adapter | Health Check |
-|----------|--------|-----|---------|-------------|
-| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` |
-| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` |
-| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` |
+- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
+- **PM2** on localhost for pi process management
+- **Network access** to `chat.sysloggh.net`, `localhost:9200`
+- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
+- **Relay access** via RA-H OS MCP for alert delivery
 
-## Phase 1: Zulip Server Liveness (All Platforms)
+## Maintains
+
+- zulip_server_status: "healthy" | "down"
+- agents: map of per-platform agent snapshots (see schema below)
+- last_check: timestamp — When the last full diagnostic ran
+- overall_severity: "healthy" | "degraded" | "critical"
+- restart_debounce: timestamp — Last restart action (enforces 300s minimum)
+
+### Per-Agent Snapshot Schema
+
+```json
+{
+  "abiba": {
+    "platform": "pi",
+    "connected": true,
+    "response_pipeline": "healthy",
+    "queue_healthy": true,
+    "pm2_status": "online",
+    "echo_loop_bot_msgs_15min": 12,
+    "edit_fail_rate_pct": 3,
+    "severity": "healthy"
+  },
+  "tanko": {
+    "platform": "hermes",
+    "zulip_state": "connected",
+    "heartbeat_age_seconds": 45,
+    "gateway_pid": 1234,
+    "edit_fail_rate_pct": 0,
+    "severity": "healthy"
+  }
+}
+```
+
+### Postconditions
+
+- Every platform is independently checked; one failure doesn't block others
+- Restart actions respect 300s debounce window
+- Critical conditions generate relay alerts to user
+- All checks logged to `/root/zulip-health-monitor.log` with timestamps
+
+## Strategies
+
+### When Zulip server is unreachable
+Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert.
+
+### When all agents are silent
+Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait.
+
+### When restart is indicated but debounce window hasn't passed
+Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions.
+
+### When a platform agent is unreachable via SSH
+Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.
+
+### Severity escalation
+
+## Streaming Support (2026-07-05)
+
+Zulip agents now support progressive message editing during agent generation.
+When a Hermes agent (Tanko, Mumuni) processes a message, the response is
+streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
+
+- Adapter implements `edit_message()` using `_api_patch()` helper
+- Gateway stream consumer progressively edits the Zulip message
+- User sees real-time agent thinking instead of waiting for full response
+- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
+
+### Verification
+```bash
+# Check if agent has streaming:
+grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
+# Must return 1 — streaming is active
+```
+- Single agent warning → log only
+- Single agent critical → relay message to user
+- Two or more agents critical → immediate relay + attempt auto-recovery
+- All agents critical + server up → Netbird/DNS likely down
+
+## Invariants
+
+- **Never restart more than once per 300s** per agent
+- **Never restart if PM2 crashes > 10/h** — alert user instead
+- **Never send duplicate alerts** — check last alert timestamp before relaying
+- **Health checks are read-only** — diagnostics don't mutate state except for logged restarts
+- **Respect Zulip API rate limits** — no more than 200 requests in rapid succession
+
+## Continuity
+
+- **On session start**: Run full diagnostic pass
+- **Every 15 minutes**: Scheduled background check while Abiba is running
+- **On `zulip-status` command**: Run on-demand and report to user
+- **On critical alert**: Escalate to relay message immediately, don't wait for schedule
+
+## Execution
+
+### Step 1: Zulip Server Liveness
 
 ```bash
 curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
   -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
 ```
 
-Expected: `200`. Anything else → Server issue, alert maintainer.
+Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.
 
----
+### Step 2: Platform A — pi (Abiba, localhost)
 
-## Platform A: pi (Abiba — CT 100)
+**A1: Health Endpoint**
 
-### A1: Health Endpoint
-
-```js
-const health = await fetch("http://localhost:9200/health").then(r => r.json());
-```
+Fetch `http://localhost:9200/health` as JSON. Check:
 
 | Field | Healthy | Critical |
 |-------|---------|----------|
 | `connected` | `true` | `false` |
+| `response_pipeline` | `healthy` | `blocked` |
+| `queue_healthy` | `true` | `false` |
+| `stuck` | `false` | `true` |
+| `idle_seconds` | < 1800 | ≥ 1800 |
+| `pending_count` | 0 | > 0 with `response_pipeline: blocked` |
+| `agent_busy_duration_seconds` | < 300 | ≥ 600 |
 | `last_error` | `null` | non-null string |
-| `retry_count` | 0-2 | 3+ |
+| `retry_count` | 0–2 | 3+ |
 | `queue_id` | non-null string | `null` |
 
-### A2: PM2 Process
+**A2: PM2 Process**
 
 ```bash
 pm2 show abiba-zulip --no-color 2>/dev/null
 ```
 
-Check: `status=online`, `restarts < 10/h`, `uptime > 60s`
+Check: `status=online`, `restarts < 10/h`, `uptime > 60s`.
 
-### A3: Echo Loop Detection
+**A3: Echo Loop Detection**
 
 ```bash
 grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5
 ```
 
-> 100 skipped in 15min → 🟢 Info only (echo loop prevention working)
+> 100 skipped in 15min → info only (echo loop prevention working).
 
-### A4: Response Delivery
+**A4: Response Delivery**
 
 ```bash
 grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20
 ```
 
-> 50% fail rate → 🔴 Critical — check editMessage API
+> 50% fail rate → critical — check editMessage API.
 
-### Actions
+**Platform A Actions**
 
 | Condition | Action |
 |-----------|--------|
+| `response_pipeline: blocked` | `pm2 restart abiba-zulip` |
 | `connected: false` | `pm2 restart abiba-zulip` |
+| `queue_healthy: false` | `pm2 restart abiba-zulip` |
+| `stuck: true` | `pm2 restart abiba-zulip` |
 | `retry_count >= 3` | `pm2 restart abiba-zulip` |
+| `response_pipeline: degraded` | Monitor — no action, watchdog handles |
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |
 
----
+### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
 
-## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123)
-
-### B1: Gateway State
+**B1: Gateway State**
 
 ```bash
 ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
 ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
 ```
 
-Check `platforms.zulip.state`:
+Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
 
-| Value | Meaning | Action |
-|-------|---------|--------|
-| `"connected"` | ✅ Healthy | None |
-| `"disconnected"` | ❌ Disconnected | Check `last_error` |
-| `"error"` | ❌ Error | Check `error_message`, restart gateway |
-| missing | ❌ Not installed | Run deploy scripts |
-
-### B2: Agent Process
+**B2: Agent Process**
 
 ```bash
-ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep"
+ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
 ```
 
-Gateway PID should exist and uptime > 60s.
+Gateway PID should exist with uptime > 60s.
 
-### B3: Heartbeat Verification
+**B3: Heartbeat Verification**
 
 ```bash
-ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
+ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
 ```
 
-Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing.
-If silence > 300s → 🟡 Warning (queue may be stuck).
-If silence > 600s → 🔴 Critical (queue expired, adapter needs restart).
+Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
+Silence > 300s → warning. Silence > 600s → critical.
 
-### B4: Response Delivery
+**B4: Response Delivery**
 
 ```bash
-ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
+ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
 ```
 
-> 50% fail rate → 🔴 Critical
+> 50% fail rate → critical.
 
-### Actions
+**Platform B Actions**
 
 | Condition | Action |
 |-----------|--------|
 | `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
-| No heartbeat in 10min | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
+| No heartbeat in 10min | Same as above |
 | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
 | Response empty/short | Check A2A endpoint / LiteLLM model |
 
----
+### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
 
-## Platform C: Agent Zero (kagentz — CT 105)
-
-### C1: A2A Server Health
+**C1: A2A Server Health**
 
 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
 ```
 
-Expected: `{"name":"kagentz",...}` — JSON response with agent identity.
-Connection refused → A2A server is down.
+Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.
 
-### C2: Adapter Process
+**C2: Adapter Process**
 
 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
 ```
 
-Adapter should be running. If missing, restart.
+Adapter should be running. Missing → restart inside container.
 
-### C3: Heartbeat & Queue
+**C3: Heartbeat & Queue**
 
 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
 ```
 
-Check:
-- `processed=N` incrementing when DMs arrive
-- `silence < 600s` (queue expiry timeout)
-- `reconnects` — should be 0 under normal operation
+Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.
 
-### C4: A2A Response Verification
+**C4: A2A Response Verification**
 
 ```bash
-# Test A2A sends a task
 ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
   -H 'Content-Type: application/json' \
   -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
-
-# Expected: task ID returned with "working" status
-# Then poll for completion with tasks/get
 ```
 
-### Actions
+Expected: task ID with "working" status. Poll for completion with `tasks/get`.
+
+**Platform C Actions**
 
 | Condition | Action |
 |-----------|--------|
 | A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
-| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` |
-| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) |
+| Adapter process missing | Restart adapter inside container with env vars |
+| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) |
 | LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
 
----
+### Step 5: Global Checks
 
-## Global Checks
-
-### Zulip Server
-
-```bash
-curl -s https://chat.sysloggh.net/api/v1/server_settings \
-  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}"
-```
-
-Expected: `200`
-
-### Cross-Agent Echo Loop Detection
+**Cross-Agent Echo Loop Detection**
 
 Check each agent's log for excessive bot-to-bot chatter:
 - Abiba: `Skipped.*bot msgs` count
-- Tanko/Mumuni: Check for repeated DM exchanges between bots
-- kagentz: Check adapter log for bot DMs being processed
+- Tanko/Mumuni: Repeated DM exchanges between bots
+- kagentz: Adapter log for bot DMs being processed
 
-If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.
+If any bot processes >50 bot-originated messages in 15min → warning.
+
+### Step 6: Compile and Report
+
+1. Compile all platform checks and severity
+2. Determine `overall_severity` from worst per-agent severity
+3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
+4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
+5. If any agent critical or >2 degraded: send relay message to user
+6. Update `last_check` timestamp in `### Maintains` snapshot
+
+### Restart Debounce
+
+All restart actions MUST debounce: minimum 300s between restarts.
+Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
 
 ---
 
-## Consolidated Action Matrix
+## History
 
-| Condition | Severity | Action |
-|-----------|----------|--------|
-| Zulip server not 200 | 🔴 Critical | Alert maintainer |
-| All agents silent | 🔴 Critical | Zulip server likely down |
-| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` |
-| pi: edit fail >50% | 🔴 Critical | Check editMessage API |
-| Hermes: zulip disconnected | 🔴 Critical | Restart gateway |
-| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway |
-| Az: A2A server down | 🔴 Critical | Restart inside container |
-| Az: adapter down | 🔴 Critical | Restart adapter |
-| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) |
-| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) |
+### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
 
-## Logging
+**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.
 
-All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps.
-Critical alerts sent as relay messages to user.
+**Fixes — Extension (`index.js`)**:
+1. **Rotation extended to 55 min** (from 25) with **±90s jitter** — avoids aligning with cron/monitor cycles
+2. **Rate-limit-aware retry** — if connection fails with 429, skip the retry loop entirely, wait 120s, try once
+
+**Fixes — Monitor (`zulip-monitor.sh`)**:
+3. **Smart Triage** instead of instant restart:
+   - **Rate-limit detection**: If error log shows recent 429s, wait — don't add more API load
+   - **Self-healing detection**: If `retry_count` is 1-2, extension is already retrying — don't interrupt
+   - **Rotation window awareness**: If `queue_age` is 25-35min, disconnection is likely transient rotation — wait
+   - **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
+   - **Debounce gate**: Skip all checks entirely if recently restarted
+
+### Gen 4 (2026-06-29) — Response Pipeline Fix
+
+**Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
+
+**Four-Layer Defense**:
+1. **Response Watchdog** — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue
+2. **Queue Health Ping** — Every 2min `/api/v1/events?dont_block=true` to detect silent expiry
+3. **Proactive Queue Rotation** — New queue every 25min + 600 empty poll reconnection threshold
+4. **Health Endpoint v2** — Added `response_pipeline`, `pending_count`, `queue_healthy`, `agent_busy_duration_seconds`
+
+### Gen 3 (2026-06-15) — Stuck Detection
+
+Added `stuck: bool` and `idle_seconds` to health endpoint. Monitor restarts on `stuck: true`. Added 300s restart debounce. Queue re-registers after 30min of no events.

zulip-mention-reliability.prose.md

@@ -1,27 +1,31 @@
 ---
 kind: responsibility
 name: zulip-mention-reliability
+status: retired
 description: >
-  Monitors whether Abiba Bot is correctly detecting and responding to
-  @mentions in Zulip stream topics. Tests periodically, diagnoses
-  failures, and attempts fixes. Same self-improving pattern as
-  litellm-self-heal.
+  RETIRED 2026-07-04 — Zulip extension decommissioned. Abiba now operates
+  exclusively through Telegram (@AbibaBot). This contract is kept for
+  historical reference only.
 ---
 
+> **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
+> PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
+> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
+> Agent Zero (kagentz) continue to use Zulip.
+
 ## Maintains
 
 - mention_response_rate: number — % of @mentions that get a response (target: >95%)
 - last_test_result: "pass" | "fail" | "degraded"
 - known_failure_modes: array — History of what went wrong and how it was fixed
 - queue_health: "healthy" | "expired" | "reconnecting"
-
-## Success Criteria
-
-- test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
-- stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
-- queue_active == true — PM2's event queue is registered and polling
-- no_echo_loop == true — Bot doesn't respond to its own messages
-- dm_always_works == true — DMs still respond even if streams are broken
+  
+  **Postconditions**
+  - test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
+  - stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
+  - queue_active == true — PM2's event queue is registered and polling
+  - no_echo_loop == true — Bot doesn't respond to its own messages
+  - dm_always_works == true — DMs still respond even if streams are broken
 
 ## Known Failure Modes (from History)
 

zulip-oidc-redirect-fix.prose.md

@@ -0,0 +1,92 @@
+---
+kind: responsibility
+name: zulip-oidc-redirect-fix
+status: active
+description: >
+  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
+  uses the internal IP (192.168.68.19) instead of the public domain
+  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
+  Survives container restarts through compose.override.yaml.
+agent: abiba
+triggers:
+  - Zulip OIDC login returns "Redirect URI Error" from Authentik
+  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
+  - After Zulip server restart, Authentik SSO login broken
+---
+
+## Maintains
+
+- zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
+- authentik-acceptance: { status: "accepted" | "rejected" }
+- patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
+
+## Detection
+
+### Rule 1: Wrong redirect_uri host
+- **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
+- **Status**: CRITICAL — Authentik will reject
+- **Trigger fix** → Execute self-heal
+
+### Rule 2: OIDC flow broken
+- **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
+- **Diagnose**: Check redirect_uri in the 302 Location header chain
+
+## Fix (Self-Heal)
+
+Two layers applied:
+
+### Layer 1: Live patch (inside container, immediate)
+```bash
+# Add to /home/zulip/deployments/current/zproject/computed_settings.py:
+SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
+
+import urllib.parse
+from social_core.strategy import BaseStrategy
+_original_absolute_uri = BaseStrategy.absolute_uri
+def _patched_absolute_uri(self, path=None):
+    from django.conf import settings
+    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
+    if path is not None:
+        return urllib.parse.urljoin(root, path)
+    return root
+BaseStrategy.absolute_uri = _patched_absolute_uri
+
+# Restart Django
+supervisorctl restart zulip-django
+```
+
+### Layer 2: Persistent fix (compose.override.yaml)
+The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
+`/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
+
+### Verification
+```bash
+STEP1=$(curl -sk -w "%{redirect_url}" \
+  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
+curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
+# Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
+# Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
+```
+
+### Rollback
+Remove the patch block from `compose.override.yaml` and restart the container:
+```bash
+docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
+```
+
+## Root Cause
+
+After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
+Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
+Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
+`Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
+Django falls back to the server's IP.
+
+The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
+`ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
+Host header.
+
+## Related Contracts
+
+- `zulip-health.prose.md` — General Zulip health monitoring
+- `zulip-self-heal.prose.md` — RETIRED (pi extension removed)

zulip-self-heal.prose.md

@@ -0,0 +1,92 @@
+---
+kind: responsibility
+name: zulip-self-heal
+status: retired
+description: >
+  RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
+  Self-healing for Zulip infrastructure continues through Hermes agents.
+  Abiba no longer monitors or manages Zulip.
+agent: abiba
+triggers:
+  - none (retired)
+---
+
+> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
+> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
+> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
+
+## Maintains
+
+- zulip-server: { status: "healthy" | "degraded" | "down" }
+- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
+- zulip-config: { syntax_valid: bool, proxy_configured: bool }
+- last_action: { timestamp, action, result }
+
+## Detection Rules
+
+### Rule 1: Zulip Server Down (502/000)
+- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
+- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
+- **Fix**: `docker restart zulip-zulip-1` if container crashed
+- **Verify**: Re-test server_settings, wait for "healthy"
+
+### Rule 2: Zulip Config Corruption (400/500 non-proxy)
+- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
+- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
+- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
+- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
+
+### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
+- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
+- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
+- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
+- **Verify**: Re-test via NetBird URL
+
+### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
+- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
+- **Fix (pi)**: `pm2 restart abiba-zulip`
+- **Fix (Hermes)**: `hermes gateway restart` on agent host
+- **Verify**: Check health endpoint for `queue_healthy: true`
+
+### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
+- **Detect**: No heartbeat in agent log for >600s
+- **Fix**: Restart gateway on affected host
+- **Verify**: Check for new heartbeat in logs within 60s
+
+### Rule 6: NetBird Tunnel Down
+- **Detect**: Server returns 502 with NetBird HTML in response
+- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
+- **Fix**: Alert user — NetBird outage requires manual intervention
+- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
+
+## Hosts & Credentials
+
+| Host | IP | User | Service |
+|------|----|------|---------|
+| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
+| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
+| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
+| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
+
+## Debounce
+
+All restart actions must debounce: minimum 300s between restarts per agent.
+Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
+
+## Reporting
+
+Every cycle produces a knowledge graph node:
+- Title: `[LEARN] zulip-self-heal: <timestamp>`
+- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
+- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
+- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
+
+## Configuration
+
+Zulip server config at 192.168.68.19:
+- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
+- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
+- Fixed values:
+  - `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
+  - `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
+  - `LOAD_BALANCER_IPS = ["192.168.68.10"]`