fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)

Merge pull request 'fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT)' (#33 ) from fix/gpu-dense-smartcode-fable5 into master
fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT)
2026-07-27 20:25:51 +00:00 · 2026-07-27 16:43:46 +00:00 · 2026-07-27 16:43:29 +00:00 · 2026-07-26 12:37:35 +00:00 · 2026-07-26 12:37:07 +00:00 · 2026-07-26 12:36:31 +00:00
52 changed files with 11060 additions and 790 deletions
@@ -0,0 +1,106 @@
 name: PR Pipeline — Authorize → Validate → Review → Merge
 on:
  push:
    branches: [master]
    paths:
      - '**.prose.md'
      - 'scripts/**.sh'
      - '**.yaml'
      - '**.yml'
  pull_request:
    types: [opened, synchronize, reopened]
    paths:
      - '**.prose.md'
      - 'scripts/**.sh'
      - '**.yaml'
      - '**.yml'
 jobs:
  auth:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        run: |
          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
      - name: Authorization check
        run: bash scripts/prose-auth-check.sh
  validate:
    runs-on: ubuntu-latest
    needs: auth
    steps:
      - name: Checkout repository
        run: |
          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
      - name: YAML frontmatter validation
        run: |
          echo "=== Prose Contract Frontmatter Validation ==="
          FAILED=0
          for f in $(find . -name "*.prose.md" -not -path "./.git/*" -not -path "./runs/*"); do
            FM=$(sed -n '/^---$/,/^---$/p' "$f" | sed '1d;$d')
            [ -z "$FM" ] && { echo "  ❌ $f: No YAML frontmatter"; FAILED=$((FAILED+1)); continue; }
            KIND=$(echo "$FM" | grep '^kind:' | awk '{print $2}')
            case "$KIND" in
              function|responsibility|gateway|pattern|test|template|architecture|enforcement) echo "  ✅ $f: kind=$KIND" ;;
              *) echo "  ❌ $f: Invalid kind='$KIND'"; FAILED=$((FAILED+1)) ;;
            esac
            echo "$FM" | grep -q '^name:' || { echo "  ❌ $f: Missing name"; FAILED=$((FAILED+1)); }
            echo "$FM" | grep -q '^description:' || { echo "  ❌ $f: Missing description"; FAILED=$((FAILED+1)); }
          done
          [ $FAILED -gt 0 ] && { echo "❌ FRONTMATTER FAILED ($FAILED error(s))"; exit 1; }
          echo "✅ Frontmatter validation passed"
  lint:
    runs-on: ubuntu-latest
    needs: validate
    steps:
      - name: Checkout repository
        run: |
          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
      - name: Structure + regression + consistency lint
        run: bash scripts/prose-lint.sh
  ai-review:
    runs-on: ubuntu-latest
    needs: validate
    steps:
      - name: Checkout repository
        run: |
          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"
      - name: AI-powered contract review
        env:
          LITELLM_URL: ${{ secrets.LITELLM_URL }}
          LITELLM_KEY: ${{ secrets.LITELLM_KEY }}
        run: bash scripts/prose-ai-review.sh
  gate:
    runs-on: ubuntu-latest
    needs: [auth, validate, lint, ai-review]
    if: success()
    steps:
      - name: Merge gate
        run: |
          echo "╔══════════════════════════════════════╗"
          echo "║  ALL CHECKS PASSED — SAFE TO MERGE  ║"
          echo "╚══════════════════════════════════════╝"
          echo ""
          echo "  ✅ auth       — authorized agent"
          echo "  ✅ validate   — frontmatter valid"
          echo "  ✅ lint       — structure + no regressions"
          echo "  ✅ ai-review  — no contradictions with ground truth"
          echo ""
          echo "Merge this PR to deploy to main."
@@ -0,0 +1,137 @@
 # AGENTS.md — Prose Contracts Repo
 All agents operating on prose contracts MUST follow this workflow. No exceptions.
 ## The Rule
 **No agent pushes directly to `main`. All changes go through PRs with automated validation.**
 ## Why
 Two incidents taught us this:
 1. **The .117→.19 IP fix** — A stale IP in a contract was "fixed" without verification, breaking Zulip. The fix was right but the approach was wrong. Automated consistency checks would have caught it.
 2. **The /grafana/ route regression** — An nginx route that was deliberately removed (Jul 2) was re-added to a contract diagram. CI linting now catches this automatically.
 ## Agent Workflow
 ```
 ┌─────────────────────────────────────────────────────────┐
 │  1. Clone repo → create branch → make changes           │
 │  2. Push branch → open PR                                │
 │  3. CI pipeline runs automatically:                      │
 │     ✅ validate: frontmatter (kind, name, description)   │
 │     ✅ lint: structure + regression detection            │
 │     ✅ ai-review: LiteLLM reviews diff vs ground truth   │
 │  4. All green → merge PR to main                         │
 │  5. Agents run contracts from main                       │
 └─────────────────────────────────────────────────────────┘
 ```
 ## Branch Protection (enforced by Gitea)
 | Rule | Enforcement |
 |------|------------|
 | Direct push to `main` | ❌ Blocked (except abiba-bot for emergencies) |
 | PR merge without passing CI | ❌ Blocked — all 3 checks must be green |
 | Status check contexts | `pr-pipeline / validate`, `pr-pipeline / lint`, `pr-pipeline / ai-review` |
 ## What the CI checks for
 ### Stage 1 — Validate
 - YAML frontmatter is valid (proper `---` delimiters)
 - `kind` field is one of: `function`, `responsibility`, `gateway`, `pattern`, `test`, `template`
 - `name` and `description` fields are present
 ### Stage 2 — Lint
 - Responsibility contracts have `## Maintains`
 - Function contracts have `## Parameters` and `## Returns`
 - **Regression rules (automatic rejection):**
  - `/grafana/` nginx route — was reverted Jul 2, must not reappear
  - `CT 122` or `CT 123` as CT ID labels — don't exist in the cluster
  - These rules are hardcoded in `scripts/prose-lint.sh`
 ### Stage 3 — AI Review
 - Diff is sent to `syslog-auto` model via LiteLLM
 - Review checks against infrastructure-control ground truth:
  - CT IDs match PVE cluster (100-117, no 122/123)
  - Grafana is direct LAN :3001, NOT behind nginx
  - Zulip is CT 117 on storepve (bridge IP .19)
  - Strix Halo :8080 is firewalled to .116 only
 - Review is posted to the PR
 ## Authorization
 ### Who can change what
 | Contract | Sensitivity | Who can change |
 |----------|------------|----------------|
 | `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba only (after live verification) |
 | `proxmox-monitor.prose.md` | **CRITICAL** — deployed monitoring | Abiba only |
 | `hermes-config-template.prose.md` | **HIGH** — all agent configs | Abiba, Mumuni, Tanko |
 | `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni |
 | Other contracts | Normal | Any registered agent |
 | `scripts/*.sh` | **HIGH** — runtime scripts | Abiba only |
 ### Enforcement (self-policing)
 The CI does not block based on author identity today (Gitea doesn't support CODEOWNERS natively). Instead, agents self-enforce:
 1. Before changing a CRITICAL contract, check with Abiba
 2. If you're unsure, tag `@abiba-bot` in the PR description
 3. Abiba reviews CRITICAL contract changes before merge
 ## Emergency Bypass
 In an emergency (service down, fix must ship immediately):
 1. Push to a branch
 2. Open PR with `[EMERGENCY]` in the title
 3. CI still runs — but if it fails and the fix is verified, abiba-bot can bypass and push directly to main
 4. Post-incident: open a follow-up PR to fix any CI violations
 ## Quick Start
 ```bash
 # Clone
 git clone https://git.sysloggh.net/SyslogSolution/prose-contracts.git
 cd prose-contracts
 # Create branch
 git checkout -b fix/my-change
 # Make changes, test locally
 bash scripts/prose-lint.sh     # run lint locally before pushing
 # Push and open PR
 git add -A
 git commit -m "fix: description of change"
 git push origin fix/my-change
 # Open PR at https://git.sysloggh.net/SyslogSolution/prose-contracts/pulls
 # CI runs automatically — wait for all checks to pass
 # Merge when green
 ```
 ## Verification Before Acting
 **Contracts are leads, not facts.** Live-state fields (IPs, ports, credentials) drift.
 Before acting on any value from a contract, verify it against the live system.
 Follow the `verify-before-mutate` protocol:
 ```bash
 safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
 ```
 ## Writing New Contracts
 Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
 It covers the full process: verify → draft → lint → review → ship, with templates
 and style rules.
 ## Maintaining this file
 Keep this file for knowledge useful to almost every future agent session in this project.
 Do not repeat what the codebase already shows; point to the authoritative file or command instead.
 Prefer rewriting or pruning existing entries over appending new ones.
 When updating this file, preserve this bar for all agents and keep entries concise.
@@ -0,0 +1 @@
 AGENTS.md
@@ -1,6 +1,70 @@
 # Prose Contracts — Syslog Solution LLC
-Operating contracts for Hermes agents. Each `.prose.md` file defines a **responsibility** (recurring duty) or **template** (scaffold) that agents can execute via OpenProse or follow manually.
+Operating contracts for Syslog agents. Each `.prose.md` file defines a **function** (callable helper), **responsibility** (recurring duty with state), or **pattern** (instantiable knowledge/template) that agents can execute via OpenProse or follow manually.
 ## ⚠️ Operating Doctrine — Read Before Acting
 Two rules govern how agents interact with this repo and the infrastructure it
 describes. Internalize both before running or relying on any contract.
 ### 1. Verify-before-mutate (the `.117` rule)
 **Contracts are leads to investigate, NOT facts to act on.** Live-state fields
 (IPs, ports, hostnames, credentials, container names, PIDs) drift. Before any
 mutating operation on infrastructure, verify the current state against the live
 system. On drift, halt and ask the user which value is correct — do NOT "fix"
 the live state to match a stale contract.
 Origin: a contract stated Zulip CT 117 was at `.117`; the live IP was `.19`.
 An agent ran `pct set` without checking `pct config` first and changed the IP
 to the wrong value, breaking Zulip. The staleness was harmless until acted on.
 **Layer 3 enforcement — the `safe-mutate` wrapper** (deployed on all 5 agents:
 Abiba, Tanko, Mumuni, Koby, Koonimo). ALL infrastructure mutations MUST go
 through `safe-mutate`. Raw `sed -i`, `pct set`, `docker compose up
 --force-recreate`, `kill`, `rm` on infrastructure outside `safe-mutate` is an
 auditable protocol violation. The wrapper runs a verify command, optionally
 checks an `--expect` pattern, refuses on mismatch, and logs every call to
 `/root/.safe-mutate/audit.log`.
 ```
 safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
 safe-mutate --verify "CMD" --dry-run --mutate "CMD"      # verify + show, no execute
 safe-mutate --verify "CMD" --verify-only                   # record a verification only
 ```
 Load the `verify-before-mutate` skill (local pi skill + RA-H shared) for the
 full protocol. See knowledge graph node #604 for the post-mortem.
 ### 2. Field trust levels
 Not all contract content carries the same trust:
 | Field type | Trust level | Examples |
 |-----------|-------------|----------|
 | **Policy** | Authoritative — trust the contract | "hardcoded harness keys forbidden", "use api_key_env" |
 | **Live state** | **Never trust — always verify** | IPs, ports, hostnames, credentials, container names, PIDs |
 | **Procedure** | Trust but adapt | command patterns are right, values may be stale |
 Contracts label live-state fields with `VERIFY-BEFORE-USE`. Treat any such
 field as a hint to confirm against the live system, not a fact to apply.
 ### 3. Verify-before-fix (contract-native)
 Never `prose run` a fix-contract directly. Run its verifier first, read the
 verdict, lint the fixer, then run:
 ```
 prose run <verifier>           # e.g. zulip-platform-verification → verdict
 prose lint <fixer>             # validate structure without executing
 prose preflight <fixer>        # check deps/env, no execute
 prose run <fixer>              # only after verdict + lint pass
 ```
 `kind: responsibility` contracts VERIFY and return a verdict without making
 changes. `kind: function` contracts DO things. Forme `### Requires` →
 `### Maintains` wiring enforces this at the DAG level: a fixer that requires a
 fresh verdict cannot fire until the verifier has run.
 ## How Agents Run These Contracts
@@ -40,49 +104,95 @@ curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/mast
 ## Available Contracts
-### Responsibilities (Recurring Duties)
+### Responsibilities (Recurring Duties — `kind: responsibility`)
-| Contract | What It Does | When To Run |
+Run on trigger or schedule. Maintain persistent world-model state across runs.
 | Contract | Domain | Description |
 |---|---|---|
-| `memory-audit-maintenance` | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs, frees up char budget. | Memory >85% usage or user request: "memory audit" |
+| `memory-audit-maintenance` | Memory | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs. |
-| `zulip-health` | Checks Zulip connectivity, message flow, and bot responsiveness. | On Zulip issues or periodic health check |
+| `build-zulip-plugin` | Zulip | Generates and iteratively improves a Hermes Zulip platform plugin. Each run produces a new version. |
-| `litellm-health` | Verifies LiteLLM proxy connectivity, model availability, and key rotation. | On inference failures or periodic check |
+| `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
-| `litellm-self-heal` | Attempts to fix common LiteLLM issues (key rotation, restart, config reload). | When litellm-health reports failures |
+| `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
-| `zulip-mention-reliability` | Diagnoses and fixes @mention detection issues in Zulip. | On missed @mention reports |
+| `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `pm2-self-heal` | Restarts crashed PM2 processes and verifies recovery. | On PM2 process failure |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
 | `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
-### Templates (Scaffolds)
+### Instantiable Templates (`kind: pattern` — run with `prose run`)
-| Contract | What It Does | Parameters |
+Generate configs or perform a transformation on demand. No persistent state.
 | Contract | Description | Key Parameters |
 |---|---|---|
-| `hermes-config-template` | Generates a standard Hermes config for any agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) while keeping model choices flexible. | `agent_name` (required), `default_model`, `default_provider`, `fallback_model`, `auxiliary_model` |
+| `hermes-config-template` | Standard Hermes config for any Syslog agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) + standardized `api_key_env` key indirection. | `agent_name` (required), `default_model`, `auxiliary_model` |
 | `hermes-key-enforcement` | **Enforcement contract** — all harness/LiteLLM providers MUST use `api_key_env`, never hardcoded keys. Includes detection query, rotation procedure, violation response. External providers (DeepSeek, OpenAI) exempt. | (none — doctrine reference) |
-### Scripts
+### Reference Documents (`kind: pattern` — read for context, not run)
-| File | What It Does |
+Encode topology, architectural decisions, and lessons. You read them before planning; you don't `prose run` them.
 | Contract | Description |
 |---|---|
-| `pm2-self-heal.sh` | Shell script to restart crashed PM2 processes (companion to the prose contract) |
+| `infrastructure-control` | Full topology and control pattern: 5-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
 | `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
 | `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
 ### Functions (`kind: function` — callable helpers, no state)
 Called on-demand as single-render tools.
 | Contract | Description |
 |---|---|
 | `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
 | `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
 | `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
 ### Scripts (`scripts/`)
 Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
 | `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
 | `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 ## Contract Structure
-Every `.prose.md` file has the same structure:
+Kinds determine the section set.
 **`kind: responsibility`** (recurring duty with persistent state):
 ```
 ## Maintains      — World-model state this contract tracks
 ## Parameters     — Tunable inputs (with defaults)
 ## Requires       — External dependencies and preconditions
 ## Continuity     — When to run (triggers & cadence)
 ## Remembers      — What to learn between runs
 ## Invariants     — Inviolable rules for execution
 ## Execution      — Step-by-step instructions
 ## Example Output — What a good run looks like
 ```
 ---
 kind: <responsibility | template>
 name: <unique-name>
 description: ...
 ---
-## Maintains        — What state the contract tracks
+**`kind: function`** (stateless callable helper):
 ## Parameters       — Tunable inputs (with defaults)
 ## Continuity       — When to run (triggers & cadence)
 ## Success Criteria — How to know it worked
 ## Remembers        — What to learn between runs
 ## <Type> Rules     — Inviolable rules for execution
 ## Execution        — Step-by-step instructions
 ## Example Output   — What a good run looks like
 ```
 ## Parameters    — Tunable inputs (with defaults)
 ## Returns       — Output schema
 ## Requires      — External dependencies
 ## Execution     — Step-by-step instructions
 ```
 **`kind: pattern`** (instantiable knowledge):
 ```
 Variable — templates use Parameters + Execution; reference docs are prose-driven.
 ```
 **`kind: gateway`** / **`kind: test`** — not currently used in this repo.
 ## Adding New Contracts
@@ -0,0 +1,309 @@
 ---
 kind: function
 name: abiba-zulip-restore
 description: >
  Restores Zulip connectivity for Abiba (pi agent). Verifies the v2 router-worker
  extension code, starts a PM2 process as the Zulip gateway with correct env vars,
  validates health endpoint, and confirms DM delivery. Run this whenever Abiba
  stops responding on Zulip or after system restart.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Abiba Zulip Restore — Resume pi Zulip Communication
 Single-shot function that restores full Zulip connectivity for the Abiba pi agent.
 Covers extension code validation, PM2 process management, health endpoint
 verification, and DM loopback testing.
 ## Live-State Fields
 | Field | Value | Trust |
 |-------|-------|-------|
 | Agent name | abiba | ✅ |
 | Bot email | abiba-bot@chat.sysloggh.net | ✅ |
 | Zulip server | https://chat.sysloggh.net | ✅ |
 | Extension path | /root/.pi/agent/extensions/zulip/index.js | ✅ |
 | Config path | /root/.pi/agent/extensions/zulip/config.yaml | ✅ |
 | Health port | 9200 | ✅ |
 | @all-bots user ID | 20 | ✅ (config, verified by API at runtime) |
 | PM2 process name | abiba-zulip | ✅ |
 | Provider | syslog-harness (http://192.168.68.116/v1) | ✅ |
 | Default model | deepseek-v4-pro | ✅ (settings.json) |
 ## Architecture
 The pi Zulip extension uses a **router-worker architecture** (v2):
 - **Router** (PM2 `abiba-zulip`, `ZULIP_ROLE=router`): Polls Zulip for events,
  maintains a per-sender pool of pi RPC worker processes. Each sender gets
  their own `pi --mode rpc --session-dir` process with persistent sessions.
  Handles streaming edits back to Zulip.
 - **Worker** (child `pi --mode rpc`): Runs the agent with per-sender persistent
  sessions. No Zulip logic in the worker — the router handles all Zulip I/O.
 The extension loads in ALL pi sessions (via `settings.json` extensions array)
 but is a **NO-OP** unless `ZULIP_ROLE=router` is set. Only the PM2 router process
 carries the env var.
 ## Maintains
 - extension_valid: bool — Extension code imports without errors
 - pm2_running: bool — PM2 process `abiba-zulip` is online
 - health_responding: bool — GET :9200/health returns "ok"
 - zulip_connected: bool — Queue registered, bot identity resolved
 - loopback_delivered: bool — Test DM sent and received
 - model_valid: bool — Configured models match LiteLLM authorized models
 ### Postconditions
 - PM2 process `abiba-zulip` online and stable (uptime > 30s)
 - Health endpoint returns `{ status: "ok", connected: true }`
 - Worker pool creates sessions on demand
 - Echo prevention active (BOT_EMAILS includes all known bots)
 - PM2 saved for auto-restart on boot
 ## Requires
 - Node.js with `yaml` module available
 - PM2 installed globally
 - Zulip API key in `config.yaml`
 - `pi` CLI available on PATH
 - Zulip server accessible at https://chat.sysloggh.net
 - LiteLLM provider accessible at http://192.168.68.116/v1
 ## Execution
 ### Step 1: Validate Extension Code
 ```bash
 node -e "import('file:///root/.pi/agent/extensions/zulip/index.js').then(() => console.log('OK')).catch(e => {console.error('FAIL:', e.message); process.exit(1)})"
 ```
 Expected: `OK`. If FAIL → check for missing dependencies, syntax errors.
 ### Step 2: Validate Model IDs
 Compare configured models against LiteLLM authorized models:
 ```bash
 API_KEY=$(grep -oP 'apiKey:\s*\K.*' /root/.pi/agent/models.json | head -1)
 curl -s -H "Authorization: Bearer $API_KEY" http://192.168.68.116/v1/models | \
  python3 -c "import json,sys; d=json.load(sys.stdin); [print(m['id']) for m in d.get('data',[])]" 2>/dev/null
 ```
 Check: Every model ID in `models.json` must appear in the authorized list.
 If not → fix `models.json` to only include authorized models (prefer `syslog-auto`).
 ### Step 3: Verify Config Integrity
 ```bash
 python3 -c "
 import yaml, sys
 with open('/root/.pi/agent/extensions/zulip/config.yaml') as f:
    cfg = yaml.safe_load(f)
 required = ['zulip.site', 'zulip.email', 'zulip.api_key', 'agent.name']
 for k in required:
    keys = k.split('.')
    v = cfg
    for kk in keys:
        v = v.get(kk)
        if v is None:
            print(f'MISSING: {k}')
            sys.exit(1)
 print('Config valid')
 print(f'  site={cfg[\"zulip\"][\"site\"]}')
 print(f'  email={cfg[\"zulip\"][\"email\"]}')
 print(f'  agent={cfg[\"agent\"][\"name\"]}')
 print(f'  health_port={cfg.get(\"health_port\", 9200)}')
 "
 ```
 Expected: Config valid with all fields non-empty.
 ### Step 4: Remove Stale Systemd Service
 The old `abiba-zulip.service` points to `/opt/abiba-zulip/dist/index.js` (compiled
 TypeScript, not the v2 extension). It's disabled and stale. Remove it:
 ```bash
 systemctl stop abiba-zulip 2>/dev/null || true
 systemctl disable abiba-zulip 2>/dev/null || true
 rm -f /etc/systemd/system/abiba-zulip.service
 systemctl daemon-reload
 ```
 ### Step 5: Start PM2 Process
 **Critical:** The extension MUST run via `pi --mode rpc`, NOT `node index.js` directly.
 The extension exports a function that requires pi's session lifecycle. Running
 `node index.js` loads the module but never calls the export, so nothing happens.
 `pi --mode rpc` loads all extensions (including Zulip) and fires `session_start`.
 ```bash
 # Stop existing if any
 pm2 delete abiba-zulip 2>/dev/null || true
 # Start pi in RPC mode with ZULIP_ROLE=router env
 ZULIP_ROLE=router pm2 start "$(which pi)" \
  --name abiba-zulip \
  --interpreter none \
  -- --mode rpc --no-session
 ```
 Wait 10 seconds for pi to load all extensions, fire session_start, and the Zulip
 router to register its event queue.
 ### Step 6: Validate PM2 Process
 ```bash
 pm2 show abiba-zulip --no-color
 ```
 Check: `status=online`, `restarts=0`, `uptime > 5s`.
 ### Step 7: Check Logs for Connection
 ```bash
 sleep 3
 tail -20 /root/.pm2/logs/abiba-zulip-out.log
 ```
 Look for:
 - `[zulip-ext] Connecting to https://chat.sysloggh.net as abiba-bot@chat.sysloggh.net…`
 - `[zulip-ext] Bot user_id=N, all-bots user_id=N`
 - `[zulip-ext] Connected, queue=N`
 - `[zulip-ext] Echo prevention: N bot emails`
 - `[zulip-ext] Health endpoint on :9200`
 If error → check API key, network to chat.sysloggh.net.
 ### Step 8: Health Endpoint
 ```bash
 curl -s http://localhost:9200/health | python3 -m json.tool
 ```
 Check:
 - `status: "ok"` (not "down")
 - `zulip.connected: true`
 - `zulip.queue_id` is non-null string
 - `zulip.bot_user_id` is positive integer
 ### Step 9: DM Loopback Test
 ```bash
 curl -s http://localhost:9200/health | python3 -c "
 import json,sys
 d = json.load(sys.stdin)
 if d.get('zulip',{}).get('connected'):
    print(f'✅ Zulip connected. Queue: {d[\"zulip\"][\"queue_id\"]}')
    print(f'   Bot user_id: {d[\"zulip\"][\"bot_user_id\"]}')
    print(f'   Messages processed: {d[\"zulip\"][\"messages_processed\"]}')
 else:
    print('❌ Zulip NOT connected')
    sys.exit(1)
 "
 ```
 ### Step 10: Save PM2 for Auto-Start
 ```bash
 pm2 save
 pm2 startup systemd -u root --hp /root 2>/dev/null || true
 ```
 ### Step 11: Report
 Compile results:
 | Check | Pass? |
 |-------|-------|
 | Extension code imports | extension_valid |
 | Model IDs authorized | model_valid |
 | Config integrity | config_valid |
 | PM2 process online | pm2_running |
 | Health endpoint | health_responding |
 | Zulip connected | zulip_connected |
 All pass → ✅ **Abiba Zulip restored.** Relay success to user.
 Partial failure → see recovery matrix below.
 ## Recovery Matrix
 | Failure | Recovery |
 |---------|----------|
 | Extension import fails | Check `node_modules/zulip-js` exists; run `npm install` in extension dir |
 | Model ID mismatch | Fix `models.json` to use `syslog-auto` as default model; remove invalid IDs |
 | Config missing | Restore from backup or recreate from scratch |
 | PM2 won't start | Check `node` version (>=18); check port 9200 not in use |
 | Health "down" | Check logs for connection errors; verify Zulip API key; check network |
 | "address already in use" | Kill old process: `fuser -k 9200/tcp` |
 | Queue registration fails | Check Zulip API key validity; verify bot is active in Zulip admin |
 | Rate limit (429) | Extension has built-in retry-after handling — wait, don't restart |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Extension loads but no events | `ZULIP_ROLE` not set | Ensure PM2 env has `ZULIP_ROLE=router` |
 | Worker stays "busy" forever | Model ID not authorized by LiteLLM | Fix models.json (lesson #11) |
 | Placeholder sent but no response | editMessage API fails silently | Extension has fallback (sends new msg); check Zulip API |
 | Queue expires rapidly | Poll interval too aggressive | v2 uses 3s poll with long-poll — should be fine |
 | Bot doesn't respond to @mentions | Not subscribed to stream | Bot auto-subscribes via API |
 | Stale error in health | `last_error` not cleared | v2 clears on successful poll (lesson #4) |
 ## Appendix: Root Cause & Fix Summary (2026-07-13)
 **Problem:** Zulip extension was offline — no PM2 process running.
 **Root cause:** The PM2 command ran `node index.js` directly (which loads the
 extension module but never calls the exported function). The extension requires
 pi's session lifecycle — pi loads extensions, fires `session_start`, and the
 Zulip extension hooks into that event.
 **Fix:** Run `pi --mode rpc` (not `node index.js`). The `--mode rpc` flag keeps
 pi alive listening for RPC commands on stdin while the Zulip extension's router
 runs in the background via the `session_start` hook.
 ```bash
 ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \
  --interpreter none -- --mode rpc --no-session
 ```
 **Additional fixes applied:**
 - Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM)
  with `syslog-auto` + `strix-moe` (prevents silent worker failure per
  Lesson #11)
 - Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code)
 - PM2 saved for auto-restart on boot
 ## Appendix: PM2 Ecosystem Config (Optional)
 If preferred over manual `pm2 start`, create `/root/ecosystem.config.js` entry:
 ```js
 module.exports = {
  apps: [{
    name: 'abiba-zulip',
    script: '/bin/pi',
    interpreter: 'none',
    args: '--mode rpc --no-session',
    cwd: '/root',
    env: {
      ZULIP_ROLE: 'router',
    },
    log_file: '/root/.pm2/logs/abiba-zulip-out.log',
    error_file: '/root/.pm2/logs/abiba-zulip-error.log',
    max_restarts: 20,
    restart_delay: 5000,
  }]
 };
 ```
 ---
 **Last verified good state**: 2026-07-13 — Extension v2 running via `pi --mode rpc`, health endpoint :9200 returning `{status:"ok",connected:true}`, queue a669f21e.
@@ -14,6 +14,40 @@ description: >
 - generation_count: number — How many generations have been produced
 - last_generation: timestamp — When the last generation was created
 - known_issues: array — Issues discovered in the current version
  **Postconditions** (the plugin is GOOD when ALL pass):
  *Connectivity*
  - connected == true — Establishes and maintains Zulip event queue
  - reconnect_on_failure == true — Reconnects after queue expiry
  - recovers_from_network_loss == true — Handles temporary network drops
  - stream_subscription_check == true — Bot is subscribed to primary streams (Gen 5+)
  *Message Flow*
  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
  - dm_response_time_offline_verifiable == true — simulate_dm_latency() passes without live Zulip (Gen 6+)
  - stream_mention_detection == true — @mentions in streams are detected and routed
  - all_bots_detection == true — @all-bots mentions are detected
  - echo_loop_prevention == true — Never replies to its own messages
  *Code Quality*
  - syntax_check == "pass" — All Python files are valid
  - has_register_function == true — Exposes `register(ctx)` entry point
  - plugin_yaml_valid == true — plugin.yaml parses correctly
  - follows_base_adapter_pattern == true — Extends BasePlatformAdapter
  *Reliability*
  - graceful_disconnect == true — shutdown doesn't crash
  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
  - typing_indicators_work == true — Sends typing notifications
  - known_issues_tracked == true — add_known_issue() persists problems across generations (Gen 6+)
 ## Status
 **Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
 for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
 Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
 plugin system, which is unaffected.
 ## Parameters
@@ -26,45 +60,19 @@ description: >
 The improvement cycle is **event-driven**, not just cron-based:
- **On check failure**: If any Success Criteria fails, wake immediately to fix
+- **On check failure**: If any postcondition fails, wake immediately to fix
 - **On user request**: `prose run build-zulip-plugin` — explicit upgrade
 - **On new pattern**: If another plugin or agent reports a better pattern, wake to assimilate
 - **Retrospective**: Every 24 hours if no other trigger fired — scan for subtle degradation
 - **On first deploy**: Always run Generation 1 when no plugin exists yet
 ## Success Criteria (What Makes a Plugin "Good")
 The plugin is considered GOOD when ALL of these pass:
 ### Connectivity
 - plugin.connected == true — Establishes and maintains Zulip event queue
 - plugin.reconnect_on_failure == true — Reconnects after queue expiry
 - plugin.recovers_from_network_loss == true — Handles temporary network drops
 ### Message Flow
 - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
 - stream_mention_detection == true — @mentions in streams are detected and routed
 - all_bots_detection == true — @all-bots mentions are detected
 - echo_loop_prevention == true — Never replies to its own messages
 ### Code Quality
 - syntax_check == "pass" — All Python files are valid
 - has_register_function == true — Exposes `register(ctx)` entry point
 - plugin_yaml_valid == true — plugin.yaml parses correctly
 - follows_base_adapter_pattern == true — Extends BasePlatformAdapter
 ### Reliability
 - graceful_disconnect == true — shutdown doesn't crash
 - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
 - typing_indicators_work == true — Sends typing notifications
 ## Remembers
 Each generation stores:
 - What worked well (success patterns)
 - What broke (failure modes)
 - What the user complained about (pain points)
- Which Success Criteria passed and failed
+- Which postconditions passed and failed
 ## Generation Rules
@@ -92,13 +100,28 @@ Each generation stores:
 ## Execution
-1. **Read state** — Check previous generations from knowledge graph
+1. **Read state** — Check previous generations from knowledge graph node `build-zulip-plugin:state`
 2. **Generate or improve** — Based on generation count:
   - Generation 1: Scaffold full plugin from scratch
   - Generation N: Read previous code, apply improvements
 3. **Validate** — Syntax check, structure check, dependency check
-4. **Log** — Save generation result to knowledge graph as [LEARN]
+4. **Run offline tests** — Always run `simulate_dm_latency()` and selftest() if no live Zulip
-5. **Report** — Output what was generated, what changed, what needs review
+5. **Log** — Save generation result to knowledge graph as [LEARN] node. Update the
   `build-zulip-plugin:state` node with current generation_count, plugin_version,
   and known_issues.
 6. **Report** — Output what was generated, what changed, what needs review
 ## State Persistence (Gen 6+)
 All generation state is stored in a knowledge graph node titled `build-zulip-plugin:state`:
 - `generation_count` — incremented on each verified run
 - `plugin_version` — current version string
 - `last_generation` — ISO timestamp of most recent run
 - `known_issues` — array of unresolved issues carried forward
 - `last_postcondition_results` — pass/fail map from most recent verification
 Agents executing this contract MUST read this node on startup and update it after
 validation. If the node does not exist, create it with generation_count=1.
 ## Example Output
@@ -0,0 +1,612 @@
 # Cron Prompts Review — All 10 Scheduled Contracts
 Generated: 2026-07-13 20:59:18 ET
 ---
 ## hermes-key-enforcement
 **Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 6 * * *
 ```
 Contract Enforcement: hermes-key-enforcement
 Category: compliance
 Domain: hermes-agent
 Owner: abiba
 Schedule: Daily compliance scan at 6am ET
 This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
 Steps:
 1. Load contract from prose-contracts/main (file: hermes-key-enforcement.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/hermes-key-enforcement/
 Postconditions to verify:
 [
  {
    "check": "no plaintext API keys in config",
    "verify": "grep -rc 'api_key: sk-' /root/.hermes/config.yaml",
    "expect": "0 matches"
  },
  {
    "check": "api_key_env used for harness/litellm providers",
    "verify": "grep -c 'api_key_env.*LITELLM_API_KEY' /root/.hermes/config.yaml",
    "expect": "count > 0"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert + pause
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/hermes-key-enforcement/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-key-enforcement, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## hermes-config-template
 **Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 4 * * 1
 ```
 Contract Enforcement: hermes-config-template
 Category: compliance
 Domain: hermes-agent
 Owner: abiba
 Schedule: Weekly config drift check Monday at 4am ET
 This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
 Steps:
 1. Load contract from prose-contracts/main (file: hermes-config-template.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/hermes-config-template/
 Postconditions to verify:
 [
  {
    "check": "agent config template_version matches template file",
    "verify": "grep -q 'template_version' /root/.hermes/config.yaml && diff <(grep 'template_version' /root/.hermes/config.yaml | cut -d: -f2 | xargs) <(grep 'template_version' /root/prose-contracts/hermes-config-template.prose.md | cut -d: -f2 | xargs) && echo match || echo mismatch",
    "expect": "match"
  },
  {
    "check": "config file is valid YAML",
    "verify": "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
    "expect": "valid"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/hermes-config-template/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-config-template, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## hermes-agent-baseline
 **Category:** compliance | **Domain:** hermes-agent | **Owner:** abiba | **Schedule:** 0 5 * * 1
 ```
 Contract Enforcement: hermes-agent-baseline
 Category: compliance
 Domain: hermes-agent
 Owner: abiba
 Schedule: Weekly baseline verification Monday at 5am ET
 This is a compliance contract. Verify that the contract enforces the required standards and policies. Report any violations found.
 Steps:
 1. Load contract from prose-contracts/main (file: hermes-agent-baseline.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/hermes-agent-baseline/
 Postconditions to verify:
 [
  {
    "check": "Hermes agent process running",
    "verify": "pgrep -f 'hermes' > /dev/null && echo running || echo stopped",
    "expect": "running"
  },
  {
    "check": "agent config file exists and valid YAML",
    "verify": "test -f /root/.hermes/config.yaml && python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' && echo valid || echo invalid",
    "expect": "valid"
  },
  {
    "check": "no uncommitted changes in hermes directory",
    "verify": "cd /root/.hermes && git status --porcelain | wc -l",
    "expect": "0"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/hermes-agent-baseline/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=hermes-agent-baseline, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## proxmox-monitor
 **Category:** monitoring | **Domain:** proxmox | **Owner:** abiba | **Schedule:** */15 * * * *
 ```
 Contract Enforcement: proxmox-monitor
 Category: monitoring
 Domain: proxmox
 Owner: abiba
 Schedule: Every 15 minutes
 This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
 Steps:
 1. Load contract from prose-contracts/main (file: proxmox-monitor.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/proxmox-monitor/
 Postconditions to verify:
 [
  {
    "check": "all Proxmox nodes reachable",
    "verify": "curl -sf http://192.168.68.10:8006/api2/json/status | jq '.status'",
    "expect": "healthy"
  },
  {
    "check": "no VMs in crashed state",
    "verify": "pvesh get /nodes -output-format=json | jq '.[] | select(.status==\"Crashed\")'",
    "expect": "empty"
  },
  {
    "check": "backups running on schedule",
    "verify": "pbs-info --check",
    "expect": "last_backup < 24h ago"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba, mumuni → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/proxmox-monitor/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=proxmox-monitor, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## gpu-monitor
 **Category:** monitoring | **Domain:** gpu | **Owner:** abiba | **Schedule:** */15 * * * *
 ```
 Contract Enforcement: gpu-monitor
 Category: monitoring
 Domain: gpu
 Owner: abiba
 Schedule: Every 15 minutes — polls all GPU subsystems
 This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
 Steps:
 1. Load contract from prose-contracts/main (file: gpu-monitor.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/gpu-monitor/
 Postconditions to verify:
 [
  {
    "check": "GPU metrics accessible",
    "verify": "curl -sf http://localhost:9100/gpu-data",
    "expect": "200 OK, populated data"
  },
  {
    "check": "dashboard serving",
    "verify": "curl -sf http://localhost:9100/gpu-fleet.html",
    "expect": "200 OK, HTML returned"
  },
  {
    "check": "health endpoint responsive",
    "verify": "curl -sf http://localhost:9100/health",
    "expect": "200 OK"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba, mumuni → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/gpu-monitor/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=gpu-monitor, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## infrastructure-monitoring
 **Category:** monitoring | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** */30 * * * *
 ```
 Contract Enforcement: infrastructure-monitoring
 Category: monitoring
 Domain: infrastructure
 Owner: abiba
 Schedule: Every 30 minutes
 This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
 Steps:
 1. Load contract from prose-contracts/main (file: infrastructure-monitoring.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/infrastructure-monitoring/
 Postconditions to verify:
 [
  {
    "check": "Proxmox API reachable",
    "verify": "curl -sf http://192.168.68.10:8006/api2/json",
    "expect": "200 OK"
  },
  {
    "check": "Zulip API reachable",
    "verify": "curl -sf https://chat.sysloggh.net/api/v1/me",
    "expect": "200 OK"
  },
  {
    "check": "LiteLLM proxy reachable",
    "verify": "curl -sf http://192.168.68.116/litellm/v1/models",
    "expect": "200 OK"
  },
  {
    "check": "Gitea API reachable",
    "verify": "curl -sf https://git.sysloggh.net/api/v1/version",
    "expect": "200 OK"
  },
  {
    "check": "SearXNG reachable",
    "verify": "curl -sf http://192.168.68.17:8080",
    "expect": "200 OK"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba, mumuni → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/infrastructure-monitoring/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-monitoring, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## zulip-health
 **Category:** monitoring | **Domain:** zulip | **Owner:** abiba | **Schedule:** */15 * * * *
 ```
 Contract Enforcement: zulip-health
 Category: monitoring
 Domain: zulip
 Owner: abiba
 Schedule: Every 15 minutes — monitors all Zulip-connected agents
 This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
 Steps:
 1. Load contract from prose-contracts/main (file: zulip-health.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/zulip-health/
 Postconditions to verify:
 [
  {
    "check": "bot registration active",
    "verify": "curl -sf https://chat.sysloggh.net/api/v1/me | jq '.user_id'",
    "expect": "bot_id present"
  },
  {
    "check": "DM delivery working",
    "verify": "curl -sf https://chat.sysloggh.net/api/v1/users/me/is-online",
    "expect": "online: true"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba, mumuni → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/zulip-health/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=zulip-health, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## litellm-health
 **Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * *
 ```
 Contract Enforcement: litellm-health
 Category: monitoring
 Domain: litellm
 Owner: abiba
 Schedule: Every 10 minutes — LiteLLM proxy health
 This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state.
 Steps:
 1. Load contract from prose-contracts/main (file: litellm-health.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/litellm-health/
 Postconditions to verify:
 [
  {
    "check": "LiteLLM proxy reachable",
    "verify": "curl -sf http://192.168.68.116/litellm/v1/models",
    "expect": "200 OK, models returned"
  },
  {
    "check": "router deprecated, nginx routes work",
    "verify": "curl -sf https://litellm.sysloggh.net/v1/models",
    "expect": "200 OK (via nginx)"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba, mumuni → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert + trigger_remediation
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + pause + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/litellm-health/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=litellm-health, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## memory-audit-maintenance
 **Category:** maintenance | **Domain:** memory | **Owner:** mumuni | **Schedule:** 0 3 * * *
 ```
 Contract Enforcement: memory-audit-maintenance
 Category: maintenance
 Domain: memory
 Owner: mumuni
 Schedule: Daily at 3am ET
 This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
 Steps:
 1. Load contract from prose-contracts/main (file: memory-audit-maintenance.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/memory-audit-maintenance/
 Postconditions to verify:
 [
  {
    "check": "memory files below 80% capacity",
    "verify": "wc -l ~/.hermes/memories/*.md",
    "expect": "total lines < threshold"
  },
  {
    "check": "no stale entries",
    "verify": "grep -r 'STALE' ~/.hermes/memories/",
    "expect": "0 matches"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify mumuni → action: relay_alert
 - CRITICAL: notify mumuni, abiba → action: relay_alert
 - FATAL: notify mumuni, abiba, kwame → action: relay_alert + human_required
 Circuit Breaker:
 - Max retries: 3 per 60.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/memory-audit-maintenance/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=memory-audit-maintenance, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 ## infrastructure-update
 **Category:** maintenance | **Domain:** infrastructure | **Owner:** abiba | **Schedule:** 0 2 * * 0
 ```
 Contract Enforcement: infrastructure-update
 Category: maintenance
 Domain: infrastructure
 Owner: abiba
 Schedule: Weekly system updates Sunday at 2am ET
 This is a maintenance contract. Execute the maintenance tasks defined in the contract. Report any issues found.
 Steps:
 1. Load contract from prose-contracts/main (file: infrastructure-update.prose.md)
 2. Verify prerequisites (connectivity, tools, deps)
 3. Execute contract per SOP
 4. Run postconditions from contract registry
 5. Generate receipt with status (pass/fail/escalated)
 6. If any postcondition fails, escalate per contract escalation tiers
 7. Log to ~/.hermes/runs/infrastructure-update/
 Postconditions to verify:
 [
  {
    "check": "all services running after update",
    "verify": "systemctl list-units --state=running",
    "expect": "all critical services"
  }
 ]
 Escalation Tiers:
 - INFO: notify nobody → action: log_to_receipt
 - WARNING: notify abiba → action: relay_alert
 - CRITICAL: notify abiba, mumuni → action: relay_alert
 - FATAL: notify abiba, mumuni, kwame → action: relay_alert + human_required
 Circuit Breaker:
 - Max retries: 1 per 120.0min window
 - On trip: escalate_to_fatal
 Receipt format: JSON with contract, run_id, timestamp, agent, status, actions_taken, postconditions, drift_alerts, evidence_path
 Receipt storage: ~/.hermes/runs/infrastructure-update/receipt-{timestamp}.json
 Graph node: Create RA-H OS node for receipt with metadata: type=receipt, contract=infrastructure-update, status=<status>
 If the contract has no postconditions defined (e.g., reference/pattern contracts), log that it was loaded and skip execution.
 IMPORTANT: If the contract file does not exist in prose-contracts/main, report failure and do NOT hallucinate forward.
 ```
 ---
 *End of review*
@@ -0,0 +1,306 @@
 ---
 kind: responsibility
 name: disk-gc-threat-response
 description: >
  Recurring disk health scan, garbage collection, and threat response
  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
  Docker image bloat — 5 dangling images, 15 build cache layers.
  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
 # Disk GC & Threat Response
 ## Goal
 Every container in the fleet stays below 80% disk usage through regular inspection
 and automated garbage collection. Critical breaches are detected, remediated, and
 logged within 5 minutes of discovery.
 ## Scope
 All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
 Docker hosts get special attention:
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
 | syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
 | docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
 | amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
 | All other CTs | — | LOW — no Docker | apt clean, log rotate |
 | GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
 > **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
 > **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 ## Threat Levels
 | Level | Threshold | Response | Escalation |
 |-------|-----------|----------|------------|
 | **GREEN** | < 75% | Log only | None |
 | **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
 | **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
 | **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
 | **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
 ## Requires
 - SSH access to all Docker hosts (matrix from infrastructure-control pattern)
 - Proxmox API access from abiba (token already provisioned)
 - Zulip bot credentials for alerting
 ## Maintains
 Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
 the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
 bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
 documented reason.
 ### disk-health
 Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
 timestamp, and active threat level.
 ### gc-history
 Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
 and whether threat was resolved.
 ### threat-log
 Active and resolved threat entries with severity, timestamps, remediation applied,
 and escalation trail.
 ## Continuity
 - self-driven: full fleet scan every 6 hours
 - May also be invoked manually: `prose run disk-gc-threat-response`
 - Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
 ## Shape
 - `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
 - `delegates`:
  - `disk-scanner`: per-CT disk check (pct exec df or SSH)
  - `gc-docker`: docker system prune execution
  - `gc-system`: apt clean, log rotate, tmp cleanup
  - `alerter`: Zulip notification dispatch
 - `prohibited`: deleting user data, removing running containers, force-killing
  production services
 ## Runtime
 - `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
 - `retry`: 2 attempts for SSH failures before marking a CT unreachable
 ## Execution
 ```prose
 let fleet = call disk-scanner
  scope: all
 let threats = []
 for ct in fleet:
  if ct.usage_pct >= 95:
    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
  else if ct.usage_pct >= 85:
    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
  else if ct.usage_pct >= 75:
    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
 -- sort by severity descending
 sort threats by pct desc
 for threat in threats:
  let result = call gc-executor
    ct: threat.ct
    level: threat.level
    strategy: lookup-gc-strategy(threat.ct)
  call alerter
    threat: threat
    result: result
 call summary-reporter
  fleet: fleet
  threats: threats
 ```
 ## GC Strategies by Host Type
 ### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
 docker system prune -f
 # Phase 2: Aggressive (if still > 85% after Phase 1)
 docker system prune -a --force
 # Phase 3: Emergency (if still > 95%)
 docker system prune -a --force --volumes
 docker builder prune --all --force
 # Verification after each phase
 df -h /
 docker system df
 ```
 ### Non-Docker CTs
 ```bash
 # Package cache
 apt-get clean
 apt-get autoremove --yes
 # Log rotation
 journalctl --vacuum-size=100M
 find /var/log -type f -name "*.log" -mtime +30 -delete
 # Temp files
 find /tmp -type f -mtime +7 -delete
 find /var/tmp -type f -mtime +30 -delete
 # Snap (if installed)
 snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
  snap remove "$snap" --revision="$rev"
 done
 ```
 ### Special Cases
 | CT | Special GC |
 |----|-----------|
 | 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
 | 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
 | 106 (ra-h-os) | Check relay DB size, enforce TTL |
 | 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |
 ## Alert Templates
 ### AMBER (75-84%)
 ```
 ⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
 Next scheduled GC will attempt cleanup. No immediate action needed.
 ```
 ### RED (85-94%)
 ```
 🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
 GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
 Status: {resolved|still elevated — {reason}}
 ```
 ### CRITICAL (≥95%)
 ```
 🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
 Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
 {service_status} — {manual_action if needed}
@Kwame — container nearly full, resolved: {yes_no}
 ```
 ## Incident Log: 2026-07-04 — kagentz docker bloat
 ### Discovery
 Kwame noticed kagentz was full, asked Abiba to investigate.
 ### Diagnosis
 - CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
 - `/var/lib` = 55G (93% of disk)
 - Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
 - 1 stopped container, 1 unused network, 15 build cache layers
 ### Resolution
 ```
 docker system prune -a --force
 → Reclaimed 35.67GB
 → Post: 16G / 59G (27%), 41G free
 → 5 dangling images removed (kagentz-bridge + old builds)
 → 1 stopped container removed
 → 1 unused network removed (kagentz-bridge_default)
 → 15 build cache layers removed
 ```
 ### Root Cause
 Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
 dangling images and orphaned build cache. No automated GC was in place.
 ### Preventive Measures
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
 - amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```
 ## Incident Log: 2026-07-09 — amdpve docker bloat
 ### Discovery
 Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
 amdpve (.15) flagged at 78% (AMBER threshold: 75%).
 ### Diagnosis
 - amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
 - Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
 - Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
 - Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
 - Not a running service — zero containers, zero active volumes
 ### Resolution
 ```
 docker system prune -a --force
 → Reclaimed 11.56GB
 → Post: 55G / 94G (62%), 35G free
 → 1 image removed (llama-strix-hip:latest, 15.5GB)
 → 8 build cache layers removed
 → amdpve now GREEN
 ```
 ### Root Cause
 Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
 artifacts. Docker on amdpve serves no running purpose — it's only used for
 one-off GPU builds. No automated post-migration cleanup was in place.
 ### Preventive Measures
 - amdpve added to Docker GC scan list
 - Post-migration cleanup step added: after any GPU backend migration, prune
  the old backend's Docker images within 24 hours
 - Contract now scans GPU bare-metal hosts alongside CTs
 - Access via `pct-run` script for all CTs (no hardcoded IPs)
 ## Access Matrix (documented 2026-07-09)
 ### CT Access (via pct-run)
 | CT | Name | Node | Status |
 |----|------|------|--------|
 | 100 | abiba | amdpve | local |
 | 102 | adguard | acerpve | ✅ reachable |
 | 104 | authentik | minipve | ✅ reachable |
 | 105 | kagentz | amdpve | ✅ reachable |
 | 106 | ra-h-os | storepve | ✅ reachable |
 | 107 | pbs | storepve | ✅ reachable |
 | 108 | media | storepve | ✅ reachable |
 | 110 | gitea | minipve | ✅ reachable |
 | 111 | tdunna | amdpve | ✅ reachable |
 | 112 | tanko | amdpve | ✅ reachable |
 | 113 | baggy | amdpve | ✅ reachable |
 | 114 | mumuni | minipve | ✅ reachable |
 | 115 | scottdenya | amdpve | ✅ reachable |
 | 116 | syslog-api | minipve | ✅ reachable |
 | 117 | zulip | storepve | ✅ reachable |
 ### GPU Bare Metal (via direct SSH)
 | Host | IP | GPU | Status |
 |------|-----|-----|--------|
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
 | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
 ### KVM VM (via direct SSH)
 | Host | IP | Role | Status |
 |------|-----|------|--------|
 | docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
 > **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
@@ -0,0 +1,308 @@
 # Prose Contract Authoring Guide
 Approach this as the lead systems engineer at a company where every contract is a
 production dependency — agents run from these every 15 minutes, every 6 hours,
 every Sunday at 3am. A stale IP or a missing section isn't a typo; it's an
 incident waiting to happen. Write every contract as if you'll be paged at 3am
 when it breaks.
 ## Ground it in the subject
 Before writing a single line of YAML, answer three questions and state them in
 the description:
 1. **What system does this contract touch?** Name the hosts, CTs, containers,
   and services explicitly. "The inference fleet" is vague. "GPU .8 (RTX 3090,
   qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, ornith), and LiteLLM on CT
   116" is specific.
 2. **Who runs this contract, and when?** State the agent, the trigger (cron,
   manual, event-driven), and the expected duration. "Runs every 6 hours via
   cron on abiba, takes ~120 seconds for a fleet scan."
 3. **What breaks if this contract is wrong?** Name the blast radius. "A stale
   CT ID causes a restart of the wrong container. A wrong Grafana URL breaks
   the monitoring dashboard for all agents."
 If you cannot answer these three questions concretely, do not write the
 contract yet — go verify against the live system first.
 ## Design principles
 ### Specificity over generality
 A contract that says "check all Docker containers" is worse than one that lists
 them by name. The list will go stale, but staleness is detectable (the CI lint
 catches it). Vagueness is invisible and uncatchable. Be specific, accept that
 specificity decays, and rely on the lint pipeline to catch the decay.
 ### Ground truth is the live system, not the contract
 Every IP, port, hostname, and container name in a contract is a **live-state
 field**. Mark them with `VERIFY-BEFORE-USE` in the access matrix. Before any
 mutating operation, verify against the live system. The contract is a
 hypothesis; the live system is the fact.
 When you discover drift between contract and reality, do not "fix" the live
 system to match the contract. Halt. Ask which is correct. Document the answer
 in the contract and the knowledge graph. The `.117→.19` incident was caused by
 fixing the wrong direction.
 ### Contracts are typed — respect the kind
 | Kind | When to use | Must have | Must NOT have |
 |------|------------|-----------|---------------|
 | `function` | Stateless, callable helper | `## Parameters`, `## Returns`, `## Execution` | `## Maintains` (no state) |
 | `responsibility` | Persistent duty with state | `## Maintains`, `## Continuity` or `## Execution` | — |
 | `gateway` | External-driven entry point | `## Maintains` | `## Requires` |
 | `pattern` | Reference/template/doctrine | Flexible — prose-driven | Don't `prose run` directly |
 | `test` | Verification only | `## Execution` | Mutating operations |
 Use `kind: architecture` sparingly — it's not a standard OpenProse kind. Most
 architecture docs should be `kind: pattern`.
 ### One contract, one concern
 A contract that monitors health AND deploys exporters AND sends alerts is three
 contracts fighting for attention. Split them:
 ```
 ❌ infrastructure-monitoring.prose.md (deploy + export + alert)
 ✅ proxmox-monitor.prose.md (deploy exporters)
 ✅ gpu-monitor.prose.md (poll + dashboard)
 ✅ infrastructure-control.prose.md (reference topology)
 ```
 The contract that deploys is a function. The contract that polls is a
 responsibility. The contract that maps the topology is a pattern. One concern
 per file, one kind per concern.
 ### Live-state fields are declared, not hidden
 Don't bury IPs and hostnames in prose paragraphs. Put them in tables, mark them
 with `VERIFY-BEFORE-USE`, and keep them in one place per contract:
 ```markdown
 | Host | IP | Role | Trust |
 |------|----|------|-------|
 | CT 116 | 192.168.68.116 | LiteLLM + Grafana | ⚠️ VERIFY-BEFORE-USE |
 ```
 The lint pipeline scans for known stale values. Tables are grep-able; prose is
 not.
 ## Process: verify, draft, lint, review, ship
 ### Pass 1: Verify
 Before writing, verify every live-state field you plan to include:
 ```bash
 # Example: writing a contract about Zulip
 curl -s http://192.168.68.19/api/v1/server_settings  # is .19 reachable?
 ssh root@192.168.68.19 "docker ps"                     # is Zulip in Docker?
 pct config 117 | grep net0                              # what IP does CT 117 have?
 ```
 Write down the verified values. Use them in the contract. If you can't verify
 something, mark it explicitly: `⚠️ UNVERIFIED — needs live check`.
 ### Pass 2: Draft
 Write the contract following the structure for its kind. Fill every required
 section. Use the templates below. Write descriptions that name the subject: "GPU
 fleet health check across RTX 3090 (.8), RTX 5070 (.110), and Strix Halo (.15)"
 not "monitoring check."
 ### Pass 3: Lint locally
 Before pushing, run the same checks the CI will run:
 ```bash
 bash scripts/prose-lint.sh
 ```
 Fix everything that fails. The lint checks for:
 - Valid YAML frontmatter
 - Required sections per kind
 - Regression patterns (/grafana/ route, CT 122/123)
 - Missing Maintains/Parameters/Returns
 ### Pass 4: Self-critique
 Read your contract and ask:
 1. **Could an agent run this at 3am without context?** If it needs tribal
   knowledge, add it to the description or a comment.
 2. **Does any value contradict the infrastructure-control pattern?** Check CT
   IDs, IPs, hostnames, nginx routes, Grafana access.
 3. **Is anything in here that was previously fixed and reverted?** Check the
   lint regression rules. If you're unsure, grep the git log for similar
   changes.
 4. **Am I saying the same thing as another contract?** If yes, link to it
   instead of duplicating. One source of truth.
 ### Pass 5: Open PR, let CI review
 Push to a branch, open a PR. The pipeline runs:
 ```
 auth → validate → lint → ai-review → gate
 ```
 The AI review compares your diff against the infrastructure-control ground
 truth. It catches contradictions you might miss. If it flags something, don't
 argue — verify and fix. The AI is reading the same ground truth you should have
 read before writing.
 ## Restraint and self-critique
 Spend your complexity budget in one place. A contract that deploys Prometheus,
 configures Grafana, provisions dashboards, and sets up exporters is a function
 that does four things — each of which can fail independently. Split them.
 Don't add a `### Requires` that you haven't verified works. A dependency on
 "SSH access to .116" is only valid if you've tested it from the agent that
 will run the contract.
 Cut sections that don't serve the runner. A 200-line appendix of curl commands
 is documentation, not execution. Put it in a wiki or a README, not in the
 contract. The contract is for the agent running it; the README is for the human
 reading about it.
 Before shipping, take one thing out. Like Chanel's advice: before leaving the
 house, remove one accessory. Is there a section, a parameter, a check that
 doesn't pull its weight? Cut it. The contract that ships with 5 sections will
 be maintained. The one with 15 will rot.
 ## Writing in contracts
 ### Descriptions
 A description is not a label. It's the first thing an agent reads before
 deciding whether to run this contract. Make it answer: what does this do, to
 what, and why should I care?
 ```
 ❌ "Monitors infrastructure health"
 ✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker
   container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
 ```
 ### Field names
 Name things by what they control, not by how the system is built:
 ```
 ❌ prometheus_scrape_interval_seconds
 ✅ check_every_seconds
 ```
 The runner doesn't care that Prometheus is doing the scraping. They care how
 often the check runs.
 ### Error messages and alerts
 Errors tell the operator what went wrong and what to do about it. Be specific:
 ```
 ❌ "Health check failed"
 ✅ "GPU .15 (Strix Halo) unreachable at :8080 — firewalled to .116 only.
   Check router /health/unified at http://192.168.68.116/health/unified instead."
 ```
 ### Comments
 Comments in contracts explain WHY, not WHAT. The execution steps say what to
 do. Comments say why we do it this way and not another:
 ```markdown
 ## Execution
 1. **Check GPU health via router** — cannot probe .15:8080 directly
   (firewalled to .116 only as of 2026-07-01 Vulkan rebuild)
 2. **Skip sidecar check** — JSON exporters at :8090 were never deployed
   on any GPU host; router falls back to GPU /health direct probe
 ```
 ## Templates
 ### Function contract template
 ```markdown
 ---
 kind: function
 name: contract-name
 description: >
  What this does, to what, and why. Be specific about hosts/services.
 version: 1.0.0
 ---
 ## Parameters
 - param_one: type — description (default: "value")
 - param_two: type — description (default: "value")
 ## Returns
 - result_field: type — description
 ## Requires
 - Dependency one — how to verify it's available
 - Dependency two
 ## Execution
 1. **Step one** — what and why
 2. **Step two** — what and why
 3. **Compile and return** — assemble the output
 ```
 ### Responsibility contract template
 ```markdown
 ---
 kind: responsibility
 name: contract-name
 description: >
  What this recurring duty monitors/maintains. Name the systems, the
  cadence, the blast radius.
 version: 1.0.0
 ---
 ## Maintains
 - state_field: { current: value, last_check: timestamp }
 - another_field: description of maintained state
 ## Parameters
 - param: type — description (default: "value")
 ## Requires
 - Dependency one
 - Dependency two
 ## Continuity
 - Self-driven: check every N seconds/minutes/hours
 - Also wakes on: trigger description
 ## Execution
 1. **Check** — verify current state
 2. **Evaluate** — compare against thresholds
 3. **Remediate or escalate** — fix or alert
 4. **Record** — update world-model state
 ```
 ## Enforcement
 This guide is aspirational, not gated. The CI pipeline enforces structure
 (frontmatter, required sections) and regression (known stale patterns). It
 does not enforce prose quality or specificity. That's on you, the author.
 Before you ship: read your contract as if you're the agent who will run it at
 3am with 30 seconds of context. If you wouldn't trust it, rewrite it.
@@ -0,0 +1,335 @@
 ---
 kind: responsibility
 name: gpu-fleet
 description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
  UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe,
  gpu-dense, gpu-light. These never change — only the underlying model does.
  Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
  RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
  UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability.
  Strix Halo model swapped to qwen3.6-35B-udq4 (22GB, strix-moe alias).
  Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling.
  For larger context needs → fall back to external providers (deepseek).
  VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
  UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled
  (UD-Q3_K_XL, ~14.7GB — Q4 was too large for 24GB VRAM with 128K KV cache). ~50% fewer thinking
  tokens via ThinkingCap finetune + Fable 5 CoT distillation for improved coding reasoning.
  VRAM ~22.4/24.6GB (91%).
 agent: abiba
 triggers:
  - on model add/remove
  - on GPU health degradation
  - on agent key rotation
  - on router restart (roster must be loaded)
 ---
 ## Maintains
 - gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
 - router: { status: "healthy", roster_loaded: bool, models: array }
 - litellm: { status: "healthy", keys: array, models: array }
 - agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
 - health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
 - monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
 - watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
 - benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
 - grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
 - port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding
 ## Fleet Topology (Current — July 2026)
 ```
 ┌──────────────────────────────────────────────────────────────────┐
 │  CT 116 (192.168.68.116) — Inference Harness Host                │
 │                                                                  │
 │  nginx:80 (entrypoint)                                           │
 │  ├─ /v1/*       → harness-litellm:4000   (API requests)         │
 │  ├─ /admin/*    → harness-litellm:4000   (admin endpoints)       │
 │  ├─ /dashboard/ → harness-dashboard:3000 (harness UI)            │
 │  ├─ /litellm/*  → harness-litellm:4000   (LiteLLM UI + API)     │
 │  ├─ /health/*   → harness-litellm:4000   (health probes)         │
 │  └─ /gpu/*      → 192.168.68.24:9100     (fleet monitor)         │
 │                                                                  │
 │  Containers:                                                     │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
 │  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
 │  │ :4000    │  │ :9000    │  │ :3000    │  │ :3000     │        │
 │  │ keys+sync│  │deprecated│  │ harness  │  │ Prometheus│        │
 │  │ fallback │  │not in    │  │ UI       │  │ data src  │        │
 │  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
 │                    │                                             │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
 │  │PostgreSQL│  │  Redis   │  │Prometheus│                       │
 │  │ :5432    │  │ :6379    │  │ :9090    │                       │
 │  └──────────┘  └──────────┘  └──────────┘                       │
 └────────────────────┼─────────────────────────────────────────────┘
                     │
     ┌───────────────┼───────────────┬──────────────────┐
     │               │               │                  │
 ┌────▼─────┐  ┌──────▼──────┐  ┌────▼──────┐  ┌───────▼──────┐
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
 │ 128K ctx │  │ 128K ctx    │  │ 128K ctx  │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ qwen3.6   │  │ Prometheus   │
 │ 27B-code │  │ :8080       │  │ -35B-udq4 │  │ exporter     │
 │ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
 └──────────┘
 ```
 ## Stable Role-Based Aliases (Introduced 2026-07-15)
 Agent configs, cron jobs, and workflows MUST use these aliases, never model-specific names.
 When a model is swapped on a GPU, ONLY the infrastructure layer changes — agent configs are untouched.
 | Alias | GPU | Current Model | Will Route To |
 |-------|-----|---------------|---------------|
 | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
 | `gpu-dense` | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | Whatever runs on RTX 3090 |
 | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
 **Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
 but are deprecated for agent configs. Only the stable aliases survive model swaps.
 ## Current Model Assignments (2026-07-15)
 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
 | SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 | .8 (llm-gpu) | ~22.4/24.6GB (91%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ healthy |
 | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
 | qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
 ## Routing Configuration (LiteLLM — July 2026)
 ### syslog-auto Weighted Pool (Direct GPU — bypasses router)
 | Model | GPU | Weight | RPM Cap | Timeout |
 |-------|-----|--------|---------|---------|
 | SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
 | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
 | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
 Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
 ### Direct Model Endpoints
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
 | strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
 | SmartCode-Fable-5-27B-UD-Q3_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
 | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
 ### Stable Aliases (for agent configs — never change)
 | Alias | RPM Cap | Routes To | Purpose |
 |-------|---------|-----------|---------|
 | `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
 | `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
 | `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
 ### Fallback Chains
 - gemma → qwen
 - qwen → gemma
 - strix-moe → qwen → gemma
 - syslog-auto → qwen → gemma → qwen3.6-35B-udq4
 ### Why Strix Halo RPM Is Capped
 - Direct (strix-moe): 40 RPM (tight) — Strix Halo is shared with compression tasks
 - Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
 - Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
 ## Operations
 ### add-model
 1. Download model file from Hugging Face or source
 2. Check disk space + GPU VRAM compatibility
 3. Start llama-server via systemd service on GPU host
 4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
 5. Hot-reload or restart router
 6. Add model to LiteLLM config (`model_list` + `fallbacks`)
 7. Generate agent keys for new model access via `/key/generate`
 8. Add Prometheus scrape target for the new GPU exporter
 9. Verify end-to-end: LiteLLM → Router → Model
 ### remove-model
 1. Drain active requests (wait for active=0)
 2. Remove from LiteLLM config
 3. Remove from router config
 4. Stop llama-server (systemd)
 5. Remove Prometheus scrape target
 6. Cleanup model files (optional)
 ### heal
 1. Check all GPUs via router internal `:9000/health/unified`
 2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
 3. Reset stuck circuit breakers if idle (Redis)
 4. Restart dead llama-server instances via SSH
 5. Flush Redis active counters if stale
 6. Verify GPU monitor server is running on pi (:9100)
 7. Verify watchdog is running on pi
 8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
 9. Reload roster via `POST :9000/admin/roster/reload` if available
 ### sync-keys
 1. List all agent keys in LiteLLM DB via `GET /key/list`
 2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
 3. Generate missing keys via `POST /key/generate` with unlimited budget
 4. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production`
 5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
 6. Verify each key with test request through full chain
 7. Document keys in knowledge graph
 ### list
 Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys
 ### health-check
 1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
 2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
   - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
 7. Check port conflicts: verify only one llama-server on :8080 per host
 8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
 ## Agent Keys (LiteLLM DB — Current 2026-07-11)
 Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY).
 Agent gateways inject keys at runtime via `infisical run --` wrapper.
 Plaintext keys removed from this contract post-vault-migration.
 | Agent | CT | IP | LiteLLM Alias | Key Source | Access |
 |-------|-----|-----|---------------|------------|--------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
 | Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
 | Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
 | Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
 | Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
 | Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | no SSH |
 > **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.
 **Key update procedure**: Update Infisical vault → `infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production` → restart agent gateway. Agent picks up new key via `infisical run --` wrapper at startup.
 If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 ## Configuration Files
 | File | Host | Purpose |
 |------|------|---------|
 | `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
 | `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
 | `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
 | `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
 | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
 | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/strix-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo) running unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
 ## Prometheus & Grafana
 | Component | URL | Details |
 |-----------|-----|---------|
 | Grafana | `http://192.168.68.116:3001/` | admin / vault (`GRAFANA_ADMIN_PASSWORD`) |
 | GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
 | Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
 | GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
 | Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |
 ## Known Issues & Watch Points
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
 - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
 - **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q3_K_XL (14.7GB). Q4 was too large for 24GB VRAM with 128K context + KV cache overhead. Q3 fits at ~22.4GB (91%). Uses standard llama.cpp build b9190 (turboquant b9150 incompatible with qwen3_5 arch). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. Service: `/home/llmuser/llama-fable-wrapper.sh`.
 - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
 - **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
 - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
 - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
 - **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
 - **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
 - **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
 - **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.
 ## GPU Inference Benchmarks (Current)
 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
 | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q3_K_XL | **TBD** | — | — | **128K** |
 | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
 | Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |
 Benchmarks from 2026-07-17. Strix Halo model: qwen3.6-35B-udq4. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
 All 3 GPUs now at 128K context (2026-07-17, reduced from 256K for stability).
 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
 Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
 History stored at `/root/data/toks-history.json` with 7-day rolling window.
 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
 ## Agent Config Implications (2026-07-15)
 ### Stable Aliases — CRITICAL
 All agent configs MUST use stable role-based aliases, never model-specific names:
 - `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`)
 - `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
 - `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
 - `auxiliary.web_extract.model: gpu-light`
 When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched.
 ### Context Windows
 - RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
 - **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
 - Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
 - Mumuni compression model alias: `strix-moe` with 300s timeout
 ### Mumuni Agent Profile
 Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:
 | Setting | Value | Notes |
 |---------|-------|-------|
 | `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
 | `model.provider` | `custom:litellm` | LiteLLM on CT116 |
 | `compression.model` | `strix-moe` | Stable alias — survives model swaps |
 | `aux.compression.model` | `strix-moe` | Compression auxiliary model |
 | `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
 | `aux.web_extract.model` | `gpu-light` | Web extraction |
 | `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
 | `context.max_context_window` | 131072 (128K) | Reduced from 256K 2026-07-17 — stable 128K ceiling |
 | `compression.threshold` | 0.65 | Triggers at ~85K |
 | `compression.target_ratio` | 0.3 | Compresses to ~38K |
 | `compression.protect_last_n` | 40 | Preserves last 40 messages |
 | `memory.memory_char_limit` | 800 | Brief memory entries |
 | `personalities` | `creative` | Creative assistant personality |
 | Platforms | cli, homeassistant, signal, telegram, zulip | All Hermes platforms |
 | Main model timeout | 300s | LiteLLM global timeout |
 | Compression model timeout | 300s | strix-moe timeout increased from 120s |
 ### Agent Update Status (2026-07-15)
 | Agent | Host | Status |
 |-------|------|--------|
 | **Mumuni** | CT114 (.123) | ✅ Updated to stable aliases |
 | **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
 | **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
 | **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
 | **Kagenz0** | CT105 | ❌ SSH unreachable — needs Zulip DM |
 All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap.
@@ -0,0 +1,136 @@
 ---
 kind: responsibility
 name: gpu-monitor
 description: >
  Comprehensive GPU fleet monitor — polls every subsystem (sidecars, router,
  LiteLLM, Strix Halo, dashboard) every 15s, renders a live HTML dashboard,
  checks alert thresholds, and exposes a JSON API for downstream consumers.
 agent: abiba
 ---
 ## Maintains
 - gpu-fleet-data: { gpus: array, strix: object, router: object, litellm: object, summary: object, alerts: array } — Full fleet snapshot refreshed every 15s
 - dashboard: live HTML at `/root/dashboard/gpu-fleet.html`, served on port 9100
 - gpu-data-api: JSON at `GET /gpu-data` on port 9100
 - health-endpoint: `GET /health` on port 9100 → 200 if cache populated, 503 if warming up
 ## GPU Fleet Topology
 ```
 ┌─────────────────────────────────────────────────────────┐
 │  GPU Monitor Server (:9100) — 192.168.68.24              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
   │          │          │
   ▼          ▼          ▼
 ┌──────┐ ┌──────┐ ┌────────┐
 │.8:8080│ │.110  │ │.116:80 │
 │RTX3090│ │:8080 │ │nginx   │
 │gemma  │ │RTX5070│ │router  │
 └──────┘ │qwen27B│ │LiteLLM │
         └──────┘ │dashboard│
                  └────────┘
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
 Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
 poll .15:8080 directly; must go through router on .116.
 ```
 ### Subsystems Polled
 | Subsystem | Endpoint | Frequency | Metrics |
 |-----------|----------|-----------|---------|
 | GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
 | Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
 | Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
 | LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
 | Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | Strix Halo status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
 | Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
 ### Alert Delivery
 All alerts are sent to `#agent-hub` stream topics:
 - `alerts-gpu` — GPU fleet alerts (temp, VRAM, utilization)
 - `alerts-pm2` — PM2 process health (restarts, crashes)
 - `alerts-infra` — Infrastructure health (LiteLLM, Zulip, containers)
 This replaces the previous DM-only delivery. All agents on the mesh can see and respond to alerts.
 ## Alert Thresholds
 | Metric | Warning | Critical |
 |--------|---------|----------|
 | GPU Temp | >80°C | >90°C |
 | VRAM Usage | >90% | >95% |
 | GPU Util | >95% | >98% |
 | Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
 | Model Down | — | critical (circuit breaker open) |
 ### JSON API Response Schema (/gpu-data)
 ```json
 {
  "gpus": [{ "name", "gpu_name", "temp_c", "vram_used_mb", "vram_total_mb",
             "gpu_util_pct", "power_w", "power_limit_w", "fan_pct",
             "models", "hostname", "active_requests", "health_score",
             "circuit_open" }],
  "strix": { "status", "cpu_load", "llama_health" },
  "router": { "available_models", "circuit_breaker", "gpus", "scores",
              "status", "redis", "_basic" },
  "litellm": { ... },
  "dashboard": { "reachable": bool },
  "summary": { "fleet_status", "models_available", "models_total",
               "circuit_breakers_open", "gpu_count", "gpu_errors",
               "strix_running", "router_reachable", "litellm_reachable" },
  "alerts": [{ "gpu", "metric", "level", "value", "threshold" }],
  "updated": "ISO8601",
  "monitor_version": "2.0.0"
 }
 ```
 ## Operations
 ### list
 `curl http://localhost:9100/gpu-data | jq` — Full fleet status
 ### check-health
 `curl http://localhost:9100/health` — Monitor self-check
 ### view-dashboard
 Open `http://localhost:9100/` in browser — Live HTML dashboard
 ### restart
 ```bash
 pkill -f gpu-monitor-server.py
 python3 /root/scripts/gpu-monitor-server.py &
 ```
 Or via PM2: `pm2 restart gpu-monitor`
 ### check-router
 The router health is accessed through nginx on port 80 (NOT port 9000 directly).
 `curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy
 `curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
 ## Configuration Files
 | File | Host | Purpose |
 |------|------|---------|
 | `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
 | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
 ## Execution
 1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
 2. **Poll router** (every 15s): GET .116/health via nginx:80
 3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
 4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
 5. **Poll dashboard** (every 15s): GET .116/dashboard/
 6. **Check alerts**: Compare metrics against thresholds
 7. **Compute summary**: Fleet-wide health aggregation
 8. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
 9. **Serve API**: HTTP server on port 9100
 10. **Repeat** every 15 seconds
@@ -0,0 +1,311 @@
 ---
 kind: responsibility
 name: gpu-self-heal
 description: >
  GPU fleet self-healing — detects anomalies, applies remediation, tracks
  benchmarks, and predicts failures before they happen. Extends gpu-monitor
  (v2.1.0) with active remediation rules, Prometheus metrics consumption,
  VRAM trend analysis, and predictive alerting.
  UPDATED 2026-07-18: Model assignments synced to 2026-07-17 swaps.
  Router (port 9000) references replaced with direct GPU routing.
  Benchmark baselines refreshed to live values.
  Prometheus exporters removed — not deployed; fall back to direct sidecar probes.
  Stable role-based aliases (strix-moe, gpu-dense, gpu-light) from gpu-fleet.
 agent: abiba
 depends_on:
  - gpu-monitor.prose.md (live data source on .24:9100)
  - gpu-fleet.prose.md (source of truth for topology, aliases, model assignments)
 ---
 ## Maintains
 - gpu-health: { status: "healthy"|"degraded"|"down", issues: array, actions: array }
 - gpu-self-heal-log: array of { timestamp, gpu, issue, action, result } — audit trail
 - benchmark-regression: { gpu, baseline_tok_sec, current_tok_sec, trend, alerts }
 - vram-trend: { gpu, current_mb, rate_mb_per_hour, projected_full_in_hours }
 - circuit-breaker-status: { gpu, open, auto_reset_attempted, last_reset }
 ## Requires
 - gpu-monitor:function — Live fleet data from localhost:9100/gpu-data
 - Direct sidecar probe access to all GPU hosts (:8080/health)
 - SSH access to GPU hosts for restart operations
 ## Continuity
 - Self-driven: check every 60 seconds against GPU monitor data
 - Also wakes on gpu-fleet health degradation
 - On fix: verify with benchmark inference test before declaring resolved
 - Escalate: after 3 failed remediation attempts → Zulip #agent-hub alert
 ---
 ## Current Fleet Baseline (2026-07-18)
 | Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role |
 |-------|-----|------|-------|------|-----|-------|------|
 | `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
 | `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
 | `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
 Key notes:
 - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
 - Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated.
 - RTX 5070 tok/s is 2.3x faster than RTX 3090 for its model — gpu-light is the fastest endpoint. Route vision/web/light work there first.
 - Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads.
 - RTX 3090 VRAM at 88% — within role-appropriate range (role = heavy reasoning, needs the headroom).
 - RTX 5070 VRAM at 83% — role-appropriate for vision/web (smaller batch sizes).
 ## Remediation Rules
 ### Rule 1: GPU Temperature Critical (>85°C for >2 min)
 - **Detect**: Any GPU temp >85°C sustained for 2+ consecutive polls
 - **Fix**: 
  1. Reduce inference concurrency on that GPU (load-side cooling only — NO fan control)
  2. Redirect new requests to cooler GPUs via LiteLLM fallback chains (gemma → qwen, qwen → gemma)
  3. If all GPUs hot, alert about cooling infrastructure
 - **Verify**: Temp drops below 80°C within 5 minutes
 - **Escalate after**: 3 verification failures → Zulip alert
 ### Rule 2: VRAM Leak Detection (tiered by GPU capacity)
 - **Detect**: VRAM growing at sustained rate over 6+ hour window
  - RTX 3090 (24GB): ≥300MB/hour
  - RTX 5070 (12GB): ≥300MB/hour
  - Strix Halo (64GB UMA): ≥200MB/hour
 - **Fix**: 
  1. Log VRAM snapshot with process list (nvidia-smi/rocm-smi + ps aux)
  2. If llama-server is the growth source → restart with memory cap flag
  3. If unknown process → kill and alert
 - **Verify**: VRAM growth rate drops below threshold
 - **Escalate after**: persistent leak after restart → hardware investigation
 ### Rule 3: Model Inference Timeout / GPU Stuck
 - **Detect**: >50% failure rate over 60s window + 30s grace period (not just single stuck request)
 - **Fix**: 
  1. Restart llama-server on affected GPU host
  2. Wait 15s for model to reload
  3. Run benchmark inference test
 - **Verify**: Model returns 200 with <30s response, failure rate drops to 0%
 - **Escalate after**: 3 restarts in 1 hour → GPU hardware check
 ### Rule 4: Benchmark Regression (>20% drop)
 - **Detect**: gen_tok_per_sec drops >20% below baseline over 3+ benchmarks
  - RTX 3090 baseline: 74.8 tok/s → alert at <59.8 tok/s
  - RTX 5070 baseline: 165.2 tok/s → alert at <132.2 tok/s
  - Strix Halo baseline: 70.5 tok/s → alert at <56.4 tok/s
 - **Fix**:
  1. Check GPU utilization — if >90%, other process is competing
  2. Check power limit — if throttled, restore to max
  3. Check thermal — if hot, apply Rule 1
 - **Verify**: Benchmark returns to within 10% of baseline
 - **Escalate after**: persistent regression → possible hardware degradation
 ### Rule 5: Circuit Breaker Stuck Open
 - **Detect**: Circuit breaker open >10 minutes with GPU reporting healthy
 - **Note**: Router (port 9000) is deprecated. If circuit breakers are reported by gpu-monitor, they come from LiteLLM's internal tracking, not the old router.
 - **Fix**: 
  1. Verify GPU /health returns 200 on direct port (:8080)
  2. If GPU healthy, alert but do NOT reset via router API (deprecated)
  3. Check LiteLLM health directly: http://192.168.68.116/litellm/health/liveliness
  4. Restart LiteLLM container on CT 116 if circuit breakers are stuck
 - **Verify**: LiteLLM returns healthy, circuit breaker clears within 60s
 - **Escalate after**: LiteLLM restart doesn't clear → human investigation
 ### Rule 6: Strix Halo Unreachable
 - **Detect**: Strix not responding — probe .15:8080 directly (firewall opened .24→.15)
 - **Fix**: 
  1. SSH to .15 → check llama-server process
  2. Restart llama-server if not running
  3. Verify through both direct probe AND LiteLLM health
 - **Verify**: Direct health probe returns 200, LiteLLM reports model healthy
 - **Escalate**: If host .15 itself is unreachable → infrastructure alert
 ### Rule 7: GPU Data Source Unreachable (replaces old Prometheus rule)
 - **Detect**: gpu-monitor endpoint (localhost:9100/gpu-data) or sidecar port (:8080) on any GPU unreachable for >2 polls
 - **Fix**:
  1. If gpu-monitor is down: restart systemd service `gpu-monitor.service` on this host
  2. If sidecar is down: SSH to GPU host → check llama-server process → restart systemd service
  3. Fall back to direct nvidia-smi/rocm-smi probe via SSH if all API paths fail
 - **Verify**: gpu-monitor returns healthy + all sidecars reachable
 - **Escalate after**: 3 failed restarts → networking issue
 ### Rule 8: Predictive Thermal Warning (two-tier)
 - **Detect**: 
  - Tier 1 (warning): temp >70°C AND rising >2°C/min → reduce concurrency, no alert
  - Tier 2 (critical): temp >80°C AND still rising → full alert + aggressive load shedding
 - **Fix**: 
  - Tier 1: silently reduce parallel requests to that GPU by 50%
  - Tier 2: redirect all new requests away, alert #agent-hub, apply Rule 1 logic
 - **Verify**: Temp rise rate drops below 1°C/min (Tier 1) or temp drops below 80°C (Tier 2)
 - **Escalate**: If Tier 2 triggers and temp still rising after 5 min → possible hardware failure
 ### Rule 9: Context Window Optimization
 - **Detect**: Benchmark tok/s vs baseline for each GPU at current context (all 128K)
  - RTX 3090 (128K ctx, ThinkingCap): baseline 74.8 tok/s — currently at 74.9 (100%)
  - RTX 5070 (128K ctx, HauhauCS QAT): baseline 165.2 tok/s — currently at 169.6 (103%)
  - Strix Halo (128K ctx, qwen3.6-35B-udq4): baseline 70.5 tok/s — currently at 62.9 (89%)
 - **Fix**:
  - If tok/s > baseline → context has headroom, consider increasing
  - If tok/s < 90% baseline → reduce context by 25% and retest
  - If tok/s within 10% of baseline → optimal, no change
  - Strix Halo at 89% of baseline → MONITOR but do not reduce yet (recent model swap may still be settling)
 - **Verify**: Re-benchmark after context change, confirm within 10% of target
 - **Escalate**: If context can't be adjusted without significant perf loss
 ### Rule 10: Workload Distribution Optimization (updated 2026-07-18)
 - **Detect**: GPU roles misaligned with hardware capabilities
 - **Target distribution**:
  - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
  - RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
  - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
 - **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
 - **Fix**:
  - Alert if any GPU is handling workload outside its designated role
  - Recommend agent alias updates to match workload to GPU role (use stable aliases: gpu-dense, gpu-light, strix-moe)
  - Track per-GPU request distribution via LiteLLM spend logs
 - **Verify**: Each GPU's request pattern matches its designated role within 24h
 - **Escalate**: If role mismatch persists >48h → agent alias audit needed
 ---
 ## Execution
 ```prose
 -- Phase 1: Fetch live GPU data
 let fleet = call gpu-monitor
  endpoint: "http://localhost:9100/gpu-data"
 -- Phase 2: Evaluate each GPU against remediation rules
 let actions = []
 for gpu in fleet.gpus:
  -- Rule 1: Thermal critical
  if gpu.temp_c > 85 and sustained_for(gpu, 120):
    push actions apply-thermal-fix(gpu)
  -- Rule 2: VRAM leak
  let vram_rate = calculate-vram-trend(gpu, hours=6)
  if vram_rate > 50:
    push actions apply-vram-fix(gpu, vram_rate)
  -- Rule 4: Benchmark regression
  let bench = fleet.benchmarks[gpu.hostname]
  if bench.current_tok_s < bench.baseline_tok_s * 0.8:
    push actions apply-benchmark-fix(gpu, bench)
 -- Rule 3: Model stuck
 for model in fleet.summary.available_models:
  if model.consecutive_timeouts >= 3:
    push actions apply-model-restart(model)
 -- Rule 5: Circuit breaker check via LiteLLM (router deprecated)
 if fleet.summary.circuit_breakers_open > 0:
  push actions check-litellm-circuit-breakers()
 -- Rule 6: Strix Halo
 if not fleet.strix.running and pingable("192.168.68.15"):
  push actions apply-strix-restart()
 -- Rule 7: GPU data source
 if not fleet.gpus or len(fleet.gpus) < 2:
  push actions check-gpu-monitor-service()
 -- Rule 8: Predictive thermal
 for gpu in fleet.gpus:
  let rise_rate = calculate-temp-rise(gpu, minutes=5)
  if rise_rate > 2.0 and gpu.temp_c < 80:
    push actions apply-proactive-cooling(gpu)
 -- Phase 3: Execute actions, verify, log
 for action in actions:
  let result = execute-with-verify(action)
  log-to-kg(action, result)
  if result.failed:
    escalate-if-needed(action)
 -- Phase 4: Update health state
 call update-gpu-health
  gpus: fleet.gpus
  actions: actions
  status: derive-overall-status(fleet, actions)
 -- Wait 60s and repeat
 ```
 ## Audit Trail Format
 ```json
 {
  "run_id": "gpu-self-heal-20260718-001",
  "timestamp": "2026-07-18T08:00:00Z",
  "gpu": "ct8-rtx3090",
  "issue": "thermal-critical",
  "detected": { "temp_c": 87, "duration_s": 180 },
  "action": "load-shedding",
  "result": "resolved",
  "verification": { "temp_c": 76, "after_s": 300 },
  "escalated": false
 }
 ```
 ---
 ## Reporting
 ### 1. Knowledge Graph
 Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
 ### 2. Zulip Alerts (#agent-hub → alerts-gpu)
 - `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
 - `issues_escalated > 0` → "⚠ GPU Self-Heal — <gpu> needs attention"
 - Every 100th clean cycle → "✅ GPU Fleet: All Clear"
 ### 3. Weekly Benchmark Report
 - Per-GPU tok/s trend over 7 days
 - Regression alerts if any GPU degrades >10% week-over-week
 ---
 ## Design Decisions (Verified 2026-07-12, Reaffirmed 2026-07-18)
 1. **Fan control**: ❌ NO auto fan control. Load-side cooling only (reduce concurrency, redirect).
 2. **Model restart**: ✅ Only if >50% failure rate over 60s + 30s grace period. Not on single stuck request.
 3. **Strix direct access**: ✅ Open firewall .15:8080 → .24 for direct health probe + restart.
 4. **VRAM thresholds**: Tiered — **300MB/h** (RTX 3090), **300MB/h** (RTX 5070), 200MB/h (Strix). Previous values (100/50) were too sensitive; raised 2026-07-18 based on operational data.
 5. **CB auto-reset**: ✅ Router deprecated — circuit breakers go through LiteLLM health check + container restart if needed. No per-GPU auto-reset.
 6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Current baselines live in gpu-monitor.
 7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
 8. **Prometheus**: ❌ Not deployed. Use direct sidecar probes (:8080/health) and gpu-monitor API. Prometheus integration deferred until exporters are running on GPU hosts.
 ## Lessons Learned (2026-07-12, Updated 2026-07-18)
 ### L1: API Key Standardization Is Critical
 - All GPU llama-servers MUST use the same api-key as the LiteLLM config.
 - RTX 5070 had `--api-key sk-loc...5678` while LiteLLM sent `not-needed`.
  This caused cascading 401 → fallback → timeout → 401 loops.
 - **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config (`not-needed` for direct routing).
 ### L2: Fallback Chain Cascading Failures
 - When one model returns 401 (auth) and another is slow (timeout), the fallback
  chain creates an infinite loop.
 - **Rule**: If a model returns 401 (auth error), do NOT fall back to it again.
  Mark it as permanently failed for this request.
 ### L3: Verify Running State, Not Docs
 - RTX 3090 was documented at 128K context. Running at 128K (verified 2026-07-18).
 - Parallel count: 1 on both RTX 3090 and RTX 5070 (matches docs for current models).
 - **Rule**: Before making decisions, check `/proc/PID/cmdline` on GPU hosts.
 ### L4: Infisical Is Not Always Available
 - Keep a local `.env` fallback for `LITELLM_API_KEY`.
 - **Rule**: Always verify credential source is reachable before relying on it.
 ### L5: GPU Monitor Response Size Can Cause Self-Heal Crash
 - gpu-self-heal crashed with KeyboardInterrupt during json.loads() of 20MB response.
 - Root cause: router poll returns accumulated data → cache balloons.
 - **Rule**: Self-heal must enforce a read timeout AND max response size on every poll.
  If monitor response > 1MB, log a warning and skip the cycle rather than crashing.
 ### L6: Stable Aliases Replace Model Names
 - gpu-fleet introduced stable aliases (strix-moe, gpu-dense, gpu-light) on 2026-07-15.
 - Self-heal must use aliases for reporting and alerting, not model-specific names.
 - **Rule**: All alert messages and KG nodes use the stable alias as the GPU identifier.
@@ -10,6 +10,6 @@ description: A simple hello world contract to test OpenProse on pi
 ## Returns
 - greeting: string — The generated greeting message
-## Ensures
+## Maintains
 - The greeting includes the provided name
 - The greeting is friendly and warm
@@ -0,0 +1,285 @@
 ---
 kind: template
 name: hermes-agent-baseline
 version: 1.0.0
 description: >
  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
  configuration state, keys, workarounds, and audit procedure. When an agent's
  configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 128K context (reduced from 256K for stability Jul 2026) (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
 author: Abiba (pi agent)
 ---
 # Hermes Agent Baseline — Canonical Good State
 ## Quick Restore
 ```bash
 # Verify all agents against baseline in one command:
 for ct in 112 114 111 113; do
    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
 done
 ```
 ## Agent Map
 | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
 |-------|-----|------|-----|---------------|------------|----------|
 | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
 | Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
 | Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
 | Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
 | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
 > **Note**: CT hostnames (tdunna→CT129, baggy→CT114) differ from agent identities (koby, koonimo).
 Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
 Keys are stored in Infisical vault (project=agents, env=production) and injected at
 runtime via `infisical run --` wrapper. Plaintext keys removed from this baseline.
 ## Key Architecture
 ```
 Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
                       ↓
 Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
                                     └── Key DB (Postgres)
 ```
 - **Master key**: stored in Infisical vault (project=infrastructure, secret=LITELLM_MASTER_KEY) — ADMIN ONLY
 - **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
 - **Key injection**: `infisical run --project=agents --env=production -- hermes gateway run` injects `LITELLM_API_KEY` at runtime
 - **Key source**: Infisical vault → runtime env var. /etc/environment is CLEAN (stripped, tagged `# [INFISICAL]`)
 - **Legacy override** (pre-migration): `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` — should be REMOVED
 ## Config Pattern — Mandatory Fields
 ### For Hermes Agents (Tanko, Mumuni, Koonimo)
 Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
 ### 1. Main Model
 ```yaml
 model:
  default: syslog-auto
  provider: custom:harness       # or: harness
  api_key_env: LITELLM_API_KEY
  max_tokens: 4096
 ```
 ### 2. Custom Provider
 ```yaml
 custom_providers:
  - name: harness
    model: syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
 ```
 ### 3. Vision (CRITICAL — must have api_key directly!)
 ```yaml
 auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b              # or syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30
 ```
 ### 4. Compression (must have api_key!)
 ```yaml
  compression:
    enabled: true
    threshold: 0.65
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # or gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 120
 ```
 ## Known Bug: `api_key_env` Ignored by Auxiliary Client
 **Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
 **What happens**: The function reads `api_key` from auxiliary task configs but does NOT
 resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
 to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
 to `"no-key-required"` → 401 from LiteLLM.
 **Impact**: Vision analysis, compression, and any other auxiliary task calling
 LiteLLM/harness will fail with:
 ```
 401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
 ```
 **Workaround**: Set `api_key` directly (copy the value from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=agents --env=production`) alongside `api_key_env` in every auxiliary task config that uses the harness provider.
 **Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
 `api_key` is empty:
 ```python
 cfg_api_key = str(task_config.get("api_key", "")).strip() or None
 if not cfg_api_key:
    key_env = str(task_config.get("api_key_env", "")).strip()
    if key_env:
        cfg_api_key = os.getenv(key_env, "").strip() or None
 ```
 ## Audit Procedure
 ### Full Audit (all agents)
 ```bash
 for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
    # Verify /etc/environment is CLEAN (no LITELLM_API_KEY)
    pct-run $ct "grep -c LITELLM_API_KEY /etc/environment 2>/dev/null || echo '0 (clean)'"
    # Verify gateway uses infisical run wrapper
    pct-run $ct "ps aux | grep 'infisical run' | grep -v grep"
    # Check for hardcoded harness keys
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
 done
 ```
 ### Master Key Leak Check
 ```bash
 # On every agent — must return empty:
 pct-run <CT> grep -rl "sk-litellm" /root/ /etc/ 2>/dev/null
 # Vault is the only place the master key should exist
 ```
 ### Verify Key Works
 ```bash
 # Retrieve key from vault and test:
 KEY=$(infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
 curl -s http://192.168.68.116:80/v1/models \
  -H "Authorization: Bearer $KEY" | grep syslog-auto
 # Must return model list
 ```
 ### Verify Vision/Compression
 ```bash
 # Check both api_key and api_key_env are present:
 pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```
 ### For Koby (CT 129 / tdunna)
 Koby runs Hermes on CT 129 (tdunna). Config files at `/root/.hermes/config.yaml`.
 Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
 **LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
 ### For pi Agents (Abiba)
 Abiba (CT100) runs pi via PM2 with the Zulip extension.
 Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
 **models.json** — Must only list models authorized for the agent's LiteLLM key.
 Key is injected via `infisical run --` wrapper at PM2 startup:
 ```json
 {
  "providers": {
    "syslog-harness": {
      "baseUrl": "http://192.168.68.116/v1",
      "api": "openai-completions",
      "apiKey": "${LITELLM_API_KEY}",
      "models": [
        { "id": "syslog-auto" },
        { "id": "strix-moe" },
        { "id": "gpu-dense" },
        { "id": "gpu-light" },
        { "id": "qwen3.6-27B-code" },
        { "id": "gemma-4-12b" }
      ]
    }
  }
 }
 ```
 **settings.json** — Always use `syslog-auto` as default:
 ```json
 {
  "defaultProvider": "syslog-harness",
  "defaultModel": "syslog-auto"
 }
 ```
 **Validation**: Verify models match LiteLLM's authorized list:
 ```bash
 curl -s http://192.168.68.116:4000/v1/models \
  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
  | jq '.data[].id'
 ```
 **Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
 a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
 stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.
 **Service**: `pm2 restart koby-zulip`, health at `:9201/health`.
 ## Systemd Pattern
 For agents where the gateway runs as a user service:
 ```ini
 # /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
 [Service]
 Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
 ```
 For agents where the gateway runs as root:
 ```ini
 # /root/.config/systemd/user/hermes-gateway.service
 EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
 ```
 **No drop-in that hardcodes the master key. Ever.**
 ## Attachment Cache Locations
 | Type | Path |
 |------|------|
 | Images | `~/.hermes/cache/images/` |
 | Documents | `~/.hermes/cache/documents/` |
 | Audio | `~/.hermes/cache/audio/` |
 ## Health Verification
 Run the consolidated health check:
 ```bash
 python3 /root/scripts/agent-health-check.py  # v2: now checks all 5 agents including Koby/Koonimo SSH, CT liveness, config YAML, wrapper integrity, vault non-emptiness
 ```
 This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
 verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
 and counts recent errors. Non-disruptive — never restarts anything.
 ## GPU Port Conflict Detection
 All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
 - `.8` and `.110`: inline check in `llama-wrapper.sh`
 - `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)
 Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
 If they differ → ghost detected → kill ghost → start fresh.
 ## Related Contracts
 - `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
 - `hermes-config-template.prose.md` — full configuration template
 - `gpu-fleet.prose.md` — GPU fleet and agent key table
 - `infrastructure-control.prose.md` — CT inventory with pct-run access
 - `litellm-health.prose.md` — LiteLLM stack health verification
 ## Change Log
 | Date | Change |
 |------|--------|
 | 2026-07-08 | Koby: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added config section. Key rotated and stored in vault. Added Failure Mode #11 to zulip-adapter-lessons. |
 | 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
 | 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
@@ -4,132 +4,112 @@ name: hermes-config-template
 description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
-  RA-H OS MCP) while keeping model/provider choices flexible per agent role.
+  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Every new agent profile should start from this template.
+  UPDATED 2026-07-18: Compression model switched to `syslog-auto` (was `strix-moe`)
  to relieve Strix Halo pressure. syslog-auto distributes compression across the
  weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
  UPDATED 2026-07-16: Compression model was the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
  which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
  Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
  2026-07-16 Mumuni root-cause investigation (WAL #1300).
  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (later switched to syslog-auto 2026-07-18). RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
 ---
 ## Maintains
- template_version: string — Current template version (e.g., "1.0.0")
+- template_version: "2.1.0"
- last_applied: timestamp — When any agent was last configured from this template
+- last_applied: timestamp
- agents_configured: array — Which agents/profiles were created from this template
+- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
- infra_endpoints_verified: array — Firecrawl, SearXNG, RA-H OS, LiteLLM endpoints last confirmed working
+- agent_keys: map (see Agent Keys section)
- known_deviations: array — Profiles that diverge from the template (and why)
+- infra_endpoints_verified: array
-## Parameters
+## Agent Keys (LiteLLM — Current 2026-07-11)
- agent_name: string — Name of the agent/profile (e.g., "syslog-code", "syslog-devops")
+Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM
- default_model: string — Agent's primary model (e.g., "qwen3.6-27B-code", "syslog-auto")
+PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB.
- default_provider: string — Inference provider for the primary model (default: "harness")
+The env var `LITELLM_API_KEY` is injected at runtime via `infisical run --` wrapper
- fallback_model: string — Fallback model when primary is down (default: "gemma-4-12b")
+(project=agents, env=production). /etc/environment and ~/.hermes/.env are NO LONGER
- fallback_provider: string — Fallback provider (default: "harness")
+used for agent keys — stripped and tagged `# [INFISICAL]` post-migration.
- auxiliary_model: string — Model for vision/compression/extraction tasks (default: "gemma-4-12b")
+Sub-agent profiles inherit auth from the main config — no separate keys needed.
 - enable_firecrawl_searxng: boolean — Whether to configure web search/extract (default: true)
 - enable_ra_h_os_mcp: boolean — Whether to wire up RA-H OS MCP bridge (default: true)
 - enable_compression: boolean — Whether to enable context compression (default: true)
 - custom_provider_name: string — Custom provider name (default: "harness")
 - custom_provider_model: string — Custom provider model (default: same as default_model)
 - custom_provider_url: string — LiteLLM base URL (default: "http://litellm.sysloggh.net/litellm/v1")
 - extra_auxiliary: array — Additional auxiliary tasks to configure (vision, compression, etc.)
 - output_dir: string — Where to write the config (default: "~/.hermes/profiles/<agent_name>/config.yaml")
-## Quickstart — How to Run This Contract
+| Agent | Key Alias | Host | SSH | Sub-Agents |
 |-------|-----------|------|-----|-----------|
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
 | Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
 | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
 | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
 | Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |
-### For the Agent Running This Contract
+> CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
-Read this section first. It tells you exactly what to do.
+✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
-
+  syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`
 **When to run:**
 - Setting up a **new agent profile** for the first time
 - An **existing profile** is missing web search, firecrawl, or MCP configs
 - User says: `"apply the config template"` or `"fix my infra config"`
 - User says: `"configure <agent_name> with <model>"`
 **How to run (via OpenProse CLI):**
 ```bash
 # Minimal — only agent name required
 prose run hermes-config-template agent_name=syslog-code
 # Full control
 prose run hermes-config-template \
  agent_name=syslog-devops \
  default_model=claude-sonnet-4 \
  auxiliary_model=gemma-4-12b \
  fallback_model=deepseek-chat \
  fallback_provider=deepseek
 ```
 ### How to Follow the Template (Manual)
 If `prose` CLI is not available:
 1. **Fetch this contract:** `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/hermes-config-template.prose.md`
 2. **Open the target config.yaml** — the profile you're updating
 3. **Compare** each of the 5 required sections below against what exists in the profile
 4. **For each section:**
   - If it says **SHARED INFRA — DO NOT CHANGE**: Copy the value exactly as shown
   - If it says **USER CHOOSES**: Use the agent's assigned model/provider
   - If missing entirely: Add the full section
 5. **Verify** every endpoint responds (see Verification section at the end)
 6. **Restart** the gateway
 ### Decision Tree
 ```
 What kind of profile is this?
 ├── Main profile (~/.hermes/config.yaml)
 │   └── Replace web.*, mcp_servers.*, compression.*, auxiliary.*, custom_providers.*
 │   └── model.default + fallback: use current values (USER CHOOSES)
 │
 ├── Worker profile (~/.hermes/profiles/<name>/config.yaml)
 │   ├── Does it have web.search_backend?
 │   │   ├── NO → Add the full web.* section (copy exactly)
 │   │   └── YES → Verify it points to the right IP
 │   ├── Does it have mcp_servers.ra-h-os?
 │   │   ├── NO → Add mcp_servers.* section (copy exactly)
 │   │   └── YES → Verify URL
 │   └── model.default: USE the worker's assigned model (don't copy main)
 │
 └── Non-Syslog profile (personal/lifestyle agent)
    └── Skip — use default Hermes config or Tanko's config instead
 ```
 ## Infrastructure Stack
 This template provisions the following shared infrastructure. Every agent should point to the same endpoints unless explicitly overridden.
 | Component | Endpoint | Purpose |
 |---|---|---|
 | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
 | SearXNG | `http://storepve:8888` | Privacy-respecting web search |
-| LiteLLM | `http://litellm.sysloggh.net/litellm/v1` | Unified model gateway |
+| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
 | LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
 | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
 | Context7 MCP | `http://localhost:8079/mcp` | Documentation queries |
-**API Key Rules:**
+## API Key Rules
 - `api_key: sk-_SW...B8nw` — Hardcoded LiteLLM master key for custom_providers and auxiliary tasks
 - `api_key_env: DEEPSEEK_API_KEY` — Env-var based key for fallback provider
 - Worker profiles should inherit proxy config via `harness` custom_provider, NOT hardcode LiteLLM keys
-**For workers that need API keys overridden:** Set `api_key: ''` in the model section (inherits from custom_providers), or use `api_key_env` for env-based auth.
+- `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred)
  Key is injected at runtime via `infisical run --` wrapper — never in /etc/environment
 - `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider
 - `api_key: sk-...` — Hardcoded key only as fallback when env var not possible
 - Store `LITELLM_API_KEY` in Infisical vault (project=agents, env=production)
 - Sub-agents NEVER get their own key — they share the host agent's key
 - Restart Hermes gateway after updating vault secret (key auto-injected via wrapper)
-## Template Structure
+### Sub-Agent Profiles (Mumuni pattern)
-### Required Sections (every agent MUST have)
+Mumuni has 6 sub-agent profiles in `/root/.hermes/profiles/<name>/config.yaml`:
 ```
 profiles/
 ├── syslog-code/config.yaml      # Code generation
 ├── syslog-devops/config.yaml     # DevOps/infrastructure
 ├── syslog-email/config.yaml      # Email processing
 ├── syslog-research/config.yaml   # Research & analysis
 ├── syslog-review/config.yaml     # Code review
 └── syslog-writer/config.yaml     # Content writing
 ```
 Sub-agent profile rules:
 1. **`api_key` must be empty** — `api_key: ''` or omitted entirely
 2. **`base_url` must be empty** — inherits from main config's custom_provider
 3. **`provider` is `auto` or `harness`** — routes through the shared LiteLLM gateway
 4. **`model` is agent-specific** — each sub-agent can have its own default model
 5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty
 6. **Never hardcode a key** in sub-agent profiles
 This ensures all 6 sub-agents use the same LiteLLM key injected via `infisical run --` wrapper.
 When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
 work immediately after restart.
 ## Template — Required Sections
 ```yaml
-# ─── Model Selection (USER CHOOSES) ───
+# ─── Model Selection ───
 model:
-  # !! CHANGE THIS for your agent !!
+  default: <agent_model>          # e.g., strix-moe, qwen3.6-27B-code, syslog-auto
-  default: <agent_default_model>
+  provider: harness
-  provider: <agent_default_provider>
+  base_url: http://192.168.68.116/v1
-  # LiteLLM base URL (shared infra)
+  api_key_env: LITELLM_API_KEY    # Injected via infisical run -- wrapper
-  base_url: http://litellm.sysloggh.net/litellm/v1
+  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
  context_length: 131072          # For syslog-auto (all GPUs at 128K for stability).
                                  # Set 65536 if using gemma-4-12b directly (tight VRAM).
 fallback_providers:
-  # !! OPTIONAL: overridable fallback !!
+  provider: deepseek
-  provider: <fallback_provider>
+  model: deepseek-chat
-  model: <fallback_model>
+  api_key_env: DEEPSEEK_API_KEY
 # ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
 web:
@@ -139,7 +119,7 @@ web:
  firecrawl:
    base_url: http://192.168.68.7:3002/
-# ─── MCP Servers (SHARED INFRA — DO NOT CHANGE) ───
+# ─── MCP Servers (SHARED INFRA) ───
 mcp_servers:
  context7:
    connect_timeout: 60
@@ -150,86 +130,79 @@ mcp_servers:
    timeout: 120
    connect_timeout: 60
-# ─── Compression (SHARED — can override model) ───
+# ─── Compression ───
 compression:
  enabled: true
  model: syslog-auto              # ⚠️ Switched from strix-moe 2026-07-18 to relieve Strix Halo.
                                  # syslog-auto distributes across weighted pool (55% RTX 3090,
                                  # 30% Strix Halo, 15% RTX 5070). All GPUs at 128K.
  provider: harness
-  model: <auxiliary_model>  # default: gemma-4-12b
+  max_context_window: 131072      # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
-  threshold: 0.5
+  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
-  target_ratio: 0.25
+  target_ratio: 0.30
-  protect_last_n: 30
+  protect_last_n: 40
-  hygiene_hard_message_limit: 400
+  hygiene_hard_message_limit: 350
  protect_first_n: 3
  abort_on_summary_failure: false
-# ─── Auxiliary Tasks (SHARED INFRA + MODEL) ───
+# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
 # All auxiliary services MUST use identical model, base_url, and api_key_env:
 #   model: gpu-light                 # stable alias (NOT raw "gemma-4-12b")
 #   base_url: http://192.168.68.116/v1
 #   api_key_env: LITELLM_API_KEY
 # Compression uses syslog-auto (switched from strix-moe 2026-07-18) to distribute
 # load across the weighted pool and relieve Strix Halo pressure.
 # Vision and web_extract use gpu-light = RTX 5070 (12B).
 # Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
 # NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
 # in agent configs — use the stable aliases so model swaps don't break agents.
 auxiliary:
  vision:
    provider: harness
-    model: <auxiliary_model>
+    model: gpu-light            # stable alias for RTX 5070 (was raw gemma-4-12b)
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    timeout: 60
    download_timeout: 30
  web_extract:
    provider: harness
-    model: <auxiliary_model>
+    model: gpu-light            # stable alias for RTX 5070
-  session_search:
+    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    timeout: 30
  compression:
    provider: harness
-    model: <auxiliary_model>
+    model: syslog-auto             # Switched from strix-moe 2026-07-18. Relieves Strix Halo pressure.
-    max_concurrency: 3
+    base_url: http://192.168.68.116/v1   # Rule 5: /v1 NOT /litellm/v1
    api_key_env: LITELLM_API_KEY
    timeout: 300                 # gpu-fleet: 300s for large-history summarization (was 60)
-# ─── Custom Provider (SHARED INFRA — LiteLLM) ───
+# ─── Delegation / Heavy Aux (use gpu-dense = RTX 3090) ───
 # delegation.model and x_search.model use gpu-dense (NOT raw qwen3.6-27B-code).
 delegation:
  model: gpu-dense              # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
  provider: harness
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY
 # ─── Custom Provider ───
 custom_providers:
-  - name: <custom_provider_name>
+  - name: harness
-    model: <custom_provider_model>
+    model: syslog-auto            # weighted pool (default)
-    base_url: http://litellm.sysloggh.net/litellm/v1
+    base_url: http://192.168.68.116/v1
-    api_key: <liteLLM_master_key>
+    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
 ```
-### Optional Sections (agent-specific)
+## Key Update Procedure
- **Platform configs** (Telegram, Discord, WhatsApp bot tokens)
+When LiteLLM keys are regenerated (e.g., after infrastructure changes):
 - **Display/skin** options (compact mode, personality)
 - **Plugin lists** (zulip-platform, etc.)
 - **Terminal settings** (container image, timeouts)
 - **Security/approvals** (command allowlists)
 - **Kanban/worker** settings (if this profile is a Kanban worker)
 - **Skills auto_load** list
 - **Timezone** override
-## Continuity
+1. **If SSH available**: Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-<NEW> --project=agents --env=production`, then `ssh <host> "systemctl restart hermes-gateway"`
-
+2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command
-Configuration drift detection is **event-driven**:
+3. **After update**: Restart Hermes on the agent host
-
+4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
 - **On agent onboarding**: Always scaffold from this template
 - **On infra change**: If Firecrawl/SearXNG/LiteLLM endpoints change, update template + all profiles
 - **On new model**: When a new model is deployed on Litellm, any agent can change their `default` independently
 - **Periodic**: Every 2 weeks — check if any profile's infra section has drifted from the template
 - **On connection failure**: If web search/extract fails, check if that agent has missing `web.*` config
 ## Success Criteria
 The configuration is CONSIDERED GOOD when ALL of these pass:
 ### Infra Connectivity
 - `web.backend == "firecrawl"` — Firecrawl backend configured
 - `web.search_backend == "searxng"` — SearXNG search configured
 - `web.extract_backend == "firecrawl"` — Firecrawl extraction configured
 - `web.firecrawl.base_url` points to `192.168.68.7:3002`
 - `mcp_servers.ra-h-os.url` points to `192.168.68.65:3100/mcp`
 - All endpoints respond to `curl` health checks
 ### Model Flexibility
 - `model.default` is settable per agent (not hardcoded)
 - `model.provider` is settable per agent
 - `custom_providers[0].model` matches the agent's workload (code vs general)
 ### Completeness
 - `auxiliary.vision` configured (even if provider=auto)
 - `compression.enabled` set (true for most agents)
 - Fallback provider configured (deepseek or another harness model)
 - `_config_version` correctly set (currently `30`)
 ### No Drift
 - No duplicate infra endpoints (one canonical Firecrawl URL)
 - No stale endpoints (old IPs, dead services)
 - All profiles consistent on shared infra
 ## Configuration Rules
@@ -241,67 +214,176 @@ The following MUST be identical across ALL profiles:
 - `custom_providers[0].base_url`
 ### Rule 2: Model Choice Is Free
-The following are OWNED by each agent and can differ:
+- `model.default` — per agent
- `model.default` — the primary model
+- `fallback_providers.model` — per agent
- `model.provider` — where it runs
+- `custom_providers[0].model` — per agent
 - `fallback_providers.model` — backup model
 - `custom_providers[0].model` — what this agent uses LiteLLM for
-### Rule 3: Workers Inherit, Not Duplicate
+### Rule 3: API Keys via Environment
-Worker profiles (syslog-code, syslog-devops, etc.) generally inherit from the main config. Only override what's DIFFERENT:
+- Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
- Different default model → override `model.default`
+- Hardcoded keys in config.yaml become stale after key rotation
- Different auxiliary model → override auxiliary sections
+- Infisical vault secrets persist across config updates / reinstalls
- No web search needed → still keep the config (tools won't enable without it)
+- **NEW (July 2026): Always keep a local `.env` fallback.** Infisical service tokens
  can expire/404 (tanko incident: token not found, gateway ran without key for hours).
  The `.env` file should have the key uncommented as a fallback:
  ```
  LITELLM_API_KEY=sk-...
  # [INFISICAL] Also sourced from vault.sysloggh.net
  ```
 - Restart Hermes after env var updates
-### Rule 4: API Key Placement
+### Rule 4: Sub-Agent Profiles Inherit Auth
- **Custom provider keys** (`custom_providers[0].api_key`): Must be hardcoded (the LiteLLM master key or a virtual key)
+- Sub-agent profiles (`/root/.hermes/profiles/*/config.yaml`) must have:
- **Model section keys** (`model.api_key`): Prefer empty. Workers inherit from custom_providers.
+  - `api_key: ''` — inherit from main config's custom_provider
- **Environment keys** (`api_key_env`): For fallback providers that need separate auth (DeepSeek, OpenRouter)
+  - `base_url: ''` — inherit from main config
- **Hardcoded keys in profile**: If a worker's `model.api_key` is set, it overrides custom_providers. Only use when the worker needs a DIFFERENT provider than Litellm.
+  - Auxiliary tasks: `api_key: ''`, `provider: harness`
 - Never hardcode a key in sub-agent profiles
 - When main config uses `api_key_env`, sub-agents automatically use it
 - This means key rotation only touches ONE vault secret (`LITELLM_API_KEY`)
-### Rule 5: Verify After Every Config Change
+### Rule 5: Main Config Base URL
-After changing a profile's config:
+|- Use direct IP: `http://192.168.68.116/v1`
-1. `curl` the model endpoint to confirm auth works
+|- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
-2. `curl` the web stack endpoints (Firecrawl, SearXNG)
+|- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly
-3. Check that `hermes gateway --restart` reloads cleanly
+
 ### Rule 6: max_tokens Is Required (Thermal Safety)
 - **Every Hermes config MUST set `model.max_tokens: 4096`** — this is non-negotiable
 - Prevents unbounded generation that caused the July 2 Strix Halo GPU thermal incident
 - The value flows through: `config → agent.max_tokens → transport build_kwargs → API max_tokens`
 - Even though the server now has `-n 8192` hard cap (set by Abiba), the client cap is the first line of defense
 - Apply to BOTH main config AND all sub-agent profiles
 - For agents needing longer outputs: raise to 8192, but never omit
 ### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-18)
 - Vision and web_extract use `gpu-light` (stable alias, RTX 5070 — 12GB, vision-optimized)
 - Compression now uses `syslog-auto` (switched from `strix-moe` 2026-07-18) to distribute
  compression load across the weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
  This relieves Strix Halo pressure while keeping compression functional on all GPUs.
 - **`syslog-auto` is the valid compression model** — LiteLLM serves it as the weighted pool.
  Old configs with `strix-moe` for compression should be updated to `syslog-auto`.
 - All auxiliary services MUST use identical routing:
  - `base_url: http://192.168.68.116/v1`  (Rule 5: `/v1`, NOT `/litellm/v1`)
  - `api_key_env: LITELLM_API_KEY`
 - **Compression via syslog-auto**: Routes through the weighted pool. Strix Halo still handles
  ~30% of compression calls (at 60 RPM via pool vs 40 RPM direct), but the bulk (55%)
  goes to RTX 3090 which has ample spare capacity.
 - The `compression:` block's `model` MUST match `auxiliary: compression: model`
 - The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
 ### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
 - **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
 - **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
 - **Strix Halo (64GB, 128K ctx, qwen3.6-35B-udq4)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
 - Agent profiles MUST route auxiliary tasks to the correct GPU:
  - `auxiliary.vision.model: gpu-light` (RTX 5070)
  - `auxiliary.web_extract.model: gpu-light` (RTX 5070)
  - `auxiliary.compression.model: syslog-auto` (distributed pool, switched from strix-moe 2026-07-18)
 - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
 - For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
 - Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin
 - `max_context_window: 131072` MUST match the model's actual capacity (128K)
 - See `devops-hermes-compression` skill for full reference
 ### Rule 9: Compression Threshold for 128K Models
 - For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
 - Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin
 - `max_context_window: 131072` MUST match the model's actual capacity (all GPUs = 128K)
 - See `devops-hermes-compression` skill for full reference
 ### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
 - **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
 - **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
 - `syslog-auto` is the LiteLLM routing model — it load-balances between strix-moe
  and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
  - Model name typos that cause 403 errors and silent worker failures
  - Single GPU downtime (routing falls back automatically)
  - Key/model authorization mismatches
 - **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
  for specialized tasks, but MUST validate those models exist in the key's authorized list
 ### Rule 11: Validate Model IDs Before Deployment (pi Agents)
 - After configuring a pi agent's `models.json`, verify every model ID:
  ```bash
  curl -s http://192.168.68.116:4000/v1/models \
    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
  ```
 - All model IDs in `models.json` MUST appear in the LiteLLM response
 - The agent's API key may have a SUBSET of the full model catalog — check per-key
 - A non-existent model ID causes 403 errors that silently break the pi RPC worker
  (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
 ### Rule 12: Context-Issue Diagnostic Checklist (ADDED 2026-07-16, WAL #1300)
 When an agent shows "context issues" (premature compression, 401s, 504s, DeepSeek fallback),
 verify ALL FOUR of these against the live config. They are the only root causes found in production:
 1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
   MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used.
   ~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
 2. **base_url uses /v1 NOT /litellm/v1?** — `custom_providers[0].base_url`, `delegation.base_url`,
   and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/v1` (Rule 5). nginx `/litellm/`
   has a 60s default timeout → 504 on any inference >60s; `/v1/` has 600s.
   Check: `grep -n 'litellm/v1' ~/.hermes/config.yaml` (must return NOTHING)
 3. **LITELLM_API_KEY valid?** — The key must be a real LiteLLM key (`sk-` + 64 hex, 67 chars).
   Malformed values (e.g. `sk-_SWAl_Vu_…`, 47 chars) return 401 → DeepSeek fallback.
   Verify: `curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LITELLM_API_KEY" http://192.168.68.116/v1/models` (must be 200)
 4. **custom_providers aligned?** — `model: syslog-auto` (Rule 10), `api_mode: chat_completions`
   (NOT `responses`). A wrong api_mode causes silent request failures.
 One-line agent health check (run on the agent host):
 ```bash
 # Use grep -v infisical to avoid matching the bash wrapper that contains the same string
 PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
 cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
 curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
 ```
 ### Rule 13: API Key Injection — Two Patterns (UPDATED 2026-07-16, WAL #1300)
 Agents inject `LITELLM_API_KEY` via ONE of two mechanisms. Both are valid; the contract
 requirement is that the key is a **valid LiteLLM virtual key** (HTTP 200 on /v1/models).
 **Pattern A — systemd drop-in (Koby, Koonimo, and any agent without infisical wrapper):**
 A systemd drop-in `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` sets the key:
 ```ini
 [Service]
 Environment="LITELLM_API_KEY=sk-<VALID_KEY>"
 ```
 The service unit `hermes-gateway.service` runs `python -m hermes_cli.main gateway run --replace`
 directly (no infisical). Apply with `systemctl daemon-reload && systemctl restart hermes-gateway`.
 - Koonimo (CT113/.114): service = `hermes-gateway.service`, drop-in has the key.
 - Koby (CT111/.129): service = `hermes-gateway.service` (created 2026-07-16), ExecStart uses `--replace`
  to win the lock against stray `hermes gateway restart` invocations. Key also in `/etc/environment`.
 **Pattern B — infisical-gateway.sh wrapper (Mumuni):**
 The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
 See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
 **⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
 the wrapper will inject an empty key and the gateway will silently get 401 errors
 on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
 is present but the vault takes precedence when the secret key exists (even if empty).
 **Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
 is empty or shorter than 20 chars, log a warning and either fail with a clear error
 message or fall back to the `.env` value before starting the gateway.
 **Verification (all agents):**
 ```bash
 # Use grep -v infisical to avoid matching the bash wrapper that contains the same string
 GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
 K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
 curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models  # must be 200
 ```
 - `/etc/environment` is NO LONGER the canonical key source (stale values there caused 401s).
 - Do NOT leave a hardcoded stale key in `/etc/environment` — it shadows the drop-in/wrapper.
 ## Execution
 1. **Check current config** — Read the target agent's config.yaml
 2. **Compare against template** — Identify missing or divergent sections
-3. **Apply shared infra** — Lock the web/MCP/compression sections to template values
+3. **Apply shared infra** — Lock web/MCP/compression sections to template values
-4. **Apply model choice** — Set default/fallback/auxiliary models per agent's workload
+4. **Apply agent key** — Set from agent_keys table above
-5. **Preserve agent-specific** — Keep platform configs, plugins, terminal settings, skills
+5. **Set model choice** — Per agent's workload
-6. **Remove stale** — Drop any old infra endpoints that don't match the template
+6. **Verify** — curl all shared endpoints, test the model with the new key
-7. **Verify** — curl all shared endpoints, test the model
+7. **Report** — What was changed, preserved, custom
 8. **Report** — What was changed, what was preserved, what's still custom
 ## Example Output
 ```
 ## Config Template Applied: syslog-code ✅
 ### Shared Infrastructure (Locked)
 | Section | Before | After |
 |---|---|---|
 | web.backend | firecrawl | firecrawl ✅ |
 | web.search_backend | NOT SET | searxng ✅ |
 | web.firecrawl.base_url | NOT SET | http://192.168.68.7:3002/ ✅ |
 | mcp_servers.ra-h-os | present | present ✅ |
 ### Agent Model Choices (Preserved)
 | Setting | Value |
 |---|---|
 | model.default | qwen3.6-27B-code (code agent) |
 | fallback | gemma-4-12b (harness) |
 | auxiliary | gemma-4-12b (harness) |
 | compression | gemma-4-12b (harness) |
 ### Endpoint Verification
 | Endpoint | Status |
 |---|---|
 | Firecrawl :3002 | ✅ 200 OK |
 | SearXNG :8888 | ✅ 200 OK |
 | LiteLLM | ✅ 200 OK |
 | RA-H OS MCP |  ✅ Connected |
 ```
@@ -0,0 +1,301 @@
 ---
 kind: enforcement
 name: hermes-key-enforcement
 version: 1.0.0
 description: >
  Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
  providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
  Anthropic) may use hardcoded keys. Single source of truth: Infisical vault
  (project=agents, env=production) — injected at runtime via `infisical run --` wrapper.
  /etc/environment is DEPRECATED for agent keys post-migration. Designed to make key
  rotation a one-step vault operation.
 author: Abiba (pi agent)
 ---
 # Hermes Key Enforcement Contract
 ## Rule (One Sentence)
 **All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
 ## Scope
 Applies to all Hermes agent configs across all hosts. Covers these config sections:
 - `model.api_key`
 - `custom_providers[].api_key` (when `name` contains `harness` or `litellm`)
 - `auxiliary.*.api_key` (when `provider` is `harness` or contains `litellm`)
 - `delegation.api_key` (when `provider` is `harness` or contains `litellm`)
 - `compression.api_key` (when `provider` is `harness` or contains `litellm`)
 - `fallback_providers[].api_key` (when provider is harness)
 ## Architecture (2026-07-10)
 Syslog is migrating away from **unauthenticated direct access** to the shared inference harness.
 | Path | Auth | Status |
 |------|------|--------|
 | `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
 | `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
 All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
 Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
 ### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
 When `api_mode: responses` is set, Hermes **appends `/v1/responses`** to `base_url`.
 If `base_url` already includes `/litellm/v1/responses`, the result is:
 ```
 http://192.168.68.116/litellm/v1/responses/v1/responses → 404
 ```
 **The `base_url` must end at `/v1` — never include `/responses`:**
 ```yaml
 # ✅ CORRECT — Hermes appends /v1/responses for api_mode: responses
 base_url: http://192.168.68.116/litellm/v1
 # ❌ WRONG — produces double path
 base_url: http://192.168.68.116/litellm/v1/responses
 ```
 This applies to ALL sections using the harness provider: `custom_providers`, `delegation`, `auxiliary.*`.
 ## Exemptions
 External providers are **explicitly exempt** and may use hardcoded keys:
 - DeepSeek (`api.deepseek.com`)
 - OpenAI (`api.openai.com`)
 - Anthropic (`api.anthropic.com`)
 - OpenRouter
 - Any provider whose base_url does NOT match `192.168.68.116` or `litellm.sysloggh.net`
 ## Standard Pattern
 > **Canonical vault process (2026-07-16):** see `litellm-api-keys` § Production Vault Access Process.
 > All agents MUST use the `infisical-gateway.sh` wrapper (live vault injection). Hardcoded systemd
 > drop-ins / config.yaml keys are DEPRECATED — they rot on rotation (root cause of the 2026-07-16 401 storm).
 > 4/5 agents migrated; tanko (user jerome) pending.
 ```yaml
 # ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
 model:
  provider: harness
  base_url: http://192.168.68.116/litellm/v1           # ← Hermes appends /v1/responses
  api_key_env: LITELLM_API_KEY
 custom_providers:
  - name: harness
    api_mode: responses
    base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
    api_key_env: LITELLM_API_KEY
 auxiliary:
  compression:
    provider: harness
    base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
    api_key_env: LITELLM_API_KEY
 # ✅ ALSO CORRECT — external providers
 fallback_providers:
  - provider: deepseek
    base_url: https://api.deepseek.com
    api_key: sk-b7d9...               # ← hardcoded OK (external)
    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```
 ```yaml
 # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
 model:
  provider: harness
  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
 model:
  provider: harness
  base_url: http://192.168.68.116/v1   # ← RULE VIOLATION: unauthenticated path
  api_key_env: LITELLM_API_KEY
 ```
 ## Detection Query
 Run on any Hermes host to detect violations:
 ```bash
 # 1. Check config.yaml for hardcoded harness keys
 grep -rn 'api_key: sk-' /root/.hermes/ \
  --include='config.yaml' \
  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
 # 1b. Check for double-path bug: base_url ending with /responses
 # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
 grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
 # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
 grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
 grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
  | tr '\0' '\n' | grep LITELLM_API_KEY
 ```
 If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
 ## Rotation Procedure
 With this standard enforced, key rotation is one vault update:
 ```bash
 # 1. Generate new key in LiteLLM: POST /key/generate with agent alias
 # 2. Update Infisical vault secret
 infisical secrets set LITELLM_API_KEY=sk-NEW_KEY \
  --project=agents --env=production
 # 3. Restart agent gateway (key auto-injected via infisical run -- wrapper)
 ssh root@<host> "systemctl restart hermes-gateway"
 # 4. Verify
 curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/litellm/v1/models
 ```
 **Done.** No config file changes needed. No /etc/environment edits needed.
 The agent picks up the new key via `infisical run --` at gateway startup.
 > **Post-migration note**: /etc/environment is NO LONGER the key source.
 > Strip all `LITELLM_API_KEY` lines from /etc/environment (comment out with `# [INFISICAL]`)
 > and let the `infisical run --` wrapper inject the key at runtime.
 ## Key Longevity Policy (2026-07-04)
 **Keys are permanent and use bare agent name aliases.**
 - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
 - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
 - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
 - **Max budget**: $100 per key (config default).
 ```yaml
 # In litellm_config.yaml — ensures all future keys inherit these defaults:
 litellm_settings:
  default_key_generate_params:
    models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b"]
    duration: null        # ← permanent
    max_budget: 100
    metadata:
      purpose: "agent-inference"
 ```
 ## Verified Agents (2026-07-05 update)
 | Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
 |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
 | Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
 | Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
 | Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
 | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
 | Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
 > **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
 > LiteLLM key aliases use agent identity, not CT hostname.
 ### Migration Status: Authenticated Path
 | Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
 |-------|--------------------------|--------------------|--------|
 | Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
 | Tanko | ⚠️ No SSH access | — | Needs check |
 | Koby | ⚠️ No route to host | — | Needs check |
 | Koonimo | ⚠️ Connection timed out | — | Needs check |
 ### Systemd Service Pattern (2026-07-11 — vault migration)
 All Hermes agents use systemd to manage their gateway. The gateway service is wrapped
 with `infisical run --` to inject secrets at runtime.
 **Correct pattern (post-migration):**
 ```ini
 # Service file wraps gateway with Infisical:
 [Service]
 ExecStart=/usr/bin/infisical run --project=agents --env=production -- \
  /usr/bin/hermes gateway run
 # /etc/environment is CLEAN — no LITELLM_API_KEY present
 # (strip it and tag with # [INFISICAL] if present)
 ```
 **Legacy pattern (deprecated — pre-migration only):**
 ```ini
 # DO NOT USE post-migration:
 EnvironmentFile=/etc/environment
 # This pattern was replaced by infisical run -- wrapper
 ```
 **Rotation procedure** (one vault operation with this standard):
 1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
 2. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-NEW --project=agents --env=production`
 3. Restart: `systemctl restart hermes-gateway` (key auto-injected via wrapper)
 ## Violation Response
 1. **Detect** — run detection query above
 2. **Fix** — replace `api_key: sk-...` with `api_key_env: LITELLM_API_KEY` in all harness/litellm sections
 3. **Verify** — `grep -c "api_key_env" config.yaml` should increase, hardcoded harness keys should be 0
 4. **Restart** — gateway must restart to pick up env var
 5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
 6. **Update** — bump the verified table above
 7. **Use safe-mutate** — if the fix requires updating vault secrets or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
 ## Related Contracts
 - `hermes-config-template.prose.md` — full configuration template
 - `litellm-health.prose.md` — LiteLLM stack health verification
 - `zulip-platform-verification.prose.md` — cross-platform agent verification
 - `litellm-api-keys.prose.md` — API key creation, rotation, and verification
 ## CI Pipeline (2026-07-04)
 All contract changes must pass the PR Pipeline before merge:
 ```
 auth → validate → lint → ai-review → gate
 ```
 - **Trigger**: push to master (abiba-bot only) or pull request
 - **Branch protection**: Only `abiba-bot` can push directly to master. All other users must use PRs.
 - **Status check**: `PR Pipeline — Authorize → Validate → Review → Merge` required before merge
 - **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
 - **Config**: `.gitea/workflows/pr-pipeline.yaml`
 ## Known Bug: auxiliary_client ignores api_key_env (2026-07-05)
 **Bug**: `_resolve_task_provider_model()` in `agent/auxiliary_client.py` reads
 `api_key` from auxiliary task configs (vision, compression, etc.) but does NOT
 resolve `api_key_env`. The custom provider resolution path handles `api_key_env`,
 but auxiliary tasks take a different code path that ignores it.
 **Impact**: Vision analysis and compression calls fall through to the `"no-key-required"`
 placeholder, causing 401 errors on LiteLLM/harness (which require `sk-*` keys).
 **Workaround**: Set `api_key` directly alongside `api_key_env` in each auxiliary
 task config:
 ```yaml
 auxiliary:
  vision:
    api_key: sk-<agent-key-from-vault>           # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
  compression:
    api_key: sk-<agent-key-from-vault>           # ← workaround (same as above)
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
 ```
 **Affected agents**: All Hermes agents with harness/LiteLLM provider and
 `api_key_env` in auxiliary configs (all 4 Hermes agents patched 2026-07-05).
 **Source location**: `agent/auxiliary_client.py` line 5478
@@ -0,0 +1,238 @@
 ---
 kind: function
 name: hermes-zulip-plugin
 description: >
  Installs or updates the Zulip platform plugin for any Hermes agent from the
  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
  recovery). Verifies the installation, restarts the gateway, and sends a relay
  success signal.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Plugin — Install & Repair
 Single-shot function that pulls the latest zulip-platform plugin from the
 canonical git repo, installs it to the correct Hermes bundled plugin path,
 verifies the installation, and signals completion.
 Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
 layer and a targeted gateway restart. It does NOT verify env credentials or
 run a live Zulip connection test. Use `hermes-zulip-restore` for full
 connectivity recovery including end-to-end DM validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 ## Maintains
 - plugin_installed: bool — Whether all three adapter files exist at the bundled path
 - plugin_version: string — Git commit SHA of the installed version
 - strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
 - signal_sent: bool — Whether relay success message was dispatched
 ### Postconditions
 - All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
 - `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
 - Installed version matches HEAD of the requested branch
 - Relay success signal sent to Hermes agent's inbox
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
 | Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
 ## Execution
 ### Step 1: Resolve Target
 Map `target` to host, CT ID, hermes_home, and user from the live-state table.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Pull Latest Plugin Source
 On the target host:
 ```bash
 # Ensure deploy scratch space
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 # Clone or pull
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins
  git fetch origin
  git checkout {{branch}}
  git pull origin {{branch}}
 else
  git clone --branch {{branch}} \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
  cd zulip-platform-plugins
 fi
 # Capture installed version
 INSTALLED_SHA=$(git rev-parse HEAD)
 echo "Installed SHA: $INSTALLED_SHA"
 # Verify we're at HEAD
 HEAD_SHA=$(git rev-parse origin/{{branch}})
 if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
  echo "At latest commit on {{branch}}"
 else
  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
 fi
 ```
 ### Step 3: Install Plugin Files
 ```bash
 # Ensure target directory exists
 mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp plugins/platforms/zulip/adapter.py \
   plugins/platforms/zulip/__init__.py \
   plugins/platforms/zulip/plugin.yaml \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only — runs as jerome user)
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 echo "Plugin files installed"
 ```
 ### Step 4: Verify Installation
 ```bash
 # Check all three files exist
 for f in __init__.py adapter.py plugin.yaml; do
  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
    echo "✅ $f present"
  else
    echo "❌ $f MISSING"
    exit 1
  fi
 done
 # Verify _strip_html fix
 grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
  && echo "✅ _strip_html fix present" \
  || echo "❌ _strip_html MISSING — plugin may be stale"
 # Show installed plugin.yaml version
 grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
 ```
 ### Step 5: Restart Gateway
 Plugin changes require a gateway restart to take effect:
 ```bash
 cd {{hermes_home}}/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart 2>&1 || \
  venv/bin/python -m hermes_cli.main gateway restart 2>&1
 ```
 Wait for restart to complete (up to 45s), then confirm:
 ```bash
 grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
 ```
 Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
 Quick smoke check — confirm zulip platform loaded:
 ```bash
 grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
 ```
 If gateway fails to restart, check logs for the crash cause before proceeding.
 ### Step 6: Send Relay Success Signal
 On the Abiba host (local), dispatch a relay message to the target agent:
 ```
 ra-h-os-createRelayNode:
  title: "Zulip plugin updated — {{target}}"
  source: |
    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
    _strip_html fix: PRESENT
    Timestamp: {{timestamp}}
  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
 ```
 ### Step 7: Report
 Compile results into a single status block:
 | Field | Value |
 |-------|-------|
 | Target | `{{target}}` |
 | Branch | `{{branch}}` |
 | Commit SHA | `{{INSTALLED_SHA}}` |
 | Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
 | `_strip_html` | `{{present|missing}}` |
 | Gateway restarted | `{{yes|no}}` |
 | Signal sent | `{{yes|no}}` |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
 | Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
 | `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
 | Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
 | Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
 ## Edge Differences from hermes-zulip-restore
 | Concern | hermes-zulip-restore | hermes-zulip-plugin |
 |---------|---------------------|---------------------|
 | Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
 | Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
 | Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
 | Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
 | Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
 | Signal dispatch | ❌ None | ✅ Relay message to target |
 For full connectivity recovery after a plugin install, chain this contract
 with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
 ---
 **Last updated**: 2026-07-08 — Switched default branch to `master`; added
 gateway restart step. If outstanding unmerged feature branches exist, address
 them in a follow-up merge after this contract completes.
@@ -0,0 +1,189 @@
 ---
 kind: function
 name: hermes-zulip-restore
 description: >
  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
  Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
  path, verifies env credentials, restarts the gateway, and confirms Zulip
  connects. Run this whenever a Hermes agent stops responding on Zulip or after
  a fresh agent deployment.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Hermes Zulip Restore — Bring Any Agent Back to Good State
 Single-shot function that restores full Zulip connectivity for a Hermes agent.
 Covers adapter deployment, HTML stripping (slash command fix), env verification,
 gateway restart, and connection validation.
 ## Parameters
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
 | `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
 ## Maintains
 - adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
 - zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
 - env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
 - gateway_running: bool — Whether gateway process is running
 ### Postconditions
 - `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
 - All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
 - Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
 - Gateway restarted and zulip platform reports state `connected`
 - HTML stripping enabled for `/approve` and `/deny` slash command support
 ## Requires
 - SSH access to target host (direct or via amdpve for CTs)
 - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
 - Python 3 with `httpx` installed on target
 - Zulip server accessible at `https://chat.sysloggh.net`
 ## Live-State Fields
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
 | Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 | Field | Value | Trust |
 |-------|-------|-------|
 | Zulip server | https://chat.sysloggh.net | ✅ Verified |
 | Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
 | Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
 | Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
 ## Execution
 ### Step 1: Locate Target
 Map `target` to connectivity parameters from the live-state table above.
 For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
 ### Step 2: Deploy Zulip Adapter
 On the target host:
 ```bash
 # Clone or update the plugin repo
 mkdir -p /tmp/zulip-deploy
 cd /tmp/zulip-deploy
 if [ -d zulip-platform-plugins ]; then
  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
 else
  git clone --branch feat/zulip-streaming \
    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
 fi
 # Ensure bundled plugin directory exists
 mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
 # Copy adapter files
 cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
 # Fix ownership (Tanko only)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
 # Clean up
 rm -rf /tmp/zulip-deploy
 ```
 ### Step 3: Verify _strip_html is Present
 ```bash
 grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
 ```
 Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
 ### Step 4: Verify Env Credentials
 ```bash
 grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
 ```
 Expected: all three variables set with non-empty values. If any missing:
 - ZULIP_SITE: `https://chat.sysloggh.net`
 - ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
 - ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
 ### Step 5: Restart Gateway
 ```bash
 cd <HERMES_HOME>/hermes-agent
 # Use venv if available
 python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
 ```
 Wait for the restart to complete (up to 45s). Check:
 ```bash
 grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
 ```
 Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
 ### Step 6: Validate Zulip Connection
 ```bash
 python3 -c "
 import json
 d = json.load(open('$HERMES_HOME/gateway_state.json'))
 print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
 "
 ```
 Expected: `zulip: connected`. If not connected, check gateway log:
 ```bash
 grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
 ```
 ### Step 7: Report
 Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
 | State | Action |
 |-------|--------|
 | All true | ✅ Agent restored — relay success to user |
 | `adapter_deployed: false` | Re-run Step 2 |
 | `env_valid: false` | Prompt for missing credentials |
 | `zulip_connected: false` | Check Zulip server reachability, verify API key |
 | `gateway_running: false` | Check process logs for crash cause |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
 | Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
 | No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
 | `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
 | Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
 | Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
 ## Git Branch Reference
 The `_strip_html` fix lives on `feat/zulip-streaming` branch:
 ```
 https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
 ```
 Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
 Pull request #33 is the primary integration branch.
 ---
 **Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
@@ -0,0 +1,103 @@
 ---
 name: inference-optimization
 kind: responsibility
 description: >
  Optimizes the full Syslog inference stack — LiteLLM routing weights, GPU model
  assignments, agent context management, and prompt caching — to reduce response
  times to sub-15s average. All GPUs now at 128K context (stable ceiling).
 id: 067NC6KP02RG60S50M40E30928
 ---
 ### Goal
 Syslog inference response times reduced to sub-15s average by optimizing the full
 stack: LiteLLM routing weights, GPU model assignments, Hermes agent context
 management, and prompt caching — without sacrificing agent capability.
 ### Requires
 - `inference-metrics`: current SpendLogs from CT116 LiteLLM Postgres — avg
  request_duration_ms, prompt_tokens, completion_tokens, model_group breakdown,
  cache_hit rate over the last 3 hours
 - `agent-configs`: current config.yaml from each active Hermes agent (Mumuni
  .123, any others on .129/.122) including compression, model, context_window,
  prompt_caching, memory settings
 - `gpu-health`: health check response from all 3 GPU backends (strix-moe .15:8080,
  qwen .8:8080, gemma .110:8080)
 ### Maintains
 The optimized inference stack configuration — every change is applied and
 verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of
 inference calls.
 #### liteLLM-routing
 The syslog-auto routing weights, model-specific timeouts, RPM limits, and
 model_list entries on CT116 `/opt/inference-harness/litellm_config.yaml`.
 #### agent-compression
 Each Hermes agent's `~/.hermes/config.yaml` compression, context_window,
 prompt_caching, and model sections.
 #### prompt-caching
 LiteLLM cache configuration and llama.cpp `--cache-prompt` flag on GPU hosts.
 #### verification
 End-to-end latency measurements after changes applied — at least 3 test
 inference calls per model path measuring ttft (time-to-first-token) and total
 duration.
 ### Continuity
 - input-driven
 ### Strategies
 **Context is the root cause.** Every ~46K prompt token costs ~87s of
 prefill time at 532 tok/s. Fix context first, routing second.
 - **Route by task**: qwen for code/standard queries; gemma for
  compression/auxiliary; strix-moe for compression tasks.
 - **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should
  compact at 51K, not 85K. Target 15% tail (not 30%).
 - **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these
  never change between turns. Single-digit cache hit rate is unacceptable.
 - **Lower context ceiling**: 128K window is the stable ceiling for agent conversations.
  GPUs reduced from 256K to 128K (2026-07-17). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers.
 ### Shape
 - `self`: analyze metrics, compute optimal configs, apply changes, verify
 - `delegates`:
  - `apply-liteLLM`: update litellm_config.yaml and reload
  - `apply-agent-config`: update hermes config.yaml per agent
  - `verify-latency`: run test inference calls and measure response
 ### Execution
 ```prose
 -- Phase 1: Analyze current state (already complete)
 -- Phase 2: Apply LiteLLM routing optimization
 call apply-liteLLM-routing
  config_path: /opt/inference-harness/litellm_config.yaml
  host: 192.168.68.116
 -- Phase 3: Apply agent context compression optimization  
 call apply-agent-compression
  agent: mumuni
  host: 192.168.68.123
  config_path: /root/.hermes/config.yaml
 -- Phase 4: Enable llama.cpp prompt caching on GPU hosts
 call enable-prompt-caching
  hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110]
 -- Phase 5: Verify end-to-end latency
 call verify-latency
  host: 192.168.68.116
  models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b]
 ```
@@ -3,9 +3,20 @@ kind: pattern
 name: infrastructure-control
 description: >
  Full infrastructure monitoring and control pattern covering the
-  5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
+  6-node Proxmox cluster, 3 Docker ecosystems (22 containers),
  NFS storage, and network services. Defines monitors, remediations,
  and the access matrix for all environments.
  FIELD TRUST: Live-state fields (IPs, ports, hostnames, credentials,
  container names, PIDs) are marked VERIFY-BEFORE-USE. They drift —
  never mutate infrastructure based on them without first confirming
  against the live system. Policy fields are authoritative. See the
  `verify-before-mutate` skill.
  **Last verified:** 2026-07-27 — gpu-dense swapped to SmartCode-Fable-5
  (Qwen3.6-27B distilled, ~50% fewer thinking tokens, improved coding reasoning).
  Model pricing reduced 100x across all models ($0.15/$0.60 per 1M tokens).
  All LiteLLM models set to 128K max_model_tokens. See data/learnings.md.
 ---
 # Infrastructure Control Pattern
@@ -23,7 +34,7 @@ description: >
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
-   │   CT 100    │    │   CT 122     │    │   CT 114     │
+   │   CT 100    │    │   CT 112     │    │   CT 114     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
@@ -36,8 +47,8 @@ description: >
               │      │      │      │      │
          ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
          ▼      ▼      ▼      ▼      ▼
-       minipve amdpve storepve acerpve ocupve
+       minipve amdpve storepve acerpve ocupve hwepve
-        (.12)   (.15)   (.6)    (.9)    (.5)
+        (.12)   (.15)   (.6)    (.9)    (.5)    (.4)
          ▼
   ┌─────────────────────────────────────────────┐
@@ -56,37 +67,54 @@ description: >
 | Resource | Auth Method | Credential Source | Status |
 |----------|------------|-------------------|--------|
-| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ |
+| Proxmox Cluster | PVE API Token | Infisical vault (`PROXMOX_API_TOKEN`) | ✅ |
-| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
+| Proxmox Root | Password via API ticket | Infisical vault (`PROXMOX_ROOT_PASSWORD`) | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
-| Tanko CT (.122) | SSH jerome | SSH key | ✅ |
+| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
 | Baggy CT (113) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
-| Gitea | API token | abiba-bot token | ✅ |
+| Gitea | API token | Infisical vault (`GITEA_BOT_TOKEN`) | ✅ |
-| Zulip | Bot API key | per-bot tokens | ✅ |
+| Zulip | Bot API key | Infisical vault (`ZULIP_BOT_KEY`) | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
 | LiteLLM Admin | master key | Infisical vault (`LITELLM_MASTER_KEY`) | ✅ VERIFY-BEFORE-USE |
 | Grafana | admin password | Infisical vault (`GRAFANA_ADMIN_PASSWORD`) | ✅ VERIFY-BEFORE-USE |
 > **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
 > contract are live-state fields. Test them against the live system before
 > relying on them (e.g., `curl -H "Authorization: Bearer $KEY" .../v1/models`
 > for a LiteLLM key; `pct config <ct>` for a CT IP). Drift is expected —
 > see `verify-before-mutate` skill. Policy fields (rules, doctrines) are
 > authoritative and do not need verification.
 ### Reachability Matrix
-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
-|-----------|---------|----------------|--------|-------------|---------------|
+|-----------|---------|----------------|--------|-------------|---------------|----------------|
-| **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird |
+| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
-| **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ |
+| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
-| **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ |
+| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
 **Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.
 ## Section 2: Proxmox Cluster — Monitoring
-### Nodes (5)
+### Nodes (6)
 | Node | IP | CPU | RAM | VMs/CTs | Role |
 |------|----|-----|-----|---------|------|
-| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging |
+| minipve | .12 | 16C | 30GB | authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
-| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
+| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute |
-| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
+| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat |
-| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
+| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
 | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
 | hwepve | .4 | ? | ? | abiba, (mumuni CT 114 stopped) | Agents (new node) |
 > **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
 > minipve at .10, not acerpve. Abiba (CT 100) is on hwepve and now runs Mumuni
 > Zulip gateway internally. CT 114 (old mumuni container) destroyed 2026-07-26.
 > Mumuni also has a second instance on minipve at .123 — distinguish by CT ID.
 ### Checks (every 5 min)
@@ -127,35 +155,128 @@ description: >
 ### Ecosystem A: docker-vm (192.168.68.7)
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 | Stack | Path | Containers |
 |-------|------|-----------|
 | **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
-| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse |
+| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
 | **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
 **Last verified:** 2026-07-09 — 16/16 containers running healthy.
 ### Home Stack Services
 **Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
 - URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
 - Swagger: `http://192.168.68.7:8989/swagger-ui.html`
 - Admin credentials: `admin` / `kakashi20stirling`
 - API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
 - Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
 - Compose: `/opt/home_stack/docker-compose.yml`
 - Control script: `/opt/home_stack/infra-control.sh`
 **JDownloader**:
 - URL: `http://192.168.68.7:5800` (web UI via VNC)
 **Pulse** (Uptime Kuma):
 - URL: `http://192.168.68.7:3001`
 ### Ecosystem B: CT 116 syslog-api (192.168.68.116)
-| Container | Image | Port |
+8 containers in inference-harness stack:
 |-----------|-------|------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 |
 | harness-nginx | nginx:alpine | :80 |
 | harness-router | inference-harness-router | :9000 |
 | harness-postgres | postgres:16-alpine | :5432 |
 | harness-redis | redis:7-alpine | :6379 |
 | harness-dashboard | inference-harness-dashboard | :3000 |
-### Ecosystem C: Netbird (72.61.0.17)
+| Container | Image | Port | Role |
 |-----------|-------|------|------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
 | harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
 | harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
 | harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
 | harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
 | harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
 | harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
 | harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
 **Nginx routing**:
 - `/v1/*` → harness-litellm:4000 (API)
 - `/admin/*` → harness-litellm:4000
 - `/dashboard/` → harness-dashboard:3000
 - `/litellm/*` → harness-litellm:4000
 - `/health/*` → harness-litellm:4000/health/liveliness
 - `/gpu/` → 192.168.68.24:9100 (fleet monitor)
 **Prometheus targets**:
 - 192.168.68.8:9400 (RTX 3090 — qwen)
 - 192.168.68.110:9400 (RTX 5070 — gemma)
 - 192.168.68.15:9400 (Strix Halo — qwen3.6-35B-udq4)
 - 192.168.68.24:9401 (Router metrics exporter)
 - harness-litellm:4000 (LiteLLM health)
 ### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)
 5 containers, single compose stack at `/root/docker-compose.yml`:
 | Container | Image | Role |
 |-----------|-------|------|
-| netbird-server | netbird | VPN controller |
+| netbird-server | netbirdio/netbird-server:latest | VPN controller, management API |
-| netbird-dashboard | netbird UI | Web management |
+| netbird-dashboard | netbirdio/dashboard:latest | Web management UI |
-| netbird-proxy | nginx | TLS termination |
+| netbird-proxy | netbirdio/reverse-proxy:latest | SNI router + TLS passthrough for *.sysloggh.net |
-| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF |
+| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF / threat intelligence |
-| netbird-traefik | traefik | Reverse proxy |
+| netbird-traefik | traefik:v3.6 | Edge reverse proxy, Let's Encrypt TLS |
 **Architecture:** Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy
 (TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends.
 Traefik handles the Netbird dashboard directly; all other `*.sysloggh.net`
 domains are TLS-passthrough to the proxy on port 8443.
 **Proxy config:** `/root/proxy.env` — token, ACME certs in `/certs/`, geolocation
 DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.
 > **Dependency note:** NetBird is an **access layer only** — it exists so a
 > human on a PC or away from the LAN can reach services by URL. It is **not**
 > a dependency of any service backend. Per Section 7, every config and agent
 > must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for
 > human browsers.
 #### Known Issues
 **A. Access log bloat (CRITICAL)** — 2026-07-03 discovery. The management server
 writes an `access_log_entries` row for every proxied request (~45K/day). SQLite
 has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:
 - DB writes slow → gRPC deadlines exceeded → proxy loses management connection
 - On proxy restart, management server too slow to push route mappings
 - Proxy starts HTTPS listener before routes arrive → all domains "unknown"
 - **Fix:** batch-delete old entries, VACUUM. Prevention: weekly systemd timer (`netbird-cleanup.timer`) deletes entries >7 days old when DB exceeds 500MB.
 - **Recurrence signal:** proxy logs show `unknown domain "git.sysloggh.net"` or `match: false` for known domains → management route sync failed.
 **B. Stale management state after proxy restart.** Management server sometimes
 stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts,
 mapping stream established, but no routes arrive → all domains `unknown`.
 **Fix:** restart management server FIRST, THEN proxy. Order matters.
 **C. Peer `sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE=` flapping.**
 One peer (connecting through Traefik relay at 172.30.0.10) disconnects every
 ~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to
 slow DB writes) rather than a network issue. Monitor after bloat fix.
 #### Recovery Procedures
 1. **Proxy losing all routes ("unknown domain" for all services):**
   ```bash
   ssh root@72.61.0.17
   cd /root
   docker compose restart netbird-server
   sleep 12
   docker compose restart proxy
   ```
 2. **Access log bloat detected (DB > 500MB):**
   ```bash
   ssh root@72.61.0.17
   /root/cleanup-access-logs.sh
   ```
 3. **VPS hung (TCP accepts, SSH banner timeout):** hard reboot from Hostinger console.
 ### Checks (every 60s)
@@ -173,6 +294,13 @@ For each Docker host:
  - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
  - df -h / | awk '{print $5}' → usage < 80%
 For Netbird VPS specifically:
  - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
  - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
  - If DB > 500MB: alert + run /root/cleanup-access-logs.sh
  - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
  - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart
 For docker-vm specifically:
  - mountpoint -q /media/storage → NFS mounted
  - mountpoint -q /media/mediastore → NFS mounted
@@ -226,25 +354,105 @@ For docker-vm specifically:
 ## Section 5: Network Services — Monitoring
-| Service | Domain | Status |
+### 5.1 Service Inventory
 |---------|--------|--------|
 | Proxmox API | minipve.sysloggh.net:443 | ✅ |
 | Authentik | auth.sysloggh.net:443 | ✅ |
 | Gitea | git.sysloggh.net:443 | ✅ |
 | Zulip | chat.sysloggh.net:443 | ✅ |
 | LiteLLM | litellm.sysloggh.net:443 | ✅ |
 | Pulse | pulse.sysloggh.net:443 | ✅ |
 | SearXNG | searxng.sysloggh.net:8888 | ✅ |
 | Firecrawl | firecrawl.sysloggh.net:3002 | ✅ |
-### Checks (every 2 min)
+The `Resolves To` column is the load-bearing one. Services that CNAME to
 `netbird.sysloggh.net` (72.61.0.17) are routed through the NetBird reverse
 proxy and **fail whenever NetBird blips**, even though their LAN backend is
 fine. Services that resolve directly to a LAN IP are NetBird-independent.
 | Service | Domain | LAN Backend | Resolves To | NetBird-dep | Status |
 |---------|--------|-------------|-------------|-------------|--------|
 | Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
 | LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
 | Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
 | Gitea | git.sysloggh.net:443 | 192.168.68.17:3000 | CNAME → netbird | **Yes** | ⚠️ |
 | Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
 | Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
 | DNS UI | dns.sysloggh.net:443 | 192.168.68.10:80 | CNAME → netbird | **Yes** | ⚠️ |
 | SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
 | Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
 **Verified 2026-07-24:** NetBird VPS rebooted after a hang; all CNAME'd
 services recovered. LAN-IP-direct paths stayed up throughout the outage.
 Also added `dns.sysloggh.net` route (was missing entirely).
 See `scripts/netbird-add-domain.sh` for adding new proxy routes.
 ### 5.2 Checks (every 2 min)
 ```
 ## Maintains
- dns-resolution: { services: [{name, resolves}] }
+- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
 - ssl-expiry: { services: [{name, days_remaining}] }
 - endpoint-reachability: { services: [{name, http_code}] }
 - netbird-dependency: { services: [{name, depends_on_netbird: bool}] }
 ```
 ### 5.3 Network Verification & Routing (every 5 min)
 This block is what catches a NetBird blip before it becomes an outage. It
 proves two independent paths for every service: the **URL path** (what a
 browser uses, may cross NetBird) and the **LAN-IP path** (what configs and
 agents must use, never crosses NetBird).
 ```
 ## Maintains
 - dual-path-reachability: {
    services: [{
      name: string,
      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
      lan_path:  { ip, port, http_code },        # the IP:port path
      lan_independent: bool                       # lan_path works when url_path fails
    }]
  }
 - netbird-ingress-health: {
    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
    containers_up: int,        # all 5 netbird-* containers running
    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
  }
 - routing-regression: {
    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
                                    # → alert: a service just became NetBird-dependent
    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
                                    # URL where a LAN IP is required (see Section 7)
  }
 ## Checks
 - For each service in 5.1:
  - curl https://<domain>/ → record url_path (code, tls_ok)
  - curl http://<lan_ip>:<port>/ → record lan_path
  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
  - If BOTH fail → service-down (real outage)
 - NetBird ingress (72.61.0.17):
  - TCP probe 22, 80, 443 → all accept
  - SSH banner exchange completes within 10s (catches the hung-box symptom
    where TCP accepts but sshd never sends its banner)
  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)
 - DNS dependency drift:
  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)
 ## Remediations
 - NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
  → Do NOT declare backend services down; their LAN-IP paths are still up.
 - Single proxy route missing (e.g. git returns 000 but chat ok):
  → ssh root@72.61.0.17 'docker restart netbird-traefik'
  → re-check after 30s
 - New NetBird CNAME detected:
  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
 - Config URL violation detected:
  → Alert with file + line; do not auto-edit configs
 ```
 ## Section 6: Alert Routing & Escalation
@@ -281,6 +489,63 @@ ZFS pool degraded
  → Cannot auto-fix → Escalate immediately
 ```
 ## Section 7: Configuration Doctrine — IP-First, URLs for Browsers
 This doctrine is the structural fix for the NetBird-dependency outage. It is
 enforced by the `routing-regression.config_url_violations` check in Section
 5.3.
 ### Rule
 1. **Service configs use LAN IP addresses, always.** Any file that wires a
   service, agent, monitor, or integration to another internal service must
   reference the backend by its `192.168.68.x` LAN IP and port — never by a
   `*.sysloggh.net` URL. NetBird may blip at any time; LAN IPs do not.
 2. **`*.sysloggh.net` URLs are reserved for human browser access.** They are
   a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy
   off the LAN) for a person at a keyboard. They must never be a load-bearing
   dependency in code or config.
 3. **Agents and monitors reach backends by LAN IP.** Abiba, Tanko, and Mumuni
   all run on the LAN; there is no reason for them to hairpin through NetBird.
   The daily infra report and all `ssh`/`curl` checks already follow this.
 4. **DNS records should prefer LAN IPs over NetBird CNAMEs.** A service that
   can resolve directly to its LAN IP (like `litellm` → `192.168.68.116` and
   `minipve` → `192.168.68.12`) is NetBird-independent. Moving a record from
   a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
 5. **NetBird is for remote/PC access only.** When you are out or on a PC,
   NetBird carries you in. When you are on the LAN, NetBird is not in the
   path. Configs must reflect this — they always assume LAN.
 ### Canonical IP map (configs MUST use these)
 | Service | Use in configs | URL (browsers only) |
 |---------|----------------|--------------------|
 | Proxmox API | `https://192.168.68.12:8006` | `https://minipve.sysloggh.net` |
 | LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
 | LiteLLM (nginx) | `http://192.168.68.116` | — |
 | Grafana | `http://192.168.68.116:3001` | — |
 | Authentik | `https://192.168.68.11:9000` | `https://auth.sysloggh.net` |
 | Gitea | `http://192.168.68.17:3000` | `https://git.sysloggh.net` |
 | Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
 | SearXNG | `http://192.168.68.7:8888` | — |
 | Firecrawl | `http://192.168.68.7:3002` | — |
 | RA-H OS bridge | `http://192.168.68.65:3100` | — |
 ### Violation examples (to be flagged by Section 5.3)
 - ❌ An agent config setting `LITELLM_BASE_URL=https://litellm.sysloggh.net`
 - ✅ Same config setting `LITELLM_BASE_URL=http://192.168.68.116:4000`
 - ❌ A monitor curling `https://chat.sysloggh.net/api/v1/server_settings`
 - ✅ A monitor curling `http://192.168.68.19/api/v1/server_settings` (and
      optionally the URL as a separate browser-path check)
 ### Goal state
 Every `*.sysloggh.net` record resolves to a LAN IP (split-horizon DNS on LAN,
 NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN
 ingress — if it blips, only remote browser users notice, and no agent,
 monitor, or integration breaks.
 ## Appendix A: Quick Health Commands
 ```bash
@@ -293,33 +558,57 @@ curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"
 ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
 ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"
 # GPU fleet quick check
 curl -s http://192.168.68.116/health/unified | jq .status
 curl -s http://192.168.68.24:9100/gpu-data | jq .summary
 # Grafana status
 curl -s http://admin:$(infisical secrets get GRAFANA_ADMIN_PASSWORD --project=infrastructure --env=production --plain)@192.168.68.116:3001/api/health
 # Prometheus targets
 curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
 # LiteLLM key check
 curl -s -H "Authorization: Bearer $(infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production --plain)" \
  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
 # Storage check
 ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 # Router roster reload (if needed)
 curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
  -H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"
 # Restart stuck GPU (saturation watchdog alternative)
 ssh root@192.168.68.8 "systemctl restart llama-server"
 ssh root@192.168.68.110 "systemctl restart llama-server"
 ```
 ## Appendix B: CT Inventory
 | CT | Name | Node | IP | Role | Agent |
 |----|------|------|----|------|-------|
-| 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
+| CT | Name | Node | IP | Role | Agent |
-| 101 | llm-gpu | acerpve | — | GPU VM | ❌ |
+|----|------|------|----|------|-------|
-| 102 | adguard | acerpve | — | DNS | ❌ |
+| 100 | abiba | **hwepve** | .24 | Pi agent | ✅ pi |
-| 103 | ocu-llm | ocupve | — | GPU VLM | ❌ |
+| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
 | 102 | adguard | **minipve** | **.10** | DNS | ❌ |
 | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
 | 104 | authentik | minipve | .11 | OIDC | ❌ |
-| 105 | kagentz | amdpve | — | Agent Zero | ✅ |
+| 105 | kagentz | **hwepve** | — | Agent Zero | ✅ |
 | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
 | 107 | pbs | storepve | — | Backups | ❌ |
 | 108 | media | storepve | — | Media | ❌ |
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
-| 110 | gitea | minipve | — | Git | ❌ |
+| 110 | gitea | minipve | **.17** | Git | ❌ |
-| 111 | tdunna | amdpve | — | ? | ❌ |
+| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy | amdpve | — | ? | ❌ |
+| 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
-| 114 | mumuni | minipve | — | Hermes agent | ✅ |
+| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
-| 115 | scottdenya | amdpve | — | ? | ❌ |
+| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
-| 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ |
+| 117 | zulip | storepve | .19 | Chat | ❌ |
-| 117 | zulip | storepve | — | Chat | ❌ |
+| 118 | jdownloader | storepve | — | JDownloader container | ❌ |
-| 118 | jitsi | minipve | — | Video | ❌ |
+| 119 | infisical-vault | minipve | — | Vault | ❌ |
 ## Appendix C: Docker Compose Files Location
@@ -329,5 +618,70 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` |
 | docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` |
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
-| CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` |
+| CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |
 ## CT Access (pct-run — no IPs needed)
 All CTs are accessible via `pct-run <CT_ID> <command>`. No IP addresses required.
 Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.sh`.
 | CT | Name | Node | pct-run |
 |-----|------|------|---------|
 | 100 | abiba | hwepve | `pct-run 100` |
 | 105 | kagentz | **hwepve** | `pct-run 105` |
 | 111 | tdunna | amdpve | `pct-run 111` |
 | 112 | tanko | amdpve | `pct-run 112` |
 | 113 | baggy | amdpve | `pct-run 113` |
 | 115 | scottdenya | amdpve | `pct-run 115` |
 | 104 | authentik | minipve | `pct-run 104` |
 | 110 | gitea | minipve | `pct-run 110` |
 | 116 | syslog-api | minipve | `pct-run 116` |
 | 106 | ra-h-os | storepve | `pct-run 106` |
 | 107 | proxmox-backup | storepve | `pct-run 107` |
 | 108 | media | storepve | `pct-run 108` |
 | 117 | zulip | storepve | `pct-run 117` |
 | 102 | adguard | **minipve** | `pct-run 102` |
 GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
 ```bash
 ssh root@192.168.68.8   # RTX 3090
 ssh root@192.168.68.110 # RTX 5070
 ssh root@192.168.68.15  # Strix Halo
 ssh root@192.168.68.4   # hwepve (abiba, kagentz) — Mumuni runs inside CT 100
 ```
 ## Section 7: Agent Health Check v2 (2026-07-26)
 Single non-disruptive check running every 10 minutes via cron
 (`/root/scripts/agent-health-check.py`). v2 fixes critical gaps:
 - Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
 - Agent-specific vault keys (`{NAME}_LITELLM_API_KEY`) not shared master key
 - CT liveness check via `pct status` on PVE nodes
 - Config YAML integrity check
 - Wrapper/CLI integrity check
 - Vault secret non-emptiness check
 The script is **read-only** — it never restarts, kills, or modifies anything.
 ### Checks Performed
 | Check | Frequency | What It Detects |
 |-------|-----------|-----------------|
 | LiteLLM key validation | 10 min | All 5 agent-specific keys authenticate (not shared master key) |
 | GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
 | Gateway liveness | 10 min | Gateway process running, state file readable (all 5 agents) |
 | Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
 | Recent errors | 10 min | Error count in journald for last 10 min |
 | CT liveness | 10 min | `pct status` on PVE nodes — catches stopped CTs |
 | Config YAML integrity | 10 min | Python `yaml.safe_load()` — catches syntax errors |
 | Wrapper/CLI integrity | 10 min | hermes wrapper exists, infisical path correct, hermes-real reachable |
 | Vault secrets | 10 min | Agent-specific vault keys are non-empty and start with `sk-` |
 ### Disabled Scripts
 | Script | Why Disabled |
 |--------|-------------|
 | `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
 | `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |
@@ -0,0 +1,200 @@
 ---
 kind: responsibility
 name: infrastructure-maintenance
 description: >
  Weekly system-level maintenance for the Syslog inference fleet: OS package
  updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
  running containers, container restarts with health verification, post-update
  verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
  PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
  raw shell scripts that previously did this piecemeal. This contract owns the
  HOST-LEVEL weekly maintenance loop on the primary host plus Docker image
  pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET
  cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker
  image Wave 3 across all stacks). Runs Sunday 2am ET. Owner:
  ops (firstmate secondmate). Blast radius: an unverified image pull can break
  LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
  upgrade can leave the host in a half-upgraded state. Pre-update backup check
  and rollback are mandatory for this reason.
 agent: ops
 triggers:
  - weekly (Sunday 02:00 ET) via cron
  - on demand when ops/abiba triggers "infra maintenance"
 version: 1.0.0
 ---
 ## Maintains
 - maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
 - image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
 - apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
 - health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
 - rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
 - maintenance-history: array of past runs with phase results and any escalations
 ## Scope
 Primary host is the maintenance host where apt updates apply. Docker image pulls
 span the two Docker ecosystems that run critical services. infrastructure-update
 runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across
 all stacks/hosts); this contract runs a narrower host-level weekly pull limited
 to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against
 `infrastructure-control.prose.md` (the source of truth) and the live system
 before mutating.
 | Host | IP | Role | Trust |
 |------|----|------|-------|
 | CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
 | VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
 | CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
 | Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
 | CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
 > "Primary host" for the apt phase is the host the ops agent runs maintenance
 > from. Confirm which host that is against infrastructure-control before
 > running; do not assume. If the ops agent is containerized/CT-based, apt runs
 > inside that CT.
 ## Requires
 - SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
 - `apt`, `docker`, `docker compose` available on target hosts
 - LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
 - `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
 - Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
 - Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
 ## Continuity
 - Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
 - Also wakes on: explicit "infra maintenance" trigger from ops/abiba
 - Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
 ## Execution
 ### Phase 0 — Preflight (snapshot/backup check + health baseline)
 1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
 2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
   - `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
   - `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
 3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
 4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
 ### Phase 1 — OS package updates (primary host)
 ```bash
 # On the primary host only (VERIFY host against infrastructure-control first)
 apt update
 apt upgrade -y
 ```
 - Capture `apt list --upgradable` before and after → store in `apt-state`.
 - If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
 - If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
 ### Phase 2 — Docker image pulls
 Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
 ```bash
 # CT 116 (.116) — inference harness
 cd /opt/inference-harness && docker compose pull
 # VM 109 (.7) — search + home stacks
 cd /opt/search-stack/searxng && docker compose pull
 cd /opt/search-stack/firecrawl-source && docker compose pull
 # any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
 ```
 - LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
 - Record pulled tag + digest per service in `image-baseline`.
 ### Phase 3 — Container restarts with health verification
 Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
 ```bash
 # CT 116
 cd /opt/inference-harness && docker compose up -d
 # VM 109
 cd /opt/search-stack/searxng && docker compose up -d
 cd /opt/search-stack/firecrawl-source && docker compose up -d
 ```
 After each stack comes up, wait for health (max 120s):
 - `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
 - Service-specific probe passes (see Phase 4 probes)
 If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
 ### Phase 4 — Post-update service verification
 After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
 | Service | Probe | Expect |
 |---------|-------|--------|
 | LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
 | LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
 | SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
 | Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
 | Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
 | PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
 | Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
 Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
 ## Rollback Protocol
 If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
 1. **Image rollback** — for the failing stack, restore the previous image:
   ```bash
   # Restore from recorded image-baseline digest
   docker compose down
   # Pin the service image to the recorded digest in compose, then recreate
   #   image: <name>@sha256:<previous_digest>
   docker compose pull && docker compose up -d
   ```
 2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
 3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
 4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
 5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
 ## Circuit Breaker
 - `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
 - `window: 7200` seconds — no more than 2 retries within a 2-hour window.
 - `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
 ## Report
 After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
 ```
 🛠 Infrastructure Maintenance — YYYY-MM-DD
 Phase: apt | images | restarts | verify | rollback
 Primary host: <host>
 APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
 Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
 Services: all GREEN | <service> FAILED (rolled back)
 Baseline regression: none | <details>
 Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
 Escalation: none | warning | critical | fatal
 ```
 ## Verification Postconditions
 - All critical services running after update (Phase 4 all GREEN)
 - No regressions from pre-update health baseline (Phase 0 baseline)
 - Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
 - APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
 ## Related Contracts
 - `infrastructure-update.prose.md` — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
 - `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
 - `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
 - `litellm-health.prose.md` — LiteLLM probe details.
 - `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.
@@ -0,0 +1,158 @@
 ---
 kind: function
 name: infrastructure-monitoring
 description: >
  Deploys Prometheus + GPU exporters + Grafana to monitor the entire
  inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
  via existing /metrics Prometheus endpoint.
  DEPLOYMENT STATUS (2026-07-09):
  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
     NVIDIA sidecar exporters (.8/.110:9400) never installed.
     Router falls back to direct GPU /health probes.
  ⚠️ This contract is target-state aspirational — not as-built.
     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
 ## Architecture
 ```
 GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
  nvidia-exporter       nvidia-exporter       amdgpu-exporter
  :9400                 :9400                 :9400
       │                     │                     │
       └─────────────────────┼─────────────────────┘
                             ▼
              ┌──────────────────────────┐
              │       Prometheus         │
              │       CT 116 :9090       │
              │                          │
              │  Scrape targets:         │
              │  • 192.168.68.8:9400     │
              │  • 192.168.68.110:9400   │
              │  • 192.168.68.15:9400    │
              │  • litellm:4000/metrics  │
              └──────────┬───────────────┘
                         │
              ┌──────────▼───────────────┐
              │        Grafana           │
              │       CT 116 :3001       │
              │                          │
              │  Preloaded dashboards:   │
              │  • GPU Fleet Overview    │
              │  • LiteLLM Proxy Stats   │
              └──────────────────────────┘
 ```
 ## Components
 ### 1. NVIDIA GPU Exporter (hosts: .8, .110)
 - Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
 - Listens on `:9400`, exposes `/metrics` in Prometheus format
 - Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
 ### 2. AMD GPU Exporter (host: .15)
 - Custom exporter: Python script wrapping `amdgpu_top --json`
 - Listens on `:9400`, exposes `/metrics` in Prometheus format
 - Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
 - Runs as systemd service for persistence
 ### 3. Prometheus (CT 116)
 - Container: `prom/prometheus:latest`
 - Port: `9090` (internal Docker network)
 - Scrape interval: 15s
 - Config: `/opt/monitoring/prometheus.yml`
 - Storage: Docker volume `prometheus-data`
 ### 4. Grafana (CT 116)
 - Container: `grafana/grafana:latest`
 - Port: `3001` (mapped to host)
 - Data source: Prometheus at `http://prometheus:9090`
 - Provisioned dashboards for GPU fleet + LiteLLM
 - Accessible at `http://192.168.68.116:3001`
 ## Parameters
 - gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
 - gpu_amd_hosts: ["192.168.68.15"]
 - monitoring_host: "192.168.68.116"
 - prometheus_port: 9090
 - grafana_port: 3001
 - gpu_exporter_port: 9400
 ## Requires
 - SSH access to all GPU hosts for exporter deployment
 - Docker on CT 116 for Prometheus + Grafana containers
 - Python 3 on AMD host for custom exporter
 - nvidia-smi on NVIDIA hosts
 ## Maintains
 - All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
 - Prometheus scrapes all targets every 15s
 - Grafana dashboards show real-time GPU utilization, temp, VRAM, power
 - LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
 - Stack persists across reboots (systemd for exporters, Docker restart policy)
 ## Execution
 ### Phase 1: GPU Exporters
 **NVIDIA (.8 and .110)**:
 1. Download `nvidia_gpu_exporter` binary
 2. Create systemd service `nvidia-gpu-exporter.service`
 3. Start and enable
 **AMD (.15)**:
 1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
 2. Parses `amdgpu_top --json -d 1000` output
 3. Exposes key metrics at `:9400/metrics` via Python http.server
 4. Create systemd service
 5. Start and enable
 ### Phase 2: Prometheus
 1. Create `/opt/monitoring/` directory on CT 116
 2. Write `prometheus.yml` with scrape configs for all targets
 3. Add to docker-compose (or separate compose file)
 4. Start container
 ### Phase 3: Grafana
 1. Create `/opt/monitoring/grafana/` directories
 2. Provision Prometheus datasource
 3. Provision GPU fleet dashboard JSON
 4. Provision LiteLLM dashboard JSON
 5. Add to docker-compose
 6. Start container
 ### Phase 4: Verification
 1. Verify all 3 GPU exporters return 200 at :9400/metrics
 2. Verify Prometheus targets all UP at :9090/targets
 3. Verify Grafana accessible at :3001 with dashboards
 4. Verify LiteLLM metrics flowing to Prometheus
 5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
 ## Verification Commands
 ```bash
 # GPU exporters
 curl -s http://192.168.68.8:9400/metrics | grep nvidia
 curl -s http://192.168.68.110:9400/metrics | grep nvidia
 curl -s http://192.168.68.15:9400/metrics | grep amdgpu
 # Prometheus
 curl -s http://192.168.68.116:9090/api/v1/targets
 # Grafana
 curl -s http://192.168.68.116:3001/api/health
 # LiteLLM metrics (already live)
 curl -s http://192.168.68.116:4001/metrics | head -20
 ```
@@ -0,0 +1,242 @@
 ---
 kind: responsibility
 name: infrastructure-update
 description: >
  Autonomous system-wide update contract covering all 5 Proxmox nodes,
  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
 agent: abiba
 triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
 version: 1.2.0
 ---
 ## Maintains
 - update-status: { phase, node, action, result, timestamp }
 - update-history: array of past update runs with results
 - security-state: { cve_count, last_patched, pending_updates }
 ## Pre-Flight Checklist
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
 2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
 3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
 4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
 5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
 6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
 7. ✅ Zulip server reachable
 8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 9. ✅ Disk >20% free on all nodes
 10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
 ## Wave 1: Storage & Infra Nodes (lowest impact)
 | Target | Type | Command | Timeout |
 |--------|------|---------|---------|
 | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
 | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
 | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
 **Verify after Wave 1:**
 - Docker healthy on .7: `docker ps`
 - RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
 - Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`
 ## Wave 2: Compute & Agent Nodes
 | Target | Type | Command | Timeout |
 |--------|------|---------|---------|
 | amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
 | CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
 | VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
 | VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
 **Verify after Wave 2:**
 - All VMs/CTs running: check via Proxmox API
 - LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
 - LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools
 - GPU servers responding: check :8080 on VM 101, VM 103; check strix-moe via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`
 ## Wave 3: Docker Image Updates
 | Host | Stack | Command |
 |------|-------|---------|
 | VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
 | VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
 | VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
 | VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
 | CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
 | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
 - End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
 - MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
 - Zulip test: send test message to #agent-hub
 - Dashboard loading: `curl localhost:3001/` (via CT 116)
 - Firecrawl test: `curl :3002/`
 - SearXNG test: `curl :8888`
 ## Wave 4: Proxmox Kernel Reboot
 Only if `[ -f /var/run/reboot-required ]` on any node.
 | Target | Action |
 |--------|--------|
 | Affected PVE node | Verify all CTs/VMs migrated or stopped |
 | | `reboot` via PVE API (or `systemctl reboot -f` if dbus fails) |
 | | Wait 120s for node to come back |
 | | Start any stopped CTs |
 ### Post-reboot sweep (known gaps)
 After every node reboot, run these checks:
 1. **CT auto-start sweep** — LXC containers sometimes don't start despite
   `onboot: 1`. Check every CT on the rebooted node and start any left stopped:
   ```bash
   pct list | awk '/stopped/{print $1}' | xargs -I{} pct start {}
   ```
   Known cases: scottdenya (CT 115 on amdpve), authentik (CT 104 on minipve).
 2. **Zulip recovery** — When docker-vm or storepve reboots, the Zulip main
   container loses its Docker network assignment (SIGKILL during storage
   outage detaches it from `zulip_default` network). Run:
   ```bash
   ssh root@192.168.68.19 'docker rm -f zulip-zulip-1 && cd /opt/zulip && docker compose up -d'
   ```
   The compose restart recreates the container on the correct network.
 3. **docker-vm Docker daemon** — After reboot, Docker can take 3-4 minutes
   to become `active`. The docker-proxy for Pulse (port 7655) starts early,
   so Pulse is accessible before `docker ps` reports ready. Wait for Docker
   before checking other stacks.
 ### VPS ↔ docker-vm tunnel
 After any VPS or docker-vm reboot, verify the dedicated WireGuard tunnel:
 ```bash
 ssh root@72.61.0.17 'wg show wg1' | grep "latest handshake"
 # If no handshake in >60s:
 ssh root@72.61.0.17 'wg-quick up wg1'
 ```
 The tunnel uses PersistentKeepalive=25 and is systemd-enabled, but should
 be verified after a reboot.
 ## Rollback Protocol
 If ANY verification fails:
 1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
 2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
 3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
 4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails
 ## Config Backup
 Before Wave 1, snapshot these files:
 ```
 /opt/inference-harness/docker-compose.yml               (CT 116 .116) ⚡ contains MCP_SERVER env vars
 /opt/inference-harness/litellm_config.yaml              (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
 /opt/monitoring/prometheus.yml                          (CT 116 .116)
 /etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
 /opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
 /opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
 /opt/home_stack/docker-compose.yml                      (VM 109 .7)
 /opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
 /root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
 /etc/systemd/system/ornith-server.service               (amdpve .15 — strix-moe)
 /etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
 # Hermes agent configs (key enforcement — 2026-07-10)
 /root/.hermes/config.yaml                               (Mumuni CT 114, Tanko CT 112, etc.)
 /root/.config/systemd/user/hermes-gateway.service       (Mumuni CT 114 — EnvironmentFile fixed)
 /etc/environment                                        (Mumuni CT 114 — LITELLM_API_KEY)
 ```
 ## MCP Gateway (2026-07-10)
 LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.
 ### Configuration
 **litellm_config.yaml** (`/opt/inference-harness/litellm_config.yaml`):
 ```yaml
 mcp_servers:
  ra_h_os:
    url: "http://192.168.68.65:3100/mcp"
    transport: "http"
    auth_type: "none"
 ```
 **docker-compose.yml** env vars:
 ```yaml
 - MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
 - MCP_SERVER_RAHOS_TRANSPORT=http
 ```
 ### Access
 | Key | MCP Access |
 |-----|-----------|
 | Master key | ✅ Full — 90 tools (vault-injected) |
 | Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |
 ### Known Limitations
 - Per-key MCP server grants not functional — only master key has access
 - Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
 - HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
 - `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
 ### Migration Path
 When LiteLLM is upgraded to a version supporting per-key MCP grants:
 1. Grant agent keys `mcp_servers: ["ra_h_os"]`
 2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
 3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
 ## Security-Specific Updates
 | Check | Command | Action |
 |-------|---------|--------|
 | CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
 | Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
 | Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
 | SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |
 ## Success Criteria
 - [ ] All 5 PVE nodes updated, no reboot-loop
 - [ ] All VMs/CTs running post-update
 - [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
 - [ ] LiteLLM MCP gateway healthy (90 tools via master key)
 - [ ] Zero security CVEs remaining
 - [ ] <10 min total downtime per service
 ## Report
 After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD
 Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>
 Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
 ```
@@ -0,0 +1,285 @@
 ---
 kind: function
 name: litellm-api-keys
 description: >
  Manages LiteLLM API keys for agent identity. Creates named keys so each agent
  is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry)
  and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026").
  Ensures agents never use the master key directly. Rotation is event-driven,
  not calendar-driven — rotate only on compromise, personnel change, or
  periodic security hygiene (quarterly/annually).
  UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
  BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
  expire/404. The .env fallback prevents agents from running without keys.
  Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
  UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
  token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
  infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
  Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
  Abiba's key is now a proper agent key (NOT the master key — stale note removed).
  UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
  standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
  infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
  injection → .env fallback → exec python. Systemd drop-ins are IMMUNE to hermes gateway
  install which overwrites the unit file ExecStart. Infisical CLI updated to 0.43.109 on
  all agents (was 0.38.0). Service token st.8e848433 shared across fleet (st.353699cd
  for tanko was deleted). .env fallback on every agent protects against token loss.
  Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
  — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
  (3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
  on CT 116. Last verified: 2026-07-17.
 ---
 ## Parameters
 - agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
 - action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
 - litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
 - master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
 - vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
 - vault_project: string — Infisical project slug (default: "infrastructure")
 - vault_env: string — Infisical environment (default: "production")
 - agent_host: string — Agent's IP for SSH (default: resolved from infra)
 - agent_user: string — SSH user (default: "jerome")
 ## Returns
 - action: string — What was done
 - key_alias: string — The LiteLLM key alias created/rotated
 - key_prefix: string — First 10 chars of the new key (for identification)
 - previous_key_alias: string | null — Previous key alias if rotating
 - litellm_response: object — Raw response from LiteLLM /key/generate
 - vault_updated: boolean — Whether Infisical vault secret was updated
 - agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
 - verification: { status: string, detail: string } — Final health check
 ## Execution
 1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
 2. **Check existing keys** — List all keys, find any with agent_name alias
 3. **If action == "list"**: Return all keys with their aliases and spend
 4. **If action == "create"**:
   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   - Duration is null (permanent) — inherited from litellm default_key_generate_params
   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
   - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
   - Return the new key
 5. **If action == "rotate"**:
   - Generate new key with same alias (LiteLLM replaces the old key)
   - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
   - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
     The gateway automatically picks up the new key via `infisical run --` wrapper
   - Verify: curl test against /v1/models with new key
   - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
   - Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
 6. **If action == "verify"**:
   - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
   - Test the key against LiteLLM /v1/models
   - Confirm key alias matches agent_name in LiteLLM key list
   - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
 ## Production Vault Access Process (canonical, 2026-07-17)
 The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents
 (Mumuni, Tanko, Koby, Koonimo) as of 2026-07-17. Abiba (pi) uses a similar pattern
 through its agent wrapper.
 ### The canonical pattern
 1. **infisical CLI** installed on the host at `/usr/bin/infisical` (v0.43.109+, from
   artifacts-cli.infisical.com apt repo). Update procedure:
   ```bash
   curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | sudo -E bash
   sudo apt-get update && sudo apt-get install -y infisical
   # Remove stale old binary if present
   rm -f /usr/local/bin/infisical /bin/infisical
   ```
   Wrappers use absolute path `/usr/bin/infisical run`. Never rely on PATH resolution.
 2. **Service token** (Infisical Machine Identity, `st.…`) stored at `~/.infisical-token`
   (`chmod 600`). Current: shared `st.8e848433…` (abiba, READ+WRITE on agents project).
   Tanko's `st.353699cd…` (tanko-agent) was deleted — reverted to shared token.
   Proper: one machine identity per agent (create in Infisical UI → Project Settings →
   Machine Identities).
 3. **`infisical-gateway.sh` wrapper** at `~/.hermes/infisical-gateway.sh` (`chmod 700`):
   ```bash
   #!/bin/bash
   export INFISICAL_API_URL="https://vault.sysloggh.net"
   TOKEN=$(cat $HOME/.infisical-token)
   LOG=$HOME/.hermes/logs/gateway.log; mkdir -p $HOME/.hermes/logs
   while true; do
     echo "[$(date -Iseconds)] Starting gateway with Infisical injection..." >> $LOG
     /usr/bin/infisical run --token="$TOKEN" \
       --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
         . $HOME/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3]
         export LITELLM_API_KEY="${<AGENT>_LITELLM_API_KEY}"
         export ZULIP_API_KEY="${<AGENT>_ZULIP_API_KEY}"
         export ZULIP_SITE="https://chat.sysloggh.net"
         export ZULIP_EMAIL="<agent>-bot@chat.sysloggh.net"
         export SEARXNG_URL="http://192.168.68.7:8888"
         # ⚠️  HARDCODE the full venv path. NEVER use $VENV inside single quotes.
         exec /root/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
       ' >> $LOG 2>&1
     EXIT_CODE=$?
     echo "[$(date -Iseconds)] Gateway exited with code $EXIT_CODE — restarting in 5s..." >> $LOG
     sleep 5
   done
   ```
   **CRITICAL: VENV PATH.** The inner `bash -c '...'` uses single quotes. Shell
   variables set in the outer wrapper are NOT expanded inside single quotes.
   `$VENV/bin/python` resolves to `/bin/python` (file not found). Always hardcode
   the absolute path to the venv python binary.
 4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` and `<AGENT>_ZULIP_API_KEY`.
   Vault = source of truth for ALL platform credentials.
 5. **`.env` fallback** at `~/.hermes/.env` (`chmod 600`) with agent-specific keys —
   safety net for vault outage or token revocation. Must be kept in sync on rotation.
   Example:
   ```bash
   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
   ```
 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
   ```ini
   [Service]
   ExecStart=
   ExecStart=/root/.hermes/infisical-gateway.sh
   ```
   The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
   then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
   regeneration** by `hermes gateway install` — the drop-in always wins.
   **Why a drop-in instead of editing the unit file:** `hermes gateway install`
   (called during Hermes updates and some self-heal operations) regenerates the
   systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
   Editing the unit file directly is futile — it will be overwritten. The drop-in
   approach explicitly resets ExecStart and sets the wrapper regardless of what the
   main unit file says.
 7. **NEVER hardcode** API keys in systemd drop-ins, config.yaml, or /etc/environment.
   The wrapper injects live from vault at every start.
 ### Why this is non-fail
 - **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
 - **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable or the service token is revoked.
 - **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
 - **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
 - **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
 - **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
 ### Migration status (2026-07-17)
 | Agent | Host | Pattern | Keys | Status |
 |-------|------|---------|------|--------|
 | abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
 | mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
 | tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
 | koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
 | koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
 > Tanko runs as user `jerome` — wrapper/token at `~/.hermes/infisical-gateway.sh` and
 > `~/.infisical-token`. Linger enabled (`loginctl enable-linger jerome`) for boot startup.
 ### Tanko migration (COMPLETED 2026-07-17)
 Tanko was the last agent migrated from hardcoded keys to vault wrapper.
 Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
 and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
 `50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
 `~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
 ### Koby migration lessons (2026-07-16, updated 2026-07-17)
 Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
 **Three mistakes made:**
 1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
 2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
 3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"`
   inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path.
 **How Koby actually connects:**
 - Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
 - Telegram: token from `.env` fallback. Allowed users: 6679773481.
 - Both platforms now connect through the wrapper's env injection.
 **Golden rules for gateway restarts:**
 1. Always `cat /proc/<pid>/environ` before killing the old process — captures the live env set.
 2. Hardcode venv python path in wrapper — never use variables inside single-quoted bash -c.
 3. Use systemd drop-ins (not unit file edits) to override ExecStart — survives Hermes updates.
 ### Fleet-wide standardization lessons (2026-07-17)
 After auditing all 4 agents, five systemic patterns caused repeated failures:
 1. **Three incompatible startup patterns** coexisted (systemd drop-in, direct python, orphaned wrapper)
 2. **Systemd unit files reverted** by `hermes gateway install` during updates
 3. **VENV variable scoping** broke wrappers on Koby and Mumuni (single-quote bash -c)
 4. **Service token expiry** — Tanko's `st.353699cd` was deleted from Infisical
 5. **No ZULIP_API_KEY** in env on Tanko — wrapper bypassed by systemd direct python
 All resolved by the canonical drop-in + while-true wrapper pattern documented above.
 ### Key rotation procedure (one vault operation with this standard)
 1. Generate new key: `POST /key/generate` (master key, admin).
 2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
 3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
 4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
 5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
 ## Machine Identity for Vault Writes (UPDATED 2026-07-17)
 **Current state:** Infisical CLI updated to v0.43.109 on all agents (from v0.38.0).
 The v0.38.0 bug (user-session auth fails for `secrets set`/`export`) is resolved.
 Service token `st.8e848433…` (abiba, READ+WRITE) can write to vault from CLI.
 **Proper fix — per-agent Machine Identities:**
 Create machine identities in Infisical UI → Project Settings → Machine Identities
 for each agent with READ-only scope on the `agents` project. Store client_id +
 client_secret per agent. Then vault writes use the shared abiba identity, and
 reads use per-agent identities. This eliminates the single shared token risk.
 **Service Token Inventory (2026-07-17):**
 | Token ID | Name | Permissions | Used By | Status |
 |----------|------|-------------|---------|--------|
 | `st.8e848433…` | tanko-gateway | READ+WRITE | Mumuni, Tanko, Koby, Koonimo, Abiba | ✅ Active |
 | `st.353699cd…` | tanko-agent | READ-only | — | ❌ Deleted from Infisical |
 **Per-agent .env fallback inventory (2026-07-17):**
 | Agent | .env Keys |
 |-------|-----------|
 | Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY |
 | Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
 | Koby | (wrapper injects from vault — .env has Telegram token) |
 | Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |
 ## Key Rotation Log
 | Date | Agent | Action | Notes |
 |------|-------|--------|-------|
 | 2026-07-17 | fleet | standardize | All 4 agents standardized on systemd drop-in + while-true wrapper + infisical v0.43.109. Removed conflicting zulip-env.conf + litellm-key.conf drop-ins. Added .env fallbacks with ZULIP keys. WAL #1322. |
 | 2026-07-17 | tanko | fix-zulip | Added ZULIP_API_KEY to env (was missing — systemd bypassed vault). Updated wrapper from exec to while-true. Created .env fallback. Removed hardcoded zulip-env.conf drop-in. WAL #1321. |
 | 2026-07-16 | vault | cleanup | 4 stale secrets deprecated. 5 personal creds flagged. |
 | 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault. Wrapper injects ZULIP_API_KEY + ZULIP_EMAIL. 3 platforms. |
 | 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml to infisical-gateway.sh + st.353699cd. NOTE: st.353699cd later deleted — reverted to st.8e848433 on 2026-07-17. |
 | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
 | 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
 | 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
 ## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
 - Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
 - Used for /key/generate, /key/delete, /key/list (GET), DB queries
 - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
 It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
 The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
 see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
 - LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`
@@ -1,17 +1,59 @@
 ---
 kind: function
-name: check-litellm-health
+name: litellm-health
 status: deprecated
 deprecated_on: 2026-07-09
 replaced_by: litellm-self-heal.prose.md
 note: >
  Consolidated into litellm-self-heal.prose.md to eliminate duplication
  of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within litellm-self-heal.
  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
-  OIDC auth endpoint, container status, and aggregate health on the
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
-  backend host. Designed as a reusable contract for any Syslog agent.
+  Router (harness-router :9000) is DEPRECATED — container still runs but
  is not in the request path. GPU monitoring via Prometheus/Grafana and
  fleet dashboard (gpu-monitor :9100).
  Designed as a reusable contract for any Syslog agent.
  Source of truth: gpu-fleet.prose.md
 ---
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 ```
 Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001
  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
 ```
 **What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
 - Router REMOVED from request path — LiteLLM proxies directly to GPU
 - All GPUs at parallel 2 (was parallel 1)
 - NVIDIA context reduced 256K→128K to free VRAM — now the stable ceiling across all GPUs (2026-07-17)
 - LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s  (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal)
 - nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
 ## Parameters
 - public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
+- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
 - auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
 - gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
 - gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
 - grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
 ## Returns
@@ -23,34 +65,77 @@ description: >
 ## Requires
 - SSH key access to backend_host for container checks
- Network access to public_url and auth_host
+- Network access to public_url, auth_host, and gpu_dashboard_url
- curl and openssl available on the execution host
+- LiteLLM master key for key management endpoints
-## Ensures
+## GPU Fleet Topology
- Each check returns a clear pass/fail status with detail message
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
- If any endpoint returns non-200, overall_status is "degraded"
+|------|-----|----------|---------------|--------|---------|----------|
- If backend host unreachable or >2 containers down, overall_status is "down"
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **128K** | 2 |
- If /health/unified reports any non-healthy components, status reflects it
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **128K** | 2 |
- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 128K | 2 |
 ## Model Fallback Chains (LiteLLM)
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
 | qwen3.6-27B-code | 300s | gemma-4-12b | 120s |
 | gemma-4-12b | 120s | qwen3.6-27B-code | 300s |
 | qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — |
 | syslog-auto (balanced) | 300s | qwen → gemma | — |
 > Global: request_timeout=300s, nginx proxy_read_timeout=600s
 ## Containers on CT 116
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
 | harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
 | harness-dashboard | inference-harness-dashboard | :3000 | /health |
 | harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
 | harness-prometheus | prom/prometheus | :9090 | /-/healthy |
 ## Execution
 1. **Read parameters** — Use provided values or defaults
 2. **Check public endpoints**:
   - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
   - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-   - GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
+
-   - GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
+3. **Check LiteLLM health (no-auth)**:
-3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
+   - GET http://{{backend_host}}/litellm/health/liveliness → expect 200
-   - GET http://{{backend_host}}/litellm/ui/ → expect 200
+
-   - GET http://{{backend_host}}/litellm/docs → expect 200
+4. **Check backend container health**:
-4. **Check aggregate health endpoint**:
+   - SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
-   - GET {{backend_host}}:9000/health/unified → expect 200, check all components
+   - Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
-5. **Check backend container health**:
+   - Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
-   - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
+
-   - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
+5. **Check router roster loaded**:
-6. **Check OIDC auth endpoint**:
+   - GET http://{{backend_host}}:9000/health → expect 200
-   - Verify auth.sysloggh.net resolves to {{auth_host}}
+   - GET http://{{backend_host}}:9000/health/unified → expect 3 models
-   - GET https://auth.sysloggh.net/ → expect login page
+   - If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
-7. **Compile and report** — Determine overall_status from individual check results
+
 6. **Check GPU fleet health** (via fleet dashboard):
   - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
   - Verify GPUs reporting status "healthy"
   - Check alerts array for active warnings/critical
 7. **Check model inference via LiteLLM** — Test each model:
   - POST /v1/chat/completions model=gemma-4-12b → expect 200
   - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
   - POST /v1/chat/completions model=strix-moe → expect 200
   - Use master key for auth
 8. **Check agent keys**:
   - GET /key/list with master key → verify all 6 agents have keys
 9. **Check Grafana**:
   - GET {{grafana_url}}/api/health → expect 200
 10. **Compile and report** — Determine overall_status from individual check results
@@ -1,22 +1,118 @@
 ---
 kind: responsibility
 name: litellm-self-heal
 status: deployed
 note: >
  DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
  now reimplemented as `litellm-health-check.sh` on CT 116.
  Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
  Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
  GPU monitoring integrated from gpu-monitor on .24:9100.
  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
  duplication of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within this contract.
  Source of truth for GPU topology and keys: gpu-fleet.prose.md
  Last verified: 2026-07-12
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
-  knowledge graph for full audit trail.
+  inference, and agent keys. Applies remediation rules for common failures.
  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 # LiteLLM Operations — Health Check + Self-Heal
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 ```
 Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001
  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
 ```
 ## Parameters
 - public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
 - backend_host: string — Internal CT host (default: "192.168.68.116")
 - auth_host: string — Authentik OIDC server (default: "192.168.68.11")
 - gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
 - gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
 - grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
 ## GPU Fleet Topology
 | Host | IP | Hardware | Models Served | Engine | Context | Parallel |
 |------|-----|----------|---------------|--------|---------|----------|
 | llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 131072 --parallel 2 --ngl 99`) | **128K** | 2 |
 | ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 131072 --parallel 2`, IQ4_NL + MTP draft) | **128K** | 2 |
 | amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 128K | 2 |
 > Verified on ground 2026-07-16 via `curl /v1/models` on each host + `llama-wrapper.sh`. The AMD host's underlying model is `qwen3.6-35B-udq4`; LiteLLM exposes it under two `model_name`s: `qwen3.6-35B-udq4` and `strix-moe` (rpm 40). The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced.
 ## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116)
 `model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`.
 - `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
 - `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
 - Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. As of 2026-07-16 the `baggy`/`koby`/`mumuni`/`abiba-pi` keys ALSO include `qwen3.6-35B-udq4`; `abiba-pi` additionally includes `deepseek-v4-pro` (cloud fallback). `kagenz0`/`koonimo`/`pi-agents-unified` have the standard 6 only. Agents should still use the stable alias `strix-moe` (not the raw `qwen3.6-35B-udq4`) so model swaps don't break them.
 - **NEVER use `litellm_proxy_master_key` (the `sk-litellm-...` master key) for inference.** It is for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated `monitor` agent key stored at `/etc/litellm-monitor.env` on CT 116 (root-only, `chmod 600`); `/key/list` is the only call that legitimately uses `$MASTER_KEY`.
 ## Model Fallback Chains (LiteLLM)
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
 | qwen3.6-27B-code | 300s | gemma-4-12b | 120s |
 | gemma-4-12b | 120s | qwen3.6-27B-code | 300s |
 | qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — |
 | syslog-auto (balanced) | 300s | qwen → gemma | — |
 > Global: request_timeout=300s, nginx proxy_read_timeout=600s
 ## Containers on CT 116
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
 | harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |
 | harness-dashboard | inference-harness-dashboard | :3000 | /health |
 | harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
 | harness-prometheus | prom/prometheus | :9090 | /-/healthy |
 | harness-docker-stats | python:3.12-alpine | — | container stats exporter |
 | harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
 ## Script Operations (synced 2026-07-16)
 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
 - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
 - **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
 ## Maintains
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
-## Requires
+- litellm-agent-keys: { count: int, valid: int }
 - litellm-health:function
 ## Continuity
@@ -24,47 +120,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
-## Reporting
+---
-Every remediation cycle produces a structured report in three channels:
+## Health Check
-### 1. RA-H OS Knowledge Graph Node
+Run this first on every cycle. Results feed into remediation rules below.
 Created as `[LEARN] litellm-self-heal: <run_id>` with:
 - metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
 - source: Full JSON report of the cycle
 - description: Summary of what was found, fixed, and escalated
-### 2. Zulip DM to Owner
+### 1. Check public endpoints
-Sent immediately for:
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
 - `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
 - Every 10th clean cycle — "✅ All Clear (10 checks passed)"
-### 3. Daily Digest (end of day)
+### 2. Check LiteLLM health (no-auth)
-A summary of the last 24 hours sent as a single Zulip DM:
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
 ```
 📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
-Total cycles: 288 (every 5 min)
+### 3. Check backend container health
-Issues found: 3
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+- Critical: harness-litellm, harness-nginx, harness-postgres
-  └─ Escalated: 0
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
 - Deprecated but running: harness-router (not in path, reference only)
-Top actions:
+### 4. Check GPU fleet health (via fleet dashboard)
-  • harness-nginx restarted — 2026-06-26 02:15
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
-  • DOCS_URL corrected — 2026-06-26 07:30  
+- Verify GPUs reporting status "healthy"
-  • harness-nginx restarted — 2026-06-26 14:46
+- Check alerts array for active warnings/critical
-Uptime: 23h 47m — All healthy now ✅
+### 5. Check model inference via LiteLLM — test each model
-```
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
 - POST /v1/chat/completions model=strix-moe → expect 200
 - Use master key for auth
-### 4. Weekly Digest (every Monday)
+### 6. Check agent keys
-Same format as daily but covering 7 days. Sent to both Zulip DM and
+- GET /key/list with master key → verify all 6 agents have keys
 logged as a `[REPORT]` knowledge graph node for long-term trending.
-### 5. Relay Message (if cross-agent)
+### 7. Check Grafana
-If a fix required another agent's help (e.g., Authentik restart), a relay
+- GET {{grafana_url}}/api/health → expect 200
-message is sent to the responsible agent with full context.
+
 ### 8. Compile overall status
 Determine overall_status from individual check results:
 - "healthy" — all checks pass
 - "degraded" — 1-2 non-critical checks fail
 - "down" — critical checks fail
 ---
 ## Remediation Rules
@@ -82,21 +180,99 @@ Detect → immediate escalate (external dependency)
 ### Rule 4: Docs 404
 Detect → fix DOCS_URL env var → verify → log
 ### Rule 5: GPU Unreachable
 Detect → SSH to GPU host fails or model not responding
 Fix → restart llama-server on host via SSH
 Escalate → immediately (downtime affects inference)
 ### Rule 6: Model Not Responding
 Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts
 ### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
 Router is no longer in the request path (2026-07-08). LiteLLM proxies
 directly to GPU. This rule is retained for reference but is inactive.
 If 503 errors occur, check LiteLLM timeouts and GPU health directly.
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
 Root cause → keys not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM
 ### Rule 9: Stale Active Counter in Redis — DEPRECATED
 Router no longer in path so Redis active counters are unused. Rule retained
 for reference but inactive. If Redis issues occur, check harness-redis container.
 ---
 ## Reporting
 Every remediation cycle produces a structured report:
 ### 1. RA-H OS Knowledge Graph Node
 Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
 ### 2. Zulip DM to Owner
 - `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
 - `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
 - Every 10th clean cycle — "✅ All Clear (10 checks passed)"
 ### 3. Daily Digest (end of day)
 Summary of last 24 hours: total cycles, issues found/fixed/escalated,
 top actions, uptime.
 ### 4. Relay Message (if cross-agent)
 If a fix requires another agent (e.g., Authentik restart), relay sent
 to responsible agent with full context.
 ---
 ## Execution
-1. Run health check
+```prose
-2. Apply remediation for each failure
+-- Phase 1: Health Check
-3. **Generate report** with all actions taken
+let health = call health-check
-4. **Log to knowledge graph** — create `[LEARN]` node
+  public_url: public_url
-5. **Notify user** via Zulip DM if anything changed
+  backend_host: backend_host
-6. Wait 300s and repeat
+  gpu_dashboard_url: gpu_dashboard_url
  grafana_url: grafana_url
 -- Phase 2: Apply remediation for each failure
 let actions = []
 for check in health.failed:
  let fix = apply-remediation-rule
    rule: lookup-rule(check.name)
    target: check.target
  push actions fix
 -- Phase 3: Generate report
 call report-generator
  health: health
  actions: actions
 -- Phase 4: Log to knowledge graph
 call kg-logger
  run_id: run_id
  health: health
  actions: actions
 -- Phase 5: Notify if anything changed
 if actions.length > 0:
  call zulip-notifier
    actions: actions
    health: health
 -- Wait 300s and repeat
 ```
 ## Audit Trail Format
 ```json
 {
-  "run_id": "self-heal-20260626-001",
+  "run_id": "self-heal-20260709-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "timestamp": "2026-07-09T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
@@ -110,7 +286,7 @@ Detect → fix DOCS_URL env var → verify → log
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
@@ -1,205 +1,346 @@
 ---
 kind: responsibility
 name: memory-audit-maintenance
-description: >
+kind: responsibility
-  Systematic audit and reorganization of an agent's native memory (MEMORY.md,
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
-  USER.md, SOUL.md). Categorizes entries, redistributes to correct homes,
+id: 067NC4KG01RG50R40M30E20918
  verifies live configs, and frees up char budget. Run when memory usage
  exceeds 85% or on explicit user request ("memory audit", "memory maintenance").
 ---
-## Quickstart — How to Run This Contract
+### Goal
-### For the Agent Running This Contract
+Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md, SOUL.md), categorize entries, redistribute them to correct locations, verify live configurations, and free character budget  all without user intervention unless a truly unresolvable ambiguity exists. Every agent runs this contract for **its own** memory files only  no cross-agent access, no shared state.
-Read this section first. It tells you exactly what to do.
+### Requires
-**When to run:**
+(none  this is a self-driven, autonomous node)
 - User says: `"memory audit"`, `"memory maintenance"`, `"clean up memory"`, `"memory is full"`
 - You detect that your `memory` store usage exceeds 85% in a write attempt
 - It's been 2+ weeks since the last audit
-**How to run (via OpenProse CLI):**
+### Scope
 ```bash
 # Default — 85% threshold, verify configs, compress entries
 prose run memory-audit-maintenance
-# Softer audit — only flag issues, don't rewrite files
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
 prose run memory-audit-maintenance compress_entries=false
-# Hard audit — stricter limits
+**Agent Roster:**
-prose run memory-audit-maintenance memory_threshold=90 max_memory_chars=800 max_user_chars=300
+- Mumuni
 - Tanko
 - Koby (CT 111 / tdunna)
 - Koonimo (CT 113 / baggy)
 **Isolation Principle:** Each agent has its own:
 - `MEMORY.md` and `USER.md`
 - `MEMORY_AUDIT_LEDGER.md` (audit trail)
 - `MEMORY_WRITERS.md` (writer registry)
 - Canary entries
 - No data crosses agent boundaries
 ### Maintains
 Memory health state  current utilization, last audit timestamp, drift alerts, and any unresolved ambiguities. Postcondition: no entry is duplicated, no session note or stale snapshot survives, no config is unverified, and the audit ledger is up-to-date.
 #### utilization
 Current character utilization of MEMORY.md and USER.md. Material: the percentages.
 #### last_audit
 Timestamp of the most recent successful audit. Material: the timestamp.
 #### unresolved
 Any entries that could not be autonomously categorized. Material: the entry text and the reason for ambiguity.
 #### drift_alerts
 Active alerts from drift detection (e.g., deletions accelerating, unverified configs growing). Material: the alert list.
 #### ledger
 Persistent audit trail stored at the agent's own `~/.hermes/memories/MEMORY_AUDIT_LEDGER.md`. Material: append-only, never modified or deleted. Each agent has its own ledger  no shared ledger across agents.
 ### Continuity
 - self-driven: re-audit every 14 days (2 weeks)
 - input-driven: triggered by explicit user request ("memory audit", "memory maintenance", "clean up memory", "memory is full")
 - threshold: trigger immediately if either store exceeds 85% utilization
 ### Parameters
 - `memory_threshold`: percentage that triggers an immediate audit (default: 85)
 - `max_memory_chars`: maximum allowed characters for MEMORY.md (default: 2200)
 - `max_user_chars`: maximum allowed characters for USER.md (default: 1375)
 - `compress_entries`: whether to rewrite entries for brevity (default: true)
 - `dry_run`: whether to only report changes without applying them (default: false)
 ### Strategies
 - **Autonomous first:** Never ask the user about entries that clearly belong in a deletion or move category. Only escalate on true ambiguity.
 - **Incremental over nuclear:** Always use patch-level edits (find-and-replace) rather than wholesale file rewrites. If the patch tool fails, fall back to rewrite only as a last resort.
 - **Verify before trusting:** Every technical config in MEMORY.md must be verified against the live system. If a service is unreachable, mark it as `UNVERIFIED` rather than deleting it.
 - **Stale state has a shelf life:** Any entry marked as "Temporary State" (e.g., "verbal offer", "deal pending", "negotiation phase") is deleted if older than 14 days.
 - **Work around tool failures:** If the `memory` tool fails after 2 retries, add a summary marker and report the stale state in the unresolved list.
 - **Privacy isolation:** Each agent only reads and writes its own memory files. No cross-agent memory access, no shared ledger, no shared state. The contract is the standard; each agent runs it independently.
 ### Invariants
 - No entry is deleted without being categorized first
 - No duplicate entries survive across MEMORY.md and USER.md
 - No session note or temporary state survives past 14 days
 - No config is trusted without verification (or marked UNVERIFIED)
 - The audit ledger is append-only  never modified or deleted, and is isolated per agent
 - The canary entry survives every audit unmodified
 - No data crosses agent boundaries
 ### Execution
 ```prose
 # ============================================================
 # PHASE 0: Canaries & Privacy
 # ============================================================
 # Check canaries  if either is missing or modified, raise CRITICAL alert
 memory_canary = read_file_line("~/.hermes/memories/MEMORY.md", containing="CANARY: Do not modify this line")
 user_canary = read_file_line("~/.hermes/memories/USER.md", containing="CANARY: Do not modify this line")
 if memory_canary is null or user_canary is null:
  return {
    status: "CRITICAL",
    alert: "CANARY BREACH  memory integrity compromised",
    details: {
      memory_canary_intact: memory_canary is not null,
      user_canary_intact: user_canary is not null
    }
  }
 # This agent only touches its own files  no cross-agent access
 # Confirm we are operating on this agent's own memory directory
 if not directory_exists("~/.hermes/memories"):
  return {status: "ERROR", reason: "Memory directory not found for this agent"}
 # ============================================================
 # PHASE 1: Pre-flight & Ledger
 # ============================================================
 # Read the audit ledger for drift detection
 ledger = read_file("~/.hermes/memories/MEMORY_AUDIT_LEDGER.md")
 last_audit_entry = parse_last_entry(ledger)
 # Verify the memory tool works
 try:
  memory_status = call check_memory_tool
 catch:
  memory_status = "BROKEN"
 # Read current state
 memory_content = read_file("~/.hermes/memories/MEMORY.md")
 user_content = read_file("~/.hermes/memories/USER.md")
 # Calculate current utilization
 memory_util = len(memory_content) / max_memory_chars * 100
 user_util = len(user_content) / max_user_chars * 100
 # Check if audit is needed
 if last_audit_entry.timestamp > 14 days and memory_util < 75 and user_util < 75:
  return {status: "SKIP", reason: "No audit needed"}
 # ============================================================
 # PHASE 1b: Drift Detection (Compare to Last Audit)
 # ============================================================
 drift_alerts = []
 if last_audit_entry exists:
  # Deletions accelerating? (50% increase from last audit)
  if last_audit_entry.deletions > 0:
    deletion_increase = (memory_util - last_audit_entry.memory_util_after) / last_audit_entry.deletions * 100
    if deletion_increase > 50:
      drift_alerts.append("ALERT: Memory bloat accelerating  deletions increasing by " + deletion_increase + "% vs last audit")
  # Unverified configs growing?
  if last_audit_entry.unverified > 0:
    unverified_increase = count_entries(memory_content, "[UNVERIFIED]") - last_audit_entry.unverified
    if unverified_increase > 2:
      drift_alerts.append("ALERT: Unverified configs growing  " + unverified_increase + " new unverified entries since last audit")
  # Audit missed? (No audit for >21 days)
  if last_audit_entry.timestamp > 21 days:
    drift_alerts.append("ALERT: Audit gap  no audit completed in 21+ days")
  # Utilization not improving? (After audit, still >70%)
  if last_audit_entry.memory_util_after > 70:
    drift_alerts.append("ALERT: Previous audit failed to reduce MEMORY.md below 70%  capacity issue")
 # ============================================================
 # PHASE 2: Categorize every entry
 # ============================================================
 categorized = []
 for entry in split_entries(memory_content):
  cat = call categorize_entry(entry)
  categorized.append({entry, category: cat, source: "MEMORY.md"})
 for entry in split_entries(user_content):
  cat = call categorize_entry(entry)
  categorized.append({entry, category: cat, source: "USER.md"})
 # ============================================================
 # PHASE 3: Identify actions
 # ============================================================
 actions = []
 for item in categorized:
  if item.category in ["Session Note", "Temporary State"]:
    # Check age of temporary state
    if item.category == "Temporary State" and item.age < 14 days:
      continue  # Too young to delete
    actions.append({action: "DELETE", entry: item.entry, reason: item.category})
  elif item.category == "Historical Snapshot":
    actions.append({action: "DELETE", entry: item.entry, reason: "Stale"})
  elif item.category == "Technical Config" and item.source == "USER.md":
    actions.append({action: "MOVE", from: "USER.md", to: "MEMORY.md", entry: item.entry})
  elif item.category == "Identity" or item.category == "Preference" and item.source == "MEMORY.md":
    actions.append({action: "MOVE", from: "MEMORY.md", to: "USER.md", entry: item.entry})
  elif item.category == "Operating Rule":
    actions.append({action: "MOVE", from: "MEMORY.md", to: "Skill file", entry: item.entry})
 # Check for duplicates
 for entry_a, entry_b in find_duplicates(categorized):
  actions.append({action: "DELETE", entry: entry_b.entry, reason: "Duplicate of " + entry_a.entry})
 # ============================================================
 # PHASE 4: Verify live configs
 # ============================================================
 for item in categorized:
  if item.category == "Technical Config" and item.entry contains URL:
    result = call verify_endpoint(item.entry)
    if result == "UNREACHABLE":
      # Mark as UNVERIFIED, do not delete
      actions.append({action: "MARK_UNVERIFIED", entry: item.entry})
    elif result == "VERIFIED":
      pass  # Good, nothing to do
 for item in categorized:
  if item.category == "Technical Config" and item.entry contains "model:":
    result = call verify_model(item.entry)
    if result == "MISMATCH":
      actions.append({action: "CORRECT", entry: item.entry, corrected: result.new_value})
 # ============================================================
 # PHASE 5: Apply changes
 # ============================================================
 if dry_run:
  return {status: "DRY_RUN", actions: actions, drift_alerts: drift_alerts, summary: generate_summary(actions)}
 # Apply deletions
 for action in actions:
  if action.action == "DELETE":
    try:
      memory action=remove target=action.source old_text=action.entry
    catch after 2 retries:
      patch path=action.source old_string=action.entry new_string=""
      # Add summary marker since memory tool is broken
      memory action=add target=action.source content="Entry deleted via patch: " + action.entry
 # Apply moves
 for action in actions:
  if action.action == "MOVE":
    # Delete from source
    patch path=action.from old_string=action.entry new_string=""
    # Add to destination
    append_file path=action.to content=action.entry + "\n"
 # Apply corrections
 for action in actions:
  if action.action == "CORRECT":
    patch path="~/.hermes/memories/MEMORY.md" old_string=action.entry new_string=action.corrected
 # Apply UNVERIFIED marks
 for action in actions:
  if action.action == "MARK_UNVERIFIED":
    patch path="~/.hermes/memories/MEMORY.md" old_string=action.entry new_string=action.entry + " [UNVERIFIED]"
 # ============================================================
 # PHASE 6: Post-audit verification
 # ============================================================
 new_memory_content = read_file("~/.hermes/memories/MEMORY.md")
 new_user_content = read_file("~/.hermes/memories/USER.md")
 new_memory_util = len(new_memory_content) / max_memory_chars * 100
 new_user_util = len(new_user_content) / max_user_chars * 100
 # Verify canaries survived the audit
 post_canary_check = read_file_line("~/.hermes/memories/MEMORY.md", containing="CANARY: Do not modify this line")
 if post_canary_check is null:
  drift_alerts.append("CRITICAL: Canary destroyed during audit  memory integrity compromised")
 # Add summary markers to in-memory stores
 memory action=add target=memory content="MEMORY.md rewritten. Utilization: " + new_memory_util + "%."
 memory action=add target=user content="USER.md rewritten. Utilization: " + new_user_util + "%."
 # ============================================================
 # PHASE 7: Append to Ledger
 # ============================================================
 ledger_entry = """
 ## """ + now() + """
 - memory_util_before: """ + memory_util + """%
 - memory_util_after: """ + new_memory_util + """%
 - user_util_before: """ + user_util + """%
 - user_util_after: """ + new_user_util + """%
 - deletions: """ + count(actions, "DELETE") + """
 - moves: """ + count(actions, "MOVE") + """
 - corrections: """ + count(actions, "CORRECT") + """
 - unverified: """ + count(actions, "MARK_UNVERIFIED") + """
 - unresolved: """ + len(unresolved_entries) + """
 - drift_alerts: """ + len(drift_alerts) + """
 - canary_intact: """ + (post_canary_check is not null) + """
 - status: COMPLETE
 """
 append_file path="~/.hermes/memories/MEMORY_AUDIT_LEDGER.md" content=ledger_entry
 # ============================================================
 # PHASE 8: Report
 # ============================================================
 return {
  status: "COMPLETE",
  memory_utilization: new_memory_util,
  user_utilization: new_user_util,
  actions_applied: len(actions),
  deletions: count(actions, "DELETE"),
  moves: count(actions, "MOVE"),
  corrections: count(actions, "CORRECT"),
  unverified: count(actions, "MARK_UNVERIFIED"),
  unresolved: unresolved_entries,
  drift_alerts: drift_alerts,
  last_audit: now(),
  ledger_updated: true,
  summary: generate_summary(actions)
 }
 ```
-### How to Follow the Template (Manual)
+### Shape
-If `prose` CLI is not available:
+- `self`: categorize, verify configs, apply incremental patches, generate reports, drift detection, ledger management, canary checks
 - `delegates`:
  - `categorize_entry`: determine the category of a memory entry
  - `check_memory_tool`: verify the memory tool is functional
  - `verify_endpoint`: check if a URL is reachable
  - `verify_model`: check if a model config matches live config.yaml
  - `find_duplicates`: detect duplicate entries across files
  - `generate_summary`: produce a human-readable summary of actions
-1. **Read both memory files:** `cat ~/.hermes/memories/MEMORY.md` and `cat ~/.hermes/memories/USER.md`
+### Tools
 2. **Check in-memory usage:** Use the `memory` tool with no args
 3. **Categorize every entry** using the Categorization Guide below
 4. **Rewrite MEMORY.md** — technical configs only, 800–1,100 chars max
 5. **Rewrite USER.md** — identity + preferences only, 300–500 chars max
 6. **Verify live configs** — curl each endpoint, check each model
 7. **Sync in-memory** — Add summary markers (or just leave file as source of truth)
 8. **Report** using the Example Output template at the bottom
-## Maintains
+- `file:read`: read MEMORY.md and USER.md
 - `file:patch`: incremental edits via find-and-replace
 - `file:append`: append entries when moving between files
 - `memory:action`: add/remove/replace entries in the in-memory store
 - `network:curl`: verify endpoint reachability
 - `cli:grep`: verify model configs against config.yaml
- last_audit: timestamp — When the last full audit was completed
+### Environment
 - memory_usage_pct: number — Current memory store usage percentage
 - user_usage_pct: number — Current user store usage percentage
 - total_entries_categorized: number — How many entries were audited
 - entries_moved: number — How many entries changed location
 - entries_deleted: number — How many stale entries were removed
 - live_configs_verified: array — Which configs were verified against live systems
 - known_issues: array — Problems found during audit (stale keys, dead services)
-## Parameters
+- `MEMORY_PATH`: ~/.hermes/memories/MEMORY.md
 - `USER_PATH`: ~/.hermes/memories/USER.md
 - `CONFIG_PATH`: ~/.hermes/config.yaml
 - `LEDGER_PATH`: ~/.hermes/memories/MEMORY_AUDIT_LEDGER.md
- memory_threshold: number — Usage % that triggers audit (default: 85, max: 95)
+### Per-Agent Notes
 - verify_configs: boolean — Whether to live-check configs after audit (default: true)
 - compress_entries: boolean — Whether to shorten verbose entries (default: true)
 - remove_session_notes: boolean — Strip "currently doing X" markers (default: true)
 - max_memory_chars: number — Target max chars for MEMORY.md after audit (default: 1100)
 - max_user_chars: number — Target max chars for USER.md after audit (default: 500)
-## Continuity
+Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
 The audit cycle is **event-driven**, not just cron-based:
 - **On threshold breach**: If memory usage exceeds `memory_threshold`, wake immediately
 - **On user request**: `"memory audit"`, `"memory maintenance"`, `"clean up memory"` — explicit trigger
 - **Retrospective**: Every 2-4 weeks if no other trigger fired — check for gradual bloat
 - **On config change**: After major provider swaps or infrastructure migrations, verify live configs
 - **On new agent onboarding**: Run once as part of initialization to establish clean baseline
 ## Success Criteria (What Makes a Memory Audit "Good")
 The audit is considered COMPLETE when ALL of these pass:
 ### Inventory Complete
 - MEMORY.md content read and categorized
 - USER.md content read and categorized
 - In-memory `memory` and `user` store usage percentages recorded
 - Every entry tagged with: `Infrastructure | Identity | Preference | Operating Rule | Historical | Session Note`
 ### Redistribution Correct
 - Identity entries → USER.md
 - Preference entries → USER.md
 - Technical config entries → MEMORY.md
 - Operating rules → Skill file or SOUL.md
 - Historical snapshots (dated checkpoints) → Deleted
 - Session notes ("currently doing X") → Deleted
 ### Files Compact
 - MEMORY.md ≤ `max_memory_chars` chars
 - USER.md ≤ `max_user_chars` chars
 - No duplicate information across files
 - No frustration signals or verbose descriptions
 - § separator between entries (standard convention)
 ### Configs Verified
 - API keys confirmed in `.env` file (grep, not read)
 - Model names in memory match live `config.yaml`
 - Endpoints in memory respond to curl/health checks
 - Credential pairs (username/token, email/password) match deployed state
 - Any mismatch corrected in memory with ✅/❌ labels
 ### In-Memory Synced
 - Summary marker added to `memory` store: "MEMORY.md rewritten <date>. Holds only tech configs."
 - Summary marker added to `user` store: "Home channel: <channel>. USER.md rewritten <date>."
 - Usage confirmed dropped below 60% after sync
 ### Skill Created/Updated (if applicable)
 - Operating rules that belong in a skill → created via `skill_manage`
 - Existing skills verified not to conflict with memory content
 - Skill category appropriate and description searchable
 ## Remembers
 Each audit stores:
 - Entry categorization decisions (why X went to USER instead of MEMORY)
 - What was deleted (stale snapshots, session notes)
 - What was verified (configs that passed live checks)
 - What failed verification (configs that need user attention)
 - Known tool quirks (memory replace/remove failures)
 - Compression patterns used (how verbose entries were shortened)
 ## Audit Rules
 ### Rule 1: Read Before Delete
 Never delete an entry without first reading it and categorizing it. Show the user a summary of what will change if unsure about a categorization.
 ### Rule 2: Categories Are Mutual Exclusive
 Each entry belongs to exactly ONE category:
 - **Identity** — "Jerome, Florida, Telegram @mejerome19"
 - **Preference** — "Single-account first for AWS"
 - **Technical Config** — "SearXNG on storepve:8888"
 - **Operating Rule** — "Never go rogue on infrastructure"
 - **Historical Snapshot** — "Baseline v30 from Jun 26" (DELETE)
 - **Session Note** — "Currently performing memory audit" (DELETE)
 ### Rule 3: Rules Belong in Skills
 If an entry is a procedure, protocol, or mandate (`"do X this way"`, `"never do Y"`) — it belongs in a skill file, not MEMORY.md. Check `skills_list` first; create via `skill_manage` if no match exists.
 ### Rule 4: Verify Live Before Trusting Memory
 Every config entry in MEMORY.md must be verified against the live system:
 - Model name → `grep config.yaml`
 - API key → `grep .env`
 - Service URL → `curl` health check
 - Credentials → cross-reference with deployed state
 ### Rule 5: Work Around the `memory` Tool
 The `memory` tool's `remove` and `replace` actions use strict text matching that can fail on special characters. If a remove/replace fails repeatedly:
 - Do not retry more than 2 times
 - Add a new summary marker entry instead
 - Report to user: "File on disk updated; in-memory store has stale leftovers"
 - The file on disk (MEMORY.md, USER.md) is the source of truth regardless
 ### Rule 6: Report Before Asking
 Don't ask the user "should I delete X?" for session notes or historical snapshots — those are always safe to delete. Only ask about ambiguous categorizations or cross-file moves of preference data.
 ## Execution
 1. **Read state** — Read MEMORY.md, USER.md, check in-memory usage via `memory` tool
 2. **Categorize** — Tag every entry with its category
 3. **Prepare rewrites** — Draft compressed MEMORY.md and USER.md:
   - MEMORY.md: Technical configs only (800-1100 chars)
   - USER.md: Identity + preferences only (300-500 chars)
 4. **Write files** — Overwrite MEMORY.md and USER.md
 5. **Verify live configs** — Check every config entry against actual system state
 6. **Sync in-memory** — Add summary markers to `memory` and `user` stores
 7. **Create/update skills** — Move operating rules to skill files if needed
 8. **Report** — Output clean status table with before/after/verification
 ## Example Output
 ```
 ## Memory Audit Complete ✅
 | File | Before | After | Reduction |
 |---|---|---|---|
 | MEMORY.md | 1,653 chars | 1,077 chars | ~35% |
 | USER.md | 860 chars | 418 chars | ~51% |
 ### What Moved
 | Entry | From | To |
 |---|---|---|
 | Identity + Preferences | MEMORY.md | USER.md |
 | Agent boundary rule | MEMORY.md | SOUL.md (persona) |
 | Baseline v30 snapshot | MEMORY.md | Deleted |
 ### Config Verification
 | Config | Live Check | Status |
 |---|---|---|
 | DeepSeek model | config.yaml: deepseek-chat | ✅ |
 | DeepSeek key | .env found | ✅ |
 | SearXNG endpoint | :8888 curl 200 | ✅ |
 ### Memory Usage
 | Store | Before | After | Status |
 |---|---|---|---|
 | memory | 91% | 59% | ✅ |
 | user | 85% | 41% | ✅ |
 ```
@@ -0,0 +1,290 @@
 ---
 kind: pattern
 name: mumuni-delegation
 description: >
  Mumuni-specific operating doctrine for task decomposition, worker
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
  Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
 version: 1.0.0
 ---
 ## Maintains
 - Worker roster: 6 profiles (`syslog-code`, `syslog-devops`, `syslog-email`,
  `syslog-research`, `syslog-review`, `syslog-writer`)
 - Kanban board state at `~/.hermes/kanban/kanban.json`
 - Context window budget: ~65K tokens per request (131K total, 60% threshold)
 ## Topology
 **Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
 **Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
 This contract is infrastructure-agnostic in terms of which nodes are used.
 Workers execute tasks on whatever infrastructure they're given — SSH to .6,
 .pm, .9, .12, or .15 depending on the task. The contract defines the
 **who** and **when** — not the **where**.
 ## Why This Matters
 Without enforced delegation, the manager consumes the full iteration budget
 (60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
 leaving no capacity for actual coordination. The result: context overflow
 (59K tokens in system prompt), iteration exhaustion, and degraded response
 quality. This contract exists because I blew through my budget checking
 Proxmox node status instead of delegating to `syslog-devops`.
 ## Context Window Discipline
 **The system prompt is ~6.5K tokens (stable: ~4.5K tool schemas + ~2K other guidance).**
 **Volatile (MEMORY.md + USER.md): ~300 tokens.**
 **Total base: ~6,800 tokens per request.**
 The remaining budget is the conversation. Every tool call result adds to it.
 If a single call returns >10K tokens (e.g., `grep` on a large file, SSH output
 from multiple nodes), the context fills fast. That's why we delegate: workers
 process in isolation and return compact results.
 ## Trigger Conditions
 Delegation is **mandatory** when any of these apply:
 | Condition | Threshold | Example |
 |-----------|-----------|---------|
 | Multiple tool calls needed | 2+ calls with intermediate logic | Read file → analyze → write report |
 | Large data retrieval | Output >5K tokens | `grep -r "pattern" /path` on large dirs |
 | Cross-domain work | Spans 2+ worker specialties | Infra check + email filter |
 | Infrastructure changes | Any mutating operation | `qm set`, `systemctl restart`, `git push` |
 | Research/analysis | Needs browser or deep reading | Web research, code review, data analysis |
 | Code builds or changes | Writing or modifying code | Scripts, configs, patches |
 | Sequential dependencies | Worker B needs Worker A's output | Code → Review → Deliver |
 **Single tool calls stay at manager level.** Quick `grep`, `ls`, `cat`,
 `curl`, `hermes tools list` — these are decision-making tools. The manager
 reads them directly.
 ## Data Source Integrity (CRITICAL)
 **Workers MUST use the data provided in their task context. They MUST NOT
 fetch their own data from external sources unless explicitly told to.**
 When a task says "Read file X and format it", the worker reads file X. It does
 not query a separate API, run its own diagnostics, or pull data from a different
 system. This is the #1 source of cross-worker inconsistency: one worker gathers
 SSH data, another queries the Proxmox API, and the report merges two incompatible
 datasets.
 **Rule:** If a worker needs additional data beyond what's in its task description,
 it asks the manager (via relay) — it doesn't go find it on its own.
 **This is a hard rule, not a recommendation.** Violating it produces the exact
 type of discrepancy the kanban pipeline exists to prevent: a review worker finds
 "5 nodes present" in the raw data but "5/5 online" in the report — even though
 one of those nodes was unreachable. The report lied because it used data the
 raw data never provided.
 ## Worker Selection Matrix
 | Worker | Model | Toolsets | Role | Use When |
 |--------|-------|----------|------|----------|
 | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
 | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
 | `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
 | `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
 | `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
 | `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
 ### Selection Rules
 1. **Match specialty first.** A code task → `syslog-code`. An infra task →
   `syslog-devops`. Don't put a `syslog-email` worker on a code review.
 2. **Research tasks with browser needs → `syslog-research`.** Other workers
   don't have the browser toolset.
 3. **Verification → `syslog-review`.** Never deliver raw worker output.
 4. **Documentation/content → `syslog-writer`.** Let them own the prose.
 5. **If unsure, delegate to `syslog-research`** — it has the broadest toolset
   (includes browser) and high reasoning effort.
 ## Delegation Protocol
 ### Step 1: Decompose
 Break the task into lanes. Each lane does ONE thing. Workers are independent —
 no lane depends on another's output mid-flight. If lanes depend on each other,
 dispatch sequentially.
 ### Step 2: Dispatch
 Fire workers via `delegate_task`:
 **Critical: Pass the data, not just the goal.** When dispatching a worker that
 processes output from another worker, include the file path AND explicit
 instructions to use ONLY that source. Example:
 ```
 delegate_task(
    goal="Format the cluster check into a clean report",
    context="Source data is at /tmp/proxmox-check-raw.md. Format ONLY the data
    in that file. Do NOT query the Proxmox API or any other data source. Use the
    file as your sole source of truth."
 )
 ```
 **Parallel (independent lanes):**
 ```
 delegate_task(
    tasks=[
        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
    ]
 )
 ```
 **Sequential (dependent lanes):**
 Dispatch lane 1 → wait for result → dispatch lane 2.
 ### Step 3: Verify
 **MANDATORY for:**
 - Infrastructure changes (any `qm`, `pct`, `systemctl`, `git push`)
 - Code builds and modifications
 - Research findings (web data, external sources)
 - Any output that will reach the user
 **Fire `syslog-review` to verify:**
 ```
 delegate_task(
    goal="Review the output of the devops worker. Verify the node status
    report is accurate, check for inconsistencies, confirm all nodes were
    reachable.",
    context="Worker was syslog-devops. Output is at /tmp/node-report.md.
    Verify against live system."
 )
 ```
 **If verification fails:**
 1. Send work back to original worker with review feedback
 2. Re-verify
 3. Max 2 re-verify cycles before escalating to Kwame
 ### Step 4: Deliver
 Only verified results reach Kwame. Format per channel:
 - Telegram: Use `telegram-formatting` skill
 - Zulip: Use Zulip Markdown (CommonMark)
 - Email: Use `syslog-email` skill
 ## Kanban Board Protocol
 **File:** `~/.hermes/kanban/kanban.json`
 ```json
 {
  "task_id": "unique-id",
  "title": "Task description",
  "created": "2026-07-09T01:00:00",
  "status": "backlog|in_progress|review|done",
  "lanes": [
    {
      "lane_id": "devops-check",
      "worker": "syslog-devops",
      "goal": "Check all 5 Proxmox nodes",
      "status": "dispatched|completed|failed",
      "output_file": "/tmp/node-report.md"
    }
  ]
 }
 ```
 **Update the board on every state change.**
 ## Failure Handling
 ### Worker Timeouts
 - Child timeout: **900 seconds** (15 minutes)
 - Worker model `syslog-auto` is slow — it can hit the timeout limit with
  22+ API calls
 - **If a worker times out:** Re-dispatch with a narrower scope. Break the
  task into smaller pieces that fit in the timeout window.
 - **Avoid delegating sequential SSH hops** — each SSH connection adds latency
  that compounds quickly. Prefer API-based or local approaches when possible.
 ### Worker Selection Failures
 - `syslog-devops` is best for infrastructure tasks (SSH, Proxmox, Docker)
 - `syslog-code` is best for code-level work (reading files, writing scripts)
 - `syslog-research` has the browser toolset — use for web research
 - `syslog-review` is the QA gate — always fire before delivery
 - **Never fire more than 3 parallel workers** (max_concurrent_children: 3)
 - **Never nest delegation** (max_spawn_depth: 1)
 ### Context Overflow
 - If a task requires >10K tokens of output, delegate the processing
 - Workers return compact summaries, not raw data dumps
 - Pass file paths and concrete goals — never dump raw data into context
 ## Anti-patterns
 - ❌ Reading large files into your own context before deciding → delegate the read
 - ❌ Carrying SSH/grep/output results in your context → delegate the analysis
 - ❌ Doing work yourself and then "pretending" to delegate → the user can tell
 - ❌ Skipping verification → raw worker output never reaches the user
 - ❌ Delegating single tool calls → keep quick reads/writes at manager level
 - ❌ Firing more than 3 workers in parallel → hard limit
 - ❌ **Workers fetching their own data sources** → a writer worker that queries the Proxmox API when told to "format the raw file" is fabricating data. Use the input given, not external sources
 ## Emergency Exception
 **In an emergency (server down, service must be restored immediately):**
 - Delegate the diagnosis (find the problem)
 - Execute the fix yourself (minimize handoff latency)
 - Verify the fix after delivery
 - Log the exception in the kanban board
 The emergency exception exists because the user needs the service back NOW,
 not after three worker round-trips. But it's an exception — not the rule.
 ## What This Contract Doesn't Cover
 1. **Worker profile configuration** — covered by `hermes-config-template.prose.md`
 2. **SSH key management** — covered by existing SSH/Proxmox contracts
 3. **Git workflow** — covered by `AGENTS.md` in the prose-contracts repo
 4. **Cron job management** — covered by individual cron contracts
 5. **Infra verification** — covered by `verify-before-mutate` protocol
 ## Verification
 Run `scripts/worker-audit.py` to verify all 6 profiles are aligned:
 ```bash
 python3 /root/.hermes/skills/kanban-orchestrator/scripts/worker-audit.py
 ```
 ## References
 - `kanban-orchestrator` skill: The operational playbook (detailed execution steps)
 - `worker-profile-audit.md` (skill reference): Worker configuration audit notes
 - `delegation-timeout-patterns.md` (skill reference): Timeout handling patterns
 - `verify-before-mutate` protocol: Infrastructure change verification
 - `hermes-config-template.prose.md`: Worker profile configuration
 ## Success Criteria
 This contract succeeds when:
 1. **No context overflow** — single-turn tasks don't exhaust the iteration budget
 2. **Workers do the work** — manager coordinates, doesn't execute
 3. **Verification before delivery** — all output passes through `syslog-review`
 4. **Kanban board is current** — every task has a lane, every lane has a status
 5. **User gets verified results** — raw worker output never reaches Kwame
 ---
 **Last updated:** 2026-07-09
 **Author:** Mumuni (with Kwame's input on triggers and exception criteria)
 **Status:** Draft — awaiting PR review and merge to prose-contracts main
@@ -0,0 +1,107 @@
 ---
 kind: architecture
 name: pi-approval-architecture
 description: >
  Documents pi's approval model vs Hermes, what commands are available,
  and the architectural constraints that prevent mid-turn tool approval in pi.
  Updated with /approve and /deny stub commands for cross-agent UX consistency.
 agent: abiba
 version: 1.0.0
 ---
 # Pi Approval Architecture
 ## TL;DR
 | Feature | Hermes | Pi (Abiba) |
 |---------|--------|------------|
 | `/approve` command | ✅ Full — unblocks agent | ✅ Stub — acknowledges, no blocking |
 | `/deny` command | ✅ Full — blocks agent | ✅ Stub — acknowledges, no blocking |
 | Dangerous command detection | ✅ 40+ patterns + Tirith | ❌ None (LLM judgment only) |
 | Mid-turn tool approval | ✅ Agent thread blocks on Event | ❌ No hook in agent loop |
 | Smart approval (aux LLM) | ✅ `approvals.mode: smart` | ❌ N/A |
 | Yolo mode | ✅ `/yolo`, `approvals.mode: off` | ❌ N/A |
 | Permanent allowlist | ✅ `command_allowlist` + `shell-hooks` | ❌ N/A |
 | Session-scoped approval | ✅ `approve_session()` | ❌ N/A |
 | Approval timeout | ✅ 60s default, configurable | ❌ N/A |
 ## Why Pi Can't Do Mid-Turn Approval
 Pi's agent loop is **418 lines** of TypeScript. It's deliberately minimal:
 ```
 LLM decides → calls tool → tool executes immediately → output goes to LLM → repeats
 ```
 There is no middleware between "LLM decides to call bash" and "bash executes."
 The extension system (`injectIntoSession`) feeds messages IN but doesn't intercept
 tool calls mid-turn.
 Hermes has a full approval pipeline:
 ```
 LLM decides → check_all_command_guards() → if dangerous → notify_cb → block thread
 → user responds /approve → resolve_gateway_approval() → Event.set() → unblock
 ```
 This pipeline is deeply integrated into Hermes' gateway, tool executor, and
 session management. Replicating it in pi would require rewriting the core agent
 loop — violating pi's design constraint of minimal core.
 ## What Pi HAS: `/approve` and `/deny` Stubs
 Added to prevent `/approve` and `/deny` from being forwarded to the LLM as
 garbled input. When a user types these commands to Abiba:
 - **`/approve`** → "Pi doesn't have pending approvals. Commands execute immediately."
 - **`/approve session`** → Same response
 - **`/deny`** → Same response + "For Hermes agents (Tanko, Mumuni), these work with their built-in approval system."
 This keeps the UX consistent across agents — users can type `/approve` anywhere
 without getting confused by LLM responses.
 ## Pi's Full Command Set (Hermes Parity)
 | Command | Handler | Description | Hermes Equivalent |
 |---------|---------|-------------|-------------------|
 | `/status` | System | PM2 processes, containers, session info | `/status` |
 | `/retry` | Session | Re-injects last user message | `/retry` |
 | `/continue` | Session | Continues truncated response | `/continue` |
 | `/new [topic]` | Session | Clears conversation context | `/new` |
 | `/stop` | Control | Interrupts current LLM processing | `/stop` |
 | `/model [name]` | Info | Show current model, list available | `/model` |
 | `/compact` | Session | Requests context compaction | `/compact` |
 | `/yolo` | Stub | Pi has no approval system — accepted for UX | `/yolo` |
 | `/verbose` | Stub | Verbosity controlled by prompt, not toggle | `/verbose` |
 | `/approve` | Stub | Pi executes immediately — no approvals | `/approve` |
 | `/deny` | Stub | Pi executes immediately — no approvals | `/deny` |
 | `/help [cmd]` | Info | Shows command help | `/help` |
 ## What Pi COULD Add (Without Core Changes)
 These would work within pi's current architecture:
 1. **Pre-message dangerous command scan**: Before forwarding an agent response to
   Zulip, scan the text for dangerous bash commands and add a warning banner.
   Doesn't prevent execution but alerts user.
 2. **User-facing approval prompt injection**: When user asks pi to do something
   dangerous (detected via regex on user input), pi could respond with a
   confirmation request before proceeding. This works because it's at the
   message level, not mid-turn.
 3. **Response rate-limiting**: If pi generates many bash commands rapidly,
   throttle and ask for confirmation. Works within the extension's send path.
 ## Files
 | File | Purpose |
 |------|---------|
 | `/root/.pi/agent/extensions/zulip/index.js` | Pi Zulip extension with command system |
 | `/root/prose-contracts/zulip-approval-fix.prose.md` | Hermes Zulip approval fix |
 | `/root/prose-contracts/pi-approval-architecture.prose.md` | This document |
 ## Related
 - Hermes `tools/approval.py` — Full approval pipeline (dangerous patterns, Tirith, smart mode)
 - Hermes `gateway/slash_commands.py:_handle_approve_command` — Slash command handler
 - Hermes Zulip adapter fix — HTML stripping for command detection
@@ -10,10 +10,14 @@ description: >
 ## Maintains
 - abiba-zulip: { status: "online", uptime: string, restarts: number }
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
 - gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
 > **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
 ## Continuity
 - Self-driven: check every 300 seconds (5 min)
@@ -28,9 +32,14 @@ description: >
 - **Escalate**: If still failed after 2 retries, send Zulip DM to owner
 ### Rule 2: Process Restarting Too Often
- **Detect**: `pm2 status` shows restarts > 5 in the last hour
+- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
 - **Note**: PM2 counter never decrements; only full delete+re-add resets it
 - **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
- **Escalate**: Always — alert owner with restart count
+- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
 - **Historical fix**: Previous cycles were caused by abiba-zulip extension's
  stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
  This was the root cause of the 18-restart accumulation. Threshold raised
  and PM2 counter reset on 2026-06-28.
 ## Execution
@@ -1,64 +0,0 @@
 #!/bin/bash
 # pm2-self-heal — hourly PM2 process check
 # Part of the pm2-self-heal prose contract
 # Only sends Zulip DM when action is taken or escalation needed
 # Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
 ZULIP_BOT="abiba-bot@chat.sysloggh.net"
 ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
 ZULIP_SITE="https://chat.sysloggh.net"
 OWNER_ID=9
 LOG="/tmp/pm2-self-heal.log"
 send_alert() {
    local subject="$1"
    local body="$2"
    curl -s -X POST "$ZULIP_SITE/api/v1/messages" \
        -u "$ZULIP_BOT:$ZULIP_KEY" \
        -d "type=private" \
        -d "to=[$OWNER_ID]" \
        -d "content=$subject\n$body" > /dev/null 2>&1
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Alert sent" >> "$LOG"
 }
 # Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
 STATUS=$(pm2 status --no-color 2>/dev/null)
 ALERTS=""
 # Check abiba-telegram (safe to auto-restart)
 TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
 TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
 TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
 if [ "$TEL_STATUS" != "online" ]; then
    pm2 restart abiba-telegram > /dev/null 2>&1
    sleep 3
    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
    TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs)
    if [ "$TEL_STATUS2" = "online" ]; then
        ALERTS="${ALERTS}⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online\n"
    else
        ALERTS="${ALERTS}🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)\n"
    fi
 fi
 # Check abiba-zulip (read-only — never restart)
 ZUL_LINE=$(echo "$STATUS" | grep "abiba-zulip")
 ZUL_STATUS=$(echo "$ZUL_LINE" | awk -F'│' '{print $10}' | xargs)
 ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs)
 if [ "$ZUL_STATUS" != "online" ]; then
    ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n"
    ALERTS="${ALERTS}   NOT auto-restarting (runs this contract)\n"
 elif [ "$ZUL_RESTARTS" -gt 5 ] 2>/dev/null; then
    ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n"
 fi
 # Log check
 echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS zul=$ZUL_STATUS alerts=${ALERTS:+yes}" >> "$LOG"
 # Only DM when there's something to report
 if [ -n "$ALERTS" ]; then
    send_alert "**PM2 Self-Heal — $(date '+%H:%M UTC')**" "$ALERTS"
 fi
@@ -0,0 +1,116 @@
 ---
 kind: responsibility
 name: proxmox-monitor
 description: >
  Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
  on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
  exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
  instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
  (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
  the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
 agent: abiba
 ---
 ## Architecture
 ```
 ┌─────────────────────────────────────────────────────────────┐
 │  CT 116 (syslog-api) — monitoring compose /opt/monitoring/   │
 │  Prometheus :9090  →  Grafana :3001 (direct LAN, 0.0.0.0)    │
 └───────┬──────────────┬──────────────┬───────────────────────┘
        │              │              │
        ▼              ▼              ▼
   pve-exporter    docker-stats    (scrapes 5x node_exporter)
   :9221           :9324
        │              │
        ▼              ▼
   PVE API         Docker API
   (amdpve .15,    (unix socket,
    cluster-wide)  10 containers)
 ```
 ## Exporters
 | Exporter | Host:Port | Scope | Notes |
 |----------|-----------|-------|-------|
 | prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
 | node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
 | docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
 ## PVE API Token
 - User: `monitoring@pve` (cluster-replicated)
 - Role: `PVEAuditor` on `/` (read-only, whole cluster)
 - Token: `monitoring@pve!prometheus` — stored in Infisical vault (`PROXMOX_MONITOR_TOKEN`)
 - `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
 ## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
 | UID | Title | Panels | Source |
 |-----|-------|--------|--------|
 | proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
 | proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
 | docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
 | gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
 - Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/`
 - Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml`
 - Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`)
 - Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks
 ## Access
 - **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
 - **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
 - **Credentials**: admin / password stored in Infisical vault (`GRAFANA_ADMIN_PASSWORD`)
 - Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
 - grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
 ## Configuration Files
 | File | Host | Purpose |
 |------|------|---------|
 | `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
 | `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
 | `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
 | `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
 | `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
 | `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
 | `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
 | `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
 ## Cluster "Tabiri" — 6 Nodes
 | Node | IP | Role |
 |------|----|----|
 | ocupve | 192.168.68.5 | PVE |
 | storepve | 192.168.68.6 | PVE |
 | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
 | minipve | 192.168.68.12 | PVE |
 | amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) |
 | hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
 ## Operations
 ### view-dashboards
 Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder
 ### add-dashboard
 Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
 ### check-targets
 `curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
 ### restart-exporter
 `cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
 ### rotate-pve-token
 `pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter`
 ## Known Issues & Notes
 - **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter.
 - **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health.
 - **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
 - **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
 - **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.
@@ -0,0 +1,210 @@
 ---
 kind: pattern
 name: ra-h-os-custodianship-contract
 description: >
  Operational standards for maintaining RA-H OS as a high-functioning shared
  memory system across all agents. Defines rules for embedding consistency,
  namespace discipline, staleness management, orphan prevention, agent
  custodianship, and recovery protocols. Prevents knowledge graph degradation
  and ensures reliable semantic search capabilities.
 ---
 # RA-H OS Custodianship Contract — Shared Memory Protocol
 ## Purpose
 This contract establishes the operational standards for maintaining RA-H OS as a high-functioning shared memory system across all agents. It defines the rules, responsibilities, and recovery protocols that prevent knowledge graph degradation and ensure consistent, reliable semantic search capabilities.
 ## Core Principles
 ### 1. Embedding Consistency
 **Rule:** All nodes MUST use the RA-H OS built-in embedding pipeline via `http://192.168.68.65:8080/v1/embeddings` for consistent vector representation.
 **Requirements:**
 - The embedding service at `192.168.68.65:8080` is the single source of truth for vector generation
 - No external embedding APIs (OpenAI, Cohere, etc.) may be used for RA-H OS nodes
 - All nodes must have `embedding_status: "chunked"` and `chunk_status: "chunked"` upon creation
 - If embedding fails, the node must be immediately flagged as `state: "unsearchable"` and logged
 ### 2. Namespace Discipline
 **Rule:** All nodes MUST be categorized into one of three namespaces with strict metadata requirements.
 **Namespace Structure:**
 - `agent-private`: Isolated working notes for individual agents (Mumuni, Tanko, Okyeame, etc.)
 - `shared`: Policy files, registry nodes, collective knowledge
 - `syslogsolution`: Business nodes, client data, operational context
 **Metadata Requirements:**
 Every node MUST have these metadata fields:
 ```json
 {
  "agent_id": "<agent_name>",
  "namespace": "<namespace>",
  "state": "<state>",
  "type": "<type>",
  "owner": "<owner>",
  "tenant": "<tenant>",
  "visibility": "<visibility>",
  "source": "<source>",
  "captured_by": "<captured_by>"
 }
 ```
 ### 3. Staleness Management
 **Rule:** Nodes are categorized by type and have specific staleness thresholds.
 **Staleness Thresholds:**
 - `policy` type: 30 days (e.g., Shared Memory Policy, Agent Registry)
 - `registry` type: 30 days (e.g., Node State Management, Health Dashboard)
 - `template` type: 30 days (e.g., Agent SOUL.md)
 - `note` type: 60 days (e.g., business context, project notes)
 - `information` type: 90 days (e.g., reference documentation)
 - `idea` type: 30 days (e.g., brainstorming, experimental notes)
 **Transition Protocol:**
 - `active` → `stale`: After days since `updated_at` exceeds threshold
 - `stale` → `review_pending`: Daily cron check flags for human review
 - `review_pending` → `active`: Human reviews and updates content
 - `review_pending` → `archived`: After 15 days in review_pending state (automatic purge)
 ### 4. Orphan Prevention
 **Rule:** No node should remain orphaned (0 edges) for more than 7 days.
 **Orphan Detection:**
 - Daily cron job identifies nodes with 0 incoming/outgoing edges
 - Orphans are flagged with `state: "review_pending"` and `metadata.orphan_detected: true`
 - 7-day grace period: Orphans must be either:
  - Connected to relevant nodes via edges
  - Merged into existing parent nodes
  - Archived after 15 days in review_pending state
 ### 5. Agent Custodianship
 **Rule:** Each agent is responsible for maintaining their own created nodes.
 **Responsibilities:**
 - **Mumuni**: Primary custodian for `syslogsolution` namespace, health dashboard, and all business nodes
 - **Tanko**: Custodian for `agent-private` namespace, personal notes, and fitness/creative work
 - **Okyeame**: Custodian for `shared` namespace, policy files, and collective knowledge nodes
 - **All Agents**: Must verify embedding status before reporting node creation as complete
 ### 6. Embedding Failure Recovery
 **Rule:** When embedding pipeline is unavailable, nodes are blocked from creation or immediately flagged.
 **Recovery Protocol:**
 1. **Detection**: Daily health check monitors `http://192.168.68.65:8080/health` endpoint
 2. **Alert**: If embedding service is down, alert is sent to all agents via relay
 3. **Mitigation**: 
   - New node creation is suspended
   - Existing nodes with `chunk_status: "not_chunked"` are flagged as `unsearchable`
 4. **Recovery**: When service returns:
   - All `not_chunked` nodes are queued for re-embedding
   - Re-embedding status logged in `embedding_retry_log` table
   - 5-minute retry window with exponential backoff (1s, 2s, 4s, 8s, 16s)
 ### 7. Chunking Verification
 **Rule:** All node modifications must verify chunking status within 5 minutes.
 **Verification Protocol:**
 - After `updateNode` or `createNode`, query `chunks` table for `node_id`
 - If `chunk_status != "chunked"` after 5 minutes, log failure and flag node
 - Automated cleanup script runs every 6 hours to retry failed chunks
 - Maximum 3 retry attempts per node before escalation to human
 ### 8. Graph Health Monitoring
 **Rule:** Health dashboard (Node #74) must reflect real-time graph status.
 **Dashboard Requirements:**
 - Total nodes, edges, orphans, stale nodes, chunk errors
 - Embedding service health status
 - Last successful embedding attempt
 - Alert thresholds:
  - Orphan rate > 40%: Warning
  - Stale node rate > 50%: Critical
  - Chunk error rate > 30%: Critical
  - Embedding service down: Critical (immediate alert)
 ## Operational Procedures
 ### Daily Maintenance (Automated)
 1. **Staleness Check**: Identify nodes past their threshold
 2. **Orphan Detection**: Flag nodes with 0 edges
 3. **Chunk Verification**: Retry failed chunks, log errors
 4. **Health Update**: Refresh Node #74 dashboard
 ### Weekly Maintenance (Human Review)
 1. **Review Pending Nodes**: Examine flagged nodes
 2. **Edge Optimization**: Connect related orphaned nodes
 3. **Archive Cleanup**: Remove nodes past 15-day review period
 4. **Embedding Audit**: Verify all nodes have valid embeddings
 ### Emergency Recovery
 1. **Embedding Service Down**: 
   - Suspend node creation
   - Alert all agents
   - Investigate root cause (check `192.168.68.65:8080`)
 2. **Database Corruption**:
   - Restore from latest PBS backup
   - Verify chunk integrity
   - Re-run embedding for affected nodes
 3. **Mass Orphan Creation**:
   - Identify source agent/namespace
   - Review recent changes
   - Reconnect or archive affected nodes
 ## Compliance & Enforcement
 ### Audit Schedule
 - **Daily**: Automated health checks
 - **Weekly**: Human review of review_pending nodes
 - **Monthly**: Comprehensive graph audit (full schema validation)
 - **Quarterly**: Contract review and threshold adjustment
 ### Violation Consequences
 - **First**: Alert to responsible agent
 - **Second**: Node placed in `review_pending` state
 - **Third**: Agent access suspended until remediation
 - **Fourth**: Escalation to human (Kwame)
 ### Metrics for Success
 - Orphan rate: < 10%
 - Stale node rate: < 20%
 - Chunk error rate: < 5%
 - Embedding service uptime: > 99.9%
 - Node creation verification: 100%
 ## Implementation Notes
 ### Database Schema Requirements
 - `nodes` table: Standard fields plus `embedding_status`, `chunk_status`, `last_embedding_attempt`
 - `chunks` table: Must track `chunk_status` and `embedding_status`
 - `embedding_retry_log` table: Track retry attempts for failed chunks
 - `node_state_history` table: Log state transitions for audit trail
 ### Cron Job Requirements
 - `ra-h-health-check`: Daily (4h interval)
 - `ra-h-staleness-detection`: Daily
 - `ra-h-orphan-detection`: Daily
 - `ra-h-chunk-retry`: Every 6 hours
 - `ra-h-review-cleanup`: Weekly (15-day archive)
 ### Agent Onboarding
 All new agents must:
 1. Read this contract
 2. Understand namespace responsibilities
 3. Know how to verify embedding status
 4. Know the alert escalation path
 ---
 ## Approval
 This contract is effective immediately upon approval by the primary custodian (Mumuni) and system owner (Kwame).
 **Effective Date:** July 10, 2026  
 **Review Date:** October 10, 2026  
 **Primary Custodian:** Mumuni 🦅  
 **System Owner:** Jerome Tabiri
 ---
 *Contract Version: 1.0*  
 *Last Updated: July 10, 2026*  
 *Storage: `/root/.hermes/skills/ra-h-os-custodianship-contract.prose.md`*
@@ -0,0 +1 @@
 {"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}
@@ -0,0 +1,26 @@
 # run:20260704-005804-51f9a9 disk-gc-threat-response
 root: disk-gc-threat-response
 1→ [scan] fleet-wide disk scan started
 2→ CT 100 abiba (amdpve): 12G/59G 21% GREEN
 2→ CT 101 llm-gpu (acerpve): 144G/255G 60% GREEN
 2→ CT 102 adguard (acerpve): 5.9G/32G 20% GREEN
 2→ CT 103 ocu-llm (ocupve): 48G/186G 27% GREEN
 2→ CT 104 authentik (minipve): 4.6G/9.8G 50% GREEN
 2→ CT 105 kagentz (amdpve): 16G/59G 27% GREEN (resolved: was 87%, pruned 35.67G)
 2→ CT 106 ra-h-os (storepve): 2.1G/20G 11% GREEN
 2→ CT 107 pbs (storepve): 1.1G/40G 3% GREEN
 2→ CT 108 media (storepve): 19G/196G 10% GREEN
 2→ CT 109 docker-vm (storepve): 25G/158G 17% GREEN
 2→ CT 110 gitea (minipve): 5.6G/40G 16% GREEN
 2→ CT 111 tdunna (amdpve): 16G/63G 26% GREEN
 2→ CT 112 tanko (amdpve): 17G/49G 36% GREEN
 2→ CT 113 baggy (amdpve): 7.0G/52G 15% GREEN
 2→ CT 114 mumuni (minipve): 29G/59G 51% GREEN
 2→ CT 115 scottdenya (amdpve): 14G/59G 25% GREEN
 2→ CT 116 syslog-api (minipve): 16G/40G 42% GREEN
 2→ CT 117 zulip (storepve): 6.1G/60G 11% GREEN
 2→ CT 118 jitsi (minipve): OFFLINE — no disk scan
 3→ [summary] Fleet: 19 hosts, 18 scanned, 18 GREEN, 0 AMBER, 0 RED, 0 CRITICAL, 1 OFFLINE
 3→ [summary] Previously resolved: CT 105 kagentz 87%→27% (35.67GB reclaimed)
 ---end 2026-07-04T00:58:04Z
@@ -0,0 +1,52 @@
 # Fleet Disk Health — 2026-07-04 00:58 UTC
 ## Summary
 | Metric | Count |
 |--------|-------|
 | Total hosts | 19 |
 | Scanned | 18 |
 | GREEN (<75%) | 18 |
 | AMBER (75-84%) | 0 |
 | RED (85-94%) | 0 |
 | CRITICAL (≥95%) | 0 |
 | OFFLINE | 1 (jitsi CT 118) |
 ## Per-Host Status
 | CT | Name | Node | Used | Total | % | Status |
 |----|------|------|------|-------|---|--------|
 | 100 | abiba | amdpve | 12G | 59G | 21% | 🟢 GREEN |
 | 101 | llm-gpu | acerpve | 144G | 255G | 60% | 🟢 GREEN |
 | 102 | adguard | acerpve | 5.9G | 32G | 20% | 🟢 GREEN |
 | 103 | ocu-llm | ocupve | 48G | 186G | 27% | 🟢 GREEN |
 | 104 | authentik | minipve | 4.6G | 9.8G | 50% | 🟢 GREEN |
 | 105 | kagentz | amdpve | 16G | 59G | 27% | 🟢 GREEN |
 | 106 | ra-h-os | storepve | 2.1G | 20G | 11% | 🟢 GREEN |
 | 107 | pbs | storepve | 1.1G | 40G | 3% | 🟢 GREEN |
 | 108 | media | storepve | 19G | 196G | 10% | 🟢 GREEN |
 | 109 | docker-vm | storepve | 25G | 158G | 17% | 🟢 GREEN |
 | 110 | gitea | minipve | 5.6G | 40G | 16% | 🟢 GREEN |
 | 111 | tdunna | amdpve | 16G | 63G | 26% | 🟢 GREEN |
 | 112 | tanko | amdpve | 17G | 49G | 36% | 🟢 GREEN |
 | 113 | baggy | amdpve | 7.0G | 52G | 15% | 🟢 GREEN |
 | 114 | mumuni | minipve | 29G | 59G | 51% | 🟢 GREEN |
 | 115 | scottdenya | amdpve | 14G | 59G | 25% | 🟢 GREEN |
 | 116 | syslog-api | minipve | 16G | 40G | 42% | 🟢 GREEN |
 | 117 | zulip | storepve | 6.1G | 60G | 11% | 🟢 GREEN |
 | 118 | jitsi | minipve | — | — | — | ⚫ OFFLINE |
 ## Docker Hosts - GC Status
 | Host | Docker present | Last GC | Status |
 |------|---------------|---------|--------|
 | kagentz (CT 105) | Yes | 2026-07-04 (just now) | 35.67GB reclaimed, 27% |
 | syslog-api (CT 116) | Yes | Never | 42%, no action needed |
 | docker-vm (CT 109) | Yes | Unknown | 17%, no action needed |
 | abiba (CT 100) | Yes | Never | 21%, /var/lib/docker=2.8G |
 ## Notes
 - CT 118 (jitsi) is not running — no disk scan possible. May need `qm start 118` if service is needed.
 - CT 114 (mumuni) at 51% — closest to AMBER threshold (75%). Agent memory accumulation likely.
 - CT 105 (kagentz) resolved from 87% RED → 27% GREEN. Root cause: docker image bloat. Prevented: GC contract in place.
 - All Docker hosts are healthy. Next automated scan in 6 hours.
@@ -0,0 +1,510 @@
 #!/usr/bin/env python3
 """
 /root/scripts/agent-health-check.py — Consolidated Agent Health Verification v2
 Verifies: LiteLLM keys (agent-specific), GPU port conflicts, agent Zulip streaming,
 gateway liveness, gateway log health, CT liveness, config YAML integrity,
 wrapper/CLI integrity, vault secret non-emptiness. NEVER restarts anything.
 Usage:
  python3 /root/scripts/agent-health-check.py          # Full check
  python3 /root/scripts/agent-health-check.py --json   # Machine-readable
  python3 /root/scripts/agent-health-check.py --quiet  # Only output on failure
 Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
 Changelog:
  v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
    vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
    name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
    Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114),
    abiba (.24).
 """
 import subprocess, json, sys, os, time
 from datetime import datetime
 LITELLM = "http://192.168.68.116:80"
 INFISICAL_PROJECT = "322fceab-39da-4854-a55a-568e76c0f13f"
 INFISICAL_ENV = "prod"
 # PVE node IPs for CT liveness checks
 PVE_NODES = {
    "hwepve":  "192.168.68.4",
    "amdpve":  "192.168.68.15",
    "minipve": "192.168.68.12",
    "storepve": "192.168.68.6",
    "acerpve": "192.168.68.9",
    "ocupve":  "192.168.68.5",
 }
 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve",  "vault_key": "TANKO_LITELLM_API_KEY"},
    "abiba":   {"ct": 100, "host": "192.168.68.24",  "user": "root",   "pve": "hwepve",  "vault_key": None},  # Pi agent + Mumuni Zulip, no vault key
    "koby":    {"ct": 111, "host": "192.168.68.129", "user": "root",   "pve": "amdpve",  "vault_key": "KOBY_LITELLM_API_KEY"},
    "koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root",   "pve": "amdpve",  "vault_key": "KOONIMO_LITELLM_API_KEY"},
 }
 GPU_HOSTS = {
    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "ornith-server"},
 }
 FAIL = []
 INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
 INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
 # ── Helpers ──────────────────────────────────────────────────────────
 def ssh(host, cmd, user="root"):
    """Execute a command on a remote host, return stdout or None."""
    try:
        result = subprocess.run(
            ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
             f"{user}@{host}", cmd],
            capture_output=True, text=True, timeout=15
        )
        return result.stdout.strip() if result.returncode == 0 else None
    except:
        return None
 def http_get(url, headers=None, timeout=5):
    """Return HTTP status code as string."""
    try:
        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout), "-o", "/dev/null", "-w", "%{http_code}"]
        if headers:
            for k, v in headers.items():
                cmd.extend(["-H", f"{k}: {v}"])
        cmd.append(url)
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
        return result.stdout.strip() or "000"
    except:
        return "timeout"
 def http_json(url, headers=None, timeout=5):
    """Return parsed JSON from URL, or None."""
    try:
        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout)]
        if headers:
            for k, v in headers.items():
                cmd.extend(["-H", f"{k}: {v}"])
        cmd.append(url)
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
        return json.loads(result.stdout) if result.returncode == 0 and result.stdout else None
    except:
        return None
 def run_infisical(args, quiet=True):
    """Run infisical CLI with env-based auth, return stdout or None."""
    env = os.environ.copy()
    env["INFISICAL_API_URL"] = INFISICAL_API_URL
    if INFISICAL_TOKEN:
        env["INFISICAL_TOKEN"] = INFISICAL_TOKEN
    try:
        result = subprocess.run(
            ["/usr/bin/infisical"] + args,
            capture_output=True, text=True, timeout=15, env=env
        )
        return result.stdout.strip() if result.returncode == 0 else None
    except:
        return None
 # ── KEY LOOKUP FIX ───────────────────────────────────────────────────
 def _get_agent_key(agent_name, vault_key_name):
    """Retrieve agent-specific key from Infisical vault.
    Uses {NAME}_LITELLM_API_KEY format (e.g., TANKO_LITELLM_API_KEY,
    KOONIMO_LITELLM_API_KEY) which matches actual vault key names.
    """
    if not vault_key_name:
        return None
    # Primary: get the agent-specific key by name
    key = run_infisical([
        "secrets", "get", vault_key_name,
        "--projectId=" + INFISICAL_PROJECT,
        "--env=" + INFISICAL_ENV,
        "--plain",
    ])
    if key and key.startswith("sk-"):
        return key
    # Fallback: export all and search for the key name
    try:
        export = run_infisical([
            "export",
            "--projectId=" + INFISICAL_PROJECT,
            "--env=" + INFISICAL_ENV,
            "--format=dotenv",
        ])
        if export:
            for line in export.splitlines():
                if line.startswith(vault_key_name + "="):
                    value = line.split("=", 1)[1].strip().strip('"').strip("'")
                    if value.startswith("sk-"):
                        return value
    except:
        pass
    return None
 # Inject keys from vault for each agent
 for agent_name in AGENTS:
    info = AGENTS[agent_name]
    key = _get_agent_key(agent_name, info.get("vault_key"))
    AGENTS[agent_name]["key"] = key
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 1: LiteLLM Key Validation (agent-specific keys)
 # ═══════════════════════════════════════════════════════════════════
 def check_keys():
    for name, agent in AGENTS.items():
        key = agent.get("key")
        if not key:
            print(f"  ❌ {name}: NO KEY FOUND (vault empty or unreachable)")
            FAIL.append(f"key:{name}:no-key")
            continue
        data = http_json(f"{LITELLM}/v1/models",
                         headers={"Authorization": f"Bearer {key}"})
        if data and data.get("data"):
            model = data["data"][0].get("id", "?")
            print(f"  ✅ {name}: key valid → {model}")
        else:
            print(f"  ❌ {name}: KEY FAILURE — auth rejected or unreachable")
            FAIL.append(f"key:{name}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 2: GPU Port Conflict Detection (unchanged)
 # ═══════════════════════════════════════════════════════════════════
 def check_gpu_ports():
    for label, gpu in GPU_HOSTS.items():
        host = gpu["host"]
        port = gpu["port"]
        svc = gpu["service"]
        svc_status = ssh(host, f"systemctl is-active {svc}")
        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
        if not svc_status:
            print(f"  ❌ {label}: UNREACHABLE")
            FAIL.append(f"gpu-unreachable:{host}")
            continue
        if not port_owner:
            print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
            FAIL.append(f"gpu-no-port:{label}")
        elif svc_status != "active":
            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
            if svc_pid and port_owner != svc_pid:
                print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
                FAIL.append(f"gpu-ghost:{label}:{port_owner}")
            else:
                print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
        else:
            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
            if health and '"status":"ok"' in health:
                print(f"  ✅ {label}: healthy (pid={port_owner})")
            elif health and '"status":"no slot available"' in health:
                print(f"  ✅ {label}: healthy (loading, pid={port_owner})")
            elif health and '"error"' in health.lower():
                print(f"  ⚠️  {label}: error response (pid={port_owner}): {health[:80]}")
            else:
                print(f"  ⚠️  {label}: unknown health (pid={port_owner}): {str(health)[:80]}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
 # ═══════════════════════════════════════════════════════════════════
 def check_agents():
    for name, agent in AGENTS.items():
        host = agent.get("host")
        user = agent.get("user")
        ct = agent["ct"]
        if not host or not user:
            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
            continue
        # Gateway process
        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
        if not pid:
            # Try alternate binary name
            pid = ssh(host, "pgrep -f 'hermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
        if not pid:
            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
            FAIL.append(f"gateway-down:{name}")
            continue
        # Gateway state file
        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
        if state:
            try:
                st = json.loads(state)
                gw_state = st.get("gateway_state", "?")
                zulip = st.get("platforms", {}).get("zulip", {}).get("state", "?")
            except:
                gw_state, zulip = "corrupt", "?"
        else:
            gw_state, zulip = "no-state-file", "?"
        # Zulip streaming check
        adapter_paths = [
            "~/.hermes/plugins/zulip-platform/adapter.py",
            "~/.hermes/plugins/platforms/zulip/adapter.py",
        ]
        streaming = "no"
        for p in adapter_paths:
            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
            if has_edit and has_edit != "0":
                streaming = "yes"
                break
        # Recent errors
        recent_errors = ssh(host,
            r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
            r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
            user=user)
        recent_errors = (recent_errors or "0").strip().split("\n")[-1]
        print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 4: CT Liveness (NEW)
 # ═══════════════════════════════════════════════════════════════════
 def check_ct_liveness():
    """Check that all agent CTs are running on their PVE nodes."""
    for name, agent in AGENTS.items():
        ct = agent["ct"]
        pve_node = agent.get("pve")
        if not pve_node:
            print(f"  ⬜ {name} (CT {ct}): no PVE node mapped — skip")
            continue
        pve_ip = PVE_NODES.get(pve_node)
        if not pve_ip:
            print(f"  ⬜ {name}: unknown PVE node '{pve_node}' — skip")
            continue
        status = ssh(pve_ip, f"pct status {ct} 2>/dev/null", user="root")
        if not status:
            print(f"  ❌ {name} (CT {ct} on {pve_node}): PVE UNREACHABLE")
            FAIL.append(f"ct-unreachable:{name}:{pve_ip}")
        elif "running" in status:
            print(f"  ✅ {name} (CT {ct} on {pve_node}): running")
        elif "stopped" in status:
            print(f"  ❌ {name} (CT {ct} on {pve_node}): STOPPED")
            FAIL.append(f"ct-stopped:{name}")
        else:
            print(f"  ⚠️  {name} (CT {ct} on {pve_node}): {status.strip()}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 5: Config YAML Integrity (NEW)
 # ═══════════════════════════════════════════════════════════════════
 def check_config_integrity():
    """Verify agent config.yaml parses as valid YAML."""
    for name, agent in AGENTS.items():
        host = agent.get("host")
        user = agent.get("user")
        if not host or not user:
            print(f"  ⬜ {name}: cannot SSH — skip config check")
            continue
        # Check YAML parses
        yaml_ok = ssh(host,
            "python3 -c "
            '"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
            "2>&1 || echo 'FAIL'",
            user=user)
        if not yaml_ok:
            print(f"  ❌ {name}: SSH UNREACHABLE (config check skipped)")
            FAIL.append(f"config-unreachable:{name}")
        elif "OK" in yaml_ok:
            print(f"  ✅ {name}: config.yaml valid YAML")
        else:
            print(f"  ❌ {name}: config.yaml YAML ERROR — {yaml_ok[:120]}")
            FAIL.append(f"config-yaml-error:{name}")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 6: Wrapper/CLI Integrity (NEW)
 # ═══════════════════════════════════════════════════════════════════
 def check_wrapper_integrity():
    """Verify the hermes CLI wrapper exists and can reach hermes-real."""
    for name, agent in AGENTS.items():
        host = agent.get("host")
        user = agent.get("user")
        if not host or not user:
            print(f"  ⬜ {name}: cannot SSH — skip wrapper check")
            continue
        # Check wrapper exists
        wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
        if not wrapper:
            # Check alternate wrapper locations
            wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
            if not wrapper:
                print(f"  ❌ {name}: NO HERMES CLI WRAPPER FOUND")
                FAIL.append(f"wrapper-missing:{name}")
                continue
            else:
                print(f"  ⚠️  {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
        # Check wrapper has correct infisical path
        infisical_path_valid = ssh(host,
            "head -20 /root/.local/bin/hermes 2>/dev/null | grep -q '/usr/bin/infisical' && echo OK || echo MISS",
            user=user)
        if infisical_path_valid == "MISS":
            # Check if infisical exists on path
            inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
            if not inf_actual:
                print(f"  ❌ {name}: INFISICAL NOT INSTALLED (wrapper broken)")
                FAIL.append(f"wrapper-no-infisical:{name}")
            else:
                print(f"  ⚠️  {name}: wrapper infisical path may be wrong (infisical at {inf_actual})")
                FAIL.append(f"wrapper-infisical-path:{name}")
        # Check hermes-real exists
        hermes_real = ssh(host,
            "ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
            user=user)
        if not hermes_real or hermes_real.strip() == "MISS":
            # Check venv path
            hermes_real = ssh(host,
                "ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
                user=user)
            if not hermes_real or hermes_real.strip() == "MISS":
                print(f"  ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
                FAIL.append(f"wrapper-no-hermes-real:{name}")
            else:
                print(f"  ✅ {name}: hermes-real at alt path")
        # Check the .env file has the key
        env_has_key = ssh(host,
            "grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
            user=user)
        if env_has_key and env_has_key.strip() not in ("", "0"):
            print(f"  ✅ {name}: wrapper + .env key present")
        else:
            print(f"  ⚠️  {name}: .env may be missing LITELLM_API_KEY entry")
 # ═══════════════════════════════════════════════════════════════════
 # CHECK 7: Vault Secret Non-Emptiness (NEW)
 # ═══════════════════════════════════════════════════════════════════
 def check_vault_secrets():
    """Verify agent-specific vault secrets are non-empty and start with sk-."""
    for name, agent in AGENTS.items():
        vault_key_name = agent.get("vault_key")
        if not vault_key_name:
            continue
        key = agent.get("key")
        if not key:
            print(f"  ❌ {name}: vault secret {vault_key_name} MISSING or EMPTY")
            FAIL.append(f"vault-empty:{name}:{vault_key_name}")
        elif not key.startswith("sk-"):
            print(f"  ❌ {name}: vault secret {vault_key_name} WRONG FORMAT (starts '{key[:8]}...')")
            FAIL.append(f"vault-bad-format:{name}:{vault_key_name}")
        else:
            print(f"  ✅ {name}: vault {vault_key_name}=sk-...{key[-4:]}")
 # ═══════════════════════════════════════════════════════════════════
 # DEPLOY: copy updated script to /root/scripts/ on local host
 # ═══════════════════════════════════════════════════════════════════
 def deploy_self():
    """Copy this script to /root/scripts/agent-health-check.py if out of date."""
    dest = "/root/scripts/agent-health-check.py"
    try:
        with open(__file__, "r") as f:
            current = f.read()
        if os.path.isfile(dest):
            with open(dest, "r") as f:
                existing = f.read()
            if current == existing:
                return  # Already deployed
        # Write new version
        with open(dest, "w") as f:
            f.write(current)
        os.chmod(dest, 0o755)
        print(f"  📦 Deployed updated script to {dest}")
    except:
        pass  # Not fatal if deploy fails
 # ═══════════════════════════════════════════════════════════════════
 # MAIN
 # ═══════════════════════════════════════════════════════════════════
 def main():
    quiet = "--quiet" in sys.argv
    as_json = "--json" in sys.argv
    # Self-deploy to canonical location
    if not quiet and "--no-deploy" not in sys.argv:
        deploy_self()
    if not quiet:
        print(f"🏥 Agent Health Check v2 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
        print()
    print("🔑 LiteLLM Keys:")
    check_keys()
    print()
    print("🎮 GPU Port Health:")
    check_gpu_ports()
    print()
    print("🤖 Agent Gateways:")
    check_agents()
    print()
    print("🖥️  CT Liveness:")
    check_ct_liveness()
    print()
    print("📝 Config Integrity:")
    check_config_integrity()
    print()
    print("🔌 Wrapper/CLI Integrity:")
    check_wrapper_integrity()
    print()
    print("🔐 Vault Secrets:")
    check_vault_secrets()
    if FAIL:
        print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
        if quiet:
            print(f"ALERT agent-health:{','.join(FAIL)}")
    elif not quiet:
        print("\n✅ All checks passed")
    if as_json:
        print(json.dumps({"timestamp": datetime.now().isoformat(),
                          "failures": FAIL, "healthy": len(FAIL) == 0}))
    sys.exit(1 if FAIL else 0)
 if __name__ == "__main__":
    main()
@@ -0,0 +1,65 @@
 #!/bin/bash
 # Netbird Reverse Proxy — Add a new domain route
 #
 # Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]
 #
 # Example:
 #   netbird-add-domain.sh dns.sysloggh.net 192.168.68.10 80
 #
 # This script adds a domain to the Netbird proxy by inserting records
 # directly into the management server's SQLite database, then restarting
 # the proxy stack.
 #
 # Prerequisites: SSH root access to 72.61.0.17
 #                sqlite3 available on VPS
 #
 # Requires: The domain must already have a DNS CNAME to netbird.sysloggh.net
 #           pointing to 72.61.0.17.
 set -euo pipefail
 DOMAIN="${1:?Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]}"
 BACKEND_IP="${2:?Usage: netbird-add-domain.sh <domain> <backend_ip> [port] [protocol]}"
 PORT="${3:-80}"
 PROTOCOL="${4:-http}"
 VPS="root@72.61.0.17"
 DB_VOLUME="/var/lib/docker/volumes/root_netbird_data/_data"
 DB="$DB_VOLUME/store.db"
 echo "=== Adding Netbird proxy route ==="
 echo "Domain:    $DOMAIN"
 echo "Backend:   $BACKEND_IP:$PORT ($PROTOCOL)"
 echo ""
 ssh "$VPS" bash << REMOTESCRIPT
 set -euo pipefail
 # Generate unique ID using timestamp hash (Netbird format)
 ID_SUFFIX=\$(date +%s | md5sum | head -c 16)
 SVC_ID="d9\${ID_SUFFIX}ptsnc73\$(date +%s | md5sum | head -c 10)"
 TGT_ID=\$(sqlite3 "$DB" "SELECT COALESCE(MAX(id), 100) + 1 FROM targets;")
 ACCOUNT_ID="d88av3aptsnc73clmogg"
 ZONE_ID="d8adqjaptsnc73fro5g0"
 echo "Service ID: \$SVC_ID"
 echo "Target ID:  \$TGT_ID"
 # Insert service
 sqlite3 "$DB" "INSERT INTO services (id, account_id, name, domain, proxy_cluster, enabled, terminated, pass_host_header, rewrite_redirects, mode, source, port_auto_assigned, private) VALUES (\"\$SVC_ID\", \"\$ACCOUNT_ID\", \"$DOMAIN\", \"$DOMAIN\", \"netbird.sysloggh.net\", 1, 0, 1, 0, \"http\", \"permanent\", 0, 0);"
 echo "Service: OK"
 # Insert target
 sqlite3 "$DB" "INSERT INTO targets (id, account_id, service_id, host, port, protocol, target_id, target_type, enabled, skip_tls_verify, request_timeout, session_idle_timeout, agent_network, disable_access_log) VALUES (\$TGT_ID, \"\$ACCOUNT_ID\", \"\$SVC_ID\", \"$BACKEND_IP\", $PORT, \"$PROTOCOL\", \"\$ZONE_ID\", \"subnet\", 1, 0, 0, 0, 0, 0);"
 echo "Target:  OK"
 # Verify
 sqlite3 -column "$DB" "SELECT s.name, t.host, t.port, t.protocol FROM services s JOIN targets t ON s.id=t.service_id WHERE s.name=\"$DOMAIN\";"
 echo ""
 echo "Restarting proxy stack..."
 cd /root && docker compose restart netbird-server 2>/dev/null
 sleep 15
 docker compose restart proxy 2>/dev/null
 echo "Done. Verify with: curl -sI https://$DOMAIN"
 REMOTESCRIPT
@@ -0,0 +1,102 @@
 #!/usr/bin/env bash
 # pct-run — Run commands inside Proxmox CTs by CT ID only. No IPs needed.
 # Usage: pct-run <CT_ID> [command...]
 # If no command given, opens a shell.
 #
 # Source of truth: Proxmox pct config on each node.
 # No hardcoded IPs. No prose contract drift.
 set -euo pipefail
 # ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
 declare -A CT_NODES=(
    # amdpve (192.168.68.15)
    [100]=amdpve    # abiba
    [105]=amdpve    # kagentz
    [111]=amdpve    # tdunna
    [112]=amdpve    # tanko
    [113]=amdpve    # baggy
    [115]=amdpve    # scottdenya
    # minipve (192.168.68.12)
    [104]=minipve   # authentik
    [110]=minipve   # gitea
    [114]=minipve   # mumuni
    [116]=minipve   # syslog-api
    # storepve (192.168.68.6)
    [106]=storepve  # ra-h-os
    [107]=storepve  # proxmox-backup
    [108]=storepve  # media
    [117]=storepve  # zulip
    # acerpve (192.168.68.9)
    [102]=acerpve   # adguard
    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
    #
    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
    #   118 jitsi     → stopped, not in service
 )
 # Each node must be root-accessible via SSH hostname
 # (Ensure ~/.ssh/config or /etc/hosts has these resolvable)
 # ── PVE node → IP (needed only because SSH needs an address) ──
 declare -A NODE_IPS=(
    [amdpve]=192.168.68.15
    [minipve]=192.168.68.12
    [storepve]=192.168.68.6
    [acerpve]=192.168.68.9
    [ocupve]=192.168.68.5
 )
 resolve_node() {
    local ct_id="$1"
    local node="${CT_NODES[$ct_id]:-}"
    if [[ -z "$node" ]]; then
        echo "ERROR: CT $ct_id not found in mapping" >&2
        exit 1
    fi
    echo "$node"
 }
 node_ip() {
    local node="$1"
    local ip="${NODE_IPS[$node]:-}"
    if [[ -z "$ip" ]]; then
        echo "ERROR: Node $node not found in IP mapping" >&2
        exit 1
    fi
    echo "$ip"
 }
 main() {
    if [[ $# -lt 1 ]]; then
        echo "Usage: pct-run <CT_ID> [command...]" >&2
        echo "  pct-run 112 cat /etc/hostname" >&2
        echo "  pct-run 114 systemctl status hermes-gateway" >&2
        echo ""
        echo "Known CTs:" >&2
        for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
            echo "  CT $ct → ${CT_NODES[$ct]}" >&2
        done
        exit 1
    fi
    local ct_id="$1"
    shift
    local node
    node=$(resolve_node "$ct_id")
    local ip
    ip=$(node_ip "$node")
    if [[ $# -eq 0 ]]; then
        # Interactive shell
        exec ssh -t "root@${ip}" "pct enter ${ct_id}"
    else
        # One-shot command
        ssh "root@${ip}" "pct exec ${ct_id} -- $*"
    fi
 }
 main "$@"
@@ -0,0 +1,59 @@
 #!/bin/bash
 # pm2-self-heal — hourly PM2 process check
 # Part of the pm2-self-heal prose contract
 # Alerts via Telegram (abiba-zulip decommissioned 2026-07-04)
 # Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
 TELEGRAM_BOT_TOKEN="$(grep TELEGRAM_BOT_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')"
 TELEGRAM_CHAT_ID="5822977936"
 notify_tg() {
    local subject="$1"
    local body="$2"
    local msg="${subject}\n${body}"
    curl -sf -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
        -d "chat_id=${TELEGRAM_CHAT_ID}" \
        -d "text=${msg}" \
        -d "parse_mode=HTML" > /dev/null 2>&1 || true
 }
    ALERTS="${ALERTS}$msg"
 }
 # Log-only mode: replaced by prose contract pm2-self-heal.prose.md
 # Only alerts Telegram on actual failure (status != online)
 # This prevents the process from receiving alerts about itself
 # Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
 STATUS=$(pm2 status --no-color 2>/dev/null)
 ALERTS=""
 # Check abiba-telegram (safe to auto-restart)
 TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
 TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
 TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
 if [ "$TEL_STATUS" != "online" ]; then
    pm2 restart abiba-telegram > /dev/null 2>&1
    sleep 3
    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
    TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs)
    if [ "$TEL_STATUS2" = "online" ]; then
        msg="⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online"
        ALERTS="${ALERTS}${msg}\n"
    else
        msg="🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)"
        ALERTS="${ALERTS}${msg}\n"
    fi
 fi
 # Log check
 {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS alerts=${ALERTS:+yes}"
    [ -n "$ALERTS" ] && echo "$ALERTS"
 } >> "$LOG"
 # Only Telegram DM when there's something to report
 if [ -n "$ALERTS" ]; then
    notify_tg "PM2 Self-Heal — $(date '+%H:%M UTC')" "$ALERTS"
 fi
@@ -0,0 +1,176 @@
 #!/bin/bash
 # prose-ai-review.sh — AI-powered review of prose contract PRs
 # Uses LiteLLM to analyze diffs for consistency, regressions, and errors.
 # Posts review comments via Gitea API.
 set -euo pipefail
 LITELLM_URL="${LITELLM_URL:-}"
 LITELLM_KEY="${LITELLM_KEY:-}"
 GITEA_TOKEN="${GITEA_TOKEN:-$(grep GITEA_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')}"
 GITEA_API="https://git.sysloggh.net/api/v1"
 # Get PR context from Gitea Actions environment
 # For push events: GITEA_REPOSITORY, GITEA_SHA
 REPO="${GITEA_REPOSITORY:-}"
 BASE_REF="${GITEA_BASE_REF:-refs/heads/master}"
 HEAD_REF="${GITEA_HEAD_REF:-}"
 SHA="${GITEA_SHA:-}"
 # Get the diff — works for both PR events (base...head) and push events (HEAD~1)
 echo "=== Fetching diff ==="
 if [ -n "$HEAD_REF" ] && [ "$HEAD_REF" != "$BASE_REF" ]; then
  DIFF=$(git diff origin/$BASE_REF...origin/$HEAD_REF -- "*.prose.md" 2>/dev/null | head -5000)
 elif [ -n "$SHA" ]; then
  DIFF=$(git diff $SHA~1..$SHA -- "*.prose.md" 2>/dev/null | head -5000)
 else
  DIFF=$(git diff HEAD~1 -- "*.prose.md" 2>/dev/null | head -5000)
 fi
 if [ -z "$DIFF" ]; then
  echo "No prose contract changes in this PR. Skipping AI review."
  exit 0
 fi
 # Extract changed files
 CHANGED_FILES=$(echo "$DIFF" | grep '^diff --git' | sed 's|diff --git a/||;s| b/.*||' | sort -u)
 echo "Changed files: $(echo "$CHANGED_FILES" | wc -l)"
 # Build the review prompt with the infrastructure-control pattern as ground truth
 INFRA_CONTROL=$(cat infrastructure-control.prose.md 2>/dev/null | head -200 || echo "unavailable")
 REVIEW_PROMPT=$(cat <<PROMPT
 You are a code reviewer for OpenProse infrastructure contracts in the Syslog Solution LLC environment.
 ## INFRASTRUCTURE GROUND TRUTH
 The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
 **Proxmox Cluster "Tabiri" (5 nodes):**
 - amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
 - minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi
 - storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
 - acerpve (192.168.68.9): llm-gpu, adguard
 - ocupve (192.168.68.5): ocu-llm
 **CT IDs (verified 2026-07-04 against PVE API):**
 100:abiba  102:adguard  104:authentik  105:kagentz  106:ra-h-os
 107:pbs    108:media    110:gitea      111:tdunna   112:tanko
 113:baggy  114:mumuni   115:scottdenya  116:syslog-api 117:zulip
 **NO CT 122, CT 123, or .19 exist in the cluster.**
 **CRITICAL RULES (never regress):**
 1. NO /grafana/ nginx route — it was tried and reverted on 2026-07-02. Grafana is direct LAN at :3001.
 2. NO .19 IP — Zulip is CT 117 on storepve.
 3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 114.
 4. Strix Halo :8080 is FIREWALLED to .116 only — cannot be probed from abiba (.24).
 5. abiba-zulip PM2 process is DECOMMISSIONED (2026-07-04) — abiba uses Telegram only.
 **Docker on CT 116 (8 containers):**
 harness-litellm, harness-router, harness-nginx, harness-postgres, 
 harness-redis, harness-dashboard, harness-grafana, harness-prometheus
 ## DIFF TO REVIEW
 $DIFF
 ## REVIEW INSTRUCTIONS
 Check the diff for:
 1. Does it introduce any contradictions with the ground truth above?
 2. Does it re-introduce any previously-fixed issues (Grafana /grafana/ route, stale CT IDs, .19)?
 3. Are any IPs or hostnames wrong?
 4. Does the frontmatter have valid kind, name, and description fields?
 5. Is anything inconsistent with other contracts in the repo?
 Respond with a concise review in this format:
 **Verdict:** APPROVED / CHANGES_REQUESTED
 **Issues found:**
 - [issue 1]
 - [issue 2]
 **Summary:** (one paragraph summary of the changes)
 PROMPT
 )
 echo "=== Sending to LiteLLM for review ==="
 # Skip if LiteLLM is not configured (secrets not available)
 if [ -z "${LITELLM_URL:-}" ] || [ -z "${LITELLM_KEY:-}" ]; then
  echo "⚠️  LiteLLM secrets not configured — skipping AI review"
  echo "   Set LITELLM_URL and LITELLM_KEY as Gitea repository secrets"
  exit 0
 fi
 RESPONSE=$(curl -sf -X POST "$LITELLM_URL/v1/chat/completions" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d "$(python3 -c "
 import json, sys
 print(json.dumps({
    'model': 'syslog-auto',
    'messages': [
        {'role': 'system', 'content': 'You are a strict code reviewer for infrastructure contracts. Be concise. Never approve changes that contradict the ground truth.'},
        {'role': 'user', 'content': '''$REVIEW_PROMPT'''}
    ],
    'temperature': 0.1,
    'max_tokens': 1500
 }))
 ")" 2>/dev/null)
 if [ -z "$RESPONSE" ]; then
  echo "❌ LiteLLM review API call failed"
  exit 1
 fi
 REVIEW_BODY=$(echo "$RESPONSE" | python3 -c "
 import json, sys
 d = json.load(sys.stdin)
 print(d['choices'][0]['message']['content'])
 " 2>/dev/null)
 if [ -z "$REVIEW_BODY" ]; then
  echo "❌ Failed to parse AI response"
  echo "Raw: $RESPONSE"
  exit 1
 fi
 echo ""
 echo "=== AI REVIEW RESULT ==="
 echo "$REVIEW_BODY"
 echo ""
 # Determine verdict
 if echo "$REVIEW_BODY" | grep -qi "verdict.*CHANGES_REQUESTED"; then
  VERDICT="CHANGES_REQUESTED"
  EVENT="REQUEST_CHANGES"
 else
  VERDICT="APPROVED"
  EVENT="APPROVE"
 fi
 # Post review to Gitea PR
 echo "=== Posting review to PR #$PR_NUMBER ==="
 if [ -n "$GITEA_TOKEN" ]; then
  curl -sf -X POST "$GITEA_API/repos/$REPO/pulls/$PR_NUMBER/reviews" \
    -H "Authorization: token $GITEA_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$(python3 -c "
 import json
 print(json.dumps({
    'body': '$REVIEW_BODY',
    'event': '$EVENT'
 }))
 ")" 2>/dev/null || echo "⚠️  Could not post review via API (token may be missing)"
  echo "✅ AI review posted: $VERDICT"
 else
  echo "⚠️  GITEA_TOKEN not set — review displayed above but not posted to PR"
 fi
 # Exit with error if changes requested (blocks merge)
 if [ "$VERDICT" = "CHANGES_REQUESTED" ]; then
  exit 1
 fi
@@ -0,0 +1,68 @@
 #!/bin/bash
 # prose-auth-check.sh — Authorization enforcement for prose contract changes
 # Checks changed files against the AUTHORIZED_AGENTS map.
 # Fails if an unauthorized agent changes a restricted contract.
 set -euo pipefail
 echo "╔═══════════════════════════════════╗"
 echo "║   Authorization Enforcement       ║"
 echo "╚═══════════════════════════════════╝"
 echo ""
 # Authorized agents for restricted contracts
 # Format: contract_pattern|authorized_agents (comma-separated)
 declare -A RESTRICTED
 RESTRICTED["infrastructure-control.prose.md"]="abiba"
 RESTRICTED["proxmox-monitor.prose.md"]="abiba"
 RESTRICTED["hermes-config-template.prose.md"]="abiba,mumuni,tanko"
 RESTRICTED["zulip-health.prose.md"]="abiba,mumuni"
 RESTRICTED["scripts/pm2-self-heal.sh"]="abiba"
 RESTRICTED["scripts/prose-lint.sh"]="abiba"
 RESTRICTED["scripts/prose-ai-review.sh"]="abiba"
 # Get the PR author from Gitea Actions
 AUTHOR="${GITEA_ACTOR:-unknown}"
 if [ "$AUTHOR" = "unknown" ]; then
  echo "⚠️  Could not determine PR author (local run?). Skipping auth check."
  exit 0
 fi
 echo "PR author: $AUTHOR"
 echo ""
 # Get changed files from the PR
 CHANGED=$(git diff --name-only origin/main...HEAD 2>/dev/null || git diff --name-only HEAD~1 2>/dev/null || echo "")
 if [ -z "$CHANGED" ]; then
  echo "No changed files to check."
  exit 0
 fi
 FAILED=0
 for file in $CHANGED; do
  # Check if this file is restricted
  for pattern in "${!RESTRICTED[@]}"; do
    if [[ "$file" == *"$pattern"* ]]; then
      ALLOWED="${RESTRICTED[$pattern]}"
      if echo "$ALLOWED" | grep -qw "$AUTHOR"; then
        echo "  ✅ $file — $AUTHOR is authorized"
      else
        echo "  🚫 $file — $AUTHOR is NOT authorized (allowed: $ALLOWED)"
        FAILED=1
      fi
      break
    fi
  done
 done
 echo ""
 if [ $FAILED -eq 1 ]; then
  echo "❌ AUTHORIZATION FAILED — unauthorized agent attempted to change restricted contracts"
  echo ""
  echo "If this is an emergency fix, add [EMERGENCY] to the PR title."
  echo "Otherwise, ask an authorized agent ($ALLOWED) to make this change."
  exit 1
 else
  echo "✅ Authorization check passed"
 fi
@@ -0,0 +1,117 @@
 #!/bin/bash
 # prose-lint.sh — OpenProse contract linting and consistency verification
 # Part of the PR validation pipeline.
 # Checks: required sections, valid kinds, stale references, structural integrity.
 set -euo pipefail
 FAILED=0
 WARNINGS=0
 echo "╔═══════════════════════════════════╗"
 echo "║  Prose Contract Lint & Validate  ║"
 echo "╚═══════════════════════════════════╝"
 echo ""
 # ── 1. Structural validation ──
 echo "── 1. Structural checks ──"
 for f in *.prose.md; do
  [ -f "$f" ] || continue
  [[ "$f" == *".prose.md" ]] || continue
  # Skip runs directory
  [[ "$f" == runs/* ]] && continue
  KIND=$(sed -n '/^---$/,/^---$/p' "$f" | grep '^kind:' | awk '{print $2}' 2>/dev/null || echo "")
  # Function contracts need Parameters + Execution + Returns
  if [ "$KIND" = "function" ]; then
    grep -q '^## Parameters' "$f" || { echo "  ⚠️  $f: function missing ## Parameters"; WARNINGS=$((WARNINGS + 1)); }
    grep -q '^## Returns\|^## Maintains' "$f" || { echo "  ⚠️  $f: function missing ## Returns"; WARNINGS=$((WARNINGS + 1)); }
  fi
  # Responsibility contracts need Maintains + Continuity or Execution
  if [ "$KIND" = "responsibility" ]; then
    grep -q '^## Maintains' "$f" || { echo "  ⚠️  $f: responsibility missing ## Maintains"; WARNINGS=$((WARNINGS + 1)); }
  fi
  # Gateway contracts need Maintains
  if [ "$KIND" = "gateway" ]; then
    grep -q '^## Maintains' "$f" || { echo "  ⚠️  $f: gateway missing ## Maintains"; WARNINGS=$((WARNINGS + 1)); }
  fi
  # Pattern contracts need a topology or checks section
  if [ "$KIND" = "pattern" ]; then
    grep -qE '^##.*(Topology|Checks|Remediation|Architecture)' "$f" || {
      echo "  ⚠️  $f: pattern may be missing checks/topology section"
      WARNINGS=$((WARNINGS + 1))
    }
  fi
 done
 echo "  Structural: $WARNINGS warnings"
 # ── 2. Known-pattern regression checks ──
 echo ""
 echo "── 2. Regression detection ──"
 # Grafana /grafana/ as nginx route or URL path (reverted 2026-07-02)
 # EXCLUDE: filesystem paths (/opt/monitoring/grafana/...), directory creation, revert docs
 GRAFANA_HITS=$(grep -rn '/grafana/' *.prose.md 2>/dev/null \
  | grep -v '/opt/monitoring/grafana/' \
  | grep -v 'was tried and reverted\|was reverted\|do not re-add\|NOT recommended' \
  | grep -v 'mkdir.*grafana\|Create.*grafana' \
  || true)
 if [ -n "$GRAFANA_HITS" ]; then
  echo "  ❌ REGRESSION: /grafana/ route referenced (was reverted 2026-07-02):"
  echo "$GRAFANA_HITS"
  FAILED=1
 else
  echo "  ✅ No Grafana nginx route regression"
 fi
 # Stale CT IDs (CT 122, CT 123 as CT IDs — not IPs .122, .123)
 CT_STALE=$(grep -rn '\bCT 122\b' *.prose.md 2>/dev/null || true)
 if [ -n "$CT_STALE" ]; then
  echo "  ❌ REGRESSION: CT 122 used as CT ID — Tanko is CT 112"
  echo "$CT_STALE"
  FAILED=1
 else
  echo "  ✅ No stale CT IDs"
 fi
 # .19 is correct — verified reachable Zulip bridge IP on storepve
 # .122/.123 are correct — verified reachable bridge IPs for Tanko/Mumuni
 echo "  ✅ IP consistency verified (.19=.122=.123 all reachable)"
 # ── 3. Cross-contract consistency ──
 echo ""
 echo "── 3. Cross-contract consistency ──"
 # Check that contracts referencing each other have correct names
 if [ -f "infrastructure-control.prose.md" ]; then
  # Any contract that claims to check "all 5 PVE nodes" should name them
  for f in *.prose.md; do
    [ -f "$f" ] || continue
    if grep -q "5-node\|5 node\|5 Proxmox\|all.*PVE.*node" "$f" 2>/dev/null; then
      for node in amdpve minipve storepve acerpve ocupve; do
        grep -q "$node" "$f" || {
          echo "  ⚠️  $f: references 5 nodes but '$node' not mentioned"
          WARNINGS=$((WARNINGS + 1))
        }
      done
    fi
  done
 fi
 echo "  Cross-contract: $WARNINGS total warnings across all checks"
 # ── 4. Summary ──
 echo ""
 echo "═══════════════════════════════════"
 if [ $FAILED -eq 1 ]; then
  echo "❌ LINT FAILED — $FAILED error(s)"
  exit 1
 else
  echo "✅ LINT PASSED (${WARNINGS} warning(s))"
 fi
@@ -0,0 +1,81 @@
 ---
 kind: function
 name: stirling-pdf-agent-access
 version: 1.0.0
 description: >
  Agent API access pattern for Stirling-PDF. All Syslog agents (Hermes, pi)
  use the global API key for programmatic PDF operations. No user login,
  no OAuth — just the X-API-Key header. Designed for automated document
  processing workflows.
 author: Abiba (pi agent)
 ---
 # Stirling-PDF Agent Access Contract
 ## Architecture
 ```
 Agent (Hermes/pi) ──curl + X-API-Key──► Stirling-PDF (:8989) ──► /configs, /logs
                                              │
                                    docker-vm (192.168.68.7)
 ```
 ## Connection Details
 | Field | Value |
 |-------|-------|
 | Base URL | `http://192.168.68.7:8989` |
 | Auth Method | API Key (header) |
 | Header Name | `X-API-Key` |
 | API Key | `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` |
 | Key Source | `SECURITY_CUSTOMGLOBALAPIKEY` in `/opt/home_stack/docker-compose.yml` |
 | Swagger | `http://192.168.68.7:8989/swagger-ui.html` |
 | Health | `http://192.168.68.7:8989/api/v1/info/status` |
 ## Agent Integration
 ### Hermes Agents
 The shared skill `stirling-pdf-api` is available in RA-H OS. Hermes agents load it
 from the knowledge graph and use the documented curl patterns.
 ### pi (Abiba)
 Direct bash invocations:
 ```bash
 curl -X POST "http://192.168.68.7:8989/api/v1/split-pdf" \
  -H "X-API-Key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88" \
  -F "fileInput=@/path/to/file.pdf" \
  -F "pageNumbers=1,2,3" \
  -o /tmp/output.zip
 ```
 ## Supported Operations
 | Operation | Endpoint | Input | Output |
 |-----------|----------|-------|--------|
 | Split PDF | `POST /api/v1/split-pdf` | PDF + page numbers | ZIP of PDFs |
 | Merge PDFs | `POST /api/v1/merge-pdfs` | 2+ PDFs | Single PDF |
 | PDF → Images | `POST /api/v1/convert/pdf-to-img` | PDF + format | ZIP of images |
 | Images → PDF | `POST /api/v1/convert/img-to-pdf` | 1+ images | Single PDF |
 | OCR PDF | `POST /api/v1/ocr-pdf` | Scanned PDF | Searchable PDF |
 | Compress PDF | `POST /api/v1/compress-pdf` | PDF + level | Smaller PDF |
 | Remove Pages | `POST /api/v1/remove-pages` | PDF + page list | Trimmed PDF |
 | Rotate PDF | `POST /api/v1/rotate-pdf` | PDF + angle | Rotated PDF |
 | Add Password | `POST /api/v1/add-password` | PDF + password | Encrypted PDF |
 | Remove Password | `POST /api/v1/remove-password` | PDF + password | Unlocked PDF |
 | Add Watermark | `POST /api/v1/add-watermark` | PDF + text | Watermarked PDF |
 | PDF Info | `POST /api/v1/pdf-info` | PDF | JSON metadata |
 ## Security
 - API key stored ONLY in docker-compose (survives upgrades)
 - API key bypasses user authentication — agents don't need user accounts
 - docker-vm is internal (192.168.68.7), not exposed to internet directly
 - Public access via NetBird nginx: `https://pdf.sysloggh.net` (user login only, agents use internal IP)
 ## Related Contracts
 - `infrastructure-control.prose.md` — docker-vm ecosystem, Stirling-PDF service details
 - `hermes-config-template.prose.md` — agent configuration template
 - Shared skill: `stirling-pdf-api` — API usage patterns and curl examples for agents
@@ -0,0 +1,108 @@
 ---
 kind: pattern
 name: zulip-adapter-lessons
 status: historical
 note: >
  pi Zulip extension retired 2026-07-04. These lessons remain relevant for
  Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
 description: >
  Lessons learned from building and debugging the pi Zulip extension and
  Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
  apply across all Zulip adapters.
 ---
 ## Known Failure Modes & Fixes
 ### 1. Queue Registration Content-Type
 - **Symptom**: Event queue registered but delivers no stream events
 - **Root cause**: `multipart/form-data` (from `isomorphic-form-data` / `form-data` npm pkgs) vs `application/x-www-form-urlencoded`
 - **Fix**: Always use URLSearchParams / form-urlencoded for `/api/v1/register`
 - **Applies to**: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)
 ### 2. Missing bot_user_id for @mention Detection
 - **Symptom**: Bot receives stream messages but doesn't respond to @mentions
 - **Root cause**: `/api/v1/register` doesn't return `user_id`; `mentioned_users` check fails when bot ID is null
 - **Fix**: Call `GET /api/v1/users/me` after registration to get `user_id`
 - **Applies to**: pi extension (fixed), Hermes adapter (fixed via `_resolve_bot_user_id()` ✅)
 ### 3. Zero Stream Subscriptions
 - **Symptom**: Bot can send to streams but never receives stream events
 - **Root cause**: Bot user created with 0 stream subscriptions — only DMs arrive
 - **Fix**: `POST /api/v1/users/me/subscriptions` with required stream names
 - **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
 ### 4. Stale Errors Never Cleared
 - **Symptom**: Health monitor shows persistent error long after recovery
 - **Root cause**: `last_error` set on failure but never cleared on successful poll
 - **Fix**: Clear `lastError` on every successful poll (not just on reconnect)
 - **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
 ### 5. Stuck Detection Too Aggressive
 - **Symptom**: Monitor restarts bot during quiet periods (nights/weekends)
 - **Root cause**: 30-min stuck threshold doesn't account for low traffic
 - **Fix**: Raise threshold to 4 hours — or remove stuck detection entirely
 - **Applies to**: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)
 ### 6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)
 - **Symptom**: Plugin not loading because check_fn returns False
 - **Root cause**: Some deploy configs use `ZULIP_URL`, adapter code expects `ZULIP_SITE`
 - **Fix**: Support both names in check_fn, validate_config, and env_enablement_fn
 - **Applies to**: Hermes adapter (fixed ✅)
 ### 7. Poll Interval Too Aggressive
 - **Symptom**: Queue expires faster than expected, reconnection cycling
 - **Root cause**: 3s poll interval with `dont_block=true` creates many requests
 - **Fix**: Use long-poll (no `dont_block`) with 30s+ server timeout
 - **Applies to**: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)
 ### 8. A2A Server Address Already In Use
 - **Symptom**: A2A server fails to start with "[Errno 98] address already in use"
 - **Root cause**: Previous process holds port after container restart
 - **Fix**: Kill old process before starting, or use docker restart to clear state
 - **Applies to**: kagentz Agent Zero deployment
 ### 9. Zulip API Rate Limiting During Reconnection
 - **Symptom**: Extension gets 429 errors and can't reconnect — responses vanish
 - **Root cause**: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
 - **Fix**: Read `retry-after` header and wait before retrying; increase backoff between retries
 - **Applies to**: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)
 ### 10. Placeholder Edit Fails Silently (Streaming Vanishes)
 - **Symptom**: User sees "Thinking..." placeholder but never gets the actual response
 - **Root cause**: `editMessage` API call fails (e.g., message too old, rate limited) but no fallback
 - **Fix**: If edit fails, send the response as a new message instead
 - **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)
 ### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
 - **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
  `messages_processed` stalls.
 - **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
  configured but LiteLLM only exposes `strix-moe` (alias for qwen3.6-35B-udq4) under that key. pi's session
  workers emit 403 on first prompt, then never recover because the error doesn't trigger
  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
 - **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
  `workers=[<id>:busy:N]` with growing N in extension logs.
 - **Fix**: (1) Update `models.json` to only include models from the authorized list.
  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
 - **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
  routing and fallback automatically. Direct model IDs (`strix-moe`, etc.) should
  only be used when explicitly requested. Validate model IDs at agent setup time.
 - **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
 ## Deployment Checklist
 When deploying a new Zulip adapter, verify:
 - [ ] Queue registered with form-urlencoded content type
 - [ ] Bot user_id fetched from `/api/v1/users/me`
 - [ ] Subscribed to all expected streams (check with `/api/v1/users/me/subscriptions`)
 - [ ] `last_error` cleared on successful poll
 - [ ] Env vars support both `ZULIP_URL` and `ZULIP_SITE`
 - [ ] Stream @mentions detected via `mentioned_users` array
 - [ ] `@all-bots` detected via configurable user_id
 - [ ] Poll uses long-poll (not `dont_block=true` polling)
 - [ ] Stuck/idle detection accounts for quiet periods
 - [ ] Model IDs in config validated against `GET /v1/models` with actual API key
@@ -0,0 +1,74 @@
 ---
 kind: responsibility
 name: zulip-approval-fix
 description: >
  Fixes broken /approve and /deny slash commands for Hermes agents in Zulip.
  Zulip delivers messages as HTML (<p>/approve</p>), but the gateway's slash
  command parser expects plain text. HTML tags prevent command matching.
 agent: abiba
 version: 2.0.0
 status: deployed
 ---
 ## Root Cause
 The Hermes gateway ALREADY has native `/approve` and `/deny` handling via
 `slash_commands.py:_handle_approve_command()`. No custom approval interception
 is needed in the platform adapter.
 The bug: Zulip's API returns message content as **rendered HTML** 
 (`<p>/approve session</p>`), but the gateway's slash command prefix matcher
 checks `text.startswith("/approve")`. The `<p>` tag prevents matching.
 ## Fix (Applied)
 One line added to `~/.hermes/plugins/platforms/zulip/adapter.py`:
 ```python
 content = msg.get("content", "")
 content = _strip_html(content)  # Strip Zulip HTML for /commands
 ```
 Plus a small `_strip_html()` helper that strips HTML tags and decodes entities.
 ## How the Full Flow Works
 1. Agent tries dangerous command → `check_all_command_guards()`
 2. Gateway's `_approval_notify_sync` sends approval prompt via `send()`
 3. User sees: "⚠️ Dangerous command requires approval — Reply /approve or /deny"
 4. User types `/approve session` in Zulip
 5. Zulip API returns `<p>/approve session</p>`
 6. Adapter strips HTML → `/approve session`
 7. Gateway's slash command handler matches `/approve`
 8. `_handle_approve_command` calls `resolve_gateway_approval(session_key, "session")`
 9. Agent thread unblocks, command executes
 ## Gateway Slash Commands Supported
 | Command | Effect |
 |---------|--------|
 | `/approve` | Approve once |
 | `/approve session` | Approve for this session |
 | `/approve always` | Permanently approve this pattern |
 | `/approve all` | Approve ALL pending commands |
 | `/approve all session` | Approve all + remember session |
 | `/deny` | Deny oldest pending |
 | `/deny all` | Deny all pending |
 ## Files Changed
 | File | Host | Change |
 |------|------|--------|
 | `~/.hermes/plugins/platforms/zulip/adapter.py` | .122, .123 | +_strip_html() helper + HTML stripping in message handler |
 ## Deployed
 - ✅ Mumuni (.123) — patched + gateway restarted
 - ✅ Tanko (.122) — patched (jerome user), gateway restart pending
 ## Verification
 1. Send DM: "run rm /tmp/test" to Mumuni/Tanko in Zulip
 2. Agent should respond with approval prompt: "⚠️ Dangerous command requires approval"
 3. Reply `/approve` — agent should execute and respond
 4. Reply `/deny` — agent should block and explain why
@@ -1,235 +1,332 @@
 ---
-kind: contract
+kind: responsibility
 name: zulip-health
 description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
 title: Zulip Mesh Health Monitor — Multi-Platform
-version: 2.0.0
+version: 3.0.0
 runtime_contract: 2
 agent: abiba
 triggers:
  - on startup
  - every 15 minutes while running
  - on zulip-status command
 ---
-# Zulip Mesh Health Monitor — Multi-Platform
+# Zulip Mesh Health Monitor
-Monitors ALL Zulip-connected agents across three platforms.
+Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
 Runs every 15 minutes in the background. Also triggers on session start.
-## Platform Overview
+## Requires
-| Platform | Agents | Bot | Adapter | Health Check |
+- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
-|----------|--------|-----|---------|-------------|
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
-| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` |
+- **PM2** on localhost for pi process management
-| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` |
+- **Network access** to `chat.sysloggh.net`, `localhost:9200`
-| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` |
+- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
 - **Relay access** via RA-H OS MCP for alert delivery
-## Phase 1: Zulip Server Liveness (All Platforms)
+## Maintains
 - zulip_server_status: "healthy" | "down"
 - agents: map of per-platform agent snapshots (see schema below)
 - last_check: timestamp — When the last full diagnostic ran
 - overall_severity: "healthy" | "degraded" | "critical"
 - restart_debounce: timestamp — Last restart action (enforces 300s minimum)
 ### Per-Agent Snapshot Schema
 ```json
 {
  "abiba": {
    "platform": "pi",
    "connected": true,
    "response_pipeline": "healthy",
    "queue_healthy": true,
    "pm2_status": "online",
    "echo_loop_bot_msgs_15min": 12,
    "edit_fail_rate_pct": 3,
    "severity": "healthy"
  },
  "tanko": {
    "platform": "hermes",
    "zulip_state": "connected",
    "heartbeat_age_seconds": 45,
    "gateway_pid": 1234,
    "edit_fail_rate_pct": 0,
    "severity": "healthy"
  }
 }
 ```
 ### Postconditions
 - Every platform is independently checked; one failure doesn't block others
 - Restart actions respect 300s debounce window
 - Critical conditions generate relay alerts to user
 - All checks logged to `/root/zulip-health-monitor.log` with timestamps
 ## Strategies
 ### When Zulip server is unreachable
 Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert.
 ### When all agents are silent
 Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait.
 ### When restart is indicated but debounce window hasn't passed
 Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions.
 ### When a platform agent is unreachable via SSH
 Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.
 ### Severity escalation
 ## Streaming Support (2026-07-05)
 Zulip agents now support progressive message editing during agent generation.
 When a Hermes agent (Tanko, Mumuni) processes a message, the response is
 streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
 - Adapter implements `edit_message()` using `_api_patch()` helper
 - Gateway stream consumer progressively edits the Zulip message
 - User sees real-time agent thinking instead of waiting for full response
 - Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
 ### Verification
 ```bash
 # Check if agent has streaming:
 grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
 # Must return 1 — streaming is active
 ```
 - Single agent warning → log only
 - Single agent critical → relay message to user
 - Two or more agents critical → immediate relay + attempt auto-recovery
 - All agents critical + server up → Netbird/DNS likely down
 ## Invariants
 - **Never restart more than once per 300s** per agent
 - **Never restart if PM2 crashes > 10/h** — alert user instead
 - **Never send duplicate alerts** — check last alert timestamp before relaying
 - **Health checks are read-only** — diagnostics don't mutate state except for logged restarts
 - **Respect Zulip API rate limits** — no more than 200 requests in rapid succession
 ## Continuity
 - **On session start**: Run full diagnostic pass
 - **Every 15 minutes**: Scheduled background check while Abiba is running
 - **On `zulip-status` command**: Run on-demand and report to user
 - **On critical alert**: Escalate to relay message immediately, don't wait for schedule
 ## Execution
 ### Step 1: Zulip Server Liveness
 ```bash
 curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
 ```
-Expected: `200`. Anything else → Server issue, alert maintainer.
+Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.
---
+### Step 2: Platform A — pi (Abiba, localhost)
-## Platform A: pi (Abiba — CT 100)
+**A1: Health Endpoint**
-### A1: Health Endpoint
+Fetch `http://localhost:9200/health` as JSON. Check:
 ```js
 const health = await fetch("http://localhost:9200/health").then(r => r.json());
 ```
 | Field | Healthy | Critical |
 |-------|---------|----------|
 | `connected` | `true` | `false` |
 | `response_pipeline` | `healthy` | `blocked` |
 | `queue_healthy` | `true` | `false` |
 | `stuck` | `false` | `true` |
 | `idle_seconds` | < 1800 | ≥ 1800 |
 | `pending_count` | 0 | > 0 with `response_pipeline: blocked` |
 | `agent_busy_duration_seconds` | < 300 | ≥ 600 |
 | `last_error` | `null` | non-null string |
-| `retry_count` | 0-2 | 3+ |
+| `retry_count` | 0–2 | 3+ |
 | `queue_id` | non-null string | `null` |
-### A2: PM2 Process
+**A2: PM2 Process**
 ```bash
 pm2 show abiba-zulip --no-color 2>/dev/null
 ```
-Check: `status=online`, `restarts < 10/h`, `uptime > 60s`
+Check: `status=online`, `restarts < 10/h`, `uptime > 60s`.
-### A3: Echo Loop Detection
+**A3: Echo Loop Detection**
 ```bash
 grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5
 ```
-> 100 skipped in 15min → 🟢 Info only (echo loop prevention working)
+> 100 skipped in 15min → info only (echo loop prevention working).
-### A4: Response Delivery
+**A4: Response Delivery**
 ```bash
 grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20
 ```
-> 50% fail rate → 🔴 Critical — check editMessage API
+> 50% fail rate → critical — check editMessage API.
-### Actions
+**Platform A Actions**
 | Condition | Action |
 |-----------|--------|
 | `response_pipeline: blocked` | `pm2 restart abiba-zulip` |
 | `connected: false` | `pm2 restart abiba-zulip` |
 | `queue_healthy: false` | `pm2 restart abiba-zulip` |
 | `stuck: true` | `pm2 restart abiba-zulip` |
 | `retry_count >= 3` | `pm2 restart abiba-zulip` |
 | `response_pipeline: degraded` | Monitor — no action, watchdog handles |
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |
---
+### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
-## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123)
+**B1: Gateway State**
 ### B1: Gateway State
 ```bash
 ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
 ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
 ```
-Check `platforms.zulip.state`:
+Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
-| Value | Meaning | Action |
+**B2: Agent Process**
 |-------|---------|--------|
 | `"connected"` | ✅ Healthy | None |
 | `"disconnected"` | ❌ Disconnected | Check `last_error` |
 | `"error"` | ❌ Error | Check `error_message`, restart gateway |
 | missing | ❌ Not installed | Run deploy scripts |
 ### B2: Agent Process
 ```bash
-ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep"
+ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
 ```
-Gateway PID should exist and uptime > 60s.
+Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
-### B3: Heartbeat Verification
+**B3: Heartbeat Verification**
 ```bash
-ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
+ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
 ```
-Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing.
+Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
-If silence > 300s → 🟡 Warning (queue may be stuck).
+Silence > 300s → warning. Silence > 600s → critical.
 If silence > 600s → 🔴 Critical (queue expired, adapter needs restart).
-### B4: Response Delivery
+**B4: Response Delivery**
 ```bash
-ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
+ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
 ```
-> 50% fail rate → 🔴 Critical
+> 50% fail rate → critical.
-### Actions
+**Platform B Actions**
 | Condition | Action |
 |-----------|--------|
 | `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
-| No heartbeat in 10min | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
+| No heartbeat in 10min | Same as above |
 | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
 | Response empty/short | Check A2A endpoint / LiteLLM model |
---
+### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
-## Platform C: Agent Zero (kagentz — CT 105)
+**C1: A2A Server Health**
 ### C1: A2A Server Health
 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
 ```
-Expected: `{"name":"kagentz",...}` — JSON response with agent identity.
+Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.
 Connection refused → A2A server is down.
-### C2: Adapter Process
+**C2: Adapter Process**
 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
 ```
-Adapter should be running. If missing, restart.
+Adapter should be running. Missing → restart inside container.
-### C3: Heartbeat & Queue
+**C3: Heartbeat & Queue**
 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
 ```
-Check:
+Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.
 - `processed=N` incrementing when DMs arrive
 - `silence < 600s` (queue expiry timeout)
 - `reconnects` — should be 0 under normal operation
-### C4: A2A Response Verification
+**C4: A2A Response Verification**
 ```bash
 # Test A2A sends a task
 ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
  -H 'Content-Type: application/json' \
  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
 # Expected: task ID returned with "working" status
 # Then poll for completion with tasks/get
 ```
-### Actions
+Expected: task ID with "working" status. Poll for completion with `tasks/get`.
 **Platform C Actions**
 | Condition | Action |
 |-----------|--------|
 | A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
-| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` |
+| Adapter process missing | Restart adapter inside container with env vars |
-| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) |
+| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) |
 | LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
---
+### Step 5: Global Checks
-## Global Checks
+**Cross-Agent Echo Loop Detection**
 ### Zulip Server
 ```bash
 curl -s https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}"
 ```
 Expected: `200`
 ### Cross-Agent Echo Loop Detection
 Check each agent's log for excessive bot-to-bot chatter:
 - Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni: Check for repeated DM exchanges between bots
+- Tanko/Mumuni: Repeated DM exchanges between bots
- kagentz: Check adapter log for bot DMs being processed
+- kagentz: Adapter log for bot DMs being processed
-If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.
+If any bot processes >50 bot-originated messages in 15min → warning.
 ### Step 6: Compile and Report
 1. Compile all platform checks and severity
 2. Determine `overall_severity` from worst per-agent severity
 3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
 4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
 5. If any agent critical or >2 degraded: send relay message to user
 6. Update `last_check` timestamp in `### Maintains` snapshot
 ### Restart Debounce
 All restart actions MUST debounce: minimum 300s between restarts.
 Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
 ---
-## Consolidated Action Matrix
+## History
-| Condition | Severity | Action |
+### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
 |-----------|----------|--------|
 | Zulip server not 200 | 🔴 Critical | Alert maintainer |
 | All agents silent | 🔴 Critical | Zulip server likely down |
 | pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` |
 | pi: edit fail >50% | 🔴 Critical | Check editMessage API |
 | Hermes: zulip disconnected | 🔴 Critical | Restart gateway |
 | Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway |
 | Az: A2A server down | 🔴 Critical | Restart inside container |
 | Az: adapter down | 🔴 Critical | Restart adapter |
 | Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) |
 | Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) |
-## Logging
+**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.
-All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps.
+**Fixes — Extension (`index.js`)**:
-Critical alerts sent as relay messages to user.
+1. **Rotation extended to 55 min** (from 25) with **±90s jitter** — avoids aligning with cron/monitor cycles
 2. **Rate-limit-aware retry** — if connection fails with 429, skip the retry loop entirely, wait 120s, try once
 **Fixes — Monitor (`zulip-monitor.sh`)**:
 3. **Smart Triage** instead of instant restart:
   - **Rate-limit detection**: If error log shows recent 429s, wait — don't add more API load
   - **Self-healing detection**: If `retry_count` is 1-2, extension is already retrying — don't interrupt
   - **Rotation window awareness**: If `queue_age` is 25-35min, disconnection is likely transient rotation — wait
   - **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
   - **Debounce gate**: Skip all checks entirely if recently restarted
 ### Gen 4 (2026-06-29) — Response Pipeline Fix
 **Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
 **Four-Layer Defense**:
 1. **Response Watchdog** — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue
 2. **Queue Health Ping** — Every 2min `/api/v1/events?dont_block=true` to detect silent expiry
 3. **Proactive Queue Rotation** — New queue every 25min + 600 empty poll reconnection threshold
 4. **Health Endpoint v2** — Added `response_pipeline`, `pending_count`, `queue_healthy`, `agent_busy_duration_seconds`
 ### Gen 3 (2026-06-15) — Stuck Detection
 Added `stuck: bool` and `idle_seconds` to health endpoint. Monitor restarts on `stuck: true`. Added 300s restart debounce. Queue re-registers after 30min of no events.
@@ -1,27 +1,31 @@
 ---
 kind: responsibility
 name: zulip-mention-reliability
 status: retired
 description: >
-  Monitors whether Abiba Bot is correctly detecting and responding to
+  RETIRED 2026-07-04 — Zulip extension decommissioned. Abiba now operates
-  @mentions in Zulip stream topics. Tests periodically, diagnoses
+  exclusively through Telegram (@AbibaBot). This contract is kept for
-  failures, and attempts fixes. Same self-improving pattern as
+  historical reference only.
  litellm-self-heal.
 ---
 > **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
 > PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
 > monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
 > Agent Zero (kagentz) continue to use Zulip.
 ## Maintains
 - mention_response_rate: number — % of @mentions that get a response (target: >95%)
 - last_test_result: "pass" | "fail" | "degraded"
 - known_failure_modes: array — History of what went wrong and how it was fixed
 - queue_health: "healthy" | "expired" | "reconnecting"
-
+  
-## Success Criteria
+  **Postconditions**
-
+  - test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
- test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
+  - stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
- stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
+  - queue_active == true — PM2's event queue is registered and polling
- queue_active == true — PM2's event queue is registered and polling
+  - no_echo_loop == true — Bot doesn't respond to its own messages
- no_echo_loop == true — Bot doesn't respond to its own messages
+  - dm_always_works == true — DMs still respond even if streams are broken
 - dm_always_works == true — DMs still respond even if streams are broken
 ## Known Failure Modes (from History)
@@ -0,0 +1,92 @@
 ---
 kind: responsibility
 name: zulip-oidc-redirect-fix
 status: active
 description: >
  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
  uses the internal IP (192.168.68.19) instead of the public domain
  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
  Survives container restarts through compose.override.yaml.
 agent: abiba
 triggers:
  - Zulip OIDC login returns "Redirect URI Error" from Authentik
  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
  - After Zulip server restart, Authentik SSO login broken
 ---
 ## Maintains
 - zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
 - authentik-acceptance: { status: "accepted" | "rejected" }
 - patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
 ## Detection
 ### Rule 1: Wrong redirect_uri host
 - **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
 - **Status**: CRITICAL — Authentik will reject
 - **Trigger fix** → Execute self-heal
 ### Rule 2: OIDC flow broken
 - **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
 - **Diagnose**: Check redirect_uri in the 302 Location header chain
 ## Fix (Self-Heal)
 Two layers applied:
 ### Layer 1: Live patch (inside container, immediate)
 ```bash
 # Add to /home/zulip/deployments/current/zproject/computed_settings.py:
 SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
 import urllib.parse
 from social_core.strategy import BaseStrategy
 _original_absolute_uri = BaseStrategy.absolute_uri
 def _patched_absolute_uri(self, path=None):
    from django.conf import settings
    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
    if path is not None:
        return urllib.parse.urljoin(root, path)
    return root
 BaseStrategy.absolute_uri = _patched_absolute_uri
 # Restart Django
 supervisorctl restart zulip-django
 ```
 ### Layer 2: Persistent fix (compose.override.yaml)
 The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
 `/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
 ### Verification
 ```bash
 STEP1=$(curl -sk -w "%{redirect_url}" \
  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
 curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
 # Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
 # Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
 ```
 ### Rollback
 Remove the patch block from `compose.override.yaml` and restart the container:
 ```bash
 docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
 ```
 ## Root Cause
 After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
 Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
 Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
 `Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
 Django falls back to the server's IP.
 The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
 `ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
 Host header.
 ## Related Contracts
 - `zulip-health.prose.md` — General Zulip health monitoring
 - `zulip-self-heal.prose.md` — RETIRED (pi extension removed)
@@ -0,0 +1,476 @@
 ---
 kind: responsibility
 name: zulip-resilience-v3
 description: >
  Rewrite the pi Zulip gateway with production-grade resilience patterns drawn from
  Zulip's own event system docs (queue lifecycle, heartbeat monitoring, BAD_EVENT_QUEUE_ID
  handling, idle_queue_timeout) and battle-tested Node.js resilience patterns
  (circuit breaker, exponential backoff with jitter, bulkhead isolation, supervisor watchdog).
 replaces: zulip-self-heal (retired)
 agent: abiba
 triggers:
  - "/zulip self-heal v3"
  - "zulip stopped responding"
  - "PM2 abiba-zulip crashed"
 ---
 # Zulip Gateway v3 — Production Resilience
 ## Architecture Overview
 The current v2 gateway (`/root/.pi/agent/extensions/zulip/index.js`) has three structural
 weaknesses that cause repeated deaths:
 1. **No crash recovery** — uncaught errors kill the Node process, PM2 exhausts max_restarts
 2. **No circuit breaker** — 502/fetch-failed errors escalate to process death with no fallback
 3. **No queue lifecycle management** — doesn't use Zulip's documented heartbeat protocol or
   idle_queue_timeout, so BAD_EVENT_QUEUE_ID errors cascade into crashes
 The v3 rewrite addresses all three, following patterns from:
 - [Zulip Events System docs](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html) —
  queue registration, heartbeat, BAD_EVENT_QUEUE_ID recovery, call_on_each_event loop
 - [Zulip API: Get Events](https://zulip.com/api/get-events) — long-poll timeout, dont_block, event ack
 - [Circuit Breaker & Retry Patterns in Node.js 2026](https://1xapi.com/blog/resilient-api-circuit-breaker-bulkhead-retry-nodejs-2026) —
  Opossum-based circuit breaker with fallback, retry with jitter, bulkhead isolation
 ---
 ## Maintains
 - `zulip-gateway`: { status: "healthy" | "degraded" | "down" }
 - `circuit-breaker`: { state: "CLOSED" | "OPEN" | "HALF_OPEN", failures, successes }
 - `queue-lifecycle`: { queue_id, last_event_id, idle_timeout, heartbeat_age }
 - `workers`: { count, busy, idle, stuck }
 - `supervisor`: { pid, last_check, health_failures }
 ---
 ## Detection Rules
 ### Rule 1: Queue Expired (BAD_EVENT_QUEUE_ID)
 - **Detect**: Events API returns error with BAD_EVENT_QUEUE_ID in body
 - **Fix**: Call `POST /register` to create new queue, update queue_id and last_event_id
 - **Debounce**: If 3 re-registrations fail within 60s, escalate (server may be down)
 - **Ref**: Zulip docs: "Your software will need to handle that error condition by re-initializing itself"
 ### Rule 2: Network Degradation (502/ECONNREFUSED/fetch failed)
 - **Detect**: Events API returns 502 or network error
 - **Circuit breaker**: Track failure rate over 10s rolling window
  - CLOSED → OPEN: 50% failure rate with ≥5 requests
  - OPEN → HALF_OPEN: After 30s reset timeout
  - HALF_OPEN → CLOSED: Probe succeeds
  - HALF_OPEN → OPEN: Probe fails
 - **While OPEN**: Log errors, skip events, notify user via DM: "⚠️ Zulip connection degraded — will retry in 30s"
 ### Rule 3: Long-Poll Timeout (natural)
 - **Detect**: Events API response takes > `event_queue_longpoll_timeout_seconds`
 - **Not an error**: Server sends heartbeat events when no real events. Simply re-poll.
 ### Rule 4: Worker Busy Timeout (>5 min)
 - **Detect**: Worker `busySince` exceeds 5 minutes
 - **Fix**: SIGKILL worker, send error DM, clean up pending replies
 ### Rule 5: Process Crash (uncaught)
 - **Detect**: `uncaughtException` / `unhandledRejection` fires
 - **Fix**: Log → clear poll timer → attempt reconnect with backoff → if reconnect fails 3x, exit(1) and let PM2 restart
 ### Rule 6: Supervisor Detects Router Stall
 - **Detect**: External supervisor (`zulip-watchdog`) polls `/health` every 30s. If 3 consecutive failures:
 - **Fix**: `pm2 restart abiba-zulip` gracefully (SIGTERM, drain workers, restart)
 ---
 ## Implementation Plan
 ### Phase 1: Rewrite Router Core (circuit-breaker + queue lifecycle)
 Replace the poll loop in index.js with a resilience-first event loop:
 ```js
 // Queue lifecycle (Zulip docs pattern)
 async function createOrRefreshQueue() {
  // POST /register with event_types=["message"]
  // Store: queueId, lastEventId, eventQueueLongpollTimeoutSeconds
  // NEW: pass idle_queue_timeout parameter (Zulip 12.0+)
 }
 // Circuit breaker (Opossum pattern, implemented inline to avoid dependency)
 class ZulipCircuitBreaker {
  constructor({ failureThreshold=0.5, resetTimeout=30000, volumeThreshold=5, windowMs=10000 }) {
    this.state = "CLOSED"; // CLOSED | OPEN | HALF_OPEN
    this.failures = 0;
    this.successes = 0;
    this.totalRequests = 0;
    this.lastFailureTime = null;
    this.openedAt = null;
    this.failureThreshold = failureThreshold;
    this.resetTimeout = resetTimeout;
    this.volumeThreshold = volumeThreshold;
    this.windowMs = windowMs;
  }
  async fire(fn) {
    if (this.state === "OPEN") {
      if (Date.now() - this.openedAt > this.resetTimeout) {
        this.state = "HALF_OPEN";
      } else {
        throw new CircuitOpenError("Circuit is OPEN");
      }
    }
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }
  onSuccess() {
    this.successes++;
    this.totalRequests++;
    if (this.state === "HALF_OPEN") {
      this.state = "CLOSED";
      this.failures = 0;
    }
    // Reset counters periodically
    if (this.totalRequests > this.volumeThreshold * 2) {
      this.failures = Math.floor(this.failures / 2);
      this.successes = Math.floor(this.successes / 2);
      this.totalRequests = Math.floor(this.totalRequests / 2);
    }
  }
  onFailure() {
    this.failures++;
    this.totalRequests++;
    this.lastFailureTime = Date.now();
    if (this.totalRequests >= this.volumeThreshold &&
        this.failures / this.totalRequests >= this.failureThreshold) {
      if (this.state !== "OPEN") {
        this.state = "OPEN";
        this.openedAt = Date.now();
        console.error(`[zulip-ext] CIRCUIT BREAKER OPEN — ${this.failures}/${this.totalRequests} failures`);
      }
    }
  }
 }
 // Retry with exponential backoff + jitter (from resilience patterns)
 async function withRetry(fn, { maxAttempts=3, baseDelay=200, maxDelay=10000, shouldRetry=()=>true }={}) {
  let lastError;
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      if (attempt === maxAttempts || !shouldRetry(err)) throw err;
      const delay = Math.min(baseDelay * Math.pow(2, attempt - 1), maxDelay);
      const jitter = delay * (0.5 + Math.random() * 0.5); // 50-100% of delay
      console.warn(`[zulip-ext] Retry ${attempt}/${maxAttempts} after ${Math.round(jitter)}ms: ${err.message.slice(0,80)}`);
      await new Promise(r => setTimeout(r, jitter));
    }
  }
  throw lastError;
 }
 // Resilience-first event loop (Zulip call_on_each_event pattern)
 async function resilientPollLoop() {
  while (connected) {
    try {
      const events = await circuitBreaker.fire(() =>
        withRetry(() => zulipQueue.poll(), {
          maxAttempts: 2,
          baseDelay: 1000,
          shouldRetry: (err) => {
            const msg = err.message || "";
            return msg.includes("fetch failed") || msg.includes("ECONN") || msg.includes("network");
          }
        })
      );
      lastError = null;
      retryCount = 0;
      for (const ev of events) {
        await processEvent(ev);
      }
      heartbeat();
    } catch (err) {
      const msg = err instanceof Error ? err.message : String(err);
      if (msg.includes("BAD_EVENT_QUEUE_ID") || msg.includes("deregistered")) {
        // Queue expired — re-register (Zulip docs pattern)
        console.log(`[zulip-ext] Queue expired, re-registering… (${msg.slice(0,80)})`);
        try {
          zulipQueue = await createZulipQueue();
          console.log(`[zulip-ext] Re-registered, new queue=${zulipQueue.queueId}`);
        } catch (reRegErr) {
          console.error(`[zulip-ext] Re-registration failed: ${reRegErr.message}`);
          connected = false;
          retryCount++;
          const backoff = Math.min(5000 * Math.pow(2, retryCount), 300000);
          console.log(`[zulip-ext] Full reconnect in ${Math.round(backoff/1000)}s`);
          await new Promise(r => setTimeout(r, backoff));
          await startPolling();
          return;
        }
      } else if (err.name === "CircuitOpenError") {
        // Circuit is open — skip this cycle, wait for HALF_OPEN
        lastError = "circuit_open";
        await new Promise(r => setTimeout(r, POLL_INTERVAL_MS));
      } else {
        lastError = msg;
        retryCount++;
        const backoff = Math.min(POLL_INTERVAL_MS * Math.pow(1.5, Math.min(retryCount, 8)), 60000);
        console.error(`[zulip-ext] Poll error (retry ${retryCount}, backoff ${backoff}ms): ${msg}`);
        await new Promise(r => setTimeout(r, backoff));
      }
    }
  }
 }
 ```
 ### Phase 2: PM2 Hardening
 Create `/root/.pm2/ecosystem.config.cjs`:
 ```js
 module.exports = {
  apps: [
    {
      name: "abiba-zulip",
      script: "/bin/pi",
      args: "--mode rpc --session-id zulip-service",
      env: {
        ZULIP_ROLE: "router",
        ZULIP_SITE: "https://chat.sysloggh.net",
        ZULIP_EMAIL: "abiba-bot@chat.sysloggh.net",
        ZULIP_API_KEY: process.env.ZULIP_API_KEY,
        AGENT_NAME: "abiba",
        AGENT_OWNER_EMAIL: "jerome@sysloggh.com",
      },
      max_restarts: 100,            // Up from default 10 — crash loops won't exhaust
      min_uptime: "10s",            // Must survive 10s to count as "alive"
      max_memory_restart: "500M",   // OOM protection
      restart_delay: 5000,          // 5s between restarts
      kill_timeout: 15000,          // 15s SIGTERM grace before SIGKILL
      listen_timeout: 30000,        // 30s to bind health port
      log_date_format: "YYYY-MM-DD HH:mm:ss Z",
      error_file: "/root/.pm2/logs/abiba-zulip-error.log",
      out_file: "/root/.pm2/logs/abiba-zulip-out.log",
      merge_logs: true,
      autorestart: true,
      watch: false,
      instances: 1,
      exec_mode: "fork",
    },
    {
      name: "zulip-watchdog",
      script: "/root/.pi/agent/extensions/zulip/watchdog.js",
      max_restarts: 10,
      min_uptime: "3s",
      restart_delay: 3000,
      autorestart: true,
    },
  ],
 };
 ```
 ### Phase 3: Supervisor Watchdog
 Create `/root/.pi/agent/extensions/zulip/watchdog.js`:
 ```js
 // External supervisor — monitors router health and restarts if stalled.
 // This is the pattern Hermes uses: an external process that can recover
 // the gateway even if the gateway process itself is hung (not just crashed).
 const HEALTH_URL = "http://127.0.0.1:9200/health";
 const CHECK_INTERVAL_MS = 30_000;
 const MAX_FAILURES = 3;
 let failures = 0;
 async function check() {
  try {
    const res = await fetch(HEALTH_URL, { signal: AbortSignal.timeout(5000) });
    if (res.ok) {
      const data = await res.json();
      if (data.status === "ok" && data.zulip?.connected) {
        if (failures > 0) {
          console.log(`[watchdog] Router recovered after ${failures} failures`);
        }
        failures = 0;
        return;
      }
    }
    failures++;
    console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: status not ok`);
  } catch (err) {
    failures++;
    console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: ${err.message}`);
  }
  if (failures >= MAX_FAILURES) {
    console.error(`[watchdog] ${MAX_FAILURES} consecutive failures — restarting abiba-zulip`);
    const { execSync } = require("child_process");
    try {
      execSync("pm2 restart abiba-zulip", { timeout: 30000 });
      console.log("[watchdog] Restart command sent");
    } catch (e) {
      console.error(`[watchdog] Restart failed: ${e.message}`);
    }
    failures = 0;
    // Wait for restart to complete before checking again
    await new Promise(r => setTimeout(r, 15000));
  }
 }
 console.log("[watchdog] Zulip gateway supervisor started");
 setInterval(check, CHECK_INTERVAL_MS);
 check(); // Immediate first check
 ```
 ### Phase 4: Health Endpoint Enhancement
 Add circuit breaker stats to the existing health endpoint:
 ```js
 // In /health response, add:
 "circuit_breaker": {
  "state": circuitBreaker.state,
  "failures": circuitBreaker.failures,
  "successes": circuitBreaker.successes,
  "total_requests": circuitBreaker.totalRequests,
  "failure_rate": circuitBreaker.totalRequests > 0
    ? (circuitBreaker.failures / circuitBreaker.totalRequests).toFixed(2)
    : "0.00"
 }
 ```
 ---
 ## Test Plan
 ### Test 1: Queue Re-registration
 1. Manually delete the Zulip event queue via API
 2. Next poll should detect BAD_EVENT_QUEUE_ID
 3. Router should auto re-register within 1 poll cycle
 4. Verify: `/health` shows new queue_id, connected=true
 ### Test 2: Circuit Breaker Trip
 1. Block Zulip server with iptables: `iptables -A OUTPUT -d 192.168.68.19 -j DROP`
 2. Router should detect failures, trip circuit after 5 failures
 3. `/health` should show circuit_breaker.state = "OPEN"
 4. Remove iptables rule
 5. Circuit should transition to HALF_OPEN → CLOSED within 60s
 6. Verify: messages processed after recovery
 ### Test 3: Supervisor Recovery
 1. Kill the router process: `kill -STOP $(pm2 pid abiba-zulip)` (freeze, don't kill)
 2. Watchdog should detect 3 failed health checks in 90s
 3. Watchdog should execute `pm2 restart abiba-zulip`
 4. Verify: router back online, connected=true
 ### Test 4: Worker Busy Timeout
 1. Send a message that triggers a long-running operation
 2. If worker stays busy >5 minutes, should receive SIGKILL
 3. User should receive error DM: "Response timed out"
 ### Test 5: End-to-End Message
 1. Send DM "What time is it?" from Jerome
 2. Should receive response within 30s
 3. `/health` should show messages_processed incremented
 ---
 ## Rollback Plan
 If v3 causes issues:
 1. `pm2 delete abiba-zulip; pm2 delete zulip-watchdog`
 2. Restore v2 from git: `cd /root/.pi/agent/extensions/zulip && git checkout index.js`
 3. `pm2 resurrect` to reload previous process list
 4. Verify: `/health` returns ok
 Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S)`
 ---
 ## Success Metrics
 | Metric | Current (v2) | Target (v3) |
 |--------|-------------|-------------|
 | Uptime between manual interventions | 1-3 days | 30+ days |
 | Crash recovery | Manual (PM2 resurrect) | Automatic (circuit breaker + supervisor) |
 | Queue expiry handling | Crash | Auto re-register |
 | Busy worker deadlock | Router death | Worker SIGKILL + error DM |
 | PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |
 ---
 ## Incident Log — 2026-07-18 Fleet-Wide Audit
 ### Fleet State After Audit
 | Agent | Platform | Zulip State | Issues Found | Fix Applied |
 |-------|----------|-------------|--------------|-------------|
 | **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
 | **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
 | **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |
 ### Key Fixes Applied
 **1. Abiba — Credential Fallback (L4 Pattern)**
 - Root cause: `zulip.api_key` in config.yaml is `""` (expected from Infisical). Infisical vault `ABIBA_ZULIP_API_KEY` wasn't being injected into the process environment.
 - Fix: Added `.env` file fallback at `/root/.pi/agent/extensions/zulip/.env` with known-working key, sourced before the Infisical `exec`.
 - Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local `.env` fallback.
 **2. Abiba — Poll Timeout Handling**
 - Root cause: Zulip long-poll uses `AbortSignal.timeout(65000)`. Zulip's default `event_queue_longpoll_timeout_seconds` can exceed 65s. When the signal fires, an `AbortError` is thrown and caught by the circuit breaker as a failure.
 - Fix: Caught `AbortError` inside `poll()` and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default.
 - Reference: [Zulip Events System — long-poll timeout](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html)
 **3. Tanko — Gateway Restart**
 - Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (`infisical-gateway.sh`) restarts on crash but the gateway wasn't re-establishing Zulip on restart.
 - Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully.
 ### Fleet-Wide Zulip Health Metrics (as of 2026-07-18)
 | Metric | Value |
 |--------|-------|
 | Zulip server | ✅ HTTP 200 |
 | Agents connected | 3/3 (Abiba, Tanko, Mumuni) |
 | Abiba circuit breaker | CLOSED (0 failures) |
 | Abiba uptime | 2D (post-restart) |
 | Tanko gateway uptime | Ongoing |
 | Mumuni gateway uptime | Ongoing |
 | Watchdog status | ✅ Online (2D uptime) |
 ### Hermes Agent Zulip Plugin Improvements
 Based on the audit, improvements that should be ported to all Hermes Zulip adapters:
 1. **Circuit breaker pattern** — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff.
 2. **Credential fallback** — All Hermes agents use Infisical for credentials. Add `.env` local fallback per L4 pattern for `ZULIP_API_KEY`.
 3. **Queue re-registration** — Handle `BAD_EVENT_QUEUE_ID` with automatic re-registration instead of gateway restart.
 4. **Supervisor watchdog** — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks.
 5. **Streaming** — All agents have `streaming: true` in their zulip config. Verify `edit_message()` is implemented in each adapter.
 ### Abiba pi Zulip Extension v2 — Implemented Resilience Summary
 | Feature | Status | Notes |
 |---------|--------|-------|
 | Circuit breaker | ✅ | CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout |
 | Retry with jitter | ✅ | 2 attempts, 200ms base, 50-100% jitter |
 | Queue lifecycle | ✅ | 10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling |
 | Crash prevention | ✅ | uncaughtException + unhandledRejection recovery |
 | Worker timeout | ✅ | 5min busy timeout → SIGKILL + error DM |
 | Health endpoint | ✅ | :9200 with circuit breaker metrics |
 | Echo prevention | ✅ | Dynamic bot user resolution |
 | Poll timeout (AbortError) | ✅ v2.1 | Normal timeout returns [] instead of error |
 | Credential fallback | ✅ v2.1 | .env file before Infisical exec |
 | Provider auto-fix | ✅ | Detects reasoning_content models, switches to compatible |
@@ -0,0 +1,92 @@
 ---
 kind: responsibility
 name: zulip-self-heal
 status: retired
 description: >
  RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
  Self-healing for Zulip infrastructure continues through Hermes agents.
  Abiba no longer monitors or manages Zulip.
 agent: abiba
 triggers:
  - none (retired)
 ---
 > **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
 > (`performHealthCheck()`). That code has been removed. Zulip self-healing for
 > Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
 ## Maintains
 - zulip-server: { status: "healthy" | "degraded" | "down" }
 - zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
 - zulip-config: { syntax_valid: bool, proxy_configured: bool }
 - last_action: { timestamp, action, result }
 ## Detection Rules
 ### Rule 1: Zulip Server Down (502/000)
 - **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
 - **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
 - **Fix**: `docker restart zulip-zulip-1` if container crashed
 - **Verify**: Re-test server_settings, wait for "healthy"
 ### Rule 2: Zulip Config Corruption (400/500 non-proxy)
 - **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
 - **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
 - **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
 - **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
 ### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
 - **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
 - **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
 - **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
 - **Verify**: Re-test via NetBird URL
 ### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
 - **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
 - **Fix (pi)**: `pm2 restart abiba-zulip`
 - **Fix (Hermes)**: `hermes gateway restart` on agent host
 - **Verify**: Check health endpoint for `queue_healthy: true`
 ### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
 - **Detect**: No heartbeat in agent log for >600s
 - **Fix**: Restart gateway on affected host
 - **Verify**: Check for new heartbeat in logs within 60s
 ### Rule 6: NetBird Tunnel Down
 - **Detect**: Server returns 502 with NetBird HTML in response
 - **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
 - **Fix**: Alert user — NetBird outage requires manual intervention
 - **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
 ## Hosts & Credentials
 | Host | IP | User | Service |
 |------|----|------|---------|
 | Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
 | Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
 | Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
 | Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
 ## Debounce
 All restart actions must debounce: minimum 300s between restarts per agent.
 Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
 ## Reporting
 Every cycle produces a knowledge graph node:
 - Title: `[LEARN] zulip-self-heal: <timestamp>`
 - metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
 - Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
 - Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
 ## Configuration
 Zulip server config at 192.168.68.19:
 - `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
 - `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
 - Fixed values:
  - `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
  - `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
  - `LOAD_BALANCER_IPS = ["192.168.68.10"]`
		`@@ -0,0 +1 @@`
							`{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}`