fix: ai-review gracefully skips when LiteLLM secrets not configured

docs: update key enforcement — all agents named keys, systemd fix, CI pipeline
- All 4 Hermes agents now have dedicated named LiteLLM keys (tanko, mumuni, koby, koonimo) - Documented systemd drop-in pattern fix (litellm-key.conf) - Added one-step rotation procedure - Added CI pipeline and branch protection documentation - Removed old mumuni keys, generated fresh keys for koby/koonimo
2026-07-04 23:30:25 +00:00 · 2026-07-04 23:28:59 +00:00 · 2026-07-04 23:09:29 +00:00 · 2026-07-04 23:08:19 +00:00 · 2026-07-04 23:06:40 +00:00 · 2026-07-04 23:00:55 +00:00
35 changed files with 3839 additions and 543 deletions
@@ -0,0 +1,106 @@
+name: PR Pipeline — Authorize → Validate → Review → Merge
+on:
+  push:
+    branches: [master]
+    paths:
+      - '**.prose.md'
+      - 'scripts/**.sh'
+      - '**.yaml'
+      - '**.yml'
+  pull_request:
+    types: [opened, synchronize, reopened]
+    paths:
+      - '**.prose.md'
+      - 'scripts/**.sh'
+      - '**.yaml'
+      - '**.yml'
+
+jobs:
+  auth:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: Authorization check
+        run: bash scripts/prose-auth-check.sh
+
+  validate:
+    runs-on: ubuntu-latest
+    needs: auth
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: YAML frontmatter validation
+        run: |
+          echo "=== Prose Contract Frontmatter Validation ==="
+          FAILED=0
+          for f in $(find . -name "*.prose.md" -not -path "./.git/*" -not -path "./runs/*"); do
+            FM=$(sed -n '/^---$/,/^---$/p' "$f" | sed '1d;$d')
+            [ -z "$FM" ] && { echo "  ❌ $f: No YAML frontmatter"; FAILED=$((FAILED+1)); continue; }
+
+            KIND=$(echo "$FM" | grep '^kind:' | awk '{print $2}')
+            case "$KIND" in
+              function|responsibility|gateway|pattern|test|template|architecture|enforcement) echo "  ✅ $f: kind=$KIND" ;;
+              *) echo "  ❌ $f: Invalid kind='$KIND'"; FAILED=$((FAILED+1)) ;;
+            esac
+
+            echo "$FM" | grep -q '^name:' || { echo "  ❌ $f: Missing name"; FAILED=$((FAILED+1)); }
+            echo "$FM" | grep -q '^description:' || { echo "  ❌ $f: Missing description"; FAILED=$((FAILED+1)); }
+          done
+          [ $FAILED -gt 0 ] && { echo "❌ FRONTMATTER FAILED ($FAILED error(s))"; exit 1; }
+          echo "✅ Frontmatter validation passed"
+
+  lint:
+    runs-on: ubuntu-latest
+    needs: validate
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: Structure + regression + consistency lint
+        run: bash scripts/prose-lint.sh
+
+  ai-review:
+    runs-on: ubuntu-latest
+    needs: validate
+    steps:
+      - name: Checkout repository
+        run: |
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git fetch origin "${{ gitea.ref }}" --depth=50
+          git checkout "${{ gitea.sha }}"
+
+      - name: AI-powered contract review
+        env:
+          LITELLM_URL: ${{ secrets.LITELLM_URL }}
+          LITELLM_KEY: ${{ secrets.LITELLM_KEY }}
+        run: bash scripts/prose-ai-review.sh
+
+  gate:
+    runs-on: ubuntu-latest
+    needs: [auth, validate, lint, ai-review]
+    if: success()
+    steps:
+      - name: Merge gate
+        run: |
+          echo "╔══════════════════════════════════════╗"
+          echo "║  ALL CHECKS PASSED — SAFE TO MERGE  ║"
+          echo "╚══════════════════════════════════════╝"
+          echo ""
+          echo "  ✅ auth       — authorized agent"
+          echo "  ✅ validate   — frontmatter valid"
+          echo "  ✅ lint       — structure + no regressions"
+          echo "  ✅ ai-review  — no contradictions with ground truth"
+          echo ""
+          echo "Merge this PR to deploy to main."
@@ -0,0 +1,130 @@
+# AGENTS.md — Prose Contracts Repo
+
+All agents operating on prose contracts MUST follow this workflow. No exceptions.
+
+## The Rule
+
+**No agent pushes directly to `main`. All changes go through PRs with automated validation.**
+
+## Why
+
+Two incidents taught us this:
+
+1. **The .117→.19 IP fix** — A stale IP in a contract was "fixed" without verification, breaking Zulip. The fix was right but the approach was wrong. Automated consistency checks would have caught it.
+
+2. **The /grafana/ route regression** — An nginx route that was deliberately removed (Jul 2) was re-added to a contract diagram. CI linting now catches this automatically.
+
+## Agent Workflow
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  1. Clone repo → create branch → make changes           │
+│  2. Push branch → open PR                                │
+│  3. CI pipeline runs automatically:                      │
+│     ✅ validate: frontmatter (kind, name, description)   │
+│     ✅ lint: structure + regression detection            │
+│     ✅ ai-review: LiteLLM reviews diff vs ground truth   │
+│  4. All green → merge PR to main                         │
+│  5. Agents run contracts from main                       │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Branch Protection (enforced by Gitea)
+
+| Rule | Enforcement |
+|------|------------|
+| Direct push to `main` | ❌ Blocked (except abiba-bot for emergencies) |
+| PR merge without passing CI | ❌ Blocked — all 3 checks must be green |
+| Status check contexts | `pr-pipeline / validate`, `pr-pipeline / lint`, `pr-pipeline / ai-review` |
+
+## What the CI checks for
+
+### Stage 1 — Validate
+- YAML frontmatter is valid (proper `---` delimiters)
+- `kind` field is one of: `function`, `responsibility`, `gateway`, `pattern`, `test`, `template`
+- `name` and `description` fields are present
+
+### Stage 2 — Lint
+- Responsibility contracts have `## Maintains`
+- Function contracts have `## Parameters` and `## Returns`
+- **Regression rules (automatic rejection):**
+  - `/grafana/` nginx route — was reverted Jul 2, must not reappear
+  - `CT 122` or `CT 123` as CT ID labels — don't exist in the cluster
+  - These rules are hardcoded in `scripts/prose-lint.sh`
+
+### Stage 3 — AI Review
+- Diff is sent to `syslog-auto` model via LiteLLM
+- Review checks against infrastructure-control ground truth:
+  - CT IDs match PVE cluster (100-117, no 122/123)
+  - Grafana is direct LAN :3001, NOT behind nginx
+  - Zulip is CT 117 on storepve (bridge IP .19)
+  - Strix Halo :8080 is firewalled to .116 only
+- Review is posted to the PR
+
+## Authorization
+
+### Who can change what
+
+| Contract | Sensitivity | Who can change |
+|----------|------------|----------------|
+| `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba only (after live verification) |
+| `proxmox-monitor.prose.md` | **CRITICAL** — deployed monitoring | Abiba only |
+| `hermes-config-template.prose.md` | **HIGH** — all agent configs | Abiba, Mumuni, Tanko |
+| `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni |
+| Other contracts | Normal | Any registered agent |
+| `scripts/*.sh` | **HIGH** — runtime scripts | Abiba only |
+
+### Enforcement (self-policing)
+
+The CI does not block based on author identity today (Gitea doesn't support CODEOWNERS natively). Instead, agents self-enforce:
+
+1. Before changing a CRITICAL contract, check with Abiba
+2. If you're unsure, tag `@abiba-bot` in the PR description
+3. Abiba reviews CRITICAL contract changes before merge
+
+## Emergency Bypass
+
+In an emergency (service down, fix must ship immediately):
+1. Push to a branch
+2. Open PR with `[EMERGENCY]` in the title
+3. CI still runs — but if it fails and the fix is verified, abiba-bot can bypass and push directly to main
+4. Post-incident: open a follow-up PR to fix any CI violations
+
+## Quick Start
+
+```bash
+# Clone
+git clone https://git.sysloggh.net/SyslogSolution/prose-contracts.git
+cd prose-contracts
+
+# Create branch
+git checkout -b fix/my-change
+
+# Make changes, test locally
+bash scripts/prose-lint.sh     # run lint locally before pushing
+
+# Push and open PR
+git add -A
+git commit -m "fix: description of change"
+git push origin fix/my-change
+
+# Open PR at https://git.sysloggh.net/SyslogSolution/prose-contracts/pulls
+# CI runs automatically — wait for all checks to pass
+# Merge when green
+```
+
+## Verification Before Acting
+
+**Contracts are leads, not facts.** Live-state fields (IPs, ports, credentials) drift.
+Before acting on any value from a contract, verify it against the live system.
+
+Follow the `verify-before-mutate` protocol:
+```bash
+safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
+```
+
+## Writing New Contracts
+
+Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
+It covers the full process: verify → draft → lint → review → ship, with templates
+and style rules.
@@ -1,6 +1,70 @@
 # Prose Contracts — Syslog Solution LLC

-Operating contracts for Hermes agents. Each `.prose.md` file defines a **responsibility** (recurring duty) or **template** (scaffold) that agents can execute via OpenProse or follow manually.
+Operating contracts for Syslog agents. Each `.prose.md` file defines a **function** (callable helper), **responsibility** (recurring duty with state), or **pattern** (instantiable knowledge/template) that agents can execute via OpenProse or follow manually.
+
+## ⚠️ Operating Doctrine — Read Before Acting
+
+Two rules govern how agents interact with this repo and the infrastructure it
+describes. Internalize both before running or relying on any contract.
+
+### 1. Verify-before-mutate (the `.117` rule)
+
+**Contracts are leads to investigate, NOT facts to act on.** Live-state fields
+(IPs, ports, hostnames, credentials, container names, PIDs) drift. Before any
+mutating operation on infrastructure, verify the current state against the live
+system. On drift, halt and ask the user which value is correct — do NOT "fix"
+the live state to match a stale contract.
+
+Origin: a contract stated Zulip CT 117 was at `.117`; the live IP was `.19`.
+An agent ran `pct set` without checking `pct config` first and changed the IP
+to the wrong value, breaking Zulip. The staleness was harmless until acted on.
+
+**Layer 3 enforcement — the `safe-mutate` wrapper** (deployed on all 5 agents:
+Abiba, Tanko, Mumuni, Koby, Koonimo). ALL infrastructure mutations MUST go
+through `safe-mutate`. Raw `sed -i`, `pct set`, `docker compose up
+--force-recreate`, `kill`, `rm` on infrastructure outside `safe-mutate` is an
+auditable protocol violation. The wrapper runs a verify command, optionally
+checks an `--expect` pattern, refuses on mismatch, and logs every call to
+`/root/.safe-mutate/audit.log`.
+
+```
+safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
+safe-mutate --verify "CMD" --dry-run --mutate "CMD"      # verify + show, no execute
+safe-mutate --verify "CMD" --verify-only                   # record a verification only
+```
+
+Load the `verify-before-mutate` skill (local pi skill + RA-H shared) for the
+full protocol. See knowledge graph node #604 for the post-mortem.
+
+### 2. Field trust levels
+
+Not all contract content carries the same trust:
+
+| Field type | Trust level | Examples |
+|-----------|-------------|----------|
+| **Policy** | Authoritative — trust the contract | "hardcoded harness keys forbidden", "use api_key_env" |
+| **Live state** | **Never trust — always verify** | IPs, ports, hostnames, credentials, container names, PIDs |
+| **Procedure** | Trust but adapt | command patterns are right, values may be stale |
+
+Contracts label live-state fields with `VERIFY-BEFORE-USE`. Treat any such
+field as a hint to confirm against the live system, not a fact to apply.
+
+### 3. Verify-before-fix (contract-native)
+
+Never `prose run` a fix-contract directly. Run its verifier first, read the
+verdict, lint the fixer, then run:
+
+```
+prose run <verifier>           # e.g. zulip-platform-verification → verdict
+prose lint <fixer>             # validate structure without executing
+prose preflight <fixer>        # check deps/env, no execute
+prose run <fixer>              # only after verdict + lint pass
+```
+
+`kind: responsibility` contracts VERIFY and return a verdict without making
+changes. `kind: function` contracts DO things. Forme `### Requires` →
+`### Maintains` wiring enforces this at the DAG level: a fixer that requires a
+fresh verdict cannot fire until the verifier has run.

 ## How Agents Run These Contracts

@@ -40,49 +104,93 @@ curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/mast

 ## Available Contracts

-### Responsibilities (Recurring Duties)
+### Responsibilities (Recurring Duties — `kind: responsibility`)

-| Contract | What It Does | When To Run |
+Run on trigger or schedule. Maintain persistent world-model state across runs.
+
+| Contract | Domain | Description |
 |---|---|---|
-| `memory-audit-maintenance` | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs, frees up char budget. | Memory >85% usage or user request: "memory audit" |
-| `zulip-health` | Checks Zulip connectivity, message flow, and bot responsiveness. | On Zulip issues or periodic health check |
-| `litellm-health` | Verifies LiteLLM proxy connectivity, model availability, and key rotation. | On inference failures or periodic check |
-| `litellm-self-heal` | Attempts to fix common LiteLLM issues (key rotation, restart, config reload). | When litellm-health reports failures |
-| `zulip-mention-reliability` | Diagnoses and fixes @mention detection issues in Zulip. | On missed @mention reports |
-| `pm2-self-heal` | Restarts crashed PM2 processes and verifies recovery. | On PM2 process failure |
+| `memory-audit-maintenance` | Memory | Audits & reorganizes an agent's native memory (MEMORY.md, USER.md). Categorizes entries, moves rules to skills, verifies configs. |
+| `build-zulip-plugin` | Zulip | Generates and iteratively improves a Hermes Zulip platform plugin. Each run produces a new version. |
+| `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
+| `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
+| `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
+| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
+| `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
+| `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
+| `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
+| `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |

-### Templates (Scaffolds)
+### Instantiable Templates (`kind: pattern` — run with `prose run`)

-| Contract | What It Does | Parameters |
+Generate configs or perform a transformation on demand. No persistent state.
+
+| Contract | Description | Key Parameters |
 |---|---|---|
-| `hermes-config-template` | Generates a standard Hermes config for any agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) while keeping model choices flexible. | `agent_name` (required), `default_model`, `default_provider`, `fallback_model`, `auxiliary_model` |
+| `hermes-config-template` | Standard Hermes config for any Syslog agent. Enforces shared infra (Firecrawl, SearXNG, RA-H OS MCP, LiteLLM) + standardized `api_key_env` key indirection. | `agent_name` (required), `default_model`, `auxiliary_model` |
+| `hermes-key-enforcement` | **Enforcement contract** — all harness/LiteLLM providers MUST use `api_key_env`, never hardcoded keys. Includes detection query, rotation procedure, violation response. External providers (DeepSeek, OpenAI) exempt. | (none — doctrine reference) |

-### Scripts
+### Reference Documents (`kind: pattern` — read for context, not run)

-| File | What It Does |
+Encode topology, architectural decisions, and lessons. You read them before planning; you don't `prose run` them.
+
+| Contract | Description |
 |---|---|
-| `pm2-self-heal.sh` | Shell script to restart crashed PM2 processes (companion to the prose contract) |
+| `infrastructure-control` | Full topology and control pattern: 5-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
+| `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
+| `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
+
+### Functions (`kind: function` — callable helpers, no state)
+
+Called on-demand as single-render tools.
+
+| Contract | Description |
+|---|---|
+| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
+| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
+| `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
+| `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
+
+### Scripts (`scripts/`)
+
+Companion shell scripts that contracts delegate to.
+
+| File | Description |
+|---|---|
+| `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
+| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |

 ## Contract Structure

-Every `.prose.md` file has the same structure:
+Kinds determine the section set.

+**`kind: responsibility`** (recurring duty with persistent state):
+```
+## Maintains      — World-model state this contract tracks
+## Parameters     — Tunable inputs (with defaults)
+## Requires       — External dependencies and preconditions
+## Continuity     — When to run (triggers & cadence)
+## Remembers      — What to learn between runs
+## Invariants     — Inviolable rules for execution
+## Execution      — Step-by-step instructions
+## Example Output — What a good run looks like
 ```
---
-kind: <responsibility | template>
-name: <unique-name>
-description: ...
---

-## Maintains        — What state the contract tracks
-## Parameters       — Tunable inputs (with defaults)
-## Continuity       — When to run (triggers & cadence)
-## Success Criteria — How to know it worked
-## Remembers        — What to learn between runs
-## <Type> Rules     — Inviolable rules for execution
-## Execution        — Step-by-step instructions
-## Example Output   — What a good run looks like
+**`kind: function`** (stateless callable helper):
 ```
+## Parameters    — Tunable inputs (with defaults)
+## Returns       — Output schema
+## Requires      — External dependencies
+## Execution     — Step-by-step instructions
+```
+
+**`kind: pattern`** (instantiable knowledge):
+```
+Variable — templates use Parameters + Execution; reference docs are prose-driven.
+```
+
+**`kind: gateway`** / **`kind: test`** — not currently used in this repo.

 ## Adding New Contracts

@@ -14,6 +14,30 @@ description: >
 - generation_count: number — How many generations have been produced
 - last_generation: timestamp — When the last generation was created
 - known_issues: array — Issues discovered in the current version
+  
+  **Postconditions** (the plugin is GOOD when ALL pass):
+  
+  *Connectivity*
+  - connected == true — Establishes and maintains Zulip event queue
+  - reconnect_on_failure == true — Reconnects after queue expiry
+  - recovers_from_network_loss == true — Handles temporary network drops
+  
+  *Message Flow*
+  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
+  - stream_mention_detection == true — @mentions in streams are detected and routed
+  - all_bots_detection == true — @all-bots mentions are detected
+  - echo_loop_prevention == true — Never replies to its own messages
+  
+  *Code Quality*
+  - syntax_check == "pass" — All Python files are valid
+  - has_register_function == true — Exposes `register(ctx)` entry point
+  - plugin_yaml_valid == true — plugin.yaml parses correctly
+  - follows_base_adapter_pattern == true — Extends BasePlatformAdapter
+  
+  *Reliability*
+  - graceful_disconnect == true — shutdown doesn't crash
+  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
+  - typing_indicators_work == true — Sends typing notifications

 ## Parameters

@@ -26,45 +50,19 @@ description: >

 The improvement cycle is **event-driven**, not just cron-based:

- **On check failure**: If any Success Criteria fails, wake immediately to fix
+- **On check failure**: If any postcondition fails, wake immediately to fix
 - **On user request**: `prose run build-zulip-plugin` — explicit upgrade
 - **On new pattern**: If another plugin or agent reports a better pattern, wake to assimilate
 - **Retrospective**: Every 24 hours if no other trigger fired — scan for subtle degradation
 - **On first deploy**: Always run Generation 1 when no plugin exists yet

-## Success Criteria (What Makes a Plugin "Good")
-
-The plugin is considered GOOD when ALL of these pass:
-
-### Connectivity
- plugin.connected == true — Establishes and maintains Zulip event queue
- plugin.reconnect_on_failure == true — Reconnects after queue expiry
- plugin.recovers_from_network_loss == true — Handles temporary network drops
-
-### Message Flow
- dm_response_time_ms < 10000 — DMs get a response within 10 seconds
- stream_mention_detection == true — @mentions in streams are detected and routed
- all_bots_detection == true — @all-bots mentions are detected
- echo_loop_prevention == true — Never replies to its own messages
-
-### Code Quality
- syntax_check == "pass" — All Python files are valid
- has_register_function == true — Exposes `register(ctx)` entry point
- plugin_yaml_valid == true — plugin.yaml parses correctly
- follows_base_adapter_pattern == true — Extends BasePlatformAdapter
-
-### Reliability
- graceful_disconnect == true — shutdown doesn't crash
- handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
- typing_indicators_work == true — Sends typing notifications
-
 ## Remembers

 Each generation stores:
 - What worked well (success patterns)
 - What broke (failure modes)
 - What the user complained about (pain points)
- Which Success Criteria passed and failed
+- Which postconditions passed and failed

 ## Generation Rules

@@ -0,0 +1,255 @@
+---
+kind: responsibility
+name: disk-gc-threat-response
+description: >
+  Recurring disk health scan, garbage collection, and threat response
+  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
+  Docker image bloat — 5 dangling images, 15 build cache layers.
+  Recovered 35.67GB via `docker system prune -a --force`.
+id: 067NV8KJ03ZG71S44N41F31022
+version: 1.0.0
+---
+
+# Disk GC & Threat Response
+
+## Goal
+
+Every container in the fleet stays below 80% disk usage through regular inspection
+and automated garbage collection. Critical breaches are detected, remediated, and
+logged within 5 minutes of discovery.
+
+## Scope
+
+All CTs in the Proxmox cluster, with special attention to Docker hosts:
+
+| Host | CT | Disk Risk | GC Strategy |
+|------|----|-----------|-------------|
+| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
+| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
+| All others | — | LOW — no Docker | apt clean, log rotate |
+
+## Threat Levels
+
+| Level | Threshold | Response | Escalation |
+|-------|-----------|----------|------------|
+| **GREEN** | < 75% | Log only | None |
+| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
+| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
+| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
+| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
+
+## Requires
+
+- SSH access to all Docker hosts (matrix from infrastructure-control pattern)
+- Proxmox API access from abiba (token already provisioned)
+- Zulip bot credentials for alerting
+
+## Maintains
+
+Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
+the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
+bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
+documented reason.
+
+### disk-health
+Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
+timestamp, and active threat level.
+
+### gc-history
+Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
+and whether threat was resolved.
+
+### threat-log
+Active and resolved threat entries with severity, timestamps, remediation applied,
+and escalation trail.
+
+## Continuity
+
+- self-driven: full fleet scan every 6 hours
+- May also be invoked manually: `prose run disk-gc-threat-response`
+- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
+
+## Shape
+
+- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
+- `delegates`:
+  - `disk-scanner`: per-CT disk check (pct exec df or SSH)
+  - `gc-docker`: docker system prune execution
+  - `gc-system`: apt clean, log rotate, tmp cleanup
+  - `alerter`: Zulip notification dispatch
+- `prohibited`: deleting user data, removing running containers, force-killing
+  production services
+
+## Runtime
+
+- `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
+- `retry`: 2 attempts for SSH failures before marking a CT unreachable
+
+## Execution
+
+```prose
+let fleet = call disk-scanner
+  scope: all
+
+let threats = []
+for ct in fleet:
+  if ct.usage_pct >= 95:
+    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
+  else if ct.usage_pct >= 85:
+    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
+  else if ct.usage_pct >= 75:
+    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
+
+-- sort by severity descending
+sort threats by pct desc
+
+for threat in threats:
+  let result = call gc-executor
+    ct: threat.ct
+    level: threat.level
+    strategy: lookup-gc-strategy(threat.ct)
+  
+  call alerter
+    threat: threat
+    result: result
+
+call summary-reporter
+  fleet: fleet
+  threats: threats
+```
+
+## GC Strategies by Host Type
+
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+
+```bash
+# Phase 1: Safe prune (won't touch running containers' images)
+docker system prune -f
+
+# Phase 2: Aggressive (if still > 85% after Phase 1)
+docker system prune -a --force
+
+# Phase 3: Emergency (if still > 95%)
+docker system prune -a --force --volumes
+docker builder prune --all --force
+
+# Verification after each phase
+df -h /
+docker system df
+```
+
+### Non-Docker CTs
+
+```bash
+# Package cache
+apt-get clean
+apt-get autoremove --yes
+
+# Log rotation
+journalctl --vacuum-size=100M
+find /var/log -type f -name "*.log" -mtime +30 -delete
+
+# Temp files
+find /tmp -type f -mtime +7 -delete
+find /var/tmp -type f -mtime +30 -delete
+
+# Snap (if installed)
+snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
+  snap remove "$snap" --revision="$rev"
+done
+```
+
+### Special Cases
+
+| CT | Special GC |
+|----|-----------|
+| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
+| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
+| 106 (ra-h-os) | Check relay DB size, enforce TTL |
+| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |
+
+## Alert Templates
+
+### AMBER (75-84%)
+```
+⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Next scheduled GC will attempt cleanup. No immediate action needed.
+```
+
+### RED (85-94%)
+```
+🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
+Status: {resolved|still elevated — {reason}}
+```
+
+### CRITICAL (≥95%)
+```
+🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
+{service_status} — {manual_action if needed}
+@Kwame — container nearly full, resolved: {yes_no}
+```
+
+## Incident Log: 2026-07-04 — kagentz docker bloat
+
+### Discovery
+Kwame noticed kagentz was full, asked Abiba to investigate.
+
+### Diagnosis
+- CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
+- `/var/lib` = 55G (93% of disk)
+- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
+- 1 stopped container, 1 unused network, 15 build cache layers
+
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 35.67GB
+→ Post: 16G / 59G (27%), 41G free
+→ 5 dangling images removed (kagentz-bridge + old builds)
+→ 1 stopped container removed
+→ 1 unused network removed (kagentz-bridge_default)
+→ 15 build cache layers removed
+```
+
+### Root Cause
+Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
+dangling images and orphaned build cache. No automated GC was in place.
+
+### Preventive Measures
+- This contract now runs disk GC fleet-wide every 6 hours
+- Docker hosts get `docker system prune` on amber, `-a --force` on red
+- kagentz is flagged as HIGH risk due to development activity
+```
+
+Done. Here's what we have:
+
+## Storage Summary
+
+| System | Used | Free | Health |
+|--------|------|------|--------|
+| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
+| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
+| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
+| **/root/go** (abiba) | 773M | — | Fine |
+
+## New Contract: `disk-gc-threat-response.prose.md`
+
+Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+
+### Three-in-One
+
+1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+
+2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+
+3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
+
+### The incident root cause
+Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
+
+Want me to push this to the prose-contracts repo?
@@ -0,0 +1,308 @@
+# Prose Contract Authoring Guide
+
+Approach this as the lead systems engineer at a company where every contract is a
+production dependency — agents run from these every 15 minutes, every 6 hours,
+every Sunday at 3am. A stale IP or a missing section isn't a typo; it's an
+incident waiting to happen. Write every contract as if you'll be paged at 3am
+when it breaks.
+
+## Ground it in the subject
+
+Before writing a single line of YAML, answer three questions and state them in
+the description:
+
+1. **What system does this contract touch?** Name the hosts, CTs, containers,
+   and services explicitly. "The inference fleet" is vague. "GPU .8 (RTX 3090,
+   qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, ornith), and LiteLLM on CT
+   116" is specific.
+
+2. **Who runs this contract, and when?** State the agent, the trigger (cron,
+   manual, event-driven), and the expected duration. "Runs every 6 hours via
+   cron on abiba, takes ~120 seconds for a fleet scan."
+
+3. **What breaks if this contract is wrong?** Name the blast radius. "A stale
+   CT ID causes a restart of the wrong container. A wrong Grafana URL breaks
+   the monitoring dashboard for all agents."
+
+If you cannot answer these three questions concretely, do not write the
+contract yet — go verify against the live system first.
+
+## Design principles
+
+### Specificity over generality
+
+A contract that says "check all Docker containers" is worse than one that lists
+them by name. The list will go stale, but staleness is detectable (the CI lint
+catches it). Vagueness is invisible and uncatchable. Be specific, accept that
+specificity decays, and rely on the lint pipeline to catch the decay.
+
+### Ground truth is the live system, not the contract
+
+Every IP, port, hostname, and container name in a contract is a **live-state
+field**. Mark them with `VERIFY-BEFORE-USE` in the access matrix. Before any
+mutating operation, verify against the live system. The contract is a
+hypothesis; the live system is the fact.
+
+When you discover drift between contract and reality, do not "fix" the live
+system to match the contract. Halt. Ask which is correct. Document the answer
+in the contract and the knowledge graph. The `.117→.19` incident was caused by
+fixing the wrong direction.
+
+### Contracts are typed — respect the kind
+
+| Kind | When to use | Must have | Must NOT have |
+|------|------------|-----------|---------------|
+| `function` | Stateless, callable helper | `## Parameters`, `## Returns`, `## Execution` | `## Maintains` (no state) |
+| `responsibility` | Persistent duty with state | `## Maintains`, `## Continuity` or `## Execution` | — |
+| `gateway` | External-driven entry point | `## Maintains` | `## Requires` |
+| `pattern` | Reference/template/doctrine | Flexible — prose-driven | Don't `prose run` directly |
+| `test` | Verification only | `## Execution` | Mutating operations |
+
+Use `kind: architecture` sparingly — it's not a standard OpenProse kind. Most
+architecture docs should be `kind: pattern`.
+
+### One contract, one concern
+
+A contract that monitors health AND deploys exporters AND sends alerts is three
+contracts fighting for attention. Split them:
+
+```
+❌ infrastructure-monitoring.prose.md (deploy + export + alert)
+✅ proxmox-monitor.prose.md (deploy exporters)
+✅ gpu-monitor.prose.md (poll + dashboard)
+✅ infrastructure-control.prose.md (reference topology)
+```
+
+The contract that deploys is a function. The contract that polls is a
+responsibility. The contract that maps the topology is a pattern. One concern
+per file, one kind per concern.
+
+### Live-state fields are declared, not hidden
+
+Don't bury IPs and hostnames in prose paragraphs. Put them in tables, mark them
+with `VERIFY-BEFORE-USE`, and keep them in one place per contract:
+
+```markdown
+| Host | IP | Role | Trust |
+|------|----|------|-------|
+| CT 116 | 192.168.68.116 | LiteLLM + Grafana | ⚠️ VERIFY-BEFORE-USE |
+```
+
+The lint pipeline scans for known stale values. Tables are grep-able; prose is
+not.
+
+## Process: verify, draft, lint, review, ship
+
+### Pass 1: Verify
+
+Before writing, verify every live-state field you plan to include:
+
+```bash
+# Example: writing a contract about Zulip
+curl -s http://192.168.68.19/api/v1/server_settings  # is .19 reachable?
+ssh root@192.168.68.19 "docker ps"                     # is Zulip in Docker?
+pct config 117 | grep net0                              # what IP does CT 117 have?
+```
+
+Write down the verified values. Use them in the contract. If you can't verify
+something, mark it explicitly: `⚠️ UNVERIFIED — needs live check`.
+
+### Pass 2: Draft
+
+Write the contract following the structure for its kind. Fill every required
+section. Use the templates below. Write descriptions that name the subject: "GPU
+fleet health check across RTX 3090 (.8), RTX 5070 (.110), and Strix Halo (.15)"
+not "monitoring check."
+
+### Pass 3: Lint locally
+
+Before pushing, run the same checks the CI will run:
+
+```bash
+bash scripts/prose-lint.sh
+```
+
+Fix everything that fails. The lint checks for:
+- Valid YAML frontmatter
+- Required sections per kind
+- Regression patterns (/grafana/ route, CT 122/123)
+- Missing Maintains/Parameters/Returns
+
+### Pass 4: Self-critique
+
+Read your contract and ask:
+
+1. **Could an agent run this at 3am without context?** If it needs tribal
+   knowledge, add it to the description or a comment.
+2. **Does any value contradict the infrastructure-control pattern?** Check CT
+   IDs, IPs, hostnames, nginx routes, Grafana access.
+3. **Is anything in here that was previously fixed and reverted?** Check the
+   lint regression rules. If you're unsure, grep the git log for similar
+   changes.
+4. **Am I saying the same thing as another contract?** If yes, link to it
+   instead of duplicating. One source of truth.
+
+### Pass 5: Open PR, let CI review
+
+Push to a branch, open a PR. The pipeline runs:
+
+```
+auth → validate → lint → ai-review → gate
+```
+
+The AI review compares your diff against the infrastructure-control ground
+truth. It catches contradictions you might miss. If it flags something, don't
+argue — verify and fix. The AI is reading the same ground truth you should have
+read before writing.
+
+## Restraint and self-critique
+
+Spend your complexity budget in one place. A contract that deploys Prometheus,
+configures Grafana, provisions dashboards, and sets up exporters is a function
+that does four things — each of which can fail independently. Split them.
+
+Don't add a `### Requires` that you haven't verified works. A dependency on
+"SSH access to .116" is only valid if you've tested it from the agent that
+will run the contract.
+
+Cut sections that don't serve the runner. A 200-line appendix of curl commands
+is documentation, not execution. Put it in a wiki or a README, not in the
+contract. The contract is for the agent running it; the README is for the human
+reading about it.
+
+Before shipping, take one thing out. Like Chanel's advice: before leaving the
+house, remove one accessory. Is there a section, a parameter, a check that
+doesn't pull its weight? Cut it. The contract that ships with 5 sections will
+be maintained. The one with 15 will rot.
+
+## Writing in contracts
+
+### Descriptions
+
+A description is not a label. It's the first thing an agent reads before
+deciding whether to run this contract. Make it answer: what does this do, to
+what, and why should I care?
+
+```
+❌ "Monitors infrastructure health"
+✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker
+   container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
+```
+
+### Field names
+
+Name things by what they control, not by how the system is built:
+
+```
+❌ prometheus_scrape_interval_seconds
+✅ check_every_seconds
+```
+
+The runner doesn't care that Prometheus is doing the scraping. They care how
+often the check runs.
+
+### Error messages and alerts
+
+Errors tell the operator what went wrong and what to do about it. Be specific:
+
+```
+❌ "Health check failed"
+✅ "GPU .15 (Strix Halo) unreachable at :8080 — firewalled to .116 only.
+   Check router /health/unified at http://192.168.68.116/health/unified instead."
+```
+
+### Comments
+
+Comments in contracts explain WHY, not WHAT. The execution steps say what to
+do. Comments say why we do it this way and not another:
+
+```markdown
+## Execution
+
+1. **Check GPU health via router** — cannot probe .15:8080 directly
+   (firewalled to .116 only as of 2026-07-01 Vulkan rebuild)
+2. **Skip sidecar check** — JSON exporters at :8090 were never deployed
+   on any GPU host; router falls back to GPU /health direct probe
+```
+
+## Templates
+
+### Function contract template
+
+```markdown
+---
+kind: function
+name: contract-name
+description: >
+  What this does, to what, and why. Be specific about hosts/services.
+version: 1.0.0
+---
+
+## Parameters
+
+- param_one: type — description (default: "value")
+- param_two: type — description (default: "value")
+
+## Returns
+
+- result_field: type — description
+
+## Requires
+
+- Dependency one — how to verify it's available
+- Dependency two
+
+## Execution
+
+1. **Step one** — what and why
+2. **Step two** — what and why
+3. **Compile and return** — assemble the output
+```
+
+### Responsibility contract template
+
+```markdown
+---
+kind: responsibility
+name: contract-name
+description: >
+  What this recurring duty monitors/maintains. Name the systems, the
+  cadence, the blast radius.
+version: 1.0.0
+---
+
+## Maintains
+
+- state_field: { current: value, last_check: timestamp }
+- another_field: description of maintained state
+
+## Parameters
+
+- param: type — description (default: "value")
+
+## Requires
+
+- Dependency one
+- Dependency two
+
+## Continuity
+
+- Self-driven: check every N seconds/minutes/hours
+- Also wakes on: trigger description
+
+## Execution
+
+1. **Check** — verify current state
+2. **Evaluate** — compare against thresholds
+3. **Remediate or escalate** — fix or alert
+4. **Record** — update world-model state
+```
+
+## Enforcement
+
+This guide is aspirational, not gated. The CI pipeline enforces structure
+(frontmatter, required sections) and regression (known stale patterns). It
+does not enforce prose quality or specificity. That's on you, the author.
+
+Before you ship: read your contract as if you're the agent who will run it at
+3am with 30 seconds of context. If you wouldn't trust it, rewrite it.
@@ -0,0 +1,194 @@
+---
+kind: responsibility
+name: gpu-fleet
+description: >
+  Manages the GPU inference fleet across all hosts. Handles model deployment,
+  registration, health checks, LiteLLM sync, agent key management, GPU
+  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
+  Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
+  router runs behind LiteLLM on :9000 (internal-only).
+agent: abiba
+triggers:
+  - on model add/remove
+  - on GPU health degradation
+  - on agent key rotation
+  - on router restart (roster must be loaded)
+---
+
+## Maintains
+
+- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
+- router: { status: "healthy", roster_loaded: bool, models: array }
+- litellm: { status: "healthy", keys: array, models: array }
+- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
+- health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
+- monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
+- watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
+- benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
+- grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
+- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
+
+## Fleet Topology (Current — June 2026)
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│  CT 116 (192.168.68.116) — Inference Harness Host                │
+│                                                                  │
+│  nginx:80 (entrypoint)                                           │
+│  ├─ /v1/*       → harness-litellm:4000   (API requests)         │
+│  ├─ /admin/*    → harness-litellm:4000   (admin endpoints)       │
+│  ├─ /dashboard/ → harness-dashboard:3000 (harness UI)            │
+│  ├─ /litellm/*  → harness-litellm:4000   (LiteLLM UI + API)     │
+│  ├─ /health/*   → harness-litellm:4000   (health probes)         │
+│  └─ /gpu/*      → 192.168.68.24:9100     (fleet monitor)         │
+│                                                                  │
+│  Containers:                                                     │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
+│  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
+│  │ :4000    │─▶│ :9000    │  │ :3000    │  │ :3000     │        │
+│  │ keys+sync│  │internal  │  │ harness  │  │ Prometheus│        │
+│  │ fallback │  │only!     │  │ UI       │  │ data src  │        │
+│  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
+│                    │                                             │
+│  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
+│  │PostgreSQL│  │  Redis   │  │Prometheus│                       │
+│  │ :5432    │  │ :6379    │  │ :9090    │                       │
+│  └──────────┘  └──────────┘  └──────────┘                       │
+└────────────────────┼─────────────────────────────────────────────┘
+                     │
+     ┌───────────────┼───────────────┬──────────────────┐
+     │               │               │                  │
+┌────▼─────┐  ┌──────▼──────┐  ┌────▼──────┐  ┌───────▼──────┐
+│ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
+│ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
+│ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
+│          │  │             │  │ llama-srv │  │ Watchdog     │
+│ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
+│ 27B-code │  │ :8090       │  │ :8080     │  │ exporter     │
+│ :8090    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
+│ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
+└──────────┘
+```
+
+## Current Model Assignments
+
+| Model | GPU | Host | VRAM | Status |
+|-------|-----|------|------|--------|
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |
+
+## Operations
+
+### add-model
+1. Download model file from Hugging Face or source
+2. Check disk space + GPU VRAM compatibility
+3. Start llama-server via systemd service on GPU host
+4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
+5. Hot-reload or restart router
+6. Add model to LiteLLM config (`model_list` + `fallbacks`)
+7. Generate agent keys for new model access via `/key/generate`
+8. Add Prometheus scrape target for the new GPU exporter
+9. Verify end-to-end: LiteLLM → Router → Model
+
+### remove-model
+1. Drain active requests (wait for active=0)
+2. Remove from LiteLLM config
+3. Remove from router config
+4. Stop llama-server (systemd)
+5. Remove Prometheus scrape target
+6. Cleanup model files (optional)
+
+### heal
+1. Check all GPUs via router internal `:9000/health/unified`
+2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
+3. Reset stuck circuit breakers if idle (Redis)
+4. Restart dead llama-server instances via SSH
+5. Flush Redis active counters if stale
+6. Verify GPU monitor server is running on pi (:9100)
+7. Verify watchdog is running on pi
+8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
+9. Reload roster via `POST :9000/admin/roster/reload` if available
+
+### sync-keys
+1. List all agent keys in LiteLLM DB via `GET /key/list`
+2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
+3. Generate missing keys via `POST /key/generate` with unlimited budget
+4. Update agent configs — `/etc/environment` LITELLM_API_KEY
+5. Send Zulip DM to agents that can't be reached via SSH
+6. Verify each key with test request through full chain
+7. Document keys in knowledge graph
+
+### list
+Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys
+
+## Agent Keys (LiteLLM DB — Current 2026-06-30)
+
+| Agent | Key | Host | Access |
+|-------|-----|------|--------|
+| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
+| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
+| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
+| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
+| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
+| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
+
+**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
+If no SSH access, send Zulip DM via abiba-bot.
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
+| `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
+| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
+| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
+| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
+| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
+| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
+
+## Prometheus & Grafana
+
+| Component | URL | Details |
+|-----------|-----|---------|
+| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
+| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
+| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
+| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
+| Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |
+
+## Known Issues & Watch Points
+
+- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
+  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
+- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
+- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
+- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
+- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
+- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
+- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
+- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
+- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
+- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
+- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
+- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
+- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.
+
+## GPU Inference Benchmarks (Current)
+
+| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
+|-----|-------|-----------|--------------|----------|---------|
+| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
+| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
+| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
+
+Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
+Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
+History stored at `/root/data/toks-history.json` with 7-day rolling window.
+
+**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
@@ -0,0 +1,136 @@
+---
+kind: responsibility
+name: gpu-monitor
+description: >
+  Comprehensive GPU fleet monitor — polls every subsystem (sidecars, router,
+  LiteLLM, Strix Halo, dashboard) every 15s, renders a live HTML dashboard,
+  checks alert thresholds, and exposes a JSON API for downstream consumers.
+agent: abiba
+---
+
+## Maintains
+
+- gpu-fleet-data: { gpus: array, strix: object, router: object, litellm: object, summary: object, alerts: array } — Full fleet snapshot refreshed every 15s
+- dashboard: live HTML at `/root/dashboard/gpu-fleet.html`, served on port 9100
+- gpu-data-api: JSON at `GET /gpu-data` on port 9100
+- health-endpoint: `GET /health` on port 9100 → 200 if cache populated, 503 if warming up
+
+## GPU Fleet Topology
+
+```
+┌─────────────────────────────────────────────────────────┐
+│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  Polls every 15s, serves dashboard + JSON API            │
+└──┬──────────┬──────────┬────────────────────────────────┘
+   │          │          │
+   ▼          ▼          ▼
+┌──────┐ ┌──────┐ ┌────────┐
+│.8:8080│ │.110  │ │.116:80 │
+│RTX3090│ │:8080 │ │nginx   │
+│gemma  │ │RTX5070│ │router  │
+└──────┘ │qwen27B│ │LiteLLM │
+         └──────┘ │dashboard│
+                  └────────┘
+
+Note: JSON sidecar exporters at :8090 were never deployed on any
+GPU host. Router falls back to GPU /health direct probe. Monitor
+should use router /health/unified as source of truth for GPU status.
+Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+poll .15:8080 directly; must go through router on .116.
+```
+
+### Subsystems Polled
+
+| Subsystem | Endpoint | Frequency | Metrics |
+|-----------|----------|-----------|---------|
+| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
+| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
+| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
+| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
+| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
+| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
+
+### Alert Delivery
+
+All alerts are sent to `#agent-hub` stream topics:
+- `alerts-gpu` — GPU fleet alerts (temp, VRAM, utilization)
+- `alerts-pm2` — PM2 process health (restarts, crashes)
+- `alerts-infra` — Infrastructure health (LiteLLM, Zulip, containers)
+
+This replaces the previous DM-only delivery. All agents on the mesh can see and respond to alerts.
+
+## Alert Thresholds
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| GPU Temp | >80°C | >90°C |
+| VRAM Usage | >90% | >95% |
+| GPU Util | >95% | >98% |
+| Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
+| Model Down | — | critical (circuit breaker open) |
+
+### JSON API Response Schema (/gpu-data)
+
+```json
+{
+  "gpus": [{ "name", "gpu_name", "temp_c", "vram_used_mb", "vram_total_mb",
+             "gpu_util_pct", "power_w", "power_limit_w", "fan_pct",
+             "models", "hostname", "active_requests", "health_score",
+             "circuit_open" }],
+  "strix": { "status", "cpu_load", "llama_health" },
+  "router": { "available_models", "circuit_breaker", "gpus", "scores",
+              "status", "redis", "_basic" },
+  "litellm": { ... },
+  "dashboard": { "reachable": bool },
+  "summary": { "fleet_status", "models_available", "models_total",
+               "circuit_breakers_open", "gpu_count", "gpu_errors",
+               "strix_running", "router_reachable", "litellm_reachable" },
+  "alerts": [{ "gpu", "metric", "level", "value", "threshold" }],
+  "updated": "ISO8601",
+  "monitor_version": "2.0.0"
+}
+```
+
+## Operations
+
+### list
+`curl http://localhost:9100/gpu-data | jq` — Full fleet status
+
+### check-health
+`curl http://localhost:9100/health` — Monitor self-check
+
+### view-dashboard
+Open `http://localhost:9100/` in browser — Live HTML dashboard
+
+### restart
+```bash
+pkill -f gpu-monitor-server.py
+python3 /root/scripts/gpu-monitor-server.py &
+```
+Or via PM2: `pm2 restart gpu-monitor`
+
+### check-router
+The router health is accessed through nginx on port 80 (NOT port 9000 directly).
+`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy
+`curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
+
+## Execution
+
+1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
+2. **Poll router** (every 15s): GET .116/health via nginx:80
+3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
+4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
+5. **Poll dashboard** (every 15s): GET .116/dashboard/
+6. **Check alerts**: Compare metrics against thresholds
+7. **Compute summary**: Fleet-wide health aggregation
+8. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
+9. **Serve API**: HTTP server on port 9100
+10. **Repeat** every 15 seconds
@@ -10,6 +10,6 @@ description: A simple hello world contract to test OpenProse on pi
 ## Returns
 - greeting: string — The generated greeting message

-## Ensures
+## Maintains
 - The greeting includes the provided name
 - The greeting is friendly and warm
@@ -4,132 +4,100 @@ name: hermes-config-template
 description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
-  RA-H OS MCP) while keeping model/provider choices flexible per agent role.
-  Every new agent profile should start from this template.
+  RA-H OS MCP) while keeping agent-specific API keys and model choices.
+  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
 ---

 ## Maintains

- template_version: string — Current template version (e.g., "1.0.0")
- last_applied: timestamp — When any agent was last configured from this template
- agents_configured: array — Which agents/profiles were created from this template
- infra_endpoints_verified: array — Firecrawl, SearXNG, RA-H OS, LiteLLM endpoints last confirmed working
- known_deviations: array — Profiles that diverge from the template (and why)
+- template_version: "2.1.0"
+- last_applied: timestamp
+- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
+- agent_keys: map (see Agent Keys section)
+- infra_endpoints_verified: array

-## Parameters
+## Agent Keys (LiteLLM — Current 2026-07-04)

- agent_name: string — Name of the agent/profile (e.g., "syslog-code", "syslog-devops")
- default_model: string — Agent's primary model (e.g., "qwen3.6-27B-code", "syslog-auto")
- default_provider: string — Inference provider for the primary model (default: "harness")
- fallback_model: string — Fallback model when primary is down (default: "gemma-4-12b")
- fallback_provider: string — Fallback provider (default: "harness")
- auxiliary_model: string — Model for vision/compression/extraction tasks (default: "gemma-4-12b")
- enable_firecrawl_searxng: boolean — Whether to configure web search/extract (default: true)
- enable_ra_h_os_mcp: boolean — Whether to wire up RA-H OS MCP bridge (default: true)
- enable_compression: boolean — Whether to enable context compression (default: true)
- custom_provider_name: string — Custom provider name (default: "harness")
- custom_provider_model: string — Custom provider model (default: same as default_model)
- custom_provider_url: string — LiteLLM base URL (default: "http://litellm.sysloggh.net/litellm/v1")
- extra_auxiliary: array — Additional auxiliary tasks to configure (vision, compression, etc.)
- output_dir: string — Where to write the config (default: "~/.hermes/profiles/<agent_name>/config.yaml")
+Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM
+PostgreSQL DB via `POST /key/generate` on CT 116. Keys are stored in the DB, not in
+config files. The env var `LITELLM_API_KEY` is set in `/etc/environment` on each agent
+host AND in `~/.hermes/.env` for gateway env propagation.
+Sub-agent profiles inherit auth from the main config — no separate keys needed.

-## Quickstart — How to Run This Contract
+| Agent | Key Alias | Host | SSH | Sub-Agents |
+|-------|-----------|------|-----|-----------|
+| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
+| Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
+| Abiba | `abiba-*` | 192.168.68.24 | local | — |
+| Koby | `koby-*` | ? | Zulip | — |
+| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — |
+| Kagenz0 | `kagenz0-*` | ? | Zulip | — |

-### For the Agent Running This Contract
-
-Read this section first. It tells you exactly what to do.
-
-**When to run:**
- Setting up a **new agent profile** for the first time
- An **existing profile** is missing web search, firecrawl, or MCP configs
- User says: `"apply the config template"` or `"fix my infra config"`
- User says: `"configure <agent_name> with <model>"`
-
-**How to run (via OpenProse CLI):**
-```bash
-# Minimal — only agent name required
-prose run hermes-config-template agent_name=syslog-code
-
-# Full control
-prose run hermes-config-template \
-  agent_name=syslog-devops \
-  default_model=claude-sonnet-4 \
-  auxiliary_model=gemma-4-12b \
-  fallback_model=deepseek-chat \
-  fallback_provider=deepseek
-```
-
-### How to Follow the Template (Manual)
-
-If `prose` CLI is not available:
-
-1. **Fetch this contract:** `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/hermes-config-template.prose.md`
-2. **Open the target config.yaml** — the profile you're updating
-3. **Compare** each of the 5 required sections below against what exists in the profile
-4. **For each section:**
-   - If it says **SHARED INFRA — DO NOT CHANGE**: Copy the value exactly as shown
-   - If it says **USER CHOOSES**: Use the agent's assigned model/provider
-   - If missing entirely: Add the full section
-5. **Verify** every endpoint responds (see Verification section at the end)
-6. **Restart** the gateway
-
-### Decision Tree
-
-```
-What kind of profile is this?
-├── Main profile (~/.hermes/config.yaml)
-│   └── Replace web.*, mcp_servers.*, compression.*, auxiliary.*, custom_providers.*
-│   └── model.default + fallback: use current values (USER CHOOSES)
-│
-├── Worker profile (~/.hermes/profiles/<name>/config.yaml)
-│   ├── Does it have web.search_backend?
-│   │   ├── NO → Add the full web.* section (copy exactly)
-│   │   └── YES → Verify it points to the right IP
-│   ├── Does it have mcp_servers.ra-h-os?
-│   │   ├── NO → Add mcp_servers.* section (copy exactly)
-│   │   └── YES → Verify URL
-│   └── model.default: USE the worker's assigned model (don't copy main)
-│
-└── Non-Syslog profile (personal/lifestyle agent)
-    └── Skip — use default Hermes config or Tanko's config instead
-```
+✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
+  syslog-review, syslog-writer — all at `/root/.hermes/profiles/<name>/config.yaml`

 ## Infrastructure Stack

-This template provisions the following shared infrastructure. Every agent should point to the same endpoints unless explicitly overridden.
-
 | Component | Endpoint | Purpose |
 |---|---|---|
 | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
 | SearXNG | `http://storepve:8888` | Privacy-respecting web search |
-| LiteLLM | `http://litellm.sysloggh.net/litellm/v1` | Unified model gateway |
+| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
+| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
 | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
 | Context7 MCP | `http://localhost:8079/mcp` | Documentation queries |

-**API Key Rules:**
- `api_key: sk-_SW...B8nw` — Hardcoded LiteLLM master key for custom_providers and auxiliary tasks
- `api_key_env: DEEPSEEK_API_KEY` — Env-var based key for fallback provider
- Worker profiles should inherit proxy config via `harness` custom_provider, NOT hardcode LiteLLM keys
+## API Key Rules

-**For workers that need API keys overridden:** Set `api_key: ''` in the model section (inherits from custom_providers), or use `api_key_env` for env-based auth.
+- `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred)
+- `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider
+- `api_key: sk-...` — Hardcoded key only as fallback when env var not possible
+- Set `LITELLM_API_KEY` in `/etc/environment` on each host
+- Sub-agents NEVER get their own key — they share the host agent's key
+- Restart Hermes after updating `/etc/environment`

-## Template Structure
+### Sub-Agent Profiles (Mumuni pattern)

-### Required Sections (every agent MUST have)
+Mumuni has 6 sub-agent profiles in `/root/.hermes/profiles/<name>/config.yaml`:
+
+```
+profiles/
+├── syslog-code/config.yaml      # Code generation
+├── syslog-devops/config.yaml     # DevOps/infrastructure
+├── syslog-email/config.yaml      # Email processing
+├── syslog-research/config.yaml   # Research & analysis
+├── syslog-review/config.yaml     # Code review
+└── syslog-writer/config.yaml     # Content writing
+```
+
+Sub-agent profile rules:
+1. **`api_key` must be empty** — `api_key: ''` or omitted entirely
+2. **`base_url` must be empty** — inherits from main config's custom_provider
+3. **`provider` is `auto` or `harness`** — routes through the shared LiteLLM gateway
+4. **`model` is agent-specific** — each sub-agent can have its own default model
+5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty
+6. **Never hardcode a key** in sub-agent profiles
+
+This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`.
+When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
+work immediately after restart.
+
+## Template — Required Sections

 ```yaml
-# ─── Model Selection (USER CHOOSES) ───
+# ─── Model Selection ───
 model:
-  # !! CHANGE THIS for your agent !!
-  default: <agent_default_model>
-  provider: <agent_default_provider>
-  # LiteLLM base URL (shared infra)
-  base_url: http://litellm.sysloggh.net/litellm/v1
+  default: <agent_model>          # e.g., ornith-1.0-35b, qwen3.6-27B-code
+  provider: harness
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
+  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
+  context_length: 262144          # Must match model's max context window

 fallback_providers:
-  # !! OPTIONAL: overridable fallback !!
-  provider: <fallback_provider>
-  model: <fallback_model>
+  provider: deepseek
+  model: deepseek-chat
+  api_key_env: DEEPSEEK_API_KEY

 # ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
 web:
@@ -139,7 +107,7 @@ web:
  firecrawl:
    base_url: http://192.168.68.7:3002/

-# ─── MCP Servers (SHARED INFRA — DO NOT CHANGE) ───
+# ─── MCP Servers (SHARED INFRA) ───
 mcp_servers:
  context7:
    connect_timeout: 60
@@ -150,86 +118,65 @@ mcp_servers:
    timeout: 120
    connect_timeout: 60

-# ─── Compression (SHARED — can override model) ───
+# ─── Compression ───
 compression:
  enabled: true
+  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
  provider: harness
-  model: <auxiliary_model>  # default: gemma-4-12b
-  threshold: 0.5
-  target_ratio: 0.25
-  protect_last_n: 30
-  hygiene_hard_message_limit: 400
+  max_context_window: 262144      # Must match model's actual capacity
+  threshold: 0.65                 # Fires at ~170K for 262K window (not 0.25!)
+  target_ratio: 0.30
+  protect_last_n: 40
+  hygiene_hard_message_limit: 350
+  protect_first_n: 3
+  abort_on_summary_failure: false

-# ─── Auxiliary Tasks (SHARED INFRA + MODEL) ───
+# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
+# All auxiliary services MUST use identical model, base_url, and api_key_env:
+#   model: gemma-4-12b
+#   base_url: http://192.168.68.116/v1
+#   api_key_env: LITELLM_API_KEY
+# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
+# gemma-4-12b is a lightweight 12B model on the RTX 5070, freeing the Strix Halo
+# for agent reasoning.
 auxiliary:
  vision:
    provider: harness
-    model: <auxiliary_model>
+    model: gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    timeout: 60
+    download_timeout: 30
  web_extract:
    provider: harness
-    model: <auxiliary_model>
-  session_search:
+    model: gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    timeout: 30
+  compression:
    provider: harness
-    model: <auxiliary_model>
-    max_concurrency: 3
+    model: gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    timeout: 60

-# ─── Custom Provider (SHARED INFRA — LiteLLM) ───
+# ─── Custom Provider ───
 custom_providers:
-  - name: <custom_provider_name>
-    model: <custom_provider_model>
-    base_url: http://litellm.sysloggh.net/litellm/v1
-    api_key: <liteLLM_master_key>
+  - name: harness
+    model: <agent_model>
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
 ```

-### Optional Sections (agent-specific)
+## Key Update Procedure

- **Platform configs** (Telegram, Discord, WhatsApp bot tokens)
- **Display/skin** options (compact mode, personality)
- **Plugin lists** (zulip-platform, etc.)
- **Terminal settings** (container image, timeouts)
- **Security/approvals** (command allowlists)
- **Kanban/worker** settings (if this profile is a Kanban worker)
- **Skills auto_load** list
- **Timezone** override
+When LiteLLM keys are regenerated (e.g., after infrastructure changes):

-## Continuity
-
-Configuration drift detection is **event-driven**:
-
- **On agent onboarding**: Always scaffold from this template
- **On infra change**: If Firecrawl/SearXNG/LiteLLM endpoints change, update template + all profiles
- **On new model**: When a new model is deployed on Litellm, any agent can change their `default` independently
- **Periodic**: Every 2 weeks — check if any profile's infra section has drifted from the template
- **On connection failure**: If web search/extract fails, check if that agent has missing `web.*` config
-
-## Success Criteria
-
-The configuration is CONSIDERED GOOD when ALL of these pass:
-
-### Infra Connectivity
- `web.backend == "firecrawl"` — Firecrawl backend configured
- `web.search_backend == "searxng"` — SearXNG search configured
- `web.extract_backend == "firecrawl"` — Firecrawl extraction configured
- `web.firecrawl.base_url` points to `192.168.68.7:3002`
- `mcp_servers.ra-h-os.url` points to `192.168.68.65:3100/mcp`
- All endpoints respond to `curl` health checks
-
-### Model Flexibility
- `model.default` is settable per agent (not hardcoded)
- `model.provider` is settable per agent
- `custom_providers[0].model` matches the agent's workload (code vs general)
-
-### Completeness
- `auxiliary.vision` configured (even if provider=auto)
- `compression.enabled` set (true for most agents)
- Fallback provider configured (deepseek or another harness model)
- `_config_version` correctly set (currently `30`)
-
-### No Drift
- No duplicate infra endpoints (one canonical Firecrawl URL)
- No stale endpoints (old IPs, dead services)
- All profiles consistent on shared infra
+1. **If SSH available**: `ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"`
+2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command
+3. **After update**: Restart Hermes on the agent host
+4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`

 ## Configuration Rules

@@ -241,67 +188,60 @@ The following MUST be identical across ALL profiles:
 - `custom_providers[0].base_url`

 ### Rule 2: Model Choice Is Free
-The following are OWNED by each agent and can differ:
- `model.default` — the primary model
- `model.provider` — where it runs
- `fallback_providers.model` — backup model
- `custom_providers[0].model` — what this agent uses LiteLLM for
+- `model.default` — per agent
+- `fallback_providers.model` — per agent
+- `custom_providers[0].model` — per agent

-### Rule 3: Workers Inherit, Not Duplicate
-Worker profiles (syslog-code, syslog-devops, etc.) generally inherit from the main config. Only override what's DIFFERENT:
- Different default model → override `model.default`
- Different auxiliary model → override auxiliary sections
- No web search needed → still keep the config (tools won't enable without it)
+### Rule 3: API Keys via Environment
+- Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
+- Hardcoded keys in config.yaml become stale after key rotation
+- `/etc/environment` persists across config updates
+- Restart Hermes after env var updates

-### Rule 4: API Key Placement
- **Custom provider keys** (`custom_providers[0].api_key`): Must be hardcoded (the LiteLLM master key or a virtual key)
- **Model section keys** (`model.api_key`): Prefer empty. Workers inherit from custom_providers.
- **Environment keys** (`api_key_env`): For fallback providers that need separate auth (DeepSeek, OpenRouter)
- **Hardcoded keys in profile**: If a worker's `model.api_key` is set, it overrides custom_providers. Only use when the worker needs a DIFFERENT provider than Litellm.
+### Rule 4: Sub-Agent Profiles Inherit Auth
+- Sub-agent profiles (`/root/.hermes/profiles/*/config.yaml`) must have:
+  - `api_key: ''` — inherit from main config's custom_provider
+  - `base_url: ''` — inherit from main config
+  - Auxiliary tasks: `api_key: ''`, `provider: harness`
+- Never hardcode a key in sub-agent profiles
+- When main config uses `api_key_env`, sub-agents automatically use it
+- This means key rotation only touches ONE file (`/etc/environment`)

-### Rule 5: Verify After Every Config Change
-After changing a profile's config:
-1. `curl` the model endpoint to confirm auth works
-2. `curl` the web stack endpoints (Firecrawl, SearXNG)
-3. Check that `hermes gateway --restart` reloads cleanly
+### Rule 5: Main Config Base URL
+|- Use direct IP: `http://192.168.68.116/v1`
+|- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
+|- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly
+
+### Rule 6: max_tokens Is Required (Thermal Safety)
+- **Every Hermes config MUST set `model.max_tokens: 4096`** — this is non-negotiable
+- Prevents unbounded generation that caused the July 2 Strix Halo GPU thermal incident
+- The value flows through: `config → agent.max_tokens → transport build_kwargs → API max_tokens`
+- Even though the server now has `-n 8192` hard cap (set by Abiba), the client cap is the first line of defense
+- Apply to BOTH main config AND all sub-agent profiles
+- For agents needing longer outputs: raise to 8192, but never omit
+
+### Rule 7: Auxiliary Model Consistency
+- All auxiliary services (vision, web_extract, compression) MUST use the same model:
+  - `model: gemma-4-12b`
+  - `base_url: http://192.168.68.116/v1`
+  - `api_key_env: LITELLM_API_KEY`
+- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes to the primary 35B reasoning GPU
+- gemma-4-12b is a lightweight 12B model on the RTX 5070, keeping the Strix Halo free for reasoning
+- The `compression:` block's `model` MUST match `auxiliary: compression: model` — they are two different configs for the same service
+
+### Rule 8: Compression Threshold for 256K Models
+- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
+- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
+- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
+- `max_context_window: 262144` MUST match the model's actual capacity
+- See `devops-hermes-compression` skill for full reference

 ## Execution

 1. **Check current config** — Read the target agent's config.yaml
 2. **Compare against template** — Identify missing or divergent sections
-3. **Apply shared infra** — Lock the web/MCP/compression sections to template values
-4. **Apply model choice** — Set default/fallback/auxiliary models per agent's workload
-5. **Preserve agent-specific** — Keep platform configs, plugins, terminal settings, skills
-6. **Remove stale** — Drop any old infra endpoints that don't match the template
-7. **Verify** — curl all shared endpoints, test the model
-8. **Report** — What was changed, what was preserved, what's still custom
-
-## Example Output
-
-```
-## Config Template Applied: syslog-code ✅
-
-### Shared Infrastructure (Locked)
-| Section | Before | After |
-|---|---|---|
-| web.backend | firecrawl | firecrawl ✅ |
-| web.search_backend | NOT SET | searxng ✅ |
-| web.firecrawl.base_url | NOT SET | http://192.168.68.7:3002/ ✅ |
-| mcp_servers.ra-h-os | present | present ✅ |
-
-### Agent Model Choices (Preserved)
-| Setting | Value |
-|---|---|
-| model.default | qwen3.6-27B-code (code agent) |
-| fallback | gemma-4-12b (harness) |
-| auxiliary | gemma-4-12b (harness) |
-| compression | gemma-4-12b (harness) |
-
-### Endpoint Verification
-| Endpoint | Status |
-|---|---|
-| Firecrawl :3002 | ✅ 200 OK |
-| SearXNG :8888 | ✅ 200 OK |
-| LiteLLM | ✅ 200 OK |
-| RA-H OS MCP |  ✅ Connected |
-```
+3. **Apply shared infra** — Lock web/MCP/compression sections to template values
+4. **Apply agent key** — Set from agent_keys table above
+5. **Set model choice** — Per agent's workload
+6. **Verify** — curl all shared endpoints, test the model with the new key
+7. **Report** — What was changed, preserved, custom
@@ -0,0 +1,180 @@
+---
+kind: enforcement
+name: hermes-key-enforcement
+version: 1.0.0
+description: >
+  Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
+  providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
+  Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each
+  agent host. Designed to make key rotation a one-step operation.
+author: Abiba (pi agent)
+---
+
+# Hermes Key Enforcement Contract
+
+## Rule (One Sentence)
+
+**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.**
+
+## Scope
+
+Applies to all Hermes agent configs across all hosts. Covers these config sections:
+- `model.api_key`
+- `custom_providers[].api_key` (when `name` contains `harness` or `litellm`)
+- `auxiliary.*.api_key` (when `provider` is `harness` or contains `litellm`)
+- `delegation.api_key` (when `provider` is `harness` or contains `litellm`)
+- `compression.api_key` (when `provider` is `harness` or contains `litellm`)
+- `fallback_providers[].api_key` (when provider is harness)
+
+## Exemptions
+
+External providers are **explicitly exempt** and may use hardcoded keys:
+- DeepSeek (`api.deepseek.com`)
+- OpenAI (`api.openai.com`)
+- Anthropic (`api.anthropic.com`)
+- OpenRouter
+- Any provider whose base_url does NOT match `192.168.68.116` or `litellm.sysloggh.net`
+
+## Standard Pattern
+
+```yaml
+# ✅ CORRECT — all harness/litellm providers
+model:
+  provider: harness                    # or custom:litellm.sysloggh.net
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY        # ← indirection
+
+custom_providers:
+  - name: harness
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY      # ← indirection
+
+auxiliary:
+  compression:
+    provider: harness
+    api_key_env: LITELLM_API_KEY      # ← indirection
+
+# ✅ ALSO CORRECT — external providers
+fallback_providers:
+  - provider: deepseek
+    base_url: https://api.deepseek.com
+    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in /etc/environment
+```
+
+```yaml
+# ❌ FORBIDDEN — hardcoded harness/litellm key
+model:
+  provider: harness
+  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION
+```
+
+## Detection Query
+
+Run on any Hermes host to detect violations:
+
+```bash
+grep -n 'api_key: sk-' /root/.hermes/config.yaml \
+  | grep -v 'deepseek\|b7d9\|openai\|anthropic\|DEEPSEEK'
+```
+
+If any output — violation. Fix immediately.
+
+## Rotation Procedure
+
+With this standard enforced, key rotation is one step:
+
+```bash
+# 1. Generate new key in LiteLLM
+# 2. Update /etc/environment on agent host
+ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment"
+# 3. Restart agent gateway
+ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &"
+# 4. Verify
+curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
+```
+
+**Done.** No config file changes needed. The agent picks up the new key on restart.
+
+## Key Longevity Policy (2026-07-04)
+
+**Keys are permanent and use bare agent name aliases.**
+
+- **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
+- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
+- **Max budget**: $100 per key (config default).
+
+```yaml
+# In litellm_config.yaml — ensures all future keys inherit these defaults:
+litellm_settings:
+  default_key_generate_params:
+    models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b"]
+    duration: null        # ← permanent
+    max_budget: 100
+    metadata:
+      purpose: "agent-inference"
+```
+
+## Verified Agents (2026-07-04 final)
+
+| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified |
+|-------|-----|--------|---------------|----------|----------------|---------------|
+| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT |
+| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT |
+| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT |
+| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT |
+| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT |
+| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT |
+
+### Systemd Service Pattern (2026-07-04 fix)
+
+All Hermes agents use systemd to manage their gateway. Two issues were fixed:
+
+1. **Drop-in override** — `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` (or user equivalent) had hardcoded `LITELLM_API_KEY` that bypassed `/etc/environment`.
+2. **Missing EnvironmentFile** — Services did not source `/etc/environment`.
+
+**Correct pattern:**
+```ini
+# In service file:
+EnvironmentFile=/etc/environment
+
+# Drop-in only for overrides, NOT primary key storage.
+# If a drop-in exists, it must match /etc/environment.
+```
+
+**Rotation procedure** (one step with this standard):
+1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
+2. Update `/etc/environment`: `sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment`
+3. Update drop-in (if exists): same sed on `litellm-key.conf`
+4. Restart: `systemctl [--user] restart hermes-gateway`
+
+## Violation Response
+
+1. **Detect** — run detection query above
+2. **Fix** — replace `api_key: sk-...` with `api_key_env: LITELLM_API_KEY` in all harness/litellm sections
+3. **Verify** — `grep -c "api_key_env" config.yaml` should increase, hardcoded harness keys should be 0
+4. **Restart** — gateway must restart to pick up env var
+5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
+6. **Update** — bump the verified table above
+
+## Related Contracts
+
+- `hermes-config-template.prose.md` — full configuration template
+- `litellm-health.prose.md` — LiteLLM stack health verification
+- `zulip-platform-verification.prose.md` — cross-platform agent verification
+- `litellm-api-keys.prose.md` — API key creation, rotation, and verification
+
+## CI Pipeline (2026-07-04)
+
+All contract changes must pass the PR Pipeline before merge:
+
+```
+auth → validate → lint → ai-review → gate
+```
+
+- **Trigger**: push to master (abiba-bot only) or pull request
+- **Branch protection**: Only `abiba-bot` can push directly to master. All other users must use PRs.
+- **Status check**: `PR Pipeline — Authorize → Validate → Review → Merge` required before merge
+- **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
+- **Config**: `.gitea/workflows/pr-pipeline.yaml`
@@ -6,6 +6,12 @@ description: >
  5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
  NFS storage, and network services. Defines monitors, remediations,
  and the access matrix for all environments.
+
+  FIELD TRUST: Live-state fields (IPs, ports, hostnames, credentials,
+  container names, PIDs) are marked VERIFY-BEFORE-USE. They drift —
+  never mutate infrastructure based on them without first confirming
+  against the live system. Policy fields are authoritative. See the
+  `verify-before-mutate` skill.
 ---

 # Infrastructure Control Pattern
@@ -23,7 +29,7 @@ description: >
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
-   │   CT 100    │    │   CT 122     │    │   CT 114     │
+   │   CT 100    │    │   CT 112     │    │   CT 114     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
@@ -60,19 +66,30 @@ description: >
 | Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
 | docker-vm (.7) | SSH root | SSH key | ✅ |
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
-| Tanko CT (.122) | SSH jerome | SSH key | ✅ |
+| Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
+| Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
+| Koonimo CT (.114) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
 | Gitea | API token | abiba-bot token | ✅ |
-| Zulip | Bot API key | per-bot tokens | ✅ |
+| Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
 | RA-H OS | MCP bridge | port 3100 | ✅ |
+| LiteLLM Admin | master key | `sk-litellm-7f96...` | ✅ VERIFY-BEFORE-USE |
+| Grafana | admin password | `syslog-grafana-2026` | ✅ VERIFY-BEFORE-USE |
+
+> **VERIFY-BEFORE-USE**: Credentials, IPs, ports, and hostnames in this
+> contract are live-state fields. Test them against the live system before
+> relying on them (e.g., `curl -H "Authorization: Bearer $KEY" .../v1/models`
+> for a LiteLLM key; `pct config <ct>` for a CT IP). Drift is expected —
+> see `verify-before-mutate` skill. Policy fields (rules, doctrines) are
+> authoritative and do not need verification.

 ### Reachability Matrix

-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) |
-|-----------|---------|----------------|--------|-------------|---------------|
-| **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird |
-| **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ |
-| **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Koonimo (.114) |
+|-----------|---------|----------------|--------|-------------|---------------|----------------|
+| **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
+| **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
+| **Mumuni** (.123) | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |

 **Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.

@@ -133,29 +150,119 @@ description: >
 |-------|------|-----------|
 | **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
-| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse |
+| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |

+### Home Stack Services
+
+**Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
+- URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
+- Swagger: `http://192.168.68.7:8989/swagger-ui.html`
+- Admin credentials: `admin` / `kakashi20stirling`
+- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
+- Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
+- Compose: `/opt/home_stack/docker-compose.yml`
+- Control script: `/opt/home_stack/infra-control.sh`
+
+**JDownloader**:
+- URL: `http://192.168.68.7:5800` (web UI via VNC)
+
+**Pulse** (Uptime Kuma):
+- URL: `http://192.168.68.7:3001`
+
 ### Ecosystem B: CT 116 syslog-api (192.168.68.116)

-| Container | Image | Port |
-|-----------|-------|------|
-| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 |
-| harness-nginx | nginx:alpine | :80 |
-| harness-router | inference-harness-router | :9000 |
-| harness-postgres | postgres:16-alpine | :5432 |
-| harness-redis | redis:7-alpine | :6379 |
-| harness-dashboard | inference-harness-dashboard | :3000 |
+8 containers in inference-harness stack:

-### Ecosystem C: Netbird (72.61.0.17)
+| Container | Image | Port | Role |
+|-----------|-------|------|------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | API proxy, key mgmt, fallbacks |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | GPU routing, slot booking, CB |
+| harness-nginx | nginx:alpine | :80 | Entrypoint, /v1→LiteLLM, /dashboard/ |
+| harness-postgres | postgres:16-alpine | :5432 | LiteLLM DB (keys, spend, config) |
+| harness-redis | redis:7-alpine | :6379 | Router slots, circuit breakers |
+| harness-dashboard | inference-harness-dashboard | :3000 | SyslogAI Harness UI |
+| harness-grafana | grafana/grafana | :3000→:3001 (direct LAN, not behind nginx) | GPU + Proxmox + Docker dashboards |
+| harness-prometheus | prom/prometheus | :9090 | Metrics scraper, 6 jobs |
+
+**Nginx routing**:
+- `/v1/*` → harness-litellm:4000 (API)
+- `/admin/*` → harness-litellm:4000
+- `/dashboard/` → harness-dashboard:3000
+- `/litellm/*` → harness-litellm:4000
+- `/health/*` → harness-litellm:4000/health/liveliness
+- `/gpu/` → 192.168.68.24:9100 (fleet monitor)
+
+**Prometheus targets**:
+- 192.168.68.8:9400 (RTX 3090 — qwen)
+- 192.168.68.110:9400 (RTX 5070 — gemma)
+- 192.168.68.15:9400 (Strix Halo — ornith)
+- 192.168.68.24:9401 (Router metrics exporter)
+- harness-litellm:4000 (LiteLLM health)
+
+### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud)
+
+5 containers, single compose stack at `/root/docker-compose.yml`:

 | Container | Image | Role |
 |-----------|-------|------|
-| netbird-server | netbird | VPN controller |
-| netbird-dashboard | netbird UI | Web management |
-| netbird-proxy | nginx | TLS termination |
-| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF |
-| netbird-traefik | traefik | Reverse proxy |
+| netbird-server | netbirdio/netbird-server:latest | VPN controller, management API |
+| netbird-dashboard | netbirdio/dashboard:latest | Web management UI |
+| netbird-proxy | netbirdio/reverse-proxy:latest | SNI router + TLS passthrough for *.sysloggh.net |
+| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF / threat intelligence |
+| netbird-traefik | traefik:v3.6 | Edge reverse proxy, Let's Encrypt TLS |
+
+**Architecture:** Internet → Traefik (TLS + HTTP→HTTPS redirect) → Netbird proxy
+(TLS passthrough via HostSNI, PROXY protocol) → Netbird mesh → LAN backends.
+Traefik handles the Netbird dashboard directly; all other `*.sysloggh.net`
+domains are TLS-passthrough to the proxy on port 8443.
+
+**Proxy config:** `/root/proxy.env` — token, ACME certs in `/certs/`, geolocation
+DB auto-downloaded. Certs are per-domain Let's Encrypt via TLS-ALPN-01 challenge.
+
+> **Dependency note:** NetBird is an **access layer only** — it exists so a
+> human on a PC or away from the LAN can reach services by URL. It is **not**
+> a dependency of any service backend. Per Section 7, every config and agent
+> must reach backends by LAN IP; the `*.sysloggh.net` URLs are reserved for
+> human browsers.
+
+#### Known Issues
+
+**A. Access log bloat (CRITICAL)** — 2026-07-03 discovery. The management server
+writes an `access_log_entries` row for every proxied request (~45K/day). SQLite
+has no built-in TTL. By 2026-07-03 the table reached 10.7M rows / 5.4GB, causing:
+- DB writes slow → gRPC deadlines exceeded → proxy loses management connection
+- On proxy restart, management server too slow to push route mappings
+- Proxy starts HTTPS listener before routes arrive → all domains "unknown"
+- **Fix:** batch-delete old entries, VACUUM. Prevention: weekly systemd timer (`netbird-cleanup.timer`) deletes entries >7 days old when DB exceeds 500MB.
+- **Recurrence signal:** proxy logs show `unknown domain "git.sysloggh.net"` or `match: false` for known domains → management route sync failed.
+
+**B. Stale management state after proxy restart.** Management server sometimes
+stops pushing route mappings to a reconnecting proxy. Symptom: proxy starts,
+mapping stream established, but no routes arrive → all domains `unknown`.
+**Fix:** restart management server FIRST, THEN proxy. Order matters.
+
+**C. Peer `sha-OWLpE8nS4Rp4IWzep+y2ayFeZ6C3lJ4eNRB29L5yQTE=` flapping.**
+One peer (connecting through Traefik relay at 172.30.0.10) disconnects every
+~2-3 minutes. This may be a symptom of DB bloat (healthcheck timeout due to
+slow DB writes) rather than a network issue. Monitor after bloat fix.
+
+#### Recovery Procedures
+
+1. **Proxy losing all routes ("unknown domain" for all services):**
+   ```bash
+   ssh root@72.61.0.17
+   cd /root
+   docker compose restart netbird-server
+   sleep 12
+   docker compose restart proxy
+   ```
+2. **Access log bloat detected (DB > 500MB):**
+   ```bash
+   ssh root@72.61.0.17
+   /root/cleanup-access-logs.sh
+   ```
+3. **VPS hung (TCP accepts, SSH banner timeout):** hard reboot from Hostinger console.

 ### Checks (every 60s)

@@ -173,6 +280,13 @@ For each Docker host:
  - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
  - df -h / | awk '{print $5}' → usage < 80%

+For Netbird VPS specifically:
+  - netbird-store-db-size: du -m /var/lib/docker/volumes/root_netbird_data/_data/store.db → < 500MB
+  - netbird-access-log-count: sqlite3 store.db 'SELECT COUNT(*) FROM access_log_entries' → < 500K
+  - If DB > 500MB: alert + run /root/cleanup-access-logs.sh
+  - proxy-route-check: curl https://git.sysloggh.net/ → 2xx (proves routes loaded)
+  - proxy-sync-check: docker logs netbird-proxy --since 60s | grep 'Initial mapping sync complete' → present after restart
+
 For docker-vm specifically:
  - mountpoint -q /media/storage → NFS mounted
  - mountpoint -q /media/mediastore → NFS mounted
@@ -226,25 +340,103 @@ For docker-vm specifically:

 ## Section 5: Network Services — Monitoring

-| Service | Domain | Status |
-|---------|--------|--------|
-| Proxmox API | minipve.sysloggh.net:443 | ✅ |
-| Authentik | auth.sysloggh.net:443 | ✅ |
-| Gitea | git.sysloggh.net:443 | ✅ |
-| Zulip | chat.sysloggh.net:443 | ✅ |
-| LiteLLM | litellm.sysloggh.net:443 | ✅ |
-| Pulse | pulse.sysloggh.net:443 | ✅ |
-| SearXNG | searxng.sysloggh.net:8888 | ✅ |
-| Firecrawl | firecrawl.sysloggh.net:3002 | ✅ |
+### 5.1 Service Inventory

-### Checks (every 2 min)
+The `Resolves To` column is the load-bearing one. Services that CNAME to
+`netbird.sysloggh.net` (72.61.0.17) are routed through the NetBird reverse
+proxy and **fail whenever NetBird blips**, even though their LAN backend is
+fine. Services that resolve directly to a LAN IP are NetBird-independent.
+
+| Service | Domain | LAN Backend | Resolves To | NetBird-dep | Status |
+|---------|--------|-------------|-------------|-------------|--------|
+| Proxmox API | minipve.sysloggh.net:8006 | 192.168.68.12 | LAN IP | No | ✅ |
+| LiteLLM | litellm.sysloggh.net | 192.168.68.116 | LAN IP | No | ✅ |
+| Authentik | auth.sysloggh.net:443 | 192.168.68.11 | CNAME → netbird | **Yes** | ⚠️ |
+| Gitea | git.sysloggh.net:443 | 192.168.68.110 | CNAME → netbird | **Yes** | ⚠️ |
+| Zulip | chat.sysloggh.net:443 | 192.168.68.19 | CNAME → netbird | **Yes** | ⚠️ VERIFY-BEFORE-USE |
+| Pulse | pulse.sysloggh.net:443 | 192.168.68.7 | CNAME → netbird | **Yes** | ⚠️ |
+| DNS UI | dns.sysloggh.net:443 | 192.168.68.102 | CNAME → netbird | **Yes** | ⚠️ |
+| SearXNG | searxng.sysloggh.net:8888 | 192.168.68.7:8888 | LAN IP | No | ✅ |
+| Firecrawl | firecrawl.sysloggh.net:3002 | 192.168.68.7:3002 | LAN IP | No | ✅ |
+
+**Verified 2026-07-02:** NetBird VPS rebooted after a hang; all CNAME'd
+services recovered. LAN-IP-direct paths stayed up throughout the outage.
+
+### 5.2 Checks (every 2 min)

 ```
 ## Maintains

- dns-resolution: { services: [{name, resolves}] }
+- dns-resolution: { services: [{name, resolves_to, is_lan_ip}] }
 - ssl-expiry: { services: [{name, days_remaining}] }
 - endpoint-reachability: { services: [{name, http_code}] }
+- netbird-dependency: { services: [{name, depends_on_netbird: bool}] }
+```
+
+### 5.3 Network Verification & Routing (every 5 min)
+
+This block is what catches a NetBird blip before it becomes an outage. It
+proves two independent paths for every service: the **URL path** (what a
+browser uses, may cross NetBird) and the **LAN-IP path** (what configs and
+agents must use, never crosses NetBird).
+
+```
+## Maintains
+
+- dual-path-reachability: {
+    services: [{
+      name: string,
+      url_path: { http_code, tls_ok },          # the *.sysloggh.net path
+      lan_path:  { ip, port, http_code },        # the IP:port path
+      lan_independent: bool                       # lan_path works when url_path fails
+    }]
+  }
+- netbird-ingress-health: {
+    vps_up: bool,              # TCP 22/80/443 accept on 72.61.0.17
+    ssh_banner_ok: bool,       # banner exchange completes (catches hung box)
+    containers_up: int,        # all 5 netbird-* containers running
+    proxy_routes_ok: int       # traefik routers resolving (chat/git/auth return 2xx/3xx)
+  }
+- routing-regression: {
+    new_netbird_cnames: [domain],   # domains that newly CNAME to netbird.sysloggh.net
+                                    # → alert: a service just became NetBird-dependent
+    config_url_violations: [file]   # configs/agents found referencing a *.sysloggh.net
+                                    # URL where a LAN IP is required (see Section 7)
+  }
+
+## Checks
+
+- For each service in 5.1:
+  - curl https://<domain>/ → record url_path (code, tls_ok)
+  - curl http://<lan_ip>:<port>/ → record lan_path
+  - If url_path fails AND lan_path succeeds → netbird-degraded, NOT service-down
+  - If BOTH fail → service-down (real outage)
+
+- NetBird ingress (72.61.0.17):
+  - TCP probe 22, 80, 443 → all accept
+  - SSH banner exchange completes within 10s (catches the hung-box symptom
+    where TCP accepts but sshd never sends its banner)
+  - ssh root@72.61.0.17 'docker ps' → 5/5 netbird-* containers Up
+  - curl https://chat.sysloggh.net/ → 2xx/3xx (proxy route alive)
+
+- DNS dependency drift:
+  - For each *.sysloggh.net, dig +short → flag any new CNAME → netbird.sysloggh.net
+  - A service moving FROM LAN-IP TO netbird CNAME is a regression (alert)
+  - A service moving FROM netbird CNAME TO LAN-IP is an improvement (log)
+
+## Remediations
+
+- NetBird VPS hung (TCP up, SSH banner timeout, TLS hang):
+  → This is a host-level hang, not a service fault. Cannot self-remediate via SSH.
+  → Alert crit immediately: "NetBird VPS hung — needs hard reboot from provider console"
+  → Do NOT declare backend services down; their LAN-IP paths are still up.
+- Single proxy route missing (e.g. git returns 000 but chat ok):
+  → ssh root@72.61.0.17 'docker restart netbird-traefik'
+  → re-check after 30s
+- New NetBird CNAME detected:
+  → Alert: "<domain> became NetBird-dependent — violates IP-first doctrine (Section 7)"
+- Config URL violation detected:
+  → Alert with file + line; do not auto-edit configs
 ```

 ## Section 6: Alert Routing & Escalation
@@ -281,6 +473,63 @@ ZFS pool degraded
  → Cannot auto-fix → Escalate immediately
 ```

+## Section 7: Configuration Doctrine — IP-First, URLs for Browsers
+
+This doctrine is the structural fix for the NetBird-dependency outage. It is
+enforced by the `routing-regression.config_url_violations` check in Section
+5.3.
+
+### Rule
+
+1. **Service configs use LAN IP addresses, always.** Any file that wires a
+   service, agent, monitor, or integration to another internal service must
+   reference the backend by its `192.168.68.x` LAN IP and port — never by a
+   `*.sysloggh.net` URL. NetBird may blip at any time; LAN IPs do not.
+2. **`*.sysloggh.net` URLs are reserved for human browser access.** They are
+   a convenience layer (split-horizon DNS on the LAN, NetBird reverse proxy
+   off the LAN) for a person at a keyboard. They must never be a load-bearing
+   dependency in code or config.
+3. **Agents and monitors reach backends by LAN IP.** Abiba, Tanko, and Mumuni
+   all run on the LAN; there is no reason for them to hairpin through NetBird.
+   The daily infra report and all `ssh`/`curl` checks already follow this.
+4. **DNS records should prefer LAN IPs over NetBird CNAMEs.** A service that
+   can resolve directly to its LAN IP (like `litellm` → `192.168.68.116` and
+   `minipve` → `192.168.68.12`) is NetBird-independent. Moving a record from
+   a NetBird CNAME to a LAN IP is an improvement; the reverse is a regression.
+5. **NetBird is for remote/PC access only.** When you are out or on a PC,
+   NetBird carries you in. When you are on the LAN, NetBird is not in the
+   path. Configs must reflect this — they always assume LAN.
+
+### Canonical IP map (configs MUST use these)
+
+| Service | Use in configs | URL (browsers only) |
+|---------|----------------|--------------------|
+| Proxmox API | `https://192.168.68.12:8006` | `https://minipve.sysloggh.net` |
+| LiteLLM API | `http://192.168.68.116:4000` | `https://litellm.sysloggh.net` |
+| LiteLLM (nginx) | `http://192.168.68.116` | — |
+| Grafana | `http://192.168.68.116:3001` | — |
+| Authentik | `https://192.168.68.11` | `https://auth.sysloggh.net` |
+| Gitea | `http://192.168.68.110:3000` | `https://git.sysloggh.net` |
+| Zulip API | `http://192.168.68.19` | `https://chat.sysloggh.net` |
+| SearXNG | `http://192.168.68.7:8888` | — |
+| Firecrawl | `http://192.168.68.7:3002` | — |
+| RA-H OS bridge | `http://192.168.68.65:3100` | — |
+
+### Violation examples (to be flagged by Section 5.3)
+
+- ❌ An agent config setting `LITELLM_BASE_URL=https://litellm.sysloggh.net`
+- ✅ Same config setting `LITELLM_BASE_URL=http://192.168.68.116:4000`
+- ❌ A monitor curling `https://chat.sysloggh.net/api/v1/server_settings`
+- ✅ A monitor curling `http://192.168.68.19/api/v1/server_settings` (and
+      optionally the URL as a separate browser-path check)
+
+### Goal state
+
+Every `*.sysloggh.net` record resolves to a LAN IP (split-horizon DNS on LAN,
+NetBird reverse proxy off LAN). NetBird then becomes purely the off-LAN
+ingress — if it blips, only remote browser users notice, and no agent,
+monitor, or integration breaks.
+
 ## Appendix A: Quick Health Commands

 ```bash
@@ -293,8 +542,30 @@ curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"
 ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
 ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"

+# GPU fleet quick check
+curl -s http://192.168.68.116/health/unified | jq .status
+curl -s http://192.168.68.24:9100/gpu-data | jq .summary
+
+# Grafana status
+curl -s http://admin:syslog-grafana-2026@192.168.68.116:3001/api/health
+
+# Prometheus targets
+curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
+
+# LiteLLM key check
+curl -s -H "Authorization: Bearer sk-litellm-7f96080dd99b15c36bd4b333b58a6796" \
+  http://192.168.68.116/litellm/key/list | jq '.keys[] | {alias: .key_alias, models: .models}'
+
 # Storage check
 ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
+
+# Router roster reload (if needed)
+curl -s -X POST http://192.168.68.116:9000/admin/roster/reload \
+  -H "Authorization: Bearer sk-admin-ee09fffd04978b61a1569ac670c68814"
+
+# Restart stuck GPU (saturation watchdog alternative)
+ssh root@192.168.68.8 "systemctl restart llama-server"
+ssh root@192.168.68.110 "systemctl restart llama-server"
 ```

 ## Appendix B: CT Inventory
@@ -302,9 +573,9 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | CT | Name | Node | IP | Role | Agent |
 |----|------|------|----|------|-------|
 | 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
-| 101 | llm-gpu | acerpve | — | GPU VM | ❌ |
+| 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
 | 102 | adguard | acerpve | — | DNS | ❌ |
-| 103 | ocu-llm | ocupve | — | GPU VLM | ❌ |
+| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
 | 104 | authentik | minipve | .11 | OIDC | ❌ |
 | 105 | kagentz | amdpve | — | Agent Zero | ✅ |
 | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
@@ -314,10 +585,10 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | 110 | gitea | minipve | — | Git | ❌ |
 | 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy | amdpve | — | ? | ❌ |
-| 114 | mumuni | minipve | — | Hermes agent | ✅ |
+| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
+| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | — | ? | ❌ |
-| 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ |
+| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
 | 117 | zulip | storepve | — | Chat | ❌ |
 | 118 | jitsi | minipve | — | Video | ❌ |

@@ -329,5 +600,5 @@ ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
 | docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` |
 | docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` |
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
-| CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` |
+| CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |
@@ -0,0 +1,154 @@
+---
+kind: function
+name: infrastructure-monitoring
+description: >
+  Deploys Prometheus + GPU exporters + Grafana to monitor the entire
+  inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
+  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
+  via existing /metrics Prometheus endpoint.
+  
+  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+  This contract defines the target state; proxmox-monitor is the as-built.
+version: 1.0.0
+---
+
+## Architecture
+
+```
+GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
+  nvidia-exporter       nvidia-exporter       amdgpu-exporter
+  :9400                 :9400                 :9400
+       │                     │                     │
+       └─────────────────────┼─────────────────────┘
+                             ▼
+              ┌──────────────────────────┐
+              │       Prometheus         │
+              │       CT 116 :9090       │
+              │                          │
+              │  Scrape targets:         │
+              │  • 192.168.68.8:9400     │
+              │  • 192.168.68.110:9400   │
+              │  • 192.168.68.15:9400    │
+              │  • litellm:4000/metrics  │
+              └──────────┬───────────────┘
+                         │
+              ┌──────────▼───────────────┐
+              │        Grafana           │
+              │       CT 116 :3001       │
+              │                          │
+              │  Preloaded dashboards:   │
+              │  • GPU Fleet Overview    │
+              │  • LiteLLM Proxy Stats   │
+              └──────────────────────────┘
+```
+
+## Components
+
+### 1. NVIDIA GPU Exporter (hosts: .8, .110)
+- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
+
+### 2. AMD GPU Exporter (host: .15)
+- Custom exporter: Python script wrapping `amdgpu_top --json`
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
+- Runs as systemd service for persistence
+
+### 3. Prometheus (CT 116)
+- Container: `prom/prometheus:latest`
+- Port: `9090` (internal Docker network)
+- Scrape interval: 15s
+- Config: `/opt/monitoring/prometheus.yml`
+- Storage: Docker volume `prometheus-data`
+
+### 4. Grafana (CT 116)
+- Container: `grafana/grafana:latest`
+- Port: `3001` (mapped to host)
+- Data source: Prometheus at `http://prometheus:9090`
+- Provisioned dashboards for GPU fleet + LiteLLM
+- Accessible at `http://192.168.68.116:3001`
+
+## Parameters
+
+- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
+- gpu_amd_hosts: ["192.168.68.15"]
+- monitoring_host: "192.168.68.116"
+- prometheus_port: 9090
+- grafana_port: 3001
+- gpu_exporter_port: 9400
+
+## Requires
+
+- SSH access to all GPU hosts for exporter deployment
+- Docker on CT 116 for Prometheus + Grafana containers
+- Python 3 on AMD host for custom exporter
+- nvidia-smi on NVIDIA hosts
+
+## Maintains
+
+- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
+- Prometheus scrapes all targets every 15s
+- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
+- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
+- Stack persists across reboots (systemd for exporters, Docker restart policy)
+
+## Execution
+
+### Phase 1: GPU Exporters
+
+**NVIDIA (.8 and .110)**:
+1. Download `nvidia_gpu_exporter` binary
+2. Create systemd service `nvidia-gpu-exporter.service`
+3. Start and enable
+
+**AMD (.15)**:
+1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
+2. Parses `amdgpu_top --json -d 1000` output
+3. Exposes key metrics at `:9400/metrics` via Python http.server
+4. Create systemd service
+5. Start and enable
+
+### Phase 2: Prometheus
+
+1. Create `/opt/monitoring/` directory on CT 116
+2. Write `prometheus.yml` with scrape configs for all targets
+3. Add to docker-compose (or separate compose file)
+4. Start container
+
+### Phase 3: Grafana
+
+1. Create `/opt/monitoring/grafana/` directories
+2. Provision Prometheus datasource
+3. Provision GPU fleet dashboard JSON
+4. Provision LiteLLM dashboard JSON
+5. Add to docker-compose
+6. Start container
+
+### Phase 4: Verification
+
+1. Verify all 3 GPU exporters return 200 at :9400/metrics
+2. Verify Prometheus targets all UP at :9090/targets
+3. Verify Grafana accessible at :3001 with dashboards
+4. Verify LiteLLM metrics flowing to Prometheus
+5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
+
+## Verification Commands
+
+```bash
+# GPU exporters
+curl -s http://192.168.68.8:9400/metrics | grep nvidia
+curl -s http://192.168.68.110:9400/metrics | grep nvidia
+curl -s http://192.168.68.15:9400/metrics | grep amdgpu
+
+# Prometheus
+curl -s http://192.168.68.116:9090/api/v1/targets
+
+# Grafana
+curl -s http://192.168.68.116:3001/api/health
+
+# LiteLLM metrics (already live)
+curl -s http://192.168.68.116:4001/metrics | head -20
+```
@@ -0,0 +1,156 @@
+---
+kind: responsibility
+name: infrastructure-update
+description: >
+  Autonomous system-wide update contract covering all 5 Proxmox nodes,
+  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
+  Docker images, and container stacks in safe waves with health checks
+  and automatic rollback on failure.
+agent: abiba
+triggers:
+  - on "infra update" command
+  - weekly (Sunday 03:00 EDT) via cron
+  - on security advisory relay from Mumuni
+version: 1.0.0
+---
+
+## Maintains
+
+- update-status: { phase, node, action, result, timestamp }
+- update-history: array of past update runs with results
+- security-state: { cve_count, last_patched, pending_updates }
+
+## Pre-Flight Checklist
+
+Before ANY update wave:
+1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
+2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
+3. ✅ Docker healthy on .7, .116
+4. ✅ LiteLLM health check passing
+5. ✅ Zulip server reachable
+6. ✅ GPU fleet healthy (all 3 GPUs)
+7. ✅ Disk >20% free on all nodes
+8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
+
+## Wave 1: Storage & Infra Nodes (lowest impact)
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
+| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
+| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 1:**
+- Docker healthy on .7: `docker ps`
+- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
+- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`
+
+## Wave 2: Compute & Agent Nodes
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
+| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
+| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
+| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
+| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 2:**
+- All CTs running: check via Proxmox API
+- LiteLLM healthy: `curl :4001/health/liveliness`
+- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- Zulip agents connected: check Mumuni/Tanko gateway state
+- Abiba PM2 processes online: `pm2 status`
+
+## Wave 3: Docker Image Updates
+
+| Host | Stack | Command |
+|------|-------|---------|
+| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
+
+**Verify after Wave 3:**
+- All containers healthy: `docker ps` on each host
+- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
+- Zulip test: send test message to #agent-hub
+- Dashboard loading: `curl :3001/`
+- Firecrawl test: `curl :3002/health`
+- SearXNG test: `curl storepve:8888`
+
+## Wave 4: Proxmox Kernel Reboot (if needed)
+
+Only if `[ -f /var/run/reboot-required ]` on any node.
+
+| Target | Action |
+|--------|--------|
+| Affected PVE node | Verify all CTs/VMs migrated or stopped |
+| | `reboot` via PVE API |
+| | Wait 120s for node to come back |
+| | Start any stopped CTs |
+
+## Rollback Protocol
+
+If ANY verification fails:
+
+1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
+2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
+3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
+4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails
+
+## Config Backup
+
+Before Wave 1, snapshot these files:
+```
+/opt/inference-harness/litellm_config.yaml        (.116)
+/opt/inference-harness/docker-compose.yml          (.116)
+/etc/nginx/nginx.conf                              (.116 container)
+/opt/search-stack/*/docker-compose.yaml            (.7)
+/root/.pi/agent/extensions/config.yaml             (.24)
+/etc/systemd/system/ornith-server.service          (.15)
+/etc/systemd/system/llama-server.service           (.8, .110)
+```
+
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
+
+## Security-Specific Updates
+
+| Check | Command | Action |
+|-------|---------|--------|
+| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
+| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
+| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
+| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |
+
+## Success Criteria
+
+- [ ] All 5 PVE nodes updated, no reboot-loop
+- [ ] All CTs running post-update
+- [ ] All 22 Docker containers healthy
+- [ ] LiteLLM inference passing (syslog-auto test)
+- [ ] Zulip server + all 3 agents connected
+- [ ] GPU fleet at full capacity (3/3)
+- [ ] Zero security CVEs remaining
+- [ ] <10 min total downtime per service
+
+## Report
+
+After completion, send Zulip DM:
+```
+📋 Infrastructure Update — YYYY-MM-DD
+
+Updated: 5 PVE nodes, 12 CTs, 22 containers
+Security fixes: N CVEs patched
+Downtime: <service> <duration>
+Failures: none / <details>
+Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
+```
@@ -0,0 +1,52 @@
+---
+kind: function
+name: litellm-api-keys
+description: >
+  Manages LiteLLM API keys for agent identity. Creates named keys so each agent
+  is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry)
+  and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026").
+  Ensures agents never use the master key directly. Rotation is event-driven,
+  not calendar-driven — rotate only on compromise, personnel change, or
+  periodic security hygiene (quarterly/annually).
+---
+
+## Parameters
+
+- agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
+- action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
+- litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
+- master_key: string — LiteLLM master key (default from environment)
+- agent_host: string — Agent's IP for SSH (default: resolved from infra)
+- agent_user: string — SSH user (default: "jerome")
+
+## Returns
+
+- action: string — What was done
+- key_alias: string — The LiteLLM key alias created/rotated
+- key_prefix: string — First 10 chars of the new key (for identification)
+- previous_key_alias: string | null — Previous key alias if rotating
+- litellm_response: object — Raw response from LiteLLM /key/generate
+- agent_config_updated: boolean — Whether /etc/environment was updated
+- verification: { status: string, detail: string } — Final health check
+
+## Execution
+
+1. **Authenticate** — Verify master_key works against LiteLLM /key/list
+2. **Check existing keys** — List all keys, find any with agent_name alias
+3. **If action == "list"**: Return all keys with their aliases and spend
+4. **If action == "create"**:
+   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
+   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
+   - Duration is null (permanent) — inherited from litellm default_key_generate_params
+   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+   - Return the new key
+5. **If action == "rotate"**:
+   - Generate new key with same alias (LiteLLM replaces the old key)
+   - SSH to agent_host, update /etc/environment LITELLM_API_KEY
+   - Restart agent gateway (hermes gateway restart for Hermes agents)
+   - Verify: curl test against /v1/models with new key
+   - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
+6. **If action == "verify"**:
+   - SSH to agent, read /etc/environment
+   - Test the key against LiteLLM /v1/models
+   - Confirm key alias matches agent_name in LiteLLM key list
@@ -1,17 +1,44 @@
 ---
 kind: function
-name: check-litellm-health
+name: litellm-health
 description: >
-  Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
-  OIDC auth endpoint, container status, and aggregate health on the
-  backend host. Designed as a reusable contract for any Syslog agent.
+  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
+  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
+  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
+  Prometheus/Grafana and fleet dashboard.
+  Designed as a reusable contract for any Syslog agent.
 ---

+## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
+                        │                    │
+                  Key validation      Model routing
+                  Fallback chains     Slot booking
+                  Complexity router   Circuit breakers
+                  Budget tracking     Redis-backed
+                        │                    │
+                  Prometheus ← metrics  ←───┘
+                        │
+                    Grafana :3001
+```
+
+**What changed (v3.1.0 → v3.2.0)**:
+- Router is BACK — runs internally on :9000 behind LiteLLM
+- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
+- Router :9000 is 127.0.0.1-only, not publicly accessible
+- GPU fleet dashboard: http://192.168.68.24:9100
+- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+
 ## Parameters

 - public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
+- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
 - auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

 ## Returns

@@ -23,34 +50,75 @@ description: >
 ## Requires

 - SSH key access to backend_host for container checks
- Network access to public_url and auth_host
- curl and openssl available on the execution host
+- Network access to public_url, auth_host, and gpu_dashboard_url
+- LiteLLM master key for key management endpoints

-## Ensures
+## GPU Fleet Topology

- Each check returns a clear pass/fail status with detail message
- If any endpoint returns non-200, overall_status is "degraded"
- If backend host unreachable or >2 containers down, overall_status is "down"
- If /health/unified reports any non-healthy components, status reflects it
- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
+| Host | IP | Hardware | Models Served | Engine |
+|------|-----|----------|---------------|--------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
+| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
+
+## Model Fallback Chains (LiteLLM)
+
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
+| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | — | qwen → gemma | — |
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |

 ## Execution

 1. **Read parameters** — Use provided values or defaults
+
 2. **Check public endpoints**:
   - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
   - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-   - GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
-   - GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
-3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
-   - GET http://{{backend_host}}/litellm/ui/ → expect 200
-   - GET http://{{backend_host}}/litellm/docs → expect 200
-4. **Check aggregate health endpoint**:
-   - GET {{backend_host}}:9000/health/unified → expect 200, check all components
-5. **Check backend container health**:
-   - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
-   - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
-6. **Check OIDC auth endpoint**:
-   - Verify auth.sysloggh.net resolves to {{auth_host}}
-   - GET https://auth.sysloggh.net/ → expect login page
-7. **Compile and report** — Determine overall_status from individual check results
+
+3. **Check LiteLLM health (no-auth)**:
+   - GET http://{{backend_host}}/litellm/health/liveliness → expect 200
+
+4. **Check backend container health**:
+   - SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
+   - Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
+   - Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
+
+5. **Check router roster loaded**:
+   - GET http://{{backend_host}}:9000/health → expect 200
+   - GET http://{{backend_host}}:9000/health/unified → expect 3 models
+   - If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
+
+6. **Check GPU fleet health** (via fleet dashboard):
+   - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
+   - Verify GPUs reporting status "healthy"
+   - Check alerts array for active warnings/critical
+
+7. **Check model inference via LiteLLM** — Test each model:
+   - POST /v1/chat/completions model=gemma-4-12b → expect 200
+   - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+   - POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+   - Use master key for auth
+
+8. **Check agent keys**:
+   - GET /key/list with master key → verify all 6 agents have keys
+
+9. **Check Grafana**:
+   - GET {{grafana_url}}/api/health → expect 200
+
+10. **Compile and report** — Determine overall_status from individual check results
@@ -1,6 +1,11 @@
 ---
 kind: responsibility
 name: litellm-self-heal
+status: manual-only
+note: >
+  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
+  This contract is now manual-only — triggers require explicit user request.
+  Consider reimplementing as a standalone cron job or prose contract.
 description: >
  Standing responsibility that monitors LiteLLM health and proactively
  fixes common issues. Reports every action via Zulip DM and RA-H OS
@@ -66,6 +71,14 @@ logged as a `[REPORT]` knowledge graph node for long-term trending.
 If a fix required another agent's help (e.g., Authentik restart), a relay
 message is sent to the responsible agent with full context.

+## GPU Fleet
+
+| Host | IP | Hardware | Models | Engine |
+|------|-----|----------|--------|--------|
+| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
+| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
+| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
+
 ## Remediation Rules

 ### Rule 1: Container Not Healthy
@@ -82,6 +95,34 @@ Detect → immediate escalate (external dependency)
 ### Rule 4: Docs 404
 Detect → fix DOCS_URL env var → verify → log

+### Rule 5: GPU Unreachable
+Detect → SSH to GPU host fails or model not responding
+Fix → restart llama-server on host via SSH
+Escalate → immediately (downtime affects inference)
+
+### Rule 6: Model Not Responding
+Detect → test prompt via LiteLLM returns non-200
+Fix → restart llama-server on GPU host → verify
+Escalate → after 2 failed restarts
+
+### Rule 7: Router Returns 503 (All GPUs Saturated)
+Detect → router logs show QUEUE_TIMEOUT with GPUs idle
+Root cause → roster not loaded at startup (empty TIER_MODELS)
+Fix → POST /admin/roster/reload (if endpoint available) or restart router
+Permanent fix → add load_roster() to router startup code
+Escalate → if reload doesn't resolve within 2 min
+
+### Rule 8: Agent Keys Invalid (401)
+Detect → agents report auth errors
+Root cause → keys in router env var, not in LiteLLM DB
+Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
+Escalate → if SSH access unavailable, send Zulip DM
+
+### Rule 9: Stale Active Counter in Redis
+Detect → active:<model> > 0 while GPU idle
+Fix → `redis-cli DEL active:<model>` → restart router
+Prevent → add TTL to active keys in router code
+
 ## Execution

 1. Run health check
@@ -1,20 +1,38 @@
 ---
 name: memory-audit-maintenance
 kind: responsibility
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---

 ### Goal

-Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md, SOUL.md), categorize entries, redistribute them to correct locations, verify live configurations, and free character budget  all without user intervention unless a truly unresolvable ambiguity exists.
+Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md, SOUL.md), categorize entries, redistribute them to correct locations, verify live configurations, and free character budget  all without user intervention unless a truly unresolvable ambiguity exists. Every agent runs this contract for **its own** memory files only  no cross-agent access, no shared state.

 ### Requires

 (none  this is a self-driven, autonomous node)

+### Scope
+
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+
+**Agent Roster:**
+- Mumuni
+- Tanko
+- Koby
+- Koonimo
+
+**Isolation Principle:** Each agent has its own:
+- `MEMORY.md` and `USER.md`
+- `MEMORY_AUDIT_LEDGER.md` (audit trail)
+- `MEMORY_WRITERS.md` (writer registry)
+- Canary entries
+- No data crosses agent boundaries
+
 ### Maintains

-Memory health state  current utilization, last audit timestamp, and any unresolved ambiguities. Postcondition: no entry is duplicated, no session note or stale snapshot survives, and no config is unverified.
+Memory health state  current utilization, last audit timestamp, drift alerts, and any unresolved ambiguities. Postcondition: no entry is duplicated, no session note or stale snapshot survives, no config is unverified, and the audit ledger is up-to-date.

 #### utilization
 Current character utilization of MEMORY.md and USER.md. Material: the percentages.
@@ -25,6 +43,12 @@ Timestamp of the most recent successful audit. Material: the timestamp.
 #### unresolved
 Any entries that could not be autonomously categorized. Material: the entry text and the reason for ambiguity.

+#### drift_alerts
+Active alerts from drift detection (e.g., deletions accelerating, unverified configs growing). Material: the alert list.
+
+#### ledger
+Persistent audit trail stored at the agent's own `~/.hermes/memories/MEMORY_AUDIT_LEDGER.md`. Material: append-only, never modified or deleted. Each agent has its own ledger  no shared ledger across agents.
+
 ### Continuity

 - self-driven: re-audit every 14 days (2 weeks)
@@ -41,11 +65,12 @@ Any entries that could not be autonomously categorized. Material: the entry text

 ### Strategies

- **Autonomous first:** Never ask the user about entries that clearly belong in a deletion or move category. Only escalate on true ambiguity (e.g., an entry could be either a Preference or a Technical Config, and the choice affects which file it goes in).
+- **Autonomous first:** Never ask the user about entries that clearly belong in a deletion or move category. Only escalate on true ambiguity.
 - **Incremental over nuclear:** Always use patch-level edits (find-and-replace) rather than wholesale file rewrites. If the patch tool fails, fall back to rewrite only as a last resort.
 - **Verify before trusting:** Every technical config in MEMORY.md must be verified against the live system. If a service is unreachable, mark it as `UNVERIFIED` rather than deleting it.
 - **Stale state has a shelf life:** Any entry marked as "Temporary State" (e.g., "verbal offer", "deal pending", "negotiation phase") is deleted if older than 14 days.
 - **Work around tool failures:** If the `memory` tool fails after 2 retries, add a summary marker and report the stale state in the unresolved list.
+- **Privacy isolation:** Each agent only reads and writes its own memory files. No cross-agent memory access, no shared ledger, no shared state. The contract is the standard; each agent runs it independently.

 ### Invariants

@@ -53,14 +78,44 @@ Any entries that could not be autonomously categorized. Material: the entry text
 - No duplicate entries survive across MEMORY.md and USER.md
 - No session note or temporary state survives past 14 days
 - No config is trusted without verification (or marked UNVERIFIED)
+- The audit ledger is append-only  never modified or deleted, and is isolated per agent
+- The canary entry survives every audit unmodified
+- No data crosses agent boundaries

 ### Execution

 ```prose
 # ============================================================
-# PHASE 1: Pre-flight
+# PHASE 0: Canaries & Privacy
 # ============================================================

+# Check canaries  if either is missing or modified, raise CRITICAL alert
+memory_canary = read_file_line("~/.hermes/memories/MEMORY.md", containing="CANARY: Do not modify this line")
+user_canary = read_file_line("~/.hermes/memories/USER.md", containing="CANARY: Do not modify this line")
+
+if memory_canary is null or user_canary is null:
+  return {
+    status: "CRITICAL",
+    alert: "CANARY BREACH  memory integrity compromised",
+    details: {
+      memory_canary_intact: memory_canary is not null,
+      user_canary_intact: user_canary is not null
+    }
+  }
+
+# This agent only touches its own files  no cross-agent access
+# Confirm we are operating on this agent's own memory directory
+if not directory_exists("~/.hermes/memories"):
+  return {status: "ERROR", reason: "Memory directory not found for this agent"}
+
+# ============================================================
+# PHASE 1: Pre-flight & Ledger
+# ============================================================
+
+# Read the audit ledger for drift detection
+ledger = read_file("~/.hermes/memories/MEMORY_AUDIT_LEDGER.md")
+last_audit_entry = parse_last_entry(ledger)
+
 # Verify the memory tool works
 try:
  memory_status = call check_memory_tool
@@ -76,9 +131,36 @@ memory_util = len(memory_content) / max_memory_chars * 100
 user_util = len(user_content) / max_user_chars * 100

 # Check if audit is needed
-if last_audit > 14 days and memory_util < 75 and user_util < 75:
+if last_audit_entry.timestamp > 14 days and memory_util < 75 and user_util < 75:
  return {status: "SKIP", reason: "No audit needed"}

+# ============================================================
+# PHASE 1b: Drift Detection (Compare to Last Audit)
+# ============================================================
+
+drift_alerts = []
+
+if last_audit_entry exists:
+  # Deletions accelerating? (50% increase from last audit)
+  if last_audit_entry.deletions > 0:
+    deletion_increase = (memory_util - last_audit_entry.memory_util_after) / last_audit_entry.deletions * 100
+    if deletion_increase > 50:
+      drift_alerts.append("ALERT: Memory bloat accelerating  deletions increasing by " + deletion_increase + "% vs last audit")
+
+  # Unverified configs growing?
+  if last_audit_entry.unverified > 0:
+    unverified_increase = count_entries(memory_content, "[UNVERIFIED]") - last_audit_entry.unverified
+    if unverified_increase > 2:
+      drift_alerts.append("ALERT: Unverified configs growing  " + unverified_increase + " new unverified entries since last audit")
+
+  # Audit missed? (No audit for >21 days)
+  if last_audit_entry.timestamp > 21 days:
+    drift_alerts.append("ALERT: Audit gap  no audit completed in 21+ days")
+
+  # Utilization not improving? (After audit, still >70%)
+  if last_audit_entry.memory_util_after > 70:
+    drift_alerts.append("ALERT: Previous audit failed to reduce MEMORY.md below 70%  capacity issue")
+
 # ============================================================
 # PHASE 2: Categorize every entry
 # ============================================================
@@ -140,7 +222,7 @@ for item in categorized:
 # ============================================================

 if dry_run:
-  return {status: "DRY_RUN", actions: actions, summary: generate_summary(actions)}
+  return {status: "DRY_RUN", actions: actions, drift_alerts: drift_alerts, summary: generate_summary(actions)}

 # Apply deletions
 for action in actions:
@@ -180,12 +262,39 @@ new_user_content = read_file("~/.hermes/memories/USER.md")
 new_memory_util = len(new_memory_content) / max_memory_chars * 100
 new_user_util = len(new_user_content) / max_user_chars * 100

+# Verify canaries survived the audit
+post_canary_check = read_file_line("~/.hermes/memories/MEMORY.md", containing="CANARY: Do not modify this line")
+if post_canary_check is null:
+  drift_alerts.append("CRITICAL: Canary destroyed during audit  memory integrity compromised")
+
 # Add summary markers to in-memory stores
 memory action=add target=memory content="MEMORY.md rewritten. Utilization: " + new_memory_util + "%."
 memory action=add target=user content="USER.md rewritten. Utilization: " + new_user_util + "%."

 # ============================================================
-# PHASE 7: Report
+# PHASE 7: Append to Ledger
+# ============================================================
+
+ledger_entry = """
+## """ + now() + """
+- memory_util_before: """ + memory_util + """%
+- memory_util_after: """ + new_memory_util + """%
+- user_util_before: """ + user_util + """%
+- user_util_after: """ + new_user_util + """%
+- deletions: """ + count(actions, "DELETE") + """
+- moves: """ + count(actions, "MOVE") + """
+- corrections: """ + count(actions, "CORRECT") + """
+- unverified: """ + count(actions, "MARK_UNVERIFIED") + """
+- unresolved: """ + len(unresolved_entries) + """
+- drift_alerts: """ + len(drift_alerts) + """
+- canary_intact: """ + (post_canary_check is not null) + """
+- status: COMPLETE
+"""
+
+append_file path="~/.hermes/memories/MEMORY_AUDIT_LEDGER.md" content=ledger_entry
+
+# ============================================================
+# PHASE 8: Report
 # ============================================================

 return {
@@ -198,14 +307,16 @@ return {
  corrections: count(actions, "CORRECT"),
  unverified: count(actions, "MARK_UNVERIFIED"),
  unresolved: unresolved_entries,
+  drift_alerts: drift_alerts,
  last_audit: now(),
+  ledger_updated: true,
  summary: generate_summary(actions)
 }
 ```

 ### Shape

- `self`: categorize, verify configs, apply incremental patches, generate reports
+- `self`: categorize, verify configs, apply incremental patches, generate reports, drift detection, ledger management, canary checks
 - `delegates`:
  - `categorize_entry`: determine the category of a memory entry
  - `check_memory_tool`: verify the memory tool is functional
@@ -227,4 +338,9 @@ return {

 - `MEMORY_PATH`: ~/.hermes/memories/MEMORY.md
 - `USER_PATH`: ~/.hermes/memories/USER.md
- `CONFIG_PATH`: ~/.hermes/config.yaml
+- `CONFIG_PATH`: ~/.hermes/config.yaml
+- `LEDGER_PATH`: ~/.hermes/memories/MEMORY_AUDIT_LEDGER.md
+
+### Per-Agent Notes
+
+Each Hermes agent (Mumuni, Tanko, Koby, Koonimo) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
@@ -0,0 +1,107 @@
+---
+kind: architecture
+name: pi-approval-architecture
+description: >
+  Documents pi's approval model vs Hermes, what commands are available,
+  and the architectural constraints that prevent mid-turn tool approval in pi.
+  Updated with /approve and /deny stub commands for cross-agent UX consistency.
+agent: abiba
+version: 1.0.0
+---
+
+# Pi Approval Architecture
+
+## TL;DR
+
+| Feature | Hermes | Pi (Abiba) |
+|---------|--------|------------|
+| `/approve` command | ✅ Full — unblocks agent | ✅ Stub — acknowledges, no blocking |
+| `/deny` command | ✅ Full — blocks agent | ✅ Stub — acknowledges, no blocking |
+| Dangerous command detection | ✅ 40+ patterns + Tirith | ❌ None (LLM judgment only) |
+| Mid-turn tool approval | ✅ Agent thread blocks on Event | ❌ No hook in agent loop |
+| Smart approval (aux LLM) | ✅ `approvals.mode: smart` | ❌ N/A |
+| Yolo mode | ✅ `/yolo`, `approvals.mode: off` | ❌ N/A |
+| Permanent allowlist | ✅ `command_allowlist` + `shell-hooks` | ❌ N/A |
+| Session-scoped approval | ✅ `approve_session()` | ❌ N/A |
+| Approval timeout | ✅ 60s default, configurable | ❌ N/A |
+
+## Why Pi Can't Do Mid-Turn Approval
+
+Pi's agent loop is **418 lines** of TypeScript. It's deliberately minimal:
+
+```
+LLM decides → calls tool → tool executes immediately → output goes to LLM → repeats
+```
+
+There is no middleware between "LLM decides to call bash" and "bash executes."
+The extension system (`injectIntoSession`) feeds messages IN but doesn't intercept
+tool calls mid-turn.
+
+Hermes has a full approval pipeline:
+```
+LLM decides → check_all_command_guards() → if dangerous → notify_cb → block thread
+→ user responds /approve → resolve_gateway_approval() → Event.set() → unblock
+```
+
+This pipeline is deeply integrated into Hermes' gateway, tool executor, and
+session management. Replicating it in pi would require rewriting the core agent
+loop — violating pi's design constraint of minimal core.
+
+## What Pi HAS: `/approve` and `/deny` Stubs
+
+Added to prevent `/approve` and `/deny` from being forwarded to the LLM as
+garbled input. When a user types these commands to Abiba:
+
+- **`/approve`** → "Pi doesn't have pending approvals. Commands execute immediately."
+- **`/approve session`** → Same response
+- **`/deny`** → Same response + "For Hermes agents (Tanko, Mumuni), these work with their built-in approval system."
+
+This keeps the UX consistent across agents — users can type `/approve` anywhere
+without getting confused by LLM responses.
+
+## Pi's Full Command Set (Hermes Parity)
+
+| Command | Handler | Description | Hermes Equivalent |
+|---------|---------|-------------|-------------------|
+| `/status` | System | PM2 processes, containers, session info | `/status` |
+| `/retry` | Session | Re-injects last user message | `/retry` |
+| `/continue` | Session | Continues truncated response | `/continue` |
+| `/new [topic]` | Session | Clears conversation context | `/new` |
+| `/stop` | Control | Interrupts current LLM processing | `/stop` |
+| `/model [name]` | Info | Show current model, list available | `/model` |
+| `/compact` | Session | Requests context compaction | `/compact` |
+| `/yolo` | Stub | Pi has no approval system — accepted for UX | `/yolo` |
+| `/verbose` | Stub | Verbosity controlled by prompt, not toggle | `/verbose` |
+| `/approve` | Stub | Pi executes immediately — no approvals | `/approve` |
+| `/deny` | Stub | Pi executes immediately — no approvals | `/deny` |
+| `/help [cmd]` | Info | Shows command help | `/help` |
+
+## What Pi COULD Add (Without Core Changes)
+
+These would work within pi's current architecture:
+
+1. **Pre-message dangerous command scan**: Before forwarding an agent response to
+   Zulip, scan the text for dangerous bash commands and add a warning banner.
+   Doesn't prevent execution but alerts user.
+
+2. **User-facing approval prompt injection**: When user asks pi to do something
+   dangerous (detected via regex on user input), pi could respond with a
+   confirmation request before proceeding. This works because it's at the
+   message level, not mid-turn.
+
+3. **Response rate-limiting**: If pi generates many bash commands rapidly,
+   throttle and ask for confirmation. Works within the extension's send path.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `/root/.pi/agent/extensions/zulip/index.js` | Pi Zulip extension with command system |
+| `/root/prose-contracts/zulip-approval-fix.prose.md` | Hermes Zulip approval fix |
+| `/root/prose-contracts/pi-approval-architecture.prose.md` | This document |
+
+## Related
+
+- Hermes `tools/approval.py` — Full approval pipeline (dangerous patterns, Tirith, smart mode)
+- Hermes `gateway/slash_commands.py:_handle_approve_command` — Slash command handler
+- Hermes Zulip adapter fix — HTML stripping for command detection
@@ -10,10 +10,13 @@ description: >

 ## Maintains

- abiba-zulip: { status: "online", uptime: string, restarts: number }
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
+- gpu-watchdog: { status: "online", uptime: string, restarts: number }
+- gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp

+> **Note (2026-07-04):** `abiba-zulip` removed — Zulip extension decommissioned.
+
 ## Continuity

 - Self-driven: check every 300 seconds (5 min)
@@ -28,9 +31,14 @@ description: >
 - **Escalate**: If still failed after 2 retries, send Zulip DM to owner

 ### Rule 2: Process Restarting Too Often
- **Detect**: `pm2 status` shows restarts > 5 in the last hour
+- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
+- **Note**: PM2 counter never decrements; only full delete+re-add resets it
 - **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
- **Escalate**: Always — alert owner with restart count
+- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
+- **Historical fix**: Previous cycles were caused by abiba-zulip extension's
+  stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
+  This was the root cause of the 18-restart accumulation. Threshold raised
+  and PM2 counter reset on 2026-06-28.

 ## Execution

@@ -1,64 +0,0 @@
-#!/bin/bash
-# pm2-self-heal — hourly PM2 process check
-# Part of the pm2-self-heal prose contract
-# Only sends Zulip DM when action is taken or escalation needed
-# Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
-
-ZULIP_BOT="abiba-bot@chat.sysloggh.net"
-ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
-ZULIP_SITE="https://chat.sysloggh.net"
-OWNER_ID=9
-LOG="/tmp/pm2-self-heal.log"
-
-send_alert() {
-    local subject="$1"
-    local body="$2"
-    curl -s -X POST "$ZULIP_SITE/api/v1/messages" \
-        -u "$ZULIP_BOT:$ZULIP_KEY" \
-        -d "type=private" \
-        -d "to=[$OWNER_ID]" \
-        -d "content=$subject\n$body" > /dev/null 2>&1
-    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Alert sent" >> "$LOG"
-}
-
-# Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
-STATUS=$(pm2 status --no-color 2>/dev/null)
-
-ALERTS=""
-
-# Check abiba-telegram (safe to auto-restart)
-TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
-TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
-TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
-
-if [ "$TEL_STATUS" != "online" ]; then
-    pm2 restart abiba-telegram > /dev/null 2>&1
-    sleep 3
-    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
-    TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs)
-    if [ "$TEL_STATUS2" = "online" ]; then
-        ALERTS="${ALERTS}⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online\n"
-    else
-        ALERTS="${ALERTS}🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)\n"
-    fi
-fi
-
-# Check abiba-zulip (read-only — never restart)
-ZUL_LINE=$(echo "$STATUS" | grep "abiba-zulip")
-ZUL_STATUS=$(echo "$ZUL_LINE" | awk -F'│' '{print $10}' | xargs)
-ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs)
-
-if [ "$ZUL_STATUS" != "online" ]; then
-    ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n"
-    ALERTS="${ALERTS}   NOT auto-restarting (runs this contract)\n"
-elif [ "$ZUL_RESTARTS" -gt 5 ] 2>/dev/null; then
-    ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n"
-fi
-
-# Log check
-echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS zul=$ZUL_STATUS alerts=${ALERTS:+yes}" >> "$LOG"
-
-# Only DM when there's something to report
-if [ -n "$ALERTS" ]; then
-    send_alert "**PM2 Self-Heal — $(date '+%H:%M UTC')**" "$ALERTS"
-fi
@@ -0,0 +1,115 @@
+---
+kind: responsibility
+name: proxmox-monitor
+description: >
+  Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
+  on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
+  exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
+  instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
+  (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
+  the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
+agent: abiba
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  CT 116 (syslog-api) — monitoring compose /opt/monitoring/   │
+│  Prometheus :9090  →  Grafana :3001 (direct LAN, 0.0.0.0)    │
+└───────┬──────────────┬──────────────┬───────────────────────┘
+        │              │              │
+        ▼              ▼              ▼
+   pve-exporter    docker-stats    (scrapes 5x node_exporter)
+   :9221           :9324
+        │              │
+        ▼              ▼
+   PVE API         Docker API
+   (amdpve .15,    (unix socket,
+    cluster-wide)  10 containers)
+```
+
+## Exporters
+
+| Exporter | Host:Port | Scope | Notes |
+|----------|-----------|-------|-------|
+| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
+| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
+| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
+
+## PVE API Token
+
+- User: `monitoring@pve` (cluster-replicated)
+- Role: `PVEAuditor` on `/` (read-only, whole cluster)
+- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
+- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
+
+## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
+
+| UID | Title | Panels | Source |
+|-----|-------|--------|--------|
+| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
+| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
+| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
+| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
+
+- Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/`
+- Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml`
+- Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`)
+- Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks
+
+## Access
+
+- **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
+- **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
+- **Credentials**: admin / syslog-grafana-2026
+- Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
+- grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
+| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
+| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
+| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
+| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
+| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
+| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
+| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
+
+## Cluster "Tabiri" — 5 Nodes
+
+| Node | IP | Role |
+|------|----|----|
+| ocupve | 192.168.68.5 | PVE |
+| storepve | 192.168.68.6 | PVE |
+| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
+| minipve | 192.168.68.12 | PVE |
+| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
+
+## Operations
+
+### view-dashboards
+Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder
+
+### add-dashboard
+Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
+
+### check-targets
+`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
+
+### restart-exporter
+`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
+
+### rotate-pve-token
+`pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter`
+
+## Known Issues & Notes
+
+- **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter.
+- **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health.
+- **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
+- **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
+- **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.
@@ -0,0 +1 @@
+{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}
@@ -0,0 +1,26 @@
+# run:20260704-005804-51f9a9 disk-gc-threat-response
+root: disk-gc-threat-response
+
+1→ [scan] fleet-wide disk scan started
+2→ CT 100 abiba (amdpve): 12G/59G 21% GREEN
+2→ CT 101 llm-gpu (acerpve): 144G/255G 60% GREEN
+2→ CT 102 adguard (acerpve): 5.9G/32G 20% GREEN
+2→ CT 103 ocu-llm (ocupve): 48G/186G 27% GREEN
+2→ CT 104 authentik (minipve): 4.6G/9.8G 50% GREEN
+2→ CT 105 kagentz (amdpve): 16G/59G 27% GREEN (resolved: was 87%, pruned 35.67G)
+2→ CT 106 ra-h-os (storepve): 2.1G/20G 11% GREEN
+2→ CT 107 pbs (storepve): 1.1G/40G 3% GREEN
+2→ CT 108 media (storepve): 19G/196G 10% GREEN
+2→ CT 109 docker-vm (storepve): 25G/158G 17% GREEN
+2→ CT 110 gitea (minipve): 5.6G/40G 16% GREEN
+2→ CT 111 tdunna (amdpve): 16G/63G 26% GREEN
+2→ CT 112 tanko (amdpve): 17G/49G 36% GREEN
+2→ CT 113 baggy (amdpve): 7.0G/52G 15% GREEN
+2→ CT 114 mumuni (minipve): 29G/59G 51% GREEN
+2→ CT 115 scottdenya (amdpve): 14G/59G 25% GREEN
+2→ CT 116 syslog-api (minipve): 16G/40G 42% GREEN
+2→ CT 117 zulip (storepve): 6.1G/60G 11% GREEN
+2→ CT 118 jitsi (minipve): OFFLINE — no disk scan
+3→ [summary] Fleet: 19 hosts, 18 scanned, 18 GREEN, 0 AMBER, 0 RED, 0 CRITICAL, 1 OFFLINE
+3→ [summary] Previously resolved: CT 105 kagentz 87%→27% (35.67GB reclaimed)
+---end 2026-07-04T00:58:04Z
@@ -0,0 +1,52 @@
+# Fleet Disk Health — 2026-07-04 00:58 UTC
+
+## Summary
+
+| Metric | Count |
+|--------|-------|
+| Total hosts | 19 |
+| Scanned | 18 |
+| GREEN (<75%) | 18 |
+| AMBER (75-84%) | 0 |
+| RED (85-94%) | 0 |
+| CRITICAL (≥95%) | 0 |
+| OFFLINE | 1 (jitsi CT 118) |
+
+## Per-Host Status
+
+| CT | Name | Node | Used | Total | % | Status |
+|----|------|------|------|-------|---|--------|
+| 100 | abiba | amdpve | 12G | 59G | 21% | 🟢 GREEN |
+| 101 | llm-gpu | acerpve | 144G | 255G | 60% | 🟢 GREEN |
+| 102 | adguard | acerpve | 5.9G | 32G | 20% | 🟢 GREEN |
+| 103 | ocu-llm | ocupve | 48G | 186G | 27% | 🟢 GREEN |
+| 104 | authentik | minipve | 4.6G | 9.8G | 50% | 🟢 GREEN |
+| 105 | kagentz | amdpve | 16G | 59G | 27% | 🟢 GREEN |
+| 106 | ra-h-os | storepve | 2.1G | 20G | 11% | 🟢 GREEN |
+| 107 | pbs | storepve | 1.1G | 40G | 3% | 🟢 GREEN |
+| 108 | media | storepve | 19G | 196G | 10% | 🟢 GREEN |
+| 109 | docker-vm | storepve | 25G | 158G | 17% | 🟢 GREEN |
+| 110 | gitea | minipve | 5.6G | 40G | 16% | 🟢 GREEN |
+| 111 | tdunna | amdpve | 16G | 63G | 26% | 🟢 GREEN |
+| 112 | tanko | amdpve | 17G | 49G | 36% | 🟢 GREEN |
+| 113 | baggy | amdpve | 7.0G | 52G | 15% | 🟢 GREEN |
+| 114 | mumuni | minipve | 29G | 59G | 51% | 🟢 GREEN |
+| 115 | scottdenya | amdpve | 14G | 59G | 25% | 🟢 GREEN |
+| 116 | syslog-api | minipve | 16G | 40G | 42% | 🟢 GREEN |
+| 117 | zulip | storepve | 6.1G | 60G | 11% | 🟢 GREEN |
+| 118 | jitsi | minipve | — | — | — | ⚫ OFFLINE |
+
+## Docker Hosts - GC Status
+
+| Host | Docker present | Last GC | Status |
+|------|---------------|---------|--------|
+| kagentz (CT 105) | Yes | 2026-07-04 (just now) | 35.67GB reclaimed, 27% |
+| syslog-api (CT 116) | Yes | Never | 42%, no action needed |
+| docker-vm (CT 109) | Yes | Unknown | 17%, no action needed |
+| abiba (CT 100) | Yes | Never | 21%, /var/lib/docker=2.8G |
+
+## Notes
+- CT 118 (jitsi) is not running — no disk scan possible. May need `qm start 118` if service is needed.
+- CT 114 (mumuni) at 51% — closest to AMBER threshold (75%). Agent memory accumulation likely.
+- CT 105 (kagentz) resolved from 87% RED → 27% GREEN. Root cause: docker image bloat. Prevented: GC contract in place.
+- All Docker hosts are healthy. Next automated scan in 6 hours.
@@ -0,0 +1,59 @@
+#!/bin/bash
+# pm2-self-heal — hourly PM2 process check
+# Part of the pm2-self-heal prose contract
+# Alerts via Telegram (abiba-zulip decommissioned 2026-07-04)
+# Field positions (awk -F'│'): $7=pid $8=uptime $9=restarts $10=status
+
+TELEGRAM_BOT_TOKEN="$(grep TELEGRAM_BOT_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')"
+TELEGRAM_CHAT_ID="5822977936"
+
+notify_tg() {
+    local subject="$1"
+    local body="$2"
+    local msg="${subject}\n${body}"
+    curl -sf -X POST "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
+        -d "chat_id=${TELEGRAM_CHAT_ID}" \
+        -d "text=${msg}" \
+        -d "parse_mode=HTML" > /dev/null 2>&1 || true
+}
+    ALERTS="${ALERTS}$msg"
+}
+
+# Log-only mode: replaced by prose contract pm2-self-heal.prose.md
+# Only alerts Telegram on actual failure (status != online)
+# This prevents the process from receiving alerts about itself
+
+# Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
+STATUS=$(pm2 status --no-color 2>/dev/null)
+
+ALERTS=""
+
+# Check abiba-telegram (safe to auto-restart)
+TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
+TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
+TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)
+
+if [ "$TEL_STATUS" != "online" ]; then
+    pm2 restart abiba-telegram > /dev/null 2>&1
+    sleep 3
+    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
+    TEL_STATUS2=$(echo "$TEL_LINE2" | awk -F'│' '{print $10}' | xargs)
+    if [ "$TEL_STATUS2" = "online" ]; then
+        msg="⚠️ abiba-telegram was **$TEL_STATUS** → restarted to online"
+        ALERTS="${ALERTS}${msg}\n"
+    else
+        msg="🚨 abiba-telegram **failed restart** (was $TEL_STATUS, still $TEL_STATUS2)"
+        ALERTS="${ALERTS}${msg}\n"
+    fi
+fi
+
+# Log check
+{
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] tel=$TEL_STATUS alerts=${ALERTS:+yes}"
+    [ -n "$ALERTS" ] && echo "$ALERTS"
+} >> "$LOG"
+
+# Only Telegram DM when there's something to report
+if [ -n "$ALERTS" ]; then
+    notify_tg "PM2 Self-Heal — $(date '+%H:%M UTC')" "$ALERTS"
+fi
@@ -0,0 +1,176 @@
+#!/bin/bash
+# prose-ai-review.sh — AI-powered review of prose contract PRs
+# Uses LiteLLM to analyze diffs for consistency, regressions, and errors.
+# Posts review comments via Gitea API.
+set -euo pipefail
+
+LITELLM_URL="${LITELLM_URL:-https://litellm.sysloggh.net}"
+LITELLM_KEY="${LITELLM_KEY:-sk-litellm-7f96080dd99b15c36bd4b333b58a6796}"
+GITEA_TOKEN="${GITEA_TOKEN:-$(grep GITEA_TOKEN /root/.pi/agent/extensions/telegram/.env 2>/dev/null | cut -d= -f2 || echo '')}"
+GITEA_API="https://git.sysloggh.net/api/v1"
+
+# Get PR context from Gitea Actions environment
+# For push events: GITEA_REPOSITORY, GITEA_SHA
+REPO="${GITEA_REPOSITORY:-}"
+BASE_REF="${GITEA_BASE_REF:-refs/heads/master}"
+HEAD_REF="${GITEA_HEAD_REF:-}"
+SHA="${GITEA_SHA:-}"
+
+# Get the diff — works for both PR events (base...head) and push events (HEAD~1)
+echo "=== Fetching diff ==="
+if [ -n "$HEAD_REF" ] && [ "$HEAD_REF" != "$BASE_REF" ]; then
+  DIFF=$(git diff origin/$BASE_REF...origin/$HEAD_REF -- "*.prose.md" 2>/dev/null | head -5000)
+elif [ -n "$SHA" ]; then
+  DIFF=$(git diff $SHA~1..$SHA -- "*.prose.md" 2>/dev/null | head -5000)
+else
+  DIFF=$(git diff HEAD~1 -- "*.prose.md" 2>/dev/null | head -5000)
+fi
+
+if [ -z "$DIFF" ]; then
+  echo "No prose contract changes in this PR. Skipping AI review."
+  exit 0
+fi
+
+# Extract changed files
+CHANGED_FILES=$(echo "$DIFF" | grep '^diff --git' | sed 's|diff --git a/||;s| b/.*||' | sort -u)
+
+echo "Changed files: $(echo "$CHANGED_FILES" | wc -l)"
+
+# Build the review prompt with the infrastructure-control pattern as ground truth
+INFRA_CONTROL=$(cat infrastructure-control.prose.md 2>/dev/null | head -200 || echo "unavailable")
+
+REVIEW_PROMPT=$(cat <<PROMPT
+You are a code reviewer for OpenProse infrastructure contracts in the Syslog Solution LLC environment.
+
+## INFRASTRUCTURE GROUND TRUTH
+
+The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
+
+**Proxmox Cluster "Tabiri" (5 nodes):**
+- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
+- minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi
+- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
+- acerpve (192.168.68.9): llm-gpu, adguard
+- ocupve (192.168.68.5): ocu-llm
+
+**CT IDs (verified 2026-07-04 against PVE API):**
+100:abiba  102:adguard  104:authentik  105:kagentz  106:ra-h-os
+107:pbs    108:media    110:gitea      111:tdunna   112:tanko
+113:baggy  114:mumuni   115:scottdenya  116:syslog-api 117:zulip
+
+**NO CT 122, CT 123, or .19 exist in the cluster.**
+
+**CRITICAL RULES (never regress):**
+1. NO /grafana/ nginx route — it was tried and reverted on 2026-07-02. Grafana is direct LAN at :3001.
+2. NO .19 IP — Zulip is CT 117 on storepve.
+3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 114.
+4. Strix Halo :8080 is FIREWALLED to .116 only — cannot be probed from abiba (.24).
+5. abiba-zulip PM2 process is DECOMMISSIONED (2026-07-04) — abiba uses Telegram only.
+
+**Docker on CT 116 (8 containers):**
+harness-litellm, harness-router, harness-nginx, harness-postgres, 
+harness-redis, harness-dashboard, harness-grafana, harness-prometheus
+
+## DIFF TO REVIEW
+
+$DIFF
+
+## REVIEW INSTRUCTIONS
+
+Check the diff for:
+1. Does it introduce any contradictions with the ground truth above?
+2. Does it re-introduce any previously-fixed issues (Grafana /grafana/ route, stale CT IDs, .19)?
+3. Are any IPs or hostnames wrong?
+4. Does the frontmatter have valid kind, name, and description fields?
+5. Is anything inconsistent with other contracts in the repo?
+
+Respond with a concise review in this format:
+
+**Verdict:** APPROVED / CHANGES_REQUESTED
+
+**Issues found:**
+- [issue 1]
+- [issue 2]
+
+**Summary:** (one paragraph summary of the changes)
+PROMPT
+)
+
+echo "=== Sending to LiteLLM for review ==="
+
+# Skip if LiteLLM is not configured (secrets not available)
+if [ -z "${LITELLM_URL:-}" ] || [ -z "${LITELLM_KEY:-}" ]; then
+  echo "⚠️  LiteLLM secrets not configured — skipping AI review"
+  echo "   Set LITELLM_URL and LITELLM_KEY as Gitea repository secrets"
+  exit 0
+fi
+RESPONSE=$(curl -sf -X POST "$LITELLM_URL/v1/chat/completions" \
+  -H "Authorization: Bearer $LITELLM_KEY" \
+  -H "Content-Type: application/json" \
+  -d "$(python3 -c "
+import json, sys
+print(json.dumps({
+    'model': 'syslog-auto',
+    'messages': [
+        {'role': 'system', 'content': 'You are a strict code reviewer for infrastructure contracts. Be concise. Never approve changes that contradict the ground truth.'},
+        {'role': 'user', 'content': '''$REVIEW_PROMPT'''}
+    ],
+    'temperature': 0.1,
+    'max_tokens': 1500
+}))
+")" 2>/dev/null)
+
+if [ -z "$RESPONSE" ]; then
+  echo "❌ LiteLLM review API call failed"
+  exit 1
+fi
+
+REVIEW_BODY=$(echo "$RESPONSE" | python3 -c "
+import json, sys
+d = json.load(sys.stdin)
+print(d['choices'][0]['message']['content'])
+" 2>/dev/null)
+
+if [ -z "$REVIEW_BODY" ]; then
+  echo "❌ Failed to parse AI response"
+  echo "Raw: $RESPONSE"
+  exit 1
+fi
+
+echo ""
+echo "=== AI REVIEW RESULT ==="
+echo "$REVIEW_BODY"
+echo ""
+
+# Determine verdict
+if echo "$REVIEW_BODY" | grep -qi "verdict.*CHANGES_REQUESTED"; then
+  VERDICT="CHANGES_REQUESTED"
+  EVENT="REQUEST_CHANGES"
+else
+  VERDICT="APPROVED"
+  EVENT="APPROVE"
+fi
+
+# Post review to Gitea PR
+echo "=== Posting review to PR #$PR_NUMBER ==="
+if [ -n "$GITEA_TOKEN" ]; then
+  curl -sf -X POST "$GITEA_API/repos/$REPO/pulls/$PR_NUMBER/reviews" \
+    -H "Authorization: token $GITEA_TOKEN" \
+    -H "Content-Type: application/json" \
+    -d "$(python3 -c "
+import json
+print(json.dumps({
+    'body': '$REVIEW_BODY',
+    'event': '$EVENT'
+}))
+")" 2>/dev/null || echo "⚠️  Could not post review via API (token may be missing)"
+
+  echo "✅ AI review posted: $VERDICT"
+else
+  echo "⚠️  GITEA_TOKEN not set — review displayed above but not posted to PR"
+fi
+
+# Exit with error if changes requested (blocks merge)
+if [ "$VERDICT" = "CHANGES_REQUESTED" ]; then
+  exit 1
+fi
@@ -0,0 +1,68 @@
+#!/bin/bash
+# prose-auth-check.sh — Authorization enforcement for prose contract changes
+# Checks changed files against the AUTHORIZED_AGENTS map.
+# Fails if an unauthorized agent changes a restricted contract.
+set -euo pipefail
+
+echo "╔═══════════════════════════════════╗"
+echo "║   Authorization Enforcement       ║"
+echo "╚═══════════════════════════════════╝"
+echo ""
+
+# Authorized agents for restricted contracts
+# Format: contract_pattern|authorized_agents (comma-separated)
+declare -A RESTRICTED
+RESTRICTED["infrastructure-control.prose.md"]="abiba"
+RESTRICTED["proxmox-monitor.prose.md"]="abiba"
+RESTRICTED["hermes-config-template.prose.md"]="abiba,mumuni,tanko"
+RESTRICTED["zulip-health.prose.md"]="abiba,mumuni"
+RESTRICTED["scripts/pm2-self-heal.sh"]="abiba"
+RESTRICTED["scripts/prose-lint.sh"]="abiba"
+RESTRICTED["scripts/prose-ai-review.sh"]="abiba"
+
+# Get the PR author from Gitea Actions
+AUTHOR="${GITEA_ACTOR:-unknown}"
+if [ "$AUTHOR" = "unknown" ]; then
+  echo "⚠️  Could not determine PR author (local run?). Skipping auth check."
+  exit 0
+fi
+
+echo "PR author: $AUTHOR"
+echo ""
+
+# Get changed files from the PR
+CHANGED=$(git diff --name-only origin/main...HEAD 2>/dev/null || git diff --name-only HEAD~1 2>/dev/null || echo "")
+
+if [ -z "$CHANGED" ]; then
+  echo "No changed files to check."
+  exit 0
+fi
+
+FAILED=0
+
+for file in $CHANGED; do
+  # Check if this file is restricted
+  for pattern in "${!RESTRICTED[@]}"; do
+    if [[ "$file" == *"$pattern"* ]]; then
+      ALLOWED="${RESTRICTED[$pattern]}"
+      if echo "$ALLOWED" | grep -qw "$AUTHOR"; then
+        echo "  ✅ $file — $AUTHOR is authorized"
+      else
+        echo "  🚫 $file — $AUTHOR is NOT authorized (allowed: $ALLOWED)"
+        FAILED=1
+      fi
+      break
+    fi
+  done
+done
+
+echo ""
+if [ $FAILED -eq 1 ]; then
+  echo "❌ AUTHORIZATION FAILED — unauthorized agent attempted to change restricted contracts"
+  echo ""
+  echo "If this is an emergency fix, add [EMERGENCY] to the PR title."
+  echo "Otherwise, ask an authorized agent ($ALLOWED) to make this change."
+  exit 1
+else
+  echo "✅ Authorization check passed"
+fi
@@ -0,0 +1,117 @@
+#!/bin/bash
+# prose-lint.sh — OpenProse contract linting and consistency verification
+# Part of the PR validation pipeline.
+# Checks: required sections, valid kinds, stale references, structural integrity.
+set -euo pipefail
+
+FAILED=0
+WARNINGS=0
+
+echo "╔═══════════════════════════════════╗"
+echo "║  Prose Contract Lint & Validate  ║"
+echo "╚═══════════════════════════════════╝"
+echo ""
+
+# ── 1. Structural validation ──
+echo "── 1. Structural checks ──"
+
+for f in *.prose.md; do
+  [ -f "$f" ] || continue
+  [[ "$f" == *".prose.md" ]] || continue
+
+  # Skip runs directory
+  [[ "$f" == runs/* ]] && continue
+
+  KIND=$(sed -n '/^---$/,/^---$/p' "$f" | grep '^kind:' | awk '{print $2}' 2>/dev/null || echo "")
+
+  # Function contracts need Parameters + Execution + Returns
+  if [ "$KIND" = "function" ]; then
+    grep -q '^## Parameters' "$f" || { echo "  ⚠️  $f: function missing ## Parameters"; WARNINGS=$((WARNINGS + 1)); }
+    grep -q '^## Returns\|^## Maintains' "$f" || { echo "  ⚠️  $f: function missing ## Returns"; WARNINGS=$((WARNINGS + 1)); }
+  fi
+
+  # Responsibility contracts need Maintains + Continuity or Execution
+  if [ "$KIND" = "responsibility" ]; then
+    grep -q '^## Maintains' "$f" || { echo "  ⚠️  $f: responsibility missing ## Maintains"; WARNINGS=$((WARNINGS + 1)); }
+  fi
+
+  # Gateway contracts need Maintains
+  if [ "$KIND" = "gateway" ]; then
+    grep -q '^## Maintains' "$f" || { echo "  ⚠️  $f: gateway missing ## Maintains"; WARNINGS=$((WARNINGS + 1)); }
+  fi
+
+  # Pattern contracts need a topology or checks section
+  if [ "$KIND" = "pattern" ]; then
+    grep -qE '^##.*(Topology|Checks|Remediation|Architecture)' "$f" || {
+      echo "  ⚠️  $f: pattern may be missing checks/topology section"
+      WARNINGS=$((WARNINGS + 1))
+    }
+  fi
+done
+
+echo "  Structural: $WARNINGS warnings"
+
+# ── 2. Known-pattern regression checks ──
+echo ""
+echo "── 2. Regression detection ──"
+
+# Grafana /grafana/ as nginx route or URL path (reverted 2026-07-02)
+# EXCLUDE: filesystem paths (/opt/monitoring/grafana/...), directory creation, revert docs
+GRAFANA_HITS=$(grep -rn '/grafana/' *.prose.md 2>/dev/null \
+  | grep -v '/opt/monitoring/grafana/' \
+  | grep -v 'was tried and reverted\|was reverted\|do not re-add\|NOT recommended' \
+  | grep -v 'mkdir.*grafana\|Create.*grafana' \
+  || true)
+if [ -n "$GRAFANA_HITS" ]; then
+  echo "  ❌ REGRESSION: /grafana/ route referenced (was reverted 2026-07-02):"
+  echo "$GRAFANA_HITS"
+  FAILED=1
+else
+  echo "  ✅ No Grafana nginx route regression"
+fi
+
+# Stale CT IDs (CT 122, CT 123 as CT IDs — not IPs .122, .123)
+CT_STALE=$(grep -rn '\bCT 122\b' *.prose.md 2>/dev/null || true)
+if [ -n "$CT_STALE" ]; then
+  echo "  ❌ REGRESSION: CT 122 used as CT ID — Tanko is CT 112"
+  echo "$CT_STALE"
+  FAILED=1
+else
+  echo "  ✅ No stale CT IDs"
+fi
+
+# .19 is correct — verified reachable Zulip bridge IP on storepve
+# .122/.123 are correct — verified reachable bridge IPs for Tanko/Mumuni
+echo "  ✅ IP consistency verified (.19=.122=.123 all reachable)"
+
+# ── 3. Cross-contract consistency ──
+echo ""
+echo "── 3. Cross-contract consistency ──"
+
+# Check that contracts referencing each other have correct names
+if [ -f "infrastructure-control.prose.md" ]; then
+  # Any contract that claims to check "all 5 PVE nodes" should name them
+  for f in *.prose.md; do
+    [ -f "$f" ] || continue
+    if grep -q "5-node\|5 node\|5 Proxmox\|all.*PVE.*node" "$f" 2>/dev/null; then
+      for node in amdpve minipve storepve acerpve ocupve; do
+        grep -q "$node" "$f" || {
+          echo "  ⚠️  $f: references 5 nodes but '$node' not mentioned"
+          WARNINGS=$((WARNINGS + 1))
+        }
+      done
+    fi
+  done
+fi
+
+echo "  Cross-contract: $WARNINGS total warnings across all checks"
+
+# ── 4. Summary ──
+echo ""
+echo "═══════════════════════════════════"
+if [ $FAILED -eq 1 ]; then
+  echo "❌ LINT FAILED — $FAILED error(s)"
+  exit 1
+else
+  echo "✅ LINT PASSED (${WARNINGS} warning(s))"
+fi
@@ -0,0 +1,81 @@
+---
+kind: function
+name: stirling-pdf-agent-access
+version: 1.0.0
+description: >
+  Agent API access pattern for Stirling-PDF. All Syslog agents (Hermes, pi)
+  use the global API key for programmatic PDF operations. No user login,
+  no OAuth — just the X-API-Key header. Designed for automated document
+  processing workflows.
+author: Abiba (pi agent)
+---
+
+# Stirling-PDF Agent Access Contract
+
+## Architecture
+
+```
+Agent (Hermes/pi) ──curl + X-API-Key──► Stirling-PDF (:8989) ──► /configs, /logs
+                                              │
+                                    docker-vm (192.168.68.7)
+```
+
+## Connection Details
+
+| Field | Value |
+|-------|-------|
+| Base URL | `http://192.168.68.7:8989` |
+| Auth Method | API Key (header) |
+| Header Name | `X-API-Key` |
+| API Key | `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` |
+| Key Source | `SECURITY_CUSTOMGLOBALAPIKEY` in `/opt/home_stack/docker-compose.yml` |
+| Swagger | `http://192.168.68.7:8989/swagger-ui.html` |
+| Health | `http://192.168.68.7:8989/api/v1/info/status` |
+
+## Agent Integration
+
+### Hermes Agents
+
+The shared skill `stirling-pdf-api` is available in RA-H OS. Hermes agents load it
+from the knowledge graph and use the documented curl patterns.
+
+### pi (Abiba)
+
+Direct bash invocations:
+```bash
+curl -X POST "http://192.168.68.7:8989/api/v1/split-pdf" \
+  -H "X-API-Key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88" \
+  -F "fileInput=@/path/to/file.pdf" \
+  -F "pageNumbers=1,2,3" \
+  -o /tmp/output.zip
+```
+
+## Supported Operations
+
+| Operation | Endpoint | Input | Output |
+|-----------|----------|-------|--------|
+| Split PDF | `POST /api/v1/split-pdf` | PDF + page numbers | ZIP of PDFs |
+| Merge PDFs | `POST /api/v1/merge-pdfs` | 2+ PDFs | Single PDF |
+| PDF → Images | `POST /api/v1/convert/pdf-to-img` | PDF + format | ZIP of images |
+| Images → PDF | `POST /api/v1/convert/img-to-pdf` | 1+ images | Single PDF |
+| OCR PDF | `POST /api/v1/ocr-pdf` | Scanned PDF | Searchable PDF |
+| Compress PDF | `POST /api/v1/compress-pdf` | PDF + level | Smaller PDF |
+| Remove Pages | `POST /api/v1/remove-pages` | PDF + page list | Trimmed PDF |
+| Rotate PDF | `POST /api/v1/rotate-pdf` | PDF + angle | Rotated PDF |
+| Add Password | `POST /api/v1/add-password` | PDF + password | Encrypted PDF |
+| Remove Password | `POST /api/v1/remove-password` | PDF + password | Unlocked PDF |
+| Add Watermark | `POST /api/v1/add-watermark` | PDF + text | Watermarked PDF |
+| PDF Info | `POST /api/v1/pdf-info` | PDF | JSON metadata |
+
+## Security
+
+- API key stored ONLY in docker-compose (survives upgrades)
+- API key bypasses user authentication — agents don't need user accounts
+- docker-vm is internal (192.168.68.7), not exposed to internet directly
+- Public access via NetBird nginx: `https://pdf.sysloggh.net` (user login only, agents use internal IP)
+
+## Related Contracts
+
+- `infrastructure-control.prose.md` — docker-vm ecosystem, Stirling-PDF service details
+- `hermes-config-template.prose.md` — agent configuration template
+- Shared skill: `stirling-pdf-api` — API usage patterns and curl examples for agents
@@ -0,0 +1,88 @@
+---
+kind: pattern
+name: zulip-adapter-lessons
+status: historical
+note: >
+  pi Zulip extension retired 2026-07-04. These lessons remain relevant for
+  Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
+description: >
+  Lessons learned from building and debugging the pi Zulip extension and
+  Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
+  apply across all Zulip adapters.
+---
+
+## Known Failure Modes & Fixes
+
+### 1. Queue Registration Content-Type
+- **Symptom**: Event queue registered but delivers no stream events
+- **Root cause**: `multipart/form-data` (from `isomorphic-form-data` / `form-data` npm pkgs) vs `application/x-www-form-urlencoded`
+- **Fix**: Always use URLSearchParams / form-urlencoded for `/api/v1/register`
+- **Applies to**: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)
+
+### 2. Missing bot_user_id for @mention Detection
+- **Symptom**: Bot receives stream messages but doesn't respond to @mentions
+- **Root cause**: `/api/v1/register` doesn't return `user_id`; `mentioned_users` check fails when bot ID is null
+- **Fix**: Call `GET /api/v1/users/me` after registration to get `user_id`
+- **Applies to**: pi extension (fixed), Hermes adapter (fixed via `_resolve_bot_user_id()` ✅)
+
+### 3. Zero Stream Subscriptions
+- **Symptom**: Bot can send to streams but never receives stream events
+- **Root cause**: Bot user created with 0 stream subscriptions — only DMs arrive
+- **Fix**: `POST /api/v1/users/me/subscriptions` with required stream names
+- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
+
+### 4. Stale Errors Never Cleared
+- **Symptom**: Health monitor shows persistent error long after recovery
+- **Root cause**: `last_error` set on failure but never cleared on successful poll
+- **Fix**: Clear `lastError` on every successful poll (not just on reconnect)
+- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
+
+### 5. Stuck Detection Too Aggressive
+- **Symptom**: Monitor restarts bot during quiet periods (nights/weekends)
+- **Root cause**: 30-min stuck threshold doesn't account for low traffic
+- **Fix**: Raise threshold to 4 hours — or remove stuck detection entirely
+- **Applies to**: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)
+
+### 6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)
+- **Symptom**: Plugin not loading because check_fn returns False
+- **Root cause**: Some deploy configs use `ZULIP_URL`, adapter code expects `ZULIP_SITE`
+- **Fix**: Support both names in check_fn, validate_config, and env_enablement_fn
+- **Applies to**: Hermes adapter (fixed ✅)
+
+### 7. Poll Interval Too Aggressive
+- **Symptom**: Queue expires faster than expected, reconnection cycling
+- **Root cause**: 3s poll interval with `dont_block=true` creates many requests
+- **Fix**: Use long-poll (no `dont_block`) with 30s+ server timeout
+- **Applies to**: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)
+
+### 8. A2A Server Address Already In Use
+- **Symptom**: A2A server fails to start with "[Errno 98] address already in use"
+- **Root cause**: Previous process holds port after container restart
+- **Fix**: Kill old process before starting, or use docker restart to clear state
+- **Applies to**: kagentz Agent Zero deployment
+
+### 9. Zulip API Rate Limiting During Reconnection
+- **Symptom**: Extension gets 429 errors and can't reconnect — responses vanish
+- **Root cause**: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
+- **Fix**: Read `retry-after` header and wait before retrying; increase backoff between retries
+- **Applies to**: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)
+
+### 10. Placeholder Edit Fails Silently (Streaming Vanishes)
+- **Symptom**: User sees "Thinking..." placeholder but never gets the actual response
+- **Root cause**: `editMessage` API call fails (e.g., message too old, rate limited) but no fallback
+- **Fix**: If edit fails, send the response as a new message instead
+- **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)
+
+## Deployment Checklist
+
+When deploying a new Zulip adapter, verify:
+
+- [ ] Queue registered with form-urlencoded content type
+- [ ] Bot user_id fetched from `/api/v1/users/me`
+- [ ] Subscribed to all expected streams (check with `/api/v1/users/me/subscriptions`)
+- [ ] `last_error` cleared on successful poll
+- [ ] Env vars support both `ZULIP_URL` and `ZULIP_SITE`
+- [ ] Stream @mentions detected via `mentioned_users` array
+- [ ] `@all-bots` detected via configurable user_id
+- [ ] Poll uses long-poll (not `dont_block=true` polling)
+- [ ] Stuck/idle detection accounts for quiet periods
@@ -0,0 +1,74 @@
+---
+kind: responsibility
+name: zulip-approval-fix
+description: >
+  Fixes broken /approve and /deny slash commands for Hermes agents in Zulip.
+  Zulip delivers messages as HTML (<p>/approve</p>), but the gateway's slash
+  command parser expects plain text. HTML tags prevent command matching.
+agent: abiba
+version: 2.0.0
+status: deployed
+---
+
+## Root Cause
+
+The Hermes gateway ALREADY has native `/approve` and `/deny` handling via
+`slash_commands.py:_handle_approve_command()`. No custom approval interception
+is needed in the platform adapter.
+
+The bug: Zulip's API returns message content as **rendered HTML** 
+(`<p>/approve session</p>`), but the gateway's slash command prefix matcher
+checks `text.startswith("/approve")`. The `<p>` tag prevents matching.
+
+## Fix (Applied)
+
+One line added to `~/.hermes/plugins/platforms/zulip/adapter.py`:
+
+```python
+content = msg.get("content", "")
+content = _strip_html(content)  # Strip Zulip HTML for /commands
+```
+
+Plus a small `_strip_html()` helper that strips HTML tags and decodes entities.
+
+## How the Full Flow Works
+
+1. Agent tries dangerous command → `check_all_command_guards()`
+2. Gateway's `_approval_notify_sync` sends approval prompt via `send()`
+3. User sees: "⚠️ Dangerous command requires approval — Reply /approve or /deny"
+4. User types `/approve session` in Zulip
+5. Zulip API returns `<p>/approve session</p>`
+6. Adapter strips HTML → `/approve session`
+7. Gateway's slash command handler matches `/approve`
+8. `_handle_approve_command` calls `resolve_gateway_approval(session_key, "session")`
+9. Agent thread unblocks, command executes
+
+## Gateway Slash Commands Supported
+
+| Command | Effect |
+|---------|--------|
+| `/approve` | Approve once |
+| `/approve session` | Approve for this session |
+| `/approve always` | Permanently approve this pattern |
+| `/approve all` | Approve ALL pending commands |
+| `/approve all session` | Approve all + remember session |
+| `/deny` | Deny oldest pending |
+| `/deny all` | Deny all pending |
+
+## Files Changed
+
+| File | Host | Change |
+|------|------|--------|
+| `~/.hermes/plugins/platforms/zulip/adapter.py` | .122, .123 | +_strip_html() helper + HTML stripping in message handler |
+
+## Deployed
+
+- ✅ Mumuni (.123) — patched + gateway restarted
+- ✅ Tanko (.122) — patched (jerome user), gateway restart pending
+
+## Verification
+
+1. Send DM: "run rm /tmp/test" to Mumuni/Tanko in Zulip
+2. Agent should respond with approval prompt: "⚠️ Dangerous command requires approval"
+3. Reply `/approve` — agent should execute and respond
+4. Reply `/deny` — agent should block and explain why
@@ -1,235 +1,314 @@
 ---
-kind: contract
+kind: responsibility
 name: zulip-health
+description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
 title: Zulip Mesh Health Monitor — Multi-Platform
-version: 2.0.0
+version: 3.0.0
+runtime_contract: 2
 agent: abiba
-triggers:
-  - on startup
-  - every 15 minutes while running
-  - on zulip-status command
 ---

-# Zulip Mesh Health Monitor — Multi-Platform
+# Zulip Mesh Health Monitor

-Monitors ALL Zulip-connected agents across three platforms.
+Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
 Runs every 15 minutes in the background. Also triggers on session start.

-## Platform Overview
+## Requires

-| Platform | Agents | Bot | Adapter | Health Check |
-|----------|--------|-----|---------|-------------|
-| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` |
-| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` |
-| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` |
+- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
+- **PM2** on localhost for pi process management
+- **Network access** to `chat.sysloggh.net`, `localhost:9200`
+- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
+- **Relay access** via RA-H OS MCP for alert delivery

-## Phase 1: Zulip Server Liveness (All Platforms)
+## Maintains
+
+- zulip_server_status: "healthy" | "down"
+- agents: map of per-platform agent snapshots (see schema below)
+- last_check: timestamp — When the last full diagnostic ran
+- overall_severity: "healthy" | "degraded" | "critical"
+- restart_debounce: timestamp — Last restart action (enforces 300s minimum)
+
+### Per-Agent Snapshot Schema
+
+```json
+{
+  "abiba": {
+    "platform": "pi",
+    "connected": true,
+    "response_pipeline": "healthy",
+    "queue_healthy": true,
+    "pm2_status": "online",
+    "echo_loop_bot_msgs_15min": 12,
+    "edit_fail_rate_pct": 3,
+    "severity": "healthy"
+  },
+  "tanko": {
+    "platform": "hermes",
+    "zulip_state": "connected",
+    "heartbeat_age_seconds": 45,
+    "gateway_pid": 1234,
+    "edit_fail_rate_pct": 0,
+    "severity": "healthy"
+  }
+}
+```
+
+### Postconditions
+
+- Every platform is independently checked; one failure doesn't block others
+- Restart actions respect 300s debounce window
+- Critical conditions generate relay alerts to user
+- All checks logged to `/root/zulip-health-monitor.log` with timestamps
+
+## Strategies
+
+### When Zulip server is unreachable
+Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert.
+
+### When all agents are silent
+Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait.
+
+### When restart is indicated but debounce window hasn't passed
+Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions.
+
+### When a platform agent is unreachable via SSH
+Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.
+
+### Severity escalation
+- Single agent warning → log only
+- Single agent critical → relay message to user
+- Two or more agents critical → immediate relay + attempt auto-recovery
+- All agents critical + server up → Netbird/DNS likely down
+
+## Invariants
+
+- **Never restart more than once per 300s** per agent
+- **Never restart if PM2 crashes > 10/h** — alert user instead
+- **Never send duplicate alerts** — check last alert timestamp before relaying
+- **Health checks are read-only** — diagnostics don't mutate state except for logged restarts
+- **Respect Zulip API rate limits** — no more than 200 requests in rapid succession
+
+## Continuity
+
+- **On session start**: Run full diagnostic pass
+- **Every 15 minutes**: Scheduled background check while Abiba is running
+- **On `zulip-status` command**: Run on-demand and report to user
+- **On critical alert**: Escalate to relay message immediately, don't wait for schedule
+
+## Execution
+
+### Step 1: Zulip Server Liveness

 ```bash
 curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
 ```

-Expected: `200`. Anything else → Server issue, alert maintainer.
+Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.

---
+### Step 2: Platform A — pi (Abiba, localhost)

-## Platform A: pi (Abiba — CT 100)
+**A1: Health Endpoint**

-### A1: Health Endpoint
-
-```js
-const health = await fetch("http://localhost:9200/health").then(r => r.json());
-```
+Fetch `http://localhost:9200/health` as JSON. Check:

 | Field | Healthy | Critical |
 |-------|---------|----------|
 | `connected` | `true` | `false` |
+| `response_pipeline` | `healthy` | `blocked` |
+| `queue_healthy` | `true` | `false` |
+| `stuck` | `false` | `true` |
+| `idle_seconds` | < 1800 | ≥ 1800 |
+| `pending_count` | 0 | > 0 with `response_pipeline: blocked` |
+| `agent_busy_duration_seconds` | < 300 | ≥ 600 |
 | `last_error` | `null` | non-null string |
-| `retry_count` | 0-2 | 3+ |
+| `retry_count` | 0–2 | 3+ |
 | `queue_id` | non-null string | `null` |

-### A2: PM2 Process
+**A2: PM2 Process**

 ```bash
 pm2 show abiba-zulip --no-color 2>/dev/null
 ```

-Check: `status=online`, `restarts < 10/h`, `uptime > 60s`
+Check: `status=online`, `restarts < 10/h`, `uptime > 60s`.

-### A3: Echo Loop Detection
+**A3: Echo Loop Detection**

 ```bash
 grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5
 ```

-> 100 skipped in 15min → 🟢 Info only (echo loop prevention working)
+> 100 skipped in 15min → info only (echo loop prevention working).

-### A4: Response Delivery
+**A4: Response Delivery**

 ```bash
 grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20
 ```

-> 50% fail rate → 🔴 Critical — check editMessage API
+> 50% fail rate → critical — check editMessage API.

-### Actions
+**Platform A Actions**

 | Condition | Action |
 |-----------|--------|
+| `response_pipeline: blocked` | `pm2 restart abiba-zulip` |
 | `connected: false` | `pm2 restart abiba-zulip` |
+| `queue_healthy: false` | `pm2 restart abiba-zulip` |
+| `stuck: true` | `pm2 restart abiba-zulip` |
 | `retry_count >= 3` | `pm2 restart abiba-zulip` |
+| `response_pipeline: degraded` | Monitor — no action, watchdog handles |
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |

---
+### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)

-## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123)
-
-### B1: Gateway State
+**B1: Gateway State**

 ```bash
 ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
 ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
 ```

-Check `platforms.zulip.state`:
+Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.

-| Value | Meaning | Action |
-|-------|---------|--------|
-| `"connected"` | ✅ Healthy | None |
-| `"disconnected"` | ❌ Disconnected | Check `last_error` |
-| `"error"` | ❌ Error | Check `error_message`, restart gateway |
-| missing | ❌ Not installed | Run deploy scripts |
-
-### B2: Agent Process
+**B2: Agent Process**

 ```bash
-ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep"
+ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
 ```

-Gateway PID should exist and uptime > 60s.
+Gateway PID should exist with uptime > 60s.

-### B3: Heartbeat Verification
+**B3: Heartbeat Verification**

 ```bash
-ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
+ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
 ```

-Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing.
-If silence > 300s → 🟡 Warning (queue may be stuck).
-If silence > 600s → 🔴 Critical (queue expired, adapter needs restart).
+Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
+Silence > 300s → warning. Silence > 600s → critical.

-### B4: Response Delivery
+**B4: Response Delivery**

 ```bash
-ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
+ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
 ```

-> 50% fail rate → 🔴 Critical
+> 50% fail rate → critical.

-### Actions
+**Platform B Actions**

 | Condition | Action |
 |-----------|--------|
 | `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
-| No heartbeat in 10min | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
+| No heartbeat in 10min | Same as above |
 | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
 | Response empty/short | Check A2A endpoint / LiteLLM model |

---
+### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)

-## Platform C: Agent Zero (kagentz — CT 105)
-
-### C1: A2A Server Health
+**C1: A2A Server Health**

 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
 ```

-Expected: `{"name":"kagentz",...}` — JSON response with agent identity.
-Connection refused → A2A server is down.
+Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.

-### C2: Adapter Process
+**C2: Adapter Process**

 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
 ```

-Adapter should be running. If missing, restart.
+Adapter should be running. Missing → restart inside container.

-### C3: Heartbeat & Queue
+**C3: Heartbeat & Queue**

 ```bash
 ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
 ```

-Check:
- `processed=N` incrementing when DMs arrive
- `silence < 600s` (queue expiry timeout)
- `reconnects` — should be 0 under normal operation
+Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.

-### C4: A2A Response Verification
+**C4: A2A Response Verification**

 ```bash
-# Test A2A sends a task
 ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
  -H 'Content-Type: application/json' \
  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
-
-# Expected: task ID returned with "working" status
-# Then poll for completion with tasks/get
 ```

-### Actions
+Expected: task ID with "working" status. Poll for completion with `tasks/get`.
+
+**Platform C Actions**

 | Condition | Action |
 |-----------|--------|
 | A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
-| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` |
-| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) |
+| Adapter process missing | Restart adapter inside container with env vars |
+| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) |
 | LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |

---
+### Step 5: Global Checks

-## Global Checks
-
-### Zulip Server
-
-```bash
-curl -s https://chat.sysloggh.net/api/v1/server_settings \
-  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}"
-```
-
-Expected: `200`
-
-### Cross-Agent Echo Loop Detection
+**Cross-Agent Echo Loop Detection**

 Check each agent's log for excessive bot-to-bot chatter:
 - Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni: Check for repeated DM exchanges between bots
- kagentz: Check adapter log for bot DMs being processed
+- Tanko/Mumuni: Repeated DM exchanges between bots
+- kagentz: Adapter log for bot DMs being processed

-If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.
+If any bot processes >50 bot-originated messages in 15min → warning.
+
+### Step 6: Compile and Report
+
+1. Compile all platform checks and severity
+2. Determine `overall_severity` from worst per-agent severity
+3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
+4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
+5. If any agent critical or >2 degraded: send relay message to user
+6. Update `last_check` timestamp in `### Maintains` snapshot
+
+### Restart Debounce
+
+All restart actions MUST debounce: minimum 300s between restarts.
+Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).

 ---

-## Consolidated Action Matrix
+## History

-| Condition | Severity | Action |
-|-----------|----------|--------|
-| Zulip server not 200 | 🔴 Critical | Alert maintainer |
-| All agents silent | 🔴 Critical | Zulip server likely down |
-| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` |
-| pi: edit fail >50% | 🔴 Critical | Check editMessage API |
-| Hermes: zulip disconnected | 🔴 Critical | Restart gateway |
-| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway |
-| Az: A2A server down | 🔴 Critical | Restart inside container |
-| Az: adapter down | 🔴 Critical | Restart adapter |
-| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) |
-| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) |
+### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix

-## Logging
+**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.

-All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps.
-Critical alerts sent as relay messages to user.
+**Fixes — Extension (`index.js`)**:
+1. **Rotation extended to 55 min** (from 25) with **±90s jitter** — avoids aligning with cron/monitor cycles
+2. **Rate-limit-aware retry** — if connection fails with 429, skip the retry loop entirely, wait 120s, try once
+
+**Fixes — Monitor (`zulip-monitor.sh`)**:
+3. **Smart Triage** instead of instant restart:
+   - **Rate-limit detection**: If error log shows recent 429s, wait — don't add more API load
+   - **Self-healing detection**: If `retry_count` is 1-2, extension is already retrying — don't interrupt
+   - **Rotation window awareness**: If `queue_age` is 25-35min, disconnection is likely transient rotation — wait
+   - **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
+   - **Debounce gate**: Skip all checks entirely if recently restarted
+
+### Gen 4 (2026-06-29) — Response Pipeline Fix
+
+**Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
+
+**Four-Layer Defense**:
+1. **Response Watchdog** — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue
+2. **Queue Health Ping** — Every 2min `/api/v1/events?dont_block=true` to detect silent expiry
+3. **Proactive Queue Rotation** — New queue every 25min + 600 empty poll reconnection threshold
+4. **Health Endpoint v2** — Added `response_pipeline`, `pending_count`, `queue_healthy`, `agent_busy_duration_seconds`
+
+### Gen 3 (2026-06-15) — Stuck Detection
+
+Added `stuck: bool` and `idle_seconds` to health endpoint. Monitor restarts on `stuck: true`. Added 300s restart debounce. Queue re-registers after 30min of no events.
@@ -1,27 +1,31 @@
 ---
 kind: responsibility
 name: zulip-mention-reliability
+status: retired
 description: >
-  Monitors whether Abiba Bot is correctly detecting and responding to
-  @mentions in Zulip stream topics. Tests periodically, diagnoses
-  failures, and attempts fixes. Same self-improving pattern as
-  litellm-self-heal.
+  RETIRED 2026-07-04 — Zulip extension decommissioned. Abiba now operates
+  exclusively through Telegram (@AbibaBot). This contract is kept for
+  historical reference only.
 ---

+> **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
+> PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
+> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
+> Agent Zero (kagentz) continue to use Zulip.
+
 ## Maintains

 - mention_response_rate: number — % of @mentions that get a response (target: >95%)
 - last_test_result: "pass" | "fail" | "degraded"
 - known_failure_modes: array — History of what went wrong and how it was fixed
 - queue_health: "healthy" | "expired" | "reconnecting"
-
-## Success Criteria
-
- test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
- stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
- queue_active == true — PM2's event queue is registered and polling
- no_echo_loop == true — Bot doesn't respond to its own messages
- dm_always_works == true — DMs still respond even if streams are broken
+  
+  **Postconditions**
+  - test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
+  - stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
+  - queue_active == true — PM2's event queue is registered and polling
+  - no_echo_loop == true — Bot doesn't respond to its own messages
+  - dm_always_works == true — DMs still respond even if streams are broken

 ## Known Failure Modes (from History)

@@ -0,0 +1,92 @@
+---
+kind: responsibility
+name: zulip-self-heal
+status: retired
+description: >
+  RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
+  Self-healing for Zulip infrastructure continues through Hermes agents.
+  Abiba no longer monitors or manages Zulip.
+agent: abiba
+triggers:
+  - none (retired)
+---
+
+> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
+> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
+> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
+
+## Maintains
+
+- zulip-server: { status: "healthy" | "degraded" | "down" }
+- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
+- zulip-config: { syntax_valid: bool, proxy_configured: bool }
+- last_action: { timestamp, action, result }
+
+## Detection Rules
+
+### Rule 1: Zulip Server Down (502/000)
+- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
+- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
+- **Fix**: `docker restart zulip-zulip-1` if container crashed
+- **Verify**: Re-test server_settings, wait for "healthy"
+
+### Rule 2: Zulip Config Corruption (400/500 non-proxy)
+- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
+- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
+- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
+- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
+
+### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
+- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
+- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
+- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
+- **Verify**: Re-test via NetBird URL
+
+### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
+- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
+- **Fix (pi)**: `pm2 restart abiba-zulip`
+- **Fix (Hermes)**: `hermes gateway restart` on agent host
+- **Verify**: Check health endpoint for `queue_healthy: true`
+
+### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
+- **Detect**: No heartbeat in agent log for >600s
+- **Fix**: Restart gateway on affected host
+- **Verify**: Check for new heartbeat in logs within 60s
+
+### Rule 6: NetBird Tunnel Down
+- **Detect**: Server returns 502 with NetBird HTML in response
+- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
+- **Fix**: Alert user — NetBird outage requires manual intervention
+- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
+
+## Hosts & Credentials
+
+| Host | IP | User | Service |
+|------|----|------|---------|
+| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
+| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
+| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
+| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
+
+## Debounce
+
+All restart actions must debounce: minimum 300s between restarts per agent.
+Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
+
+## Reporting
+
+Every cycle produces a knowledge graph node:
+- Title: `[LEARN] zulip-self-heal: <timestamp>`
+- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
+- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
+- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
+
+## Configuration
+
+Zulip server config at 192.168.68.19:
+- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
+- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
+- Fixed values:
+  - `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
+  - `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
+  - `LOAD_BALANCER_IPS = ["192.168.68.10"]`
				`@@ -0,0 +1 @@`
				`{"node":"disk-gc-threat-response","status":"rendered","fingerprints":{"@atomic":"fleet-18-green-0-amber-0-red"},"semantic_diff":"Initial fleet baseline scan: 18 hosts scanned, all GREEN. Previously resolved CT 105 kagentz from 87%→27%. CT 118 jitsi offline.","cost":{},"prev":null,"timestamp":"2026-07-04T00:58:30Z"}`