fix: cron alignment + VM→bare metal drift

- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line

Live actions: gpu-monitor-server added to PM2 (persistent across reboots)

README.md

@@ -115,10 +115,11 @@ Run on trigger or schedule. Maintain persistent world-model state across runs.
 | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
 | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
 | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
+| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
 
 ### Instantiable Templates (`kind: pattern` — run with `prose run`)
@@ -146,8 +147,9 @@ Called on-demand as single-render tools.
 
 | Contract | Description |
 |---|---|
-| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
-| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
+| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
+| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
+| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
 
@@ -158,8 +160,8 @@ Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
-| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
-| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 
 ## Contract Structure
 

disk-gc-threat-response.prose.md

@@ -3,10 +3,11 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
   Recurring disk health scan, garbage collection, and threat response
-  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
   2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
   Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB via `docker system prune -a --force`.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
+  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
 
 ## Scope
 
-All CTs in the Proxmox cluster, with special attention to Docker hosts:
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
+Docker hosts get special attention:
 
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
+| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All others | — | LOW — no Docker | apt clean, log rotate |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
+| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
+> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 
 ## Threat Levels
 
@@ -123,7 +130,7 @@ call summary-reporter
 
 ## GC Strategies by Host Type
 
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -224,32 +231,76 @@ dangling images and orphaned build cache. No automated GC was in place.
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
+- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```
 
-Done. Here's what we have:
+## Incident Log: 2026-07-09 — amdpve docker bloat
 
-## Storage Summary
+### Discovery
+Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
+amdpve (.15) flagged at 78% (AMBER threshold: 75%).
 
-| System | Used | Free | Health |
-|--------|------|------|--------|
-| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
-| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
-| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
-| **/root/go** (abiba) | 773M | — | Fine |
+### Diagnosis
+- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
+- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
+- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
+- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
+  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
+- Not a running service — zero containers, zero active volumes
 
-## New Contract: `disk-gc-threat-response.prose.md`
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 11.56GB
+→ Post: 55G / 94G (62%), 35G free
+→ 1 image removed (llama-strix-hip:latest, 15.5GB)
+→ 8 build cache layers removed
+→ amdpve now GREEN
+```
 
-Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+### Root Cause
+Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
+artifacts. Docker on amdpve serves no running purpose — it's only used for
+one-off GPU builds. No automated post-migration cleanup was in place.
 
-### Three-in-One
+### Preventive Measures
+- amdpve added to Docker GC scan list
+- Post-migration cleanup step added: after any GPU backend migration, prune
+  the old backend's Docker images within 24 hours
+- Contract now scans GPU bare-metal hosts alongside CTs
+- Access via `pct-run` script for all CTs (no hardcoded IPs)
 
-1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+## Access Matrix (documented 2026-07-09)
 
-2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+### CT Access (via pct-run)
+| CT | Name | Node | Status |
+|----|------|------|--------|
+| 100 | abiba | amdpve | local |
+| 102 | adguard | acerpve | ✅ reachable |
+| 104 | authentik | minipve | ✅ reachable |
+| 105 | kagentz | amdpve | ✅ reachable |
+| 106 | ra-h-os | storepve | ✅ reachable |
+| 107 | pbs | storepve | ✅ reachable |
+| 108 | media | storepve | ✅ reachable |
+| 110 | gitea | minipve | ✅ reachable |
+| 111 | tdunna | amdpve | ✅ reachable |
+| 112 | tanko | amdpve | ✅ reachable |
+| 113 | baggy | amdpve | ✅ reachable |
+| 114 | mumuni | minipve | ✅ reachable |
+| 115 | scottdenya | amdpve | ✅ reachable |
+| 116 | syslog-api | minipve | ✅ reachable |
+| 117 | zulip | storepve | ✅ reachable |
 
-3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
+### GPU Bare Metal (via direct SSH)
+| Host | IP | GPU | Status |
+|------|-----|-----|--------|
+| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
+| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
+| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
 
-### The incident root cause
-Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
+### KVM VM (via direct SSH)
+| Host | IP | Role | Status |
+|------|-----|------|--------|
+| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
 
-Want me to push this to the prose-contracts repo?
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.

gpu-fleet.prose.md

@@ -142,7 +142,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 |-------|-----|-----|-----|--------|
 | Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
 | Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
-| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
 | Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
 | Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
 | Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
@@ -159,10 +159,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
 

gpu-monitor.prose.md

@@ -19,7 +19,7 @@ agent: abiba
 
 ```
 ┌─────────────────────────────────────────────────────────┐
-│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  GPU Monitor Server (:9100) — 192.168.68.24              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
    │          │          │
@@ -35,7 +35,7 @@ agent: abiba
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
-Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
 poll .15:8080 directly; must go through router on .116.
 ```
 
@@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
 
 | File | Host | Purpose |
 |------|------|---------|
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
 
 ## Execution

hermes-zulip-plugin.prose.md

@@ -0,0 +1,237 @@
+---
+kind: function
+name: hermes-zulip-plugin
+description: >
+  Installs or updates the Zulip platform plugin for any Hermes agent from the
+  canonical zulip-platform-plugins repo (master branch). Ensures the agent runs
+  the latest adapter with all Zulip chat fixes (_strip_html, streaming, event
+  recovery). Verifies the installation, restarts the gateway, and sends a relay
+  success signal.
+agent: abiba
+version: 1.0.0
+status: active
+runtime_contract: 2
+---
+
+# Hermes Zulip Plugin — Install & Repair
+
+Single-shot function that pulls the latest zulip-platform plugin from the
+canonical git repo, installs it to the correct Hermes bundled plugin path,
+verifies the installation, and signals completion.
+
+Distinct from `hermes-zulip-restore`: this contract is focused on the plugin
+layer and a targeted gateway restart. It does NOT verify env credentials or
+run a live Zulip connection test. Use `hermes-zulip-restore` for full
+connectivity recovery including end-to-end DM validation.
+
+## Parameters
+
+| Param | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
+| `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
+
+## Maintains
+
+- plugin_installed: bool — Whether all three adapter files exist at the bundled path
+- plugin_version: string — Git commit SHA of the installed version
+- strip_html_present: bool — Whether `_strip_html` fix is in the installed adapter
+- signal_sent: bool — Whether relay success message was dispatched
+
+### Postconditions
+
+- All three adapter files (`__init__.py`, `adapter.py`, `plugin.yaml`) present in `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/`
+- `_strip_html` function exists in `adapter.py` (slash-command fix, commit `55ca15d`+)
+- Installed version matches HEAD of the requested branch
+- Relay success signal sent to Hermes agent's inbox
+
+## Requires
+
+- SSH access to target host (direct or via amdpve for CTs)
+- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
+- Python 3 with `httpx` installed on target
+
+## Live-State Fields
+
+| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
+|------|-----|---------|-------------|-------------|------|
+| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
+| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
+
+| Field | Value | Trust |
+|-------|-------|-------|
+| Git repo | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
+| Default branch | `master` (contains all merged fixes including `feat/zulip-streaming`) | ✅ Verified |
+| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
+| Adapter files | `__init__.py`, `adapter.py`, `plugin.yaml` | ✅ Verified |
+| Strip-html commit | `55ca15d` (minimum) | ✅ Verified |
+
+## Execution
+
+### Step 1: Resolve Target
+
+Map `target` to host, CT ID, hermes_home, and user from the live-state table.
+For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
+
+### Step 2: Pull Latest Plugin Source
+
+On the target host:
+
+```bash
+# Ensure deploy scratch space
+mkdir -p /tmp/zulip-deploy
+cd /tmp/zulip-deploy
+
+# Clone or pull
+if [ -d zulip-platform-plugins ]; then
+  cd zulip-platform-plugins
+  git fetch origin
+  git checkout {{branch}}
+  git pull origin {{branch}}
+else
+  git clone --branch {{branch}} \
+    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
+  cd zulip-platform-plugins
+fi
+
+# Capture installed version
+INSTALLED_SHA=$(git rev-parse HEAD)
+echo "Installed SHA: $INSTALLED_SHA"
+
+# Verify we're at HEAD
+HEAD_SHA=$(git rev-parse origin/{{branch}})
+if [ "$INSTALLED_SHA" = "$HEAD_SHA" ]; then
+  echo "At latest commit on {{branch}}"
+else
+  echo "WARNING: not at HEAD — $INSTALLED_SHA vs $HEAD_SHA"
+fi
+```
+
+### Step 3: Install Plugin Files
+
+```bash
+# Ensure target directory exists
+mkdir -p {{hermes_home}}/hermes-agent/plugins/platforms/zulip
+
+# Copy adapter files
+cp plugins/platforms/zulip/adapter.py \
+   plugins/platforms/zulip/__init__.py \
+   plugins/platforms/zulip/plugin.yaml \
+   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+
+# Fix ownership (Tanko only — runs as jerome user)
+[ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
+  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+
+echo "Plugin files installed"
+```
+
+### Step 4: Verify Installation
+
+```bash
+# Check all three files exist
+for f in __init__.py adapter.py plugin.yaml; do
+  if [ -f "{{hermes_home}}/hermes-agent/plugins/platforms/zulip/$f" ]; then
+    echo "✅ $f present"
+  else
+    echo "❌ $f MISSING"
+    exit 1
+  fi
+done
+
+# Verify _strip_html fix
+grep -q "_strip_html" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/adapter.py \
+  && echo "✅ _strip_html fix present" \
+  || echo "❌ _strip_html MISSING — plugin may be stale"
+
+# Show installed plugin.yaml version
+grep "^version:" {{hermes_home}}/hermes-agent/plugins/platforms/zulip/plugin.yaml || true
+```
+
+### Step 5: Restart Gateway
+
+Plugin changes require a gateway restart to take effect:
+
+```bash
+cd {{hermes_home}}/hermes-agent
+# Use venv if available
+python3 -m hermes_cli.main gateway restart 2>&1 || \
+  venv/bin/python -m hermes_cli.main gateway restart 2>&1
+```
+
+Wait for restart to complete (up to 45s), then confirm:
+
+```bash
+grep "Gateway running" {{hermes_home}}/logs/gateway.log | tail -1
+```
+
+Expected: `Gateway running with N platform(s)` where N > 1 (includes zulip).
+
+Quick smoke check — confirm zulip platform loaded:
+
+```bash
+grep -E "zulip.*loaded|zulip.*registered" {{hermes_home}}/logs/gateway.log | tail -3
+```
+
+If gateway fails to restart, check logs for the crash cause before proceeding.
+
+### Step 6: Send Relay Success Signal
+
+On the Abiba host (local), dispatch a relay message to the target agent:
+
+```
+ra-h-os-createRelayNode:
+  title: "Zulip plugin updated — {{target}}"
+  source: |
+    Plugin installed from {{branch}} @ {{INSTALLED_SHA}}
+    All 3 adapter files verified at {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
+    _strip_html fix: PRESENT
+    Timestamp: {{timestamp}}
+
+  description: "hermes-zulip-plugin completed for {{target}} — plugin layer healthy"
+```
+
+### Step 7: Report
+
+Compile results into a single status block:
+
+| Field | Value |
+|-------|-------|
+| Target | `{{target}}` |
+| Branch | `{{branch}}` |
+| Commit SHA | `{{INSTALLED_SHA}}` |
+| Files installed | `__init__.py`, `adapter.py`, `plugin.yaml` |
+| `_strip_html` | `{{present|missing}}` |
+| Gateway restarted | `{{yes|no}}` |
+| Signal sent | `{{yes|no}}` |
+
+## Known Failure Modes
+
+| Symptom | Root Cause | Recovery |
+|---------|-----------|----------|
+| Git clone fails | No network or repo unreachable | Check VPN/network, verify repo URL |
+| Permission denied on copy | Wrong user for target | Use correct user (jerome for Tanko, root for others) |
+| `_strip_html` missing after install | Branch doesn't include commit `55ca15d` | Switch to `feat/zulip-streaming` branch |
+| Plugin files missing after copy | Target directory doesn't exist | Ensure `mkdir -p` ran successfully |
+| Relay signal fails | MCP bridge unreachable | Signal manually via `ra-h-os-createRelayNode` |
+
+## Edge Differences from hermes-zulip-restore
+
+| Concern | hermes-zulip-restore | hermes-zulip-plugin |
+|---------|---------------------|---------------------|
+| Env credential check | ✅ Full ZULIP_* verification | ❌ Out of scope |
+| Gateway restart | ✅ Full restart + state validation | ✅ Targeted restart + smoke check |
+| Live connection test | ✅ Validates `zulip.state = connected` | ❌ Out of scope |
+| Plugin deploy | ✅ Includes deploy as one step | ✅ Primary purpose |
+| Version tracking | ❌ Implicit | ✅ Explicit SHA capture |
+| Signal dispatch | ❌ None | ✅ Relay message to target |
+
+For full connectivity recovery after a plugin install, chain this contract
+with `hermes-zulip-restore` (skip its Step 2 to avoid redundant deploy).
+
+---
+
+**Last updated**: 2026-07-08 — Switched default branch to `master`; added
+gateway restart step. If outstanding unmerged feature branches exist, address
+them in a follow-up merge after this contract completes.

hermes-zulip-restore.prose.md

@@ -0,0 +1,188 @@
+---
+kind: function
+name: hermes-zulip-restore
+description: >
+  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
+  Koby CT111). Deploys the zulip-platform adapter to the correct bundled plugin
+  path, verifies env credentials, restarts the gateway, and confirms Zulip
+  connects. Run this whenever a Hermes agent stops responding on Zulip or after
+  a fresh agent deployment.
+agent: abiba
+version: 1.0.0
+status: active
+runtime_contract: 2
+---
+
+# Hermes Zulip Restore — Bring Any Agent Back to Good State
+
+Single-shot function that restores full Zulip connectivity for a Hermes agent.
+Covers adapter deployment, HTML stripping (slash command fix), env verification,
+gateway restart, and connection validation.
+
+## Parameters
+
+| Param | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
+
+## Maintains
+
+- adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
+- zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
+- env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
+- gateway_running: bool — Whether gateway process is running
+
+### Postconditions
+
+- `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
+- All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
+- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
+- Gateway restarted and zulip platform reports state `connected`
+- HTML stripping enabled for `/approve` and `/deny` slash command support
+
+## Requires
+
+- SSH access to target host (direct or via amdpve for CTs)
+- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
+- Python 3 with `httpx` installed on target
+- Zulip server accessible at `https://chat.sysloggh.net`
+
+## Live-State Fields
+
+| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
+|------|-----|---------|-------------|-------------|------|
+| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
+| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
+
+| Field | Value | Trust |
+|-------|-------|-------|
+| Zulip server | https://chat.sysloggh.net | ✅ Verified |
+| Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
+| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
+| Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
+
+## Execution
+
+### Step 1: Locate Target
+
+Map `target` to connectivity parameters from the live-state table above.
+For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
+
+### Step 2: Deploy Zulip Adapter
+
+On the target host:
+
+```bash
+# Clone or update the plugin repo
+mkdir -p /tmp/zulip-deploy
+cd /tmp/zulip-deploy
+if [ -d zulip-platform-plugins ]; then
+  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
+else
+  git clone --branch feat/zulip-streaming \
+    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
+fi
+
+# Ensure bundled plugin directory exists
+mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
+
+# Copy adapter files
+cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
+   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
+   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
+   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
+
+# Fix ownership (Tanko only)
+chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
+
+# Clean up
+rm -rf /tmp/zulip-deploy
+```
+
+### Step 3: Verify _strip_html is Present
+
+```bash
+grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
+```
+Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
+
+### Step 4: Verify Env Credentials
+
+```bash
+grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
+```
+
+Expected: all three variables set with non-empty values. If any missing:
+- ZULIP_SITE: `https://chat.sysloggh.net`
+- ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
+- ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
+
+### Step 5: Restart Gateway
+
+```bash
+cd <HERMES_HOME>/hermes-agent
+# Use venv if available
+python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
+```
+
+Wait for the restart to complete (up to 45s). Check:
+
+```bash
+grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
+```
+
+Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
+
+### Step 6: Validate Zulip Connection
+
+```bash
+python3 -c "
+import json
+d = json.load(open('$HERMES_HOME/gateway_state.json'))
+print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
+"
+```
+
+Expected: `zulip: connected`. If not connected, check gateway log:
+
+```bash
+grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
+```
+
+### Step 7: Report
+
+Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
+
+| State | Action |
+|-------|--------|
+| All true | ✅ Agent restored — relay success to user |
+| `adapter_deployed: false` | Re-run Step 2 |
+| `env_valid: false` | Prompt for missing credentials |
+| `zulip_connected: false` | Check Zulip server reachability, verify API key |
+| `gateway_running: false` | Check process logs for crash cause |
+
+## Known Failure Modes
+
+| Symptom | Root Cause | Recovery |
+|---------|-----------|----------|
+| Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
+| Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
+| No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
+| `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
+| Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
+| Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
+
+## Git Branch Reference
+
+The `_strip_html` fix lives on `feat/zulip-streaming` branch:
+```
+https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
+```
+
+Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
+Pull request #33 is the primary integration branch.
+
+---
+
+**Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.

infrastructure-control.prose.md

@@ -144,7 +144,7 @@ description: >
 
 ### Ecosystem A: docker-vm (192.168.68.7)
 
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
+| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
+
+**Last verified:** 2026-07-09 — 16/16 containers running healthy.
 
 ### Home Stack Services
 

infrastructure-monitoring.prose.md

@@ -7,10 +7,14 @@ description: >
   from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
   via existing /metrics Prometheus endpoint.
   
-  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
-  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
-  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
-  This contract defines the target state; proxmox-monitor is the as-built.
+  DEPLOYMENT STATUS (2026-07-09):
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
+     NVIDIA sidecar exporters (.8/.110:9400) never installed.
+     Router falls back to direct GPU /health probes.
+  ⚠️ This contract is target-state aspirational — not as-built.
+     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
 

infrastructure-update.prose.md

@@ -24,10 +24,11 @@ version: 1.1.0
 
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
-4. ✅ LiteLLM health check passing
-5. ✅ Zulip server reachable
+2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
+4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+5. ✅ LiteLLM health check passing
+6. ✅ Zulip server reachable
 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

litellm-api-keys.prose.md

@@ -8,6 +8,10 @@ description: >
   Ensures agents never use the master key directly. Rotation is event-driven,
   not calendar-driven — rotate only on compromise, personnel change, or
   periodic security hygiene (quarterly/annually).
+  
+  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
+  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
+  on CT 116. Last verified: 2026-07-09.
 ---
 
 ## Parameters
@@ -38,7 +42,8 @@ description: >
    - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
    - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
    - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
+   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
    - Return the new key
 5. **If action == "rotate"**:
    - Generate new key with same alias (LiteLLM replaces the old key)

litellm-health.prose.md

@@ -1,35 +1,50 @@
 ---
 kind: function
 name: litellm-health
+status: deprecated
+deprecated_on: 2026-07-09
+replaced_by: litellm-self-heal.prose.md
+note: >
+  Consolidated into litellm-self-heal.prose.md to eliminate duplication
+  of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within litellm-self-heal.
+  This file is retained for reference only — use litellm-self-heal instead.
 description: >
-  Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
-  nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
-  Router runs as internal-only backend behind LiteLLM. GPU monitoring via
-  Prometheus/Grafana and fleet dashboard.
+  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
+  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
+  Router (harness-router :9000) is DEPRECATED — container still runs but
+  is not in the request path. GPU monitoring via Prometheus/Grafana and
+  fleet dashboard (gpu-monitor :9100).
   Designed as a reusable contract for any Syslog agent.
+  
+  Source of truth: gpu-fleet.prose.md
 ---
 
-## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 
 ```
-Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
-                        │                    │
-                  Key validation      Model routing
-                  Fallback chains     Slot booking
-                  Complexity router   Circuit breakers
-                  Budget tracking     Redis-backed
-                        │                    │
-                  Prometheus ← metrics  ←───┘
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
+                        │
+                  Key validation
+                  Fallback chains
+                  Budget tracking
+                        │
+                  Prometheus ← metrics
                         │
                     Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
 ```
 
-**What changed (v3.1.0 → v3.2.0)**:
-- Router is BACK — runs internally on :9000 behind LiteLLM
-- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
-- Router :9000 is 127.0.0.1-only, not publicly accessible
-- GPU fleet dashboard: http://192.168.68.24:9100
-- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
+**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
+- Router REMOVED from request path — LiteLLM proxies directly to GPU
+- All GPUs at parallel 2 (was parallel 1)
+- NVIDIA context reduced 256K→128K to free VRAM
+- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
+- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
 
 ## Parameters
 
@@ -55,27 +70,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se
 
 ## GPU Fleet Topology
 
-| Host | IP | Hardware | Models Served | Engine |
-|------|-----|----------|---------------|--------|
-| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
-| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
-| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
+|------|-----|----------|---------------|--------|---------|----------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
 
 ## Model Fallback Chains (LiteLLM)
 
 | Primary | Timeout | Fallback | Timeout |
 |---------|---------|----------|---------|
-| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
-| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
 | ornith-1.0-35b | 120s | qwen → gemma | — |
-| syslog-auto (balanced) | — | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
 
 ## Containers on CT 116
 
 | Container | Image | Port | Health Check |
 |-----------|-------|------|-------------|
 | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
-| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
 | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
 | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
 | harness-redis | redis:7-alpine | :6379 | PING |

litellm-self-heal.prose.md

@@ -6,22 +6,92 @@ note: >
   Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
   This contract is now manual-only — triggers require explicit user request.
   Consider reimplementing as a standalone cron job or prose contract.
+  
+  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
+  duplication of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within this contract.
+  
+  Source of truth for GPU topology and keys: gpu-fleet.prose.md
+  Last verified: 2026-07-09
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
-  knowledge graph for full audit trail.
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
+  inference, and agent keys. Applies remediation rules for common failures.
+  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 
+# LiteLLM Operations — Health Check + Self-Heal
+
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
+                        │
+                  Key validation
+                  Fallback chains
+                  Budget tracking
+                        │
+                  Prometheus ← metrics
+                        │
+                    Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
+```
+
+## Parameters
+
+- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
+- backend_host: string — Internal CT host (default: "192.168.68.116")
+- auth_host: string — Authentik OIDC server (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
+
+## GPU Fleet Topology
+
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
+|------|-----|----------|---------------|--------|---------|----------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
+
+## Model Fallback Chains (LiteLLM)
+
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
+| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
+| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
+
 ## Maintains
 
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
-## Requires
-
-- litellm-health:function
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
+- litellm-agent-keys: { count: int, valid: int }
 
 ## Continuity
 
@@ -29,55 +99,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
 
-## Reporting
+---
 
-Every remediation cycle produces a structured report in three channels:
+## Health Check
 
-### 1. RA-H OS Knowledge Graph Node
-Created as `[LEARN] litellm-self-heal: <run_id>` with:
-- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
-- source: Full JSON report of the cycle
-- description: Summary of what was found, fixed, and escalated
+Run this first on every cycle. Results feed into remediation rules below.
 
-### 2. Zulip DM to Owner
-Sent immediately for:
-- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
-- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
-- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+### 1. Check public endpoints
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
 
-### 3. Daily Digest (end of day)
-A summary of the last 24 hours sent as a single Zulip DM:
-```
-📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
+### 2. Check LiteLLM health (no-auth)
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
 
-Total cycles: 288 (every 5 min)
-Issues found: 3
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
-  └─ Escalated: 0
+### 3. Check backend container health
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
+- Critical: harness-litellm, harness-nginx, harness-postgres
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
+- Deprecated but running: harness-router (not in path, reference only)
 
-Top actions:
-  • harness-nginx restarted — 2026-06-26 02:15
-  • DOCS_URL corrected — 2026-06-26 07:30  
-  • harness-nginx restarted — 2026-06-26 14:46
+### 4. Check GPU fleet health (via fleet dashboard)
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
+- Verify GPUs reporting status "healthy"
+- Check alerts array for active warnings/critical
 
-Uptime: 23h 47m — All healthy now ✅
-```
+### 5. Check model inference via LiteLLM — test each model
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
+- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+- Use master key for auth
 
-### 4. Weekly Digest (every Monday)
-Same format as daily but covering 7 days. Sent to both Zulip DM and
-logged as a `[REPORT]` knowledge graph node for long-term trending.
+### 6. Check agent keys
+- GET /key/list with master key → verify all 6 agents have keys
 
-### 5. Relay Message (if cross-agent)
-If a fix required another agent's help (e.g., Authentik restart), a relay
-message is sent to the responsible agent with full context.
+### 7. Check Grafana
+- GET {{grafana_url}}/api/health → expect 200
 
-## GPU Fleet
+### 8. Compile overall status
+Determine overall_status from individual check results:
+- "healthy" — all checks pass
+- "degraded" — 1-2 non-critical checks fail
+- "down" — critical checks fail
 
-| Host | IP | Hardware | Models | Engine |
-|------|-----|----------|--------|--------|
-| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
-| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
-| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
+---
 
 ## Remediation Rules
 
@@ -105,39 +169,89 @@ Detect → test prompt via LiteLLM returns non-200
 Fix → restart llama-server on GPU host → verify
 Escalate → after 2 failed restarts
 
-### Rule 7: Router Returns 503 (All GPUs Saturated)
-Detect → router logs show QUEUE_TIMEOUT with GPUs idle
-Root cause → roster not loaded at startup (empty TIER_MODELS)
-Fix → POST /admin/roster/reload (if endpoint available) or restart router
-Permanent fix → add load_roster() to router startup code
-Escalate → if reload doesn't resolve within 2 min
+### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
+Router is no longer in the request path (2026-07-08). LiteLLM proxies
+directly to GPU. This rule is retained for reference but is inactive.
+If 503 errors occur, check LiteLLM timeouts and GPU health directly.
 
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
-Root cause → keys in router env var, not in LiteLLM DB
+Root cause → keys not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM
 
-### Rule 9: Stale Active Counter in Redis
-Detect → active:<model> > 0 while GPU idle
-Fix → `redis-cli DEL active:<model>` → restart router
-Prevent → add TTL to active keys in router code
+### Rule 9: Stale Active Counter in Redis — DEPRECATED
+Router no longer in path so Redis active counters are unused. Rule retained
+for reference but inactive. If Redis issues occur, check harness-redis container.
+
+---
+
+## Reporting
+
+Every remediation cycle produces a structured report:
+
+### 1. RA-H OS Knowledge Graph Node
+Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
+
+### 2. Zulip DM to Owner
+- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
+- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+
+### 3. Daily Digest (end of day)
+Summary of last 24 hours: total cycles, issues found/fixed/escalated,
+top actions, uptime.
+
+### 4. Relay Message (if cross-agent)
+If a fix requires another agent (e.g., Authentik restart), relay sent
+to responsible agent with full context.
+
+---
 
 ## Execution
 
-1. Run health check
-2. Apply remediation for each failure
-3. **Generate report** with all actions taken
-4. **Log to knowledge graph** — create `[LEARN]` node
-5. **Notify user** via Zulip DM if anything changed
-6. Wait 300s and repeat
+```prose
+-- Phase 1: Health Check
+let health = call health-check
+  public_url: public_url
+  backend_host: backend_host
+  gpu_dashboard_url: gpu_dashboard_url
+  grafana_url: grafana_url
+
+-- Phase 2: Apply remediation for each failure
+let actions = []
+for check in health.failed:
+  let fix = apply-remediation-rule
+    rule: lookup-rule(check.name)
+    target: check.target
+  push actions fix
+
+-- Phase 3: Generate report
+call report-generator
+  health: health
+  actions: actions
+
+-- Phase 4: Log to knowledge graph
+call kg-logger
+  run_id: run_id
+  health: health
+  actions: actions
+
+-- Phase 5: Notify if anything changed
+if actions.length > 0:
+  call zulip-notifier
+    actions: actions
+    health: health
+
+-- Wait 300s and repeat
+```
 
 ## Audit Trail Format
 
 ```json
 {
-  "run_id": "self-heal-20260626-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "run_id": "self-heal-20260709-001",
+  "timestamp": "2026-07-09T14:00:00Z",
   "duration_ms": 1234,
   "checks_passed": 7,
   "checks_failed": 0,
@@ -151,7 +265,7 @@ Prevent → add TTL to active keys in router code
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
   "issues_found": 1,
   "issues_fixed": 1,
   "actions": [

pm2-self-heal.prose.md

@@ -12,6 +12,7 @@ description: >
 
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
+- gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp
 

scripts/pct-run.sh

@@ -30,6 +30,12 @@ declare -A CT_NODES=(
     # acerpve (192.168.68.9)
     [102]=acerpve   # adguard
     # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
+    #
+    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
+    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
+    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
+    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
+    #   118 jitsi     → stopped, not in service
 )
 
 # Each node must be root-accessible via SSH hostname

zulip-oidc-redirect-fix.prose.md

@@ -0,0 +1,92 @@
+---
+kind: responsibility
+name: zulip-oidc-redirect-fix
+status: active
+description: >
+  Fixes Zulip OIDC authentication when the redirect_uri sent to Authentik
+  uses the internal IP (192.168.68.19) instead of the public domain
+  (chat.sysloggh.net). Applied via monkey-patch in ZULIP_CUSTOM_SETTINGS.
+  Survives container restarts through compose.override.yaml.
+agent: abiba
+triggers:
+  - Zulip OIDC login returns "Redirect URI Error" from Authentik
+  - redirect_uri in OAuth URL shows 192.168.68.19 instead of chat.sysloggh.net
+  - After Zulip server restart, Authentik SSO login broken
+---
+
+## Maintains
+
+- zulip-oidc: { redirect_uri: "https://chat.sysloggh.net/complete/oidc/", scheme: "https", host: "chat.sysloggh.net" }
+- authentik-acceptance: { status: "accepted" | "rejected" }
+- patched-strategy: { module: "social_core.strategy.BaseStrategy", method: "absolute_uri", root: "ROOT_DOMAIN_URI" }
+
+## Detection
+
+### Rule 1: Wrong redirect_uri host
+- **Detect**: `curl -sk -L "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" 2>&1 | grep "redirect_uri=https://192.168.68.19"`
+- **Status**: CRITICAL — Authentik will reject
+- **Trigger fix** → Execute self-heal
+
+### Rule 2: OIDC flow broken
+- **Detect**: `curl -sk -o /dev/null -w "%{http_code}" "https://chat.sysloggh.net/accounts/login/social/oidc/authentik"` → chain ends at Authentik 400
+- **Diagnose**: Check redirect_uri in the 302 Location header chain
+
+## Fix (Self-Heal)
+
+Two layers applied:
+
+### Layer 1: Live patch (inside container, immediate)
+```bash
+# Add to /home/zulip/deployments/current/zproject/computed_settings.py:
+SOCIAL_AUTH_REDIRECT_IS_HTTPS = True
+
+import urllib.parse
+from social_core.strategy import BaseStrategy
+_original_absolute_uri = BaseStrategy.absolute_uri
+def _patched_absolute_uri(self, path=None):
+    from django.conf import settings
+    root = getattr(settings, "ROOT_DOMAIN_URI", "https://chat.sysloggh.net")
+    if path is not None:
+        return urllib.parse.urljoin(root, path)
+    return root
+BaseStrategy.absolute_uri = _patched_absolute_uri
+
+# Restart Django
+supervisorctl restart zulip-django
+```
+
+### Layer 2: Persistent fix (compose.override.yaml)
+The patch is baked into the `ZULIP_CUSTOM_SETTINGS` env var in
+`/opt/zulip/compose.override.yaml`. Survives Docker container restarts.
+
+### Verification
+```bash
+STEP1=$(curl -sk -w "%{redirect_url}" \
+  "https://chat.sysloggh.net/accounts/login/social/oidc/authentik" -o /dev/null)
+curl -sk -D- "$STEP1" -o /dev/null 2>&1 | grep "redirect_uri="
+# Expected: redirect_uri=https://chat.sysloggh.net/complete/oidc/
+# Wrong:    redirect_uri=https://192.168.68.19/complete/oidc/
+```
+
+### Rollback
+Remove the patch block from `compose.override.yaml` and restart the container:
+```bash
+docker compose -f /opt/zulip/compose.yaml -f /opt/zulip/compose.override.yaml up -d zulip
+```
+
+## Root Cause
+
+After Zulip restart, `social-auth-core` computes the OIDC `redirect_uri` via
+Django's `request.build_absolute_uri()` → `request.get_host()`. The upstream
+Netbird/Traefik proxy (72.61.0.17) forwards `Host: 192.168.68.19` instead of
+`Host: chat.sysloggh.net`, and without `HTTP_HOST` in nginx's `uwsgi_params`,
+Django falls back to the server's IP.
+
+The monkey-patch overrides `BaseStrategy.absolute_uri()` to always use
+`ROOT_DOMAIN_URI` (`https://chat.sysloggh.net`) regardless of the request's
+Host header.
+
+## Related Contracts
+
+- `zulip-health.prose.md` — General Zulip health monitoring
+- `zulip-self-heal.prose.md` — RETIRED (pi extension removed)