fix: cron alignment + VM→bare metal drift

- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line

Live actions: gpu-monitor-server added to PM2 (persistent across reboots)

README.md

@@ -115,10 +115,11 @@ Run on trigger or schedule. Maintain persistent world-model state across runs.
 | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. |
 | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. |
 | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. |
-| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. |
+| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) |
 | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. |
 | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. |
 | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. |
+| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). |
 | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. |
 
 ### Instantiable Templates (`kind: pattern` — run with `prose run`)
@@ -146,8 +147,9 @@ Called on-demand as single-render tools.
 
 | Contract | Description |
 |---|---|
-| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. |
+| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. |
-| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. |
+| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. |
+| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. |
 | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. |
 | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. |
 
@@ -158,8 +160,8 @@ Companion shell scripts that contracts delegate to.
 | File | Description |
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
-| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
+| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
 
 ## Contract Structure
 

gpu-fleet.prose.md

@@ -142,7 +142,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 |-------|-----|-----|-----|--------|
 | Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
 | Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
-| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
 | Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
 | Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
 | Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
@@ -159,10 +159,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
 

gpu-monitor.prose.md

@@ -19,7 +19,7 @@ agent: abiba
 
 ```
 ┌─────────────────────────────────────────────────────────┐
-│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  GPU Monitor Server (:9100) — 192.168.68.24              │
 │  Polls every 15s, serves dashboard + JSON API            │
 └──┬──────────┬──────────┬────────────────────────────────┘
    │          │          │
@@ -35,7 +35,7 @@ agent: abiba
 Note: JSON sidecar exporters at :8090 were never deployed on any
 GPU host. Router falls back to GPU /health direct probe. Monitor
 should use router /health/unified as source of truth for GPU status.
-Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
+Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
 poll .15:8080 directly; must go through router on .116.
 ```
 
@@ -118,8 +118,8 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
 
 | File | Host | Purpose |
 |------|------|---------|
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | Monitor server v2.0.0 |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
 
 ## Execution

infrastructure-update.prose.md

@@ -24,10 +24,11 @@ version: 1.1.0
 
 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
-4. ✅ LiteLLM health check passing
+4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
-5. ✅ Zulip server reachable
+5. ✅ LiteLLM health check passing
+6. ✅ Zulip server reachable
 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

litellm-health.prose.md

@@ -1,6 +1,14 @@
 ---
 kind: function
 name: litellm-health
+status: deprecated
+deprecated_on: 2026-07-09
+replaced_by: litellm-self-heal.prose.md
+note: >
+  Consolidated into litellm-self-heal.prose.md to eliminate duplication
+  of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within litellm-self-heal.
+  This file is retained for reference only — use litellm-self-heal instead.
 description: >
   Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
   nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.

litellm-self-heal.prose.md

@@ -7,24 +7,91 @@ note: >
   This contract is now manual-only — triggers require explicit user request.
   Consider reimplementing as a standalone cron job or prose contract.
   
-  Source of truth for GPU topology: gpu-fleet.prose.md
+  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
+  duplication of architecture diagrams, GPU topology, timeout tables, and container
+  lists. Health check is now § Health Check within this contract.
+  
+  Source of truth for GPU topology and keys: gpu-fleet.prose.md
   Last verified: 2026-07-09
 description: >
-  Standing responsibility that monitors LiteLLM health and proactively
+  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  fixes common issues. Reports every action via Zulip DM and RA-H OS
+  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
-  knowledge graph for full audit trail.
+  inference, and agent keys. Applies remediation rules for common failures.
+  Reports every action via Zulip DM and RA-H OS knowledge graph.
 ---
 
+# LiteLLM Operations — Health Check + Self-Heal
+
+## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
+
+```
+Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
+                        │
+                  Key validation
+                  Fallback chains
+                  Budget tracking
+                        │
+                  Prometheus ← metrics
+                        │
+                    Grafana :3001
+
+  harness-router :9000 — DEPRECATED, container still runs but
+  NOT in request path. nginx routes /v1 → LiteLLM directly.
+  Router slot booking + circuit breakers replaced by
+  LiteLLM native fallbacks + timeouts.
+```
+
+## Parameters
+
+- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
+- backend_host: string — Internal CT host (default: "192.168.68.116")
+- auth_host: string — Authentik OIDC server (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
+- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
+- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
+
+## GPU Fleet Topology
+
+| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
+|------|-----|----------|---------------|--------|---------|----------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
+| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
+
+## Model Fallback Chains (LiteLLM)
+
+| Primary | Timeout | Fallback | Timeout |
+|---------|---------|----------|---------|
+| qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
+| gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
+| ornith-1.0-35b | 120s | qwen → gemma | — |
+| syslog-auto (balanced) | 90s | qwen → gemma | — |
+
+> Global: request_timeout=300s, nginx proxy_read_timeout=600s
+
+## Containers on CT 116
+
+| Container | Image | Port | Health Check |
+|-----------|-------|------|-------------|
+| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
+| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
+| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
+| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
+| harness-redis | redis:7-alpine | :6379 | PING |
+| harness-dashboard | inference-harness-dashboard | :3000 | /health |
+| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
+| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
+| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
+| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
+
 ## Maintains
 
 - litellm-admin-ui: { status: "healthy", last_check: timestamp }
 - litellm-api-docs: { status: "healthy", last_check: timestamp }
 - litellm-containers: { status: "healthy", last_check: timestamp }
 - litellm-oidc: { status: "healthy", last_check: timestamp }
-
+- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
-## Requires
+- litellm-agent-keys: { count: int, valid: int }
-
-- litellm-health:function
 
 ## Continuity
 
@@ -32,55 +99,49 @@ description: >
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures
 
-## Reporting
+---
 
-Every remediation cycle produces a structured report in three channels:
+## Health Check
 
-### 1. RA-H OS Knowledge Graph Node
+Run this first on every cycle. Results feed into remediation rules below.
-Created as `[LEARN] litellm-self-heal: <run_id>` with:
-- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
-- source: Full JSON report of the cycle
-- description: Summary of what was found, fixed, and escalated
 
-### 2. Zulip DM to Owner
+### 1. Check public endpoints
-Sent immediately for:
+- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
-- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
-- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
 
-### 3. Daily Digest (end of day)
+### 2. Check LiteLLM health (no-auth)
-A summary of the last 24 hours sent as a single Zulip DM:
+- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
-```
-📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
 
-Total cycles: 288 (every 5 min)
+### 3. Check backend container health
-Issues found: 3
+- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
-  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+- Critical: harness-litellm, harness-nginx, harness-postgres
-  └─ Escalated: 0
+- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
+- Deprecated but running: harness-router (not in path, reference only)
 
-Top actions:
+### 4. Check GPU fleet health (via fleet dashboard)
-  • harness-nginx restarted — 2026-06-26 02:15
+- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
-  • DOCS_URL corrected — 2026-06-26 07:30  
+- Verify GPUs reporting status "healthy"
-  • harness-nginx restarted — 2026-06-26 14:46
+- Check alerts array for active warnings/critical
 
-Uptime: 23h 47m — All healthy now ✅
+### 5. Check model inference via LiteLLM — test each model
-```
+- POST /v1/chat/completions model=gemma-4-12b → expect 200
+- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
+- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
+- Use master key for auth
 
-### 4. Weekly Digest (every Monday)
+### 6. Check agent keys
-Same format as daily but covering 7 days. Sent to both Zulip DM and
+- GET /key/list with master key → verify all 6 agents have keys
-logged as a `[REPORT]` knowledge graph node for long-term trending.
 
-### 5. Relay Message (if cross-agent)
+### 7. Check Grafana
-If a fix required another agent's help (e.g., Authentik restart), a relay
+- GET {{grafana_url}}/api/health → expect 200
-message is sent to the responsible agent with full context.
 
-## GPU Fleet
+### 8. Compile overall status
+Determine overall_status from individual check results:
+- "healthy" — all checks pass
+- "degraded" — 1-2 non-critical checks fail
+- "down" — critical checks fail
 
-| Host | IP | Hardware | Models | Engine | Context |
+---
-|------|-----|----------|--------|--------|---------|
-| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b | llama-server :8080 (Vulkan) | 256K |
-| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 (systemd) | 128K |
-| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 (systemd) | 128K |
 
 ## Remediation Rules
 
@@ -115,30 +176,82 @@ If 503 errors occur, check LiteLLM timeouts and GPU health directly.
 
 ### Rule 8: Agent Keys Invalid (401)
 Detect → agents report auth errors
-Root cause → keys in router env var, not in LiteLLM DB
+Root cause → keys not in LiteLLM DB
 Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
 Escalate → if SSH access unavailable, send Zulip DM
 
-### Rule 9: Stale Active Counter in Redis
+### Rule 9: Stale Active Counter in Redis — DEPRECATED
-Detect → active:<model> > 0 while GPU idle
+Router no longer in path so Redis active counters are unused. Rule retained
-Fix → `redis-cli DEL active:<model>` → restart router
+for reference but inactive. If Redis issues occur, check harness-redis container.
-Prevent → add TTL to active keys in router code
+
+---
+
+## Reporting
+
+Every remediation cycle produces a structured report:
+
+### 1. RA-H OS Knowledge Graph Node
+Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
+
+### 2. Zulip DM to Owner
+- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
+- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+
+### 3. Daily Digest (end of day)
+Summary of last 24 hours: total cycles, issues found/fixed/escalated,
+top actions, uptime.
+
+### 4. Relay Message (if cross-agent)
+If a fix requires another agent (e.g., Authentik restart), relay sent
+to responsible agent with full context.
+
+---
 
 ## Execution
 
-1. Run health check
+```prose
-2. Apply remediation for each failure
+-- Phase 1: Health Check
-3. **Generate report** with all actions taken
+let health = call health-check
-4. **Log to knowledge graph** — create `[LEARN]` node
+  public_url: public_url
-5. **Notify user** via Zulip DM if anything changed
+  backend_host: backend_host
-6. Wait 300s and repeat
+  gpu_dashboard_url: gpu_dashboard_url
+  grafana_url: grafana_url
+
+-- Phase 2: Apply remediation for each failure
+let actions = []
+for check in health.failed:
+  let fix = apply-remediation-rule
+    rule: lookup-rule(check.name)
+    target: check.target
+  push actions fix
+
+-- Phase 3: Generate report
+call report-generator
+  health: health
+  actions: actions
+
+-- Phase 4: Log to knowledge graph
+call kg-logger
+  run_id: run_id
+  health: health
+  actions: actions
+
+-- Phase 5: Notify if anything changed
+if actions.length > 0:
+  call zulip-notifier
+    actions: actions
+    health: health
+
+-- Wait 300s and repeat
+```
 
 ## Audit Trail Format
 
 ```json
 {
-  "run_id": "self-heal-20260626-001",
+  "run_id": "self-heal-20260709-001",
-  "timestamp": "2026-06-26T14:00:00Z",
+  "timestamp": "2026-07-09T14:00:00Z",
   "duration_ms": 1234,
   "checks_passed": 7,
   "checks_failed": 0,
@@ -152,7 +265,7 @@ Prevent → add TTL to active keys in router code
 With failures:
 ```json
 {
-  "run_id": "self-heal-20260626-002",
+  "run_id": "self-heal-20260709-002",
   "issues_found": 1,
   "issues_fixed": 1,
   "actions": [

pm2-self-heal.prose.md

@@ -12,6 +12,7 @@ description: >
 
 - abiba-telegram: { status: "online", uptime: string, restarts: number }
 - gpu-watchdog: { status: "online", uptime: string, restarts: number }
+- gpu-monitor: { status: "online", uptime: string, restarts: number }
 - gitea-runner: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp