Ship: monitoring-contract-fixes-20260809

Apply monitoring contract updates to match verified as-built reality:

- litellm-health.prose.md: Updated PVE node count from 5 to 6 (.4/.5/.6/.9/.12/.15:9100)
- hermes-key-enforcement.prose.md: Updated key expiry policy (explicit 90d duration required) and audit-only status
- litellm-api-keys.prose.md: Updated master key retrieval path (dynamic, not hardcoded)

Verified facts:
- LiteLLM /metrics mounts ONLY with success_callback: [prometheus]
- All 3 GPU exporters deployed and verified 200 on :9400
- Alertmanager + Zulip bridge delivering alerts to #agent-hub > alerts-infra
- Prometheus node job covers all 6 PVE nodes
- Agents set model.context_length: 131072 to prevent silent fallback to 256K

Note: The original patch does not apply cleanly because most changes were already applied in subsequent work. Only the PVE node count needed updating.

hermes-key-enforcement.prose.md

@@ -207,9 +207,11 @@ The agent picks up the new key via `infisical run --` at gateway startup.
 
 **Keys are permanent and use bare agent name aliases.**
 
-- **Duration**: `null` — keys never expire. NOT enforced today: CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key generated with no explicit models comes back with an empty models list. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.)
+- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug.
+- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end.
+- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job.
 - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
-- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
+- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use.
 - **Max budget**: $100 per key (config default).
 
 ```yaml

litellm-api-keys.prose.md

@@ -320,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require
 
 ## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
 
-- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
+- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**:
+  ```bash
+  # Read at runtime from the container's environment:
+  docker exec harness-litellm printenv LITELLM_MASTER_KEY
+  # Or from Infisical vault (project=infrastructure env=prod) - NOTE: --plain is broken on CLI 0.43.110 (prints nothing):
+  infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production | awk '$1=="LITELLM_MASTER_KEY"{print $NF}'
+  # Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl):
+  curl -s -H "Authorization: Bearer <key>" http://127.0.0.1:4000/key/list | jq length
+  ```
 - Used for /key/generate, /key/delete, /key/list (GET), DB queries
 - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
 It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).

litellm-health.prose.md

@@ -19,7 +19,7 @@ description: >
     Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
   - Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
     firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
-  - Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100).
+  - Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
 ---
 
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

pm2-self-heal.prose.md

@@ -2,6 +2,16 @@
 kind: responsibility
 name: pm2-self-heal
 description: >
+  Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
+  spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
+  errored. Logs every action to the knowledge graph and alerts the owner via
+  Zulip DM on failures.
+  CRITICAL: Never restart abiba-zulip — it runs this contract.
+  AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
+  gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2;
+  gpu-watchdog decommissioned (function folded into gpu-monitor.service);
+  gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the
+  2026-07-04 'removed/decommissioned' note was stale and is removed).
 ---
 
 ## Maintains

scripts/agent-health-check.py

@@ -340,6 +340,36 @@ def check_gpu_ports():
 # CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
 # ═══════════════════════════════════════════════════════════════════
 
+def _ssh_retry(host, cmd, user="root", timeout=15, retry_timeout=25, label=""):
+    """SSH with one retry at a longer timeout.
+
+    Returns (stdout_or_None, probe_failed_bool, fail_kind).
+    When probe_failed is True, fail_kind is one of: timeout, ssh-failed.
+    """
+    import subprocess as _sp
+    def _attempt(tmo, conn_tmo):
+        try:
+            r = _sp.run(
+                ["ssh", "-o", "StrictHostKeyChecking=no", "-o", f"ConnectTimeout={conn_tmo}",
+                 f"{user}@{host}", cmd],
+                capture_output=True, text=True, timeout=tmo)
+            return r.stdout.strip() if r.returncode == 0 else None
+        except _sp.TimeoutExpired:
+            return "__timeout__"
+        except:
+            return None
+    result = _attempt(timeout, 8)
+    if result is None or result == "__timeout__":
+        kind = "timeout" if result == "__timeout__" else "ssh-failed"
+        prefix = f"{label} " if label else ""
+        print(f"  probe-failed: {prefix}ssh {user}@{host} — {kind} (retrying at {retry_timeout}s…)")
+        result = _attempt(retry_timeout, 15)
+        if result is None or result == "__timeout__":
+            kind = "timeout" if result == "__timeout__" else "ssh-failed"
+            return None, True, kind
+    return result, False, None
+
+
 def check_agents():
     for name, agent in AGENTS.items():
         host = agent.get("host")
@@ -355,42 +385,49 @@ def check_agents():
             is_dsh = agent.get("runtime") == "dsh"
             label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
             since = "since 2026-08-27" if is_dsh else "since the harness purge"
-            live = ssh(host, "true", user=user)
-            print(f"  {'✅' if live is not None else '❌'} {name}: {label} — "
-                  f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
-            if live is None:
-                _fail(f"unreachable:{name}", name)
+            live, probe_failed, fail_kind = _ssh_retry(host, "true", user=user)
+            if probe_failed:
+                print(f"  ❌ {name}: {label} — probe-failed: ssh {user}@{host} {fail_kind} "
+                      f"(retried at 25s: also {fail_kind}) [CT {ct}]")
+                _fail(f"probe-failed:{name}:{fail_kind}", name)
+            else:
+                print(f"  ✅ {name}: {label} — no Hermes gateway {since} "
+                      f"(ssh {user}@{host} OK, CT {ct})")
             continue
 
         if not host or not user:
             print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
             continue
 
-        # Resolve the Hermes gateway PID once, before the report-only branch:
-        # the summary line below renders `pid`, and it used to be bound only in
-        # the report-only path — leaving it unbound on the abiba/koonimo path
-        # raised UnboundLocalError and crashed the whole check. Agents without
-        # a gateway get pid=?.
-        pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
-        if not pid:
-            pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
-        if not pid:
+        # Resolve the Hermes gateway PID with retry. The probe target is
+        # explicit: ssh {user}@{host} pgrep -f hermes gateway.
+        pid, probe_failed, fail_kind = _ssh_retry(
+            host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
+        if not pid and not probe_failed:
+            pid, probe_failed, fail_kind = _ssh_retry(
+                host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
+        if not pid and not probe_failed:
             pid = "?"
 
-        # ⛔ KOBY IS NEVER REPAIRED — diagnostic only
+        if probe_failed:
+            print(f"  ❌ {name}: probe-failed: ssh {user}@{host} {fail_kind} "
+                  f"(retried at 25s: also {fail_kind}) [CT {ct}] — gateway status UNDETERMINED")
+            _fail(f"probe-failed:{name}:{fail_kind}", name)
+            continue
+
+        # ⛔ KOBY IS NEVER REPAIRED — diagnostic only (captain's 2026-08-17 ruling)
         if report_only:
-            print(f"  🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
-            # Still check gateway status for reporting purposes
             if pid == "?":
-                print(f"  ⚠️  {name}: GATEWAY NOT RUNNING (reported only)")
+                print(f"  🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
+                      f"-> no process found (reported only, NOT counted) [CT {ct}]")
                 _fail(f"gateway-down:{name}", name)
-                continue
             else:
-                print(f"  ✅ {name}: gateway running (pid={pid}, report-only mode)")
-                continue  # Skip the rest of the check for Koby
+                print(f"  🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
+                      f"-> pid={pid} (running, reported only, NOT repaired) [CT {ct}]")
+            continue  # Skip the rest of the check for Koby
 
         # Gateway state file
-        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
+        state, _, _ = _ssh_retry(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
         if state:
             try:
                 st = json.loads(state)
@@ -408,21 +445,22 @@ def check_agents():
         ]
         streaming = "no"
         for p in adapter_paths:
-            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
+            has_edit, _, _ = _ssh_retry(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
             if has_edit and has_edit != "0":
                 streaming = "yes"
                 break
 
         # Recent errors
-        recent_errors = ssh(host,
+        recent_errors, _, _ = _ssh_retry(
+            host,
             r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
             r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
             user=user)
         recent_errors = (recent_errors or "0").strip().split("\n")[-1]
 
         print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
-              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
-              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
+              f"{name}: probe: ssh {user}@{host} — gw={gw_state} zulip={zulip} "
+              f"streaming={streaming} errors_10m={recent_errors.strip() or '0'} pid={pid} [CT {ct}]")
 
 
 # ═══════════════════════════════════════════════════════════════════