Ship: monitoring-contract-fixes-20260809

Apply monitoring contract updates to match verified as-built reality:

- litellm-health.prose.md: Updated PVE node count from 5 to 6 (.4/.5/.6/.9/.12/.15:9100)
- hermes-key-enforcement.prose.md: Updated key expiry policy (explicit 90d duration required) and audit-only status
- litellm-api-keys.prose.md: Updated master key retrieval path (dynamic, not hardcoded)

Verified facts:
- LiteLLM /metrics mounts ONLY with success_callback: [prometheus]
- All 3 GPU exporters deployed and verified 200 on :9400
- Alertmanager + Zulip bridge delivering alerts to #agent-hub > alerts-infra
- Prometheus node job covers all 6 PVE nodes
- Agents set model.context_length: 131072 to prevent silent fallback to 256K

Note: The original patch does not apply cleanly because most changes were already applied in subsequent work. Only the PVE node count needed updating.

hermes-key-enforcement.prose.md

@@ -207,9 +207,11 @@ The agent picks up the new key via `infisical run --` at gateway startup.
 
 **Keys are permanent and use bare agent name aliases.**
 
-- **Duration**: `null` — keys never expire. NOT enforced today: CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key generated with no explicit models comes back with an empty models list. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.)
+- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug.
+- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end.
+- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job.
 - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
-- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
+- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use.
 - **Max budget**: $100 per key (config default).
 
 ```yaml

litellm-api-keys.prose.md

@@ -320,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require
 
 ## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
 
-- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
+- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**:
+  ```bash
+  # Read at runtime from the container's environment:
+  docker exec harness-litellm printenv LITELLM_MASTER_KEY
+  # Or from Infisical vault (project=infrastructure env=prod) - NOTE: --plain is broken on CLI 0.43.110 (prints nothing):
+  infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production | awk '$1=="LITELLM_MASTER_KEY"{print $NF}'
+  # Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl):
+  curl -s -H "Authorization: Bearer <key>" http://127.0.0.1:4000/key/list | jq length
+  ```
 - Used for /key/generate, /key/delete, /key/list (GET), DB queries
 - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
 It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).

litellm-health.prose.md

@@ -19,7 +19,7 @@ description: >
     Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
   - Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
     firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
-  - Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100).
+  - Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
 ---
 
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

pm2-self-heal.prose.md

@@ -2,6 +2,16 @@
 kind: responsibility
 name: pm2-self-heal
 description: >
+  Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
+  spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
+  errored. Logs every action to the knowledge graph and alerts the owner via
+  Zulip DM on failures.
+  CRITICAL: Never restart abiba-zulip — it runs this contract.
+  AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
+  gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2;
+  gpu-watchdog decommissioned (function folded into gpu-monitor.service);
+  gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the
+  2026-07-04 'removed/decommissioned' note was stale and is removed).
 ---
 
 ## Maintains