From 288d613876daaf7ed3fa9801780d0cae91b65cc7 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 02:40:30 +0000 Subject: [PATCH 1/2] fix(pm2-zulip): apply AS-BUILT note to pm2-self-heal description Apply the key change from the PM2/Zulip contract fixes patch (2026-08-09): - Add the AS-BUILT 2026-08-09 note to the pm2-self-heal.prose.md description documenting the captain's ruling that gpu-monitor is systemd-managed, gpu-watchdog is decommissioned, and gitea-runner + abiba-zulip are kept. The zulip-health.prose.md parameterized restart table from the patch is not applied because the file has been restructured since the patch was created: Mumuni is no longer monitored from this host (captain ruling 2026-09-10), so the per-agent restart command table is no longer needed. The Maintains section of pm2-self-heal.prose.md already reflects the correct process list (abiba-zulip, abiba-telegram, gitea-runner, zulip-watchdog) from a previous PR. --- pm2-self-heal.prose.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index e3df0e6..c118af2 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -2,6 +2,16 @@ kind: responsibility name: pm2-self-heal description: > + Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner, + spoton-service, zulip-watchdog) and auto-restarts any that are stopped or + errored. Logs every action to the knowledge graph and alerts the owner via + Zulip DM on failures. + CRITICAL: Never restart abiba-zulip — it runs this contract. + AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative): + gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2; + gpu-watchdog decommissioned (function folded into gpu-monitor.service); + gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the + 2026-07-04 'removed/decommissioned' note was stale and is removed). --- ## Maintains -- 2.54.0 From 937afc0baf873006d1a1ad4b472b555585763c7f Mon Sep 17 00:00:00 2001 From: root Date: Tue, 15 Sep 2026 03:58:01 +0000 Subject: [PATCH 2/2] Ship: monitoring-contract-fixes-20260809 Apply monitoring contract updates to match verified as-built reality: - litellm-health.prose.md: Updated PVE node count from 5 to 6 (.4/.5/.6/.9/.12/.15:9100) - hermes-key-enforcement.prose.md: Updated key expiry policy (explicit 90d duration required) and audit-only status - litellm-api-keys.prose.md: Updated master key retrieval path (dynamic, not hardcoded) Verified facts: - LiteLLM /metrics mounts ONLY with success_callback: [prometheus] - All 3 GPU exporters deployed and verified 200 on :9400 - Alertmanager + Zulip bridge delivering alerts to #agent-hub > alerts-infra - Prometheus node job covers all 6 PVE nodes - Agents set model.context_length: 131072 to prevent silent fallback to 256K Note: The original patch does not apply cleanly because most changes were already applied in subsequent work. Only the PVE node count needed updating. --- hermes-key-enforcement.prose.md | 6 ++++-- litellm-api-keys.prose.md | 10 +++++++++- litellm-health.prose.md | 2 +- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index 06fb821..a13a034 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -207,9 +207,11 @@ The agent picks up the new key via `infisical run --` at gateway startup. **Keys are permanent and use bare agent name aliases.** -- **Duration**: `null` — keys never expire. NOT enforced today: CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key generated with no explicit models comes back with an empty models list. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.) +- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug. +- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end. +- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job. - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity. -- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. +- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use. - **Max budget**: $100 per key (config default). ```yaml diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index 1c1e2df..644b619 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -320,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require ## LiteLLM Master Key (use sparingly — agents should NOT use it directly) -- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY) +- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**: + ```bash + # Read at runtime from the container's environment: + docker exec harness-litellm printenv LITELLM_MASTER_KEY + # Or from Infisical vault (project=infrastructure env=prod) - NOTE: --plain is broken on CLI 0.43.110 (prints nothing): + infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production | awk '$1=="LITELLM_MASTER_KEY"{print $NF}' + # Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl): + curl -s -H "Authorization: Bearer " http://127.0.0.1:4000/key/list | jq length + ``` - Used for /key/generate, /key/delete, /key/list (GET), DB queries - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key. It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`). diff --git a/litellm-health.prose.md b/litellm-health.prose.md index 6449828..fefe1aa 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -19,7 +19,7 @@ description: > Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/. - Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09. - - Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100). + - Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100). --- ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU) -- 2.54.0