fix: correct field name to allowed_mcp_servers

PR #118 finding F1 (low): The deployed LiteLLM on CT 116 uses
allowed_mcp_servers (193 occurrences in installed package), not
bare allowed_mcp. One-word doc fix.

.gitignore

@@ -1 +1,2 @@
 __pycache__/
+state/host-disk-bands.json

agent-health-check.prose.md

@@ -0,0 +1,135 @@
+---
+kind: responsibility
+name: agent-health-check
+description: >
+  Consolidated agent health verification for the LiteLLM + GPU + Zulip +
+  gateway fleet. Wraps scripts/agent-health-check.py (v4). Runs every 4 hours
+  via cron and on-demand via "run contract: agent-health-check". Verifies:
+  LiteLLM key validity, GPU port conflicts, agent Zulip streaming, gateway
+  liveness, CT liveness, gateway log health, config YAML integrity,
+  wrapper/CLI integrity, vault secret non-emptiness. NEVER restarts anything.
+title: Agent Health Check — Consolidated
+version: 1.0.0
+runtime_contract: 2
+agent: abiba
+---
+
+# Agent Health Check
+
+Consolidated health verification for the LiteLLM + GPU + Zulip + gateway fleet.
+Runs every 4 hours (2, 6, 10, 14, 18, 22 UTC at :35) via cron (`35 2,6,10,14,18,22 * * *`) and on-demand. Never restarts anything — detects and reports only.
+
+**Cadence rationale** (2026-08-28 decision): monitoring dispatches moved from hourly to every 4 hours to reduce probe load on GPU hosts while keeping detection latency acceptable (up to 4 hours).
+
+## Requires
+
+- **LiteLLM admin key** for key validation (retrieved from `/root/.pi/agent/env.sh`)
+- **SSH access** to GPU hosts (.8, .110, .15) and agent CTs (.122, .129, .114, .24)
+- **Python 3** for script execution
+- **Network access** to LiteLLM (:4000), GPU exporters (:9400), and gateway endpoints
+
+## Maintains
+
+- last_check: timestamp — When the last full diagnostic ran
+- overall_severity: "healthy" | "degraded" | "critical"
+- liteLLM_keys: map of agent → key validity
+- gpu_ports: map of host → port conflict status
+- agents: map of agent → streaming health + gateway liveness
+- ct_liveness: map of CT → active status
+- config_integrity: map of config file → valid/invalid
+
+## Execution
+
+### check-health
+
+**RUN LIVE, NEVER ECHO — every dispatch must execute the script with real tool
+calls; never repeat a prior report unless a live probe fails.**
+
+```bash
+# Run the consolidated health check script
+python3 /root/scripts/agent-health-check.py --json
+```
+
+**Report format**: Begin every report with the **absolute path the script
+executed from** so a stale-consumer report is distinguishable from a real fault
+at read time. Summarize actual results from each check. Apply the standing probe
+rules: any HTTP status = ALIVE; only 000/timeout/refused = probe-failed.
+
+**Expected output**: JSON with `overall_severity` field. If `healthy`, report
+"Agent health check: OK". If `degraded` or `critical`, report the specific
+failures and their severity.
+
+**Mandatory report legs** (2026-09-17 decision, 1295.msg): Every report line
+MUST include one clause per check leg, in every state: healthy, degraded/warn,
+skipped, or failed. A missing leg must never look the same as a healthy leg.
+Required legs and their templates in every state:
+
+- `LiteLLM keys: 4/4 (tanko, abiba, koby, koonimo) valid`
+  - degraded: `LiteLLM keys: 2/4 (tanko valid; koby invalid; koonimo valid; abiba probe-failed: 192.168.68.116:4000 timeout)`
+  - skipped: `LiteLLM keys: SKIPPED (LiteLLM router unreachable)`
+- `GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy`
+  - skipped: `GPU ports: SKIPPED (no SSH access to GPU hosts)`
+  - degraded/warn: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 degraded: svc=inactive, port owned by 1234; strixhalo healthy)`
+  - failed: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 probe-failed: 192.168.68.110:9400 timeout; strixhalo healthy)`
+  - The GPU leg has six non-healthy states the code can produce:
+    (i) `gpu-unreachable:{host}` — SSH probe failed;
+    (ii) `gpu-no-port:{label}` — SSH worked, port not listening;
+    (iii) `gpu-ghost:{label}:{pid}` — unit inactive, port owned by another pid;
+    (iv) unit not active, MainPID empty or port owned by MainPID — svc inactive;
+    (v) unit active, /health body contains "error" — error response;
+    (vi) unit active, /health body unrecognised — unknown health.
+    In every case the failing host and reason must be named.
+- `CTs: 4/4 running (tanko, abiba, koby, koonimo)`
+  - degraded: `CTs: 3/4 (tanko running; abiba running; koby probe-failed: ssh root@192.168.68.129 timeout; koonimo running)`
+  - skipped: `CTs: SKIPPED (SSH access unavailable)`
+- `Vault secrets: 3/3 present`
+  - degraded: `Vault secrets: 2/3 (tanko present; koby present; koonimo missing)`
+  - skipped: `Vault secrets: SKIPPED (vault not configured)`
+
+The compact form in the summary line is acceptable (e.g. `rtx5070 timeout`) as
+long as the host is identifiable from context; the full `probe-failed: <target>
+<kind>` form is required when a leg reports a failure in the detail section.
+
+### Probe Shape (per standing rules from 1150.msg)
+
+1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the
+   service answered — report the code, never "down". A redirect is not a failure.
+   Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
+2. **A failed probe is never a service verdict.** Print
+   `probe-failed: <target> <kind>` naming the exact URL/host/port and the failure
+   kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only
+   then report.
+3. **Say which probe produced each number.** "Grafana: 000" is unusable;
+   "Grafana http://192.168.68.116:3001/api/health -> connection timeout after 10s
+   (retried at 25s: also timeout)" is actionable.
+
+## Strategies
+
+### When LiteLLM keys are invalid
+Report the specific agent + key name. Do not attempt to fix — credential
+rotation is a separate operation.
+
+### When GPU port conflicts are detected
+Report the conflicting ports and processes. Do not kill processes — that's a
+destructive action requiring captain approval.
+
+### When gateway liveness is degraded
+Report the specific CT + gateway status. Do not restart unless the restart
+debounce window has passed.
+
+### When CT liveness is down
+Report the specific CT. Do not restart — that's a destructive action.
+
+### When config YAML is invalid
+Report the specific file + parse error. Do not fix — that's a config change.
+
+### When gateway log health is degraded
+Report the specific gateway + log health status (error patterns, stale connections, connectivity issues). Do not restart — that's a destructive action.
+
+Note: the script may perform additional diagnostics beyond the seven contract checks listed under Execution.
+
+## Continuity
+
+- **Every 4 hours** (2, 6, 10, 14, 18, 22 UTC at :35): Scheduled cron check while Abiba is running
+- **On `agent-health` command**: Run on-demand and report to user
+- **On critical alert**: Escalate to relay message immediately

agent-zero-fix-summary.md

@@ -16,8 +16,8 @@ litellm.exceptions.AuthenticationError: OpenrouterException -
 ```
 **Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.
 
-**Old Key**: `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`  
-**New Key**: `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`  
+**Old Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
+**New Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
 **New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
 
 ### 2. Telegram Bot Conflict (CRITICAL)
@@ -48,14 +48,14 @@ McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 sec
 ```bash
 # Container .env update
 sudo docker exec agent-zero bash -c '
-sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=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|" /a0/usr/.env
+sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»|" /a0/usr/.env
 '
 ```
 
 **Verification**:
 ```bash
 curl -s https://openrouter.ai/api/v1/auth/key \
-  -H "Authorization: Bearer sk-or-v1-0af3f3..." | python3 -m json.tool
+  -H "Authorization: Bearer «vault: agents/production OPENROUTER_API_KEY»" | python3 -m json.tool
 ```
 Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.
 
@@ -140,7 +140,7 @@ Added section:
 
 | Component | Status | Details |
 |-----------|--------|---------|
-| **OpenRouter Key** | ✅ Valid | `sk-or-v1-0af3f3…`, user verified |
+| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY»` user verified, |
 | **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
 | **MCP Services** | ✅ Working | No timeouts after key fix |
 | **Container** | ✅ Running | PID 3320, uptime 16+ hours |

agent-zero-openrouter-key.prose.md

@@ -54,7 +54,7 @@ description: >
    ```
 
 4. **Return status**
-   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-0af", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
+   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-synthetic...", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
    - If OpenRouter returns 401: `{ key_status: "invalid", detail: "User not found" }`
    - If vault secret is missing: `{ vault_synced: false }`
 
@@ -89,8 +89,8 @@ description: >
 
 | Field | Value |
 |-------|-------|
-| **Key Prefix** | `sk-or-v1-0af3f3` |
-| **Full Key** | `«redacted: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»` (in vault + /a0/usr/.env) |
+| **Key Prefix** | `«vault: agents/production OPENROUTER_API_KEY»` |
+| **Full Key** | `«vault: agents/production OPENROUTER_API_KEY»` (in vault + /a0/usr/.env) |
 | **OpenRouter User** | `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` |
 | **Free Tier** | No |
 | **Monthly Usage** | 0 (as of 2026-09-01) |
@@ -101,7 +101,7 @@ description: >
 
 | Date | Action | Notes |
 |------|--------|-------|
-| 2026-09-01 | fix-401 | Old key `sk-or-v1-036e5ca5…` returned 401 "User not found". Replaced with new key `sk-or-v1-0af3f3…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
+| 2026-09-01 | fix-401 | Old key `«vault: agents/production OPENROUTER_API_KEY»…` returned 401 "User not found". Replaced with new key `«vault: agents/production OPENROUTER_API_KEY»…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
 
 ## Infrastructure References
 

audit-hermes-config.py

@@ -262,6 +262,44 @@ def audit(path):
                 f"{field_path} = {value!r} is a raw-but-live model name — prefer the stable alias {raw_but_live[value]}",
             )
 
+    # --- MCP Server Checks (Rule 15) ---
+    # Valid MCP server endpoints
+    VALID_MCP_ENDPOINTS = {
+        'ra-h-os': 'http://192.168.68.65:3100/mcp',
+        'litellm': 'https://litellm.sysloggh.net/mcp',
+    }
+    
+    # Check MCP servers if they exist
+    mcp_servers = cfg.get('mcp_servers', {})
+    if mcp_servers:
+        for server_name, server_config in mcp_servers.items():
+            url = server_config.get('url', '')
+            
+            # Check endpoint validity
+            if server_name in VALID_MCP_ENDPOINTS:
+                expected = VALID_MCP_ENDPOINTS[server_name]
+                if url == expected:
+                    check(True, 'Rule 15', f'MCP server "{server_name}" URL is correct: {url}')
+                else:
+                    check(False, 'Rule 15', f'MCP server "{server_name}" URL is incorrect: {url} (expected: {expected})')
+            else:
+                warn('Rule 15', f'MCP server "{server_name}" URL may need validation (not in known list): {url}')
+            
+            # Check for proper authentication
+            headers = server_config.get('headers', {})
+            has_auth = False
+            for key, value in headers.items():
+                if 'key' in key.lower() or 'auth' in key.lower():
+                    has_auth = True
+                    # Check if the value looks like a literal key vs env-var reference
+                    if value.startswith('Bearer ') and value[7:].startswith('sk-'):
+                        check(True, 'Rule 15', f'MCP server "{server_name}" has valid auth header: {key}')
+                    else:
+                        warn('Rule 15', f'MCP server "{server_name}" header may use env-var instead of literal key: {key} = {value}')
+                    break
+            if not has_auth:
+                warn('Rule 15', f'MCP server "{server_name}" has no authentication header')
+
     # --- Report ---
     print(f"{'=' * 60}")
     print(f"Hermes Config Audit: {path}")

disk-gc-threat-response.prose.md

@@ -43,7 +43,7 @@ Docker hosts get special attention:
 
 > **Note:** CT 118 is now jdownloader (active on storepve). CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 
-## Threat Levels
+## Threat Levels (GUEST filesystems)
 
 | Level | Threshold | Response | Escalation |
 |-------|-----------|----------|------------|
@@ -53,6 +53,39 @@ Docker hosts get special attention:
 | **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
 | **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
 
+## Host Filesystem Thresholds (HOST nodes — separate bands from guest bands)
+
+Host filesystems have their own risk profile and their own bands. A host root near full is a real risk (backup staging writes to it, thin-pool metadata pressure), a media volume near full is a capacity decision for the owner, and the backup datastore near full breaks backups. These are **report-only** — no automatic deletion of media or datastore content ever.
+
+| Level | Threshold | Response | Escalation |
+|-------|-----------|----------|------------|
+| **HOST-WARN** | 85% | Name the volume + % + absolute free space in the scan output | None |
+| **HOST-AMBER** | 90% | Name the volume + % + absolute free space; flag for owner attention | Zulip DM to owner (state-change only) |
+| **HOST-RED** | 95% | Name the volume + % + absolute free space; flag for immediate owner attention | Zulip DM + channel alert (state-change only) |
+
+**Escalations are STATE-CHANGE driven, not per-run.** A volume alerts ONCE when it enters a higher band (GREEN->WARN, WARN->AMBER, AMBER->RED) and ONCE when it drops back down (a recovery notice). While a volume stays in the same band, it is reported in the scan output only — no DM, no channel alert. This prevents the same 96% easystore2 from re-DMing the owner on every 6-hour scan and drowning a real warning in noise.
+
+**State lives in a small JSON state file:** the scanner resolves it to an absolute path from the script's location — `$(dirname "$0")/state/host-disk-bands.json` (i.e., the `state/` directory next to the `scripts/` directory in the prose-contracts repo). The state file is gitignored runtime state — the scanner creates it on first run. Keyed by `host/volume` → last-seen band. The scanner reads the prior band, compares to the current band, and DMs only on a transition. The state file is written after every scan. (Chosen over a periodic digest because the scan already runs every 6h and a transition is genuinely new, actionable state that warrants an immediate DM — but only once.)
+
+**Volume naming rule:** Every host line MUST name the volume and what lives on it. Example output:
+
+```
+storepve /media/easystore2 (media): 96% (3.5T/3.7T, 177G free) -> HOST-RED
+storepve /media/reanim (media): 86% (797G/932G, 135G free) -> HOST-WARN
+storepve /media/mediastore (media): 77% (5.3T/7.3T, 1.6T free) -> GREEN
+storepve /dev/mapper/pve-root (host-root): 81% (73G/94G, 17G free) -> GREEN
+storepve tank (pbs-datastore): 1% (128K/12T, 12T free) -> GREEN
+```
+
+**First-run behavior:** When the state file does not yet exist (first scan), the current band of every volume is recorded as the baseline WITHOUT alerting — a first run would otherwise DM every already-elevated volume at once. From the second run onward, transitions alert.
+
+**Action classes by volume type:**
+- **host-root**: near full = real risk (backup staging, thin-pool metadata, journald). HOST-AMBER or above → owner must investigate.
+- **media** (/media/*): near full = capacity decision for the owner. HOST-AMBER or above → report only, never auto-delete.
+- **pbs-datastore** (tank, ZFS): near full = breaks Proxmox Backup Server. HOST-RED → escalate immediately.
+
+**Justification (measured 2026-09-15, firstmate):** storepve (192.168.68.6) shows /media/easystore2 at 96%, /media/reanim at 86%, /dev/mapper/pve-root at 81%, and tank at 1%. The previous scan reported "0/19 guests >80%, no GC action needed" while easystore2 sat at 96% — the host filesystems were printed but never banded and never acted on. Two incidents this weekend (amdpve host root filled during container backup staging, acerpve root went read-only when its thin pool errored) showed the host filesystem is the thing that breaks, not the guest's.
+
 ## Requires
 
 - SSH access to all Docker hosts (matrix from infrastructure-control pattern)
@@ -128,6 +161,10 @@ from the `report_only_guests` YAML block above.
 
 ## Execution
 
+### Host filesystems: report-only, NEVER auto-delete
+
+Host filesystems are always report-only — no automatic deletion of media or datastore content. A host root near full requires investigation by the owner, but the executor must never delete content on a host filesystem. This restriction is absolute.
+
 ### Hard gate: report-only guests (READ THIS BEFORE RUNNING GC)
 
 **CT 111 / hostname `tdunna` / 192.168.68.129 is DETECT-AND-REPORT-ONLY.** It belongs to Theo.
@@ -243,6 +280,12 @@ done
 
 ## Alert Templates
 
+### HOST-WARN / HOST-AMBER / HOST-RED (host filesystems, report-only)
+```
+⚠️ Host Disk — {hostname} {volume} ({volume_type}): {pct}% ({used}/{total}, {free} free) -> {level}
+Action: {volume_type-specific action}
+```
+
 ### AMBER (75-84%)
 ```
 ⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)

hermes-config-template.prose.md

@@ -5,7 +5,8 @@ description: >
   Standard Hermes configuration template for Syslog Solution LLC agents.
   Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
   RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident.
+  UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation)
+  to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident.
   Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
   2026-07-16 Mumuni root-cause investigation (WAL #1300).
   UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
@@ -145,6 +146,12 @@ mcp_servers:
     url: http://192.168.68.65:3100/mcp
     timeout: 120
     connect_timeout: 60
+  litellm:
+    url: https://litellm.sysloggh.net/mcp
+    headers:
+      x-litellm-api-key: "Bearer <AGENT_KEY>"   # Rule 15: must be a REAL key (sk-...), not an env-var name
+    # Note: MCP endpoint requires Accept: application/json, text/event-stream header
+    # This is handled by the MCP client library; don't add to config
 
 # ─── Compression ───
 compression:
@@ -217,6 +224,40 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes):
 3. **After update**: Restart Hermes on the agent host
 4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
 
+## MCP Server Configuration
+
+MCP server entries in `mcp_servers:` must follow the format shown in the Template section.
+
+**Header requirements (Rule 15):**
+- Use `headers:` field with a `x-litellm-api-key` entry
+- The value must be `"Bearer <REAL_KEY>"` where `<REAL_KEY>` is a literal LiteLLM virtual key
+- Do NOT use env-var references like `$LITELLM_API_KEY` — they resolve to empty strings in
+  the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident)
+
+**Key source:**
+- Keys are stored in the Infisical vault (project=agents, env=production)
+- For template-based config generation: substitute the agent's key from the agent_keys table
+- For manual config updates: retrieve the key from the vault and insert the literal value
+
+**Verification (2026-08-07):**
+- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with agent virtual key
+- Confirmed: 200 response with `serverInfo.name: "litellm-mcp-server"`
+- Confirmed: tools/list returns 200 (MCP endpoint accessible with virtual keys)
+- Note: Per-key MCP grants are now supported (verified 2026-09-18), resolving the earlier
+  contradiction with infrastructure-update.prose.md (which now reflects the update)
+- Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models)
+
+**Key rotation note:**
+- MCP headers use literal keys (not env-vars), so they do NOT auto-rotate with the vault
+- After key rotation, MCP server headers must be regenerated with the new key value
+- This is a manual step: update the `x-litellm-api-key` header in each config file
+- TODO: Consider adding MCP header regeneration to the Key Update Procedure or a generation hook
+
+**NetBird dependency:**
+- `litellm.sysloggh.net` is a NetBird endpoint (see Rule 5 and Infrastructure Stack table)
+- NetBird outages cause 502 errors on MCP requests, not auth failures
+- Diagnose: if MCP requests fail with 502, check NetBird status before investigating keys
+
 ## Violation Classification
 
 When reporting findings, separate POLICY observations from FAULT findings:
@@ -455,14 +496,28 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
 - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
   before and after any config change to catch this and all other rule violations.
 
-### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07)
-- Every MCP server entry must point at the correct endpoint:
-  - ra-h-os = http://192.168.68.65:3100/mcp
-  - litellm = https://litellm.sysloggh.net/mcp
-- MCP entries must carry a REAL key value in the header.
-  - Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP
-    endpoints and result in "Malformed API Key" floods.
-  - Ensure the header value is the actual key (e.g., `sk-...`).
+### Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07)
+
+**Endpoint validation:**
+- ra-h-os must point to `http://192.168.68.65:3100/mcp`
+- litellm must point to `https://litellm.sysloggh.net/mcp`
+- Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had
+  ra-h-os pointing to litellm's endpoint)
+
+**Header validation:**
+- Every MCP entry with authentication must carry a `headers:` field
+- The header value must be a REAL key (e.g., `Bearer sk-abc123...`), NOT an env-var name
+- Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause
+  "Malformed API Key" floods (401 errors in agent gateway logs)
+- Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models)
+- Verify MCP access: test the MCP initialize handshake against the MCP endpoint (not just /v1/models)
+  ```bash
+  curl -s -X POST -H "x-litellm-api-key: Bearer <KEY>" -H "Accept: application/json, text/event-stream" \
+    https://litellm.sysloggh.net/mcp -d '{"jsonrpc":"2.0","id":1,"method":"initialize",...}' \
+    | jq '.data.result.serverInfo'  # should show serverInfo.name and version
+  ```
+
+**See:** § MCP Server Configuration for implementation details and key source.
 
 ## Execution
 

hermes-key-enforcement.prose.md

@@ -101,7 +101,7 @@ auxiliary:
 fallback_providers:
   - provider: deepseek
     base_url: https://api.deepseek.com
-    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key: sk-synthetic-external-example   # ← hardcoded OK (external, synthetic example)
     api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```
 
@@ -109,7 +109,7 @@ fallback_providers:
 # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
 model:
   provider: harness
-  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
+  api_key: sk-synthetic-example-12345   # ← RULE VIOLATION: hardcoded key (synthetic example)
 
 model:
   provider: harness
@@ -139,6 +139,16 @@ scripts/hermes-reachability-check.sh <host> "api_key: sk-" "/root/.hermes/"
 
 When reporting findings, separate POLICY observations from FAULT findings:
 
+### ACCEPTABLE PATTERN
+Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's shape is the canonical example). A plaintext key inside a `config.yaml` or any `config.yaml.bak-*` file is a violation — the backup files are not part of the runtime credential path and are not watched by the scanner, so a key in them is stale clutter that a future reader can mistake for a working key.
+
+**Fix procedure** (when a backup file is found with a plaintext key):
+1. Move the file out of the scanned tree (e.g., `mv /root/.hermes/config.yaml.bak-* /root/hermes-config-backups/`) — do NOT delete the file, just move it so the scanner pattern no longer matches.
+2. Re-run the reachability check to confirm COMPLIANT.
+3. Report the before/after check output and the commands you ran.
+
+**Rationale**: Moving the file preserves history without leaving a credential where a scanner trips over it. Deleting the file loses the historical context. Keeping it in place means the next scan will report it as a finding and waste time re-deciding.
+
 ### POLICY (observation only, not a fault)
 - Agent uses a non-internal-harness provider (e.g., direct DeepSeek, Tencent, OpenRouter)
 - Config text has a field that looks unusual but the agent's calls are succeeding
@@ -172,7 +182,7 @@ grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
 
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
 grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null
 
 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \

infrastructure-control.prose.md

@@ -181,8 +181,8 @@ description: >
 **Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
 - URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
 - Swagger: `http://192.168.68.7:8989/swagger-ui.html`
-- Admin credentials: `admin` / `kakashi20stirling`
-- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
+- Admin credentials: `«vault: infrastructure/production STIRLING_ADMIN_USER»` / `«vault: infrastructure/production STIRLING_ADMIN_PASSWORD»`
+- API key: `«vault: infrastructure/production STIRLING_API_KEY»`
 - Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
 - Compose: `/opt/home_stack/docker-compose.yml`
 - Control script: `/opt/home_stack/infra-control.sh`
@@ -636,7 +636,7 @@ monitor, or integration breaks.
 ```bash
 # Full cluster status
 PVE="https://minipve.sysloggh.net"
-AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
+AUTH="Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"
 curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"
 
 # Docker health from Abiba

infrastructure-update.prose.md

@@ -208,19 +208,19 @@ mcp_servers:
 | Key | MCP Access |
 |-----|-----------|
 | Master key | ✅ Full — 90 tools (vault-injected) |
-| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.99.1 |
+| Agent keys (mumuni, tanko, etc.) | ✅ Per-key grants supported (as of 2026-09-18 verification) |
 
 ### Known Limitations
-- Per-key MCP server grants not functional — only master key has access
+- ~~Per-key MCP server grants not functional — only master key has access~~ (resolved 2026-09-18: per-key grants now work)
 - Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
 - HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
 - `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
 
-### Migration Path
-When LiteLLM is upgraded to a version supporting per-key MCP grants:
-1. Grant agent keys `mcp_servers: ["ra_h_os"]`
-2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
-3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
+### Migration Path (COMPLETED 2026-09-18)
+Per-key MCP grants are now supported:
+1. ✅ Agent keys granted MCP access via `allowed_mcp_servers` field
+2. ✅ Hermes `mcp_servers.litellm.url` set to `https://litellm.sysloggh.net/mcp`
+3. ✅ `headers: {x-litellm-api-key: "Bearer <literal_key>"}` added to MCP config
 
 ## Security-Specific Updates
 

litellm-api-keys.prose.md

@@ -144,8 +144,8 @@ through its agent wrapper.
    safety net for vault outage or token revocation. Must be kept in sync on rotation.
    Example:
    ```bash
-   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
-   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
+   MUMUNI_LITELLM_API_KEY=«vault: agents/production LITELLM_API_KEY»
+   MUMUNI_ZULIP_API_KEY=«vault: agents/production ZULIP_API_KEY»
    ```
 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
    ```ini
@@ -191,7 +191,7 @@ through its agent wrapper.
 ### Tanko migration (COMPLETED 2026-07-17)
 
 Tanko was the last agent migrated from hardcoded keys to vault wrapper.
-Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
+Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-synthetic-tanko-example…`)
 and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
 `50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
 `~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
@@ -271,13 +271,13 @@ not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update ma
 UI bootstrap, model selection) is built around OpenRouter's native authentication.
 
 **Key Storage:**
-- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=sk-or-v1-…`)
+- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»…`)
 - **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production)
 - **Fallback**: The container's .env is the primary source; vault sync is optional
   (unlike fleet agents which require vault injection)
 
 **Current Key (2026-09-01):**
-- **Prefix**: `sk-or-v1-0af3f3…`
+- **Prefix**: `«vault: agents/production OPENROUTER_API_KEY»`
 - **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
 - **Plan**: Paid (not free tier)
 - **Usage**: 0 (as of 2026-09-01)

litellm-self-heal.prose.md

@@ -116,7 +116,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-
 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
 - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). The current fleet roster is owned by the script changelog (`scripts/agent-health-check.py`); mumuni is no longer probed from this host. Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
-- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
+- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (hardcoded key → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM (deprecated key, no live usage).
 
 ## Maintains
 

scripts/agent-health-check.py

@@ -122,10 +122,8 @@ def _fail(key, agent_name=None):
 
 
 INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
-INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
-
-# Fallback: if no env token, read the shared vault token file
 if not INFISICAL_TOKEN:
+    # Fallback: read the shared vault token file
     _token_path = os.path.expanduser("~/.infisical-token")
     if os.path.isfile(_token_path):
         try:
@@ -133,6 +131,7 @@ if not INFISICAL_TOKEN:
                 INFISICAL_TOKEN = _f.read().strip()
         except (OSError, UnicodeDecodeError):
             pass
+INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
 
 # ── Helpers ──────────────────────────────────────────────────────────
 

scripts/daily-infra-report.py

@@ -16,14 +16,16 @@ from email.mime.text import MIMEText
 from email.mime.multipart import MIMEMultipart
 
 PVE = "https://192.168.68.12:8006"
-AUTH = "Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
+AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"
 
 # ── Shared credentials —─
 
 ZULIP_SITE = "https://chat.sysloggh.net"
 ZULIP_EMAIL = "abiba-bot@chat.sysloggh.net"
-ZULIP_KEY = "cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
-ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_KEY}"
+ZULIP_API_KEY = os.environ.get("ZULIP_API_KEY", "")
+if not ZULIP_API_KEY:
+    raise SystemExit("ZULIP_API_KEY not set — refusing to run with no credential")
+ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_API_KEY}"
 
 LITELLM_PUBLIC = "https://litellm.sysloggh.net"
 LITELLM_BACKEND = "192.168.68.116"
@@ -669,7 +671,10 @@ def send_email(html_content, subject_prefix=""):
     msg.attach(MIMEText(html_content, "html"))
     
     try:
-        EMAIL_PASSWORD = "rgbuomwcydxwbszd"
+        EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
+        if not EMAIL_PASSWORD:
+            print("EMAIL_PASSWORD not set — refusing to send email", file=sys.stderr)
+            sys.exit(1)
         GMAIL_EMAIL = "jtabiri@gmail.com"
         
         server = smtplib.SMTP("smtp.gmail.com", 587)

scripts/disk-gc-scan.py

@@ -153,6 +153,25 @@ GPU_HOSTS = [
 CONNECT_TIMEOUT = 5
 SSH_OPTS = "-o BatchMode=yes -o ConnectTimeout=" + str(CONNECT_TIMEOUT)
 
+# Host filesystem thresholds (from contract)
+HOST_THRESHOLDS = {
+    "WARN": 85,
+    "AMBER": 90,
+    "RED": 95,
+}
+
+# State file path (absolute, so execution context doesn't matter)
+STATE_FILE = pathlib.Path(__file__).resolve().parent.parent / "state" / "host-disk-bands.json"
+
+# PVE nodes to probe for host filesystems
+HOST_NODES = [
+    {"hostname": "acerpve", "ip": "192.168.68.9"},
+    {"hostname": "amdpve", "ip": "192.168.68.15"},
+    {"hostname": "storepve", "ip": "192.168.68.6"},
+    {"hostname": "minipve", "ip": "192.168.68.12"},
+    {"hostname": "ocupve", "ip": "192.168.68.5"},
+]
+
 
 def run_cmd(cmd: str, timeout: int = 30) -> tuple[int, str, str]:
     """Run a command and return (exit_code, stdout, stderr)."""
@@ -298,6 +317,151 @@ def scan_fleet() -> list[dict]:
     return results
 
 
+def classify_band(usage_pct: float) -> str:
+    """Classify a percentage into a band."""
+    if usage_pct >= HOST_THRESHOLDS["RED"]:
+        return "HOST-RED"
+    elif usage_pct >= HOST_THRESHOLDS["AMBER"]:
+        return "HOST-AMBER"
+    elif usage_pct >= HOST_THRESHOLDS["WARN"]:
+        return "HOST-WARN"
+    else:
+        return "GREEN"
+
+
+def probe_host_filesystems() -> tuple[list[dict], dict[str, str]]:
+    """Probe host filesystems on all PVE nodes.
+    
+    Returns:
+        - List of host filesystem results
+        - Dict of volume_key -> current_band (for state file)
+    """
+    results = []
+    current_bands = {}
+    
+    for node in HOST_NODES:
+        ip = node["ip"]
+        hostname = node["hostname"]
+        
+        # Probe df for host filesystems
+        probe_cmd = f'ssh {SSH_OPTS} root@{ip} "df -hP / /media/* tank 2>/dev/null | tail -n +2"'
+        exit_code, stdout, stderr = run_cmd(probe_cmd, timeout=15)
+        
+        if exit_code != 0:
+            results.append({
+                "target": f"{hostname} ({ip})",
+                "hostname": hostname,
+                "ip": ip,
+                "reachable": False,
+                "volumes": [],
+                "probe_cmd": probe_cmd,
+                "failure_kind": "ssh-error",
+            })
+            continue
+        
+        # Parse df output and classify each volume
+        volumes = []
+        for line in stdout.splitlines():
+            parts = line.split()
+            if len(parts) < 6:
+                continue
+            
+            dev, size, used, avail, pct_str, mount = parts[:6]
+            pct = float(pct_str.rstrip("%"))
+            band = classify_band(pct)
+            
+            # Volume type classification
+            if mount.startswith("/media/"):
+                vol_type = "media"
+            elif mount == "/" or "pve" in dev:
+                vol_type = "host-root"
+            elif mount == "tank" or "tank" in mount:
+                vol_type = "pbs-datastore"
+            else:
+                vol_type = "other"
+            
+            # Volume key for state file (host/volume)
+            volume_key = f"{hostname}/{mount}"
+            current_bands[volume_key] = band
+            
+            volumes.append({
+                "mount": mount,
+                "device": dev,
+                "size": size,
+                "used": used,
+                "avail": avail,
+                "pct": pct,
+                "band": band,
+                "type": vol_type,
+            })
+        
+        results.append({
+            "target": f"{hostname} ({ip})",
+            "hostname": hostname,
+            "ip": ip,
+            "reachable": True,
+            "volumes": volumes,
+            "probe_cmd": probe_cmd,
+            "failure_kind": None,
+        })
+    
+    return results, current_bands
+
+
+def read_state_file() -> Optional[dict[str, str]]:
+    """Read the state file if it exists."""
+    if not STATE_FILE.exists():
+        return None
+    try:
+        with open(STATE_FILE) as f:
+            return json.load(f)
+    except (json.JSONDecodeError, IOError) as e:
+        print(f"⚠️  State file exists but unreadable: {e}", file=sys.stderr)
+        return {}
+
+
+def write_state_file(bands: dict[str, str]) -> None:
+    """Write the state file."""
+    STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
+    try:
+        with open(STATE_FILE, "w") as f:
+            json.dump(bands, f, indent=2)
+    except IOError as e:
+        print(f"⚠️  State file write failed: {e}", file=sys.stderr)
+
+
+def detect_transitions(current_bands: dict[str, str], prior_bands: Optional[dict[str, str]]) -> list[dict]:
+    """Detect band transitions (current vs. prior)."""
+    if prior_bands is None:
+        # First run — no transitions, just establish baseline
+        return []
+    
+    transitions = []
+    # Check for volumes that moved to a higher band (escalation)
+    for volume, current_band in current_bands.items():
+        prior_band = prior_bands.get(volume, "GREEN")
+        
+        # Band ordering: GREEN < HOST-WARN < HOST-AMBER < HOST-RED
+        band_order = {"GREEN": 0, "HOST-WARN": 1, "HOST-AMBER": 2, "HOST-RED": 3}
+        
+        if band_order[current_band] > band_order[prior_band]:
+            transitions.append({
+                "type": "escalation",
+                "volume": volume,
+                "from": prior_band,
+                "to": current_band,
+            })
+        elif band_order[current_band] < band_order[prior_band]:
+            transitions.append({
+                "type": "recovery",
+                "volume": volume,
+                "from": prior_band,
+                "to": current_band,
+            })
+    
+    return transitions
+
+
 def render_results(results: list[dict]) -> str:
     """Render scan results in human-readable format."""
     lines = []
@@ -316,22 +480,88 @@ def render_results(results: list[dict]) -> str:
     return "\n".join(lines)
 
 
+def render_host_results(results: list[dict], transitions: list[dict], prior_bands: Optional[dict[str, str]]) -> str:
+    """Render host filesystem results in human-readable format."""
+    lines = []
+    lines.append("")
+    lines.append("=== Host Filesystem Bands ===")
+    lines.append("")
+    
+    # Render transitions first (they're the actionable alerts)
+    if prior_bands is None:
+        lines.append("  (first run — recording baseline, no alerts)")
+    elif not transitions:
+        lines.append("  (no band changes since last scan)")
+    else:
+        for t in transitions:
+            volume, from_band, to_band = t["volume"], t["from"], t["to"]
+            if t["type"] == "escalation":
+                lines.append(f"  ⚠️  {volume}: {from_band} → {to_band} (ESCALATION)")
+            else:
+                lines.append(f"  ✅ {volume}: {from_band} → {to_band} (RECOVERY)")
+    
+    # Render all volumes with their bands
+    lines.append("")
+    for node_result in results:
+        if not node_result["reachable"]:
+            lines.append(f"  ❌ {node_result['target']}: UNREACHABLE ({node_result['failure_kind']})")
+            continue
+        
+        lines.append(f"  {node_result['target']}:")
+        for vol in node_result["volumes"]:
+            lines.append(f"    {vol['mount']} ({vol['type']}): {vol['pct']}% ({vol['used']}/{vol['size']}, {vol['avail']} free) -> {vol['band']}")
+    
+    return "\n".join(lines)
+
+
 def main() -> int:
     import argparse
 
-    ap = argparse.ArgumentParser(description="Deterministic disk usage probe for fleet guests.")
+    ap = argparse.ArgumentParser(description="Deterministic disk usage probe for fleet guests and host filesystems.")
     ap.add_argument("--json", action="store_true", help="machine-readable output")
+    ap.add_argument("--hosts-only", action="store_true", help="scan host filesystems only")
+    ap.add_argument("--guests-only", action="store_true", help="scan guests only (skip host filesystems)")
     args = ap.parse_args()
 
-    results = scan_fleet()
+    # Scan guests (unless --hosts-only)
+    guest_results = []
+    if not args.hosts_only:
+        guest_results = scan_fleet()
+
+    # Scan host filesystems (unless --guests-only)
+    host_results = []
+    current_bands = {}
+    if not args.guests_only:
+        host_results, current_bands = probe_host_filesystems()
+        
+        # Read prior state and detect transitions
+        prior_bands = read_state_file()
+        transitions = detect_transitions(current_bands, prior_bands)
+        
+        # Write new state
+        write_state_file(current_bands)
+    else:
+        prior_bands = None
+        transitions = []
 
     if args.json:
-        print(json.dumps(results, indent=2))
+        # JSON output
+        output = {
+            "guests": guest_results,
+            "hosts": host_results,
+            "transitions": transitions,
+            "prior_bands": prior_bands,
+        }
+        print(json.dumps(output, indent=2))
     else:
-        print(render_results(results))
+        # Human-readable output
+        if guest_results:
+            print(render_results(guest_results))
+        
+        if host_results:
+            print(render_host_results(host_results, transitions, prior_bands))
 
-    # Exit 0 if all guests probed (reachable or not), 1 if any probe error
-    # (a probe error means the probe itself failed, not just that the guest was unreachable)
+    # Exit 0 if all probed (reachable or not), 1 if any probe error
     return 0
 
 

scripts/litellm-health-check.py

@@ -197,16 +197,16 @@ def check_admin_key_list():
     # Try to parse the response
     try:
         data = json.loads(stdout)
-        # Response is a dict with "keys" field
+        # Response is a dict with "keys" (paginated list) and "total_count" fields
         if isinstance(data, dict) and "keys" in data:
-            key_count = len(data["keys"])
+            key_count = data.get("total_count", len(data["keys"]))
         elif isinstance(data, list):
             key_count = len(data)
         else:
             key_count = 0
         if key_count == 0:
             return "Admin Key List", False, "admin-call-failed (empty response)"
-        return "Admin Key List", True, str(key_count) + " keys"
+        return "Admin Key List", True, str(key_count) + " total (" + str(len(data.get("keys", []) if isinstance(data, dict) else data)) + " on page 1)" if isinstance(data, dict) else str(key_count) + " total"
     except Exception as e:
         return "Admin Key List", False, "admin-call-failed (unparseable: " + str(e) + ")"
 

scripts/zulip-monitor.sh

@@ -7,9 +7,11 @@
 # agent leg is retired — see the note after the Tanko leg.
 set -euo pipefail
 
+# Credentials sourced from environment variable ZULIP_API_KEY (set by vault-backed start script)
+# Never fall back to a literal key
+ZULIP_API_KEY="${ZULIP_API_KEY:?ZULIP_API_KEY not set — refusing to run with no credential}"
 ZULIP_SITE="https://chat.sysloggh.net"
 ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
-ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
 OWNER_ZULIP_ID="9"
 
 
@@ -27,12 +29,12 @@ notify() {
   local form
   form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
   curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
+    -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
     -d "${form}" > /dev/null 2>&1 || true
   # Zulip stream post to #agent-hub on topic 'zulip-health'
   local stream_content="${severity} Zulip Monitor: ${msg}"
   curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
+    -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
     -d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
     > /dev/null 2>&1 \
     || echo "  WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)" >> "$LOG"
@@ -41,7 +43,7 @@ notify() {
 # ── Global: Zulip Server ──
 SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
   https://chat.sysloggh.net/api/v1/server_settings \
-  -u 'abiba-bot@chat.sysloggh.net:cKTDMZAPW08dk3zl05sStzO7HRztzyn8' 2>/dev/null) || SERVER_CODE="000"
+  -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" 2>/dev/null) || SERVER_CODE="000"
 SERVER_CODE=$(printf '%s' "$SERVER_CODE" | tr -d '[:space:]')
 [ -n "$SERVER_CODE" ] || SERVER_CODE="000"
 if [ "$SERVER_CODE" != "200" ]; then

stirling-pdf-agent-access.prose.md

@@ -27,7 +27,7 @@ Agent (Hermes/pi) ──curl + X-API-Key──► Stirling-PDF (:8989) ──►
 | Base URL | `http://192.168.68.7:8989` |
 | Auth Method | API Key (header) |
 | Header Name | `X-API-Key` |
-| API Key | `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` |
+| API Key | `«vault: infrastructure/production STIRLING_API_KEY»` |
 | Key Source | `SECURITY_CUSTOMGLOBALAPIKEY` in `/opt/home_stack/docker-compose.yml` |
 | Swagger | `http://192.168.68.7:8989/swagger-ui.html` |
 | Health | `http://192.168.68.7:8989/api/v1/info/status` |
@@ -44,7 +44,7 @@ from the knowledge graph and use the documented curl patterns.
 Direct bash invocations:
 ```bash
 curl -X POST "http://192.168.68.7:8989/api/v1/split-pdf" \
-  -H "X-API-Key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88" \
+  -H "X-API-Key: «vault: infrastructure/production STIRLING_API_KEY»" \
   -F "fileInput=@/path/to/file.pdf" \
   -F "pageNumbers=1,2,3" \
   -o /tmp/output.zip