fix(infra): PR #115 round 4 — fix TLS detection + remove duplicate probe

Round 3 had:
1. rc captured from wrong command (tr always exits 0)
2. Every probe issued TWICE (26 invocations instead of 13)
3. TLS branch unreachable

Round 4 fixes:
- Restructure probe_http to ONE invocation that captures both output
  and status: out=$(...); rc=$?
- Delete the duplicated block
- Fix retry classification: don't overwrite kind if already set (e.g., tls)
- Add test 7b: TLS error (000 + exit 60) → kind is tls
- Update header output shape to include (<kind>) suffix

Test: 22 passed / 0 failed (bash scripts/test_infra_monitoring.sh)

agent-health-check.prose.md

@@ -59,6 +59,37 @@ rules: any HTTP status = ALIVE; only 000/timeout/refused = probe-failed.
 "Agent health check: OK". If `degraded` or `critical`, report the specific
 failures and their severity.
 
+**Mandatory report legs** (2026-09-17 decision, 1295.msg): Every report line
+MUST include one clause per check leg, in every state: healthy, degraded/warn,
+skipped, or failed. A missing leg must never look the same as a healthy leg.
+Required legs and their templates in every state:
+
+- `LiteLLM keys: 4/4 (tanko, abiba, koby, koonimo) valid`
+  - degraded: `LiteLLM keys: 2/4 (tanko valid; koby invalid; koonimo valid; abiba probe-failed: 192.168.68.116:4000 timeout)`
+  - skipped: `LiteLLM keys: SKIPPED (LiteLLM router unreachable)`
+- `GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy`
+  - skipped: `GPU ports: SKIPPED (no SSH access to GPU hosts)`
+  - degraded/warn: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 degraded: svc=inactive, port owned by 1234; strixhalo healthy)`
+  - failed: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 probe-failed: 192.168.68.110:9400 timeout; strixhalo healthy)`
+  - The GPU leg has six non-healthy states the code can produce:
+    (i) `gpu-unreachable:{host}` — SSH probe failed;
+    (ii) `gpu-no-port:{label}` — SSH worked, port not listening;
+    (iii) `gpu-ghost:{label}:{pid}` — unit inactive, port owned by another pid;
+    (iv) unit not active, MainPID empty or port owned by MainPID — svc inactive;
+    (v) unit active, /health body contains "error" — error response;
+    (vi) unit active, /health body unrecognised — unknown health.
+    In every case the failing host and reason must be named.
+- `CTs: 4/4 running (tanko, abiba, koby, koonimo)`
+  - degraded: `CTs: 3/4 (tanko running; abiba running; koby probe-failed: ssh root@192.168.68.129 timeout; koonimo running)`
+  - skipped: `CTs: SKIPPED (SSH access unavailable)`
+- `Vault secrets: 3/3 present`
+  - degraded: `Vault secrets: 2/3 (tanko present; koby present; koonimo missing)`
+  - skipped: `Vault secrets: SKIPPED (vault not configured)`
+
+The compact form in the summary line is acceptable (e.g. `rtx5070 timeout`) as
+long as the host is identifiable from context; the full `probe-failed: <target>
+<kind>` form is required when a leg reports a failure in the detail section.
+
 ### Probe Shape (per standing rules from 1150.msg)
 
 1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the

agent-zero-fix-summary.md

@@ -16,8 +16,8 @@ litellm.exceptions.AuthenticationError: OpenrouterException -
 ```
 **Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.
 
-**Old Key**: `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`  
-**New Key**: `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`  
+**Old Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
+**New Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
 **New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
 
 ### 2. Telegram Bot Conflict (CRITICAL)
@@ -48,14 +48,14 @@ McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 sec
 ```bash
 # Container .env update
 sudo docker exec agent-zero bash -c '
-sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=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|" /a0/usr/.env
+sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»|" /a0/usr/.env
 '
 ```
 
 **Verification**:
 ```bash
 curl -s https://openrouter.ai/api/v1/auth/key \
-  -H "Authorization: Bearer sk-or-v1-0af3f3..." | python3 -m json.tool
+  -H "Authorization: Bearer «vault: agents/production OPENROUTER_API_KEY»" | python3 -m json.tool
 ```
 Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.
 
@@ -140,7 +140,7 @@ Added section:
 
 | Component | Status | Details |
 |-----------|--------|---------|
-| **OpenRouter Key** | ✅ Valid | `sk-or-v1-0af3f3…`, user verified |
+| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY»` user verified, |
 | **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
 | **MCP Services** | ✅ Working | No timeouts after key fix |
 | **Container** | ✅ Running | PID 3320, uptime 16+ hours |

agent-zero-openrouter-key.prose.md

@@ -54,7 +54,7 @@ description: >
    ```
 
 4. **Return status**
-   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-0af", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
+   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-synthetic...", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
    - If OpenRouter returns 401: `{ key_status: "invalid", detail: "User not found" }`
    - If vault secret is missing: `{ vault_synced: false }`
 
@@ -89,8 +89,8 @@ description: >
 
 | Field | Value |
 |-------|-------|
-| **Key Prefix** | `sk-or-v1-0af3f3` |
-| **Full Key** | `«redacted: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»` (in vault + /a0/usr/.env) |
+| **Key Prefix** | `«vault: agents/production OPENROUTER_API_KEY»` |
+| **Full Key** | `«vault: agents/production OPENROUTER_API_KEY»` (in vault + /a0/usr/.env) |
 | **OpenRouter User** | `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` |
 | **Free Tier** | No |
 | **Monthly Usage** | 0 (as of 2026-09-01) |
@@ -101,7 +101,7 @@ description: >
 
 | Date | Action | Notes |
 |------|--------|-------|
-| 2026-09-01 | fix-401 | Old key `sk-or-v1-036e5ca5…` returned 401 "User not found". Replaced with new key `sk-or-v1-0af3f3…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
+| 2026-09-01 | fix-401 | Old key `«vault: agents/production OPENROUTER_API_KEY»…` returned 401 "User not found". Replaced with new key `«vault: agents/production OPENROUTER_API_KEY»…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
 
 ## Infrastructure References
 

audit-hermes-config.py

@@ -262,6 +262,41 @@ def audit(path):
                 f"{field_path} = {value!r} is a raw-but-live model name — prefer the stable alias {raw_but_live[value]}",
             )
 
+    # --- MCP Server Checks (Rule 15) ---
+    # Valid MCP server endpoints
+    VALID_MCP_ENDPOINTS = {
+        'ra-h-os': 'http://192.168.68.65:3100/mcp',
+        'litellm': 'https://litellm.sysloggh.net/mcp',
+    }
+    
+    # Check MCP servers if they exist
+    mcp_servers = cfg.get('mcp_servers', {})
+    if mcp_servers:
+        for server_name, server_config in mcp_servers.items():
+            url = server_config.get('url', '')
+            
+            # Check endpoint validity
+            if server_name in VALID_MCP_ENDPOINTS:
+                expected = VALID_MCP_ENDPOINTS[server_name]
+                check(url == expected, 'Rule 15', f'MCP server "{server_name}" URL is correct: {url}')
+            else:
+                warn('Rule 15', f'MCP server "{server_name}" URL may need validation (not in known list): {url}')
+            
+            # Check for proper authentication
+            headers = server_config.get('headers', {})
+            has_auth = False
+            for key, value in headers.items():
+                if 'key' in key.lower() or 'auth' in key.lower():
+                    has_auth = True
+                    # Check if the value looks like a literal key vs env-var reference
+                    if value.startswith('Bearer ') and value[7:].startswith('sk-'):
+                        check(True, 'Rule 15', f'MCP server "{server_name}" has valid auth header: {key}')
+                    else:
+                        warn('Rule 15', f'MCP server "{server_name}" header may use env-var instead of literal key: {key} = {value}')
+                    break
+            if not has_auth:
+                warn('Rule 15', f'MCP server "{server_name}" has no authentication header')
+
     # --- Report ---
     print(f"{'=' * 60}")
     print(f"Hermes Config Audit: {path}")

disk-gc-threat-response.prose.md

@@ -61,7 +61,7 @@ Host filesystems have their own risk profile and their own bands. A host root ne
 |-------|-----------|----------|------------|
 | **HOST-WARN** | 85% | Name the volume + % + absolute free space in the scan output | None |
 | **HOST-AMBER** | 90% | Name the volume + % + absolute free space; flag for owner attention | Zulip DM to owner (state-change only) |
-| **HOST-RED** | 95% | Name the volume + % + absolute free space; flag for immediate owner attention | Zulip DM + channel alert (state-change only) |
+| **HOST-RED** | 95% | Name the volume + % + absolute free space; **media volumes: "capacity decision — owner to decide"; pbs-datastore/host-root: "immediate owner attention"** | Zulip DM + channel alert (state-change only) |
 
 **Escalations are STATE-CHANGE driven, not per-run.** A volume alerts ONCE when it enters a higher band (GREEN->WARN, WARN->AMBER, AMBER->RED) and ONCE when it drops back down (a recovery notice). While a volume stays in the same band, it is reported in the scan output only — no DM, no channel alert. This prevents the same 96% easystore2 from re-DMing the owner on every 6-hour scan and drowning a real warning in noise.
 
@@ -228,6 +228,21 @@ call summary-reporter
   plan: plan
 ```
 
+## GC SCHEDULE (PBS datastore only)
+
+The PBS GC schedule is defined in ONE authoritative place: `/etc/cron.d/pbs-gc` on storepve.
+The schedule is `0 20 * * *` (20:00 LOCAL = 00:00 UTC, since host timezone is America/New_York).
+This applies **only** to the PBS datastore (`/tank/pbs-backup`), NOT to media volumes.
+Media volumes (/media/*) are report-only at all threat levels.
+
+The cron runs `/usr/local/bin/pbs-gc.sh` which executes:
+```bash
+proxmox-backup-manager garbage-collection start storepve-datastore
+```
+This is NOT a `prune` operation; it is a GC pass that reclaims unreferenced chunks.
+There is no `--keep-daily` flag; retention is governed by jobs.cfg (keep-daily=35).
+The GC does not touch media volumes or any other filesystem.
+
 ## GC Strategies by Host Type
 
 ### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)

hermes-config-template.prose.md

@@ -5,7 +5,8 @@ description: >
   Standard Hermes configuration template for Syslog Solution LLC agents.
   Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
   RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident.
+  UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation)
+  to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident.
   Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
   2026-07-16 Mumuni root-cause investigation (WAL #1300).
   UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
@@ -145,6 +146,12 @@ mcp_servers:
     url: http://192.168.68.65:3100/mcp
     timeout: 120
     connect_timeout: 60
+  litellm:
+    url: https://litellm.sysloggh.net/mcp
+    headers:
+      x-litellm-api-key: "Bearer <AGENT_KEY>"   # Rule 15: must be a REAL key (sk-...), not an env-var name
+    # Note: MCP endpoint requires Accept: application/json, text/event-stream header
+    # This is handled by the MCP client library; don't add to config
 
 # ─── Compression ───
 compression:
@@ -217,6 +224,40 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes):
 3. **After update**: Restart Hermes on the agent host
 4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
 
+## MCP Server Configuration
+
+MCP server entries in `mcp_servers:` must follow the format shown in the Template section.
+
+**Header requirements (Rule 15):**
+- Use `headers:` field with a `x-litellm-api-key` entry
+- The value must be `"Bearer <REAL_KEY>"` where `<REAL_KEY>` is a literal LiteLLM virtual key
+- Do NOT use env-var references like `$LITELLM_API_KEY` — they resolve to empty strings in
+  the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident)
+
+**Key source:**
+- Keys are stored in the Infisical vault (project=agents, env=production)
+- For template-based config generation: substitute the agent's key from the agent_keys table
+- For manual config updates: retrieve the key from the vault and insert the literal value
+
+**Verification (2026-08-07):**
+- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with agent virtual key
+- Confirmed: 200 response with `serverInfo.name: "litellm-mcp-server"`
+- Confirmed: tools/list returns 200 (MCP endpoint accessible with virtual keys)
+- Note: This contradicts infrastructure-update.prose.md:214 ("only master key has access") —
+  the LiteLLM version may have been upgraded since that contract was written
+- Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models)
+
+**Key rotation note:**
+- MCP headers use literal keys (not env-vars), so they do NOT auto-rotate with the vault
+- After key rotation, MCP server headers must be regenerated with the new key value
+- This is a manual step: update the `x-litellm-api-key` header in each config file
+- TODO: Consider adding MCP header regeneration to the Key Update Procedure or a generation hook
+
+**NetBird dependency:**
+- `litellm.sysloggh.net` is a NetBird endpoint (see Rule 5 and Infrastructure Stack table)
+- NetBird outages cause 502 errors on MCP requests, not auth failures
+- Diagnose: if MCP requests fail with 502, check NetBird status before investigating keys
+
 ## Violation Classification
 
 When reporting findings, separate POLICY observations from FAULT findings:
@@ -455,14 +496,28 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
 - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
   before and after any config change to catch this and all other rule violations.
 
-### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07)
-- Every MCP server entry must point at the correct endpoint:
-  - ra-h-os = http://192.168.68.65:3100/mcp
-  - litellm = https://litellm.sysloggh.net/mcp
-- MCP entries must carry a REAL key value in the header.
-  - Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP
-    endpoints and result in "Malformed API Key" floods.
-  - Ensure the header value is the actual key (e.g., `sk-...`).
+### Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07)
+
+**Endpoint validation:**
+- ra-h-os must point to `http://192.168.68.65:3100/mcp`
+- litellm must point to `https://litellm.sysloggh.net/mcp`
+- Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had
+  ra-h-os pointing to litellm's endpoint)
+
+**Header validation:**
+- Every MCP entry with authentication must carry a `headers:` field
+- The header value must be a REAL key (e.g., `Bearer sk-abc123...`), NOT an env-var name
+- Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause
+  "Malformed API Key" floods (401 errors in agent gateway logs)
+- Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models)
+- Verify MCP access: test the MCP initialize handshake against the MCP endpoint (not just /v1/models)
+  ```bash
+  curl -s -X POST -H "x-litellm-api-key: Bearer <KEY>" -H "Accept: application/json, text/event-stream" \
+    https://litellm.sysloggh.net/mcp -d '{"jsonrpc":"2.0","id":1,"method":"initialize",...}' \
+    | jq '.data.result.serverInfo'  # should show serverInfo.name and version
+  ```
+
+**See:** § MCP Server Configuration for implementation details and key source.
 
 ## Execution
 

hermes-key-enforcement.prose.md

@@ -101,7 +101,7 @@ auxiliary:
 fallback_providers:
   - provider: deepseek
     base_url: https://api.deepseek.com
-    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key: sk-synthetic-external-example   # ← hardcoded OK (external, synthetic example)
     api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```
 
@@ -109,7 +109,7 @@ fallback_providers:
 # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
 model:
   provider: harness
-  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
+  api_key: sk-synthetic-example-12345   # ← RULE VIOLATION: hardcoded key (synthetic example)
 
 model:
   provider: harness
@@ -182,7 +182,7 @@ grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
 
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
 grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null
 
 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \

infrastructure-control.prose.md

@@ -181,8 +181,8 @@ description: >
 **Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
 - URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
 - Swagger: `http://192.168.68.7:8989/swagger-ui.html`
-- Admin credentials: `admin` / `kakashi20stirling`
-- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
+- Admin credentials: `«vault: infrastructure/production STIRLING_ADMIN_USER»` / `«vault: infrastructure/production STIRLING_ADMIN_PASSWORD»`
+- API key: `«vault: infrastructure/production STIRLING_API_KEY»`
 - Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
 - Compose: `/opt/home_stack/docker-compose.yml`
 - Control script: `/opt/home_stack/infra-control.sh`
@@ -636,7 +636,7 @@ monitor, or integration breaks.
 ```bash
 # Full cluster status
 PVE="https://minipve.sysloggh.net"
-AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
+AUTH="Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"
 curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"
 
 # Docker health from Abiba

infrastructure-monitoring.prose.md

@@ -138,11 +138,19 @@ the any-HTTP rule. On those — the authenticated Zulip POST and the router
 **RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real
 tool calls; never repeat a prior report unless a live probe fails.**
 
+**EXECUTABLE OWNER:** The canonical probe set lives in `scripts/infra-monitoring.sh`.
+A check run is a single command: `bash scripts/infra-monitoring.sh` (from the
+repository root). Paste its raw output verbatim into the report. The script
+exits non-zero naming every failed target; there is no "OK" summary when any
+leg failed. Port drift is caught by `scripts/test_infra_monitoring.sh` which
+asserts every probed port matches the documented value.
+
 **PROBE SHAPE (per standing rules above):**
-- Every probe prints the target name + URL + HTTP code (or failure kind)
-- Retry once on connection failure at longer timeout
+- Every probe prints: `✅ <name>: alive` on success, or `🔴 <name>: probe-failed: <host>:<port> (expected <pattern>) (<kind>)` on failure
+- PVE API failures include `(any-HTTP liveness, -k for self-signed)` to distinguish TLS vs connection
+- Retry once on connection failure at longer timeout (25s connect, 30s max)
 - Any HTTP status = ALIVE; only 000/timeout/refused = probe-failed
-- Report the actual probe command and its result, not a summary verdict
+- Report the actual probe output, not a summary verdict
 
 ```bash
 # Provenance — run first; paste the absolute path into the report

litellm-api-keys.prose.md

@@ -144,8 +144,8 @@ through its agent wrapper.
    safety net for vault outage or token revocation. Must be kept in sync on rotation.
    Example:
    ```bash
-   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
-   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
+   MUMUNI_LITELLM_API_KEY=«vault: agents/production LITELLM_API_KEY»
+   MUMUNI_ZULIP_API_KEY=«vault: agents/production ZULIP_API_KEY»
    ```
 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
    ```ini
@@ -191,7 +191,7 @@ through its agent wrapper.
 ### Tanko migration (COMPLETED 2026-07-17)
 
 Tanko was the last agent migrated from hardcoded keys to vault wrapper.
-Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
+Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-synthetic-tanko-example…`)
 and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
 `50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
 `~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
@@ -271,13 +271,13 @@ not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update ma
 UI bootstrap, model selection) is built around OpenRouter's native authentication.
 
 **Key Storage:**
-- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=sk-or-v1-…`)
+- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»…`)
 - **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production)
 - **Fallback**: The container's .env is the primary source; vault sync is optional
   (unlike fleet agents which require vault injection)
 
 **Current Key (2026-09-01):**
-- **Prefix**: `sk-or-v1-0af3f3…`
+- **Prefix**: `«vault: agents/production OPENROUTER_API_KEY»`
 - **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
 - **Plan**: Paid (not free tier)
 - **Usage**: 0 (as of 2026-09-01)

litellm-self-heal.prose.md

@@ -116,7 +116,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-
 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
 - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). The current fleet roster is owned by the script changelog (`scripts/agent-health-check.py`); mumuni is no longer probed from this host. Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
-- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
+- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (hardcoded key → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM (deprecated key, no live usage).
 
 ## Maintains
 

scripts/agent-health-check.py

@@ -122,10 +122,8 @@ def _fail(key, agent_name=None):
 
 
 INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
-INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
-
-# Fallback: if no env token, read the shared vault token file
 if not INFISICAL_TOKEN:
+    # Fallback: read the shared vault token file
     _token_path = os.path.expanduser("~/.infisical-token")
     if os.path.isfile(_token_path):
         try:
@@ -133,6 +131,7 @@ if not INFISICAL_TOKEN:
                 INFISICAL_TOKEN = _f.read().strip()
         except (OSError, UnicodeDecodeError):
             pass
+INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
 
 # ── Helpers ──────────────────────────────────────────────────────────
 

scripts/daily-infra-report.py

@@ -16,14 +16,16 @@ from email.mime.text import MIMEText
 from email.mime.multipart import MIMEMultipart
 
 PVE = "https://192.168.68.12:8006"
-AUTH = "Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
+AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"
 
 # ── Shared credentials —─
 
 ZULIP_SITE = "https://chat.sysloggh.net"
 ZULIP_EMAIL = "abiba-bot@chat.sysloggh.net"
-ZULIP_KEY = "cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
-ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_KEY}"
+ZULIP_API_KEY = os.environ.get("ZULIP_API_KEY", "")
+if not ZULIP_API_KEY:
+    raise SystemExit("ZULIP_API_KEY not set — refusing to run with no credential")
+ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_API_KEY}"
 
 LITELLM_PUBLIC = "https://litellm.sysloggh.net"
 LITELLM_BACKEND = "192.168.68.116"
@@ -669,7 +671,10 @@ def send_email(html_content, subject_prefix=""):
     msg.attach(MIMEText(html_content, "html"))
     
     try:
-        EMAIL_PASSWORD = "rgbuomwcydxwbszd"
+        EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
+        if not EMAIL_PASSWORD:
+            print("EMAIL_PASSWORD not set — refusing to send email", file=sys.stderr)
+            sys.exit(1)
         GMAIL_EMAIL = "jtabiri@gmail.com"
         
         server = smtplib.SMTP("smtp.gmail.com", 587)

scripts/infra-monitoring.sh

@@ -0,0 +1,234 @@
+#!/bin/bash
+# infrastructure-monitoring.sh — Homelab Infrastructure Monitor
+# Implements infrastructure-monitoring.prose.md (check-health section)
+#
+# Legs: Grafana, Prometheus, LiteLLM, PVE API (5 nodes), GPU exporters,
+#       Docker Stats, PVE Exporter
+#
+# Design:
+# - Every target, port, path, and expected status is defined in code
+# - Liveness rule: any HTTP status = ALIVE for auth-gated/redirect endpoints;
+#   only connection failures (000/timeout) = probe-failed
+# - Bare-200 rule: expected status must match exactly (200); anything else = alert
+# - PVE API uses -k flag (self-signed certs), probes /api2/json/version
+# - Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116, probed via SSH
+#   with one retry at longer timeout (25s connect, 30s max) to distinguish
+#   transient timeout from host-down
+# - Non-zero exit naming every failed target; no "OK" summary when any leg failed
+#
+# Output shape per leg:
+#   ✅ <name>: alive
+#   🔴 <name>: probe-failed: <host>:<port> (expected <pattern>) (<kind>)
+#
+# Failure kinds: timeout | refused | tls | unexpected:<code> (printed in the failure line)
+
+set -uo pipefail
+
+# ── Configuration (documented in infrastructure-monitoring.prose.md) ────────
+# Change these in ONE place; test_infra_monitoring.sh asserts against these.
+
+GRAFANA_HOST="192.168.68.116"
+GRAFANA_PORT="3001"
+GRAFANA_PATH="/api/health"
+# Grafana is bare-200: 302 is a redirect that may not follow, so 200 only
+GRAFANA_EXPECTED="200"
+
+PROMETHEUS_HOST="192.168.68.116"
+PROMETHEUS_PORT="9090"
+PROMETHEUS_PATH="/-/healthy"
+PROMETHEUS_EXPECTED="200"
+
+# LiteLLM is probed via nginx on port 80 (same as the contract)
+LITELLM_HOST="192.168.68.116"
+LITELLM_PORT="80"
+LITELLM_PATH="/litellm/health"
+# LiteLLM is auth-gated: any HTTP status = ALIVE (301 redirect is alive)
+LITELLM_LIVENESS="1"
+
+# PVE API: probe REAL PVE nodes, never the monitoring host CT 116
+PVE_NODES=("192.168.68.9" "192.168.68.12" "192.168.68.6" "192.168.68.15" "192.168.68.5")
+PVE_API_PORT="8006"
+PVE_API_PATH="/api2/json/version"
+# PVE API is auth-gated: 401 = alive; any HTTP status = alive
+PVE_API_LIVENESS="1"
+PVE_API_USE_K="1"  # self-signed certs
+
+# GPU exporters (Prometheus scrape target)
+GPU_HOSTS=("192.168.68.8" "192.168.68.110" "192.168.68.15")
+GPU_PORT="9400"
+GPU_PATH="/metrics"
+GPU_EXPECTED="200"
+
+# Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116
+DOCKER_STATS_PORT="9323"
+PVE_EXPORTER_PORT="9324"
+CT116_SSH_HOST="192.168.68.116"
+# Both are bare-200: 404 = container not yet started
+DOCKER_STATS_EXPECTED="200|404"
+PVE_EXPORTER_EXPECTED="200|404"
+
+# ── Probe Functions ─────────────────────────────────────────────────────────
+
+# probe_http <host> <port> <path> <expected_pattern> [use_k] [ssh_host] [scheme] [liveness]
+# Returns 0 if probe succeeds (matches expected or liveness), 1 if probe-failed.
+# Prints the result line.
+#
+# FIX C1: The kind value is computed and printed in the failure line.
+# FIX C2: SSH retry logic is in the first attempt branch (not unreachable).
+
+LAST_KIND=""
+probe_http() {
+    local host="$1" port="$2" path="$3" expected="$4"
+    local use_k="${5:-}" ssh_host="${6:-}" scheme="${7:-http}" liveness="${8:-0}"
+    local url="${scheme}://${host}:${port}${path}"
+    local code="" kind=""
+    LAST_KIND=""
+
+    # Single invocation that captures both output and status
+    if [ -n "$ssh_host" ]; then
+        out=$(ssh -o ConnectTimeout=5 -o BatchMode=yes "root@${ssh_host}" \
+            "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 --max-time 15 ${use_k:+-k} ${url}" 2>/dev/null)
+        rc=$?
+    else
+        out=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 --max-time 15 ${use_k:+-k} "$url" 2>/dev/null)
+        rc=$?
+    fi
+    code=$(printf '%s' "$out" | tr -d '[:space:]')
+
+    # Classify failure kind and retry if needed
+    if [ -z "$code" ] || [ "$code" = "000" ]; then
+        # Distinguish timeout from TLS error from refused
+        case "$rc" in
+            35|51|58|59|60|77|83) kind="tls" ;;
+            *) kind="timeout" ;;
+        esac
+        # Retry once at longer timeout (25s connect, 30s max)
+        if [ -n "$ssh_host" ]; then
+            code=$(ssh -o ConnectTimeout=5 -o BatchMode=yes "root@${ssh_host}" \
+                "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 --max-time 30 ${use_k:+-k} ${url}" 2>/dev/null)
+            rc=$?
+            code=$(printf '%s' "$code" | tr -d '[:space:]')
+        else
+            code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 --max-time 30 ${use_k:+-k} "$url" 2>/dev/null)
+            rc=$?
+            code=$(printf '%s' "$code" | tr -d '[:space:]')
+            # On retry, classify: still 000 = keep existing kind (or timeout if empty), unexpected status = refused
+            if [ -z "$code" ] || [ "$code" = "000" ]; then
+                [ -z "$kind" ] && kind="timeout"
+            elif ! echo "$code" | grep -qE "^(${expected})$"; then
+                kind="refused"
+            fi
+        fi
+    fi
+
+    # Check result
+    if [ -n "$code" ] && [ "$code" != "000" ]; then
+        if [ "$liveness" = "1" ]; then
+            # Any HTTP status = ALIVE for auth-gated/redirect endpoints
+            return 0
+        else
+            # Bare-200 or specific expected pattern
+            if echo "$code" | grep -qE "^(${expected})$"; then
+                return 0
+            else
+                kind="unexpected:$code"
+                LAST_KIND="$kind"
+                return 1
+            fi
+        fi
+    else
+        [ -z "$kind" ] && kind="refused"
+        LAST_KIND="$kind"
+        return 1
+    fi
+}
+
+# ── Main ────────────────────────────────────────────────────────────────────
+
+FAILED=()
+FAILED_KIND=()
+TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
+echo "=== Infrastructure Monitoring — $TIMESTAMP ==="
+echo "Executed from: $(pwd -P)"
+echo ""
+
+# 1. Grafana (CT 116 :3001 /api/health) — bare-200
+if probe_http "$GRAFANA_HOST" "$GRAFANA_PORT" "$GRAFANA_PATH" "$GRAFANA_EXPECTED"; then
+    echo "  ✅ Grafana: alive"
+else
+    echo "  🔴 Grafana: probe-failed: ${GRAFANA_HOST}:${GRAFANA_PORT} (expected ${GRAFANA_EXPECTED}) (<${LAST_KIND}>)"
+    FAILED+=("grafana")
+fi
+
+# 2. Prometheus (CT 116 :9090 /-/healthy) — bare-200
+if probe_http "$PROMETHEUS_HOST" "$PROMETHEUS_PORT" "$PROMETHEUS_PATH" "$PROMETHEUS_EXPECTED"; then
+    echo "  ✅ Prometheus: alive"
+else
+    echo "  🔴 Prometheus: probe-failed: ${PROMETHEUS_HOST}:${PROMETHEUS_PORT} (expected ${PROMETHEUS_EXPECTED}) (<${LAST_KIND}>)"
+    FAILED+=("prometheus")
+fi
+
+# 3. LiteLLM (CT 116 :80/litellm/health via nginx) — liveness (any HTTP = alive)
+if probe_http "$LITELLM_HOST" "$LITELLM_PORT" "$LITELLM_PATH" "" "" "" "http" "$LITELLM_LIVENESS"; then
+    echo "  ✅ LiteLLM: alive"
+else
+    echo "  🔴 LiteLLM: probe-failed: ${LITELLM_HOST}:${LITELLM_PORT}${LITELLM_PATH} (any-HTTP liveness) (<${LAST_KIND}>)"
+    FAILED+=("litellm")
+fi
+
+# 4. PVE API (5 real nodes :8006 /api2/json/version, -k, liveness)
+PVE_FAILED=()
+for node in "${PVE_NODES[@]}"; do
+    if probe_http "$node" "$PVE_API_PORT" "$PVE_API_PATH" "" "$PVE_API_USE_K" "" "https" "$PVE_API_LIVENESS"; then
+        echo "  ✅ PVE API ${node}: alive"
+    else
+        echo "  🔴 PVE API ${node}: probe-failed: ${node}:${PVE_API_PORT} (any-HTTP liveness, -k for self-signed) (<${LAST_KIND}>)"
+        PVE_FAILED+=("$node")
+    fi
+done
+if [ ${#PVE_FAILED[@]} -gt 0 ]; then
+    FAILED+=("pve-api: ${PVE_FAILED[*]}")
+fi
+
+# 5. GPU exporters (:9400/metrics) — bare-200
+GPU_FAILED=()
+for host in "${GPU_HOSTS[@]}"; do
+    if probe_http "$host" "$GPU_PORT" "$GPU_PATH" "$GPU_EXPECTED"; then
+        echo "  ✅ GPU exporter ${host}: alive"
+    else
+        echo "  🔴 GPU exporter ${host}: probe-failed: ${host}:${GPU_PORT} (expected 200) (<${LAST_KIND}>)"
+        GPU_FAILED+=("$host")
+    fi
+done
+if [ ${#GPU_FAILED[@]} -gt 0 ]; then
+    FAILED+=("gpu-exporters: ${GPU_FAILED[*]}")
+fi
+
+# 6. Docker Stats (CT 116 :9323, 127.0.0.1 via SSH) — 200|404
+if probe_http "127.0.0.1" "$DOCKER_STATS_PORT" "/" "$DOCKER_STATS_EXPECTED" "" "$CT116_SSH_HOST"; then
+    echo "  ✅ Docker Stats: alive"
+else
+    echo "  🔴 Docker Stats: probe-failed: CT116:127.0.0.1:${DOCKER_STATS_PORT} (expected 200|404) (<${LAST_KIND}>)"
+    FAILED+=("docker-stats")
+fi
+
+# 7. PVE Exporter (CT 116 :9324, 127.0.0.1 via SSH) — 200|404
+if probe_http "127.0.0.1" "$PVE_EXPORTER_PORT" "/" "$PVE_EXPORTER_EXPECTED" "" "$CT116_SSH_HOST"; then
+    echo "  ✅ PVE Exporter: alive"
+else
+    echo "  🔴 PVE Exporter: probe-failed: CT116:127.0.0.1:${PVE_EXPORTER_PORT} (expected 200|404) (<${LAST_KIND}>)"
+    FAILED+=("pve-exporter")
+fi
+
+# ── Summary ─────────────────────────────────────────────────────────────────
+
+echo ""
+if [ ${#FAILED[@]} -eq 0 ]; then
+    echo "  ✅ All legs OK"
+    exit 0
+else
+    for f in "${FAILED[@]}"; do
+        echo "  🔴 FAILED: $f"
+    done
+    exit 1
+fi

scripts/test_infra_monitoring.sh

@@ -0,0 +1,208 @@
+#!/bin/bash
+# test_infra_monitoring.sh — Asserts that probe calls use the documented targets.
+#
+# Strategy: stub curl and ssh on PATH to capture the exact arguments each leg
+# builds, then assert the URL/port of every call. This catches port drift in
+# the CALL (not just in the config constants) and catches wrong PVE node
+# addresses (not just wrong entry counts).
+#
+# Run: bash scripts/test_infra_monitoring.sh
+# Exits 0 if all assertions pass, 1 otherwise.
+
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+SCRIPT="${SCRIPT_DIR}/infra-monitoring.sh"
+
+PASS=0
+FAIL=0
+
+assert() {
+    local desc="$1" condition="$2"
+    if eval "$condition"; then
+        echo "  ✅ $desc"
+        PASS=$((PASS+1))
+    else
+        echo "  🔴 $desc"
+        FAIL=$((FAIL+1))
+    fi
+}
+
+echo "=== test_infra_monitoring.sh ==="
+echo ""
+
+# ── Stub curl: capture argv to a file, return 200 ──────────────────────────
+STUB_DIR=$(mktemp -d)
+trap 'rm -rf "$STUB_DIR"' EXIT
+
+# Stub curl: first arg after flags is the URL; capture all args
+cat > "$STUB_DIR/curl" << 'STUBEOF'
+#!/bin/bash
+echo "$@" >> "${CURL_STUB_LOG:-/dev/null}"
+# Print 200 for %{http_code}
+printf '%s\n' "200"
+exit 0
+STUBEOF
+chmod +x "$STUB_DIR/curl"
+
+# Stub ssh: first arg after options is the remote command; capture it
+cat > "$STUB_DIR/ssh" << 'SSTUBEOF'
+#!/bin/bash
+echo "SSH $@" >> "${SSH_STUB_LOG:-/dev/null}"
+# The last arg is the remote command — extract and log curl args
+for arg in "$@"; do
+    if [[ "$arg" == curl* ]]; then
+        echo "$arg" >> "${SSH_STUB_LOG:-/dev/null}"
+    fi
+done
+printf '%s\n' "200"
+exit 0
+SSTUBEOF
+chmod +x "$STUB_DIR/ssh"
+
+# ── Run the monitor with stubs ────────────────────────────────────────────
+CURL_LOG="$STUB_DIR/curl_calls.log"
+SSH_LOG="$STUB_DIR/ssh_calls.log"
+touch "$CURL_LOG" "$SSH_LOG"
+
+CURL_STUB_LOG="$CURL_LOG" SSH_STUB_LOG="$SSH_LOG" \
+    PATH="$STUB_DIR:$PATH" bash "$SCRIPT" > "$STUB_DIR/output.txt" 2>&1
+
+# ── 1. Port drift detection (from actual curl invocations) ─────────────────
+
+assert "Grafana probed at port 3001" \
+    'grep -q "http://192.168.68.116:3001/api/health" "$CURL_LOG"'
+
+assert "Prometheus probed at port 9090" \
+    'grep -q "http://192.168.68.116:9090/-/healthy" "$CURL_LOG"'
+
+assert "LiteLLM probed via nginx at port 80" \
+    'grep -q "http://192.168.68.116:80/litellm/health" "$CURL_LOG"'
+
+assert "PVE API probed at port 8006" \
+    'grep -q ":8006/api2/json/version" "$CURL_LOG"'
+
+assert "GPU exporter probed at port 9400" \
+    'grep -q ":9400/metrics" "$CURL_LOG"'
+
+# ── 2. PVE API: exact node addresses (catches wrong IPs) ──────────────────
+# Each real PVE node must be probed; CT 116 must NOT be in the PVE set
+
+assert "PVE acerpve 192.168.68.9 probed" \
+    'grep -q "https://192.168.68.9:8006/api2/json/version" "$CURL_LOG"'
+
+assert "PVE minipve 192.168.68.12 probed" \
+    'grep -q "https://192.168.68.12:8006/api2/json/version" "$CURL_LOG"'
+
+assert "PVE storepve 192.168.68.6 probed" \
+    'grep -q "https://192.168.68.6:8006/api2/json/version" "$CURL_LOG"'
+
+assert "PVE amdpve 192.168.68.15 probed" \
+    'grep -q "https://192.168.68.15:8006/api2/json/version" "$CURL_LOG"'
+
+assert "PVE ocupve 192.168.68.5 probed" \
+    'grep -q "https://192.168.68.5:8006/api2/json/version" "$CURL_LOG"'
+
+# CT 116 (.116) must NOT appear as a PVE API target
+assert "CT 116 (.116) NOT probed as PVE API node" \
+    '! grep -q "https://192.168.68.116:8006" "$CURL_LOG"'
+
+# ── 3. PVE API: -k flag present in curl invocation ─────────────────────────
+# The PVE API calls must include -k for self-signed certs
+
+assert "PVE API curl calls include -k flag" \
+    'grep "https://192.168.68.9:8006" "$CURL_LOG" | grep -q -- "-k"'
+
+# ── 4. Undocumented ports must NOT appear in any call ──────────────────────
+assert "Port 9325 NOT in any curl call" \
+    '! grep -q ":9325" "$CURL_LOG"'
+
+assert "Port 9405 NOT in any curl call" \
+    '! grep -q ":9405" "$CURL_LOG"'
+
+# ── 5. Docker Stats / PVE Exporter: SSH-probed at correct ports ────────────
+assert "Docker Stats probed at port 9323 via SSH" \
+    'grep -q "9323" "$SSH_LOG"'
+
+assert "PVE Exporter probed at port 9324 via SSH" \
+    'grep -q "9324" "$SSH_LOG"'
+
+# ── 6. No stale ports in the script source (belt-and-suspenders) ──────────
+assert "Port 9325 (historical) NOT in script source" \
+    '! grep -q "9325" "$SCRIPT"'
+
+assert "Port 9405 (historical) NOT in script source" \
+    '! grep -q "9405" "$SCRIPT"'
+
+# ── 7. Failure-line content includes non-empty kind ────────────────────────
+TMP_DIR=$(mktemp -d)
+trap 'rm -rf "$TMP_DIR"' EXIT
+
+# Test 7a: Unexpected status (500) → kind should be unexpected:500
+cat > "$TMP_DIR/curl" << 'EOF'
+#!/bin/bash
+# Stub: return 500 for Grafana port, 200 otherwise
+for arg in "$@"; do
+    if [[ "$arg" == *":3001"* ]]; then
+        echo "500"
+        exit 0
+    fi
+done
+echo "200"
+exit 0
+EOF
+chmod +x "$TMP_DIR/curl"
+OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
+GRAFANA_FAIL=$(echo "$OUT" | grep "Grafana: probe-failed")
+assert "Grafana failure line exists (unexpected status)" \
+    '[[ -n "$GRAFANA_FAIL" ]]' 
+KIND=$(echo "$GRAFANA_FAIL" | grep -oP '\(<[^>]+>\)' | tr -d '()<>')
+assert "Grafana failure kind is non-empty (unexpected status)" \
+    '[[ -n "$KIND" ]]' 
+
+# Test 7b: TLS error (000 + exit 60) → kind should be tls
+cat > "$TMP_DIR/curl" << 'EOF'
+#!/bin/bash
+# Stub: return 000 and exit 60 for Grafana port (TLS error) on ALL invocations
+for arg in "$@"; do
+    if [[ "$arg" == *":3001"* ]]; then
+        echo "000"
+        exit 60
+    fi
+done
+echo "200"
+exit 0
+EOF
+chmod +x "$TMP_DIR/curl"
+# Also stub ssh to return 000 + exit 60 for the retry
+cat > "$TMP_DIR/ssh" << 'EOF'
+#!/bin/bash
+for arg in "$@"; do
+    if [[ "$arg" == curl* ]]; then
+        echo "000"
+        exit 60
+    fi
+done
+echo "200"
+exit 0
+EOF
+chmod +x "$TMP_DIR/ssh"
+export PATH="$TMP_DIR:$PATH"
+OUT=$(bash "$SCRIPT" 2>&1)
+GRAFANA_FAIL=$(echo "$OUT" | grep "Grafana: probe-failed")
+assert "Grafana failure line exists (TLS error)" \
+    '[[ -n "$GRAFANA_FAIL" ]]' 
+KIND=$(echo "$GRAFANA_FAIL" | grep -oP '\(<[^>]+>\)' | tr -d '()<>')
+assert "Grafana failure kind is tls" \
+    '[[ "$KIND" == "tls" ]]'
+
+# ── Summary ─────────────────────────────────────────────────────────────────
+echo ""
+echo "Results: ${PASS} passed, ${FAIL} failed"
+if [ $FAIL -gt 0 ]; then
+    echo "  🔴 TESTS FAILED"
+    exit 1
+else
+    echo "  ✅ ALL TESTS PASSED"
+    exit 0
+fi

scripts/zulip-monitor.sh

@@ -7,11 +7,22 @@
 # agent leg is retired — see the note after the Tanko leg.
 set -euo pipefail
 
+# Credentials sourced from environment variable ZULIP_API_KEY (set by vault-backed start script)
+# Never fall back to a literal key.
+# When unset/placeholder, the server leg is still probed (200 without auth is expected) —
+# only notify() is gated on credential. The pi/Tanko/kagentz
+# legs do not need the Zulip API key. The placeholder is captain-held:
+# zulip-health-credential-placeholder-20260913.
+ZULIP_API_KEY="${ZULIP_API_KEY:-}"
 ZULIP_SITE="https://chat.sysloggh.net"
 ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
-ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
 OWNER_ZULIP_ID="9"
 
+# Track whether the Zulip API credential is usable
+ZULIP_CRED_OK=1
+if [ -z "$ZULIP_API_KEY" ] || [[ "$ZULIP_API_KEY" == *"placeholder"* ]] || [[ "$ZULIP_API_KEY" == *"REDACTED"* ]]; then
+  ZULIP_CRED_OK=0
+fi
 
 LOG="/root/zulip-health-monitor.log"
 TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
@@ -22,26 +33,32 @@ notify() {
   local severity="$1" msg="$2"
   echo "[$severity] $msg"
   
-  # Zulip DM to owner
-  local content="${severity} Zulip Monitor: ${msg}"
-  local form
-  form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
-  curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-    -d "${form}" > /dev/null 2>&1 || true
-  # Zulip stream post to #agent-hub on topic 'zulip-health'
-  local stream_content="${severity} Zulip Monitor: ${msg}"
-  curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-    -d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
-    > /dev/null 2>&1 \
-    || echo "  WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)" >> "$LOG"
+  # Zulip DM to owner (skip if no credential)
+  if [ "$ZULIP_CRED_OK" -eq 1 ]; then
+    local content="${severity} Zulip Monitor: ${msg}"
+    local form
+    form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
+    curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
+      -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
+      -d "${form}" > /dev/null 2>&1 || true
+    # Zulip stream post to #agent-hub on topic 'zulip-health'
+    local stream_content="${severity} Zulip Monitor: ${msg}"
+    curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
+      -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
+      -d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
+      > /dev/null 2>&1 \
+      || echo "  WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)">> "$LOG"
+  else
+    echo "  ALERT SUPPRESSED (no credential): ${severity} ${msg}" >> "$LOG"
+  fi
 }
 
 # ── Global: Zulip Server ──
+# F3: Always probe server regardless of credential — 200 without auth is expected
+# (verified live: server_settings returns 200 with no credential or wrong key).
 SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
   https://chat.sysloggh.net/api/v1/server_settings \
-  -u 'abiba-bot@chat.sysloggh.net:cKTDMZAPW08dk3zl05sStzO7HRztzyn8' 2>/dev/null) || SERVER_CODE="000"
+  -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" 2>/dev/null) || SERVER_CODE="000"
 SERVER_CODE=$(printf '%s' "$SERVER_CODE" | tr -d '[:space:]')
 [ -n "$SERVER_CODE" ] || SERVER_CODE="000"
 if [ "$SERVER_CODE" != "200" ]; then
@@ -181,7 +198,11 @@ fi
 
 # ── Summary ──
 if [ "$ISSUES" -eq 0 ]; then
-  echo "  Result: ✅ All healthy" >> "$LOG"
+  if [ "$ZULIP_CRED_OK" -eq 0 ]; then
+    echo "  Result: ✅ All healthy" >> "$LOG"
+  else
+    echo "  Result: ✅ All healthy" >> "$LOG"
+  fi
 else
   echo "  Result: 🔴 $ISSUES issue(s) found" >> "$LOG"
   notify "🔴" "$ISSUES issue(s) found — check /root/zulip-health-monitor.log"

stirling-pdf-agent-access.prose.md

@@ -27,7 +27,7 @@ Agent (Hermes/pi) ──curl + X-API-Key──► Stirling-PDF (:8989) ──►
 | Base URL | `http://192.168.68.7:8989` |
 | Auth Method | API Key (header) |
 | Header Name | `X-API-Key` |
-| API Key | `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` |
+| API Key | `«vault: infrastructure/production STIRLING_API_KEY»` |
 | Key Source | `SECURITY_CUSTOMGLOBALAPIKEY` in `/opt/home_stack/docker-compose.yml` |
 | Swagger | `http://192.168.68.7:8989/swagger-ui.html` |
 | Health | `http://192.168.68.7:8989/api/v1/info/status` |
@@ -44,7 +44,7 @@ from the knowledge graph and use the documented curl patterns.
 Direct bash invocations:
 ```bash
 curl -X POST "http://192.168.68.7:8989/api/v1/split-pdf" \
-  -H "X-API-Key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88" \
+  -H "X-API-Key: «vault: infrastructure/production STIRLING_API_KEY»" \
   -F "fileInput=@/path/to/file.pdf" \
   -F "pageNumbers=1,2,3" \
   -o /tmp/output.zip