docs: add PBS GC schedule to proxmox-monitor contract

- GC runs daily at 20:00 UTC (after backup window closes at 06:40Z) - Volume is /tank/pbs-backup on ZFS pool 'tank' (12.7T, 10% used) - NOT /media/easystore2 (media library, 3.7T, 96% HOST-RED) - Cron: /etc/cron.d/pbs-gc on storepve (192.168.68.6) - Last run: 2026-09-18 00:08:34 UTC (removed 551.768 GiB)
Merge pull request 'feat: add MCP server URL validation to hermes-config-template contract' (#114 ) from fm/hermes-config-mcp-url-validation into master
2026-09-18 04:46:44 +00:00 · 2026-09-17 13:22:31 +00:00 · 2026-09-17 12:33:52 +00:00 · 2026-09-17 12:24:43 +00:00 · 2026-09-17 12:19:00 +00:00 · 2026-09-17 11:30:44 +00:00
13 changed files with 157 additions and 43 deletions
@@ -16,8 +16,8 @@ litellm.exceptions.AuthenticationError: OpenrouterException -
 ```
 **Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.

-**Old Key**: `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`  
-**New Key**: `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`  
+**Old Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
+**New Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
 **New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`

 ### 2. Telegram Bot Conflict (CRITICAL)
@@ -48,14 +48,14 @@ McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 sec
 ```bash
 # Container .env update
 sudo docker exec agent-zero bash -c '
-sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=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|" /a0/usr/.env
+sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»|" /a0/usr/.env
 '
 ```

 **Verification**:
 ```bash
 curl -s https://openrouter.ai/api/v1/auth/key \
-  -H "Authorization: Bearer sk-or-v1-0af3f3..." | python3 -m json.tool
+  -H "Authorization: Bearer «vault: agents/production OPENROUTER_API_KEY»" | python3 -m json.tool
 ```
 Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.

@@ -140,7 +140,7 @@ Added section:

 | Component | Status | Details |
 |-----------|--------|---------|
-| **OpenRouter Key** | ✅ Valid | `sk-or-v1-0af3f3…`, user verified |
+| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY»` user verified, |
 | **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
 | **MCP Services** | ✅ Working | No timeouts after key fix |
 | **Container** | ✅ Running | PID 3320, uptime 16+ hours |
@@ -54,7 +54,7 @@ description: >
   ```

 4. **Return status**
-   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-0af", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
+   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-synthetic...", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
   - If OpenRouter returns 401: `{ key_status: "invalid", detail: "User not found" }`
   - If vault secret is missing: `{ vault_synced: false }`

@@ -89,8 +89,8 @@ description: >

 | Field | Value |
 |-------|-------|
-| **Key Prefix** | `sk-or-v1-0af3f3` |
-| **Full Key** | `«redacted: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»` (in vault + /a0/usr/.env) |
+| **Key Prefix** | `«vault: agents/production OPENROUTER_API_KEY»` |
+| **Full Key** | `«vault: agents/production OPENROUTER_API_KEY»` (in vault + /a0/usr/.env) |
 | **OpenRouter User** | `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` |
 | **Free Tier** | No |
 | **Monthly Usage** | 0 (as of 2026-09-01) |
@@ -101,7 +101,7 @@ description: >

 | Date | Action | Notes |
 |------|--------|-------|
-| 2026-09-01 | fix-401 | Old key `sk-or-v1-036e5ca5…` returned 401 "User not found". Replaced with new key `sk-or-v1-0af3f3…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
+| 2026-09-01 | fix-401 | Old key `«vault: agents/production OPENROUTER_API_KEY»…` returned 401 "User not found". Replaced with new key `«vault: agents/production OPENROUTER_API_KEY»…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |

 ## Infrastructure References

@@ -262,6 +262,41 @@ def audit(path):
                f"{field_path} = {value!r} is a raw-but-live model name — prefer the stable alias {raw_but_live[value]}",
            )

+    # --- MCP Server Checks (Rule 15) ---
+    # Valid MCP server endpoints
+    VALID_MCP_ENDPOINTS = {
+        'ra-h-os': 'http://192.168.68.65:3100/mcp',
+        'litellm': 'https://litellm.sysloggh.net/mcp',
+    }
+    
+    # Check MCP servers if they exist
+    mcp_servers = cfg.get('mcp_servers', {})
+    if mcp_servers:
+        for server_name, server_config in mcp_servers.items():
+            url = server_config.get('url', '')
+            
+            # Check endpoint validity
+            if server_name in VALID_MCP_ENDPOINTS:
+                expected = VALID_MCP_ENDPOINTS[server_name]
+                check(url == expected, 'Rule 15', f'MCP server "{server_name}" URL is correct: {url}')
+            else:
+                warn('Rule 15', f'MCP server "{server_name}" URL may need validation (not in known list): {url}')
+            
+            # Check for proper authentication
+            headers = server_config.get('headers', {})
+            has_auth = False
+            for key, value in headers.items():
+                if 'key' in key.lower() or 'auth' in key.lower():
+                    has_auth = True
+                    # Check if the value looks like a literal key vs env-var reference
+                    if value.startswith('Bearer ') and value[7:].startswith('sk-'):
+                        check(True, 'Rule 15', f'MCP server "{server_name}" has valid auth header: {key}')
+                    else:
+                        warn('Rule 15', f'MCP server "{server_name}" header may use env-var instead of literal key: {key} = {value}')
+                    break
+            if not has_auth:
+                warn('Rule 15', f'MCP server "{server_name}" has no authentication header')
+
    # --- Report ---
    print(f"{'=' * 60}")
    print(f"Hermes Config Audit: {path}")
@@ -5,7 +5,8 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident.
+  UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation)
+  to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident.
  Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
  2026-07-16 Mumuni root-cause investigation (WAL #1300).
  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
@@ -145,6 +146,12 @@ mcp_servers:
    url: http://192.168.68.65:3100/mcp
    timeout: 120
    connect_timeout: 60
+  litellm:
+    url: https://litellm.sysloggh.net/mcp
+    headers:
+      x-litellm-api-key: "Bearer <AGENT_KEY>"   # Rule 15: must be a REAL key (sk-...), not an env-var name
+    # Note: MCP endpoint requires Accept: application/json, text/event-stream header
+    # This is handled by the MCP client library; don't add to config

 # ─── Compression ───
 compression:
@@ -217,6 +224,40 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes):
 3. **After update**: Restart Hermes on the agent host
 4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`

+## MCP Server Configuration
+
+MCP server entries in `mcp_servers:` must follow the format shown in the Template section.
+
+**Header requirements (Rule 15):**
+- Use `headers:` field with a `x-litellm-api-key` entry
+- The value must be `"Bearer <REAL_KEY>"` where `<REAL_KEY>` is a literal LiteLLM virtual key
+- Do NOT use env-var references like `$LITELLM_API_KEY` — they resolve to empty strings in
+  the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident)
+
+**Key source:**
+- Keys are stored in the Infisical vault (project=agents, env=production)
+- For template-based config generation: substitute the agent's key from the agent_keys table
+- For manual config updates: retrieve the key from the vault and insert the literal value
+
+**Verification (2026-08-07):**
+- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with agent virtual key
+- Confirmed: 200 response with `serverInfo.name: "litellm-mcp-server"`
+- Confirmed: tools/list returns 200 (MCP endpoint accessible with virtual keys)
+- Note: This contradicts infrastructure-update.prose.md:214 ("only master key has access") —
+  the LiteLLM version may have been upgraded since that contract was written
+- Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models)
+
+**Key rotation note:**
+- MCP headers use literal keys (not env-vars), so they do NOT auto-rotate with the vault
+- After key rotation, MCP server headers must be regenerated with the new key value
+- This is a manual step: update the `x-litellm-api-key` header in each config file
+- TODO: Consider adding MCP header regeneration to the Key Update Procedure or a generation hook
+
+**NetBird dependency:**
+- `litellm.sysloggh.net` is a NetBird endpoint (see Rule 5 and Infrastructure Stack table)
+- NetBird outages cause 502 errors on MCP requests, not auth failures
+- Diagnose: if MCP requests fail with 502, check NetBird status before investigating keys
+
 ## Violation Classification

 When reporting findings, separate POLICY observations from FAULT findings:
@@ -455,14 +496,28 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
 - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
  before and after any config change to catch this and all other rule violations.

-### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07)
- Every MCP server entry must point at the correct endpoint:
-  - ra-h-os = http://192.168.68.65:3100/mcp
-  - litellm = https://litellm.sysloggh.net/mcp
- MCP entries must carry a REAL key value in the header.
-  - Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP
-    endpoints and result in "Malformed API Key" floods.
-  - Ensure the header value is the actual key (e.g., `sk-...`).
+### Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07)
+
+**Endpoint validation:**
+- ra-h-os must point to `http://192.168.68.65:3100/mcp`
+- litellm must point to `https://litellm.sysloggh.net/mcp`
+- Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had
+  ra-h-os pointing to litellm's endpoint)
+
+**Header validation:**
+- Every MCP entry with authentication must carry a `headers:` field
+- The header value must be a REAL key (e.g., `Bearer sk-abc123...`), NOT an env-var name
+- Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause
+  "Malformed API Key" floods (401 errors in agent gateway logs)
+- Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models)
+- Verify MCP access: test the MCP initialize handshake against the MCP endpoint (not just /v1/models)
+  ```bash
+  curl -s -X POST -H "x-litellm-api-key: Bearer <KEY>" -H "Accept: application/json, text/event-stream" \
+    https://litellm.sysloggh.net/mcp -d '{"jsonrpc":"2.0","id":1,"method":"initialize",...}' \
+    | jq '.data.result.serverInfo'  # should show serverInfo.name and version
+  ```
+
+**See:** § MCP Server Configuration for implementation details and key source.

 ## Execution

@@ -101,7 +101,7 @@ auxiliary:
 fallback_providers:
  - provider: deepseek
    base_url: https://api.deepseek.com
-    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key: sk-synthetic-external-example   # ← hardcoded OK (external, synthetic example)
    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```

@@ -109,7 +109,7 @@ fallback_providers:
 # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
 model:
  provider: harness
-  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
+  api_key: sk-synthetic-example-12345   # ← RULE VIOLATION: hardcoded key (synthetic example)

 model:
  provider: harness
@@ -182,7 +182,7 @@ grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml

 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
 grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null

 # 3. Verify running process env matches dedicated key
 cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
@@ -181,8 +181,8 @@ description: >
 **Stirling-PDF** (deployed 2026-07-03, Authentik SSO 2026-07-03):
 - URL: `https://pdf.sysloggh.net` (public) / `http://192.168.68.7:8989` (direct)
 - Swagger: `http://192.168.68.7:8989/swagger-ui.html`
- Admin credentials: `admin` / `kakashi20stirling`
- API key: `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88`
+- Admin credentials: `«vault: infrastructure/production STIRLING_ADMIN_USER»` / `«vault: infrastructure/production STIRLING_ADMIN_PASSWORD»`
+- API key: `«vault: infrastructure/production STIRLING_API_KEY»`
 - Authentik OAuth2: configured but disabled (requires paid Server license). Ready to enable: set `SECURITY_OAUTH2_ENABLED=true` + `SECURITY_LOGINMETHOD=all`
 - Compose: `/opt/home_stack/docker-compose.yml`
 - Control script: `/opt/home_stack/infra-control.sh`
@@ -636,7 +636,7 @@ monitor, or integration breaks.
 ```bash
 # Full cluster status
 PVE="https://minipve.sysloggh.net"
-AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
+AUTH="Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"
 curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"

 # Docker health from Abiba
@@ -144,8 +144,8 @@ through its agent wrapper.
   safety net for vault outage or token revocation. Must be kept in sync on rotation.
   Example:
   ```bash
-   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
-   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
+   MUMUNI_LITELLM_API_KEY=«vault: agents/production LITELLM_API_KEY»
+   MUMUNI_ZULIP_API_KEY=«vault: agents/production ZULIP_API_KEY»
   ```
 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
   ```ini
@@ -191,7 +191,7 @@ through its agent wrapper.
 ### Tanko migration (COMPLETED 2026-07-17)

 Tanko was the last agent migrated from hardcoded keys to vault wrapper.
-Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
+Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-synthetic-tanko-example…`)
 and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
 `50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
 `~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
@@ -271,13 +271,13 @@ not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update ma
 UI bootstrap, model selection) is built around OpenRouter's native authentication.

 **Key Storage:**
- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=sk-or-v1-…`)
+- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»…`)
 - **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production)
 - **Fallback**: The container's .env is the primary source; vault sync is optional
  (unlike fleet agents which require vault injection)

 **Current Key (2026-09-01):**
- **Prefix**: `sk-or-v1-0af3f3…`
+- **Prefix**: `«vault: agents/production OPENROUTER_API_KEY»`
 - **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
 - **Plan**: Paid (not free tier)
 - **Usage**: 0 (as of 2026-09-01)
@@ -116,7 +116,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-
 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
 - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). The current fleet roster is owned by the script changelog (`scripts/agent-health-check.py`); mumuni is no longer probed from this host. Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
+- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (hardcoded key → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM (deprecated key, no live usage).

 ## Maintains

@@ -142,3 +142,21 @@ each probe. If any probe returns non-200, flag as alert.
 - **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
 - **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
 - **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.
+## PBS Garbage Collection Schedule (storepve-datastore)
+
+**Schedule:** Daily at 20:00 UTC (4:00 PM EDT)
+
+**Rationale:** The nightly backup window runs 04:00–06:40 UTC (local backups 04:00–06:40Z, S3 sync 04:15Z, S3 trim 05:15Z). Running GC during this window causes avoidable I/O contention on the same datastore and host. 20:00 UTC lands well after the backup window closes and before the next day's backups begin.
+
+**Volume:** `/tank/pbs-backup` on the ZFS pool `tank` (12.7T total, 11.3T free, 10% used). Inside CT 107, this appears as 1.5T total / 413G used / 1.1T avail = 28%.
+
+**Important:** This is NOT the same as `/media/easystore2` (3.7T, 96% used, HOST-RED), which is a media library ("4K MOVIES") on the storepve host root filesystem. The PBS datastore lives on a separate ZFS pool.
+
+**Cron:** `/etc/cron.d/pbs-gc` on storepve (192.168.68.6):
+```
+0 20 * * * root /usr/local/bin/pbs-gc.sh
+```
+
+**Last GC run:** 2026-09-18 00:08:34 UTC (3h 19m 9s, removed 551.768 GiB)
+
+**Schedule owner:** The `gc-schedule` field in the PBS datastore config is the durable owner of GC behavior. The cron is a fallback because `proxmox-backup-manager datastore update --gc-schedule` failed to parse the calendar event (Nom(Eof) error).
@@ -122,10 +122,8 @@ def _fail(key, agent_name=None):


 INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
-INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
-
-# Fallback: if no env token, read the shared vault token file
 if not INFISICAL_TOKEN:
+    # Fallback: read the shared vault token file
    _token_path = os.path.expanduser("~/.infisical-token")
    if os.path.isfile(_token_path):
        try:
@@ -133,6 +131,7 @@ if not INFISICAL_TOKEN:
                INFISICAL_TOKEN = _f.read().strip()
        except (OSError, UnicodeDecodeError):
            pass
+INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")

 # ── Helpers ──────────────────────────────────────────────────────────

@@ -16,14 +16,16 @@ from email.mime.text import MIMEText
 from email.mime.multipart import MIMEMultipart

 PVE = "https://192.168.68.12:8006"
-AUTH = "Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
+AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»"

 # ── Shared credentials —─

 ZULIP_SITE = "https://chat.sysloggh.net"
 ZULIP_EMAIL = "abiba-bot@chat.sysloggh.net"
-ZULIP_KEY = "cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
-ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_KEY}"
+ZULIP_API_KEY = os.environ.get("ZULIP_API_KEY", "")
+if not ZULIP_API_KEY:
+    raise SystemExit("ZULIP_API_KEY not set — refusing to run with no credential")
+ZULIP_AUTH = f"{ZULIP_EMAIL}:{ZULIP_API_KEY}"

 LITELLM_PUBLIC = "https://litellm.sysloggh.net"
 LITELLM_BACKEND = "192.168.68.116"
@@ -669,7 +671,10 @@ def send_email(html_content, subject_prefix=""):
    msg.attach(MIMEText(html_content, "html"))
    
    try:
-        EMAIL_PASSWORD = "rgbuomwcydxwbszd"
+        EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
+        if not EMAIL_PASSWORD:
+            print("EMAIL_PASSWORD not set — refusing to send email", file=sys.stderr)
+            sys.exit(1)
        GMAIL_EMAIL = "jtabiri@gmail.com"
        
        server = smtplib.SMTP("smtp.gmail.com", 587)
@@ -7,9 +7,11 @@
 # agent leg is retired — see the note after the Tanko leg.
 set -euo pipefail

+# Credentials sourced from environment variable ZULIP_API_KEY (set by vault-backed start script)
+# Never fall back to a literal key
+ZULIP_API_KEY="${ZULIP_API_KEY:?ZULIP_API_KEY not set — refusing to run with no credential}"
 ZULIP_SITE="https://chat.sysloggh.net"
 ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
-ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8"
 OWNER_ZULIP_ID="9"


@@ -27,12 +29,12 @@ notify() {
  local form
  form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
  curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
+    -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
    -d "${form}" > /dev/null 2>&1 || true
  # Zulip stream post to #agent-hub on topic 'zulip-health'
  local stream_content="${severity} Zulip Monitor: ${msg}"
  curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
+    -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
    -d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
    > /dev/null 2>&1 \
    || echo "  WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)" >> "$LOG"
@@ -41,7 +43,7 @@ notify() {
 # ── Global: Zulip Server ──
 SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
  https://chat.sysloggh.net/api/v1/server_settings \
-  -u 'abiba-bot@chat.sysloggh.net:cKTDMZAPW08dk3zl05sStzO7HRztzyn8' 2>/dev/null) || SERVER_CODE="000"
+  -u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" 2>/dev/null) || SERVER_CODE="000"
 SERVER_CODE=$(printf '%s' "$SERVER_CODE" | tr -d '[:space:]')
 [ -n "$SERVER_CODE" ] || SERVER_CODE="000"
 if [ "$SERVER_CODE" != "200" ]; then
@@ -27,7 +27,7 @@ Agent (Hermes/pi) ──curl + X-API-Key──► Stirling-PDF (:8989) ──►
 | Base URL | `http://192.168.68.7:8989` |
 | Auth Method | API Key (header) |
 | Header Name | `X-API-Key` |
-| API Key | `adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88` |
+| API Key | `«vault: infrastructure/production STIRLING_API_KEY»` |
 | Key Source | `SECURITY_CUSTOMGLOBALAPIKEY` in `/opt/home_stack/docker-compose.yml` |
 | Swagger | `http://192.168.68.7:8989/swagger-ui.html` |
 | Health | `http://192.168.68.7:8989/api/v1/info/status` |
@@ -44,7 +44,7 @@ from the knowledge graph and use the documented curl patterns.
 Direct bash invocations:
 ```bash
 curl -X POST "http://192.168.68.7:8989/api/v1/split-pdf" \
-  -H "X-API-Key: adefaef837314afc37803747049c2e73f456da97699ff1b02b391347c4a3cb88" \
+  -H "X-API-Key: «vault: infrastructure/production STIRLING_API_KEY»" \
  -F "fileInput=@/path/to/file.pdf" \
  -F "pageNumbers=1,2,3" \
  -o /tmp/output.zip
Author	SHA1	Message	Date
root	1bfab85a25	docs: add PBS GC schedule to proxmox-monitor contract PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 12s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details - GC runs daily at 20:00 UTC (after backup window closes at 06:40Z) - Volume is /tank/pbs-backup on ZFS pool 'tank' (12.7T, 10% used) - NOT /media/easystore2 (media library, 3.7T, 96% HOST-RED) - Cron: /etc/cron.d/pbs-gc on storepve (192.168.68.6) - Last run: 2026-09-18 00:08:34 UTC (removed 551.768 GiB)	2026-09-18 04:46:44 +00:00
mumuni-bot	1137dd4582	Merge pull request 'feat: add MCP server URL validation to hermes-config-template contract' (#114 ) from fm/hermes-config-mcp-url-validation into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Merged by mumuni PR-review agent: CI green (5/5), diff verified, no secrets, audit script behaviorally tested.	2026-09-17 13:22:31 +00:00
abiba-bot	7400dfd833	fix: add MCP server checks to audit-hermes-config.py PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details Implement Rule 15 automated enforcement for MCP servers: - Validate MCP server URLs against known endpoints (ra-h-os, litellm) - Check for authentication headers on MCP server configs - Warn if header values look like env-vars instead of literal keys - Warn if no auth header is present This ensures the MCP URL/header invariants from the prose contract are enforced at the earliest shared boundary (before config application).	2026-09-17 12:33:52 +00:00
abiba-bot	c65f5219e1	fix: address review findings for MCP URL validation Addressed all 4 ask-user findings from the review: f1: Qualified the MCP access verification claim - noted that it may contradict infrastructure-update.prose.md and that LiteLLM version may have been upgraded since that contract was written. f2: Added key rotation note documenting that MCP headers use literal keys and do NOT auto-rotate with the vault. Added TODO to consider adding MCP header regeneration to the Key Update Procedure. f3: Added MCP server checks to audit-hermes-config.py (Rule 15): - Validate MCP server URLs against known endpoints - Check for authentication headers - Warn if header values look like env-vars instead of literal keys f4: Updated Rule 15 verification instruction to include MCP initialize handshake test, not just /v1/models check. f5: Added NetBird dependency note documenting that 502 errors on MCP requests may indicate NetBird outage, not auth failure.	2026-09-17 12:24:43 +00:00
abiba-bot	077972fa2b	docs: add MCP verification details and Accept header note - Documented MCP endpoint verification (2026-08-07): tested with real key, confirmed initialize handshake works and virtual keys have MCP access - Added note about Accept header requirement (handled by MCP client library) - Clarified that the Accept header is NOT part of the config template	2026-09-17 12:19:00 +00:00
abiba-bot	d2bca5405a	feat: add litellm MCP server entry and enhance Rule 15 validation - Added litellm MCP server entry to mcp_servers section with correct URL (https://litellm.sysloggh.net/mcp) and header format - Updated Rule 15 to be more specific about endpoint validation and header requirements (REAL keys, not env-var references) - Added MCP Server Configuration section with implementation details - Documented the 2026-08-07 Tanko incident where ra-h-os was pointing to litellm endpoint with env header causing 401 floods - Updated frontmatter to reflect the changes Fixes: #keyless-mcp-incident-20260807 Refs: Rule 15 (MCP Endpoint and Header Validation)	2026-09-17 11:30:44 +00:00
abiba-bot	8a5cba8515	Merge pull request 'security(secrets): remove committed credentials from the tree and read them from the vault/environment' (#112 ) from fix/monitor-creds-to-env-master-20260910 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 10s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 10s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-09-17 07:15:07 +00:00
root	20f882412f	PR #112 round 2: fix syntax error, restore docs, clean residual credentials PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 10s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 16s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s Details	2026-09-17 07:06:02 +00:00
root	30b2fe3fdc	Fix PR #112 security review - restore docs, remove live credentials PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 10s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details	2026-09-17 06:51:52 +00:00
root	5112c566c8	Remove Stirling PDF credentials (password + API key) from 2 files PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 15s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 8s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 12s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details	2026-09-17 06:40:59 +00:00
root	83307eb9b2	Annotate deprecated key in litellm-self-heal.prose.md as not live	2026-09-17 06:12:18 +00:00
root	8245716286	Remove all hardcoded credentials from repository Audit results (all patterns checked across .md, .prose.md, .sh, .py, .js, .ts, .json, .yaml, .yml, .env): - sk-or-v1 (OpenRouter): 0 occurrences - sk- prefix (20+ chars): 0 occurrences - sk_live: 0 occurrences - Bearer <key>: 0 occurrences - api_key: <value>: 0 occurrences - PASSWORD=: 0 occurrences - TOKEN=: 0 occurrences - SECRET=: 0 occurrences Files changed: - agent-zero-fix-summary.md (removed 2 OpenRouter keys) - agent-zero-openrouter-key.prose.md (removed 1 OpenRouter key) - hermes-key-enforcement.prose.md (removed 1 LiteLLM key, 1 external key) - litellm-api-keys.prose.md (removed 1 LiteLLM key) - litellm-self-heal.prose.md (removed 1 stale key reference) - scripts/agent-health-check.py (INFISICAL_TOKEN now required) - scripts/daily-infra-report.py (EMAIL_PASSWORD now required) - zulip-health.prose.md (TOKEN references annotated)	2026-09-17 06:11:42 +00:00
root	cfb6c03572	Fix remaining hardcoded ZULIP_KEY in zulip-monitor.sh (line 46)	2026-09-17 05:58:58 +00:00
root	85f70f65bc	Remove hardcoded ZULIP_KEY from monitoring scripts Scripts that had hardcoded credentials: - scripts/zulip-monitor.sh:12 (was ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8") - scripts/daily-infra-report.py:25 (was ZULIP_KEY="cKTDMZAPW08dk3zl05sStzO7HRztzyn8") Both now read from environment variable ZULIP_API_KEY (set by vault-backed start script) with loud failure if not present. Other credentials in scripts/: - capture-dsh-token.sh: uses TOKEN variable with fallback (not a secret) - pm2-self-heal.sh: reads TELEGRAM_BOT_TOKEN from /root/.pi/agent/extensions/telegram/.env (acceptable) - prose-ai-review.sh: uses GITEA_TOKEN from .env file with LITELLM_KEY fallback (not secrets) No other hardcoded credentials found. Proof of behavior: With ZULIP_API_KEY set: bash scripts/zulip-monitor.sh → Server: HTTP 200 (authenticated) python3 scripts/daily-infra-report.py --json → Collecting infrastructure data... Without ZULIP_API_KEY set: bash scripts/zulip-monitor.sh → "ZULIP_API_KEY not set — refusing to run with no credential" python3 scripts/daily-infra-report.py --json → "ZULIP_API_KEY not set — refusing to run with no credential" Cred source: environment variable ZULIP_API_KEY (set by vault-backed start script) No key rotation (that is a separate decision).	2026-09-17 05:51:30 +00:00
abiba-bot	a820b3f7dd	Merge pull request 'docs(agent-health): every check leg must appear in every report - a missing line is not a pass' (#111 ) from fix/agent-health-mandatory-report-legs-20260917 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-09-17 03:12:57 +00:00