2026-09-17 07:15:09 +00:00
8 changed files with 25 additions and 33 deletions
@@ -16,8 +16,8 @@ litellm.exceptions.AuthenticationError: OpenrouterException -
 ```
 **Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.

-**Old Key**: `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`  
-**New Key**: `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`  
+**Old Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
+**New Key**: `«vault: agents/production OPENROUTER_API_KEY»`  
 **New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`

 ### 2. Telegram Bot Conflict (CRITICAL)
@@ -48,14 +48,14 @@ McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 sec
 ```bash
 # Container .env update
 sudo docker exec agent-zero bash -c '
-sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=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|" /a0/usr/.env
+sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»|" /a0/usr/.env
 '
 ```

 **Verification**:
 ```bash
 curl -s https://openrouter.ai/api/v1/auth/key \
-  -H "Authorization: Bearer sk-or-v1-0af3f3..." | python3 -m json.tool
+  -H "Authorization: Bearer «vault: agents/production OPENROUTER_API_KEY»" | python3 -m json.tool
 ```
 Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.

@@ -140,7 +140,7 @@ Added section:

 | Component | Status | Details |
 |-----------|--------|---------|
-| **OpenRouter Key** | ✅ Valid | `sk-or-v1-0af3f3…`, user verified |
+| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY» user verified |
 | **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
 | **MCP Services** | ✅ Working | No timeouts after key fix |
 | **Container** | ✅ Running | PID 3320, uptime 16+ hours |
@@ -54,7 +54,7 @@ description: >
   ```

 4. **Return status**
-   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-0af", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
+   - If all checks pass: `{ key_status: "valid", key_prefix: "«vault: agents/production OPENROUTER_API_KEY»", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
   - If OpenRouter returns 401: `{ key_status: "invalid", detail: "User not found" }`
   - If vault secret is missing: `{ vault_synced: false }`

@@ -89,8 +89,8 @@ description: >

 | Field | Value |
 |-------|-------|
-| **Key Prefix** | `sk-or-v1-0af3f3` |
-| **Full Key** | `«redacted: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»` (in vault + /a0/usr/.env) |
+| **Key Prefix** | `«vault: agents/production OPENROUTER_API_KEY»` |
+| **Full Key** | `«redacted:«vault: agents/production OPENROUTER_API_KEY»»` (in vault + /a0/usr/.env) |
 | **OpenRouter User** | `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` |
 | **Free Tier** | No |
 | **Monthly Usage** | 0 (as of 2026-09-01) |
@@ -101,7 +101,7 @@ description: >

 | Date | Action | Notes |
 |------|--------|-------|
-| 2026-09-01 | fix-401 | Old key `sk-or-v1-036e5ca5…` returned 401 "User not found". Replaced with new key `sk-or-v1-0af3f3…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
+| 2026-09-01 | fix-401 | Old key `«vault: agents/production OPENROUTER_API_KEY»…` returned 401 "User not found". Replaced with new key `«vault: agents/production OPENROUTER_API_KEY»…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |

 ## Infrastructure References

@@ -101,7 +101,7 @@ auxiliary:
 fallback_providers:
  - provider: deepseek
    base_url: https://api.deepseek.com
-    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key: «vault: external/production LITELLM_API_KEY»   # ← hardcoded OK (external) OK (external)
    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```

@@ -109,7 +109,7 @@ fallback_providers:
 # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
 model:
  provider: harness
-  api_key: sk-Flc62smlegyMEaSo1ka8JA   # ← RULE VIOLATION: hardcoded key
+  api_key: «vault: agents/production LITELLM_API_KEY»   # ← RULE VIOLATION: hardcoded key

 model:
  provider: harness
@@ -144,7 +144,7 @@ through its agent wrapper.
   safety net for vault outage or token revocation. Must be kept in sync on rotation.
   Example:
   ```bash
-   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
+   MUMUNI_LITELLM_API_KEY=«vault: agents/production LITELLM_API_KEY»
   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
   ```
 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
@@ -191,21 +191,7 @@ through its agent wrapper.
 ### Tanko migration (COMPLETED 2026-07-17)

 Tanko was the last agent migrated from hardcoded keys to vault wrapper.
-Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
-and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
-`50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
-`~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
-
-### Koby migration lessons (2026-07-16, updated 2026-07-17)
-
-Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
-**Three mistakes made:**
-1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
-   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
-   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
-2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
-   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
-3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"`
+Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: «vault: agents/production LITELLM_API_KEY»"$VENV/bin/python"`
   inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path.

 **How Koby actually connects:**
@@ -271,13 +257,13 @@ not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update ma
 UI bootstrap, model selection) is built around OpenRouter's native authentication.

 **Key Storage:**
- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=sk-or-v1-…`)
+- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»`)
 - **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production)
 - **Fallback**: The container's .env is the primary source; vault sync is optional
  (unlike fleet agents which require vault injection)

 **Current Key (2026-09-01):**
- **Prefix**: `sk-or-v1-0af3f3…`
+- **Prefix**: `«vault: agents/production OPENROUTER_API_KEY»`
 - **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
 - **Plan**: Paid (not free tier)
 - **Usage**: 0 (as of 2026-09-01)
@@ -116,7 +116,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-
 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
 - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). The current fleet roster is owned by the script changelog (`scripts/agent-health-check.py`); mumuni is no longer probed from this host. Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
+- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`«vault: agents/production LITELLM_API_KEY»` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.

 ## Maintains

@@ -122,6 +122,9 @@ def _fail(key, agent_name=None):


 INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
+if not INFISICAL_TOKEN:
+    print("INFISICAL_TOKEN not set — refusing to run with no credential", file=sys.stderr)
+    sys.exit(1)
 INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")

 # Fallback: if no env token, read the shared vault token file
@@ -671,7 +671,10 @@ def send_email(html_content, subject_prefix=""):
    msg.attach(MIMEText(html_content, "html"))
    
    try:
-        EMAIL_PASSWORD = "rgbuomwcydxwbszd"
+        EMAIL_PASSWORD = os.environ.get("EMAIL_PASSWORD") or os.environ.get("SMTP_PASSWORD") or os.environ.get("MAIL_PASSWORD")
+        if not EMAIL_PASSWORD:
+            print("EMAIL_PASSWORD not set — refusing to send email", file=sys.stderr)
+            sys.exit(1)
        GMAIL_EMAIL = "jtabiri@gmail.com"
        
        server = smtplib.SMTP("smtp.gmail.com", 587)
@@ -410,7 +410,7 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
 # Expected: 000

 # 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to).
-TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token")
+TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") # ← source: dsh-web launch-token (retrieved from CT 112)
 ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
  -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
 ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
@@ -446,7 +446,7 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
 # manual run may no-op on the flock, so poll until the include carries a token
 # the running process accepts (bounded wait) before the mint+reuse check.
 for i in $(seq 1 60); do
-  TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
+  TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'") # ← source: dsh-web launch-token (retrieved from CT 112)
  CODE=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
    -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'")
  [ "$CODE" = "303" ] && break