diff --git a/agent-zero-fix-summary.md b/agent-zero-fix-summary.md index e0484b0..98ad44a 100644 --- a/agent-zero-fix-summary.md +++ b/agent-zero-fix-summary.md @@ -140,7 +140,7 @@ Added section: | Component | Status | Details | |-----------|--------|---------| -| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY» user verified | +| **OpenRouter Key** | ✅ Valid | `«vault: agents/production OPENROUTER_API_KEY»` user verified, | | **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared | | **MCP Services** | ✅ Working | No timeouts after key fix | | **Container** | ✅ Running | PID 3320, uptime 16+ hours | diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index 1343698..d2320d0 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -101,7 +101,7 @@ auxiliary: fallback_providers: - provider: deepseek base_url: https://api.deepseek.com - api_key: «vault: external/production LITELLM_API_KEY» # ← hardcoded OK (external) OK (external) + api_key: «vault: external/production LITELLM_API_KEY» # ← hardcoded OK (external) api_key_env: DEEPSEEK_API_KEY # ← also OK if set in environment (vault or /etc/environment) ``` @@ -109,7 +109,7 @@ fallback_providers: # ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom) model: provider: harness - api_key: «vault: agents/production LITELLM_API_KEY» # ← RULE VIOLATION: hardcoded key + api_key: sk-synthetic-example-12345 # ← RULE VIOLATION: hardcoded key (synthetic example) model: provider: harness diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index 58c21dd..7144fb0 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -636,7 +636,7 @@ monitor, or integration breaks. ```bash # Full cluster status PVE="https://minipve.sysloggh.net" -AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3" +AUTH="Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»" curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH" # Docker health from Abiba diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index 90d9eaf..0f7fb22 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -145,7 +145,7 @@ through its agent wrapper. Example: ```bash MUMUNI_LITELLM_API_KEY=«vault: agents/production LITELLM_API_KEY» - MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt + MUMUNI_ZULIP_API_KEY=«vault: agents/production ZULIP_API_KEY» ``` 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`: ```ini @@ -191,7 +191,21 @@ through its agent wrapper. ### Tanko migration (COMPLETED 2026-07-17) Tanko was the last agent migrated from hardcoded keys to vault wrapper. -Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: «vault: agents/production LITELLM_API_KEY»"$VENV/bin/python"` +Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`) +and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in +`50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at +`~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault. + +### Koby migration lessons (2026-07-16, updated 2026-07-17) + +Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper. +**Three mistakes made:** +1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed + in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were + inherited from the pre-migration gateway env, not stored in any file. Lost on restart. +2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials. + Agents need ALL their platform env vars. Missing vars cause silent adapter failures. +3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"` inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path. **How Koby actually connects:** @@ -257,13 +271,13 @@ not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update ma UI bootstrap, model selection) is built around OpenRouter's native authentication. **Key Storage:** -- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»`) +- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=«vault: agents/production OPENROUTER_API_KEY»…`) - **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production) - **Fallback**: The container's .env is the primary source; vault sync is optional (unlike fleet agents which require vault injection) **Current Key (2026-09-01):** -- **Prefix**: `«vault: agents/production OPENROUTER_API_KEY»` +- **Prefix**: `«vault: agents/production OPENROUTER_API_KEY»`…` - **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` - **Plan**: Paid (not free tier) - **Usage**: 0 (as of 2026-09-01) diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 4f84cc7..6ea155a 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -116,7 +116,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu- - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`). - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault). - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is **report-only** (captain's 2026-08-17 ruling) — its findings go to the `--json` `report_only` array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (`script=` + `cwd=`). The current fleet roster is owned by the script changelog (`scripts/agent-health-check.py`); mumuni is no longer probed from this host. Legacy `tdunna`/`baggy` replaced with canonical agent hostnames. -- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`«vault: agents/production LITELLM_API_KEY»` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM (deprecated key, no live usage). +- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (hardcoded key → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM (deprecated key, no live usage). ## Maintains diff --git a/scripts/agent-health-check.py b/scripts/agent-health-check.py index 1e6797c..62b1f06 100755 --- a/scripts/agent-health-check.py +++ b/scripts/agent-health-check.py @@ -123,19 +123,13 @@ def _fail(key, agent_name=None): INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN") if not INFISICAL_TOKEN: - print("INFISICAL_TOKEN not set — refusing to run with no credential", file=sys.stderr) - sys.exit(1) -INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net") - -# Fallback: if no env token, read the shared vault token file -if not INFISICAL_TOKEN: + # Fallback: read the shared vault token file _token_path = os.path.expanduser("~/.infisical-token") if os.path.isfile(_token_path): try: with open(_token_path) as _f: INFISICAL_TOKEN = _f.read().strip() - except (OSError, UnicodeDecodeError): - pass +INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net") # ── Helpers ────────────────────────────────────────────────────────── diff --git a/scripts/daily-infra-report.py b/scripts/daily-infra-report.py index 486b05e..e9cba4e 100755 --- a/scripts/daily-infra-report.py +++ b/scripts/daily-infra-report.py @@ -16,7 +16,7 @@ from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart PVE = "https://192.168.68.12:8006" -AUTH = "Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3" +AUTH = "Authorization: PVEAPIToken=«vault: infrastructure/production PVE_API_TOKEN»" # ── Shared credentials —─ diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 3410d46..54229f8 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -410,7 +410,7 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \ # Expected: 000 # 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to). -TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") # ← source: dsh-web launch-token (retrieved from CT 112) +TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token") ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \ -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'" ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \ @@ -446,7 +446,7 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \ # manual run may no-op on the flock, so poll until the include carries a token # the running process accepts (bounded wait) before the mint+reuse check. for i in $(seq 1 60); do - TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'") # ← source: dsh-web launch-token (retrieved from CT 112) + TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'") CODE=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \ -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'") [ "$CODE" = "303" ] && break