diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index 8575d24..f8b6ef6 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -20,9 +20,20 @@ description: > Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. Abiba's key is now a proper agent key (NOT the master key — stale note removed). + UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo) + standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) → + infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key + injection → .env fallback → exec python. Systemd drop-ins are IMMUNE to hermes gateway + install which overwrites the unit file ExecStart. Infisical CLI updated to 0.43.109 on + all agents (was 0.38.0). Service token st.8e848433 shared across fleet (st.353699cd + for tanko was deleted). .env fallback on every agent protects against token loss. + Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers + — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently. + (3) Capture /proc//environ before gateway restarts to preserve running env set. + Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml - on CT 116. Last verified: 2026-07-16. + on CT 116. Last verified: 2026-07-17. --- ## Parameters @@ -74,88 +85,147 @@ description: > - Confirm key alias matches agent_name in LiteLLM key list - Verify agent gateway uses vault wrapper: `cat /proc//cmdline` shows `infisical run` -## Production Vault Access Process (canonical, 2026-07-16) +## Production Vault Access Process (canonical, 2026-07-17) -The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending — -runs as user `jerome`, not systemd root, needs user-scope adaptation). +The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents +(Mumuni, Tanko, Koby, Koonimo) as of 2026-07-17. Abiba (pi) uses a similar pattern +through its agent wrapper. ### The canonical pattern -1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`). -2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`). - - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project. - - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities). -3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`): +1. **infisical CLI** installed on the host at `/usr/bin/infisical` (v0.43.109+, from + artifacts-cli.infisical.com apt repo). Update procedure: + ```bash + curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | sudo -E bash + sudo apt-get update && sudo apt-get install -y infisical + # Remove stale old binary if present + rm -f /usr/local/bin/infisical /bin/infisical + ``` + Wrappers use absolute path `/usr/bin/infisical run`. Never rely on PATH resolution. +2. **Service token** (Infisical Machine Identity, `st.…`) stored at `~/.infisical-token` + (`chmod 600`). Current: shared `st.8e848433…` (abiba, READ+WRITE on agents project). + Tanko's `st.353699cd…` (tanko-agent) was deleted — reverted to shared token. + Proper: one machine identity per agent (create in Infisical UI → Project Settings → + Machine Identities). +3. **`infisical-gateway.sh` wrapper** at `~/.hermes/infisical-gateway.sh` (`chmod 700`): ```bash #!/bin/bash export INFISICAL_API_URL="https://vault.sysloggh.net" - TOKEN=$(cat /root/.infisical-token) - LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs + TOKEN=$(cat $HOME/.infisical-token) + LOG=$HOME/.hermes/logs/gateway.log; mkdir -p $HOME/.hermes/logs while true; do - infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \ + echo "[$(date -Iseconds)] Starting gateway with Infisical injection..." >> $LOG + /usr/bin/infisical run --token="$TOKEN" \ + --projectId=322fceab-39da-4854-a55a-568e76c0f13f \ --env=prod --domain=https://vault.sysloggh.net -- bash -c ' - . /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only - export LITELLM_API_KEY="$_LITELLM_API_KEY" - exec /bin/python -m hermes_cli.main gateway run + . $HOME/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] + export LITELLM_API_KEY="${_LITELLM_API_KEY}" + export ZULIP_API_KEY="${_ZULIP_API_KEY}" + export ZULIP_SITE="https://chat.sysloggh.net" + export ZULIP_EMAIL="-bot@chat.sysloggh.net" + export SEARXNG_URL="http://192.168.68.7:8888" + # ⚠️ HARDCODE the full venv path. NEVER use $VENV inside single quotes. + exec /root/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run ' >> $LOG 2>&1 - sleep 5 # restart on exit + EXIT_CODE=$? + echo "[$(date -Iseconds)] Gateway exited with code $EXIT_CODE — restarting in 5s..." >> $LOG + sleep 5 done ``` -4. **Agent key in vault** as `_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth. -5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation. -6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot). -7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault. + **CRITICAL: VENV PATH.** The inner `bash -c '...'` uses single quotes. Shell + variables set in the outer wrapper are NOT expanded inside single quotes. + `$VENV/bin/python` resolves to `/bin/python` (file not found). Always hardcode + the absolute path to the venv python binary. +4. **Agent key in vault** as `_LITELLM_API_KEY` and `_ZULIP_API_KEY`. + Vault = source of truth for ALL platform credentials. +5. **`.env` fallback** at `~/.hermes/.env` (`chmod 600`) with agent-specific keys — + safety net for vault outage or token revocation. Must be kept in sync on rotation. + Example: + ```bash + MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw + MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt + ``` +6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`: + ```ini + [Service] + ExecStart= + ExecStart=/root/.hermes/infisical-gateway.sh + ``` + The `ExecStart=` (empty reset) clears any ExecStart from the main unit file, + then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file + regeneration** by `hermes gateway install` — the drop-in always wins. + + **Why a drop-in instead of editing the unit file:** `hermes gateway install` + (called during Hermes updates and some self-heal operations) regenerates the + systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`. + Editing the unit file directly is futile — it will be overwritten. The drop-in + approach explicitly resets ExecStart and sets the wrapper regardless of what the + main unit file says. +7. **NEVER hardcode** API keys in systemd drop-ins, config.yaml, or /etc/environment. + The wrapper injects live from vault at every start. ### Why this is non-fail - **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits. -- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable. -- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway. -- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source. +- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable or the service token is revoked. +- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense. +- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection. +- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session. +- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source. -### Migration status (2026-07-16) +### Migration status (2026-07-17) -| Agent | Host | Pattern | Vault key | Status | -|-------|------|---------|-----------|--------| -| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed | -| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed | -| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram | -| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed | -> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent. -<<<<<<< HEAD -| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed | -======= -| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper | ->>>>>>> origin/master +| Agent | Host | Pattern | Keys | Status | +|-------|------|---------|------|--------| +| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed | +| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback | +| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback | +| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed | +| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed | -### Tanko migration (pending) +> Tanko runs as user `jerome` — wrapper/token at `~/.hermes/infisical-gateway.sh` and +> `~/.infisical-token`. Linger enabled (`loginctl enable-linger jerome`) for boot startup. -Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in -`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced). -Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`) -with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering -enabled (`loginctl enable-linger jerome`) so the user service runs without a login session. +### Tanko migration (COMPLETED 2026-07-17) -### Koby migration lessons (2026-07-16) +Tanko was the last agent migrated from hardcoded keys to vault wrapper. +Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`) +and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in +`50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at +`~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault. + +### Koby migration lessons (2026-07-16, updated 2026-07-17) Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper. -**Two mistakes I made that broke the agent:** +**Three mistakes made:** 1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were inherited from the pre-migration gateway env, not stored in any file. Lost on restart. 2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials. Agents need ALL their platform env vars. Missing vars cause silent adapter failures. +3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"` + inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path. -**How Koby actually connects (2026-07-16):** +**How Koby actually connects:** - Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`). - Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config). -- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026). - Allowed users: 6679773481. Home channel: 6679773481. +- Telegram: token from `.env` fallback. Allowed users: 6679773481. - Both platforms now connect through the wrapper's env injection. -**Golden rule for gateway restarts:** always `cat /proc//environ` before killing the old -process — captures the live env set. Especially important when migrating gateways between -injection mechanisms. +**Golden rules for gateway restarts:** +1. Always `cat /proc//environ` before killing the old process — captures the live env set. +2. Hardcode venv python path in wrapper — never use variables inside single-quoted bash -c. +3. Use systemd drop-ins (not unit file edits) to override ExecStart — survives Hermes updates. + +### Fleet-wide standardization lessons (2026-07-17) + +After auditing all 4 agents, five systemic patterns caused repeated failures: +1. **Three incompatible startup patterns** coexisted (systemd drop-in, direct python, orphaned wrapper) +2. **Systemd unit files reverted** by `hermes gateway install` during updates +3. **VENV variable scoping** broke wrappers on Koby and Mumuni (single-quote bash -c) +4. **Service token expiry** — Tanko's `st.353699cd` was deleted from Infisical +5. **No ZULIP_API_KEY** in env on Tanko — wrapper bypassed by systemd direct python + +All resolved by the canonical drop-in + while-true wrapper pattern documented above. ### Key rotation procedure (one vault operation with this standard) @@ -165,69 +235,41 @@ injection mechanisms. 4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live. 5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200. -## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300) +## Machine Identity for Vault Writes (UPDATED 2026-07-17) -**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com). -In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace -key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt -repo only ships 0.38.0, so `apt upgrade` does not help. +**Current state:** Infisical CLI updated to v0.43.109 on all agents (from v0.38.0). +The v0.38.0 bug (user-session auth fails for `secrets set`/`export`) is resolved. +Service token `st.8e848433…` (abiba, READ+WRITE) can write to vault from CLI. -**Proper fix — Machine Identity (Infisical automation best practice):** -Create a machine identity with READ+WRITE scope on the `agents` project (project_id= -`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely. -Then vault writes work from any host: -```bash -# Get a machine-identity access token -TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \ - -H 'Content-Type: application/json' \ - -d '{"clientId":"","clientSecret":""}' | jq -r .accessToken) -# Write a secret via REST API v3 - curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \ - -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d '{"environment":"prod","secretValue":"sk-","workspaceId":"","type":"shared"}' -# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ... -``` -<<<<<<< HEAD -**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE) -can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM. -4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from -hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored. +**Proper fix — per-agent Machine Identities:** +Create machine identities in Infisical UI → Project Settings → Machine Identities +for each agent with READ-only scope on the `agents` project. Store client_id + +client_secret per agent. Then vault writes use the shared abiba identity, and +reads use per-agent identities. This eliminates the single shared token risk. -**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI. -Client secret needed to use universal auth for vault writes from automation. +**Service Token Inventory (2026-07-17):** +| Token ID | Name | Permissions | Used By | Status | +|----------|------|-------------|---------|--------| +| `st.8e848433…` | tanko-gateway | READ+WRITE | Mumuni, Tanko, Koby, Koonimo, Abiba | ✅ Active | +| `st.353699cd…` | tanko-agent | READ-only | — | ❌ Deleted from Infisical | -**Service Token Inventory (2026-07-16):** -| Token ID | Name | Permissions | Used By | -|----------|------|-------------|---------| -| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo | -| `st.353699cd…` | tanko-agent | READ-only | Tanko | -======= -Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings → -Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity -and store its credentials in the vault itself (or a root-only file).** - -**Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The -infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `_LITELLM_API_KEY` -overrides a stale vault value. - -**2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE) -can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`, -koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` / -`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again. -Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix -so the shared service token isn't reused across hosts — but it is no longer blocking. ->>>>>>> origin/master +**Per-agent .env fallback inventory (2026-07-17):** +| Agent | .env Keys | +|-------|-----------| +| Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY | +| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY | +| Koby | (wrapper injects from vault — .env has Telegram token) | +| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY | ## Key Rotation Log | Date | Agent | Action | Notes | |------|-------|--------|-------| -<<<<<<< HEAD -| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. | -| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. | -| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. | -======= ->>>>>>> origin/master +| 2026-07-17 | fleet | standardize | All 4 agents standardized on systemd drop-in + while-true wrapper + infisical v0.43.109. Removed conflicting zulip-env.conf + litellm-key.conf drop-ins. Added .env fallbacks with ZULIP keys. WAL #1322. | +| 2026-07-17 | tanko | fix-zulip | Added ZULIP_API_KEY to env (was missing — systemd bypassed vault). Updated wrapper from exec to while-true. Created .env fallback. Removed hardcoded zulip-env.conf drop-in. WAL #1321. | +| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated. 5 personal creds flagged. | +| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault. Wrapper injects ZULIP_API_KEY + ZULIP_EMAIL. 3 platforms. | +| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml to infisical-gateway.sh + st.353699cd. NOTE: st.353699cd later deleted — reverted to st.8e848433 on 2026-07-17. | | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. | | 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. | | 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |