--- kind: function name: litellm-api-keys description: > Manages LiteLLM API keys for agent identity. Creates named keys so each agent is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry) and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026"). Ensures agents never use the master key directly. Rotation is event-driven, not calendar-driven — rotate only on compromise, personnel change, or periodic security hygiene (quarterly/annually). UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production) BUT each agent host MUST keep a local .env fallback. Infisical service tokens can expire/404. The .env fallback prevents agents from running without keys. Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours. UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed. Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. Abiba's key is now a proper agent key (NOT the master key — stale note removed). Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml on CT 116. Last verified: 2026-07-16. --- ## Parameters - agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni") - action: "create" | "rotate" | "verify" | "list" — What to do (default: "create") - litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000") - master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY) - vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net") - vault_project: string — Infisical project slug (default: "infrastructure") - vault_env: string — Infisical environment (default: "production") - agent_host: string — Agent's IP for SSH (default: resolved from infra) - agent_user: string — SSH user (default: "jerome") ## Returns - action: string — What was done - key_alias: string — The LiteLLM key alias created/rotated - key_prefix: string — First 10 chars of the new key (for identification) - previous_key_alias: string | null — Previous key alias if rotating - litellm_response: object — Raw response from LiteLLM /key/generate - vault_updated: boolean — Whether Infisical vault secret was updated - agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration) - verification: { status: string, detail: string } — Final health check ## Execution 1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project= --env=`, verify against LiteLLM /key/list 2. **Check existing keys** — List all keys, find any with agent_name alias 3. **If action == "list"**: Return all keys with their aliases and spend 4. **If action == "create"**: - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date) - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" } - Duration is null (permanent) — inherited from litellm default_key_generate_params - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"] - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed). - Return the new key 5. **If action == "rotate"**: - Generate new key with same alias (LiteLLM replaces the old key) - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY= --project= --env=` - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process) The gateway automatically picks up the new key via `infisical run --` wrapper - Verify: curl test against /v1/models with new key - Rotation policy: on-demand only (compromise, departure, quarterly hygiene) - Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper. 6. **If action == "verify"**: - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project= --env=` - Test the key against LiteLLM /v1/models - Confirm key alias matches agent_name in LiteLLM key list - Verify agent gateway uses vault wrapper: `cat /proc//cmdline` shows `infisical run` ## Production Vault Access Process (canonical, 2026-07-16) The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending — runs as user `jerome`, not systemd root, needs user-scope adaptation). ### The canonical pattern 1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`). 2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`). - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project. - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities). 3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`): ```bash #!/bin/bash export INFISICAL_API_URL="https://vault.sysloggh.net" TOKEN=$(cat /root/.infisical-token) LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs while true; do infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \ --env=prod --domain=https://vault.sysloggh.net -- bash -c ' . /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only export LITELLM_API_KEY="$_LITELLM_API_KEY" exec /bin/python -m hermes_cli.main gateway run ' >> $LOG 2>&1 sleep 5 # restart on exit done ``` 4. **Agent key in vault** as `_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth. 5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation. 6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot). 7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault. ### Why this is non-fail - **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits. - **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable. - **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway. - **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source. ### Migration status (2026-07-16) | Agent | Host | Pattern | Vault key | Status | |-------|------|---------|-----------|--------| | abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed | | mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed | | koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram | | koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed | > **Baggy = Koonimo (CT 113).** `BAGGY_LITELLM_API_KEY` and `KOONIMO_LITELLM_API_KEY` are the same agent under different aliases. Synced 2026-07-16: both now `sk-OEK7z26n6…`. TODO: delete `BAGGY_LITELLM_API_KEY` from vault (needs Infisical UI — shared secrets can't be deleted via service token). Only `KOONIMO_LITELLM_API_KEY` should exist long-term. | tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper | ### Tanko migration (pending) Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced). Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`) with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering enabled (`loginctl enable-linger jerome`) so the user service runs without a login session. ### Koby migration lessons (2026-07-16) Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper. **Two mistakes I made that broke the agent:** 1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were inherited from the pre-migration gateway env, not stored in any file. Lost on restart. 2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials. Agents need ALL their platform env vars. Missing vars cause silent adapter failures. **How Koby actually connects (2026-07-16):** - Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`). Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config). - Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026). Allowed users: 6679773481. Home channel: 6679773481. - Both platforms now connect through the wrapper's env injection. **Golden rule for gateway restarts:** always `cat /proc//environ` before killing the old process — captures the live env set. Especially important when migrating gateways between injection mechanisms. ### Key rotation procedure (one vault operation with this standard) 1. Generate new key: `POST /key/generate` (master key, admin). 2. Update vault: `infisical secrets set _LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`. 3. Update `.env` fallback: `echo '_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`. 4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live. 5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200. ## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300) **Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com). In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt repo only ships 0.38.0, so `apt upgrade` does not help. **Proper fix — Machine Identity (Infisical automation best practice):** Create a machine identity with READ+WRITE scope on the `agents` project (project_id= `322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely. Then vault writes work from any host: ```bash # Get a machine-identity access token TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \ -H 'Content-Type: application/json' \ -d '{"clientId":"","clientSecret":""}' | jq -r .accessToken) # Write a secret via REST API v3 curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \ -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ -d '{"environment":"prod","secretValue":"sk-","workspaceId":"","type":"shared"}' # OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ... ``` Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings → Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity and store its credentials in the vault itself (or a root-only file).** **Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `_LITELLM_API_KEY` overrides a stale vault value. **2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE) can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`, koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` / `KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again. Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix so the shared service token isn't reused across hosts — but it is no longer blocking. ## Key Rotation Log | Date | Agent | Action | Notes | |------|-------|--------|-------| | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. | | 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. | | 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. | ## LiteLLM Master Key (use sparingly — agents should NOT use it directly) - Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY) - Used for /key/generate, /key/delete, /key/list (GET), DB queries - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key. It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`). The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference — see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference". - LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`