prose-contracts/litellm-api-keys.prose.md

kind, name, description

kind	name	description
function	litellm-api-keys	Manages LiteLLM API keys for agent identity. Creates named keys so each agent is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry) and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026"). Ensures agents never use the master key directly. Rotation is event-driven, not calendar-driven — rotate only on compromise, personnel change, or periodic security hygiene (quarterly/annually). UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production) BUT each agent host MUST keep a local .env fallback. Infisical service tokens can expire/404. The .env fallback prevents agents from running without keys. Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours. UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed. Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. Abiba's key is now a proper agent key (NOT the master key — stale note removed). UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo) standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) → infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key injection → .env fallback → exec python. Systemd drop-ins are IMMUNE to hermes gateway install which overwrites the unit file ExecStart. Infisical CLI updated to 0.43.109 on all agents (was 0.38.0). Service token st.8e848433 shared across fleet (st.353699cd for tanko was deleted). .env fallback on every agent protects against token loss. Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently. (3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set. Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml on CT 116. Last verified: 2026-07-17.

Parameters

• agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
• action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
• litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
• master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
• vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
• vault_project: string — Infisical project slug (default: "infrastructure")
• vault_env: string — Infisical environment (default: "production")
• agent_host: string — Agent's IP for SSH (default: resolved from infra)
• agent_user: string — SSH user (default: "jerome")

Returns

• action: string — What was done
• key_alias: string — The LiteLLM key alias created/rotated
• key_prefix: string — First 10 chars of the new key (for identification)
• previous_key_alias: string | null — Previous key alias if rotating
• litellm_response: object — Raw response from LiteLLM /key/generate
• vault_updated: boolean — Whether Infisical vault secret was updated
• agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
• verification: { status: string, detail: string } — Final health check

Execution

1. Authenticate — Retrieve master key from Infisical vault via infisical export --project=<vault_project> --env=<vault_env>, verify against LiteLLM /key/list
2. Check existing keys — List all keys, find any with agent_name alias
3. If action == "list": Return all keys with their aliases and spend
4. If action == "create":
   • Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   • Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   • Duration is null (permanent) — inherited from litellm default_key_generate_params
   • Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
   • Note: ornith-1.0-35b is NOT a valid LiteLLM model name (use strix-moe, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
   • Return the new key
5. If action == "rotate":
   • Generate new key with same alias (LiteLLM replaces the old key)
   • Update secret in Infisical vault: infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>
   • Restart agent gateway (Hermes: systemctl restart hermes-gateway; pi: restart PM2 process) The gateway automatically picks up the new key via infisical run -- wrapper
   • Verify: curl test against /v1/models with new key
   • Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
   • Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
6. If action == "verify":
   • Retrieve key from Infisical vault: infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>
   • Test the key against LiteLLM /v1/models
   • Confirm key alias matches agent_name in LiteLLM key list
   • Verify agent gateway uses vault wrapper: cat /proc/<pid>/cmdline shows infisical run

Production Vault Access Process (canonical, 2026-07-17)

The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents (Mumuni, Tanko, Koby, Koonimo) as of 2026-07-17. Abiba (pi) uses a similar pattern through its agent wrapper.

The canonical pattern

1. infisical CLI installed on the host at /usr/bin/infisical (v0.43.109+, from artifacts-cli.infisical.com apt repo). Update procedure:

   curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | sudo -E bash
   sudo apt-get update && sudo apt-get install -y infisical
   # Remove stale old binary if present
   rm -f /usr/local/bin/infisical /bin/infisical
   

   Wrappers use absolute path /usr/bin/infisical run. Never rely on PATH resolution.

2. Service token (Infisical Machine Identity, st.…) stored at ~/.infisical-token (chmod 600). Current: shared st.8e848433… (abiba, READ+WRITE on agents project). Tanko's st.353699cd… (tanko-agent) was deleted — reverted to shared token. Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).

3. infisical-gateway.sh wrapper at ~/.hermes/infisical-gateway.sh (chmod 700):

   #!/bin/bash
   export INFISICAL_API_URL="https://vault.sysloggh.net"
   TOKEN=$(cat $HOME/.infisical-token)
   LOG=$HOME/.hermes/logs/gateway.log; mkdir -p $HOME/.hermes/logs
   while true; do
     echo "[$(date -Iseconds)] Starting gateway with Infisical injection..." >> $LOG
     /usr/bin/infisical run --token="$TOKEN" \
       --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
         . $HOME/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3]
         export LITELLM_API_KEY="${<AGENT>_LITELLM_API_KEY}"
         export ZULIP_API_KEY="${<AGENT>_ZULIP_API_KEY}"
         export ZULIP_SITE="https://chat.sysloggh.net"
         export ZULIP_EMAIL="<agent>-bot@chat.sysloggh.net"
         export SEARXNG_URL="http://192.168.68.7:8888"
         # ⚠️  HARDCODE the full venv path. NEVER use $VENV inside single quotes.
         exec /root/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
       ' >> $LOG 2>&1
     EXIT_CODE=$?
     echo "[$(date -Iseconds)] Gateway exited with code $EXIT_CODE — restarting in 5s..." >> $LOG
     sleep 5
   done
   

   CRITICAL: VENV PATH. The inner bash -c '...' uses single quotes. Shell variables set in the outer wrapper are NOT expanded inside single quotes. $VENV/bin/python resolves to /bin/python (file not found). Always hardcode the absolute path to the venv python binary.

4. Agent key in vault as <AGENT>_LITELLM_API_KEY and <AGENT>_ZULIP_API_KEY. Vault = source of truth for ALL platform credentials.

5. .env fallback at ~/.hermes/.env (chmod 600) with agent-specific keys — safety net for vault outage or token revocation. Must be kept in sync on rotation. Example:

   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
   

6. systemd drop-in at ~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf:

   [Service]
   ExecStart=
   ExecStart=/root/.hermes/infisical-gateway.sh
   

   The ExecStart= (empty reset) clears any ExecStart from the main unit file, then the second ExecStart= sets the wrapper. This drop-in survives unit file regeneration by hermes gateway install — the drop-in always wins.

   Why a drop-in instead of editing the unit file: hermes gateway install (called during Hermes updates and some self-heal operations) regenerates the systemd unit file with ExecStart=/path/to/python -m hermes_cli.main gateway run. Editing the unit file directly is futile — it will be overwritten. The drop-in approach explicitly resets ExecStart and sets the wrapper regardless of what the main unit file says.

7. NEVER hardcode API keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault at every start.

Why this is non-fail

• No rot: keys pulled live from vault at every gateway start. Rotation = one infisical secrets set + systemctl restart. No per-host file edits.
• Survives vault outage: the .env fallback (Rule 3) keeps the gateway running if Infisical is unreachable or the service token is revoked.
• Survives gateway crash: the wrapper's while true + systemd Restart=always revive the gateway. Two-layer defense.
• Survives Hermes updates: systemd drop-in overrides unit file ExecStart — hermes gateway install cannot break the vault injection.
• Survives reboot: systemd user service + loginctl enable-linger ensures gateway starts at boot without a login session.
• Auditable: cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); infisical secrets shows the vault source.

Migration status (2026-07-17)

Agent	Host	Pattern	Keys	Status
abiba	.24	pi agent wrapper	ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY	✅ vault-backed
mumuni	.24 (CT100 abiba)	Pi Hermes gateway (no systemd)	MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY	✅ vault-backed + .env fallback
tanko	.122	systemd drop-in + while-true wrapper + st.8e848433 (user jerome)	TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY	✅ vault-backed + .env fallback
koby	.129	systemd drop-in + while-true wrapper + st.8e848433	KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot)	✅ vault-backed
koonimo	.114	systemd drop-in + while-true wrapper + st.8e848433	KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY	✅ vault-backed

Tanko runs as user jerome — wrapper/token at ~/.hermes/infisical-gateway.sh and ~/.infisical-token. Linger enabled (loginctl enable-linger jerome) for boot startup.

Tanko migration (COMPLETED 2026-07-17)

Tanko was the last agent migrated from hardcoded keys to vault wrapper. Previously: key hardcoded in /home/jerome/.hermes/config.yaml (api_key: sk-CggiHWlamQy…) and zulip-env.conf systemd drop-in. Now: user-scope systemd service with drop-in 50-vault-wrapper.conf, infisical-gateway.sh wrapper with while-true loop, token at ~/.infisical-token, .env fallback at ~/.hermes/.env. Keys injected live from vault.

Koby migration lessons (2026-07-16, updated 2026-07-17)

Migrated Koby from hardcoded systemd drop-in → infisical-gateway.sh wrapper. Three mistakes made:

1. Overwrote /root/.hermes/.env without backing it up. The Zulip API key only existed in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
2. Only injected LITELLM_API_KEY in the wrapper — forgot Zulip + Telegram credentials. Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
3. (2026-07-17 fix) VENV variable in single-quoted bash -c: exec "$VENV/bin/python" inside single quotes resolved to exec "/bin/python" (file not found). Hardcoded full path.

How Koby actually connects:

• Zulip: shares Tanko's bot (tanko-bot@chat.sysloggh.net, TANKO_ZULIP_API_KEY=5PeD6f3zo…).
• Telegram: token from .env fallback. Allowed users: 6679773481.
• Both platforms now connect through the wrapper's env injection.

Golden rules for gateway restarts:

1. Always cat /proc/<pid>/environ before killing the old process — captures the live env set.
2. Hardcode venv python path in wrapper — never use variables inside single-quoted bash -c.
3. Use systemd drop-ins (not unit file edits) to override ExecStart — survives Hermes updates.

Fleet-wide standardization lessons (2026-07-17)

After auditing all 4 agents, five systemic patterns caused repeated failures:

1. Three incompatible startup patterns coexisted (systemd drop-in, direct python, orphaned wrapper)
2. Systemd unit files reverted by hermes gateway install during updates
3. VENV variable scoping broke wrappers on Koby and Mumuni (single-quote bash -c)
4. Service token expiry — Tanko's st.353699cd was deleted from Infisical
5. No ZULIP_API_KEY in env on Tanko — wrapper bypassed by systemd direct python

All resolved by the canonical drop-in + while-true wrapper pattern documented above.

Key rotation procedure (one vault operation with this standard)

1. Generate new key: POST /key/generate (master key, admin).
2. Update vault: infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net.
3. Update .env fallback: echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env.
4. Restart: systemctl restart hermes-gateway. The wrapper pulls the new key live.
5. Verify: curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models → 200.

Machine Identity for Vault Writes (UPDATED 2026-07-17)

Current state: Infisical CLI updated to v0.43.109 on all agents (from v0.38.0). The v0.38.0 bug (user-session auth fails for secrets set/export) is resolved. Service token st.8e848433… (abiba, READ+WRITE) can write to vault from CLI.

Proper fix — per-agent Machine Identities: Create machine identities in Infisical UI → Project Settings → Machine Identities for each agent with READ-only scope on the agents project. Store client_id + client_secret per agent. Then vault writes use the shared abiba identity, and reads use per-agent identities. This eliminates the single shared token risk.

Service Token Inventory (2026-07-17):

Token ID	Name	Permissions	Used By	Status
st.8e848433…	tanko-gateway	READ+WRITE	Mumuni, Tanko, Koby, Koonimo, Abiba	✅ Active
st.353699cd…	tanko-agent	READ-only	—	❌ Deleted from Infisical

Per-agent .env fallback inventory (2026-07-17):

Agent	.env Keys
Mumuni	MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY
Tanko	TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY
Koby	(wrapper injects from vault — .env has Telegram token)
Koonimo	KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY

Key Rotation Log

Date	Agent	Action	Notes
2026-07-17	fleet	standardize	All 4 agents standardized on systemd drop-in + while-true wrapper + infisical v0.43.109. Removed conflicting zulip-env.conf + litellm-key.conf drop-ins. Added .env fallbacks with ZULIP keys. WAL #1322.
2026-07-17	tanko	fix-zulip	Added ZULIP_API_KEY to env (was missing — systemd bypassed vault). Updated wrapper from exec to while-true. Created .env fallback. Removed hardcoded zulip-env.conf drop-in. WAL #1321.
2026-07-16	vault	cleanup	4 stale secrets deprecated. 5 personal creds flagged.
2026-07-16	koonimo	add-zulip	Added KOONIMO_ZULIP_API_KEY to vault. Wrapper injects ZULIP_API_KEY + ZULIP_EMAIL. 3 platforms.
2026-07-16	tanko	migrate	Migrated from hardcoded config.yaml to infisical-gateway.sh + st.353699cd. NOTE: st.353699cd later deleted — reverted to st.8e848433 on 2026-07-17.
2026-07-16	mumuni	rotate	Old key malformed (sk-SWAl_Vu, 47 chars, not LiteLLM format) → 401. Deleted old mumuni key (token 15cbca18…), generated fresh (alias mumuni, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300.
2026-07-16	koby	rotate	Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old koby key, generated fresh (alias koby). New key sk-BqRRMboTI… in systemd drop-in hermes-gateway.service.d/litellm-key.conf + /etc/environment. Created hermes-gateway.service unit (was missing — gateway wasn't persistent) with --replace. Verified HTTP 200, Telegram connected.
2026-07-16	baggy (koonimo)	rotate	Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old baggy key, generated fresh (alias baggy, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in hermes-gateway.service.d/litellm-key.conf. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected.

LiteLLM Master Key (use sparingly — agents should NOT use it directly)

• Master key: sk-litellm-7f96080dd99b15c36bd4b333b58a6796 (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
• Used for /key/generate, /key/delete, /key/list (GET), DB queries
• Known violation (RESOLVED 2026-07-16): Abiba's LITELLM_API_KEY was previously the master key. It is now a dedicated agent key sk-sxbphLvk1OU… (vault secret ABIBA_LITELLM_API_KEY, alias abiba-pi). The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference — see litellm-self-heal § "NEVER use litellm_proxy_master_key for inference".
• LiteLLM key DB: harness-postgres container on CT116, table "LiteLLM_VerificationToken" (columns: token, key_alias, key_name, created_at, expires). Query: docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"