prose-contracts/litellm-api-keys.prose.md

kind, name, description

kind	name	description
function	litellm-api-keys	Manages LiteLLM API keys for agent identity. Creates named keys so each agent is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry) and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026"). Ensures agents never use the master key directly. Rotation is event-driven, not calendar-driven — rotate only on compromise, personnel change, or periodic security hygiene (quarterly/annually). UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production) BUT each agent host MUST keep a local .env fallback. Infisical service tokens can expire/404. The .env fallback prevents agents from running without keys. Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours. UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed. Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. Abiba's key is now a proper agent key (NOT the master key — stale note removed). Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml on CT 116. Last verified: 2026-07-16.

Parameters

• agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
• action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
• litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
• master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
• vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
• vault_project: string — Infisical project slug (default: "infrastructure")
• vault_env: string — Infisical environment (default: "production")
• agent_host: string — Agent's IP for SSH (default: resolved from infra)
• agent_user: string — SSH user (default: "jerome")

Returns

• action: string — What was done
• key_alias: string — The LiteLLM key alias created/rotated
• key_prefix: string — First 10 chars of the new key (for identification)
• previous_key_alias: string | null — Previous key alias if rotating
• litellm_response: object — Raw response from LiteLLM /key/generate
• vault_updated: boolean — Whether Infisical vault secret was updated
• agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
• verification: { status: string, detail: string } — Final health check

Execution

1. Authenticate — Retrieve master key from Infisical vault via infisical export --project=<vault_project> --env=<vault_env>, verify against LiteLLM /key/list
2. Check existing keys — List all keys, find any with agent_name alias
3. If action == "list": Return all keys with their aliases and spend
4. If action == "create":
   • Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   • Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   • Duration is null (permanent) — inherited from litellm default_key_generate_params
   • Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
   • Note: ornith-1.0-35b is NOT a valid LiteLLM model name (use strix-moe, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
   • Return the new key
5. If action == "rotate":
   • Generate new key with same alias (LiteLLM replaces the old key)
   • Update secret in Infisical vault: infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>
   • Restart agent gateway (Hermes: systemctl restart hermes-gateway; pi: restart PM2 process) The gateway automatically picks up the new key via infisical run -- wrapper
   • Verify: curl test against /v1/models with new key
   • Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
   • Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
6. If action == "verify":
   • Retrieve key from Infisical vault: infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>
   • Test the key against LiteLLM /v1/models
   • Confirm key alias matches agent_name in LiteLLM key list
   • Verify agent gateway uses vault wrapper: cat /proc/<pid>/cmdline shows infisical run

Production Vault Access Process (canonical, 2026-07-16)

The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending — runs as user jerome, not systemd root, needs user-scope adaptation).

The canonical pattern

1. infisical CLI installed on the host (/usr/local/bin/infisical or /usr/bin/infisical).
2. Service token (Infisical Machine Identity, st.…) stored root-only at /root/.infisical-token (chmod 600).
   • Interim: the shared abiba service token (st.8e848433…) has READ+WRITE on the agents project.
   • Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
3. infisical-gateway.sh wrapper at /root/.hermes/infisical-gateway.sh (chmod 700):

   #!/bin/bash
   export INFISICAL_API_URL="https://vault.sysloggh.net"
   TOKEN=$(cat /root/.infisical-token)
   LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
   while true; do
     infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
         . /root/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3] safety net only
         export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
         exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
       ' >> $LOG 2>&1
     sleep 5   # restart on exit
   done
   

4. Agent key in vault as <AGENT>_LITELLM_API_KEY (e.g. KOBY_LITELLM_API_KEY). Vault = source of truth.
5. .env fallback at /root/.hermes/.env (chmod 600) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
6. systemd service hermes-gateway.service with ExecStart=/root/.hermes/infisical-gateway.sh. NO litellm-key.conf drop-in (those hardcode keys and rot).
7. NEVER hardcode LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.

Why this is non-fail

• No rot: keys pulled live from vault at every gateway start. Rotation = one infisical secrets set + systemctl restart. No per-host file edits.
• Survives vault outage: the .env fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
• Survives gateway crash: the wrapper's while true + systemd Restart=on-failure revive the gateway.
• Auditable: cat /proc/$(pgrep hermes_cli)/environ shows the live key; infisical secrets shows the vault source.

Migration status (2026-07-16)

Agent	Host	Pattern	Vault key	Status
abiba	.24	infisical run (pi agent wrapper, service token)	ABIBA_LITELLM_API_KEY	✅ vault-backed
mumuni	.123	infisical-gateway.sh + user-login machine identity	MUMUNI_LITELLM_API_KEY	✅ vault-backed
koby	.129	infisical-gateway.sh + service token (migrated 2026-07-16)	KOBY_LITELLM_API_KEY	✅ vault-backed, Zulip (tanko-bot@) + Telegram
koonimo	.114	infisical-gateway.sh + service token (migrated 2026-07-16)	KOONIMO_LITELLM_API_KEY	✅ vault-backed

Baggy = Koonimo (CT 113). BAGGY_LITELLM_API_KEY and KOONIMO_LITELLM_API_KEY MUST mirror each other in the vault — they are the same agent under different aliases. | tanko | .122 | hardcoded in config.yaml (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |

Tanko migration (pending)

Tanko runs the gateway as user jerome (not root/systemd), with the key hardcoded in /home/jerome/.hermes/config.yaml (api_key: sk-CggiHWlamQy…, valid but not vault-sourced). Migration: create a user-scope systemd service (~/.config/systemd/user/hermes-gateway.service) with infisical-gateway.sh wrapper in jerome's home, token at ~/.infisical-token, lingering enabled (loginctl enable-linger jerome) so the user service runs without a login session.

Koby migration lessons (2026-07-16)

Migrated Koby from hardcoded systemd drop-in → infisical-gateway.sh wrapper. Two mistakes I made that broke the agent:

1. Overwrote /root/.hermes/.env without backing it up. The Zulip API key only existed in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
2. Only injected LITELLM_API_KEY in the wrapper — forgot Zulip + Telegram credentials. Agents need ALL their platform env vars. Missing vars cause silent adapter failures.

How Koby actually connects (2026-07-16):

• Zulip: shares Tanko's bot (tanko-bot@chat.sysloggh.net, TANKO_ZULIP_API_KEY=5PeD6f3zo…). Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
• Telegram: token 828640… recovered from .env.bak-20260603 (18KB backup from June 2026). Allowed users: 6679773481. Home channel: 6679773481.
• Both platforms now connect through the wrapper's env injection.

Golden rule for gateway restarts: always cat /proc/<pid>/environ before killing the old process — captures the live env set. Especially important when migrating gateways between injection mechanisms.

Key rotation procedure (one vault operation with this standard)

1. Generate new key: POST /key/generate (master key, admin).
2. Update vault: infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net.
3. Update .env fallback: echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env.
4. Restart: systemctl restart hermes-gateway. The wrapper pulls the new key live.
5. Verify: curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models → 200.

Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)

Problem: The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com). In CLI v0.38.0, infisical secrets set / infisical export fail with "project id missing" / "workspace key 404" — a known bug where user-session auth works for run but NOT for secrets set. The apt repo only ships 0.38.0, so apt upgrade does not help.

Proper fix — Machine Identity (Infisical automation best practice): Create a machine identity with READ+WRITE scope on the agents project (project_id= 322fceab-39da-4854-a55a-568e76c0f13f, env prod). Store client_id + client_secret securely. Then vault writes work from any host:

# Get a machine-identity access token
TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
  -H 'Content-Type: application/json' \
  -d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
# Write a secret via REST API v3
 curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...

Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings → Machine Identities, or an admin API call. TODO: create abiba-automation machine identity and store its credentials in the vault itself (or a root-only file).

Interim (working now): the .env fallback (hermes-config-template Rule 3/13). The infisical-gateway.sh wrapper sources ~/.hermes/.env, so its <AGENT>_LITELLM_API_KEY overrides a stale vault value.

2026-07-16 UPDATE — vault is now SYNCED. The abiba service token (st.8e848433…, READ+WRITE) can write to the vault, so the session-13 rotated keys (mumuni sk-OzuWsoX2…, koby sk-BqRRMboTI…, koonimo sk-OEK7z26n6E…) are now in the vault as MUMUNI_LITELLM_API_KEY / KOBY_LITELLM_API_KEY / KOONIMO_LITELLM_API_KEY and validate 200 against LiteLLM. The vault is the source of truth again. Creating a dedicated abiba-automation machine identity (via UI) is still the proper long-term fix so the shared service token isn't reused across hosts — but it is no longer blocking.

Key Rotation Log

Date	Agent	Action	Notes
2026-07-16	mumuni	rotate	Old key malformed (sk-SWAl_Vu, 47 chars, not LiteLLM format) → 401. Deleted old mumuni key (token 15cbca18…), generated fresh (alias mumuni, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300.
2026-07-16	koby	rotate	Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old koby key, generated fresh (alias koby). New key sk-BqRRMboTI… in systemd drop-in hermes-gateway.service.d/litellm-key.conf + /etc/environment. Created hermes-gateway.service unit (was missing — gateway wasn't persistent) with --replace. Verified HTTP 200, Telegram connected.
2026-07-16	baggy (koonimo)	rotate	Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old baggy key, generated fresh (alias baggy, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in hermes-gateway.service.d/litellm-key.conf. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected.

LiteLLM Master Key (use sparingly — agents should NOT use it directly)

• Master key: sk-litellm-7f96080dd99b15c36bd4b333b58a6796 (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
• Used for /key/generate, /key/delete, /key/list (GET), DB queries
• Known violation (RESOLVED 2026-07-16): Abiba's LITELLM_API_KEY was previously the master key. It is now a dedicated agent key sk-sxbphLvk1OU… (vault secret ABIBA_LITELLM_API_KEY, alias abiba-pi). The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference — see litellm-self-heal § "NEVER use litellm_proxy_master_key for inference".
• LiteLLM key DB: harness-postgres container on CT116, table "LiteLLM_VerificationToken" (columns: token, key_alias, key_name, created_at, expires). Query: docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"