prose-contracts/litellm-api-keys.prose.md

---
kind: function
name: litellm-api-keys
description: >
  Manages LiteLLM API keys for agent identity. Creates named keys so each agent
  is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry)
  and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026").
  Ensures agents never use the master key directly. Rotation is event-driven,
  not calendar-driven — rotate only on compromise, personnel change, or
  periodic security hygiene (quarterly/annually).

  UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
  BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
  expire/404. The .env fallback prevents agents from running without keys.
  Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.

  UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
  token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
  infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
  Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
  Abiba's key is now a proper agent key (NOT the master key — stale note removed).

  UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
  standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
  infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
  injection → .env fallback → exec python. Systemd drop-ins are IMMUNE to hermes gateway
  install which overwrites the unit file ExecStart. Infisical CLI updated to 0.43.109 on
  all agents (was 0.38.0). Service token st.8e848433 shared across fleet (st.353699cd
  for tanko was deleted). .env fallback on every agent protects against token loss.
  Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
  — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
  (3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.

  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
  on CT 116. Last verified: 2026-07-17.
---

## Parameters

- agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
- action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
- litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
- master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
- vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
- vault_project: string — Infisical project slug (default: "infrastructure")
- vault_env: string — Infisical environment (default: "production")
- agent_host: string — Agent's IP for SSH (default: resolved from infra)
- agent_user: string — SSH user (default: "jerome")

## Returns

- action: string — What was done
- key_alias: string — The LiteLLM key alias created/rotated
- key_prefix: string — First 10 chars of the new key (for identification)
- previous_key_alias: string | null — Previous key alias if rotating
- litellm_response: object — Raw response from LiteLLM /key/generate
- vault_updated: boolean — Whether Infisical vault secret was updated
- agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
- verification: { status: string, detail: string } — Final health check

## Execution

1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
2. **Check existing keys** — List all keys, find any with agent_name alias
3. **If action == "list"**: Return all keys with their aliases and spend
4. **If action == "create"**:
   - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
   - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
   - Duration is null (permanent) — inherited from litellm default_key_generate_params
   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
   - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
   - Return the new key
5. **If action == "rotate"**:
   - Generate new key with same alias (LiteLLM replaces the old key)
   - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
   - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
     The gateway automatically picks up the new key via `infisical run --` wrapper
   - Verify: curl test against /v1/models with new key
   - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
   - Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
6. **If action == "verify"**:
   - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
   - Test the key against LiteLLM /v1/models
   - Confirm key alias matches agent_name in LiteLLM key list
   - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`

## Production Vault Access Process (canonical, 2026-07-17)

The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents
(Mumuni, Tanko, Koby, Koonimo) as of 2026-07-17. Abiba (pi) uses a similar pattern
through its agent wrapper.

### The canonical pattern

1. **infisical CLI** installed on the host at `/usr/bin/infisical` (v0.43.109+, from
   artifacts-cli.infisical.com apt repo). Update procedure:
   ```bash
   curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | sudo -E bash
   sudo apt-get update && sudo apt-get install -y infisical
   # Remove stale old binary if present
   rm -f /usr/local/bin/infisical /bin/infisical
   ```
   Wrappers use absolute path `/usr/bin/infisical run`. Never rely on PATH resolution.
2. **Service token** (Infisical Machine Identity, `st.…`) stored at `~/.infisical-token`
   (`chmod 600`). Current: shared `st.8e848433…` (abiba, READ+WRITE on agents project).
   Tanko's `st.353699cd…` (tanko-agent) was deleted — reverted to shared token.
   Proper: one machine identity per agent (create in Infisical UI → Project Settings →
   Machine Identities).
3. **`infisical-gateway.sh` wrapper** at `~/.hermes/infisical-gateway.sh` (`chmod 700`):
   ```bash
   #!/bin/bash
   export INFISICAL_API_URL="https://vault.sysloggh.net"
   TOKEN=$(cat $HOME/.infisical-token)
   LOG=$HOME/.hermes/logs/gateway.log; mkdir -p $HOME/.hermes/logs
   while true; do
     echo "[$(date -Iseconds)] Starting gateway with Infisical injection..." >> $LOG
     /usr/bin/infisical run --token="$TOKEN" \
       --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
         . $HOME/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3]
         export LITELLM_API_KEY="${<AGENT>_LITELLM_API_KEY}"
         export ZULIP_API_KEY="${<AGENT>_ZULIP_API_KEY}"
         export ZULIP_SITE="https://chat.sysloggh.net"
         export ZULIP_EMAIL="<agent>-bot@chat.sysloggh.net"
         export SEARXNG_URL="http://192.168.68.7:8888"
         # ⚠️  HARDCODE the full venv path. NEVER use $VENV inside single quotes.
         exec /root/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
       ' >> $LOG 2>&1
     EXIT_CODE=$?
     echo "[$(date -Iseconds)] Gateway exited with code $EXIT_CODE — restarting in 5s..." >> $LOG
     sleep 5
   done
   ```
   **CRITICAL: VENV PATH.** The inner `bash -c '...'` uses single quotes. Shell
   variables set in the outer wrapper are NOT expanded inside single quotes.
   `$VENV/bin/python` resolves to `/bin/python` (file not found). Always hardcode
   the absolute path to the venv python binary.
4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` and `<AGENT>_ZULIP_API_KEY`.
   Vault = source of truth for ALL platform credentials.
5. **`.env` fallback** at `~/.hermes/.env` (`chmod 600`) with agent-specific keys —
   safety net for vault outage or token revocation. Must be kept in sync on rotation.
   Example:
   ```bash
   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
   ```
6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
   ```ini
   [Service]
   ExecStart=
   ExecStart=/root/.hermes/infisical-gateway.sh
   ```
   The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
   then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
   regeneration** by `hermes gateway install` — the drop-in always wins.

   **Why a drop-in instead of editing the unit file:** `hermes gateway install`
   (called during Hermes updates and some self-heal operations) regenerates the
   systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
   Editing the unit file directly is futile — it will be overwritten. The drop-in
   approach explicitly resets ExecStart and sets the wrapper regardless of what the
   main unit file says.
7. **NEVER hardcode** API keys in systemd drop-ins, config.yaml, or /etc/environment.
   The wrapper injects live from vault at every start.

### Why this is non-fail

- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable or the service token is revoked.
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.

### Migration status (2026-07-17)

| Agent | Host | Pattern | Keys | Status |
|-------|------|---------|------|--------|
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |

> Tanko runs as user `jerome` — wrapper/token at `~/.hermes/infisical-gateway.sh` and
> `~/.infisical-token`. Linger enabled (`loginctl enable-linger jerome`) for boot startup.

### Tanko migration (COMPLETED 2026-07-17)

Tanko was the last agent migrated from hardcoded keys to vault wrapper.
Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
`50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
`~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.

### Koby migration lessons (2026-07-16, updated 2026-07-17)

Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
**Three mistakes made:**
1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"`
   inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path.

**How Koby actually connects:**
- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
- Telegram: token from `.env` fallback. Allowed users: 6679773481.
- Both platforms now connect through the wrapper's env injection.

**Golden rules for gateway restarts:**
1. Always `cat /proc/<pid>/environ` before killing the old process — captures the live env set.
2. Hardcode venv python path in wrapper — never use variables inside single-quoted bash -c.
3. Use systemd drop-ins (not unit file edits) to override ExecStart — survives Hermes updates.

### Fleet-wide standardization lessons (2026-07-17)

After auditing all 4 agents, five systemic patterns caused repeated failures:
1. **Three incompatible startup patterns** coexisted (systemd drop-in, direct python, orphaned wrapper)
2. **Systemd unit files reverted** by `hermes gateway install` during updates
3. **VENV variable scoping** broke wrappers on Koby and Mumuni (single-quote bash -c)
4. **Service token expiry** — Tanko's `st.353699cd` was deleted from Infisical
5. **No ZULIP_API_KEY** in env on Tanko — wrapper bypassed by systemd direct python

All resolved by the canonical drop-in + while-true wrapper pattern documented above.

### Key rotation procedure (one vault operation with this standard)

1. Generate new key: `POST /key/generate` (master key, admin).
2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.

## Machine Identity for Vault Writes (UPDATED 2026-07-17)

**Current state:** Infisical CLI updated to v0.43.109 on all agents (from v0.38.0).
The v0.38.0 bug (user-session auth fails for `secrets set`/`export`) is resolved.
Service token `st.8e848433…` (abiba, READ+WRITE) can write to vault from CLI.

**Proper fix — per-agent Machine Identities:**
Create machine identities in Infisical UI → Project Settings → Machine Identities
for each agent with READ-only scope on the `agents` project. Store client_id +
client_secret per agent. Then vault writes use the shared abiba identity, and
reads use per-agent identities. This eliminates the single shared token risk.

**Service Token Inventory (2026-07-17):**
| Token ID | Name | Permissions | Used By | Status |
|----------|------|-------------|---------|--------|
| `st.8e848433…` | tanko-gateway | READ+WRITE | Mumuni, Tanko, Koby, Koonimo, Abiba | ✅ Active |
| `st.353699cd…` | tanko-agent | READ-only | — | ❌ Deleted from Infisical |

**Per-agent .env fallback inventory (2026-07-17):**
| Agent | .env Keys |
|-------|-----------|
| Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY |
| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
| Koby | (wrapper injects from vault — .env has Telegram token) |
| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |

## Key Rotation Log

| Date | Agent | Action | Notes |
|------|-------|--------|-------|
| 2026-07-17 | fleet | standardize | All 4 agents standardized on systemd drop-in + while-true wrapper + infisical v0.43.109. Removed conflicting zulip-env.conf + litellm-key.conf drop-ins. Added .env fallbacks with ZULIP keys. WAL #1322. |
| 2026-07-17 | tanko | fix-zulip | Added ZULIP_API_KEY to env (was missing — systemd bypassed vault). Updated wrapper from exec to while-true. Created .env fallback. Removed hardcoded zulip-env.conf drop-in. WAL #1321. |
| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated. 5 personal creds flagged. |
| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault. Wrapper injects ZULIP_API_KEY + ZULIP_EMAIL. 3 platforms. |
| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml to infisical-gateway.sh + st.353699cd. NOTE: st.353699cd later deleted — reverted to st.8e848433 on 2026-07-17. |
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |

## LiteLLM Master Key (use sparingly — agents should NOT use it directly)

- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`