114 lines
7.5 KiB
Markdown
114 lines
7.5 KiB
Markdown
---
|
|
kind: function
|
|
name: litellm-api-keys
|
|
description: >
|
|
Manages LiteLLM API keys for agent identity. Creates named keys so each agent
|
|
is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry)
|
|
and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026").
|
|
Ensures agents never use the master key directly. Rotation is event-driven,
|
|
not calendar-driven — rotate only on compromise, personnel change, or
|
|
periodic security hygiene (quarterly/annually).
|
|
|
|
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
|
|
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
|
|
expire/404. The .env fallback prevents agents from running without keys.
|
|
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
|
|
|
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
|
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
|
on CT 116. Last verified: 2026-07-12.
|
|
---
|
|
|
|
## Parameters
|
|
|
|
- agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
|
|
- action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
|
|
- litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
|
|
- master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
|
|
- vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
|
|
- vault_project: string — Infisical project slug (default: "infrastructure")
|
|
- vault_env: string — Infisical environment (default: "production")
|
|
- agent_host: string — Agent's IP for SSH (default: resolved from infra)
|
|
- agent_user: string — SSH user (default: "jerome")
|
|
|
|
## Returns
|
|
|
|
- action: string — What was done
|
|
- key_alias: string — The LiteLLM key alias created/rotated
|
|
- key_prefix: string — First 10 chars of the new key (for identification)
|
|
- previous_key_alias: string | null — Previous key alias if rotating
|
|
- litellm_response: object — Raw response from LiteLLM /key/generate
|
|
- vault_updated: boolean — Whether Infisical vault secret was updated
|
|
- agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
|
|
- verification: { status: string, detail: string } — Final health check
|
|
|
|
## Execution
|
|
|
|
1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
|
|
2. **Check existing keys** — List all keys, find any with agent_name alias
|
|
3. **If action == "list"**: Return all keys with their aliases and spend
|
|
4. **If action == "create"**:
|
|
- Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
|
|
- Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
|
|
- Duration is null (permanent) — inherited from litellm default_key_generate_params
|
|
- Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
|
|
- Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
|
|
- Return the new key
|
|
5. **If action == "rotate"**:
|
|
- Generate new key with same alias (LiteLLM replaces the old key)
|
|
- Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
|
|
- Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
|
|
The gateway automatically picks up the new key via `infisical run --` wrapper
|
|
- Verify: curl test against /v1/models with new key
|
|
- Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
|
|
- Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
|
|
6. **If action == "verify"**:
|
|
- Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
|
|
- Test the key against LiteLLM /v1/models
|
|
- Confirm key alias matches agent_name in LiteLLM key list
|
|
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
|
|
|
|
## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
|
|
|
|
**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
|
|
In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace
|
|
key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt
|
|
repo only ships 0.38.0, so `apt upgrade` does not help.
|
|
|
|
**Proper fix — Machine Identity (Infisical automation best practice):**
|
|
Create a machine identity with READ+WRITE scope on the `agents` project (project_id=
|
|
`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely.
|
|
Then vault writes work from any host:
|
|
```bash
|
|
# Get a machine-identity access token
|
|
TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
|
|
# Write a secret via REST API v3
|
|
curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
|
|
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
|
-d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
|
|
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
|
|
```
|
|
Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
|
|
Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
|
|
and store its credentials in the vault itself (or a root-only file).**
|
|
|
|
**Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
|
|
infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `MUMUNI_LITELLM_API_KEY` overrides
|
|
the stale vault value. This is fully functional and contract-sanctioned — the vault sync above is
|
|
only for consistency so .env and vault never drift.
|
|
|
|
## Key Rotation Log
|
|
|
|
| Date | Agent | Action | Notes |
|
|
|------|-------|--------|-------|
|
|
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
|
|
|
|
## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
|
|
|
|
- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
|
|
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
|
|
- **Known violation:** Abiba's LITELLM_API_KEY IS the master key (should be a dedicated `abiba` virtual key). TODO: generate `abiba` virtual key and stop using master key directly.
|
|
- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`
|