root
79d4a73895
- gpu-fleet: Corrected architecture (direct GPU, no router in path). Updated context values (RTX 3090=256K, not 128K). Added api-key standardization requirement. - gpu-self-heal: Added Lessons Learned section with 5 critical findings: L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed) L2: Fallback chain cascading failure loop detection L3: Verify running state, not documentation L4: Infisical fallback requirement (.env must have uncommented key) L5: Zulip event queue can silently die after ~40 reconnects - litellm-self-heal: Updated status manual-only→deployed, cron schedule - litellm-api-keys: Added Infisical token expiry warning + .env fallback - hermes-config-template: Rule 3 updated with .env fallback requirement
70 lines
4.1 KiB
Markdown
70 lines
4.1 KiB
Markdown
---
|
|
kind: function
|
|
name: litellm-api-keys
|
|
description: >
|
|
Manages LiteLLM API keys for agent identity. Creates named keys so each agent
|
|
is identifiable in LiteLLM logs/spend tracking. Keys are permanent (no expiry)
|
|
and use the agent's bare name as alias (e.g., "tanko", not "tanko-jul2026").
|
|
Ensures agents never use the master key directly. Rotation is event-driven,
|
|
not calendar-driven — rotate only on compromise, personnel change, or
|
|
periodic security hygiene (quarterly/annually).
|
|
|
|
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
|
|
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
|
|
expire/404. The .env fallback prevents agents from running without keys.
|
|
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
|
|
|
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
|
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
|
on CT 116. Last verified: 2026-07-09.
|
|
---
|
|
|
|
## Parameters
|
|
|
|
- agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
|
|
- action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
|
|
- litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
|
|
- master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
|
|
- vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
|
|
- vault_project: string — Infisical project slug (default: "infrastructure")
|
|
- vault_env: string — Infisical environment (default: "production")
|
|
- agent_host: string — Agent's IP for SSH (default: resolved from infra)
|
|
- agent_user: string — SSH user (default: "jerome")
|
|
|
|
## Returns
|
|
|
|
- action: string — What was done
|
|
- key_alias: string — The LiteLLM key alias created/rotated
|
|
- key_prefix: string — First 10 chars of the new key (for identification)
|
|
- previous_key_alias: string | null — Previous key alias if rotating
|
|
- litellm_response: object — Raw response from LiteLLM /key/generate
|
|
- vault_updated: boolean — Whether Infisical vault secret was updated
|
|
- agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
|
|
- verification: { status: string, detail: string } — Final health check
|
|
|
|
## Execution
|
|
|
|
1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
|
|
2. **Check existing keys** — List all keys, find any with agent_name alias
|
|
3. **If action == "list"**: Return all keys with their aliases and spend
|
|
4. **If action == "create"**:
|
|
- Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
|
|
- Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
|
|
- Duration is null (permanent) — inherited from litellm default_key_generate_params
|
|
- Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
|
|
- Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
|
|
- Return the new key
|
|
5. **If action == "rotate"**:
|
|
- Generate new key with same alias (LiteLLM replaces the old key)
|
|
- Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
|
|
- Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
|
|
The gateway automatically picks up the new key via `infisical run --` wrapper
|
|
- Verify: curl test against /v1/models with new key
|
|
- Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
|
|
- Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
|
|
6. **If action == "verify"**:
|
|
- Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
|
|
- Test the key against LiteLLM /v1/models
|
|
- Confirm key alias matches agent_name in LiteLLM key list
|
|
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
|