docs: lessons learned from 2026-07-12 session

- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement

litellm-api-keys.prose.md

@@ -9,6 +9,11 @@ description: >
   not calendar-driven — rotate only on compromise, personnel change, or
   periodic security hygiene (quarterly/annually).
   
+  UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
+  BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
+  expire/404. The .env fallback prevents agents from running without keys.
+  Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
+  
   Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
   Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
   on CT 116. Last verified: 2026-07-09.
@@ -19,7 +24,10 @@ description: >
 - agent_name: string — The agent to manage keys for (e.g., "tanko", "mumuni")
 - action: "create" | "rotate" | "verify" | "list" — What to do (default: "create")
 - litellm_host: string — LiteLLM admin endpoint (default: "192.168.68.116:4000")
-- master_key: string — LiteLLM master key (default from environment)
+- master_key: string — LiteLLM master key (default from Infisical vault: project=infrastructure, env=production, secret=LITELLM_MASTER_KEY)
+- vault_url: string — Infisical vault URL (default: "https://vault.sysloggh.net")
+- vault_project: string — Infisical project slug (default: "infrastructure")
+- vault_env: string — Infisical environment (default: "production")
 - agent_host: string — Agent's IP for SSH (default: resolved from infra)
 - agent_user: string — SSH user (default: "jerome")
 
@@ -30,12 +38,13 @@ description: >
 - key_prefix: string — First 10 chars of the new key (for identification)
 - previous_key_alias: string | null — Previous key alias if rotating
 - litellm_response: object — Raw response from LiteLLM /key/generate
-- agent_config_updated: boolean — Whether /etc/environment was updated
+- vault_updated: boolean — Whether Infisical vault secret was updated
+- agent_config_updated: boolean — Legacy: whether /etc/environment was updated (deprecated, always false post-migration)
 - verification: { status: string, detail: string } — Final health check
 
 ## Execution
 
-1. **Authenticate** — Verify master_key works against LiteLLM /key/list
+1. **Authenticate** — Retrieve master key from Infisical vault via `infisical export --project=<vault_project> --env=<vault_env>`, verify against LiteLLM /key/list
 2. **Check existing keys** — List all keys, find any with agent_name alias
 3. **If action == "list"**: Return all keys with their aliases and spend
 4. **If action == "create"**:
@@ -47,11 +56,14 @@ description: >
    - Return the new key
 5. **If action == "rotate"**:
    - Generate new key with same alias (LiteLLM replaces the old key)
-   - SSH to agent_host, update /etc/environment LITELLM_API_KEY
-   - Restart agent gateway (hermes gateway restart for Hermes agents)
+   - Update secret in Infisical vault: `infisical secrets set LITELLM_API_KEY=<new_key> --project=<vault_project> --env=<vault_env>`
+   - Restart agent gateway (Hermes: `systemctl restart hermes-gateway`; pi: restart PM2 process)
+     The gateway automatically picks up the new key via `infisical run --` wrapper
    - Verify: curl test against /v1/models with new key
    - Rotation policy: on-demand only (compromise, departure, quarterly hygiene)
+   - Note: /etc/environment is NO LONGER used for LiteLLM keys. Agents inject keys at runtime via vault wrapper.
 6. **If action == "verify"**:
-   - SSH to agent, read /etc/environment
+   - Retrieve key from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=<vault_project> --env=<vault_env>`
    - Test the key against LiteLLM /v1/models
    - Confirm key alias matches agent_name in LiteLLM key list
+   - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`