vault: BAGGY_LITELLM_API_KEY deleted — one secret per agent

Baggy = Koonimo (CT 113). The vault had two divergent keys:
- BAGGY_LITELLM_API_KEY = sk-QT-kDt8Szo... (stale, 401)
- KOONIMO_LITELLM_API_KEY = sk-OEK7z26n6... (valid, 200, session-13 rotation)

Synced BAGGY to match KOONIMO (the valid key).
Added contract note: both secrets MUST mirror each other.
This was the root cause of Koonimo's 401 after migration —
the ghost process had BAGGY in its env instead of KOONIMO.

hermes-config-template.prose.md

@@ -143,23 +143,25 @@ compression:
 
 # ─── Auxiliary Tasks (CONSISTENCY RULE) ───
 # All auxiliary services MUST use identical model, base_url, and api_key_env:
-#   model: gemma-4-12b
+#   model: gpu-light                 # stable alias (NOT raw "gemma-4-12b")
 #   base_url: http://192.168.68.116/v1
 #   api_key_env: LITELLM_API_KEY
 # Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
-# gemma-4-12b is a lightweight 12B model on the RTX 5070, freeing the Strix Halo
-# for agent reasoning.
+# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
+# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
+# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
+# in agent configs — use the stable aliases so model swaps don't break agents.
 auxiliary:
   vision:
     provider: harness
-    model: gemma-4-12b
+    model: gpu-light            # stable alias for RTX 5070 (was raw gemma-4-12b)
     base_url: http://192.168.68.116/v1
     api_key_env: LITELLM_API_KEY
     timeout: 60
     download_timeout: 30
   web_extract:
     provider: harness
-    model: gemma-4-12b
+    model: gpu-light            # stable alias for RTX 5070
     base_url: http://192.168.68.116/v1
     api_key_env: LITELLM_API_KEY
     timeout: 30
@@ -170,10 +172,19 @@ auxiliary:
     api_key_env: LITELLM_API_KEY
     timeout: 300                 # gpu-fleet: 300s for large-history summarization (was 60)
 
+# ─── Delegation / Heavy Aux (use gpu-dense = RTX 3090) ───
+# delegation.model and x_search.model use gpu-dense (NOT raw qwen3.6-27B-code).
+
+delegation:
+  model: gpu-dense              # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
+  provider: harness
+  base_url: http://192.168.68.116/v1
+  api_key_env: LITELLM_API_KEY
+
 # ─── Custom Provider ───
 custom_providers:
   - name: harness
-    model: <agent_model>
+    model: syslog-auto            # weighted pool (default)
     base_url: http://192.168.68.116/v1
     api_key_env: LITELLM_API_KEY
     api_mode: chat_completions

hermes-key-enforcement.prose.md

@@ -72,6 +72,11 @@ External providers are **explicitly exempt** and may use hardcoded keys:
 
 ## Standard Pattern
 
+> **Canonical vault process (2026-07-16):** see `litellm-api-keys` § Production Vault Access Process.
+> All agents MUST use the `infisical-gateway.sh` wrapper (live vault injection). Hardcoded systemd
+> drop-ins / config.yaml keys are DEPRECATED — they rot on rotation (root cause of the 2026-07-16 401 storm).
+> 4/5 agents migrated; tanko (user jerome) pending.
+
 ```yaml
 # ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
 model:

litellm-api-keys.prose.md

@@ -14,9 +14,15 @@ description: >
   expire/404. The .env fallback prevents agents from running without keys.
   Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
   
+  UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
+  token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
+  infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
+  Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
+  Abiba's key is now a proper agent key (NOT the master key — stale note removed).
+  
   Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
   Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
-  on CT 116. Last verified: 2026-07-12.
+  on CT 116. Last verified: 2026-07-16.
 ---
 
 ## Parameters
@@ -68,6 +74,93 @@ description: >
    - Confirm key alias matches agent_name in LiteLLM key list
    - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
 
+## Production Vault Access Process (canonical, 2026-07-16)
+
+The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
+runs as user `jerome`, not systemd root, needs user-scope adaptation).
+
+### The canonical pattern
+
+1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
+2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
+   - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
+   - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
+3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
+   ```bash
+   #!/bin/bash
+   export INFISICAL_API_URL="https://vault.sysloggh.net"
+   TOKEN=$(cat /root/.infisical-token)
+   LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
+   while true; do
+     infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
+       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
+         . /root/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3] safety net only
+         export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
+         exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
+       ' >> $LOG 2>&1
+     sleep 5   # restart on exit
+   done
+   ```
+4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
+5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
+6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
+7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
+
+### Why this is non-fail
+
+- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
+- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
+- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
+- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
+
+### Migration status (2026-07-16)
+
+| Agent | Host | Pattern | Vault key | Status |
+|-------|------|---------|-----------|--------|
+| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
+| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
+| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
+| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
+> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
+| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
+
+### Tanko migration (pending)
+
+Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
+`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
+Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
+with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
+enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
+
+### Koby migration lessons (2026-07-16)
+
+Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
+**Two mistakes I made that broke the agent:**
+1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
+   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
+   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
+2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
+   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
+
+**How Koby actually connects (2026-07-16):**
+- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
+  Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
+- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
+  Allowed users: 6679773481. Home channel: 6679773481.
+- Both platforms now connect through the wrapper's env injection.
+
+**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old
+process — captures the live env set. Especially important when migrating gateways between
+injection mechanisms.
+
+### Key rotation procedure (one vault operation with this standard)
+
+1. Generate new key: `POST /key/generate` (master key, admin).
+2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
+3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
+4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
+5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
+
 ## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
 
 **Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
@@ -95,9 +188,15 @@ Machine Identities, or an admin API call. **TODO: create `abiba-automation` mach
 and store its credentials in the vault itself (or a root-only file).**
 
 **Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
-infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `MUMUNI_LITELLM_API_KEY` overrides
-the stale vault value. This is fully functional and contract-sanctioned — the vault sync above is
-only for consistency so .env and vault never drift.
+infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `<AGENT>_LITELLM_API_KEY`
+overrides a stale vault value.
+
+**2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE)
+can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`,
+koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` /
+`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
+Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
+so the shared service token isn't reused across hosts — but it is no longer blocking.
 
 ## Key Rotation Log
 
@@ -111,5 +210,8 @@ only for consistency so .env and vault never drift.
 
 - Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
 - Used for /key/generate, /key/delete, /key/list (GET), DB queries
-- **Known violation:** Abiba's LITELLM_API_KEY IS the master key (should be a dedicated `abiba` virtual key). TODO: generate `abiba` virtual key and stop using master key directly.
+- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
+It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
+The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
+see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
 - LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`

litellm-self-heal.prose.md

@@ -69,7 +69,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
 
 - `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
 - `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
-- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. Note `qwen3.6-35B-udq4` is NOT in the agent-key allowlist — agents must use `strix-moe` to reach the AMD model.
+- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. As of 2026-07-16 the `baggy`/`koby`/`mumuni`/`abiba-pi` keys ALSO include `qwen3.6-35B-udq4`; `abiba-pi` additionally includes `deepseek-v4-pro` (cloud fallback). `kagenz0`/`koonimo`/`pi-agents-unified` have the standard 6 only. Agents should still use the stable alias `strix-moe` (not the raw `qwen3.6-35B-udq4`) so model swaps don't break them.
+- **NEVER use `litellm_proxy_master_key` (the `sk-litellm-...` master key) for inference.** It is for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated `monitor` agent key stored at `/etc/litellm-monitor.env` on CT 116 (root-only, `chmod 600`); `/key/list` is the only call that legitimately uses `$MASTER_KEY`.
 
 ## Model Fallback Chains (LiteLLM)