Contracts: Mumuni context-fix (WAL #1300) — strix-moe, 256K all GPUs, Rule 12/13, machine-identity vault procedure

litellm-api-keys.prose.md

@@ -51,8 +51,8 @@ description: >
    - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
    - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
    - Duration is null (permanent) — inherited from litellm default_key_generate_params
-   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
-   - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
+   - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
+   - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
    - Return the new key
 5. **If action == "rotate"**:
    - Generate new key with same alias (LiteLLM replaces the old key)
@@ -67,3 +67,47 @@ description: >
    - Test the key against LiteLLM /v1/models
    - Confirm key alias matches agent_name in LiteLLM key list
    - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
+
+## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
+
+**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
+In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace
+key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt
+repo only ships 0.38.0, so `apt upgrade` does not help.
+
+**Proper fix — Machine Identity (Infisical automation best practice):**
+Create a machine identity with READ+WRITE scope on the `agents` project (project_id=
+`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely.
+Then vault writes work from any host:
+```bash
+# Get a machine-identity access token
+TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
+  -H 'Content-Type: application/json' \
+  -d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
+# Write a secret via REST API v3
+ curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
+  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
+  -d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
+# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
+```
+Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
+Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
+and store its credentials in the vault itself (or a root-only file).**
+
+**Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
+infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `MUMUNI_LITELLM_API_KEY` overrides
+the stale vault value. This is fully functional and contract-sanctioned — the vault sync above is
+only for consistency so .env and vault never drift.
+
+## Key Rotation Log
+
+| Date | Agent | Action | Notes |
+|------|-------|--------|-------|
+| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
+
+## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
+
+- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
+- Used for /key/generate, /key/delete, /key/list (GET), DB queries
+- **Known violation:** Abiba's LITELLM_API_KEY IS the master key (should be a dedicated `abiba` virtual key). TODO: generate `abiba` virtual key and stop using master key directly.
+- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`