Contracts: Rule 13 two key-injection patterns + koby/baggy rotation log (WAL #1300)

hermes-config-template.prose.md

@@ -320,23 +320,35 @@ cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set
 curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
 ```
 
-### Rule 13: .env Fallback Enforcement (ADDED 2026-07-16, WAL #1300)
-- The infisical-gateway.sh wrapper MUST source `~/.hermes/.env` BEFORE exporting `LITELLM_API_KEY`,
-  so the local .env overrides a stale Infisical vault value:
-  ```bash
-  -- bash -c "
-    . /root/.hermes/.env  # [FALLBACK Rule 3/13] local .env overrides stale vault value
-    export LITELLM_API_KEY=\"\$MUMUNI_LITELLM_API_KEY\"
-    ...
-  ```
-- `~/.hermes/.env` MUST contain `MUMUNI_LITELLM_API_KEY=sk-<valid>` (or the agent's equivalent).
-- This is sanctioned by Rule 3: "Infisical service tokens can expire/404; the .env fallback prevents
-  agents from running without keys."
-- The systemd unit `hermes-gateway.service` ExecStart MUST be `/root/.hermes/infisical-gateway.sh`
-  (the wrapper), NOT `python -m hermes_cli.main gateway run` directly (direct exec breaks ALL env
-  injection → "No messaging platforms enabled").
-- To sync the Infisical vault (when write access is available): see `litellm-api-keys.prose.md`
-  § Machine Identity for Vault Writes.
+### Rule 13: API Key Injection — Two Patterns (UPDATED 2026-07-16, WAL #1300)
+
+Agents inject `LITELLM_API_KEY` via ONE of two mechanisms. Both are valid; the contract
+requirement is that the key is a **valid LiteLLM virtual key** (HTTP 200 on /v1/models).
+
+**Pattern A — systemd drop-in (Koby, Koonimo, and any agent without infisical wrapper):**
+A systemd drop-in `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` sets the key:
+```ini
+[Service]
+Environment="LITELLM_API_KEY=sk-<VALID_KEY>"
+```
+The service unit `hermes-gateway.service` runs `python -m hermes_cli.main gateway run --replace`
+directly (no infisical). Apply with `systemctl daemon-reload && systemctl restart hermes-gateway`.
+- Koonimo (CT113/.114): service = `hermes-gateway.service`, drop-in has the key.
+- Koby (CT111/.129): service = `hermes-gateway.service` (created 2026-07-16), ExecStart uses `--replace`
+  to win the lock against stray `hermes gateway restart` invocations. Key also in `/etc/environment`.
+
+**Pattern B — infisical-gateway.sh wrapper (Mumuni):**
+The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
+See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
+
+**Verification (all agents):**
+```bash
+GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
+K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
+curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models  # must be 200
+```
+- `/etc/environment` is NO LONGER the canonical key source (stale values there caused 401s).
+- Do NOT leave a hardcoded stale key in `/etc/environment` — it shadows the drop-in/wrapper.
 
 ## Execution
 

litellm-api-keys.prose.md

@@ -104,6 +104,8 @@ only for consistency so .env and vault never drift.
 | Date | Agent | Action | Notes |
 |------|-------|--------|-------|
 | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
+| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
+| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
 
 ## LiteLLM Master Key (use sparingly — agents should NOT use it directly)