docs: hermes-agent-baseline — canonical good-state reference

Single source of truth for all 4 Hermes agents. Captures:
- Agent map (CT, IP, key, alias)
- Key architecture and config patterns
- Mandatory api_key workaround with explanation
- Known bug: auxiliary_client ignores api_key_env
- Audit procedure (full audit, leak check, key verification)
- Systemd patterns
- Attachment cache locations

hermes-agent-baseline.prose.md

@@ -0,0 +1,194 @@
+---
+kind: reference
+name: hermes-agent-baseline
+version: 1.0.0
+description: >
+  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
+  configuration state, keys, workarounds, and audit procedure. When an agent's
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-05.
+author: Abiba (pi agent)
+---
+
+# Hermes Agent Baseline — Canonical Good State
+
+## Quick Restore
+
+```bash
+# Verify all agents against baseline in one command:
+for ct in 112 114 111 113; do
+    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
+done
+```
+
+## Agent Map
+
+| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias |
+|-------|-----|------|-----|-------------|---------------|
+| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` |
+| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` |
+| Tdunna | 111 | amdpve | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | `tdunna` |
+| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` |
+
+Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
+
+## Key Architecture
+
+```
+Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
+                                                     └── Key DB (Postgres)
+```
+
+- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
+- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
+- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
+- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
+
+## Config Pattern — Mandatory Fields
+
+Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
+
+### 1. Main Model
+```yaml
+model:
+  default: syslog-auto
+  provider: custom:harness       # or: harness
+  api_key_env: LITELLM_API_KEY
+  max_tokens: 4096
+```
+
+### 2. Custom Provider
+```yaml
+custom_providers:
+  - name: harness
+    model: syslog-auto
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_mode: chat_completions
+```
+
+### 3. Vision (CRITICAL — must have api_key directly!)
+```yaml
+auxiliary:
+  vision:
+    provider: harness
+    model: gemma-4-12b              # or syslog-auto
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    timeout: 60
+    download_timeout: 30
+```
+
+### 4. Compression (must have api_key!)
+```yaml
+  compression:
+    enabled: true
+    threshold: 0.65
+    target_ratio: 0.3
+    provider: harness
+    model: syslog-auto               # or gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    timeout: 120
+```
+
+## Known Bug: `api_key_env` Ignored by Auxiliary Client
+
+**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
+
+**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
+resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
+to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
+to `"no-key-required"` → 401 from LiteLLM.
+
+**Impact**: Vision analysis, compression, and any other auxiliary task calling
+LiteLLM/harness will fail with:
+```
+401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
+```
+
+**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
+`api_key_env` in every auxiliary task config that uses the harness provider.
+
+**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
+`api_key` is empty:
+```python
+cfg_api_key = str(task_config.get("api_key", "")).strip() or None
+if not cfg_api_key:
+    key_env = str(task_config.get("api_key_env", "")).strip()
+    if key_env:
+        cfg_api_key = os.getenv(key_env, "").strip() or None
+```
+
+## Audit Procedure
+
+### Full Audit (all agents)
+```bash
+for ct in 112 114 111 113; do
+    echo "=== CT $ct ==="
+    pct-run $ct grep LITELLM_API_KEY /etc/environment
+    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
+    echo ""
+done
+```
+
+### Master Key Leak Check
+```bash
+# On every agent:
+pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
+# Must return empty
+```
+
+### Verify Key Works
+```bash
+curl -s http://192.168.68.116:80/v1/models \
+  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
+# Must return model list
+```
+
+### Verify Vision/Compression
+```bash
+# Check both api_key and api_key_env are present:
+pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
+# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
+```
+
+## Systemd Pattern
+
+For agents where the gateway runs as a user service:
+```ini
+# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
+[Service]
+Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
+```
+
+For agents where the gateway runs as root:
+```ini
+# /root/.config/systemd/user/hermes-gateway.service
+EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
+```
+
+**No drop-in that hardcodes the master key. Ever.**
+
+## Attachment Cache Locations
+
+| Type | Path |
+|------|------|
+| Images | `~/.hermes/cache/images/` |
+| Documents | `~/.hermes/cache/documents/` |
+| Audio | `~/.hermes/cache/audio/` |
+
+## Related Contracts
+
+- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
+- `hermes-config-template.prose.md` — full configuration template
+- `gpu-fleet.prose.md` — GPU fleet and agent key table
+- `infrastructure-control.prose.md` — CT inventory with pct-run access
+- `litellm-health.prose.md` — LiteLLM stack health verification
+
+## Change Log
+
+| Date | Change |
+|------|--------|
+| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |