From 2ea9cac23a3d72152a3d1f356cb31866e2960878 Mon Sep 17 00:00:00 2001 From: root Date: Sun, 5 Jul 2026 23:37:56 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20hermes-agent-baseline=20=E2=80=94=20can?= =?UTF-8?q?onical=20good-state=20reference?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single source of truth for all 4 Hermes agents. Captures: - Agent map (CT, IP, key, alias) - Key architecture and config patterns - Mandatory api_key workaround with explanation - Known bug: auxiliary_client ignores api_key_env - Audit procedure (full audit, leak check, key verification) - Systemd patterns - Attachment cache locations --- hermes-agent-baseline.prose.md | 194 +++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 hermes-agent-baseline.prose.md diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md new file mode 100644 index 0000000..4126fee --- /dev/null +++ b/hermes-agent-baseline.prose.md @@ -0,0 +1,194 @@ +--- +kind: reference +name: hermes-agent-baseline +version: 1.0.0 +description: > + Canonical known-good baseline for all Syslog Hermes agents. Captures the exact + configuration state, keys, workarounds, and audit procedure. When an agent's + configuration goes sideways, restore from this baseline. Last verified 2026-07-05. +author: Abiba (pi agent) +--- + +# Hermes Agent Baseline — Canonical Good State + +## Quick Restore + +```bash +# Verify all agents against baseline in one command: +for ct in 112 114 111 113; do + echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found" +done +``` + +## Agent Map + +| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | +|-------|-----|------|-----|-------------|---------------| +| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | +| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | +| Tdunna | 111 | amdpve | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | `tdunna` | +| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | + +Access: `pct-run ` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. + +## Key Architecture + +``` +Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server) + └── Key DB (Postgres) +``` + +- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs +- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name +- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this) +- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match) + +## Config Pattern — Mandatory Fields + +Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have: + +### 1. Main Model +```yaml +model: + default: syslog-auto + provider: custom:harness # or: harness + api_key_env: LITELLM_API_KEY + max_tokens: 4096 +``` + +### 2. Custom Provider +```yaml +custom_providers: + - name: harness + model: syslog-auto + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY + api_mode: chat_completions +``` + +### 3. Vision (CRITICAL — must have api_key directly!) +```yaml +auxiliary: + vision: + provider: harness + model: gemma-4-12b # or syslog-auto + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY + api_key: # ← MANDATORY workaround + timeout: 60 + download_timeout: 30 +``` + +### 4. Compression (must have api_key!) +```yaml + compression: + enabled: true + threshold: 0.65 + target_ratio: 0.3 + provider: harness + model: syslog-auto # or gemma-4-12b + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY + api_key: # ← MANDATORY workaround + timeout: 120 +``` + +## Known Bug: `api_key_env` Ignored by Auxiliary Client + +**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478) + +**What happens**: The function reads `api_key` from auxiliary task configs but does NOT +resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves +to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through +to `"no-key-required"` → 401 from LiteLLM. + +**Impact**: Vision analysis, compression, and any other auxiliary task calling +LiteLLM/harness will fail with: +``` +401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-' +``` + +**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside +`api_key_env` in every auxiliary task config that uses the harness provider. + +**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when +`api_key` is empty: +```python +cfg_api_key = str(task_config.get("api_key", "")).strip() or None +if not cfg_api_key: + key_env = str(task_config.get("api_key_env", "")).strip() + if key_env: + cfg_api_key = os.getenv(key_env, "").strip() or None +``` + +## Audit Procedure + +### Full Audit (all agents) +```bash +for ct in 112 114 111 113; do + echo "=== CT $ct ===" + pct-run $ct grep LITELLM_API_KEY /etc/environment + pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env + echo "" +done +``` + +### Master Key Leak Check +```bash +# On every agent: +pct-run grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null +# Must return empty +``` + +### Verify Key Works +```bash +curl -s http://192.168.68.116:80/v1/models \ + -H "Authorization: Bearer " | grep syslog-auto +# Must return model list +``` + +### Verify Vision/Compression +```bash +# Check both api_key and api_key_env are present: +pct-run grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key +# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY +``` + +## Systemd Pattern + +For agents where the gateway runs as a user service: +```ini +# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf +[Service] +Environment="LITELLM_API_KEY=sk-..." # must match /etc/environment +``` + +For agents where the gateway runs as root: +```ini +# /root/.config/systemd/user/hermes-gateway.service +EnvironmentFile=/etc/environment # sources LITELLM_API_KEY +``` + +**No drop-in that hardcodes the master key. Ever.** + +## Attachment Cache Locations + +| Type | Path | +|------|------| +| Images | `~/.hermes/cache/images/` | +| Documents | `~/.hermes/cache/documents/` | +| Audio | `~/.hermes/cache/audio/` | + +## Related Contracts + +- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query +- `hermes-config-template.prose.md` — full configuration template +- `gpu-fleet.prose.md` — GPU fleet and agent key table +- `infrastructure-control.prose.md` — CT inventory with pct-run access +- `litellm-health.prose.md` — LiteLLM stack health verification + +## Change Log + +| Date | Change | +|------|--------| +| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |