docs: hermes-agent-baseline — canonical good-state reference
Single source of truth for all 4 Hermes agents. Captures: - Agent map (CT, IP, key, alias) - Key architecture and config patterns - Mandatory api_key workaround with explanation - Known bug: auxiliary_client ignores api_key_env - Audit procedure (full audit, leak check, key verification) - Systemd patterns - Attachment cache locations
This commit is contained in:
root
@@ -0,0 +1,194 @@
|
|||||||
|
---
|
||||||
|
kind: reference
|
||||||
|
name: hermes-agent-baseline
|
||||||
|
version: 1.0.0
|
||||||
|
description: >
|
||||||
|
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
||||||
|
configuration state, keys, workarounds, and audit procedure. When an agent's
|
||||||
|
configuration goes sideways, restore from this baseline. Last verified 2026-07-05.
|
||||||
|
author: Abiba (pi agent)
|
||||||
|
---
|
||||||
|
|
||||||
|
# Hermes Agent Baseline — Canonical Good State
|
||||||
|
|
||||||
|
## Quick Restore
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Verify all agents against baseline in one command:
|
||||||
|
for ct in 112 114 111 113; do
|
||||||
|
echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Agent Map
|
||||||
|
|
||||||
|
| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias |
|
||||||
|
|-------|-----|------|-----|-------------|---------------|
|
||||||
|
| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` |
|
||||||
|
| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` |
|
||||||
|
| Tdunna | 111 | amdpve | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | `tdunna` |
|
||||||
|
| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` |
|
||||||
|
|
||||||
|
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
|
||||||
|
|
||||||
|
## Key Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
|
||||||
|
└── Key DB (Postgres)
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
|
||||||
|
- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
|
||||||
|
- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
|
||||||
|
- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
|
||||||
|
|
||||||
|
## Config Pattern — Mandatory Fields
|
||||||
|
|
||||||
|
Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
|
||||||
|
|
||||||
|
### 1. Main Model
|
||||||
|
```yaml
|
||||||
|
model:
|
||||||
|
default: syslog-auto
|
||||||
|
provider: custom:harness # or: harness
|
||||||
|
api_key_env: LITELLM_API_KEY
|
||||||
|
max_tokens: 4096
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Custom Provider
|
||||||
|
```yaml
|
||||||
|
custom_providers:
|
||||||
|
- name: harness
|
||||||
|
model: syslog-auto
|
||||||
|
base_url: http://192.168.68.116/v1
|
||||||
|
api_key_env: LITELLM_API_KEY
|
||||||
|
api_mode: chat_completions
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Vision (CRITICAL — must have api_key directly!)
|
||||||
|
```yaml
|
||||||
|
auxiliary:
|
||||||
|
vision:
|
||||||
|
provider: harness
|
||||||
|
model: gemma-4-12b # or syslog-auto
|
||||||
|
base_url: http://192.168.68.116/v1
|
||||||
|
api_key_env: LITELLM_API_KEY
|
||||||
|
api_key: <ACTUAL_KEY_FROM_/etc/environment> # ← MANDATORY workaround
|
||||||
|
timeout: 60
|
||||||
|
download_timeout: 30
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Compression (must have api_key!)
|
||||||
|
```yaml
|
||||||
|
compression:
|
||||||
|
enabled: true
|
||||||
|
threshold: 0.65
|
||||||
|
target_ratio: 0.3
|
||||||
|
provider: harness
|
||||||
|
model: syslog-auto # or gemma-4-12b
|
||||||
|
base_url: http://192.168.68.116/v1
|
||||||
|
api_key_env: LITELLM_API_KEY
|
||||||
|
api_key: <ACTUAL_KEY_FROM_/etc/environment> # ← MANDATORY workaround
|
||||||
|
timeout: 120
|
||||||
|
```
|
||||||
|
|
||||||
|
## Known Bug: `api_key_env` Ignored by Auxiliary Client
|
||||||
|
|
||||||
|
**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
|
||||||
|
|
||||||
|
**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
|
||||||
|
resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
|
||||||
|
to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
|
||||||
|
to `"no-key-required"` → 401 from LiteLLM.
|
||||||
|
|
||||||
|
**Impact**: Vision analysis, compression, and any other auxiliary task calling
|
||||||
|
LiteLLM/harness will fail with:
|
||||||
|
```
|
||||||
|
401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
|
||||||
|
`api_key_env` in every auxiliary task config that uses the harness provider.
|
||||||
|
|
||||||
|
**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
|
||||||
|
`api_key` is empty:
|
||||||
|
```python
|
||||||
|
cfg_api_key = str(task_config.get("api_key", "")).strip() or None
|
||||||
|
if not cfg_api_key:
|
||||||
|
key_env = str(task_config.get("api_key_env", "")).strip()
|
||||||
|
if key_env:
|
||||||
|
cfg_api_key = os.getenv(key_env, "").strip() or None
|
||||||
|
```
|
||||||
|
|
||||||
|
## Audit Procedure
|
||||||
|
|
||||||
|
### Full Audit (all agents)
|
||||||
|
```bash
|
||||||
|
for ct in 112 114 111 113; do
|
||||||
|
echo "=== CT $ct ==="
|
||||||
|
pct-run $ct grep LITELLM_API_KEY /etc/environment
|
||||||
|
pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
```
|
||||||
|
|
||||||
|
### Master Key Leak Check
|
||||||
|
```bash
|
||||||
|
# On every agent:
|
||||||
|
pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
|
||||||
|
# Must return empty
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Key Works
|
||||||
|
```bash
|
||||||
|
curl -s http://192.168.68.116:80/v1/models \
|
||||||
|
-H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
|
||||||
|
# Must return model list
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Vision/Compression
|
||||||
|
```bash
|
||||||
|
# Check both api_key and api_key_env are present:
|
||||||
|
pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
|
||||||
|
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
## Systemd Pattern
|
||||||
|
|
||||||
|
For agents where the gateway runs as a user service:
|
||||||
|
```ini
|
||||||
|
# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
|
||||||
|
[Service]
|
||||||
|
Environment="LITELLM_API_KEY=sk-..." # must match /etc/environment
|
||||||
|
```
|
||||||
|
|
||||||
|
For agents where the gateway runs as root:
|
||||||
|
```ini
|
||||||
|
# /root/.config/systemd/user/hermes-gateway.service
|
||||||
|
EnvironmentFile=/etc/environment # sources LITELLM_API_KEY
|
||||||
|
```
|
||||||
|
|
||||||
|
**No drop-in that hardcodes the master key. Ever.**
|
||||||
|
|
||||||
|
## Attachment Cache Locations
|
||||||
|
|
||||||
|
| Type | Path |
|
||||||
|
|------|------|
|
||||||
|
| Images | `~/.hermes/cache/images/` |
|
||||||
|
| Documents | `~/.hermes/cache/documents/` |
|
||||||
|
| Audio | `~/.hermes/cache/audio/` |
|
||||||
|
|
||||||
|
## Related Contracts
|
||||||
|
|
||||||
|
- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
|
||||||
|
- `hermes-config-template.prose.md` — full configuration template
|
||||||
|
- `gpu-fleet.prose.md` — GPU fleet and agent key table
|
||||||
|
- `infrastructure-control.prose.md` — CT inventory with pct-run access
|
||||||
|
- `litellm-health.prose.md` — LiteLLM stack health verification
|
||||||
|
|
||||||
|
## Change Log
|
||||||
|
|
||||||
|
| Date | Change |
|
||||||
|
|------|--------|
|
||||||
|
| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
|
||||||
Reference in New Issue
Block a user