prose-contracts/hermes-agent-baseline.prose.md

---
kind: template
name: hermes-agent-baseline
version: 1.0.0
description: >
  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
  configuration state, keys, workarounds, and audit procedure. When an agent's
  configuration goes sideways, restore from this baseline. Last verified 2026-07-16. RTX 3090/5070 at 128K (reduced from 256K for stability Jul 2026); Strix Halo at 256K (2026-09-12) (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
author: Abiba (pi agent)
---

# Hermes Agent Baseline — Canonical Good State

## Reachability Detection

Before checking agent baseline, verify the host is reachable and can be audited. Use the shared reachability helper from the clone root:

```bash
# Run on each host to check reachability (Tanko, Mumuni, Koonimo, Koby)
scripts/hermes-reachability-check.sh <host> "api_key:" "/root/.hermes/config.yaml"
# Example: scripts/hermes-reachability-check.sh 192.168.68.122 "api_key:" "/root/.hermes/config.yaml"

# Expected outcomes:
# - UNREACHABLE: SSH connection failed (host is down)
# - VIOLATION: SSH succeeded and found matches (report the finding)
# - COMPLIANT: SSH succeeded and found no matches (no api_key in config)
#
# NOTE: The bug this replaces was deriving reachability from the remote grep's exit code.
# The correct pattern: remote side always succeeds (grep ...; true), so ssh status = connection only.
```

## Quick Restore

```bash
# Verify all agents against baseline in one command:
for ct in 112 114 111 113; do
    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
done
```

## Agent Map

| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------|
| Koby | 111 | storepve | .129 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |

> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
> CT 111 (tdunna, 192.168.68.129, storepve) is report-only — Theo's box; alert only, never garbage-collect.

Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
Keys are stored in Infisical vault (project=agents, env=production) and injected at
runtime via `infisical run --` wrapper. Plaintext keys removed from this baseline.

## Key Architecture

```
Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
                       ↓
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)
                                     └── Key DB (Postgres)
```

- **Master key**: stored in Infisical vault (project=infrastructure, secret=LITELLM_MASTER_KEY) — ADMIN ONLY
- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key injection**: `infisical run --project=agents --env=production -- hermes gateway run` injects `LITELLM_API_KEY` at runtime
- **Key source**: Infisical vault → runtime env var. /etc/environment is CLEAN (stripped, tagged `# [INFISICAL]`)
- **Legacy override** (pre-migration): `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` — should be REMOVED

## Config Pattern — Mandatory Fields

### For Hermes Agents (Mumuni, Koonimo)

Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)

### 1. Main Model
```yaml
model:
  default: syslog-auto
  provider: custom:harness       # or: harness
  api_key_env: LITELLM_API_KEY
  max_tokens: 4096
```

### 2. Custom Provider
```yaml
custom_providers:
  - name: harness
    model: syslog-auto
    base_url: http://192.168.68.116/litellm/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
```

### 3. Vision (CRITICAL — must have api_key directly!)
```yaml
auxiliary:
  vision:
    provider: harness
    model: gpu-vision              # RTX 5070 stable alias (Rule 8; do not use syslog-auto for aux)
    base_url: http://192.168.68.116/litellm/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30
```

### 4. Compression (must have api_key!)
```yaml
  compression:
    enabled: true
    threshold: 0.65
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # Rule 7: compression must be syslog-auto
    base_url: http://192.168.68.116/litellm/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 120
```

## Violation Classification

When reporting findings, separate POLICY observations from FAULT findings:

### POLICY (observation only, not a fault)
- Agent uses a non-internal-harness provider (e.g., direct DeepSeek, Tencent, OpenRouter)
- Config text has a field that looks unusual but the agent's calls are succeeding
- Example: "POLICY: Koonimo uses deepseek directly; calls succeeding in last hour"

### FAULT (requires request-level evidence)
- Agent's calls are failing with auth errors (401/403 in logs)
- Agent's config has no valid API key AND calls are failing
- Example: "FAULT: Koby's LiteLLM key expired; 401 observed at 2026-09-14 11:42:00"

### Rules
1. Do NOT infer the runtime's credential resolution from config text alone.
2. Require request-level evidence before calling something a FAULT: an observed auth failure in the agent's log, or the absence of successful calls in the window.
3. If calls are succeeding, the correct output is "POLICY: uses <provider> directly; calls succeeding" - not a violation.
4. State what you OBSERVED, not what the field implies.

## Known Bug: `api_key_env` Ignored by Auxiliary Client

**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)

**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
to `"no-key-required"` → 401 from LiteLLM.

**Impact**: Vision analysis, compression, and any other auxiliary task calling
LiteLLM/harness will fail with:
```
401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
```

**Workaround**: Set `api_key` directly (copy the value from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=agents --env=production`) alongside `api_key_env` in every auxiliary task config that uses the harness provider.

**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
`api_key` is empty:
```python
cfg_api_key = str(task_config.get("api_key", "")).strip() or None
if not cfg_api_key:
    key_env = str(task_config.get("api_key_env", "")).strip()
    if key_env:
        cfg_api_key = os.getenv(key_env, "").strip() or None
```

## Audit Procedure

### Full Audit (all agents)
```bash
for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
    # Verify /etc/environment is CLEAN (no LITELLM_API_KEY)
    pct-run $ct "grep -c LITELLM_API_KEY /etc/environment 2>/dev/null || echo '0 (clean)'"
    # Verify gateway uses infisical run wrapper
    pct-run $ct "ps aux | grep 'infisical run' | grep -v grep"
    # Check for hardcoded harness keys
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
done
```

### Master Key Leak Check
```bash
# On every agent — must return empty:
pct-run <CT> grep -rl "sk-litellm" /root/ /etc/ 2>/dev/null
# Vault is the only place the master key should exist
```

### Verify Key Works
```bash
# Retrieve key from vault and test:
KEY=$(infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
curl -s http://192.168.68.116:80/v1/models \
  -H "Authorization: Bearer $KEY" | grep syslog-auto
# Must return model list
```

### Verify Vision/Compression
```bash
# Check both api_key and api_key_env are present:
pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
```

### For Koby (CT 111 / tdunna) — **REPORT-ONLY MODE**

Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.

**⛔ KOBY IS NEVER REPAIRED (2026-08-17, Captain)**: Diagnostic only — detect and report, never fix on .129.
No heal step, no restart, no key rotation, no config edit, no memory rewrite, no disk GC, no service touch, no process kill — ever.
If a health check shows Koby degraded, **DO NOT** execute any repair action. Instead, report to Zulip and let Theo fix it.

**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.

### For pi Agents (Abiba)

Abiba (CT100) runs pi via PM2 with the Zulip extension.
Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.

**models.json** — Must only list models authorized for the agent's LiteLLM key.
`/v1/models` is key-scoped and the live registry is CT 116
`/opt/inference-harness/litellm_config.yaml`; treat the list below as a snapshot and re-read
the registry before applying. Key is injected via `infisical run --` wrapper at PM2 startup:
```json
{
  "providers": {
    "syslog-harness": {
      "baseUrl": "http://192.168.68.116/v1",
      "api": "openai-completions",
      "apiKey": "${LITELLM_API_KEY}",
      "models": [
        { "id": "syslog-auto" },
        { "id": "strix-moe" },
        { "id": "gpu-dense" },
        { "id": "gpu-vision" }
      ]
    }
  }
}
```

**settings.json** — Always use `syslog-auto` as default:
```json
{
  "defaultProvider": "syslog-harness",
  "defaultModel": "syslog-auto"
}
```

**Validation**: Verify models match LiteLLM's authorized list:
```bash
curl -s http://192.168.68.116:4000/v1/models \
  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
  | jq '.data[].id'
```

**Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.

**Service**: `pm2 restart koby-zulip`, health at `:9201/health`.

## Systemd Pattern

For agents where the gateway runs as a user service:
```ini
# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
[Service]
Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
```

For agents where the gateway runs as root:
```ini
# /root/.config/systemd/user/hermes-gateway.service
EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
```

**No drop-in that hardcodes the master key. Ever.**

## Attachment Cache Locations

| Type | Path |
|------|------|
| Images | `~/.hermes/cache/images/` |
| Documents | `~/.hermes/cache/documents/` |
| Audio | `~/.hermes/cache/audio/` |

## Health Verification

Run the consolidated health check:
```bash
python3 /root/scripts/agent-health-check.py
```
This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
and counts recent errors. Non-disruptive — never restarts anything.

## GPU Port Conflict Detection

All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
- `.8` and `.110`: inline check in `llama-wrapper.sh`
- `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)

Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
If they differ → ghost detected → kill ghost → start fresh.

## Related Contracts

- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
- `hermes-config-template.prose.md` — full configuration template
- `gpu-fleet.prose.md` — GPU fleet and agent key table
- `infrastructure-control.prose.md` — CT inventory with pct-run access
- `litellm-health.prose.md` — LiteLLM stack health verification

## Change Log

| Date | Change |
|------|--------|
| 2026-07-08 | Koby: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added config section. Key rotated and stored in vault. Added Failure Mode #11 to zulip-adapter-lessons. |
| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |