prose-contracts/hermes-agent-baseline.prose.md

---
kind: template
name: hermes-agent-baseline
version: 1.0.0
description: >
  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
  configuration state, keys, workarounds, and audit procedure. When an agent's
  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
author: Abiba (pi agent)
---

# Hermes Agent Baseline — Canonical Good State

## Quick Restore

```bash
# Verify all agents against baseline in one command:
for ct in 112 114 111 113; do
    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
done
```

## Agent Map

| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
|-------|-----|------|-----|-------------|---------------|----------|
| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |

Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.

## Key Architecture

```
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
                                                     └── Key DB (Postgres)
```

- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)

## Config Pattern — Mandatory Fields

### For Hermes Agents (Tanko, Mumuni, Baggy)

Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:

### 1. Main Model
```yaml
model:
  default: syslog-auto
  provider: custom:harness       # or: harness
  api_key_env: LITELLM_API_KEY
  max_tokens: 4096
```

### 2. Custom Provider
```yaml
custom_providers:
  - name: harness
    model: syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
```

### 3. Vision (CRITICAL — must have api_key directly!)
```yaml
auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b              # or syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30
```

### 4. Compression (must have api_key!)
```yaml
  compression:
    enabled: true
    threshold: 0.65
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # or gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 120
```

## Known Bug: `api_key_env` Ignored by Auxiliary Client

**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)

**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
to `"no-key-required"` → 401 from LiteLLM.

**Impact**: Vision analysis, compression, and any other auxiliary task calling
LiteLLM/harness will fail with:
```
401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
```

**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
`api_key_env` in every auxiliary task config that uses the harness provider.

**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
`api_key` is empty:
```python
cfg_api_key = str(task_config.get("api_key", "")).strip() or None
if not cfg_api_key:
    key_env = str(task_config.get("api_key_env", "")).strip()
    if key_env:
        cfg_api_key = os.getenv(key_env, "").strip() or None
```

## Audit Procedure

### Full Audit (all agents)
```bash
for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
    pct-run $ct grep LITELLM_API_KEY /etc/environment
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
done
```

### Master Key Leak Check
```bash
# On every agent:
pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
# Must return empty
```

### Verify Key Works
```bash
curl -s http://192.168.68.116:80/v1/models \
  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
# Must return model list
```

### Verify Vision/Compression
```bash
# Check both api_key and api_key_env are present:
pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
```

### For pi Agents (Tdunna)

Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.

**models.json** — Must only list models authorized for the agent's LiteLLM key:
```json
{
  "providers": {
    "syslog-harness": {
      "baseUrl": "http://192.168.68.116/v1",
      "api": "openai-completions",
      "apiKey": "sk-...",
      "models": [
        { "id": "syslog-auto" },
        { "id": "ornith-1.0-35b" },
        { "id": "qwen3.6-27B-code" },
        { "id": "gemma-4-12b" }
      ]
    }
  }
}
```

**settings.json** — Always use `syslog-auto` as default:
```json
{
  "defaultProvider": "syslog-harness",
  "defaultModel": "syslog-auto"
}
```

**Validation**: Verify models match LiteLLM's authorized list:
```bash
curl -s http://192.168.68.116:4000/v1/models \
  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
  | jq '.data[].id'
```

**Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.

**Service**: `pm2 restart koby-zulip`, health at `:9201/health`.

## Systemd Pattern

For agents where the gateway runs as a user service:
```ini
# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
[Service]
Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
```

For agents where the gateway runs as root:
```ini
# /root/.config/systemd/user/hermes-gateway.service
EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
```

**No drop-in that hardcodes the master key. Ever.**

## Attachment Cache Locations

| Type | Path |
|------|------|
| Images | `~/.hermes/cache/images/` |
| Documents | `~/.hermes/cache/documents/` |
| Audio | `~/.hermes/cache/audio/` |

## Health Verification

Run the consolidated health check:
```bash
python3 /root/scripts/agent-health-check.py
```
This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
and counts recent errors. Non-disruptive — never restarts anything.

## GPU Port Conflict Detection

All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
- `.8` and `.110`: inline check in `llama-wrapper.sh`
- `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)

Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
If they differ → ghost detected → kill ghost → start fresh.

## Related Contracts

- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
- `hermes-config-template.prose.md` — full configuration template
- `gpu-fleet.prose.md` — GPU fleet and agent key table
- `infrastructure-control.prose.md` — CT inventory with pct-run access
- `litellm-health.prose.md` — LiteLLM stack health verification

## Change Log

| Date | Change |
|------|--------|
| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |