prose-contracts/hermes-agent-baseline.prose.md

kind, name, version, description, author

kind	name	version	description	author
template	hermes-agent-baseline	1.0.0	Canonical known-good baseline for all Syslog Hermes agents. Captures the exact configuration state, keys, workarounds, and audit procedure. When an agent's configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.	Abiba (pi agent)

Hermes Agent Baseline — Canonical Good State

Quick Restore

# Verify all agents against baseline in one command:
for ct in 112 114 111 113; do
    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
done

Agent Map

Agent	CT	Node	IP	LiteLLM Key	LiteLLM Alias	Platform
Tanko	112	amdpve	.122	sk-CggiHWlamQyShxWC3Hx6uw	tanko	Hermes
Mumuni	114	minipve	.123	sk-VrqCNlwUgzoNGOpikJ7nwQ	mumuni	Hermes
Tdunna	111	amdpve	srv1079750	sk-Qvzi4uYQBhlSK_XstEhcyQ	tdunna	pi
Baggy	113	amdpve	?	sk-krnw_zGBwvvL5b7l2t-s-A	baggy	Hermes

Access: pct-run <CT_ID> <command> — no IPs needed. GPU hosts (.8, .110, .15) use SSH.

Key Architecture

Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
                                                     └── Key DB (Postgres)

• Master key: sk-litellm-7f96080dd99b15c36bd4b333b58a6796 — ADMIN ONLY, never in agent configs
• Agent keys: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
• Key source: /etc/environment → LITELLM_API_KEY=sk-... (systemd service sources this)
• Override: /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf (if present, must match)

Config Pattern — Mandatory Fields

For Hermes Agents (Tanko, Mumuni, Baggy)

Every agent's /root/.hermes/config.yaml (or /home/jerome/.hermes/config.yaml) MUST have:

1. Main Model

model:
  default: syslog-auto
  provider: custom:harness       # or: harness
  api_key_env: LITELLM_API_KEY
  max_tokens: 4096

2. Custom Provider

custom_providers:
  - name: harness
    model: syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions

3. Vision (CRITICAL — must have api_key directly!)

auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b              # or syslog-auto
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 60
    download_timeout: 30

4. Compression (must have api_key!)

  compression:
    enabled: true
    threshold: 0.65
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # or gemma-4-12b
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
    timeout: 120

Known Bug: api_key_env Ignored by Auxiliary Client

Bug location: agent/auxiliary_client.py → _resolve_task_provider_model() (line ~5478)

What happens: The function reads api_key from auxiliary task configs but does NOT resolve api_key_env. If only api_key_env is set (no api_key), the key resolves to None, and the explicit_base_url branch in resolve_provider_client falls through to "no-key-required" → 401 from LiteLLM.

Impact: Vision analysis, compression, and any other auxiliary task calling LiteLLM/harness will fail with:

401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'

Workaround: Set api_key directly (copy the value from /etc/environment) alongside api_key_env in every auxiliary task config that uses the harness provider.

Permanent fix: Patch _resolve_task_provider_model() to resolve api_key_env when api_key is empty:

cfg_api_key = str(task_config.get("api_key", "")).strip() or None
if not cfg_api_key:
    key_env = str(task_config.get("api_key_env", "")).strip()
    if key_env:
        cfg_api_key = os.getenv(key_env, "").strip() or None

Audit Procedure

Full Audit (all agents)

for ct in 112 114 111 113; do
    echo "=== CT $ct ==="
    pct-run $ct grep LITELLM_API_KEY /etc/environment
    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
    echo ""
done

Master Key Leak Check

# On every agent:
pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
# Must return empty

Verify Key Works

curl -s http://192.168.68.116:80/v1/models \
  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
# Must return model list

Verify Vision/Compression

# Check both api_key and api_key_env are present:
pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY

For pi Agents (Tdunna)

Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture). Config files: ~/.pi/agent/models.json, ~/.pi/agent/settings.json.

models.json — Must only list models authorized for the agent's LiteLLM key:

{
  "providers": {
    "syslog-harness": {
      "baseUrl": "http://192.168.68.116/v1",
      "api": "openai-completions",
      "apiKey": "sk-...",
      "models": [
        { "id": "syslog-auto" },
        { "id": "ornith-1.0-35b" },
        { "id": "qwen3.6-27B-code" },
        { "id": "gemma-4-12b" }
      ]
    }
  }
}

settings.json — Always use syslog-auto as default:

{
  "defaultProvider": "syslog-harness",
  "defaultModel": "syslog-auto"
}

Validation: Verify models match LiteLLM's authorized list:

curl -s http://192.168.68.116:4000/v1/models \
  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
  | jq '.data[].id'

Stuck worker detection: In PM2 logs, workers=[<id>:busy:N] with growing N indicates a stuck worker (model error, no agent_end emitted). Fix: correct models.json, delete stale sessions from ~/.pi/agent/sessions/zulip/, restart PM2.

Service: pm2 restart koby-zulip, health at :9201/health.

Systemd Pattern

For agents where the gateway runs as a user service:

# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
[Service]
Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment

For agents where the gateway runs as root:

# /root/.config/systemd/user/hermes-gateway.service
EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY

No drop-in that hardcodes the master key. Ever.

Attachment Cache Locations

Type	Path
Images	~/.hermes/cache/images/
Documents	~/.hermes/cache/documents/
Audio	~/.hermes/cache/audio/

Health Verification

Run the consolidated health check:

python3 /root/scripts/agent-health-check.py

This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes), verifies gateway liveness, confirms Zulip streaming (edit_message present), and counts recent errors. Non-disruptive — never restarts anything.

GPU Port Conflict Detection

All 3 GPU hosts have pre-start ghost detection in their launch wrappers:

• .8 and .110: inline check in llama-wrapper.sh
• .15: /usr/local/bin/port-cleanup.sh (ExecStartPre, replaces blanket pkill)

Detection pattern: ss -tlnp on port 8080 → compare pid against systemctl MainPID. If they differ → ghost detected → kill ghost → start fresh.

Related Contracts

• hermes-key-enforcement.prose.md — key policy, rotation, detection query
• hermes-config-template.prose.md — full configuration template
• gpu-fleet.prose.md — GPU fleet and agent key table
• infrastructure-control.prose.md — CT inventory with pct-run access
• litellm-health.prose.md — LiteLLM stack health verification

Change Log

Date	Change
2026-07-08	Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons.
2026-07-06	Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni.
2026-07-05	Baseline created. All 4 agents audited, master key removed, api_key workaround applied