prose-contracts/hermes-config-template.prose.md

---
kind: pattern
name: hermes-config-template
description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
---

## Maintains

- template_version: "2.0.0"
- last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
- agent_keys: managed in LiteLLM DB — current aliases in Key Management section below
- infra_endpoints_verified: array

## Key Management

**Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in
`/etc/environment` on each agent host, never hardcoded in contracts or config.yaml.
This contract defines the procedure; it does NOT store plaintext keys.

### Current key aliases (2026-07-02 rotation)
- `tanko-v2`, `mumuni-v2`, `abiba-v2`, `koby-v2`, `koonimo-v2`, `kagenz0-v2`
- `synthetic-monitor` (daily infra report)

### Verify: all keys working
```bash
# Run from Abiba against each agent host
for agent in tanko mumuni; do
  host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123)
  key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2")
  code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models)
  printf "  %-10s -> %s\n" "$agent" "$code"
done
```

### Rotate: generate new key, deploy, verify
```bash
AGENT=mumuni && ALIAS="${AGENT}-v2"

# 1. Generate new key in LiteLLM (master key required)
KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \
  -H 'Authorization: Bearer sk-litellm-7f960...' \
  -H 'Content-Type: application/json' \
  -d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))')

# 2. Verify new key
curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models

# 3. Deploy: update /etc/environment on agent host
ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment"

# 4. Restart agent (systemd or kill + relaunch)
ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)"
```

### Key deployment rules
1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml
2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness
3. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access
4. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly
5. **After rotating any key:** restart the corresponding agent gateway
6. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login
7. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid)

### Mumuni sub-agent profiles (6)
Sub-agents in `/root/.hermes/profiles/<name>/config.yaml`:
`syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`,
`syslog-review`, `syslog-writer`.

All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`.
When the main key is rotated, all 7 configs (main + 6 subs) stay valid —
only `/etc/environment` needs updating.

## Infrastructure Stack

| Component | Endpoint | Purpose |
|---|---|---|
| Firecrawl | `http://192.168.68.7:3002/` | Web content extraction |
| SearXNG | `http://storepve:8888` | Privacy-respecting web search |
| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) |
| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) |
| RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge |
| Context7 MCP | `http://localhost:8079/mcp` | Documentation queries |

## API Key Rules

- `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred)
- `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider
- `api_key: sk-...` — Hardcoded key only as fallback when env var not possible
- Set `LITELLM_API_KEY` in `/etc/environment` on each host
- Sub-agents NEVER get their own key — they share the host agent's key
- Restart Hermes after updating `/etc/environment`

### Sub-Agent Profiles (Mumuni pattern)

Mumuni has 6 sub-agent profiles in `/root/.hermes/profiles/<name>/config.yaml`:

```
profiles/
├── syslog-code/config.yaml      # Code generation
├── syslog-devops/config.yaml     # DevOps/infrastructure
├── syslog-email/config.yaml      # Email processing
├── syslog-research/config.yaml   # Research & analysis
├── syslog-review/config.yaml     # Code review
└── syslog-writer/config.yaml     # Content writing
```

Sub-agent profile rules:
1. **`api_key` must be empty** — `api_key: ''` or omitted entirely
2. **`base_url` must be empty** — inherits from main config's custom_provider
3. **`provider` is `auto` or `harness`** — routes through the shared LiteLLM gateway
4. **`model` is agent-specific** — each sub-agent can have its own default model
5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty
6. **Never hardcode a key** in sub-agent profiles

This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`.
When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
work immediately after restart.

## Template — Required Sections

```yaml
# ─── Model Selection ───
model:
  default: <agent_model>          # e.g., qwen3.6-27B-code, syslog-auto
  provider: harness
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Set in /etc/environment

fallback_providers:
  provider: deepseek
  model: deepseek-chat
  api_key_env: DEEPSEEK_API_KEY

# ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
web:
  backend: firecrawl
  search_backend: searxng
  extract_backend: firecrawl
  firecrawl:
    base_url: http://192.168.68.7:3002/

# ─── MCP Servers (SHARED INFRA) ───
mcp_servers:
  context7:
    connect_timeout: 60
    timeout: 300
    url: http://localhost:8079/mcp
  ra-h-os:
    url: http://192.168.68.65:3100/mcp
    timeout: 120
    connect_timeout: 60

# ─── Compression ───
compression:
  enabled: true
  provider: harness
  model: gemma-4-12b
  api_key_env: LITELLM_API_KEY
  threshold: 0.5
  target_ratio: 0.25
  protect_last_n: 30
  hygiene_hard_message_limit: 400

# ─── Auxiliary Tasks ───
auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b
    api_key_env: LITELLM_API_KEY
  web_extract:
    provider: harness
    model: gemma-4-12b
    api_key_env: LITELLM_API_KEY
  session_search:
    provider: harness
    model: gemma-4-12b
    max_concurrency: 3
    api_key_env: LITELLM_API_KEY

# ─── Custom Provider ───
custom_providers:
  - name: harness
    model: <agent_model>
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
```

## Key Update Procedure

When LiteLLM keys are regenerated (e.g., after infrastructure changes):

1. **If SSH available**: `ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"`
2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command
3. **After update**: Restart Hermes on the agent host
4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`

## Configuration Rules

### Rule 1: Shared Infra Is Locked
The following MUST be identical across ALL profiles:
- `web.backend`, `web.search_backend`, `web.extract_backend`
- `web.firecrawl.base_url`
- `mcp_servers.ra-h-os.url`
- `custom_providers[0].base_url`

### Rule 2: Model Choice Is Free
- `model.default` — per agent
- `fallback_providers.model` — per agent
- `custom_providers[0].model` — per agent

### Rule 3: API Keys via Environment
- Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys
- Hardcoded keys in config.yaml become stale after key rotation
- `/etc/environment` persists across config updates
- Restart Hermes after env var updates

### Rule 4: Sub-Agent Profiles Inherit Auth
- Sub-agent profiles (`/root/.hermes/profiles/*/config.yaml`) must have:
  - `api_key: ''` — inherit from main config's custom_provider
  - `base_url: ''` — inherit from main config
  - Auxiliary tasks: `api_key: ''`, `provider: harness`
- Never hardcode a key in sub-agent profiles
- When main config uses `api_key_env`, sub-agents automatically use it
- This means key rotation only touches ONE file (`/etc/environment`)

### Rule 5: Main Config Base URL
- Use direct IP: `http://192.168.68.116/v1`
- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down
- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly

## Execution

1. **Check current config** — Read the target agent's config.yaml
2. **Compare against template** — Identify missing or divergent sections
3. **Apply shared infra** — Lock web/MCP/compression sections to template values
4. **Apply agent key** — Set from agent_keys table above
5. **Set model choice** — Per agent's workload
6. **Verify** — curl all shared endpoints, test the model with the new key
7. **Report** — What was changed, preserved, custom