prose-contracts/hermes-config-template.prose.md

kind, name, description

kind	name	description
pattern	hermes-config-template	Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix), Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key). Previous: full key audit — mumuni-v3 + context7-mcp registered, /etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM keys mapped with active/inactive status.

Maintains

• template_version: "2.0.0"
• last_applied: timestamp
• agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
• agent_keys: managed in LiteLLM DB — current aliases in Key Management section below
• infra_endpoints_verified: array

Agent topology

Agent	CT	IP	Key Alias	Startup	Zulip Owner
Tanko	112	192.168.68.122	tanko-v3	systemd user + .env	Jerome only ✅
Mumuni	114	192.168.68.123	mumuni-v3	systemd user + .env	Jerome only ✅
Koonimo	113	192.168.68.114	koonimo-v2	systemd (Discord/Telegram)	N/A
Koby	111	192.168.68.129	koby-v3	systemd user (Discord/Telegram)	N/A
Abiba	—	192.168.68.24	abiba	pi harness	N/A
Kagenz0	?	?	kagenz0	unknown	not verified

Zulip ownership rules (ALL agents)

Every Zulip-enabled agent MUST only respond to its owner. This is enforced via two env vars in the agent's .env file:

ZULIP_ALLOW_ALL_USERS=false
ZULIP_ALLOWED_USERS=jerome@sysloggh.com

• ZULIP_ALLOW_ALL_USERS=false — disables the default open-to-everyone mode
• ZULIP_ALLOWED_USERS=jerome@sysloggh.com — only Jerome can trigger responses

This applies regardless of whether the agent uses .env directly (Mumuni) or a systemd drop-in (Tanko). Both mechanisms work — the key is that the env vars reach the Hermes process.

Key Management

Source of truth: LiteLLM PostgreSQL DB. Plaintext keys live ONLY in /etc/environment on each agent host, never hardcoded in contracts or config.yaml. This contract defines the procedure; it does NOT store plaintext keys.

Current key aliases (2026-07-03 audit)

Active keys (by agent):

Agent	Alias	Key (last 8)	Source	Status
Tanko	tanko-v3	...JN2OsJw	Process env	Active (needs restart → v2)
Tanko	tanko-v2	...1ka8JA	/etc/environment	Registered, idle
Mumuni	mumuni-v3	...OiWU_B8nw	Process + /etc/env	✅ Aligned
Koonimo	koonimo-v3	...A_wIOA	Config (hardcoded)	Active
Koonimo	koonimo-v2	...R7FAobA	/etc/environment	Registered, idle
Koby	koby-v3	...P4DrWw	/etc/environment	✅ Registered
Koby	koby-hermes	...syslog-koby	Config (deprecated)	Active legacy
Abiba	abiba	...aLgMw	pi config	✅ Active
Kagenz0	kagenz0	...Uggg	unknown	Last active Jul 1
Context7	context7-mcp	...ZPh6VA	Process env	✅ Registered

Unused registrations: abiba-v2, kagenz0-v2, koonimo-v2, synthetic-monitor, UI Dashboard, unnamed key

2026-07-03 alignment: Tanko/Koonimo configs updated to use api_key_env: LITELLM_API_KEY (was hardcoded sk-syslog-*). Base URLs corrected to /v1 (was /litellm/v1). All active keys now registered in LiteLLM. Tanko and Koonimo need Hermes gateway restart for config changes to take effect.

Key lineage

Agent	Alias	Key (last 8)	Status	Date
Tanko	tanko-v3	...JN2OsJw	active (process)	2026-07-03
Tanko	tanko-v2	...1ka8JA	idle (in /etc/env)	2026-07-02
Mumuni	mumuni-v3	...OiWU_B8nw	active	2026-07-03
Mumuni	mumuni-v2	...HcnuOPtQ	superseded	2026-07-02
Mumuni	mumuni	...62da807e	superseded	2026-06-30
Koonimo	koonimo-v3	...A_wIOA	active (hardcoded)	2026-07-03
Koonimo	koonimo-v2	...R7FAobA	idle (in /etc/env)	2026-07-02
Koby	koby-v3	...P4DrWw	active (/etc/env)	2026-07-03
Koby	koby-hermes	...syslog-koby	legacy (in config)	2026-06-30

2026-07-03 incidents:

• Mumuni: double-B8nw — process had ...POiWU_B8nw, /etc/environment had ...VDtHWd-XiWU_B8nw. Wrong one registered first.
• Context7 MCP: sk-XY2aUfvy2BIs6kp1ZPh6VA was never registered — the original error Kwame reported.
• Tanko: process key sk-3ZsdWJbbNSo9zSnJN2OsJw unregistered. Config used deprecated sk-syslog-tanko. Both fixed.
• Koonimo: hardcoded sk-KFnE3Td6aKEBZxwVA_wIOA unregistered. Base URL used old /litellm/v1 path. Both fixed.
• Koby: /etc/environment key sk-x5BnZUUKvz3r8EdLP4DrWw unregistered. Config used deprecated sk-syslog-koby. Fixed.

Verify: all keys working

# Run from Abiba against each agent host
for agent in tanko mumuni; do
  host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123)
  key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2")
  code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models)
  printf "  %-10s -> %s\n" "$agent" "$code"
done

Rotate: generate new key, deploy, verify

AGENT=mumuni && ALIAS="${AGENT}-v3"

# 1. Generate new key in LiteLLM (master key required)
KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \
  -H 'Authorization: Bearer sk-litellm-7f960...' \
  -H 'Content-Type: application/json' \
  -d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))')

# 2. Verify new key
curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models

# 3. Deploy: update /etc/environment on agent host
ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment"

# 4. Restart agent (systemd or kill + relaunch)
ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)"

Key deployment rules

1. Plaintext keys live ONLY in /etc/environment — never in a contract or config.yaml
2. All configs use api_key_env: LITELLM_API_KEY — indirection prevents staleness
3. custom_providers[].api_key MUST use api_key_env — hardcoded keys shadow env vars and bypass rotation
4. LiteLLM DB is the authoritative key list — key aliases with active tokens define who has access
5. Rotate when: LiteLLM container redeployed, any key 401s, or quarterly
6. After rotating any key: restart the corresponding agent gateway
7. Agents without SSH (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login
8. Verification is automated by litellm-self-heal contract (Rule 8: Agent Keys Invalid)

Model name enforcement (2026-07-03)

CRITICAL: Hermes model: auto in any config section gets sent to LiteLLM as the literal model name "auto", which LiteLLM rejects with 403 because the registered model is syslog-auto, not auto.

Must-fix pattern — any auxiliary section with model: auto:

# ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden)
auxiliary:
  approval:
    model: auto
    provider: auto
  skills_hub:
    model: auto
    provider: auto

# ✅ FIXED — registered model name
auxiliary:
  approval:
    model: syslog-auto
    provider: harness
    api_key_env: LITELLM_API_KEY
    base_url: http://192.168.68.116/v1

Detection: grep -rn "model: auto" /root/.hermes/config.yaml on any Hermes host — if any results found in auxiliary sections, fix immediately.

Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT)

Agent	CT	Key Alias	Config Status	Key Verified
Tanko	112	tanko-v2	⚠️ Hardcoded (correct key, no api_key_env)	✅ 200
Mumuni	114	mumuni-v3	✅ Clean (api_key_env)	✅ 200
Koby	111	koby-v3	✅ Clean (11 fixes applied)	✅ 200
Koonimo	113	koonimo-v2	✅ Clean (3 fixes applied)	✅ 200
Abiba	100	abiba	✅ pi native	✅ 200
Kagenz0	105	kagenz0-v2	❌ DOWN (no hermes, no config)	—

Fix summary (2026-07-03 second pass)

Koby (11 fixes):

• 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract)
• custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto
• delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env

Mumuni (1 fix):

• delegation: model:auto → syslog-auto + provider:harness + api_key_env

Koonimo (3 fixes, first pass):

• auxiliary.approval: model:auto → syslog-auto, provider:auto → harness
• auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness
• custom_providers.harness: hardcoded koonimo-v3 → api_key_env

Tanko (flagged, not urgent):

• Config has hardcoded tanko-v2 key (correct) instead of api_key_env
• DeepSeek fallback key (sk-b7d9...) is intentional — external service
• Will need fix before next key rotation

Mumuni sub-agent profiles (6)

Sub-agents in /root/.hermes/profiles/<name>/config.yaml: syslog-code, syslog-devops, syslog-email, syslog-research, syslog-review, syslog-writer.

All sub-agent profiles inherit auth: api_key: '', base_url: ''. When the main key is rotated, all 7 configs (main + 6 subs) stay valid — only /etc/environment needs updating.

Infrastructure Stack

Component	Endpoint	Purpose
Firecrawl	http://192.168.68.7:3002/	Web content extraction
SearXNG	http://192.168.68.7:8888	Privacy-respecting web search
LiteLLM	http://192.168.68.116/v1	Unified model gateway (via nginx)
LiteLLM (NetBird)	https://litellm.sysloggh.net/v1	Alternative (may have 502 issues)
RA-H OS MCP	http://192.168.68.65:3100/mcp	Knowledge graph bridge
Context7 MCP	http://localhost:8079/mcp	Documentation queries

API Key Rules

• api_key_env: LITELLM_API_KEY — Use env var for main model auth (preferred)
• api_key: '' — Sub-agents leave empty to inherit from main config's custom_provider
• api_key: sk-... — Hardcoded key only as fallback when env var not possible
• Set LITELLM_API_KEY in /etc/environment on each host
• Sub-agents NEVER get their own key — they share the host agent's key
• Restart Hermes after updating /etc/environment

Sub-Agent Profiles (Mumuni pattern)

Mumuni has 6 sub-agent profiles in /root/.hermes/profiles/<name>/config.yaml:

profiles/
├── syslog-code/config.yaml      # Code generation
├── syslog-devops/config.yaml     # DevOps/infrastructure
├── syslog-email/config.yaml      # Email processing
├── syslog-research/config.yaml   # Research & analysis
├── syslog-review/config.yaml     # Code review
└── syslog-writer/config.yaml     # Content writing

Sub-agent profile rules:

1. api_key must be empty — api_key: '' or omitted entirely
2. base_url must be empty — inherits from main config's custom_provider
3. provider is auto or harness — routes through the shared LiteLLM gateway
4. model is agent-specific — each sub-agent can have its own default model
5. Auxiliary tasks (vision, compression, etc.) also leave api_key empty
6. Never hardcode a key in sub-agent profiles

This ensures all 6 sub-agents use the same LiteLLM key set in /etc/environment. When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs) work immediately after restart.

Template — Required Sections

# ─── Model Selection ───
model:
  default: <agent_model>          # e.g., qwen3.6-27B-code, syslog-auto
  provider: harness
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Set in /etc/environment

fallback_providers:
  provider: deepseek
  model: deepseek-chat
  api_key_env: DEEPSEEK_API_KEY

# ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
web:
  backend: firecrawl
  search_backend: searxng
  extract_backend: firecrawl
  firecrawl:
    base_url: http://192.168.68.7:3002/

# ─── MCP Servers (SHARED INFRA) ───
mcp_servers:
  context7:
    connect_timeout: 60
    timeout: 300
    url: http://localhost:8079/mcp
  ra-h-os:
    url: http://192.168.68.65:3100/mcp
    timeout: 120
    connect_timeout: 60

# ─── Compression ───
compression:
  enabled: true
  provider: harness
  model: gemma-4-12b
  api_key_env: LITELLM_API_KEY
  threshold: 0.5
  target_ratio: 0.25
  protect_last_n: 30
  hygiene_hard_message_limit: 400

# ─── Auxiliary Tasks ───
auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b
    api_key_env: LITELLM_API_KEY
  web_extract:
    provider: harness
    model: gemma-4-12b
    api_key_env: LITELLM_API_KEY
  session_search:
    provider: harness
    model: gemma-4-12b
    max_concurrency: 3
    api_key_env: LITELLM_API_KEY

# ─── Custom Provider ───
custom_providers:
  - name: harness
    model: <agent_model>
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions

Key Update Procedure

When LiteLLM keys are regenerated (e.g., after infrastructure changes):

1. If SSH available: ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"
2. If SSH unavailable: Send Zulip DM via abiba-bot with update command
3. After update: Restart Hermes on the agent host
4. Verify: curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models

Configuration Rules

Rule 1: Shared Infra Is Locked

The following MUST be identical across ALL profiles:

• web.backend, web.search_backend, web.extract_backend
• web.firecrawl.base_url
• mcp_servers.ra-h-os.url
• custom_providers[0].base_url

Rule 2: Model Choice Is Free

• model.default — per agent
• fallback_providers.model — per agent
• custom_providers[0].model — per agent

Rule 3: API Keys via Environment

• Prefer api_key_env: LITELLM_API_KEY over hardcoded keys
• Hardcoded keys in config.yaml become stale after key rotation
• /etc/environment persists across config updates
• Restart Hermes after env var updates

Rule 4: Sub-Agent Profiles Inherit Auth

• Sub-agent profiles (/root/.hermes/profiles/*/config.yaml) must have:
  • api_key: '' — inherit from main config's custom_provider
  • base_url: '' — inherit from main config
  • Auxiliary tasks: api_key: '', provider: harness
• Never hardcode a key in sub-agent profiles
• When main config uses api_key_env, sub-agents automatically use it
• This means key rotation only touches ONE file (/etc/environment)

Rule 5: Main Config Base URL

• Use direct IP: http://192.168.68.116/v1
• NOT the NetBird URL (litellm.sysloggh.net) — can cause 502 when NetBird is down
• NOT the old path (/litellm/v1) — nginx now routes /v1 directly

Execution

1. Check current config — Read the target agent's config.yaml
2. Compare against template — Identify missing or divergent sections
3. Apply shared infra — Lock web/MCP/compression sections to template values
4. Apply agent key — Set from agent_keys table above
5. Set model choice — Per agent's workload
6. Verify — curl all shared endpoints, test the model with the new key
7. Report — What was changed, preserved, custom