Files
prose-contracts/hermes-config-template.prose.md
T
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00

7.4 KiB

kind, name, description
kind name description
template hermes-config-template Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.

Maintains

  • template_version: "2.0.0"
  • last_applied: timestamp
  • agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
  • agent_keys: map (see Agent Keys section)
  • infra_endpoints_verified: array

Agent Keys (LiteLLM — Current 2026-06-30)

Each agent has a unique LiteLLM API key. Keys are stored in LiteLLM's PostgreSQL DB. The env var LITELLM_API_KEY is set in /etc/environment on each agent host. Sub-agent profiles inherit auth from the main config — no separate keys needed.

Agent Key Host SSH Sub-Agents
Tanko sk-3ZsdWJbbNSo9zSnJN2OsJw 192.168.68.122 jerome@.122
Mumuni sk-XY2aUfvy2BIs6kp1ZPh6VA 192.168.68.123 root@.123 6 profiles ✱
Abiba sk-kxbPgbnV2Zkn6TKQbAEcEg 192.168.68.24 local
Koby sk-wb86PvcZXWkjShAIBxrpDg ? Zulip
Koonimo sk-Prx_vRXYb2VEzDmhPCbNeg 192.168.68.114 Zulip
Kagenz0 sk-Dh4CDkaHebMLEp8qqq20qA ? Zulip

✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research, syslog-review, syslog-writer — all at /root/.hermes/profiles/<name>/config.yaml

Infrastructure Stack

Component Endpoint Purpose
Firecrawl http://192.168.68.7:3002/ Web content extraction
SearXNG http://storepve:8888 Privacy-respecting web search
LiteLLM http://192.168.68.116/v1 Unified model gateway (via nginx)
LiteLLM (NetBird) https://litellm.sysloggh.net/v1 Alternative (may have 502 issues)
RA-H OS MCP http://192.168.68.65:3100/mcp Knowledge graph bridge
Context7 MCP http://localhost:8079/mcp Documentation queries

API Key Rules

  • api_key_env: LITELLM_API_KEY — Use env var for main model auth (preferred)
  • api_key: '' — Sub-agents leave empty to inherit from main config's custom_provider
  • api_key: sk-... — Hardcoded key only as fallback when env var not possible
  • Set LITELLM_API_KEY in /etc/environment on each host
  • Sub-agents NEVER get their own key — they share the host agent's key
  • Restart Hermes after updating /etc/environment

Sub-Agent Profiles (Mumuni pattern)

Mumuni has 6 sub-agent profiles in /root/.hermes/profiles/<name>/config.yaml:

profiles/
├── syslog-code/config.yaml      # Code generation
├── syslog-devops/config.yaml     # DevOps/infrastructure
├── syslog-email/config.yaml      # Email processing
├── syslog-research/config.yaml   # Research & analysis
├── syslog-review/config.yaml     # Code review
└── syslog-writer/config.yaml     # Content writing

Sub-agent profile rules:

  1. api_key must be emptyapi_key: '' or omitted entirely
  2. base_url must be empty — inherits from main config's custom_provider
  3. provider is auto or harness — routes through the shared LiteLLM gateway
  4. model is agent-specific — each sub-agent can have its own default model
  5. Auxiliary tasks (vision, compression, etc.) also leave api_key empty
  6. Never hardcode a key in sub-agent profiles

This ensures all 6 sub-agents use the same LiteLLM key set in /etc/environment. When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs) work immediately after restart.

Template — Required Sections

# ─── Model Selection ───
model:
  default: <agent_model>          # e.g., qwen3.6-27B-code, syslog-auto
  provider: harness
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Set in /etc/environment

fallback_providers:
  provider: deepseek
  model: deepseek-chat
  api_key_env: DEEPSEEK_API_KEY

# ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
web:
  backend: firecrawl
  search_backend: searxng
  extract_backend: firecrawl
  firecrawl:
    base_url: http://192.168.68.7:3002/

# ─── MCP Servers (SHARED INFRA) ───
mcp_servers:
  context7:
    connect_timeout: 60
    timeout: 300
    url: http://localhost:8079/mcp
  ra-h-os:
    url: http://192.168.68.65:3100/mcp
    timeout: 120
    connect_timeout: 60

# ─── Compression ───
compression:
  enabled: true
  provider: harness
  model: gemma-4-12b
  api_key_env: LITELLM_API_KEY
  threshold: 0.5
  target_ratio: 0.25
  protect_last_n: 30
  hygiene_hard_message_limit: 400

# ─── Auxiliary Tasks ───
auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b
    api_key_env: LITELLM_API_KEY
  web_extract:
    provider: harness
    model: gemma-4-12b
    api_key_env: LITELLM_API_KEY
  session_search:
    provider: harness
    model: gemma-4-12b
    max_concurrency: 3
    api_key_env: LITELLM_API_KEY

# ─── Custom Provider ───
custom_providers:
  - name: harness
    model: <agent_model>
    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions

Key Update Procedure

When LiteLLM keys are regenerated (e.g., after infrastructure changes):

  1. If SSH available: ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment"
  2. If SSH unavailable: Send Zulip DM via abiba-bot with update command
  3. After update: Restart Hermes on the agent host
  4. Verify: curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models

Configuration Rules

Rule 1: Shared Infra Is Locked

The following MUST be identical across ALL profiles:

  • web.backend, web.search_backend, web.extract_backend
  • web.firecrawl.base_url
  • mcp_servers.ra-h-os.url
  • custom_providers[0].base_url

Rule 2: Model Choice Is Free

  • model.default — per agent
  • fallback_providers.model — per agent
  • custom_providers[0].model — per agent

Rule 3: API Keys via Environment

  • Prefer api_key_env: LITELLM_API_KEY over hardcoded keys
  • Hardcoded keys in config.yaml become stale after key rotation
  • /etc/environment persists across config updates
  • Restart Hermes after env var updates

Rule 4: Sub-Agent Profiles Inherit Auth

  • Sub-agent profiles (/root/.hermes/profiles/*/config.yaml) must have:
    • api_key: '' — inherit from main config's custom_provider
    • base_url: '' — inherit from main config
    • Auxiliary tasks: api_key: '', provider: harness
  • Never hardcode a key in sub-agent profiles
  • When main config uses api_key_env, sub-agents automatically use it
  • This means key rotation only touches ONE file (/etc/environment)

Rule 5: Main Config Base URL

  • Use direct IP: http://192.168.68.116/v1
  • NOT the NetBird URL (litellm.sysloggh.net) — can cause 502 when NetBird is down
  • NOT the old path (/litellm/v1) — nginx now routes /v1 directly

Execution

  1. Check current config — Read the target agent's config.yaml
  2. Compare against template — Identify missing or divergent sections
  3. Apply shared infra — Lock web/MCP/compression sections to template values
  4. Apply agent key — Set from agent_keys table above
  5. Set model choice — Per agent's workload
  6. Verify — curl all shared endpoints, test the model with the new key
  7. Report — What was changed, preserved, custom