--- kind: pattern name: hermes-config-template description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. Updated 2026-07-03 (19:14 EDT): Full sweep — Koby (11 fixes), Mumuni (1 fix), Koonimo (3 fixes). All agents now CLEAN: zero model:auto, zero deprecated sk-syslog-* keys. Tanko has hardcoded tanko-v2 key (correct, but should use api_key_env for rotation safety). Kagenz0 is DOWN (no config, no key). Previous: full key audit — mumuni-v3 + context7-mcp registered, /etc/environment aligned, double-B8nw incident documented, all 17 LiteLLM keys mapped with active/inactive status. --- ## Maintains - template_version: "2.0.0" - last_applied: timestamp - agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"] - agent_keys: managed in LiteLLM DB — current aliases in Key Management section below - infra_endpoints_verified: array ### Agent topology | Agent | CT | IP | Key Alias | Startup | Zulip Owner | |-------|-----|-----|-----------|---------|-------------| | Tanko | 112 | 192.168.68.122 | `tanko-v3` | systemd user + `.env` | Jerome only ✅ | | Mumuni | 114 | 192.168.68.123 | `mumuni-v3` | systemd user + `.env` | Jerome only ✅ | | Koonimo | 113 | 192.168.68.114 | `koonimo-v2` | systemd (Discord/Telegram) | N/A | | Koby | 111 | 192.168.68.129 | `koby-v3` | systemd user (Discord/Telegram) | N/A | | Abiba | — | 192.168.68.24 | `abiba` | `pi` harness | N/A | | Kagenz0 | ? | ? | `kagenz0` | unknown | not verified | ### Zulip ownership rules (ALL agents) **Every Zulip-enabled agent MUST only respond to its owner.** This is enforced via two env vars in the agent's `.env` file: ```bash ZULIP_ALLOW_ALL_USERS=false ZULIP_ALLOWED_USERS=jerome@sysloggh.com ``` - `ZULIP_ALLOW_ALL_USERS=false` — disables the default open-to-everyone mode - `ZULIP_ALLOWED_USERS=jerome@sysloggh.com` — only Jerome can trigger responses This applies regardless of whether the agent uses `.env` directly (Mumuni) or a systemd drop-in (Tanko). Both mechanisms work — the key is that the env vars reach the Hermes process. ## Key Management **Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in `/etc/environment` on each agent host, never hardcoded in contracts or config.yaml. This contract defines the procedure; it does NOT store plaintext keys. ### Current key aliases (2026-07-03 audit) **Active keys (by agent):** | Agent | Alias | Key (last 8) | Source | Status | |-------|-------|-------------|--------|--------| | Tanko | `tanko-v3` | ...JN2OsJw | Process env | Active (needs restart → v2) | | Tanko | `tanko-v2` | ...1ka8JA | `/etc/environment` | Registered, idle | | Mumuni | `mumuni-v3` | ...OiWU_B8nw | Process + /etc/env | ✅ Aligned | | Koonimo | `koonimo-v3` | ...A_wIOA | Config (hardcoded) | Active | | Koonimo | `koonimo-v2` | ...R7FAobA | `/etc/environment` | Registered, idle | | Koby | `koby-v3` | ...P4DrWw | `/etc/environment` | ✅ Registered | | Koby | `koby-hermes` | ...syslog-koby | Config (deprecated) | Active legacy | | Abiba | `abiba` | ...aLgMw | pi config | ✅ Active | | Kagenz0 | `kagenz0` | ...Uggg | unknown | Last active Jul 1 | | Context7 | `context7-mcp` | ...ZPh6VA | Process env | ✅ Registered | **Unused registrations:** `abiba-v2`, `kagenz0-v2`, `koonimo-v2`, `synthetic-monitor`, `UI Dashboard`, unnamed key > **2026-07-03 alignment**: Tanko/Koonimo configs updated to use `api_key_env: > LITELLM_API_KEY` (was hardcoded `sk-syslog-*`). Base URLs corrected to > `/v1` (was `/litellm/v1`). All active keys now registered in LiteLLM. > Tanko and Koonimo need Hermes gateway restart for config changes to take > effect. ### Key lineage | Agent | Alias | Key (last 8) | Status | Date | |-------|-------|-------------|--------|------| | Tanko | `tanko-v3` | ...JN2OsJw | **active** (process) | 2026-07-03 | | Tanko | `tanko-v2` | ...1ka8JA | idle (in /etc/env) | 2026-07-02 | | Mumuni | `mumuni-v3` | ...OiWU_B8nw | **active** | 2026-07-03 | | Mumuni | `mumuni-v2` | ...HcnuOPtQ | superseded | 2026-07-02 | | Mumuni | `mumuni` | ...62da807e | superseded | 2026-06-30 | | Koonimo | `koonimo-v3` | ...A_wIOA | **active** (hardcoded) | 2026-07-03 | | Koonimo | `koonimo-v2` | ...R7FAobA | idle (in /etc/env) | 2026-07-02 | | Koby | `koby-v3` | ...P4DrWw | **active** (/etc/env) | 2026-07-03 | | Koby | `koby-hermes` | ...syslog-koby | legacy (in config) | 2026-06-30 | > **2026-07-03 incidents**: > - Mumuni: double-B8nw — process had `...POiWU_B8nw`, /etc/environment > had `...VDtHWd-XiWU_B8nw`. Wrong one registered first. > - Context7 MCP: `sk-XY2aUfvy2BIs6kp1ZPh6VA` was never registered — the > original error Kwame reported. > - Tanko: process key `sk-3ZsdWJbbNSo9zSnJN2OsJw` unregistered. Config > used deprecated `sk-syslog-tanko`. Both fixed. > - Koonimo: hardcoded `sk-KFnE3Td6aKEBZxwVA_wIOA` unregistered. Base URL > used old `/litellm/v1` path. Both fixed. > - Koby: `/etc/environment` key `sk-x5BnZUUKvz3r8EdLP4DrWw` unregistered. > Config used deprecated `sk-syslog-koby`. Fixed. ### Verify: all keys working ```bash # Run from Abiba against each agent host for agent in tanko mumuni; do host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123) key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2") code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models) printf " %-10s -> %s\n" "$agent" "$code" done ``` ### Rotate: generate new key, deploy, verify ```bash AGENT=mumuni && ALIAS="${AGENT}-v3" # 1. Generate new key in LiteLLM (master key required) KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \ -H 'Authorization: Bearer sk-litellm-7f960...' \ -H 'Content-Type: application/json' \ -d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))') # 2. Verify new key curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models # 3. Deploy: update /etc/environment on agent host ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment" # 4. Restart agent (systemd or kill + relaunch) ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)" ``` ### Key deployment rules 1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml 2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness 3. **`custom_providers[].api_key` MUST use `api_key_env`** — hardcoded keys shadow env vars and bypass rotation 4. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access 5. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly 6. **After rotating any key:** restart the corresponding agent gateway 7. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login 8. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid) ### Model name enforcement (2026-07-03) **CRITICAL**: Hermes `model: auto` in any config section gets sent to LiteLLM as the literal model name "auto", which LiteLLM rejects with 403 because the registered model is `syslog-auto`, not `auto`. **Must-fix pattern** — any auxiliary section with `model: auto`: ```yaml # ❌ BROKEN — sends model "auto" to LiteLLM (403 Forbidden) auxiliary: approval: model: auto provider: auto skills_hub: model: auto provider: auto # ✅ FIXED — registered model name auxiliary: approval: model: syslog-auto provider: harness api_key_env: LITELLM_API_KEY base_url: http://192.168.68.116/v1 ``` **Detection**: `grep -rn "model: auto" /root/.hermes/config.yaml` on any Hermes host — if any results found in auxiliary sections, fix immediately. ### Key-to-agent mapping (2026-07-03 final sweep, 19:14 EDT) | Agent | CT | Key Alias | Config Status | Key Verified | |-------|-----|-----------|---------------|-------------| | Tanko | 112 | tanko-v2 | ⚠️ Hardcoded (correct key, no api_key_env) | ✅ 200 | | Mumuni | 114 | mumuni-v3 | ✅ Clean (api_key_env) | ✅ 200 | | Koby | 111 | koby-v3 | ✅ Clean (11 fixes applied) | ✅ 200 | | Koonimo | 113 | koonimo-v2 | ✅ Clean (3 fixes applied) | ✅ 200 | | Abiba | 100 | abiba | ✅ pi native | ✅ 200 | | Kagenz0 | 105 | kagenz0-v2 | ❌ DOWN (no hermes, no config) | — | ### Fix summary (2026-07-03 second pass) **Koby (11 fixes):** - 7 auxiliary sections: model:auto → model:syslog-auto (compression, flush_memories, mcp, session_search, skills_hub, vision, web_extract) - custom_providers.harness: deprecated sk-syslog-koby → api_key_env, model:auto → syslog-auto - delegation: model:auto → syslog-auto, deprecated sk-syslog-koby → api_key_env **Mumuni (1 fix):** - delegation: model:auto → syslog-auto + provider:harness + api_key_env **Koonimo (3 fixes, first pass):** - auxiliary.approval: model:auto → syslog-auto, provider:auto → harness - auxiliary.skills_hub: model:auto → syslog-auto, provider:auto → harness - custom_providers.harness: hardcoded koonimo-v3 → api_key_env **Tanko (flagged, not urgent):** - Config has hardcoded tanko-v2 key (correct) instead of api_key_env - DeepSeek fallback key (sk-b7d9...) is intentional — external service - Will need fix before next key rotation ### Mumuni sub-agent profiles (6) Sub-agents in `/root/.hermes/profiles//config.yaml`: `syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`, `syslog-review`, `syslog-writer`. All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`. When the main key is rotated, all 7 configs (main + 6 subs) stay valid — only `/etc/environment` needs updating. ## Infrastructure Stack | Component | Endpoint | Purpose | |---|---|---| | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction | | SearXNG | `http://192.168.68.7:8888` | Privacy-respecting web search | | LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) | | LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) | | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge | | Context7 MCP | `http://localhost:8079/mcp` | Documentation queries | ## API Key Rules - `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred) - `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider - `api_key: sk-...` — Hardcoded key only as fallback when env var not possible - Set `LITELLM_API_KEY` in `/etc/environment` on each host - Sub-agents NEVER get their own key — they share the host agent's key - Restart Hermes after updating `/etc/environment` ### Sub-Agent Profiles (Mumuni pattern) Mumuni has 6 sub-agent profiles in `/root/.hermes/profiles//config.yaml`: ``` profiles/ ├── syslog-code/config.yaml # Code generation ├── syslog-devops/config.yaml # DevOps/infrastructure ├── syslog-email/config.yaml # Email processing ├── syslog-research/config.yaml # Research & analysis ├── syslog-review/config.yaml # Code review └── syslog-writer/config.yaml # Content writing ``` Sub-agent profile rules: 1. **`api_key` must be empty** — `api_key: ''` or omitted entirely 2. **`base_url` must be empty** — inherits from main config's custom_provider 3. **`provider` is `auto` or `harness`** — routes through the shared LiteLLM gateway 4. **`model` is agent-specific** — each sub-agent can have its own default model 5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty 6. **Never hardcode a key** in sub-agent profiles This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`. When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs) work immediately after restart. ## Template — Required Sections ```yaml # ─── Model Selection ─── model: default: # e.g., qwen3.6-27B-code, syslog-auto provider: harness base_url: http://192.168.68.116/v1 api_key_env: LITELLM_API_KEY # Set in /etc/environment fallback_providers: provider: deepseek model: deepseek-chat api_key_env: DEEPSEEK_API_KEY # ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ─── web: backend: firecrawl search_backend: searxng extract_backend: firecrawl firecrawl: base_url: http://192.168.68.7:3002/ # ─── MCP Servers (SHARED INFRA) ─── mcp_servers: context7: connect_timeout: 60 timeout: 300 url: http://localhost:8079/mcp ra-h-os: url: http://192.168.68.65:3100/mcp timeout: 120 connect_timeout: 60 # ─── Compression ─── compression: enabled: true provider: harness model: gemma-4-12b api_key_env: LITELLM_API_KEY threshold: 0.5 target_ratio: 0.25 protect_last_n: 30 hygiene_hard_message_limit: 400 # ─── Auxiliary Tasks ─── auxiliary: vision: provider: harness model: gemma-4-12b api_key_env: LITELLM_API_KEY web_extract: provider: harness model: gemma-4-12b api_key_env: LITELLM_API_KEY session_search: provider: harness model: gemma-4-12b max_concurrency: 3 api_key_env: LITELLM_API_KEY # ─── Custom Provider ─── custom_providers: - name: harness model: base_url: http://192.168.68.116/v1 api_key_env: LITELLM_API_KEY api_mode: chat_completions ``` ## Key Update Procedure When LiteLLM keys are regenerated (e.g., after infrastructure changes): 1. **If SSH available**: `ssh "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-/' /etc/environment"` 2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command 3. **After update**: Restart Hermes on the agent host 4. **Verify**: `curl -H "Authorization: Bearer sk-" http://192.168.68.116/v1/models` ## Configuration Rules ### Rule 1: Shared Infra Is Locked The following MUST be identical across ALL profiles: - `web.backend`, `web.search_backend`, `web.extract_backend` - `web.firecrawl.base_url` - `mcp_servers.ra-h-os.url` - `custom_providers[0].base_url` ### Rule 2: Model Choice Is Free - `model.default` — per agent - `fallback_providers.model` — per agent - `custom_providers[0].model` — per agent ### Rule 3: API Keys via Environment - Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys - Hardcoded keys in config.yaml become stale after key rotation - `/etc/environment` persists across config updates - Restart Hermes after env var updates ### Rule 4: Sub-Agent Profiles Inherit Auth - Sub-agent profiles (`/root/.hermes/profiles/*/config.yaml`) must have: - `api_key: ''` — inherit from main config's custom_provider - `base_url: ''` — inherit from main config - Auxiliary tasks: `api_key: ''`, `provider: harness` - Never hardcode a key in sub-agent profiles - When main config uses `api_key_env`, sub-agents automatically use it - This means key rotation only touches ONE file (`/etc/environment`) ### Rule 5: Main Config Base URL - Use direct IP: `http://192.168.68.116/v1` - NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down - NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly ## Execution 1. **Check current config** — Read the target agent's config.yaml 2. **Compare against template** — Identify missing or divergent sections 3. **Apply shared infra** — Lock web/MCP/compression sections to template values 4. **Apply agent key** — Set from agent_keys table above 5. **Set model choice** — Per agent's workload 6. **Verify** — curl all shared endpoints, test the model with the new key 7. **Report** — What was changed, preserved, custom