diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 873ad56..46c4a2a 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -1,135 +1,140 @@ --- -kind: template +kind: pattern name: hermes-config-template description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, - RA-H OS MCP) while keeping model/provider choices flexible per agent role. - Every new agent profile should start from this template. + RA-H OS MCP) while keeping agent-specific API keys and model choices. + Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online. --- ## Maintains -- template_version: string — Current template version (e.g., "1.0.0") -- last_applied: timestamp — When any agent was last configured from this template -- agents_configured: array — Which agents/profiles were created from this template -- infra_endpoints_verified: array — Firecrawl, SearXNG, RA-H OS, LiteLLM endpoints last confirmed working -- known_deviations: array — Profiles that diverge from the template (and why) +- template_version: "2.0.0" +- last_applied: timestamp +- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"] +- agent_keys: managed in LiteLLM DB — current aliases in Key Management section below +- infra_endpoints_verified: array -## Parameters +## Key Management -- agent_name: string — Name of the agent/profile (e.g., "syslog-code", "syslog-devops") -- default_model: string — Agent's primary model (e.g., "qwen3.6-27B-code", "syslog-auto") -- default_provider: string — Inference provider for the primary model (default: "harness") -- fallback_model: string — Fallback model when primary is down (default: "gemma-4-12b") -- fallback_provider: string — Fallback provider (default: "harness") -- auxiliary_model: string — Model for vision/compression/extraction tasks (default: "gemma-4-12b") -- enable_firecrawl_searxng: boolean — Whether to configure web search/extract (default: true) -- enable_ra_h_os_mcp: boolean — Whether to wire up RA-H OS MCP bridge (default: true) -- enable_compression: boolean — Whether to enable context compression (default: true) -- custom_provider_name: string — Custom provider name (default: "harness") -- custom_provider_model: string — Custom provider model (default: same as default_model) -- custom_provider_url: string — LiteLLM base URL (default: "http://litellm.sysloggh.net/litellm/v1") -- extra_auxiliary: array — Additional auxiliary tasks to configure (vision, compression, etc.) -- output_dir: string — Where to write the config (default: "~/.hermes/profiles//config.yaml") +**Source of truth:** LiteLLM PostgreSQL DB. Plaintext keys live ONLY in +`/etc/environment` on each agent host, never hardcoded in contracts or config.yaml. +This contract defines the procedure; it does NOT store plaintext keys. -## Quickstart — How to Run This Contract +### Current key aliases (2026-07-02 rotation) +- `tanko-v2`, `mumuni-v2`, `abiba-v2`, `koby-v2`, `koonimo-v2`, `kagenz0-v2` +- `synthetic-monitor` (daily infra report) -### For the Agent Running This Contract - -Read this section first. It tells you exactly what to do. - -**When to run:** -- Setting up a **new agent profile** for the first time -- An **existing profile** is missing web search, firecrawl, or MCP configs -- User says: `"apply the config template"` or `"fix my infra config"` -- User says: `"configure with "` - -**How to run (via OpenProse CLI):** +### Verify: all keys working ```bash -# Minimal — only agent name required -prose run hermes-config-template agent_name=syslog-code - -# Full control -prose run hermes-config-template \ - agent_name=syslog-devops \ - default_model=claude-sonnet-4 \ - auxiliary_model=gemma-4-12b \ - fallback_model=deepseek-chat \ - fallback_provider=deepseek +# Run from Abiba against each agent host +for agent in tanko mumuni; do + host=192.168.68.$([ "$agent" = tanko ] && echo 122 || echo 123) + key=$(ssh root@$host "grep LITELLM_API_KEY /etc/environment | cut -d= -f2") + code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $key" http://192.168.68.116/v1/models) + printf " %-10s -> %s\n" "$agent" "$code" +done ``` -### How to Follow the Template (Manual) +### Rotate: generate new key, deploy, verify +```bash +AGENT=mumuni && ALIAS="${AGENT}-v2" -If `prose` CLI is not available: +# 1. Generate new key in LiteLLM (master key required) +KEY=$(curl -s -X POST http://192.168.68.116:4001/key/generate \ + -H 'Authorization: Bearer sk-litellm-7f960...' \ + -H 'Content-Type: application/json' \ + -d "{\"key_alias\":\"$ALIAS\",\"models\":[\"gemma-4-12b\",\"qwen3.6-27B-code\",\"ornith-1.0-35b\",\"syslog-auto\"],\"max_budget\":1000}" | python3 -c 'import sys,json; print(json.load(sys.stdin).get("key"))') -1. **Fetch this contract:** `curl -sL https://git.sysloggh.net/SyslogSolution/prose-contracts/raw/branch/master/hermes-config-template.prose.md` -2. **Open the target config.yaml** — the profile you're updating -3. **Compare** each of the 5 required sections below against what exists in the profile -4. **For each section:** - - If it says **SHARED INFRA — DO NOT CHANGE**: Copy the value exactly as shown - - If it says **USER CHOOSES**: Use the agent's assigned model/provider - - If missing entirely: Add the full section -5. **Verify** every endpoint responds (see Verification section at the end) -6. **Restart** the gateway +# 2. Verify new key +curl -s -H "Authorization: Bearer $KEY" http://192.168.68.116/v1/models -### Decision Tree +# 3. Deploy: update /etc/environment on agent host +ssh root@192.168.68.123 "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=$KEY/' /etc/environment" +# 4. Restart agent (systemd or kill + relaunch) +ssh root@192.168.68.123 "sudo systemctl restart hermes 2>/dev/null || (pkill -f hermes_cli.main; cd /root/.hermes && nohup python -m hermes_cli.main gateway run > /dev/null 2>&1 &)" ``` -What kind of profile is this? -├── Main profile (~/.hermes/config.yaml) -│ └── Replace web.*, mcp_servers.*, compression.*, auxiliary.*, custom_providers.* -│ └── model.default + fallback: use current values (USER CHOOSES) -│ -├── Worker profile (~/.hermes/profiles//config.yaml) -│ ├── Does it have web.search_backend? -│ │ ├── NO → Add the full web.* section (copy exactly) -│ │ └── YES → Verify it points to the right IP -│ ├── Does it have mcp_servers.ra-h-os? -│ │ ├── NO → Add mcp_servers.* section (copy exactly) -│ │ └── YES → Verify URL -│ └── model.default: USE the worker's assigned model (don't copy main) -│ -└── Non-Syslog profile (personal/lifestyle agent) - └── Skip — use default Hermes config or Tanko's config instead -``` + +### Key deployment rules +1. **Plaintext keys live ONLY in `/etc/environment`** — never in a contract or config.yaml +2. **All configs use `api_key_env: LITELLM_API_KEY`** — indirection prevents staleness +3. **LiteLLM DB is the authoritative key list** — key aliases with active tokens define who has access +4. **Rotate when:** LiteLLM container redeployed, any key 401s, or quarterly +5. **After rotating any key:** restart the corresponding agent gateway +6. **Agents without SSH** (Koby, Koonimo, Kagenz0): deploy via Zulip DM or manual login +7. **Verification is automated** by litellm-self-heal contract (Rule 8: Agent Keys Invalid) + +### Mumuni sub-agent profiles (6) +Sub-agents in `/root/.hermes/profiles//config.yaml`: +`syslog-code`, `syslog-devops`, `syslog-email`, `syslog-research`, +`syslog-review`, `syslog-writer`. + +All sub-agent profiles inherit auth: `api_key: ''`, `base_url: ''`. +When the main key is rotated, all 7 configs (main + 6 subs) stay valid — +only `/etc/environment` needs updating. ## Infrastructure Stack -This template provisions the following shared infrastructure. Every agent should point to the same endpoints unless explicitly overridden. - | Component | Endpoint | Purpose | |---|---|---| | Firecrawl | `http://192.168.68.7:3002/` | Web content extraction | | SearXNG | `http://storepve:8888` | Privacy-respecting web search | -| LiteLLM | `http://litellm.sysloggh.net/litellm/v1` | Unified model gateway | +| LiteLLM | `http://192.168.68.116/v1` | Unified model gateway (via nginx) | +| LiteLLM (NetBird) | `https://litellm.sysloggh.net/v1` | Alternative (may have 502 issues) | | RA-H OS MCP | `http://192.168.68.65:3100/mcp` | Knowledge graph bridge | | Context7 MCP | `http://localhost:8079/mcp` | Documentation queries | -**API Key Rules:** -- `api_key: sk-_SW...B8nw` — Hardcoded LiteLLM master key for custom_providers and auxiliary tasks -- `api_key_env: DEEPSEEK_API_KEY` — Env-var based key for fallback provider -- Worker profiles should inherit proxy config via `harness` custom_provider, NOT hardcode LiteLLM keys +## API Key Rules -**For workers that need API keys overridden:** Set `api_key: ''` in the model section (inherits from custom_providers), or use `api_key_env` for env-based auth. +- `api_key_env: LITELLM_API_KEY` — Use env var for main model auth (preferred) +- `api_key: ''` — Sub-agents leave empty to inherit from main config's custom_provider +- `api_key: sk-...` — Hardcoded key only as fallback when env var not possible +- Set `LITELLM_API_KEY` in `/etc/environment` on each host +- Sub-agents NEVER get their own key — they share the host agent's key +- Restart Hermes after updating `/etc/environment` -## Template Structure +### Sub-Agent Profiles (Mumuni pattern) -### Required Sections (every agent MUST have) +Mumuni has 6 sub-agent profiles in `/root/.hermes/profiles//config.yaml`: + +``` +profiles/ +├── syslog-code/config.yaml # Code generation +├── syslog-devops/config.yaml # DevOps/infrastructure +├── syslog-email/config.yaml # Email processing +├── syslog-research/config.yaml # Research & analysis +├── syslog-review/config.yaml # Code review +└── syslog-writer/config.yaml # Content writing +``` + +Sub-agent profile rules: +1. **`api_key` must be empty** — `api_key: ''` or omitted entirely +2. **`base_url` must be empty** — inherits from main config's custom_provider +3. **`provider` is `auto` or `harness`** — routes through the shared LiteLLM gateway +4. **`model` is agent-specific** — each sub-agent can have its own default model +5. **Auxiliary tasks** (vision, compression, etc.) also leave `api_key` empty +6. **Never hardcode a key** in sub-agent profiles + +This ensures all 6 sub-agents use the same LiteLLM key set in `/etc/environment`. +When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs) +work immediately after restart. + +## Template — Required Sections ```yaml -# ─── Model Selection (USER CHOOSES) ─── +# ─── Model Selection ─── model: - # !! CHANGE THIS for your agent !! - default: - provider: - # LiteLLM base URL (shared infra) - base_url: http://litellm.sysloggh.net/litellm/v1 + default: # e.g., qwen3.6-27B-code, syslog-auto + provider: harness + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY # Set in /etc/environment fallback_providers: - # !! OPTIONAL: overridable fallback !! - provider: - model: + provider: deepseek + model: deepseek-chat + api_key_env: DEEPSEEK_API_KEY # ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ─── web: @@ -139,7 +144,7 @@ web: firecrawl: base_url: http://192.168.68.7:3002/ -# ─── MCP Servers (SHARED INFRA — DO NOT CHANGE) ─── +# ─── MCP Servers (SHARED INFRA) ─── mcp_servers: context7: connect_timeout: 60 @@ -150,86 +155,50 @@ mcp_servers: timeout: 120 connect_timeout: 60 -# ─── Compression (SHARED — can override model) ─── +# ─── Compression ─── compression: enabled: true provider: harness - model: # default: gemma-4-12b + model: gemma-4-12b + api_key_env: LITELLM_API_KEY threshold: 0.5 target_ratio: 0.25 protect_last_n: 30 hygiene_hard_message_limit: 400 -# ─── Auxiliary Tasks (SHARED INFRA + MODEL) ─── +# ─── Auxiliary Tasks ─── auxiliary: vision: provider: harness - model: + model: gemma-4-12b + api_key_env: LITELLM_API_KEY web_extract: provider: harness - model: + model: gemma-4-12b + api_key_env: LITELLM_API_KEY session_search: provider: harness - model: + model: gemma-4-12b max_concurrency: 3 + api_key_env: LITELLM_API_KEY -# ─── Custom Provider (SHARED INFRA — LiteLLM) ─── +# ─── Custom Provider ─── custom_providers: - - name: - model: - base_url: http://litellm.sysloggh.net/litellm/v1 - api_key: + - name: harness + model: + base_url: http://192.168.68.116/v1 + api_key_env: LITELLM_API_KEY api_mode: chat_completions ``` -### Optional Sections (agent-specific) +## Key Update Procedure -- **Platform configs** (Telegram, Discord, WhatsApp bot tokens) -- **Display/skin** options (compact mode, personality) -- **Plugin lists** (zulip-platform, etc.) -- **Terminal settings** (container image, timeouts) -- **Security/approvals** (command allowlists) -- **Kanban/worker** settings (if this profile is a Kanban worker) -- **Skills auto_load** list -- **Timezone** override +When LiteLLM keys are regenerated (e.g., after infrastructure changes): -## Continuity - -Configuration drift detection is **event-driven**: - -- **On agent onboarding**: Always scaffold from this template -- **On infra change**: If Firecrawl/SearXNG/LiteLLM endpoints change, update template + all profiles -- **On new model**: When a new model is deployed on Litellm, any agent can change their `default` independently -- **Periodic**: Every 2 weeks — check if any profile's infra section has drifted from the template -- **On connection failure**: If web search/extract fails, check if that agent has missing `web.*` config - -## Success Criteria - -The configuration is CONSIDERED GOOD when ALL of these pass: - -### Infra Connectivity -- `web.backend == "firecrawl"` — Firecrawl backend configured -- `web.search_backend == "searxng"` — SearXNG search configured -- `web.extract_backend == "firecrawl"` — Firecrawl extraction configured -- `web.firecrawl.base_url` points to `192.168.68.7:3002` -- `mcp_servers.ra-h-os.url` points to `192.168.68.65:3100/mcp` -- All endpoints respond to `curl` health checks - -### Model Flexibility -- `model.default` is settable per agent (not hardcoded) -- `model.provider` is settable per agent -- `custom_providers[0].model` matches the agent's workload (code vs general) - -### Completeness -- `auxiliary.vision` configured (even if provider=auto) -- `compression.enabled` set (true for most agents) -- Fallback provider configured (deepseek or another harness model) -- `_config_version` correctly set (currently `30`) - -### No Drift -- No duplicate infra endpoints (one canonical Firecrawl URL) -- No stale endpoints (old IPs, dead services) -- All profiles consistent on shared infra +1. **If SSH available**: `ssh "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-/' /etc/environment"` +2. **If SSH unavailable**: Send Zulip DM via abiba-bot with update command +3. **After update**: Restart Hermes on the agent host +4. **Verify**: `curl -H "Authorization: Bearer sk-" http://192.168.68.116/v1/models` ## Configuration Rules @@ -241,67 +210,36 @@ The following MUST be identical across ALL profiles: - `custom_providers[0].base_url` ### Rule 2: Model Choice Is Free -The following are OWNED by each agent and can differ: -- `model.default` — the primary model -- `model.provider` — where it runs -- `fallback_providers.model` — backup model -- `custom_providers[0].model` — what this agent uses LiteLLM for +- `model.default` — per agent +- `fallback_providers.model` — per agent +- `custom_providers[0].model` — per agent -### Rule 3: Workers Inherit, Not Duplicate -Worker profiles (syslog-code, syslog-devops, etc.) generally inherit from the main config. Only override what's DIFFERENT: -- Different default model → override `model.default` -- Different auxiliary model → override auxiliary sections -- No web search needed → still keep the config (tools won't enable without it) +### Rule 3: API Keys via Environment +- Prefer `api_key_env: LITELLM_API_KEY` over hardcoded keys +- Hardcoded keys in config.yaml become stale after key rotation +- `/etc/environment` persists across config updates +- Restart Hermes after env var updates -### Rule 4: API Key Placement -- **Custom provider keys** (`custom_providers[0].api_key`): Must be hardcoded (the LiteLLM master key or a virtual key) -- **Model section keys** (`model.api_key`): Prefer empty. Workers inherit from custom_providers. -- **Environment keys** (`api_key_env`): For fallback providers that need separate auth (DeepSeek, OpenRouter) -- **Hardcoded keys in profile**: If a worker's `model.api_key` is set, it overrides custom_providers. Only use when the worker needs a DIFFERENT provider than Litellm. +### Rule 4: Sub-Agent Profiles Inherit Auth +- Sub-agent profiles (`/root/.hermes/profiles/*/config.yaml`) must have: + - `api_key: ''` — inherit from main config's custom_provider + - `base_url: ''` — inherit from main config + - Auxiliary tasks: `api_key: ''`, `provider: harness` +- Never hardcode a key in sub-agent profiles +- When main config uses `api_key_env`, sub-agents automatically use it +- This means key rotation only touches ONE file (`/etc/environment`) -### Rule 5: Verify After Every Config Change -After changing a profile's config: -1. `curl` the model endpoint to confirm auth works -2. `curl` the web stack endpoints (Firecrawl, SearXNG) -3. Check that `hermes gateway --restart` reloads cleanly +### Rule 5: Main Config Base URL +- Use direct IP: `http://192.168.68.116/v1` +- NOT the NetBird URL (`litellm.sysloggh.net`) — can cause 502 when NetBird is down +- NOT the old path (`/litellm/v1`) — nginx now routes `/v1` directly ## Execution 1. **Check current config** — Read the target agent's config.yaml 2. **Compare against template** — Identify missing or divergent sections -3. **Apply shared infra** — Lock the web/MCP/compression sections to template values -4. **Apply model choice** — Set default/fallback/auxiliary models per agent's workload -5. **Preserve agent-specific** — Keep platform configs, plugins, terminal settings, skills -6. **Remove stale** — Drop any old infra endpoints that don't match the template -7. **Verify** — curl all shared endpoints, test the model -8. **Report** — What was changed, what was preserved, what's still custom - -## Example Output - -``` -## Config Template Applied: syslog-code ✅ - -### Shared Infrastructure (Locked) -| Section | Before | After | -|---|---|---| -| web.backend | firecrawl | firecrawl ✅ | -| web.search_backend | NOT SET | searxng ✅ | -| web.firecrawl.base_url | NOT SET | http://192.168.68.7:3002/ ✅ | -| mcp_servers.ra-h-os | present | present ✅ | - -### Agent Model Choices (Preserved) -| Setting | Value | -|---|---| -| model.default | qwen3.6-27B-code (code agent) | -| fallback | gemma-4-12b (harness) | -| auxiliary | gemma-4-12b (harness) | -| compression | gemma-4-12b (harness) | - -### Endpoint Verification -| Endpoint | Status | -|---|---| -| Firecrawl :3002 | ✅ 200 OK | -| SearXNG :8888 | ✅ 200 OK | -| LiteLLM | ✅ 200 OK | -| RA-H OS MCP | ✅ Connected | -``` +3. **Apply shared infra** — Lock web/MCP/compression sections to template values +4. **Apply agent key** — Set from agent_keys table above +5. **Set model choice** — Per agent's workload +6. **Verify** — curl all shared endpoints, test the model with the new key +7. **Report** — What was changed, preserved, custom