- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
7.4 KiB
kind, name, description
| kind | name | description |
|---|---|---|
| template | hermes-config-template | Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online. |
Maintains
- template_version: "2.0.0"
- last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
- agent_keys: map (see Agent Keys section)
- infra_endpoints_verified: array
Agent Keys (LiteLLM — Current 2026-06-30)
Each agent has a unique LiteLLM API key. Keys are stored in LiteLLM's PostgreSQL DB.
The env var LITELLM_API_KEY is set in /etc/environment on each agent host.
Sub-agent profiles inherit auth from the main config — no separate keys needed.
| Agent | Key | Host | SSH | Sub-Agents |
|---|---|---|---|---|
| Tanko | sk-3ZsdWJbbNSo9zSnJN2OsJw |
192.168.68.122 | jerome@.122 | — |
| Mumuni | sk-XY2aUfvy2BIs6kp1ZPh6VA |
192.168.68.123 | root@.123 | 6 profiles ✱ |
| Abiba | sk-kxbPgbnV2Zkn6TKQbAEcEg |
192.168.68.24 | local | — |
| Koby | sk-wb86PvcZXWkjShAIBxrpDg |
? | Zulip | — |
| Koonimo | sk-Prx_vRXYb2VEzDmhPCbNeg |
192.168.68.114 | Zulip | — |
| Kagenz0 | sk-Dh4CDkaHebMLEp8qqq20qA |
? | Zulip | — |
✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
syslog-review, syslog-writer — all at /root/.hermes/profiles/<name>/config.yaml
Infrastructure Stack
| Component | Endpoint | Purpose |
|---|---|---|
| Firecrawl | http://192.168.68.7:3002/ |
Web content extraction |
| SearXNG | http://storepve:8888 |
Privacy-respecting web search |
| LiteLLM | http://192.168.68.116/v1 |
Unified model gateway (via nginx) |
| LiteLLM (NetBird) | https://litellm.sysloggh.net/v1 |
Alternative (may have 502 issues) |
| RA-H OS MCP | http://192.168.68.65:3100/mcp |
Knowledge graph bridge |
| Context7 MCP | http://localhost:8079/mcp |
Documentation queries |
API Key Rules
api_key_env: LITELLM_API_KEY— Use env var for main model auth (preferred)api_key: ''— Sub-agents leave empty to inherit from main config's custom_providerapi_key: sk-...— Hardcoded key only as fallback when env var not possible- Set
LITELLM_API_KEYin/etc/environmenton each host - Sub-agents NEVER get their own key — they share the host agent's key
- Restart Hermes after updating
/etc/environment
Sub-Agent Profiles (Mumuni pattern)
Mumuni has 6 sub-agent profiles in /root/.hermes/profiles/<name>/config.yaml:
profiles/
├── syslog-code/config.yaml # Code generation
├── syslog-devops/config.yaml # DevOps/infrastructure
├── syslog-email/config.yaml # Email processing
├── syslog-research/config.yaml # Research & analysis
├── syslog-review/config.yaml # Code review
└── syslog-writer/config.yaml # Content writing
Sub-agent profile rules:
api_keymust be empty —api_key: ''or omitted entirelybase_urlmust be empty — inherits from main config's custom_providerproviderisautoorharness— routes through the shared LiteLLM gatewaymodelis agent-specific — each sub-agent can have its own default model- Auxiliary tasks (vision, compression, etc.) also leave
api_keyempty - Never hardcode a key in sub-agent profiles
This ensures all 6 sub-agents use the same LiteLLM key set in /etc/environment.
When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs)
work immediately after restart.
Template — Required Sections
# ─── Model Selection ───
model:
default: <agent_model> # e.g., qwen3.6-27B-code, syslog-auto
provider: harness
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # Set in /etc/environment
fallback_providers:
provider: deepseek
model: deepseek-chat
api_key_env: DEEPSEEK_API_KEY
# ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
web:
backend: firecrawl
search_backend: searxng
extract_backend: firecrawl
firecrawl:
base_url: http://192.168.68.7:3002/
# ─── MCP Servers (SHARED INFRA) ───
mcp_servers:
context7:
connect_timeout: 60
timeout: 300
url: http://localhost:8079/mcp
ra-h-os:
url: http://192.168.68.65:3100/mcp
timeout: 120
connect_timeout: 60
# ─── Compression ───
compression:
enabled: true
provider: harness
model: gemma-4-12b
api_key_env: LITELLM_API_KEY
threshold: 0.5
target_ratio: 0.25
protect_last_n: 30
hygiene_hard_message_limit: 400
# ─── Auxiliary Tasks ───
auxiliary:
vision:
provider: harness
model: gemma-4-12b
api_key_env: LITELLM_API_KEY
web_extract:
provider: harness
model: gemma-4-12b
api_key_env: LITELLM_API_KEY
session_search:
provider: harness
model: gemma-4-12b
max_concurrency: 3
api_key_env: LITELLM_API_KEY
# ─── Custom Provider ───
custom_providers:
- name: harness
model: <agent_model>
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_mode: chat_completions
Key Update Procedure
When LiteLLM keys are regenerated (e.g., after infrastructure changes):
- If SSH available:
ssh <host> "sudo sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-<NEW>/' /etc/environment" - If SSH unavailable: Send Zulip DM via abiba-bot with update command
- After update: Restart Hermes on the agent host
- Verify:
curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models
Configuration Rules
Rule 1: Shared Infra Is Locked
The following MUST be identical across ALL profiles:
web.backend,web.search_backend,web.extract_backendweb.firecrawl.base_urlmcp_servers.ra-h-os.urlcustom_providers[0].base_url
Rule 2: Model Choice Is Free
model.default— per agentfallback_providers.model— per agentcustom_providers[0].model— per agent
Rule 3: API Keys via Environment
- Prefer
api_key_env: LITELLM_API_KEYover hardcoded keys - Hardcoded keys in config.yaml become stale after key rotation
/etc/environmentpersists across config updates- Restart Hermes after env var updates
Rule 4: Sub-Agent Profiles Inherit Auth
- Sub-agent profiles (
/root/.hermes/profiles/*/config.yaml) must have:api_key: ''— inherit from main config's custom_providerbase_url: ''— inherit from main config- Auxiliary tasks:
api_key: '',provider: harness
- Never hardcode a key in sub-agent profiles
- When main config uses
api_key_env, sub-agents automatically use it - This means key rotation only touches ONE file (
/etc/environment)
Rule 5: Main Config Base URL
- Use direct IP:
http://192.168.68.116/v1 - NOT the NetBird URL (
litellm.sysloggh.net) — can cause 502 when NetBird is down - NOT the old path (
/litellm/v1) — nginx now routes/v1directly
Execution
- Check current config — Read the target agent's config.yaml
- Compare against template — Identify missing or divergent sections
- Apply shared infra — Lock web/MCP/compression sections to template values
- Apply agent key — Set from agent_keys table above
- Set model choice — Per agent's workload
- Verify — curl all shared endpoints, test the model with the new key
- Report — What was changed, preserved, custom