prose-contracts/hermes-config-template.prose.md

kind, name, description

kind	name	description
template	hermes-config-template	Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation) to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident. Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the 2026-07-16 Mumuni root-cause investigation (WAL #1300). UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).

Maintains

• template_version: "2.1.0"
• last_applied: timestamp
• agents_configured: ["mumuni", "abiba", "koby", "koonimo", "kagenz0"] # tanko removed 2026-08-27 (now on DSH/DeepSeek Harness)
• agent_keys: map (see Agent Keys section)
• infra_endpoints_verified: array

Reachability Detection

Before auditing the config template, verify the host is reachable and can be checked. Use the shared reachability helper from the clone root:

# Run on each host to check reachability (Tanko, Mumuni, Koonimo, Koby)
scripts/hermes-reachability-check.sh <host> "base_url:" "/root/.hermes/config.yaml"
# Example: scripts/hermes-reachability-check.sh 192.168.68.122 "base_url:" "/root/.hermes/config.yaml"

# Expected outcomes:
# - UNREACHABLE: SSH connection failed (host is down)
# - VIOLATION: SSH succeeded and found matches (report the finding)
# - COMPLIANT: SSH succeeded and found no matches (no base_url in config)
#
# NOTE: The bug this replaces was deriving reachability from the remote grep's exit code.
# The correct pattern: remote side always succeeds (grep ...; true), so ssh status = connection only.

Agent Keys (LiteLLM — Current 2026-07-11)

Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM PostgreSQL DB via POST /key/generate on CT 116. Keys are stored in the DB. The env var LITELLM_API_KEY is injected at runtime via infisical run -- wrapper (project=agents, env=production). /etc/environment and ~/.hermes/.env are NO LONGER used for agent keys — stripped and tagged # [INFISICAL] post-migration. Sub-agent profiles inherit auth from the main config — no separate keys needed.

Agent	Key Alias	Host	SSH	Sub-Agents
Abiba	abiba-pi	192.168.68.24	local	—
Koby	koby	CT 111 (tdunna)	Zulip	—
Koonimo	koonimo	CT 113 (baggy)	SSH root	—
Kagenz0	kagenz0-*	?	Zulip	—

CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).

✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research, syslog-review, syslog-writer — all at /root/.hermes/profiles/<name>/config.yaml

Infrastructure Stack

Component	Endpoint	Purpose
Firecrawl	http://192.168.68.7:3002/	Web content extraction
SearXNG	http://192.168.68.7:8888	Privacy-respecting web search
LiteLLM	http://192.168.68.116/v1	Unified model gateway (via nginx)
LiteLLM (NetBird)	https://litellm.sysloggh.net/v1	Alternative (may have 502 issues)
RA-H OS MCP	http://192.168.68.65:3100/mcp	Knowledge graph bridge
Context7 MCP	http://localhost:8079/mcp	Documentation queries

API Key Rules

• api_key_env: LITELLM_API_KEY — Use env var for main model auth (preferred) Key is injected at runtime via infisical run -- wrapper — never in /etc/environment
• api_key: '' — Sub-agents leave empty to inherit from main config's custom_provider
• api_key: sk-... — Hardcoded key only as fallback when env var not possible
• Store LITELLM_API_KEY in Infisical vault (project=agents, env=production)
• Sub-agents NEVER get their own key — they share the host agent's key
• Restart Hermes gateway after updating vault secret (key auto-injected via wrapper)

Sub-Agent Profiles (Mumuni pattern)

Mumuni has 6 sub-agent profiles in /root/.hermes/profiles/<name>/config.yaml:

profiles/
├── syslog-code/config.yaml      # Code generation
├── syslog-devops/config.yaml     # DevOps/infrastructure
├── syslog-email/config.yaml      # Email processing
├── syslog-research/config.yaml   # Research & analysis
├── syslog-review/config.yaml     # Code review
└── syslog-writer/config.yaml     # Content writing

Sub-agent profile rules:

1. api_key must be empty — api_key: '' or omitted entirely
2. base_url must be empty — inherits from main config's custom_provider
3. provider is auto or harness — routes through the shared LiteLLM gateway
4. model is agent-specific — each sub-agent can have its own default model
5. Auxiliary tasks (vision, compression, etc.) also leave api_key empty
6. Never hardcode a key in sub-agent profiles

This ensures all 6 sub-agents use the same LiteLLM key injected via infisical run -- wrapper. When the key is rotated, only the env var needs updating — all 7 configs (main + 6 subs) work immediately after restart.

Template — Required Sections

# ─── Model Selection ───
model:
  default: <agent_model>          # e.g., strix-moe, gpu-dense, syslog-auto
  provider: harness
  base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK
  api_key_env: LITELLM_API_KEY    # Injected via infisical run -- wrapper
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
  context_length: 131072          # Conservative floor for syslog-auto (NVIDIA hosts 128K; Strix Halo 256K).
                                  # ⚠️ MANDATORY: Hermes probes unknown models from 256K
                                  # and falls back to 256K when /v1/models lacks a context
                                  # field (llama-server does). Without this override, agents
                                  # silently run syslog-auto at 256K (verified 2026-08-09).
                                  # Set 65536 if pinning a single model directly (tight VRAM).

fallback_providers:
  provider: deepseek
  model: deepseek-chat
  api_key_env: DEEPSEEK_API_KEY

# ─── Web Stack (SHARED INFRA — DO NOT CHANGE) ───
web:
  backend: firecrawl
  search_backend: searxng
  extract_backend: firecrawl
  firecrawl:
    base_url: http://192.168.68.7:3002/

# ─── MCP Servers (SHARED INFRA) ───
mcp_servers:
  context7:
    connect_timeout: 60
    timeout: 300
    url: http://localhost:8079/mcp
  ra-h-os:
    url: http://192.168.68.65:3100/mcp
    timeout: 120
    connect_timeout: 60
  litellm:
    url: https://litellm.sysloggh.net/mcp
    headers:
      x-litellm-api-key: "Bearer <AGENT_KEY>"   # Rule 15: must be a REAL key (sk-...), not an env-var name
    # Note: MCP endpoint requires Accept: application/json, text/event-stream header
    # This is handled by the MCP client library; don't add to config

# ─── Compression ───
compression:
  enabled: true
  model: syslog-auto               # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
  provider: harness
  max_context_window: 131072      # MUST stay at the syslog-auto pool floor: NVIDIA hosts are 128K, Strix Halo 256K (2026-09-12).
  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
  target_ratio: 0.30
  protect_last_n: 40
  hygiene_hard_message_limit: 350
  protect_first_n: 3
  abort_on_summary_failure: false

# ─── Auxiliary Tasks (CONSISTENCY RULE) ───
# All auxiliary services MUST use identical model, base_url, and api_key_env:
#   model: gpu-vision                # stable alias (NOT a raw model name)
#   base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
#   api_key_env: LITELLM_API_KEY
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
# gpu-vision = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
# NEVER use retired model names (qwen3.6-27B-code, qwen3.6-35B-udq4; gemma-4-12b is retired
# and no longer resolves) in agent configs — use the stable aliases so model swaps don't break agents.
auxiliary:
  vision:
    provider: harness
    model: gpu-vision           # stable alias for RTX 5070
    base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
    api_key_env: LITELLM_API_KEY
    timeout: 60
    download_timeout: 30
  web_extract:
    provider: harness
    model: gpu-vision           # stable alias for RTX 5070
    base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
    api_key_env: LITELLM_API_KEY
    timeout: 30
  compression:
    provider: harness
    model: syslog-auto              # MUST match compression.model above. Stable alias for Strix Halo (weighted pool).
    base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK
    api_key_env: LITELLM_API_KEY
    timeout: 300                 # gpu-fleet: 300s for large-history summarization (was 60)

# ─── Delegation / Heavy Aux (use gpu-dense = RTX 3090) ───
# delegation.model and x_search.model use gpu-dense (NOT retired raw name).

delegation:
  model: gpu-dense              # stable alias for RTX 3090
  provider: harness
  base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
  api_key_env: LITELLM_API_KEY

# ─── Custom Provider ───
custom_providers:
  - name: harness
    model: syslog-auto            # weighted pool (default)
    base_url: http://192.168.68.116/litellm/v1   # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions

Key Update Procedure

When LiteLLM keys are regenerated (e.g., after infrastructure changes):

1. If SSH available: Update Infisical vault: infisical secrets set LITELLM_API_KEY=sk-<NEW> --project=agents --env=production, then ssh <host> "systemctl restart hermes-gateway"
2. If SSH unavailable: Send Zulip DM via abiba-bot with update command
3. After update: Restart Hermes on the agent host
4. Verify: curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models

MCP Server Configuration

MCP server entries in mcp_servers: must follow the format shown in the Template section.

Header requirements (Rule 15):

• Use headers: field with a x-litellm-api-key entry
• The value must be "Bearer <REAL_KEY>" where <REAL_KEY> is a literal LiteLLM virtual key
• Do NOT use env-var references like $LITELLM_API_KEY — they resolve to empty strings in the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident)

Key source:

• Keys are stored in the Infisical vault (project=agents, env=production)
• For template-based config generation: substitute the agent's key from the agent_keys table
• For manual config updates: retrieve the key from the vault and insert the literal value

Verification (2026-08-07):

• Tested MCP initialize handshake against litellm.sysloggh.net/mcp with agent virtual key
• Confirmed: 200 response with serverInfo.name: "litellm-mcp-server"
• Confirmed: tools/list returns 200 (MCP endpoint accessible with virtual keys)
• Note: Per-key MCP grants are now supported (verified 2026-09-18), resolving the earlier contradiction with infrastructure-update.prose.md (which now reflects the update)
• Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models)

Key rotation note:

• MCP headers use literal keys (not env-vars), so they do NOT auto-rotate with the vault
• After key rotation, MCP server headers must be regenerated with the new key value
• This is a manual step: update the x-litellm-api-key header in each config file
• TODO: Consider adding MCP header regeneration to the Key Update Procedure or a generation hook

NetBird dependency:

• litellm.sysloggh.net is a NetBird endpoint (see Rule 5 and Infrastructure Stack table)
• NetBird outages cause 502 errors on MCP requests, not auth failures
• Diagnose: if MCP requests fail with 502, check NetBird status before investigating keys

Violation Classification

When reporting findings, separate POLICY observations from FAULT findings:

POLICY (observation only, not a fault)

• Agent uses a non-internal-harness provider (e.g., direct DeepSeek, Tencent, OpenRouter)
• Config text has a field that looks unusual but the agent's calls are succeeding
• Example: "POLICY: Koonimo uses deepseek directly; calls succeeding in last hour"

FAULT (requires request-level evidence)

• Agent's calls are failing with auth errors (401/403 in logs)
• Agent's config has no valid API key AND calls are failing
• Example: "FAULT: Koby's LiteLLM key expired; 401 observed at 2026-09-14 11:42:00"

Rules

1. Do NOT infer the runtime's credential resolution from config text alone.
2. Require request-level evidence before calling something a FAULT: an observed auth failure in the agent's log, or the absence of successful calls in the window.
3. If calls are succeeding, the correct output is "POLICY: uses directly; calls succeeding" - not a violation.
4. State what you OBSERVED, not what the field implies.

Configuration Rules

Rule 1: Shared Infra Is Locked

The following MUST be identical across ALL profiles:

• web.backend, web.search_backend, web.extract_backend
• web.firecrawl.base_url
• mcp_servers.ra-h-os.url
• custom_providers[0].base_url

Rule 2: Model Choice Is Free

• model.default — per agent
• fallback_providers.model — per agent
• custom_providers[0].model — per agent

Rule 3: API Keys via Environment

• Prefer api_key_env: LITELLM_API_KEY over hardcoded keys
• Hardcoded keys in config.yaml become stale after key rotation
• Infisical vault secrets persist across config updates / reinstalls
• NEW (July 2026): Always keep a local .env fallback. Infisical service tokens can expire/404 (tanko incident: token not found, gateway ran without key for hours). The .env file should have the key uncommented as a fallback:

  LITELLM_API_KEY=sk-...
  # [INFISICAL] Also sourced from vault.sysloggh.net
  

• Restart Hermes after env var updates

Rule 4: Sub-Agent Profiles Inherit Auth

• Sub-agent profiles (/root/.hermes/profiles/*/config.yaml) must have:
  • api_key: '' — inherit from main config's custom_provider
  • base_url: '' — inherit from main config
  • Auxiliary tasks: api_key: '', provider: harness
• Never hardcode a key in sub-agent profiles
• When main config uses api_key_env, sub-agents automatically use it
• This means key rotation only touches ONE vault secret (LITELLM_API_KEY)

Rule 5: Main Config Base URL (UPDATED 2026-08-09)

|- Use the authenticated LiteLLM path: http://192.168.68.116/litellm/v1 (canonical, captain-approved migration) |- Legacy http://192.68.68.116/v1 also works — nginx fronts BOTH paths with key auth (verified 2026-08-09: 401 without key, 200 with key, on both /v1 and /litellm/v1) |- Both locations have proxy_read_timeout 600s (verified in harness-nginx nginx.conf) — the old "60s timeout on /litellm/" claim was stale and is retracted |- NOT the NetBird URL (litellm.sysloggh.net) — can cause 502 when NetBird is down

Rule 6: max_tokens Is Required (Thermal Safety)

• Every Hermes config MUST set model.max_tokens: 4096 — this is non-negotiable
• Prevents unbounded generation that caused the July 2 Strix Halo GPU thermal incident
• The value flows through: config → agent.max_tokens → transport build_kwargs → API max_tokens
• Even though the server now has -n 8192 hard cap (set by Abiba), the client cap is the first line of defense
• Apply to BOTH main config AND all sub-agent profiles
• For agents needing longer outputs: raise to 8192, but never omit

Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)

• Vision and web_extract use gpu-vision (RTX 5070 — 12GB, vision-optimized)
• Compression uses syslog-auto — the Strix Halo weighted pool (64GB, 256K ctx, compression-optimized); do NOT pin compression.model to strix-moe (audit Rule 7 rejects it)
• ornith-1.0-35b is NOT a valid compression model name — LiteLLM does not serve it (do not restate the served model list here — CT 116 /opt/inference-harness/litellm_config.yaml is the single source of truth for models, aliases, weights and fallbacks). Old configs with ornith-1.0-35b cause 403/model-not-found on compression calls.
• OPERATIONAL DECISION (2026-07-23): Use syslog-auto for compression across all agents. The syslog-auto alias routes to the Strix Halo, but uses the weighted pool instead of pinning to strix-moe directly. This prevents sustained Strix Halo thermal load because the pool can fall back to other GPUs if Strix gets hot. Both compression.model and auxiliary.compression.model MUST be syslog-auto.
• All auxiliary services MUST use identical routing:
  • base_url: http://192.168.68.116/litellm/v1 (Rule 5, 2026-08-09: canonical authenticated; /v1 also OK)
  • api_key_env: LITELLM_API_KEY
• Do NOT use syslog-auto for vision/web_extract — it routes unpredictably; compression is the deliberate exception (see the OPERATIONAL DECISION above)
• Compression on Strix Halo: The strix-moe alias routes to Strix Halo (64GB UMA, 256K context) — the designated compression GPU. This frees the RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
• The compression: block's model MUST match auxiliary: compression: model
• The compression: max_context_window: 131072 MUST stay at the syslog-auto pool floor (NVIDIA hosts 128K; Strix Halo 256K)

Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)

• RTX 3090 (24GB, 128K ctx, gpu-dense): Heavy reasoning, code gen, long conversations
• RTX 5070 (12GB, 128K ctx, gpu-vision): Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
• Strix Halo (64GB, 256K ctx, syslog-auto): Context compression, summarization, long docs
• Agent profiles MUST route auxiliary tasks to the correct GPU:
  • auxiliary.vision.model: gpu-vision (RTX 5070)
  • auxiliary.web_extract.model: gpu-vision (RTX 5070)
  • auxiliary.compression.model: syslog-auto (Strix Halo)
• Default model (model.default) and custom_provider remain syslog-auto for auto-routing
• For 128K context window: threshold: 0.65 (fires at ~85K tokens)
• Do NOT use threshold: 0.25 — this fires at 65K, causing premature context loss
• Do NOT use threshold: 0.80 — this delays until ~105K, leaving only 23K margin
• max_context_window: 131072 MUST stay at the pool floor (NVIDIA hosts 128K; Strix Halo 256K)
• See devops-hermes-compression skill for full reference

Rule 9: Compression Threshold for 128K Models

• For 128K context window: threshold: 0.65 (fires at ~85K tokens)
• Do NOT use threshold: 0.25 — this fires at 65K, causing premature context loss
• Do NOT use threshold: 0.80 — this delays until ~105K, leaving only 23K margin
• max_context_window: 131072 MUST stay at the pool floor (NVIDIA hosts 128K; Strix Halo 256K)
• See devops-hermes-compression skill for full reference

Rule 10: Default Model Must Be syslog-auto (All Agents)

• Koby Exception: Per captain ruling 2026-08-11, Koby is a DeepSeek-primary external agent; its primary model remains deepseek-v4-flash (via api.deepseek.com to preserve DeepSeek-specific reasoning, while other sections follow Rule 10.
• Hermes agents: model.default: syslog-auto, custom_providers[0].model: syslog-auto
• pi agents: defaultModel: syslog-auto in settings.json, first model in models.json
• syslog-auto is the LiteLLM routing model — it load-balances across the live pool (see CT 116 /opt/inference-harness/litellm_config.yaml for the current members and weights). Using it protects against:
  • Model name typos that cause 403 errors and silent worker failures
  • Single GPU downtime (routing falls back automatically)
  • Key/model authorization mismatches
• Exception: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models for specialized tasks, but MUST validate those models exist in the key's authorized list

Rule 11: Validate Model IDs Before Deployment (pi Agents)

• After configuring a pi agent's models.json, verify every model ID:

  curl -s http://192.168.68.116:4000/v1/models \
    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
  

• All model IDs in models.json MUST appear in the LiteLLM response
• The agent's API key may have a SUBSET of the full model catalog — check per-key
• A non-existent model ID causes 403 errors that silently break the pi RPC worker (no agent_end emitted, worker stays "busy", Zulip messages pile up unprocessed)

Rule 12: Context-Issue Diagnostic Checklist (ADDED 2026-07-16, WAL #1300)

When an agent shows "context issues" (premature compression, 401s, 504s, DeepSeek fallback), verify ALL FOUR of these against the live config. They are the only root causes found in production:

1. max_context_window correct? — BOTH compression.max_context_window AND context.max_context_window MUST be 131072 (the syslog-auto pool floor: NVIDIA hosts are 128K; Strix Halo is 256K). A 262144 client window can route to a 128K NVIDIA host and fail, so it must NOT be used. ~83K instead of ~170K. Check: grep -n max_context_window ~/.hermes/config.yaml
2. base_url uses authenticated path? — custom_providers[0].base_url, delegation.base_url, and ALL auxiliary.*.base_url MUST be http://192.168.68.116/litellm/v1 (Rule 5, canonical) or http://192.168.68.116/v1 (legacy, still authenticated via nginx). BOTH verified 200 with key + 600s proxy_read_timeout on 2026-08-09. Never bare :4000 direct. Check: grep -nE 'base_url: http://192.168.68.116(:4000)?/v1' ~/.hermes/config.yaml — the ONLY paths allowed are /v1 or /litellm/v1 (both via nginx :80). :4000 or missing litellm/v1/v1 prefix = violation.
3. LITELLM_API_KEY valid? — The key must be a real LiteLLM key (sk- + 64 hex, 67 chars). Malformed values (e.g. sk-_SWAl_Vu_…, 47 chars) return 401 → DeepSeek fallback. Verify: curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LITELLM_API_KEY" http://192.168.68.116/v1/models (must be 200)
4. custom_providers aligned? — model: syslog-auto (Rule 10), api_mode: chat_completions (NOT responses). A wrong api_mode causes silent request failures.

One-line agent health check (run on the agent host):

# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models

Rule 13: API Key Injection — Two Patterns (UPDATED 2026-07-16, WAL #1300)

Rule 14: Hermes Context Detection Uses max_model_tokens, NOT max_input_tokens

CRITICAL: Hermes context detection reads max_model_tokens (128K), NOT max_input_tokens (64K cap).

• Abiba and Hermes agents: max_model_tokens: 131072 (128K) — unlimited context
• Crewmates (ops, tune, verify, auth-keys, build): max_input_tokens: 64000 (64K) — capped
• If you see max_input_tokens: 64000 in an Abiba/Hermes config, that's a mistake
• Using max_input_tokens for Hermes agents causes premature context loss
• Check: grep -n 'max_model_tokens\|max_input_tokens' ~/.hermes/config.yaml
• Expected output: max_model_tokens: 131072 (not max_input_tokens)

Agents inject LITELLM_API_KEY via ONE of two mechanisms. Both are valid; the contract requirement is that the key is a valid LiteLLM virtual key (HTTP 200 on /v1/models).

Pattern A — systemd drop-in (Koby, Koonimo, and any agent without infisical wrapper): A systemd drop-in /etc/systemd/system/hermes-gateway.service.d/litellm-key.conf sets the key:

[Service]
Environment="LITELLM_API_KEY=sk-<VALID_KEY>"

The service unit hermes-gateway.service runs python -m hermes_cli.main gateway run --replace directly (no infisical). Apply with systemctl daemon-reload && systemctl restart hermes-gateway.

• Koonimo (CT113/.114): service = hermes-gateway.service, drop-in has the key.
• Koby (CT111/.129): service = hermes-gateway.service (created 2026-07-16), ExecStart uses --replace to win the lock against stray hermes gateway restart invocations. Key also in /etc/environment.

Pattern B — infisical-gateway.sh wrapper (Mumuni): The wrapper sources ~/.hermes/.env then exports LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY". See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.

⚠️ Vault empty-key guard: If the vault stores the secret as an empty string, the wrapper will inject an empty key and the gateway will silently get 401 errors on all LiteLLM requests (triggering silent DeepSeek fallback). The .env fallback is present but the vault takes precedence when the secret key exists (even if empty).

Fix: The wrapper MUST validate the key length after injection. If LITELLM_API_KEY is empty or shorter than 20 chars, log a warning and either fail with a clear error message or fall back to the .env value before starting the gateway.

Verification (all agents):

# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models  # must be 200

• /etc/environment is NO LONGER the canonical key source (stale values there caused 401s).
• Do NOT leave a hardcoded stale key in /etc/environment — it shadows the drop-in/wrapper.

Rule 14: Provider Name Must Match custom_providers Name (ADDED 2026-07-19, WAL #1471)

• model.provider MUST be harness (the custom_providers[0].name), NOT the literal string custom
• When provider: custom, Hermes' _get_named_custom_provider("custom") returns None (no provider is named "custom" — it is named "harness"), causing a fall-through to the generic resolution path (source: env/config) at runtime_provider.py:1156
• The generic path builds api_key_candidates from model.api_key (empty), host-gated OLLAMA/OPENAI/OPENROUTER keys, and _host_derived_api_key (returns "" for IP addresses)
• The generic path does NOT resolve model.api_key_env or custom_providers.key_env — LITELLM_API_KEY is never read, producing api_key = "no-key-required" → HTTP 401
• The named custom provider path (source: custom_provider:harness) DOES read key_env — but only triggers when provider matches the custom_providers[0].name
• All sections MUST use provider: harness: model, compression, auxiliary.vision, auxiliary.web_extract, auxiliary.compression, delegation
• Only fallback_providers uses a different provider (deepseek) for true fallback diversity
• Diagnostic: If you see source: env/config in a request dump or log, the provider name is wrong. It should be source: custom_provider:harness.
• Audit script: Run python3 /root/prose-contracts/audit-hermes-config.py <config.yaml> before and after any config change to catch this and all other rule violations.

Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07)

Endpoint validation:

• ra-h-os must point to http://192.168.68.65:3100/mcp
• litellm must point to https://litellm.sysloggh.net/mcp
• Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had ra-h-os pointing to litellm's endpoint)

Header validation:

• Every MCP entry with authentication must carry a headers: field
• The header value must be a REAL key (e.g., Bearer sk-abc123...), NOT an env-var name
• Env-var names like LITELLM_API_KEY do NOT resolve in static MCP configs and cause "Malformed API Key" floods (401 errors in agent gateway logs)
• Verify: header value should match a valid LiteLLM key (test with curl against /v1/models)
• Verify MCP access: test the MCP initialize handshake against the MCP endpoint (not just /v1/models)

  curl -s -X POST -H "x-litellm-api-key: Bearer <KEY>" -H "Accept: application/json, text/event-stream" \
    https://litellm.sysloggh.net/mcp -d '{"jsonrpc":"2.0","id":1,"method":"initialize",...}' \
    | jq '.data.result.serverInfo'  # should show serverInfo.name and version
  

See: § MCP Server Configuration for implementation details and key source.

Execution

1. Check current config — Read the target agent's config.yaml
2. Compare against template — Identify missing or divergent sections
3. Apply shared infra — Lock web/MCP/compression sections to template values
4. Apply agent key — Set from agent_keys table above
5. Set model choice — Per agent's workload
6. Verify — curl all shared endpoints, test the model with the new key
7. Report — What was changed, preserved, custom