From d2bca5405a9ed2c836edf8d251c75763f2c170cf Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Thu, 17 Sep 2026 11:30:44 +0000 Subject: [PATCH] feat: add litellm MCP server entry and enhance Rule 15 validation - Added litellm MCP server entry to mcp_servers section with correct URL (https://litellm.sysloggh.net/mcp) and header format - Updated Rule 15 to be more specific about endpoint validation and header requirements (REAL keys, not env-var references) - Added MCP Server Configuration section with implementation details - Documented the 2026-08-07 Tanko incident where ra-h-os was pointing to litellm endpoint with env header causing 401 floods - Updated frontmatter to reflect the changes Fixes: #keyless-mcp-incident-20260807 Refs: Rule 15 (MCP Endpoint and Header Validation) --- hermes-config-template.prose.md | 46 ++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 0a1a6ef..c40d298 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -5,7 +5,8 @@ description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. - UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident. + UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation) + to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident. Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the 2026-07-16 Mumuni root-cause investigation (WAL #1300). UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13). @@ -145,6 +146,10 @@ mcp_servers: url: http://192.168.68.65:3100/mcp timeout: 120 connect_timeout: 60 + litellm: + url: https://litellm.sysloggh.net/mcp + headers: + x-litellm-api-key: "Bearer " # Rule 15: must be a REAL key (sk-...), not an env-var name # ─── Compression ─── compression: @@ -217,6 +222,21 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes): 3. **After update**: Restart Hermes on the agent host 4. **Verify**: `curl -H "Authorization: Bearer sk-" http://192.168.68.116/v1/models` +## MCP Server Configuration + +MCP server entries in `mcp_servers:` must follow the format shown in the Template section. + +**Header requirements (Rule 15):** +- Use `headers:` field with a `x-litellm-api-key` entry +- The value must be `"Bearer "` where `` is a literal LiteLLM virtual key +- Do NOT use env-var references like `$LITELLM_API_KEY` — they resolve to empty strings in + the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident) + +**Key source:** +- Keys are stored in the Infisical vault (project=agents, env=production) +- For template-based config generation: substitute the agent's key from the agent_keys table +- For manual config updates: retrieve the key from the vault and insert the literal value + ## Violation Classification When reporting findings, separate POLICY observations from FAULT findings: @@ -455,14 +475,22 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192. - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py ` before and after any config change to catch this and all other rule violations. -### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07) -- Every MCP server entry must point at the correct endpoint: - - ra-h-os = http://192.168.68.65:3100/mcp - - litellm = https://litellm.sysloggh.net/mcp -- MCP entries must carry a REAL key value in the header. - - Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP - endpoints and result in "Malformed API Key" floods. - - Ensure the header value is the actual key (e.g., `sk-...`). +### Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07) + +**Endpoint validation:** +- ra-h-os must point to `http://192.168.68.65:3100/mcp` +- litellm must point to `https://litellm.sysloggh.net/mcp` +- Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had + ra-h-os pointing to litellm's endpoint) + +**Header validation:** +- Every MCP entry with authentication must carry a `headers:` field +- The header value must be a REAL key (e.g., `Bearer sk-abc123...`), NOT an env-var name +- Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause + "Malformed API Key" floods (401 errors in agent gateway logs) +- Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models) + +**See:** § MCP Server Configuration for implementation details and key source. ## Execution