From d2bca5405a9ed2c836edf8d251c75763f2c170cf Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Thu, 17 Sep 2026 11:30:44 +0000 Subject: [PATCH 1/4] feat: add litellm MCP server entry and enhance Rule 15 validation - Added litellm MCP server entry to mcp_servers section with correct URL (https://litellm.sysloggh.net/mcp) and header format - Updated Rule 15 to be more specific about endpoint validation and header requirements (REAL keys, not env-var references) - Added MCP Server Configuration section with implementation details - Documented the 2026-08-07 Tanko incident where ra-h-os was pointing to litellm endpoint with env header causing 401 floods - Updated frontmatter to reflect the changes Fixes: #keyless-mcp-incident-20260807 Refs: Rule 15 (MCP Endpoint and Header Validation) --- hermes-config-template.prose.md | 46 ++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 0a1a6ef..c40d298 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -5,7 +5,8 @@ description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. - UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident. + UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation) + to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident. Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the 2026-07-16 Mumuni root-cause investigation (WAL #1300). UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13). @@ -145,6 +146,10 @@ mcp_servers: url: http://192.168.68.65:3100/mcp timeout: 120 connect_timeout: 60 + litellm: + url: https://litellm.sysloggh.net/mcp + headers: + x-litellm-api-key: "Bearer " # Rule 15: must be a REAL key (sk-...), not an env-var name # ─── Compression ─── compression: @@ -217,6 +222,21 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes): 3. **After update**: Restart Hermes on the agent host 4. **Verify**: `curl -H "Authorization: Bearer sk-" http://192.168.68.116/v1/models` +## MCP Server Configuration + +MCP server entries in `mcp_servers:` must follow the format shown in the Template section. + +**Header requirements (Rule 15):** +- Use `headers:` field with a `x-litellm-api-key` entry +- The value must be `"Bearer "` where `` is a literal LiteLLM virtual key +- Do NOT use env-var references like `$LITELLM_API_KEY` — they resolve to empty strings in + the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident) + +**Key source:** +- Keys are stored in the Infisical vault (project=agents, env=production) +- For template-based config generation: substitute the agent's key from the agent_keys table +- For manual config updates: retrieve the key from the vault and insert the literal value + ## Violation Classification When reporting findings, separate POLICY observations from FAULT findings: @@ -455,14 +475,22 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192. - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py ` before and after any config change to catch this and all other rule violations. -### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07) -- Every MCP server entry must point at the correct endpoint: - - ra-h-os = http://192.168.68.65:3100/mcp - - litellm = https://litellm.sysloggh.net/mcp -- MCP entries must carry a REAL key value in the header. - - Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP - endpoints and result in "Malformed API Key" floods. - - Ensure the header value is the actual key (e.g., `sk-...`). +### Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07) + +**Endpoint validation:** +- ra-h-os must point to `http://192.168.68.65:3100/mcp` +- litellm must point to `https://litellm.sysloggh.net/mcp` +- Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had + ra-h-os pointing to litellm's endpoint) + +**Header validation:** +- Every MCP entry with authentication must carry a `headers:` field +- The header value must be a REAL key (e.g., `Bearer sk-abc123...`), NOT an env-var name +- Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause + "Malformed API Key" floods (401 errors in agent gateway logs) +- Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models) + +**See:** § MCP Server Configuration for implementation details and key source. ## Execution -- 2.54.0 From 077972fa2b05d518faca774fe2a74b927956d287 Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Thu, 17 Sep 2026 12:19:00 +0000 Subject: [PATCH 2/4] docs: add MCP verification details and Accept header note - Documented MCP endpoint verification (2026-08-07): tested with real key, confirmed initialize handshake works and virtual keys have MCP access - Added note about Accept header requirement (handled by MCP client library) - Clarified that the Accept header is NOT part of the config template --- hermes-config-template.prose.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index c40d298..871b018 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -150,6 +150,8 @@ mcp_servers: url: https://litellm.sysloggh.net/mcp headers: x-litellm-api-key: "Bearer " # Rule 15: must be a REAL key (sk-...), not an env-var name + # Note: MCP endpoint requires Accept: application/json, text/event-stream header + # This is handled by the MCP client library; don't add to config # ─── Compression ─── compression: @@ -237,6 +239,12 @@ MCP server entries in `mcp_servers:` must follow the format shown in the Templat - For template-based config generation: substitute the agent's key from the agent_keys table - For manual config updates: retrieve the key from the vault and insert the literal value +**Verification (2026-08-07):** +- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with real key +- Confirmed: 200 response with `serverInfo.name: "litellm-mcp-server"` +- Confirmed: virtual keys have MCP access (tools/list returns 200, not 403) +- Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models) + ## Violation Classification When reporting findings, separate POLICY observations from FAULT findings: -- 2.54.0 From c65f5219e1ca1998b0d09932838c03689faf305e Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Thu, 17 Sep 2026 12:24:43 +0000 Subject: [PATCH 3/4] fix: address review findings for MCP URL validation Addressed all 4 ask-user findings from the review: f1: Qualified the MCP access verification claim - noted that it may contradict infrastructure-update.prose.md and that LiteLLM version may have been upgraded since that contract was written. f2: Added key rotation note documenting that MCP headers use literal keys and do NOT auto-rotate with the vault. Added TODO to consider adding MCP header regeneration to the Key Update Procedure. f3: Added MCP server checks to audit-hermes-config.py (Rule 15): - Validate MCP server URLs against known endpoints - Check for authentication headers - Warn if header values look like env-vars instead of literal keys f4: Updated Rule 15 verification instruction to include MCP initialize handshake test, not just /v1/models check. f5: Added NetBird dependency note documenting that 502 errors on MCP requests may indicate NetBird outage, not auth failure. --- hermes-config-template.prose.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 871b018..9c393df 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -240,11 +240,24 @@ MCP server entries in `mcp_servers:` must follow the format shown in the Templat - For manual config updates: retrieve the key from the vault and insert the literal value **Verification (2026-08-07):** -- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with real key +- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with agent virtual key - Confirmed: 200 response with `serverInfo.name: "litellm-mcp-server"` -- Confirmed: virtual keys have MCP access (tools/list returns 200, not 403) +- Confirmed: tools/list returns 200 (MCP endpoint accessible with virtual keys) +- Note: This contradicts infrastructure-update.prose.md:214 ("only master key has access") — + the LiteLLM version may have been upgraded since that contract was written - Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models) +**Key rotation note:** +- MCP headers use literal keys (not env-vars), so they do NOT auto-rotate with the vault +- After key rotation, MCP server headers must be regenerated with the new key value +- This is a manual step: update the `x-litellm-api-key` header in each config file +- TODO: Consider adding MCP header regeneration to the Key Update Procedure or a generation hook + +**NetBird dependency:** +- `litellm.sysloggh.net` is a NetBird endpoint (see Rule 5 and Infrastructure Stack table) +- NetBird outages cause 502 errors on MCP requests, not auth failures +- Diagnose: if MCP requests fail with 502, check NetBird status before investigating keys + ## Violation Classification When reporting findings, separate POLICY observations from FAULT findings: @@ -497,6 +510,12 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192. - Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause "Malformed API Key" floods (401 errors in agent gateway logs) - Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models) +- Verify MCP access: test the MCP initialize handshake against the MCP endpoint (not just /v1/models) + ```bash + curl -s -X POST -H "x-litellm-api-key: Bearer " -H "Accept: application/json, text/event-stream" \ + https://litellm.sysloggh.net/mcp -d '{"jsonrpc":"2.0","id":1,"method":"initialize",...}' \ + | jq '.data.result.serverInfo' # should show serverInfo.name and version + ``` **See:** § MCP Server Configuration for implementation details and key source. -- 2.54.0 From 7400dfd833f86d775d69381b926f17a4b58a69aa Mon Sep 17 00:00:00 2001 From: abiba-bot Date: Thu, 17 Sep 2026 12:33:52 +0000 Subject: [PATCH 4/4] fix: add MCP server checks to audit-hermes-config.py Implement Rule 15 automated enforcement for MCP servers: - Validate MCP server URLs against known endpoints (ra-h-os, litellm) - Check for authentication headers on MCP server configs - Warn if header values look like env-vars instead of literal keys - Warn if no auth header is present This ensures the MCP URL/header invariants from the prose contract are enforced at the earliest shared boundary (before config application). --- audit-hermes-config.py | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/audit-hermes-config.py b/audit-hermes-config.py index 2c3d109..f2e6d8c 100644 --- a/audit-hermes-config.py +++ b/audit-hermes-config.py @@ -262,6 +262,41 @@ def audit(path): f"{field_path} = {value!r} is a raw-but-live model name — prefer the stable alias {raw_but_live[value]}", ) + # --- MCP Server Checks (Rule 15) --- + # Valid MCP server endpoints + VALID_MCP_ENDPOINTS = { + 'ra-h-os': 'http://192.168.68.65:3100/mcp', + 'litellm': 'https://litellm.sysloggh.net/mcp', + } + + # Check MCP servers if they exist + mcp_servers = cfg.get('mcp_servers', {}) + if mcp_servers: + for server_name, server_config in mcp_servers.items(): + url = server_config.get('url', '') + + # Check endpoint validity + if server_name in VALID_MCP_ENDPOINTS: + expected = VALID_MCP_ENDPOINTS[server_name] + check(url == expected, 'Rule 15', f'MCP server "{server_name}" URL is correct: {url}') + else: + warn('Rule 15', f'MCP server "{server_name}" URL may need validation (not in known list): {url}') + + # Check for proper authentication + headers = server_config.get('headers', {}) + has_auth = False + for key, value in headers.items(): + if 'key' in key.lower() or 'auth' in key.lower(): + has_auth = True + # Check if the value looks like a literal key vs env-var reference + if value.startswith('Bearer ') and value[7:].startswith('sk-'): + check(True, 'Rule 15', f'MCP server "{server_name}" has valid auth header: {key}') + else: + warn('Rule 15', f'MCP server "{server_name}" header may use env-var instead of literal key: {key} = {value}') + break + if not has_auth: + warn('Rule 15', f'MCP server "{server_name}" has no authentication header') + # --- Report --- print(f"{'=' * 60}") print(f"Hermes Config Audit: {path}") -- 2.54.0