feat: add litellm MCP server entry and enhance Rule 15 validation

- Added litellm MCP server entry to mcp_servers section with correct URL
  (https://litellm.sysloggh.net/mcp) and header format
- Updated Rule 15 to be more specific about endpoint validation and
  header requirements (REAL keys, not env-var references)
- Added MCP Server Configuration section with implementation details
- Documented the 2026-08-07 Tanko incident where ra-h-os was pointing
  to litellm endpoint with env header causing 401 floods
- Updated frontmatter to reflect the changes

Fixes: #keyless-mcp-incident-20260807
Refs: Rule 15 (MCP Endpoint and Header Validation)
This commit is contained in:
2026-09-17 11:30:44 +00:00
parent 8a5cba8515
commit d2bca5405a
+37 -9
View File
@@ -5,7 +5,8 @@ description: >
Standard Hermes configuration template for Syslog Solution LLC agents. Standard Hermes configuration template for Syslog Solution LLC agents.
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices. RA-H OS MCP) while keeping agent-specific API keys and model choices.
UPDATED 2026-08-07: Added Rule 15 (MCP Validation) from the 2026-08-07 keyless-MCP incident. UPDATED 2026-08-07: Added litellm MCP server entry; updated Rule 15 (MCP Validation)
to enforce REAL key headers (not env-vars) from the 2026-08-07 keyless-MCP incident.
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
2026-07-16 Mumuni root-cause investigation (WAL #1300). 2026-07-16 Mumuni root-cause investigation (WAL #1300).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13). UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
@@ -145,6 +146,10 @@ mcp_servers:
url: http://192.168.68.65:3100/mcp url: http://192.168.68.65:3100/mcp
timeout: 120 timeout: 120
connect_timeout: 60 connect_timeout: 60
litellm:
url: https://litellm.sysloggh.net/mcp
headers:
x-litellm-api-key: "Bearer <AGENT_KEY>" # Rule 15: must be a REAL key (sk-...), not an env-var name
# ─── Compression ─── # ─── Compression ───
compression: compression:
@@ -217,6 +222,21 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes):
3. **After update**: Restart Hermes on the agent host 3. **After update**: Restart Hermes on the agent host
4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models` 4. **Verify**: `curl -H "Authorization: Bearer sk-<KEY>" http://192.168.68.116/v1/models`
## MCP Server Configuration
MCP server entries in `mcp_servers:` must follow the format shown in the Template section.
**Header requirements (Rule 15):**
- Use `headers:` field with a `x-litellm-api-key` entry
- The value must be `"Bearer <REAL_KEY>"` where `<REAL_KEY>` is a literal LiteLLM virtual key
- Do NOT use env-var references like `$LITELLM_API_KEY` — they resolve to empty strings in
the static config and cause "Malformed API Key" errors (2026-08-07 Tanko incident)
**Key source:**
- Keys are stored in the Infisical vault (project=agents, env=production)
- For template-based config generation: substitute the agent's key from the agent_keys table
- For manual config updates: retrieve the key from the vault and insert the literal value
## Violation Classification ## Violation Classification
When reporting findings, separate POLICY observations from FAULT findings: When reporting findings, separate POLICY observations from FAULT findings:
@@ -455,14 +475,22 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
- **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>` - **Audit script**: Run `python3 /root/prose-contracts/audit-hermes-config.py <config.yaml>`
before and after any config change to catch this and all other rule violations. before and after any config change to catch this and all other rule violations.
### Rule 15: MCP Endpoint and Header Validation (ADDED 2026-08-07) ### Rule 15: MCP Endpoint and Header Validation (UPDATED 2026-08-07)
- Every MCP server entry must point at the correct endpoint:
- ra-h-os = http://192.168.68.65:3100/mcp **Endpoint validation:**
- litellm = https://litellm.sysloggh.net/mcp - ra-h-os must point to `http://192.168.68.65:3100/mcp`
- MCP entries must carry a REAL key value in the header. - litellm must point to `https://litellm.sysloggh.net/mcp`
- Avoid using env-var names like LITELLM_API_KEY in the header; they do not resolve for MCP - Mismatched endpoints cause silent failures (e.g., 2026-08-07 incident: Tanko's config had
endpoints and result in "Malformed API Key" floods. ra-h-os pointing to litellm's endpoint)
- Ensure the header value is the actual key (e.g., `sk-...`).
**Header validation:**
- Every MCP entry with authentication must carry a `headers:` field
- The header value must be a REAL key (e.g., `Bearer sk-abc123...`), NOT an env-var name
- Env-var names like `LITELLM_API_KEY` do NOT resolve in static MCP configs and cause
"Malformed API Key" floods (401 errors in agent gateway logs)
- Verify: header value should match a valid LiteLLM key (test with `curl` against /v1/models)
**See:** § MCP Server Configuration for implementation details and key source.
## Execution ## Execution