Addressed all 4 ask-user findings from the review:
f1: Qualified the MCP access verification claim - noted that it may
contradict infrastructure-update.prose.md and that LiteLLM version may
have been upgraded since that contract was written.
f2: Added key rotation note documenting that MCP headers use literal keys
and do NOT auto-rotate with the vault. Added TODO to consider adding
MCP header regeneration to the Key Update Procedure.
f3: Added MCP server checks to audit-hermes-config.py (Rule 15):
- Validate MCP server URLs against known endpoints
- Check for authentication headers
- Warn if header values look like env-vars instead of literal keys
f4: Updated Rule 15 verification instruction to include MCP initialize
handshake test, not just /v1/models check.
f5: Added NetBird dependency note documenting that 502 errors on MCP
requests may indicate NetBird outage, not auth failure.
- Documented MCP endpoint verification (2026-08-07): tested with real key,
confirmed initialize handshake works and virtual keys have MCP access
- Added note about Accept header requirement (handled by MCP client library)
- Clarified that the Accept header is NOT part of the config template
- Added litellm MCP server entry to mcp_servers section with correct URL
(https://litellm.sysloggh.net/mcp) and header format
- Updated Rule 15 to be more specific about endpoint validation and
header requirements (REAL keys, not env-var references)
- Added MCP Server Configuration section with implementation details
- Documented the 2026-08-07 Tanko incident where ra-h-os was pointing
to litellm endpoint with env header causing 401 floods
- Updated frontmatter to reflect the changes
Fixes: #keyless-mcp-incident-20260807
Refs: Rule 15 (MCP Endpoint and Header Validation)
Contract design defect: 'uses a non-harness provider' (POLICY) and
'cannot authenticate' (FAULT) were printed as the same violation class.
A policy observation must never be phrased as if the agent were broken.
Changes:
1. Added 'Violation Classification' section to all three contracts
2. Separated POLICY (observation only) from FAULT (requires request-level evidence)
3. Rules:
- Do NOT infer runtime credential resolution from config text alone
- Require request-level evidence before calling a FAULT: observed auth failure
or absence of successful calls
- If calls are succeeding, output is 'POLICY: uses <provider> directly; calls
succeeding' - not a violation
- State what you OBSERVED, not what the field implies
Files changed (3):
- hermes-key-enforcement.prose.md
- hermes-config-template.prose.md
- hermes-agent-baseline.prose.md
The helper was correct but dead code - nothing called it. This commit:
1. Makes the helper runnable standalone: scripts/hermes-reachability-check.sh <host> <pattern> <path>
2. Wires all THREE contracts to it:
- hermes-key-enforcement.prose.md
- hermes-config-template.prose.md
- hermes-agent-baseline.prose.md
3. Each contract now explicitly instructs to run the helper and interpret the three outcomes
4. States that the bug this replaces was deriving reachability from the remote grep's exit code
Files changed (4):
- scripts/hermes-reachability-check.sh (standalone mode added)
- hermes-key-enforcement.prose.md (reachability section added)
- hermes-config-template.prose.md (reachability section added)
- hermes-agent-baseline.prose.md (reachability section added)
- Remove retired names (qwen3.6-27B-code, qwen3.6-35B-udq4) from live alias claims
- Update Strix Halo model to Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (strix-moe, 256K ctx)
- Fix litellm-health step 7 probe to gpu-vision (monitor key scoped)
- Move qwen3.6-27B-code/35B-udq4 from raw-but-live to non-resolving in audit
- Fold in pm2-self-heal: remove spoton-service (live PM2 set is 4/4)
- Update hermes templates, key enforcement, timeout tables to live names
audit-hermes-config.py Rule 8 required auxiliary.vision.model and
auxiliary.web_extract.model to equal the retired 'gpu-light', so a config
adopting the live canonical 'gpu-vision' FAILED our own audit - the audit was
enforcing a dead alias (400 Invalid model name). Rule 8 now requires
gpu-vision; retired names gpu-light/crew-auto join the raw-name rejection set;
the guidance message names the live aliases.
Sweep of the remaining references: gpu-self-heal stops canonicalizing
gpu-light; hermes-config-template, hermes-agent-baseline, hermes-key-enforcement,
inference-optimization, litellm-client-timeouts and gpu-fleet now use the live
gpu-vision alias. Where a file restated model/rpm/weight/fallback state it now
points at CT 116 /opt/inference-harness/litellm_config.yaml instead of
duplicating it. koby's .129 config is report-only and recorded, not edited.
Adds tests/test_audit_hermes_config_alias.py: executes the audit CLI and asserts
gpu-vision passes while gpu-light and gemma-4-12b fail.
Mumuni migrated off Abiba CT100 (192.168.68.24, purged) to dedicated
CT kagentz (192.168.68.14) on minipve. Verified live on kagentz:
- Gateway: systemd unit hermes-gateway.service (User=hermes), active
- Hermes home: /home/hermes/.hermes
- gateway_state.json present at ~/.hermes/gateway_state.json
Updates Mumuni rows/references in 9 contracts. Abiba-owned refs
(.24 pi agent, GPU monitor, infrastructure-control CRITICAL file)
and historical runs/ logs intentionally left unchanged.
Per relay #738 follow-ups. Refs #735-#738.
storepve (192.168.68.6) has no service on :8888. SearXNG runs on VM 109
(docker-vm) at 192.168.68.7:8888 (verified HTTP 200). Aligns with every other
contract reference. Missed in PR #39.
- Updated hermes-config-template.prose.md: all references to strix-moe for
compression changed to syslog-auto to match operational decision on 2026-07-23
(prevents sustained Strix Halo thermal load via weighted pool).
- Updated audit-hermes-config.py Rule 7 to expect syslog-auto instead of
strix-moe, ensuring Abiba's next run validates against the correct baseline.
Rule 14: model.provider MUST be 'harness' (custom_providers[0].name), NOT 'custom'.
When provider: custom, Hermes falls through to generic resolution path that
ignores key_env, producing 'no-key-required' → HTTP 401.
audit-hermes-config.py: encodes all 14 contract rules as automated checks.
Run before and after any Hermes config change.
Root cause: WAL #1471 (2026-07-19 Mumuni 401 incident)
Agent configs aligned to stable aliases (gpu-light/gpu-dense/strix-moe/syslog-auto):
- Tanko fixed: base_url :4000->/v1 (nginx), max_context_window 131072->262144,
aux.compression gemma->strix-moe, aux gemma->gpu-light
- Mumuni aligned: aux gemma->gpu-light, delegation/x_search qwen->gpu-dense
- Koby/Koonimo already used gpu-light (validated)
- Zero raw model names remain in any agent config
No-master-key rule: litellm_proxy_master_key (sk-litellm-...) is admin-only
(/key/list, /key/generate). All inference uses agent keys. Health-check model
tests use dedicated monitor key (/etc/litellm-monitor.env on CT 116).
Fixed: daily-infra-report.py, gpu-self-heal.py, litellm-health-check.sh.
Fixed wrong key-scope note: qwen3.6-35B-udq4 IS in scope for baggy/koby/mumuni/
abiba-pi keys (not 'NOT in allowlist' as previously documented).
hermes-key-enforcement.prose.md:
- Add IP column to verified agents table
- Fix Tanko status: was using master key via systemd drop-in (not compliant)
- Add detection query step 2: check systemd drop-ins for master key leaks
- Add step 3: verify running process env against dedicated key
- Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)
gpu-fleet.prose.md:
- Add CT ID column to agent table for cross-reference
- Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
- Update Abiba LiteLLM key to match actual models.json key
- Set Koonimo IP to unknown (was .114 — incorrect per user)
- Fix pi-specific paths .24 → .65
hermes-config-template.prose.md:
- Set Koonimo IP to unknown
infrastructure-control.prose.md:
- Set Koonimo IP to unknown
- Fix Koonimo CT reference
- hermes-key-enforcement.prose.md: NEW contract enforcing api_key_env for all harness/LiteLLM providers, exempting external providers
- hermes-config-template.prose.md: Updated with full fix log (15 total fixes across Koby/Koonimo/Mumuni/Tanko), model:auto detection rule, key-to-agent mapping
- infrastructure-control.prose.md: Added Stirling-PDF service details (port 8989, credentials, API key, Swagger URL) replacing Bento-PDF
- REMOVED hardcoded Agent Keys table (keys became stale after LiteLLM downgrade).
- ADDED Key Management section: verify loop, rotate + deploy procedure, key
deployment rules (plaintext in /etc/environment only, api_key_env pattern),
current aliases (2026-07-02 rotation).
- LiteLLM DB is the single source of truth for keys; contracts never store plaintext.
- Mumuni sub-agent profiles documented (6, all inherit auth from main config).
abiba-bot
root
jerome