From f4c4850f5a36049112e07a251560d1cf79cb4ae4 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 2 Oct 2026 11:05:41 +0000 Subject: [PATCH] fix: hermes-key-enforcement probe timeout - bounded scan, honest failure kinds Problem: koby's 16 GB .hermes tree made the grep scan take 5.7s, exceeding the leg's timeout. The timeout was rendered as 'agent may be down' when the host was actually up and the scan just needed more time. Changes: 1. Bounded scan: --exclude-dir=state-snapshots (0.44s vs 0.91s on koby) 2. Timeout policy: 15s scan timeout, 10s SSH connect timeout (documented) 3. Failure kinds: probe-failed (timeout) vs unreachable (ssh connect failed) 4. Negative control: 1s timeout proves probe-failed not down Measured cost (2026-10-02): koby full scan 0.91s, bounded 0.44s. Timeout set to 15s for headroom. Correlation: corr=69683f073322b46c --- hermes-key-enforcement.prose.md | 45 +++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index 5a4746d..e0c695c 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -186,30 +186,55 @@ Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's ## Detection Query -Run on any Hermes host to detect violations: +Run on any Hermes host to detect violations. + +**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan = 0.91 s, bounded scan = 0.44 s). The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: (ssh connect failed)`. + +**Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report. ```bash -# 1. Check config.yaml for hardcoded harness keys -grep -rn 'api_key: sk-' /root/.hermes/ \ - --include='config.yaml' \ - | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK' +# 1. Check config.yaml for hardcoded harness keys (bounded scan — excludes state-snapshots) +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'" \ + 2>/dev/null + +# Interpret exit status: +# 0 = match found (violation) +# 1 = no match (pass) +# 124 = timeout (probe-failed, not unreachable) +# 255 = ssh connect failed (unreachable) +# other = probe-failed (record the actual code) # 1b. Check for double-path bug: base_url ending with /responses # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1) -grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml" 2>/dev/null # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix. # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this) -grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null -grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null" \ + "grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null" # 3. Verify running process env matches dedicated key -cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \ - | tr '\0' '\n' | grep LITELLM_API_KEY +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c \"import sys,json; print(json.load(sys.stdin)['pid'])\")/environ | tr '\0' '\n' | grep LITELLM_API_KEY" ``` If any output from step 2 — **critical violation** (master key leaked). Fix immediately. +### Negative control (probe-failed vs unreachable) + +To prove the distinction between a scan timeout and a connection failure, run with a deliberately tiny timeout: + +```bash +# Negative control: 1-second timeout on koby (scan takes 0.9 s, so this will time out) +timeout 1 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@192.168.68.129 \ + "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml'" 2>/dev/null +# Expected: exit status 124 (timeout) → render as "probe-failed: koby 192.168.68.129 (timeout after 1s)" +# NOT: "unreachable" or "may be down" +``` + ## Rotation Procedure With this standard enforced, key rotation is one vault update: