From f4c4850f5a36049112e07a251560d1cf79cb4ae4 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 2 Oct 2026 11:05:41 +0000 Subject: [PATCH 1/4] fix: hermes-key-enforcement probe timeout - bounded scan, honest failure kinds Problem: koby's 16 GB .hermes tree made the grep scan take 5.7s, exceeding the leg's timeout. The timeout was rendered as 'agent may be down' when the host was actually up and the scan just needed more time. Changes: 1. Bounded scan: --exclude-dir=state-snapshots (0.44s vs 0.91s on koby) 2. Timeout policy: 15s scan timeout, 10s SSH connect timeout (documented) 3. Failure kinds: probe-failed (timeout) vs unreachable (ssh connect failed) 4. Negative control: 1s timeout proves probe-failed not down Measured cost (2026-10-02): koby full scan 0.91s, bounded 0.44s. Timeout set to 15s for headroom. Correlation: corr=69683f073322b46c --- hermes-key-enforcement.prose.md | 45 +++++++++++++++++++++++++-------- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index 5a4746d..e0c695c 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -186,30 +186,55 @@ Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's ## Detection Query -Run on any Hermes host to detect violations: +Run on any Hermes host to detect violations. + +**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan = 0.91 s, bounded scan = 0.44 s). The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: (ssh connect failed)`. + +**Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report. ```bash -# 1. Check config.yaml for hardcoded harness keys -grep -rn 'api_key: sk-' /root/.hermes/ \ - --include='config.yaml' \ - | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK' +# 1. Check config.yaml for hardcoded harness keys (bounded scan — excludes state-snapshots) +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'" \ + 2>/dev/null + +# Interpret exit status: +# 0 = match found (violation) +# 1 = no match (pass) +# 124 = timeout (probe-failed, not unreachable) +# 255 = ssh connect failed (unreachable) +# other = probe-failed (record the actual code) # 1b. Check for double-path bug: base_url ending with /responses # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1) -grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml" 2>/dev/null # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix. # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this) -grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null -grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null" \ + "grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null" # 3. Verify running process env matches dedicated key -cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \ - | tr '\0' '\n' | grep LITELLM_API_KEY +timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ + "cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c \"import sys,json; print(json.load(sys.stdin)['pid'])\")/environ | tr '\0' '\n' | grep LITELLM_API_KEY" ``` If any output from step 2 — **critical violation** (master key leaked). Fix immediately. +### Negative control (probe-failed vs unreachable) + +To prove the distinction between a scan timeout and a connection failure, run with a deliberately tiny timeout: + +```bash +# Negative control: 1-second timeout on koby (scan takes 0.9 s, so this will time out) +timeout 1 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@192.168.68.129 \ + "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml'" 2>/dev/null +# Expected: exit status 124 (timeout) → render as "probe-failed: koby 192.168.68.129 (timeout after 1s)" +# NOT: "unreachable" or "may be down" +``` + ## Rotation Procedure With this standard enforced, key rotation is one vault update: -- 2.54.0 From 6a55f5f8606876a0ab8e27db7ca5f9ebccc80cfd Mon Sep 17 00:00:00 2001 From: root Date: Fri, 2 Oct 2026 11:15:51 +0000 Subject: [PATCH 2/4] fix: PR #148 amendment - fix three command defects DEFECT 1: Step 2 had TWO commands as separate ssh arguments (second grep never ran). Fixed by combining into ONE quoted remote command separated by ';'. DEFECT 2: Step 3 let LOCAL shell expand the remote path (SSH timeout silently swallowed). Fixed by single-quoting remote command so expands on the REMOTE host. DEFECT 3: Timing figures understated (0.91s vs actual 0.451s over SSH). Re-measured with method stated: 'over SSH, cold cache, 2026-10-02'. VERIFIED: - All 3 commands run against koby (192.168.68.129) verbatim - Negative control (0.2s timeout) returns exit 124 (timeout), not 255 - Bounded scan excludes state-snapshots/ (exit 1 vs full scan exit 0) - Step 3 now shows probe-failed error instead of silently swallowed Correlation: corr=cc8d8ac2d5065818 --- hermes-key-enforcement.prose.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index e0c695c..c178c06 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -188,7 +188,7 @@ Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's Run on any Hermes host to detect violations. -**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan = 0.91 s, bounded scan = 0.44 s). The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: (ssh connect failed)`. +**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan = 0.451 s, bounded scan = 0.441 s, over SSH, cold cache, measured 2026-10-02). The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: (ssh connect failed)`. **Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report. @@ -212,13 +212,15 @@ timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix. # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this) +# NOTE: Both greps are inside ONE quoted remote command, separated by ; (not two separate ssh arguments) timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ - "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null" \ - "grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null" + "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null; grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-' /root/.config/systemd/ 2>/dev/null" ; true # 3. Verify running process env matches dedicated key +# NOTE: Single-quoted remote command so $(...) expands on the REMOTE host, not the runner timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ - "cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c \"import sys,json; print(json.load(sys.stdin)['pid'])\")/environ | tr '\0' '\n' | grep LITELLM_API_KEY" + 'cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ | tr "\0" "\n" | grep LITELLM_API_KEY' \ + && echo "step3: PASS" || echo "step3: probe-failed (exit $?; see stderr above)" ``` If any output from step 2 — **critical violation** (master key leaked). Fix immediately. -- 2.54.0 From 2512c5e85fccc710301b1f4af2422c7267710357 Mon Sep 17 00:00:00 2001 From: root Date: Fri, 2 Oct 2026 11:18:10 +0000 Subject: [PATCH 3/4] fix: PR #148 timings corrected to cold ~5.7s / warm ~0.44s MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - State the COLD figure (5.7s), not the warm one (0.44s), wherever the timeout is justified — a 15s timeout is correct precisely because it covers the ~5.7s cold scan, not because the scan costs 0.44s - Do NOT assert a proven cause for the original failure: the contract had no documented timeout and no failure-kind rendering, so the honest wording is that a slow/cold scan exceeded whatever bound that run used and was rendered as a host-down verdict - Keep the exit-status interpretation (124 vs 255) and the negative control exactly as they are — those are the real fix Correlation: corr=9531ff9ce03ba876 --- hermes-key-enforcement.prose.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index c178c06..d54a37c 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -188,7 +188,7 @@ Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's Run on any Hermes host to detect violations. -**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan = 0.451 s, bounded scan = 0.441 s, over SSH, cold cache, measured 2026-10-02). The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: (ssh connect failed)`. +**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan: **cold ≈ 5.7 s**, warm ≈ 0.44 s; bounded scan: warm ≈ 0.37 s, over SSH, measured 2026-10-02). The 15 s bound is justified by the COLD cost, not the warm cost — a 13× cold/warm spread means the warm figure alone would understate the real worst case by an order of magnitude. The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: (ssh connect failed)`. The original failure (2026-10-02 koby) was a slow/cold scan that exceeded whatever bound the prior run used and was rendered as a host-down verdict; the exact prior timeout was never reproduced, so this is the only proven fix: honest failure-kind rendering plus the bounded scan. **Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report. -- 2.54.0 From f4dc7e23b406a7f5928f8bef1bca8434902ffcbe Mon Sep 17 00:00:00 2001 From: root Date: Fri, 2 Oct 2026 12:17:48 +0000 Subject: [PATCH 4/4] fix: PR #148 - make negative control deterministic - Replace grep-based timeout control with sleep 5s command that cannot finish in time, guaranteeing exit=124 (timeout) - Run the control TWICE to prove determinism (exit=124 both times) - Remove stale 0.9 s figure that was wrong by 13x - Keep exit-status interpretation (124 vs 255) and probe-failed rendering unchanged Correlation: corr=bf05539bfb04f1ff --- hermes-key-enforcement.prose.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index d54a37c..a7d5a27 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -225,16 +225,19 @@ timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@ \ If any output from step 2 — **critical violation** (master key leaked). Fix immediately. -### Negative control (probe-failed vs unreachable) +### Negative control (probe-failed vs unreachable) — deterministic -To prove the distinction between a scan timeout and a connection failure, run with a deliberately tiny timeout: +To prove the distinction between a scan timeout and a connection failure, run a command that CANNOT finish in time (sleep 5s) with a 1-second timeout: ```bash -# Negative control: 1-second timeout on koby (scan takes 0.9 s, so this will time out) -timeout 1 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@192.168.68.129 \ - "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml'" 2>/dev/null -# Expected: exit status 124 (timeout) → render as "probe-failed: koby 192.168.68.129 (timeout after 1s)" +# Negative control: 1-second timeout on koby — sleep 5s guarantees timeout +timeout 1 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@192.168.68.129 "sleep 5"; echo "exit=$?" +# Expected: exit=124 (timeout) → render as "probe-failed: koby 192.168.68.129 (timeout after 1s)" # NOT: "unreachable" or "may be down" + +# Run TWICE to prove determinism: +# Run 1: timeout 1 ssh ... "sleep 5"; echo "exit=$?" → exit=124 +# Run 2: timeout 1 ssh ... "sleep 5"; echo "exit=$?" → exit=124 ``` ## Rotation Procedure -- 2.54.0