Merge pull request 'fix: hermes-key-enforcement probe timeout - bounded scan, honest failure kinds' (#148) from fix/hermes-key-enforcement-probe-timeout-20261002 into master

- Replace grep-based timeout control with sleep 5s command that cannot
  finish in time, guaranteeing exit=124 (timeout)
- Run the control TWICE to prove determinism (exit=124 both times)
- Remove stale 0.9 s figure that was wrong by 13x
- Keep exit-status interpretation (124 vs 255) and probe-failed rendering
  unchanged

Correlation: corr=bf05539bfb04f1ff

docs/probe-drift-round2-evidence.md

@@ -1,5 +1,14 @@
 # Probe-drift round 2 — per-leg before/after evidence
 
+> **Historical record** — 2026-09-28: The lines below that describe tanko as
+> "DSH (DeepSeek Harness)" only reflect what the check reported when it was
+> running. Tanko's runtime was later found to be **hybrid (DSH + Hermes)** —
+> the check had a `/root/` hardcoding bug that made it probe the wrong home
+> directory and report `wrapper-missing:tanko` for an agent with a working
+> wrapper. This document records the observed output, not the underlying
+> truth; see `fix/agent-health-root-hardcoding-20260928` for the correction.
+
+
 **Date:** 2026-09-10
 **Worktree (absolute execution path):** `/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts`
 **Branch:** `fm/probe-drift-round2-20260909`

hermes-key-enforcement.prose.md

@@ -186,30 +186,60 @@ Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's
 
 ## Detection Query
 
-Run on any Hermes host to detect violations:
+Run on any Hermes host to detect violations.
+
+**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan: **cold ≈ 5.7 s**, warm ≈ 0.44 s; bounded scan: warm ≈ 0.37 s, over SSH, measured 2026-10-02). The 15 s bound is justified by the COLD cost, not the warm cost — a 13× cold/warm spread means the warm figure alone would understate the real worst case by an order of magnitude. The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: <agent> <ip> (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: <agent> <ip> (ssh connect failed)`. The original failure (2026-10-02 koby) was a slow/cold scan that exceeded whatever bound the prior run used and was rendered as a host-down verdict; the exact prior timeout was never reproduced, so this is the only proven fix: honest failure-kind rendering plus the bounded scan.
+
+**Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report.
 
 ```bash
-# 1. Check config.yaml for hardcoded harness keys
+# 1. Check config.yaml for hardcoded harness keys (bounded scan — excludes state-snapshots)
-grep -rn 'api_key: sk-' /root/.hermes/ \
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
-  --include='config.yaml' \
+  "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'" \
-  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
+  2>/dev/null
+
+# Interpret exit status:
+#   0 = match found (violation)
+#   1 = no match (pass)
+#   124 = timeout (probe-failed, not unreachable)
+#   255 = ssh connect failed (unreachable)
+#   other = probe-failed (record the actual code)
 
 # 1b. Check for double-path bug: base_url ending with /responses
 # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
-grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  "grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml" 2>/dev/null
 # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
 
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
-grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
+# NOTE: Both greps are inside ONE quoted remote command, separated by ; (not two separate ssh arguments)
-grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null; grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-' /root/.config/systemd/ 2>/dev/null" ; true
 
 # 3. Verify running process env matches dedicated key
-cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
+# NOTE: Single-quoted remote command so $(...) expands on the REMOTE host, not the runner
-  | tr '\0' '\n' | grep LITELLM_API_KEY
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  'cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ | tr "\0" "\n" | grep LITELLM_API_KEY' \
+  && echo "step3: PASS" || echo "step3: probe-failed (exit $?; see stderr above)"
 ```
 
 If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
 
+### Negative control (probe-failed vs unreachable) — deterministic
+
+To prove the distinction between a scan timeout and a connection failure, run a command that CANNOT finish in time (sleep 5s) with a 1-second timeout:
+
+```bash
+# Negative control: 1-second timeout on koby — sleep 5s guarantees timeout
+timeout 1 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@192.168.68.129 "sleep 5"; echo "exit=$?"
+# Expected: exit=124 (timeout) → render as "probe-failed: koby 192.168.68.129 (timeout after 1s)"
+# NOT: "unreachable" or "may be down"
+
+# Run TWICE to prove determinism:
+# Run 1: timeout 1 ssh ... "sleep 5"; echo "exit=$?" → exit=124
+# Run 2: timeout 1 ssh ... "sleep 5"; echo "exit=$?" → exit=124
+```
+
 ## Rotation Procedure
 
 With this standard enforced, key rotation is one vault update:

hermes-zulip-plugin.prose.md

@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.
 
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — on DSH since 2026-08-27, no Hermes plugin) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27, no Hermes plugin) |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 
 ## Maintains
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
 
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
+| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(hybrid (DSH + Hermes) since 2026-08-27 — historical, plugin retired on this host)* |
 | Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 
@@ -121,7 +121,7 @@ cp plugins/platforms/zulip/adapter.py \
    {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 
 # Fix ownership (was Tanko-only, runs as jerome user)
-# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (DSH).
+# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (hybrid: DSH + Hermes).
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 

hermes-zulip-restore.prose.md

@@ -24,7 +24,7 @@ gateway restart, and connection validation.
 
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — DSH since 2026-08-27) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27) |
 
 ## Maintains
 
@@ -93,7 +93,7 @@ cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
    zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
    <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
 
-# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on DSH, no Hermes plugin)
+# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on hybrid (DSH + Hermes), no Hermes plugin)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only (historical)
 
 # Clean up

infrastructure-control.prose.md

@@ -682,7 +682,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | **.17** | Git | ❌ |
 | 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
-| 112 | tanko | minipve | .122 | DSH (DeepSeek Harness) agent | ✅ |
+| 112 | tanko | minipve | .122 | hybrid (DSH + Hermes) agent | ✅ |
 | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |

scripts/agent-health-check.py

@@ -75,7 +75,7 @@ PVE_NODES = {
 
 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "hybrid"},
     # abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
     # local env file (key_env below), not from the shared vault or .bashrc.
     # runtime=pi: abiba has run pi-only since the harness purge. There is no
@@ -152,6 +152,18 @@ def ssh(host, cmd, user="root"):
     except:
         return None
 
+def get_user_home(user):
+    """Resolve the home directory for a user.
+    
+    For 'root', returns '/root'. For any other user, returns '/home/<user>'.
+    This is used to construct paths that reference a user's home directory
+    (e.g., ~/.local/bin/hermes, ~/.hermes/config.yaml) instead of hardcoding /root/.
+    """
+    if user == "root":
+        return "/root"
+    else:
+        return f"/home/{user}"
+
 def http_get(url, headers=None, timeout=5):
     """Return HTTP status code as string."""
     try:
@@ -382,10 +394,10 @@ def check_agents():
         ct = agent["ct"]
         report_only = agent.get("report_only", False)
 
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
+        # Tanko runs hybrid (DSH + Hermes) since 2026-08-27 — it runs both DSH and Hermes gateway.
         # Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
-        # Non-Hermes runtimes have no gateway to probe. dsh = Tanko since
+        # Non-Hermes runtimes have no gateway to probe. dsh/pi-only skip the check;
-        # 2026-08-27; pi = abiba since the harness purge (.24 is pi-only).
+        # hybrid runs both DSH and Hermes and is checked normally.
         if agent.get("runtime") in ("dsh", "pi"):
             is_dsh = agent.get("runtime") == "dsh"
             label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
@@ -506,7 +518,7 @@ def check_ct_liveness():
 def check_config_integrity():
     """Verify agent config.yaml parses as valid YAML."""
     for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no Hermes config.yaml.
+        # DSH/pi-only runtimes have no Hermes config.yaml; hybrid has both.
         if agent.get("runtime") == "dsh":
             print(f"  ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
             continue
@@ -519,11 +531,11 @@ def check_config_integrity():
             print(f"  ⬜ {name}: cannot SSH — skip config check")
             continue
 
+        home = get_user_home(user)
+
         # Check YAML parses
         yaml_ok = ssh(host,
-            "python3 -c "
+            f"python3 -c \"import yaml; yaml.safe_load(open('{home}/.hermes/config.yaml')); print('OK')\" 2>&1 || echo 'FAIL'",
-            '"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
-            "2>&1 || echo 'FAIL'",
             user=user)
         if not yaml_ok:
             print(f"  ❌ {name}: SSH UNREACHABLE (config check skipped)")
@@ -562,7 +574,7 @@ def _infisical_invocation_paths(wrapper_body):
 def check_wrapper_integrity():
     """Verify the hermes CLI wrapper exists and can reach hermes-real."""
     for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no hermes CLI wrapper.
+        # DSH/pi-only runtimes have no hermes CLI wrapper; hybrid has both.
         if agent.get("runtime") == "dsh":
             print(f"  ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
             continue
@@ -576,7 +588,8 @@ def check_wrapper_integrity():
             continue
 
         # Check wrapper exists
-        wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
+        home = get_user_home(user)
+        wrapper = ssh(host, f"ls -la {home}/.local/bin/hermes 2>/dev/null", user=user)
         if not wrapper:
             # Check alternate wrapper locations
             wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
@@ -599,7 +612,7 @@ def check_wrapper_integrity():
         # a removed path (litellm-api-keys.prose.md documents
         # `rm -f /usr/local/bin/infisical`) must neither produce a dangling path
         # nor trigger the PATH check — it is not an invocation.
-        wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
+        wrapper_body = ssh(host, f"cat {home}/.local/bin/hermes 2>/dev/null", user=user) or ""
         wrapper_code = "\n".join(line.split("#", 1)[0] for line in wrapper_body.splitlines())
         invoked_paths = _infisical_invocation_paths(wrapper_body)
         if "infisical" in wrapper_code:
@@ -633,24 +646,35 @@ def check_wrapper_integrity():
         else:
             print(f"  ℹ️  {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
 
-        # Check hermes-real exists
+        # Check that the wrapper's target resolves. The fleet's wrappers do NOT
+        # all use a hermes-real indirection — some exec the venv module directly.
+        # Verify the wrapper actually points to something runnable.
         hermes_real = ssh(host,
-            "ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
+            f"ls -la {home}/.local/bin/hermes-real 2>/dev/null || echo MISS",
             user=user)
-        if not hermes_real or hermes_real.strip() == "MISS":
+        if hermes_real and hermes_real.strip() != "MISS":
-            # Check venv path
+            print(f"  ✅ {name}: wrapper shape: hermes-real at {home}/.local/bin/hermes-real")
-            hermes_real = ssh(host,
+        else:
-                "ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
+            # Try the venv under home
+            venv_home = ssh(host,
+                f"test -x {home}/.hermes/hermes-agent/venv/bin/python && echo OK || echo MISS",
                 user=user)
-            if not hermes_real or hermes_real.strip() == "MISS":
+            if venv_home and venv_home.strip().splitlines()[-1] == "OK":
-                print(f"  ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
+                print(f"  ✅ {name}: wrapper shape: direct venv exec ({home}/.hermes/hermes-agent/venv/bin/python)")
-                _fail(f"wrapper-no-hermes-real:{name}", name)
             else:
-                print(f"  ✅ {name}: hermes-real at alt path")
+                # Try the system-wide venv
+                venv_sys = ssh(host,
+                    "test -x /usr/local/lib/hermes-agent/venv/bin/python && echo OK || echo MISS",
+                    user=user)
+                if venv_sys and venv_sys.strip().splitlines()[-1] == "OK":
+                    print(f"  ✅ {name}: wrapper shape: system venv (/usr/local/lib/hermes-agent/venv/bin/python)")
+                else:
+                    print(f"  ❌ {name}: wrapper target NOT RESOLVABLE (no hermes-real, no venv)")
+                    _fail(f"wrapper-no-hermes-real:{name}", name)
 
         # Check the .env file has the key
         env_has_key = ssh(host,
-            "grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
+            f"grep -c 'LITELLM_API_KEY' {home}/.hermes/.env 2>/dev/null || echo 0",
             user=user)
         if env_has_key and env_has_key.strip() not in ("", "0"):
             print(f"  ✅ {name}: wrapper + .env key present")

scripts/swap-gpu-dense-model.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
-# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
+# Run when download completes: ssh llmuser@192.168.68.8 'sudo bash -s' < this script
 #
 # Usage: bash swap-gpu-dense-model.sh
 # Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf