Merge branch 'master' into fix/agent-health-root-hardcoding-20260928

The fleet's wrappers do not all use a hermes-real indirection. Some (tanko,
mumuni) exec the venv module directly. This check now:
  1. Tries hermes-real at {home}/.local/bin (koonimo's shape)
  2. Tries the venv under {home}/.hermes/hermes-agent/venv (tanko/mumuni shape)
  3. Tries /usr/local/lib/hermes-agent/venv (legacy system-wide shape)

Each match is reported with which shape it matched, so a genuinely broken
wrapper is still a failure while a different-but-valid shape is not.

docs/probe-drift-round2-evidence.md

@@ -1,5 +1,14 @@
 # Probe-drift round 2 — per-leg before/after evidence
 
+> **Historical record** — 2026-09-28: The lines below that describe tanko as
+> "DSH (DeepSeek Harness)" only reflect what the check reported when it was
+> running. Tanko's runtime was later found to be **hybrid (DSH + Hermes)** —
+> the check had a `/root/` hardcoding bug that made it probe the wrong home
+> directory and report `wrapper-missing:tanko` for an agent with a working
+> wrapper. This document records the observed output, not the underlying
+> truth; see `fix/agent-health-root-hardcoding-20260928` for the correction.
+
+
 **Date:** 2026-09-10
 **Worktree (absolute execution path):** `/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts`
 **Branch:** `fm/probe-drift-round2-20260909`

hermes-agent-baseline.prose.md

@@ -70,10 +70,10 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)
 
 ## Config Pattern — Mandatory Fields
 
-### For Hermes Agents (Mumuni, Koonimo)
+### For Hermes Agents (Mumuni, Koonimo, Tanko-hybrid)
 
 Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
-(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
+(Tanko is hybrid — runs both DSH and Hermes since 2026-08-27, so its Hermes config is also checked.)
 
 ### 1. Main Model
 ```yaml
@@ -297,7 +297,7 @@ Run the consolidated health check:
 ```bash
 python3 /root/scripts/agent-health-check.py
 ```
-This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
+This validates each agent's live LiteLLM key against the gateway, including tanko, which runs HYBRID (DSH + Hermes) since 2026-08-27; detects GPU port conflicts (ghost processes),
 verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
 and counts recent errors. Non-disruptive — never restarts anything.
 

hermes-zulip-plugin.prose.md

@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.
 
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — on DSH since 2026-08-27, no Hermes plugin) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27, no Hermes plugin) |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 
 ## Maintains
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
 
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
+| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(hybrid (DSH + Hermes) since 2026-08-27 — historical, plugin retired on this host)* |
 | Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 
@@ -121,7 +121,7 @@ cp plugins/platforms/zulip/adapter.py \
    {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 
 # Fix ownership (was Tanko-only, runs as jerome user)
-# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (DSH).
+# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (hybrid: DSH + Hermes).
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 

hermes-zulip-restore.prose.md

@@ -24,7 +24,7 @@ gateway restart, and connection validation.
 
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — DSH since 2026-08-27) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27) |
 
 ## Maintains
 
@@ -93,7 +93,7 @@ cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
    zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
    <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
 
-# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on DSH, no Hermes plugin)
+# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on hybrid (DSH + Hermes), no Hermes plugin)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only (historical)
 
 # Clean up

infrastructure-control.prose.md

@@ -682,7 +682,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | **.17** | Git | ❌ |
 | 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
-| 112 | tanko | minipve | .122 | DSH (DeepSeek Harness) agent | ✅ |
+| 112 | tanko | minipve | .122 | hybrid (DSH + Hermes) agent | ✅ |
 | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |

scripts/agent-health-check.py

@@ -75,7 +75,7 @@ PVE_NODES = {
 
 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "hybrid"},
     # abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
     # local env file (key_env below), not from the shared vault or .bashrc.
     # runtime=pi: abiba has run pi-only since the harness purge. There is no
@@ -152,6 +152,18 @@ def ssh(host, cmd, user="root"):
     except:
         return None
 
+def get_user_home(user):
+    """Resolve the home directory for a user.
+    
+    For 'root', returns '/root'. For any other user, returns '/home/<user>'.
+    This is used to construct paths that reference a user's home directory
+    (e.g., ~/.local/bin/hermes, ~/.hermes/config.yaml) instead of hardcoding /root/.
+    """
+    if user == "root":
+        return "/root"
+    else:
+        return f"/home/{user}"
+
 def http_get(url, headers=None, timeout=5):
     """Return HTTP status code as string."""
     try:
@@ -382,10 +394,10 @@ def check_agents():
         ct = agent["ct"]
         report_only = agent.get("report_only", False)
 
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
+        # Tanko runs hybrid (DSH + Hermes) since 2026-08-27 — it runs both DSH and Hermes gateway.
         # Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
-        # Non-Hermes runtimes have no gateway to probe. dsh = Tanko since
-        # 2026-08-27; pi = abiba since the harness purge (.24 is pi-only).
+        # Non-Hermes runtimes have no gateway to probe. dsh/pi-only skip the check;
+        # hybrid runs both DSH and Hermes and is checked normally.
         if agent.get("runtime") in ("dsh", "pi"):
             is_dsh = agent.get("runtime") == "dsh"
             label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
@@ -506,7 +518,7 @@ def check_ct_liveness():
 def check_config_integrity():
     """Verify agent config.yaml parses as valid YAML."""
     for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no Hermes config.yaml.
+        # DSH/pi-only runtimes have no Hermes config.yaml; hybrid has both.
         if agent.get("runtime") == "dsh":
             print(f"  ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
             continue
@@ -519,11 +531,11 @@ def check_config_integrity():
             print(f"  ⬜ {name}: cannot SSH — skip config check")
             continue
 
+        home = get_user_home(user)
+
         # Check YAML parses
         yaml_ok = ssh(host,
-            "python3 -c "
-            '"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
-            "2>&1 || echo 'FAIL'",
+            f"python3 -c \"import yaml; yaml.safe_load(open('{home}/.hermes/config.yaml')); print('OK')\" 2>&1 || echo 'FAIL'",
             user=user)
         if not yaml_ok:
             print(f"  ❌ {name}: SSH UNREACHABLE (config check skipped)")
@@ -562,7 +574,7 @@ def _infisical_invocation_paths(wrapper_body):
 def check_wrapper_integrity():
     """Verify the hermes CLI wrapper exists and can reach hermes-real."""
     for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no hermes CLI wrapper.
+        # DSH/pi-only runtimes have no hermes CLI wrapper; hybrid has both.
         if agent.get("runtime") == "dsh":
             print(f"  ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
             continue
@@ -576,7 +588,8 @@ def check_wrapper_integrity():
             continue
 
         # Check wrapper exists
-        wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
+        home = get_user_home(user)
+        wrapper = ssh(host, f"ls -la {home}/.local/bin/hermes 2>/dev/null", user=user)
         if not wrapper:
             # Check alternate wrapper locations
             wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
@@ -599,7 +612,7 @@ def check_wrapper_integrity():
         # a removed path (litellm-api-keys.prose.md documents
         # `rm -f /usr/local/bin/infisical`) must neither produce a dangling path
         # nor trigger the PATH check — it is not an invocation.
-        wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
+        wrapper_body = ssh(host, f"cat {home}/.local/bin/hermes 2>/dev/null", user=user) or ""
         wrapper_code = "\n".join(line.split("#", 1)[0] for line in wrapper_body.splitlines())
         invoked_paths = _infisical_invocation_paths(wrapper_body)
         if "infisical" in wrapper_code:
@@ -633,24 +646,35 @@ def check_wrapper_integrity():
         else:
             print(f"  ℹ️  {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
 
-        # Check hermes-real exists
+        # Check that the wrapper's target resolves. The fleet's wrappers do NOT
+        # all use a hermes-real indirection — some exec the venv module directly.
+        # Verify the wrapper actually points to something runnable.
         hermes_real = ssh(host,
-            "ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
+            f"ls -la {home}/.local/bin/hermes-real 2>/dev/null || echo MISS",
             user=user)
-        if not hermes_real or hermes_real.strip() == "MISS":
-            # Check venv path
-            hermes_real = ssh(host,
-                "ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
+        if hermes_real and hermes_real.strip() != "MISS":
+            print(f"  ✅ {name}: wrapper shape: hermes-real at {home}/.local/bin/hermes-real")
+        else:
+            # Try the venv under home
+            venv_home = ssh(host,
+                f"test -x {home}/.hermes/hermes-agent/venv/bin/python && echo OK || echo MISS",
                 user=user)
-            if not hermes_real or hermes_real.strip() == "MISS":
-                print(f"  ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
-                _fail(f"wrapper-no-hermes-real:{name}", name)
+            if venv_home and venv_home.strip().splitlines()[-1] == "OK":
+                print(f"  ✅ {name}: wrapper shape: direct venv exec ({home}/.hermes/hermes-agent/venv/bin/python)")
             else:
-                print(f"  ✅ {name}: hermes-real at alt path")
+                # Try the system-wide venv
+                venv_sys = ssh(host,
+                    "test -x /usr/local/lib/hermes-agent/venv/bin/python && echo OK || echo MISS",
+                    user=user)
+                if venv_sys and venv_sys.strip().splitlines()[-1] == "OK":
+                    print(f"  ✅ {name}: wrapper shape: system venv (/usr/local/lib/hermes-agent/venv/bin/python)")
+                else:
+                    print(f"  ❌ {name}: wrapper target NOT RESOLVABLE (no hermes-real, no venv)")
+                    _fail(f"wrapper-no-hermes-real:{name}", name)
 
         # Check the .env file has the key
         env_has_key = ssh(host,
-            "grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
+            f"grep -c 'LITELLM_API_KEY' {home}/.hermes/.env 2>/dev/null || echo 0",
             user=user)
         if env_has_key and env_has_key.strip() not in ("", "0"):
             print(f"  ✅ {name}: wrapper + .env key present")

scripts/swap-gpu-dense-model.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
-# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
+# Run when download completes: ssh llmuser@192.168.68.8 'sudo bash -s' < this script
 #
 # Usage: bash swap-gpu-dense-model.sh
 # Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf