Merge branch 'master' into fix/agent-health-root-hardcoding-20260928

The fleet's wrappers do not all use a hermes-real indirection. Some (tanko,
mumuni) exec the venv module directly. This check now:
  1. Tries hermes-real at {home}/.local/bin (koonimo's shape)
  2. Tries the venv under {home}/.hermes/hermes-agent/venv (tanko/mumuni shape)
  3. Tries /usr/local/lib/hermes-agent/venv (legacy system-wide shape)

Each match is reported with which shape it matched, so a genuinely broken
wrapper is still a failure while a different-but-valid shape is not.

agent-health-check.prose.md

@@ -24,7 +24,7 @@ Runs every 4 hours (2, 6, 10, 14, 18, 22 UTC at :35) via cron (`35 2,6,10,14,18,
 ## Requires
 
 - **LiteLLM admin key** for key validation (retrieved from `/root/.pi/agent/env.sh`)
-- **SSH access** to GPU hosts (.8, .110, .15) and agent CTs (.122, .129, .114, .24)
+- **SSH access** to GPU hosts — `llmuser` on .8 (owns `llama-server`), `root` on .110 and .15 — and agent CTs (.122, .129, .114, .24)
 - **Python 3** for script execution
 - **Network access** to LiteLLM (:4000), GPU exporters (:9400), and gateway endpoints
 

docs/probe-drift-round2-evidence.md

@@ -1,5 +1,14 @@
 # Probe-drift round 2 — per-leg before/after evidence
 
+> **Historical record** — 2026-09-28: The lines below that describe tanko as
+> "DSH (DeepSeek Harness)" only reflect what the check reported when it was
+> running. Tanko's runtime was later found to be **hybrid (DSH + Hermes)** —
+> the check had a `/root/` hardcoding bug that made it probe the wrong home
+> directory and report `wrapper-missing:tanko` for an agent with a working
+> wrapper. This document records the observed output, not the underlying
+> truth; see `fix/agent-health-root-hardcoding-20260928` for the correction.
+
+
 **Date:** 2026-09-10
 **Worktree (absolute execution path):** `/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts`
 **Branch:** `fm/probe-drift-round2-20260909`

hermes-agent-baseline.prose.md

@@ -70,10 +70,10 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)
 
 ## Config Pattern — Mandatory Fields
 
-### For Hermes Agents (Mumuni, Koonimo)
+### For Hermes Agents (Mumuni, Koonimo, Tanko-hybrid)
 
 Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
-(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
+(Tanko is hybrid — runs both DSH and Hermes since 2026-08-27, so its Hermes config is also checked.)
 
 ### 1. Main Model
 ```yaml
@@ -297,7 +297,7 @@ Run the consolidated health check:
 ```bash
 python3 /root/scripts/agent-health-check.py
 ```
-This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
+This validates each agent's live LiteLLM key against the gateway, including tanko, which runs HYBRID (DSH + Hermes) since 2026-08-27; detects GPU port conflicts (ghost processes),
 verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
 and counts recent errors. Non-disruptive — never restarts anything.
 

hermes-zulip-plugin.prose.md

@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.
 
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — on DSH since 2026-08-27, no Hermes plugin) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27, no Hermes plugin) |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |
 
 ## Maintains
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
 
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
+| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(hybrid (DSH + Hermes) since 2026-08-27 — historical, plugin retired on this host)* |
 | Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
 
@@ -121,7 +121,7 @@ cp plugins/platforms/zulip/adapter.py \
    {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 
 # Fix ownership (was Tanko-only, runs as jerome user)
-# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (DSH).
+# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (hybrid: DSH + Hermes).
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/
 

hermes-zulip-restore.prose.md

@@ -24,7 +24,7 @@ gateway restart, and connection validation.
 
 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — DSH since 2026-08-27) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27) |
 
 ## Maintains
 
@@ -93,7 +93,7 @@ cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
    zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
    <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
 
-# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on DSH, no Hermes plugin)
+# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on hybrid (DSH + Hermes), no Hermes plugin)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only (historical)
 
 # Clean up

infrastructure-control.prose.md

@@ -682,7 +682,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | **.17** | Git | ❌ |
 | 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
-| 112 | tanko | minipve | .122 | DSH (DeepSeek Harness) agent | ✅ |
+| 112 | tanko | minipve | .122 | hybrid (DSH + Hermes) agent | ✅ |
 | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |

scripts/agent-health-check.py

@@ -50,6 +50,11 @@ Changelog:
     (kagentz CT 105 on minipve, .14, dedicated `hermes` user) and is monitored
     from her side. This script must not probe mumuni or .24 — the v2 changelog
     roster line was the last reference still placing her at .24 / CT100.
+  v6 (2026-09-28): .8 GPU health probe now runs as `llmuser` instead of `root`.
+    Root SSH to .8 was lost when the guest was rebuilt, so every .8 leg read as
+    UNREACHABLE for a healthy host. llmuser owns llama-server and can read
+    `systemctl is-active`, `systemctl show -p MainPID`, and the :8080 pid.
+    .110 and .15 keep the default `root` user.
 """
 
 import subprocess, json, sys, os, time, re, io, contextlib
@@ -70,7 +75,7 @@ PVE_NODES = {
 
 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "hybrid"},
     # abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
     # local env file (key_env below), not from the shared vault or .bashrc.
     # runtime=pi: abiba has run pi-only since the harness purge. There is no
@@ -95,7 +100,7 @@ AGENTS = {
 #   .110 rtx5070 (ocu-llm VM) -> llama-server.service    (active)
 #   .15  strixhalo (amdpve)   -> strix-server.service    (active)
 GPU_HOSTS = {
-    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-chat-api.service"},
+    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-chat-api.service", "user": "llmuser"},
     "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server.service"},
     "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "strix-server.service"},
 }
@@ -147,6 +152,18 @@ def ssh(host, cmd, user="root"):
     except:
         return None
 
+def get_user_home(user):
+    """Resolve the home directory for a user.
+    
+    For 'root', returns '/root'. For any other user, returns '/home/<user>'.
+    This is used to construct paths that reference a user's home directory
+    (e.g., ~/.local/bin/hermes, ~/.hermes/config.yaml) instead of hardcoding /root/.
+    """
+    if user == "root":
+        return "/root"
+    else:
+        return f"/home/{user}"
+
 def http_get(url, headers=None, timeout=5):
     """Return HTTP status code as string."""
     try:
@@ -300,13 +317,14 @@ def check_gpu_ports():
         host = gpu["host"]
         port = gpu["port"]
         svc = gpu["service"]
+        user = gpu.get("user", "root")  # default root, overridden per-host where needed
 
         # `systemctl is-active` exits non-zero when the unit is inactive or
         # missing, which the ssh() helper would swallow as an SSH failure and
         # report as UNREACHABLE. `|| true` keeps the real state word so we can
         # tell "unit inactive" from "host unreachable".
-        svc_status = ssh(host, f"systemctl is-active {svc} || true")
-        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
+        svc_status = ssh(host, f"systemctl is-active {svc} || true", user=user)
+        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1", user=user)
 
         if not svc_status:
             print(f"  ❌ {label}: UNREACHABLE")
@@ -317,14 +335,14 @@ def check_gpu_ports():
             print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
             FAIL.append(f"gpu-no-port:{label}")
         elif svc_status != "active":
-            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
+            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2", user=user)
             if svc_pid and port_owner != svc_pid:
                 print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
                 FAIL.append(f"gpu-ghost:{label}:{port_owner}")
             else:
                 print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
         else:
-            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
+            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health", user=user)
             if health and '"status":"ok"' in health:
                 print(f"  ✅ {label}: healthy (pid={port_owner})")
             elif health and '"status":"no slot available"' in health:
@@ -376,10 +394,10 @@ def check_agents():
         ct = agent["ct"]
         report_only = agent.get("report_only", False)
 
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
+        # Tanko runs hybrid (DSH + Hermes) since 2026-08-27 — it runs both DSH and Hermes gateway.
         # Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
-        # Non-Hermes runtimes have no gateway to probe. dsh = Tanko since
-        # 2026-08-27; pi = abiba since the harness purge (.24 is pi-only).
+        # Non-Hermes runtimes have no gateway to probe. dsh/pi-only skip the check;
+        # hybrid runs both DSH and Hermes and is checked normally.
         if agent.get("runtime") in ("dsh", "pi"):
             is_dsh = agent.get("runtime") == "dsh"
             label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
@@ -500,7 +518,7 @@ def check_ct_liveness():
 def check_config_integrity():
     """Verify agent config.yaml parses as valid YAML."""
     for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no Hermes config.yaml.
+        # DSH/pi-only runtimes have no Hermes config.yaml; hybrid has both.
         if agent.get("runtime") == "dsh":
             print(f"  ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
             continue
@@ -513,11 +531,11 @@ def check_config_integrity():
             print(f"  ⬜ {name}: cannot SSH — skip config check")
             continue
 
+        home = get_user_home(user)
+
         # Check YAML parses
         yaml_ok = ssh(host,
-            "python3 -c "
-            '"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
-            "2>&1 || echo 'FAIL'",
+            f"python3 -c \"import yaml; yaml.safe_load(open('{home}/.hermes/config.yaml')); print('OK')\" 2>&1 || echo 'FAIL'",
             user=user)
         if not yaml_ok:
             print(f"  ❌ {name}: SSH UNREACHABLE (config check skipped)")
@@ -556,7 +574,7 @@ def _infisical_invocation_paths(wrapper_body):
 def check_wrapper_integrity():
     """Verify the hermes CLI wrapper exists and can reach hermes-real."""
     for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no hermes CLI wrapper.
+        # DSH/pi-only runtimes have no hermes CLI wrapper; hybrid has both.
         if agent.get("runtime") == "dsh":
             print(f"  ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
             continue
@@ -570,7 +588,8 @@ def check_wrapper_integrity():
             continue
 
         # Check wrapper exists
-        wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
+        home = get_user_home(user)
+        wrapper = ssh(host, f"ls -la {home}/.local/bin/hermes 2>/dev/null", user=user)
         if not wrapper:
             # Check alternate wrapper locations
             wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
@@ -593,7 +612,7 @@ def check_wrapper_integrity():
         # a removed path (litellm-api-keys.prose.md documents
         # `rm -f /usr/local/bin/infisical`) must neither produce a dangling path
         # nor trigger the PATH check — it is not an invocation.
-        wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
+        wrapper_body = ssh(host, f"cat {home}/.local/bin/hermes 2>/dev/null", user=user) or ""
         wrapper_code = "\n".join(line.split("#", 1)[0] for line in wrapper_body.splitlines())
         invoked_paths = _infisical_invocation_paths(wrapper_body)
         if "infisical" in wrapper_code:
@@ -627,24 +646,35 @@ def check_wrapper_integrity():
         else:
             print(f"  ℹ️  {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
 
-        # Check hermes-real exists
+        # Check that the wrapper's target resolves. The fleet's wrappers do NOT
+        # all use a hermes-real indirection — some exec the venv module directly.
+        # Verify the wrapper actually points to something runnable.
         hermes_real = ssh(host,
-            "ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
+            f"ls -la {home}/.local/bin/hermes-real 2>/dev/null || echo MISS",
             user=user)
-        if not hermes_real or hermes_real.strip() == "MISS":
-            # Check venv path
-            hermes_real = ssh(host,
-                "ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
+        if hermes_real and hermes_real.strip() != "MISS":
+            print(f"  ✅ {name}: wrapper shape: hermes-real at {home}/.local/bin/hermes-real")
+        else:
+            # Try the venv under home
+            venv_home = ssh(host,
+                f"test -x {home}/.hermes/hermes-agent/venv/bin/python && echo OK || echo MISS",
                 user=user)
-            if not hermes_real or hermes_real.strip() == "MISS":
-                print(f"  ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
-                _fail(f"wrapper-no-hermes-real:{name}", name)
+            if venv_home and venv_home.strip().splitlines()[-1] == "OK":
+                print(f"  ✅ {name}: wrapper shape: direct venv exec ({home}/.hermes/hermes-agent/venv/bin/python)")
             else:
-                print(f"  ✅ {name}: hermes-real at alt path")
+                # Try the system-wide venv
+                venv_sys = ssh(host,
+                    "test -x /usr/local/lib/hermes-agent/venv/bin/python && echo OK || echo MISS",
+                    user=user)
+                if venv_sys and venv_sys.strip().splitlines()[-1] == "OK":
+                    print(f"  ✅ {name}: wrapper shape: system venv (/usr/local/lib/hermes-agent/venv/bin/python)")
+                else:
+                    print(f"  ❌ {name}: wrapper target NOT RESOLVABLE (no hermes-real, no venv)")
+                    _fail(f"wrapper-no-hermes-real:{name}", name)
 
         # Check the .env file has the key
         env_has_key = ssh(host,
-            "grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
+            f"grep -c 'LITELLM_API_KEY' {home}/.hermes/.env 2>/dev/null || echo 0",
             user=user)
         if env_has_key and env_has_key.strip() not in ("", "0"):
             print(f"  ✅ {name}: wrapper + .env key present")

scripts/swap-gpu-dense-model.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
-# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
+# Run when download completes: ssh llmuser@192.168.68.8 'sudo bash -s' < this script
 #
 # Usage: bash swap-gpu-dense-model.sh
 # Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf

tests/test_probe_drift.py

@@ -124,6 +124,39 @@ def test_tanko_ct112_is_probed_on_minipve(ahc, monkeypatch, capsys):
         ahc.REPORT_ONLY.clear()
 
 
+def test_gpu_rtx3090_probe_uses_llmuser_not_root(ahc, monkeypatch, capsys):
+    # 2026-09-28: root SSH to .8 was lost when the guest was rebuilt; llmuser
+    # owns llama-server and can read systemctl status and the :8080 pid. A root
+    # probe reads as UNREACHABLE for a healthy host (the reported bug). Execute
+    # check_gpu_ports() against an SSH boundary that only accepts llmuser@.8 and
+    # assert the .8 leg does not produce the false UNREACHABLE failure.
+    seen = []
+
+    def fake_ssh(host, cmd, user="root"):
+        seen.append((host, user))
+        if host == "192.168.68.8" and user != "llmuser":
+            return None  # root SSH denied -> baseline false UNREACHABLE
+        if cmd.startswith("systemctl is-active"):
+            return "active"
+        if cmd.startswith("ss -tlnp"):
+            return "48351"
+        if cmd.startswith("curl"):
+            return '{"status":"ok"}'
+        return None
+
+    monkeypatch.setattr(ahc, "ssh", fake_ssh)
+    ahc.FAIL.clear()
+    try:
+        ahc.check_gpu_ports()
+        out = capsys.readouterr().out
+        assert "gpu-unreachable:192.168.68.8" not in ahc.FAIL
+        assert "\u2705 gpu-rtx3090 (.8): healthy" in out
+        assert ("192.168.68.8", "llmuser") in seen
+        assert not any(host == "192.168.68.8" and user == "root" for host, user in seen)
+    finally:
+        ahc.FAIL.clear()
+
+
 def test_report_only_legs_never_count_as_failures(ahc):
     for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
         ahc.FAIL.clear()