Merge branch 'master' into fix/agent-health-root-hardcoding-20260928

fix(alignment): accept multiple wrapper shapes in hermes-real check
The fleet's wrappers do not all use a hermes-real indirection. Some (tanko, mumuni) exec the venv module directly. This check now: 1. Tries hermes-real at {home}/.local/bin (koonimo's shape) 2. Tries the venv under {home}/.hermes/hermes-agent/venv (tanko/mumuni shape) 3. Tries /usr/local/lib/hermes-agent/venv (legacy system-wide shape) Each match is reported with which shape it matched, so a genuinely broken wrapper is still a failure while a different-but-valid shape is not.
2026-09-28 23:03:53 +00:00 · 2026-09-28 22:23:08 +00:00 · 2026-09-28 21:20:42 +00:00 · 2026-09-28 20:48:49 +00:00 · 2026-09-28 12:59:50 +00:00 · 2026-09-28 12:57:00 +00:00
21 changed files with 449 additions and 102 deletions
@@ -24,7 +24,7 @@ Runs every 4 hours (2, 6, 10, 14, 18, 22 UTC at :35) via cron (`35 2,6,10,14,18,
 ## Requires

 - **LiteLLM admin key** for key validation (retrieved from `/root/.pi/agent/env.sh`)
- **SSH access** to GPU hosts (.8, .110, .15) and agent CTs (.122, .129, .114, .24)
+- **SSH access** to GPU hosts — `llmuser` on .8 (owns `llama-server`), `root` on .110 and .15 — and agent CTs (.122, .129, .114, .24)
 - **Python 3** for script execution
 - **Network access** to LiteLLM (:4000), GPU exporters (:9400), and gateway endpoints

@@ -94,7 +94,16 @@ def audit(path):
        cfg = yaml.safe_load(f)

    model = cfg.get("model", {})
-    fb = cfg.get("fallback_providers", {})
+    fb_raw = cfg.get("fallback_providers", {})
+    # Normalize: fallback_providers may be a dict (single provider) or a list of dicts
+    # (one entry per fallback). Both shapes are valid; we must handle both without crashing.
+    if isinstance(fb_raw, dict):
+        fb_entries = [fb_raw]
+    elif isinstance(fb_raw, list):
+        fb_entries = fb_raw
+    else:
+        fb_entries = [fb_raw]  # Let it fail the check below as malformed
+    fb = fb_entries[0] if fb_entries else {}
    comp = cfg.get("compression", {})
    aux = cfg.get("auxiliary", {})
    deleg = cfg.get("delegation", {})
@@ -207,22 +216,32 @@ def audit(path):
        "Rule 14",
        f"delegation.provider must be 'harness' (got {deleg.get('provider')!r})",
    )
-    check(
-        fb.get("provider") == "deepseek",
-        "Rule 14",
-        f"fallback_providers.provider must be 'deepseek' (got {fb.get('provider')!r}) — "
-        f"true fallback diversity, not same endpoint as primary",
-    )
-    check(
-        fb.get("model") == "deepseek-v4-flash",
-        "Rule 14",
-        f"fallback_providers.model must be 'deepseek-v4-flash' (got {fb.get('model')!r})",
-    )
-    check(
-        fb.get("api_key_env") == "DEEPSEEK_API_KEY",
-        "Rule 14",
-        f"fallback_providers.api_key_env must be DEEPSEEK_API_KEY (got {fb.get('api_key_env')!r})",
-    )
+    # Check each fallback entry. A malformed entry (not a mapping) is a VIOLATION, not a crash.
+    for idx, entry in enumerate(fb_entries):
+        prefix = f"fallback_providers[{idx}]"
+        if not isinstance(entry, dict):
+            check(
+                False,
+                "Rule 14",
+                f"{prefix} must be a mapping (got {type(entry).__name__})",
+            )
+            continue
+        check(
+            entry.get("provider") == "deepseek",
+            "Rule 14",
+            f"{prefix}.provider must be 'deepseek' (got {entry.get('provider')!r}) — "
+            f"true fallback diversity, not same endpoint as primary",
+        )
+        check(
+            entry.get("model") == "deepseek-v4-flash",
+            "Rule 14",
+            f"{prefix}.model must be 'deepseek-v4-flash' (got {entry.get('model')!r})",
+        )
+        check(
+            entry.get("api_key_env") == "DEEPSEEK_API_KEY",
+            "Rule 14",
+            f"{prefix}.api_key_env must be DEEPSEEK_API_KEY (got {entry.get('api_key_env')!r})",
+        )

    # --- custom_providers sanity ---
    check(
@@ -643,7 +643,7 @@ contracts:
    timeout: 120
    requires:
    - Zulip API key for abiba-bot@chat.sysloggh.net
-    - SSH access to amdpve (192.168.68.15) for Tanko (CT 112) and the Agent Zero Docker host (.14)
+    - SSH access to minipve (192.168.68.12) for Tanko (CT 112) and the Agent Zero Docker host (.14)
  verification:
    postconditions:
    - check: bot registration active
@@ -414,7 +414,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 | 108 | media | storepve | lxc | ✅ reachable |
 | 110 | gitea | minipve | lxc | ✅ reachable |
 | 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) |
-| 112 | tanko | amdpve | lxc | ✅ reachable |
+| 112 | tanko | minipve | lxc | ✅ reachable |
 | 113 | baggy | amdpve | lxc | ✅ reachable |
 | 115 | scottdenya | amdpve | lxc | ✅ reachable |
 | 116 | syslog-api | minipve | lxc | ✅ reachable |
@@ -1,5 +1,14 @@
 # Probe-drift round 2 — per-leg before/after evidence

+> **Historical record** — 2026-09-28: The lines below that describe tanko as
+> "DSH (DeepSeek Harness)" only reflect what the check reported when it was
+> running. Tanko's runtime was later found to be **hybrid (DSH + Hermes)** —
+> the check had a `/root/` hardcoding bug that made it probe the wrong home
+> directory and report `wrapper-missing:tanko` for an agent with a working
+> wrapper. This document records the observed output, not the underlying
+> truth; see `fix/agent-health-root-hardcoding-20260928` for the correction.
+
+
 **Date:** 2026-09-10
 **Worktree (absolute execution path):** `/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts`
 **Branch:** `fm/probe-drift-round2-20260909`
@@ -70,10 +70,10 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)

 ## Config Pattern — Mandatory Fields

-### For Hermes Agents (Mumuni, Koonimo)
+### For Hermes Agents (Mumuni, Koonimo, Tanko-hybrid)

 Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
-(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
+(Tanko is hybrid — runs both DSH and Hermes since 2026-08-27, so its Hermes config is also checked.)

 ### 1. Main Model
 ```yaml
@@ -297,7 +297,7 @@ Run the consolidated health check:
 ```bash
 python3 /root/scripts/agent-health-check.py
 ```
-This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
+This validates each agent's live LiteLLM key against the gateway, including tanko, which runs HYBRID (DSH + Hermes) since 2026-08-27; detects GPU port conflicts (ghost processes),
 verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
 and counts recent errors. Non-disruptive — never restarts anything.

@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.

 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — on DSH since 2026-08-27, no Hermes plugin) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27, no Hermes plugin) |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |

 ## Maintains
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.

 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
+| Tanko | CT112 | minipve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(hybrid (DSH + Hermes) since 2026-08-27 — historical, plugin retired on this host)* |
 | Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |

@@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation.
 ### Step 1: Resolve Target

 Map `target` to host, CT ID, hermes_home, and user from the live-state table.
-For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
+For CT112 route through `ssh root@minipve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.

 ### Step 2: Pull Latest Plugin Source

@@ -121,7 +121,7 @@ cp plugins/platforms/zulip/adapter.py \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/

 # Fix ownership (was Tanko-only, runs as jerome user)
-# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (DSH).
+# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (hybrid: DSH + Hermes).
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/

@@ -24,7 +24,7 @@ gateway restart, and connection validation.

 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — DSH since 2026-08-27) |
+| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — hybrid (DSH + Hermes) since 2026-08-27) |

 ## Maintains

@@ -67,7 +67,7 @@ gateway restart, and connection validation.
 ### Step 1: Locate Target

 Map `target` to connectivity parameters from the live-state table above.
-For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
+For CT112 route through `ssh root@minipve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.

 ### Step 2: Deploy Zulip Adapter

@@ -93,7 +93,7 @@ cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/

-# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on DSH, no Hermes plugin)
+# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on hybrid (DSH + Hermes), no Hermes plugin)
 chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only (historical)

 # Clean up
@@ -105,8 +105,8 @@ description: >

 | Node | IP | CPU | RAM | VMs/CTs | Role |
 |------|----|-----|-----|---------|------|
-| minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
-| amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute |
+| minipve | .12 | 16C | 30GB | abiba, tanko, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
+| amdpve | .15 | 32C | 62GB | kagentz, baggy, scottdenya, adguard2 | Agents, compute |
 | storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat |
 | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
 | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
@@ -682,7 +682,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | **.17** | Git | ❌ |
 | 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
-| 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ |
+| 112 | tanko | minipve | .122 | hybrid (DSH + Hermes) agent | ✅ |
 | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
@@ -712,7 +712,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
 | 100 | abiba | minipve | `pct-run 100` |
 | 105 | kagentz | amdpve | `pct-run 105` |
 | 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) |
-| 112 | tanko | amdpve | `pct-run 112` |
+| 112 | tanko | minipve | `pct-run 112` |
 | 113 | baggy | amdpve | `pct-run 113` |
 | 115 | scottdenya | amdpve | `pct-run 115` |
 | 104 | authentik | minipve | `pct-run 104` |
@@ -59,7 +59,7 @@ Before ANY update wave:
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
 | CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
-| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
+| CT 112 (tanko, minipve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
 | VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
 | VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
@@ -50,6 +50,11 @@ Changelog:
    (kagentz CT 105 on minipve, .14, dedicated `hermes` user) and is monitored
    from her side. This script must not probe mumuni or .24 — the v2 changelog
    roster line was the last reference still placing her at .24 / CT100.
+  v6 (2026-09-28): .8 GPU health probe now runs as `llmuser` instead of `root`.
+    Root SSH to .8 was lost when the guest was rebuilt, so every .8 leg read as
+    UNREACHABLE for a healthy host. llmuser owns llama-server and can read
+    `systemctl is-active`, `systemctl show -p MainPID`, and the :8080 pid.
+    .110 and .15 keep the default `root` user.
 """

 import subprocess, json, sys, os, time, re, io, contextlib
@@ -70,7 +75,7 @@ PVE_NODES = {

 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve",  "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "minipve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "hybrid"},
    # abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
    # local env file (key_env below), not from the shared vault or .bashrc.
    # runtime=pi: abiba has run pi-only since the harness purge. There is no
@@ -95,7 +100,7 @@ AGENTS = {
 #   .110 rtx5070 (ocu-llm VM) -> llama-server.service    (active)
 #   .15  strixhalo (amdpve)   -> strix-server.service    (active)
 GPU_HOSTS = {
-    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-chat-api.service"},
+    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-chat-api.service", "user": "llmuser"},
    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server.service"},
    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "strix-server.service"},
 }
@@ -147,6 +152,18 @@ def ssh(host, cmd, user="root"):
    except:
        return None

+def get_user_home(user):
+    """Resolve the home directory for a user.
+    
+    For 'root', returns '/root'. For any other user, returns '/home/<user>'.
+    This is used to construct paths that reference a user's home directory
+    (e.g., ~/.local/bin/hermes, ~/.hermes/config.yaml) instead of hardcoding /root/.
+    """
+    if user == "root":
+        return "/root"
+    else:
+        return f"/home/{user}"
+
 def http_get(url, headers=None, timeout=5):
    """Return HTTP status code as string."""
    try:
@@ -300,13 +317,14 @@ def check_gpu_ports():
        host = gpu["host"]
        port = gpu["port"]
        svc = gpu["service"]
+        user = gpu.get("user", "root")  # default root, overridden per-host where needed

        # `systemctl is-active` exits non-zero when the unit is inactive or
        # missing, which the ssh() helper would swallow as an SSH failure and
        # report as UNREACHABLE. `|| true` keeps the real state word so we can
        # tell "unit inactive" from "host unreachable".
-        svc_status = ssh(host, f"systemctl is-active {svc} || true")
-        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
+        svc_status = ssh(host, f"systemctl is-active {svc} || true", user=user)
+        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1", user=user)

        if not svc_status:
            print(f"  ❌ {label}: UNREACHABLE")
@@ -317,14 +335,14 @@ def check_gpu_ports():
            print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
            FAIL.append(f"gpu-no-port:{label}")
        elif svc_status != "active":
-            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
+            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2", user=user)
            if svc_pid and port_owner != svc_pid:
                print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
                FAIL.append(f"gpu-ghost:{label}:{port_owner}")
            else:
                print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
        else:
-            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
+            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health", user=user)
            if health and '"status":"ok"' in health:
                print(f"  ✅ {label}: healthy (pid={port_owner})")
            elif health and '"status":"no slot available"' in health:
@@ -376,10 +394,10 @@ def check_agents():
        ct = agent["ct"]
        report_only = agent.get("report_only", False)

-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
+        # Tanko runs hybrid (DSH + Hermes) since 2026-08-27 — it runs both DSH and Hermes gateway.
        # Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
-        # Non-Hermes runtimes have no gateway to probe. dsh = Tanko since
-        # 2026-08-27; pi = abiba since the harness purge (.24 is pi-only).
+        # Non-Hermes runtimes have no gateway to probe. dsh/pi-only skip the check;
+        # hybrid runs both DSH and Hermes and is checked normally.
        if agent.get("runtime") in ("dsh", "pi"):
            is_dsh = agent.get("runtime") == "dsh"
            label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
@@ -500,7 +518,7 @@ def check_ct_liveness():
 def check_config_integrity():
    """Verify agent config.yaml parses as valid YAML."""
    for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no Hermes config.yaml.
+        # DSH/pi-only runtimes have no Hermes config.yaml; hybrid has both.
        if agent.get("runtime") == "dsh":
            print(f"  ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
            continue
@@ -513,11 +531,11 @@ def check_config_integrity():
            print(f"  ⬜ {name}: cannot SSH — skip config check")
            continue

+        home = get_user_home(user)
+
        # Check YAML parses
        yaml_ok = ssh(host,
-            "python3 -c "
-            '"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
-            "2>&1 || echo 'FAIL'",
+            f"python3 -c \"import yaml; yaml.safe_load(open('{home}/.hermes/config.yaml')); print('OK')\" 2>&1 || echo 'FAIL'",
            user=user)
        if not yaml_ok:
            print(f"  ❌ {name}: SSH UNREACHABLE (config check skipped)")
@@ -556,7 +574,7 @@ def _infisical_invocation_paths(wrapper_body):
 def check_wrapper_integrity():
    """Verify the hermes CLI wrapper exists and can reach hermes-real."""
    for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no hermes CLI wrapper.
+        # DSH/pi-only runtimes have no hermes CLI wrapper; hybrid has both.
        if agent.get("runtime") == "dsh":
            print(f"  ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
            continue
@@ -570,7 +588,8 @@ def check_wrapper_integrity():
            continue

        # Check wrapper exists
-        wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
+        home = get_user_home(user)
+        wrapper = ssh(host, f"ls -la {home}/.local/bin/hermes 2>/dev/null", user=user)
        if not wrapper:
            # Check alternate wrapper locations
            wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
@@ -593,7 +612,7 @@ def check_wrapper_integrity():
        # a removed path (litellm-api-keys.prose.md documents
        # `rm -f /usr/local/bin/infisical`) must neither produce a dangling path
        # nor trigger the PATH check — it is not an invocation.
-        wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
+        wrapper_body = ssh(host, f"cat {home}/.local/bin/hermes 2>/dev/null", user=user) or ""
        wrapper_code = "\n".join(line.split("#", 1)[0] for line in wrapper_body.splitlines())
        invoked_paths = _infisical_invocation_paths(wrapper_body)
        if "infisical" in wrapper_code:
@@ -627,24 +646,35 @@ def check_wrapper_integrity():
        else:
            print(f"  ℹ️  {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")

-        # Check hermes-real exists
+        # Check that the wrapper's target resolves. The fleet's wrappers do NOT
+        # all use a hermes-real indirection — some exec the venv module directly.
+        # Verify the wrapper actually points to something runnable.
        hermes_real = ssh(host,
-            "ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
+            f"ls -la {home}/.local/bin/hermes-real 2>/dev/null || echo MISS",
            user=user)
-        if not hermes_real or hermes_real.strip() == "MISS":
-            # Check venv path
-            hermes_real = ssh(host,
-                "ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
+        if hermes_real and hermes_real.strip() != "MISS":
+            print(f"  ✅ {name}: wrapper shape: hermes-real at {home}/.local/bin/hermes-real")
+        else:
+            # Try the venv under home
+            venv_home = ssh(host,
+                f"test -x {home}/.hermes/hermes-agent/venv/bin/python && echo OK || echo MISS",
                user=user)
-            if not hermes_real or hermes_real.strip() == "MISS":
-                print(f"  ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
-                _fail(f"wrapper-no-hermes-real:{name}", name)
+            if venv_home and venv_home.strip().splitlines()[-1] == "OK":
+                print(f"  ✅ {name}: wrapper shape: direct venv exec ({home}/.hermes/hermes-agent/venv/bin/python)")
            else:
-                print(f"  ✅ {name}: hermes-real at alt path")
+                # Try the system-wide venv
+                venv_sys = ssh(host,
+                    "test -x /usr/local/lib/hermes-agent/venv/bin/python && echo OK || echo MISS",
+                    user=user)
+                if venv_sys and venv_sys.strip().splitlines()[-1] == "OK":
+                    print(f"  ✅ {name}: wrapper shape: system venv (/usr/local/lib/hermes-agent/venv/bin/python)")
+                else:
+                    print(f"  ❌ {name}: wrapper target NOT RESOLVABLE (no hermes-real, no venv)")
+                    _fail(f"wrapper-no-hermes-real:{name}", name)

        # Check the .env file has the key
        env_has_key = ssh(host,
-            "grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
+            f"grep -c 'LITELLM_API_KEY' {home}/.hermes/.env 2>/dev/null || echo 0",
            user=user)
        if env_has_key and env_has_key.strip() not in ("", "0"):
            print(f"  ✅ {name}: wrapper + .env key present")
@@ -101,8 +101,6 @@ GUESTS: list[Guest] = [
    # amdpve (192.168.68.15)
    Guest(ct_id="105", hostname="kagentz", ip="192.168.68.105", node="amdpve",
          access_method="ssh-host", probe_target="kagentz (CT 105, amdpve)"),
-    Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="amdpve",
-          access_method="pct-run", probe_target="tanko (CT 112, amdpve)"),
    Guest(ct_id="113", hostname="baggy", ip="192.168.68.113", node="amdpve",
          access_method="pct-run", probe_target="baggy (CT 113, amdpve)"),
    Guest(ct_id="115", hostname="scottdenya", ip="192.168.68.115", node="amdpve",
@@ -110,6 +108,8 @@ GUESTS: list[Guest] = [
    Guest(ct_id="120", hostname="adguard2", ip="192.168.68.120", node="amdpve",
          access_method="pct-run", probe_target="adguard2 (CT 120, amdpve)"),
    # minipve (192.168.68.12)
+    Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="minipve",
+          access_method="pct-run", probe_target="tanko (CT 112, minipve)"),
    Guest(ct_id="100", hostname="abiba", ip="192.168.68.100", node="minipve",
          access_method="pct-run", probe_target="abiba (CT 100, minipve)"),
    Guest(ct_id="102", hostname="adguard", ip="192.168.68.102", node="minipve",
@@ -12,7 +12,6 @@ set -euo pipefail
 declare -A CT_NODES=(
    # amdpve (192.168.68.15)
    [105]=amdpve    # kagentz (was hwepve — corrected 2026-09-12; live per pvesh)
-    [112]=amdpve    # tanko
    [113]=amdpve    # baggy
    [115]=amdpve    # scottdenya
    [120]=amdpve    # adguard2 (added 2026-09-12)
@@ -21,6 +20,7 @@ declare -A CT_NODES=(
    [102]=minipve   # adguard (was acerpve)
    [104]=minipve   # authentik
    [110]=minipve   # gitea
+    [112]=minipve   # tanko (was amdpve — migrated 2026-09-27; live per pvesh)
    [116]=minipve   # syslog-api
    [119]=minipve   # infisical-vault
    # storepve (192.168.68.6)
@@ -47,8 +47,8 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
 The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:

 **Proxmox Cluster "Tabiri" (5 nodes):**
- amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2
- minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault
+- amdpve (192.168.68.15): kagentz, baggy, scottdenya, adguard2
+- minipve (192.168.68.12): abiba, tanko, adguard, authentik, gitea, syslog-api, infisical-vault
 - storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna
 - acerpve (192.168.68.9): llm-gpu
 - ocupve (192.168.68.5): ocu-llm
@@ -1,6 +1,6 @@
 #!/bin/bash
 # swap-gpu-dense-model.sh — Swap RTX 3090 from qwen3.6-27B-code to SmartCode-Fable-5
-# Run when download completes: ssh root@192.168.68.8 'bash -s' < this script
+# Run when download completes: ssh llmuser@192.168.68.8 'sudo bash -s' < this script
 #
 # Usage: bash swap-gpu-dense-model.sh
 # Requires: new model at /home/llmuser/models/SmartCode-Fable-5-27B-UD-Q4_K_XL.gguf
@@ -135,16 +135,16 @@ case "$PI_VERDICT" in
 esac
 # -- abiba-leg-end

-# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
+# ── Platform B: Tanko (DSH dsh-web on minipve CT 112) ──
 # Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
-# key availability varies — so probes run from the amdpve vantage via `pct exec`.
-# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
-# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
+# key availability varies — so probes run from the minipve vantage via `pct exec`.
+# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on minipve
+# (192.168.68.12). The gateway binds 127.0.0.1:3080 loopback-only by design — a
 # remote :3080 probe is refused and is NOT a fault.
-TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
+TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.12 \
  "pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
 [ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
-TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
+TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.12 \
  "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
 [ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"

@@ -0,0 +1,236 @@
+"""Regression test for the fallback_providers list-shape crash in audit-hermes-config.py.
+
+WHY THIS FILE EXISTS: audit-hermes-config.py assumed `fallback_providers` was always a dict
+(single provider). Two live agents (koby, koonimo) carry it as a LIST of dicts (one entry per
+fallback), so the script crashed with:
+
+    File "audit-hermes-config.py", line 211, in audit
+        fb.get("provider") == "deepseek",
+    AttributeError: 'list' object has no attribute 'get'
+
+Both are REAL agent configs, so this is not a malformed-input case — the script simply could not
+audit two of the four agents it exists to audit. Until fixed, the key-hygiene check had no
+coverage for half the fleet while appearing to run.
+
+These tests execute the real CLI (`python3 audit-hermes-config.py <config>`) and assert:
+1. A config whose `fallback_providers` is a LIST of valid dicts does NOT crash (exit code is 0 or 1,
+   never a traceback/AttributeError).
+2. A config whose `fallback_providers` contains a MALFORMED entry (a list element that is not a
+   mapping) reports a VIOLATION naming the offending entry, NOT an uncaught exception.
+3. The dict shape still works (existing tests must stay green).
+
+No network, vault, or SSH access is required.
+"""
+from __future__ import annotations
+
+import pathlib
+import subprocess
+import sys
+
+ROOT = pathlib.Path(__file__).resolve().parent.parent
+AUDIT = ROOT / "audit-hermes-config.py"
+
+# A valid config where fallback_providers is a LIST of dicts (the real koby/koonimo shape).
+# One entry, well-formed: provider=deepseek, model=deepseek-v4-flash, api_key_env=DEEPSEEK_API_KEY.
+# This must produce a real verdict (PASS or FAIL) without crashing.
+LIST_SHAPE_VALID = """
+model:
+  api_key: ""
+  api_key_env: LITELLM_API_KEY
+  base_url: http://192.168.68.116/litellm/v1
+  max_tokens: 4096
+  default: syslog-auto
+  provider: harness
+fallback_providers:
+  - provider: deepseek
+    model: deepseek-v4-flash
+    api_key_env: DEEPSEEK_API_KEY
+compression:
+  model: syslog-auto
+  provider: harness
+  threshold: 0.65
+  max_context_window: 131072
+auxiliary:
+  vision:
+    model: gpu-vision
+    provider: harness
+  web_extract:
+    model: gpu-vision
+    provider: harness
+  compression:
+    model: syslog-auto
+    provider: harness
+delegation:
+  provider: harness
+custom_providers:
+  - name: harness
+    key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/litellm/v1
+"""
+
+# A valid config where fallback_providers is a LIST with TWO entries (multiple fallbacks).
+# Both entries well-formed. Must not crash and should produce a real verdict.
+LIST_SHAPE_MULTI = """
+model:
+  api_key: ""
+  api_key_env: LITELLM_API_KEY
+  base_url: http://192.168.68.116/litellm/v1
+  max_tokens: 4096
+  default: syslog-auto
+  provider: harness
+fallback_providers:
+  - provider: deepseek
+    model: deepseek-v4-flash
+    api_key_env: DEEPSEEK_API_KEY
+  - provider: deepseek
+    model: deepseek-v4-flash
+    api_key_env: DEEPSEEK_API_KEY
+compression:
+  model: syslog-auto
+  provider: harness
+  threshold: 0.65
+  max_context_window: 131072
+auxiliary:
+  vision:
+    model: gpu-vision
+    provider: harness
+  web_extract:
+    model: gpu-vision
+    provider: harness
+  compression:
+    model: syslog-auto
+    provider: harness
+delegation:
+  provider: harness
+custom_providers:
+  - name: harness
+    key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/litellm/v1
+"""
+
+# A config where fallback_providers is a LIST containing a MALFORMED entry:
+# one element is a plain string, not a mapping. The checker must report a VIOLATION
+# naming the offending entry (fallback_providers[1]) and NOT crash.
+LIST_SHAPE_MALFORMED = """
+model:
+  api_key: ""
+  api_key_env: LITELLM_API_KEY
+  base_url: http://192.168.68.116/litellm/v1
+  max_tokens: 4096
+  default: syslog-auto
+  provider: harness
+fallback_providers:
+  - provider: deepseek
+    model: deepseek-v4-flash
+    api_key_env: DEEPSEEK_API_KEY
+  - "not-a-mapping"
+compression:
+  model: syslog-auto
+  provider: harness
+  threshold: 0.65
+  max_context_window: 131072
+auxiliary:
+  vision:
+    model: gpu-vision
+    provider: harness
+  web_extract:
+    model: gpu-vision
+    provider: harness
+  compression:
+    model: syslog-auto
+    provider: harness
+delegation:
+  provider: harness
+custom_providers:
+  - name: harness
+    key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/litellm/v1
+"""
+
+# The original DICT shape (single provider) must still work — existing behaviour preserved.
+DICT_SHAPE_VALID = """
+model:
+  api_key: ""
+  api_key_env: LITELLM_API_KEY
+  base_url: http://192.168.68.116/litellm/v1
+  max_tokens: 4096
+  default: syslog-auto
+  provider: harness
+fallback_providers:
+  provider: deepseek
+  model: deepseek-v4-flash
+  api_key_env: DEEPSEEK_API_KEY
+compression:
+  model: syslog-auto
+  provider: harness
+  threshold: 0.65
+  max_context_window: 131072
+auxiliary:
+  vision:
+    model: gpu-vision
+    provider: harness
+  web_extract:
+    model: gpu-vision
+    provider: harness
+  compression:
+    model: syslog-auto
+    provider: harness
+delegation:
+  provider: harness
+custom_providers:
+  - name: harness
+    key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/litellm/v1
+"""
+
+
+def _run_config(tmp_path, name, text):
+    cfg = tmp_path / name
+    cfg.write_text(text)
+    proc = subprocess.run(
+        [sys.executable, str(AUDIT), str(cfg)],
+        capture_output=True, text=True,
+    )
+    return proc.returncode, proc.stdout, proc.stderr
+
+
+def test_list_shape_single_entry_does_not_crash(tmp_path):
+    """A LIST with one valid dict must not raise AttributeError; exit 0 (PASS)."""
+    code, out, err = _run_config(tmp_path, "list-single.yaml", LIST_SHAPE_VALID)
+    # Must NOT be a crash (traceback). A clean run exits 0 (PASS) or 1 (FAIL), never 2+ (exception).
+    assert code in (0, 1), f"Expected clean exit 0 or 1, got {code}\nSTDOUT:\n{out}\nSTDERR:\n{err}"
+    assert "AttributeError" not in err, f"Crashed with AttributeError:\n{err}"
+    assert "Traceback" not in err, f"Crashed with uncaught exception:\n{err}"
+    # The valid single-entry list should PASS (all rules satisfied).
+    assert code == 0, f"Expected PASS but got {code}\n{out}"
+    assert "RESULT: PASS" in out
+
+
+def test_list_shape_multiple_entries_does_not_crash(tmp_path):
+    """A LIST with two valid dicts must not raise AttributeError; exit 0 (PASS)."""
+    code, out, err = _run_config(tmp_path, "list-multi.yaml", LIST_SHAPE_MULTI)
+    assert code in (0, 1), f"Expected clean exit 0 or 1, got {code}\nSTDOUT:\n{out}\nSTDERR:\n{err}"
+    assert "AttributeError" not in err, f"Crashed with AttributeError:\n{err}"
+    assert "Traceback" not in err, f"Crashed with uncaught exception:\n{err}"
+    assert code == 0, f"Expected PASS but got {code}\n{out}"
+    assert "RESULT: PASS" in out
+
+
+def test_list_shape_malformed_entry_reports_violation_not_crash(tmp_path):
+    """A LIST containing a non-mapping element must be a reported VIOLATION, not a crash."""
+    code, out, err = _run_config(tmp_path, "list-malformed.yaml", LIST_SHAPE_MALFORMED)
+    # Must NOT be a crash.
+    assert "AttributeError" not in err, f"Crashed with AttributeError:\n{err}"
+    assert "Traceback" not in err, f"Crashed with uncaught exception:\n{err}"
+    # Should be a FAIL (exit 1) because the malformed entry is a violation.
+    assert code == 1, f"Expected FAIL (exit 1) but got {code}\n{out}"
+    assert "RESULT: FAIL" in out
+    # The violation must name the offending entry (fallback_providers[1]).
+    assert "fallback_providers[1]" in out, f"Violation did not name the offending entry:\n{out}"
+
+
+def test_dict_shape_still_passes(tmp_path):
+    """The original DICT shape (single provider) must still PASS — existing behaviour preserved."""
+    code, out, err = _run_config(tmp_path, "dict-valid.yaml", DICT_SHAPE_VALID)
+    assert code == 0, f"Expected PASS but got {code}\n{out}\nSTDERR:\n{err}"
+    assert "RESULT: PASS" in out
@@ -49,7 +49,7 @@ HEALTH_CONTRACT = ROOT / "zulip-health.prose.md"
 CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"

 MUMUNI_IP = "192.168.68.24"       # Mumuni's old (decommissioned) deployment
-TANKO_VANTAGE = "192.168.68.15"   # amdpve — Tanko CT 112 via pct exec
+TANKO_VANTAGE = "192.168.68.12"   # minipve — Tanko CT 112 via pct exec
 AGENT_ZERO_HOST = "192.168.68.14"  # kagentz host, Agent Zero docker


@@ -82,7 +82,7 @@ done
 printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
 cmd="${*: -1}"
 case "$host" in
-  192.168.68.15)
+  192.168.68.12)
    case "$cmd" in
      *"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
      *curl*)                  printf '%s' "$TANKO_HTTP" ;;
@@ -104,6 +104,59 @@ def test_koby_ct111_is_on_storepve(ahc):
    assert ahc.AGENTS["koby"]["pve"] == "storepve"


+def test_tanko_ct112_is_probed_on_minipve(ahc, monkeypatch, capsys):
+    # CT 112 (tanko) was live-migrated to minipve (.12) on 2026-09-27; the
+    # amdpve mapping made `pct status 112` fail and read as ct-unreachable.
+    # Execute the probe and assert the host the script actually contacts.
+    probes = []
+    monkeypatch.setattr(
+        ahc, "ssh",
+        lambda host, cmd, user="root": probes.append((host, cmd)) or "status: running",
+    )
+    ahc.FAIL.clear()
+    ahc.REPORT_ONLY.clear()
+    try:
+        ahc.check_ct_liveness()
+        tanko_hosts = [h for h, cmd in probes if cmd == "pct status 112 2>/dev/null"]
+        assert tanko_hosts == ["192.168.68.12"]
+    finally:
+        ahc.FAIL.clear()
+        ahc.REPORT_ONLY.clear()
+
+
+def test_gpu_rtx3090_probe_uses_llmuser_not_root(ahc, monkeypatch, capsys):
+    # 2026-09-28: root SSH to .8 was lost when the guest was rebuilt; llmuser
+    # owns llama-server and can read systemctl status and the :8080 pid. A root
+    # probe reads as UNREACHABLE for a healthy host (the reported bug). Execute
+    # check_gpu_ports() against an SSH boundary that only accepts llmuser@.8 and
+    # assert the .8 leg does not produce the false UNREACHABLE failure.
+    seen = []
+
+    def fake_ssh(host, cmd, user="root"):
+        seen.append((host, user))
+        if host == "192.168.68.8" and user != "llmuser":
+            return None  # root SSH denied -> baseline false UNREACHABLE
+        if cmd.startswith("systemctl is-active"):
+            return "active"
+        if cmd.startswith("ss -tlnp"):
+            return "48351"
+        if cmd.startswith("curl"):
+            return '{"status":"ok"}'
+        return None
+
+    monkeypatch.setattr(ahc, "ssh", fake_ssh)
+    ahc.FAIL.clear()
+    try:
+        ahc.check_gpu_ports()
+        out = capsys.readouterr().out
+        assert "gpu-unreachable:192.168.68.8" not in ahc.FAIL
+        assert "\u2705 gpu-rtx3090 (.8): healthy" in out
+        assert ("192.168.68.8", "llmuser") in seen
+        assert not any(host == "192.168.68.8" and user == "root" for host, user in seen)
+    finally:
+        ahc.FAIL.clear()
+
+
 def test_report_only_legs_never_count_as_failures(ahc):
    for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
        ahc.FAIL.clear()
@@ -34,7 +34,7 @@ ROOT = pathlib.Path(__file__).resolve().parents[1]
 ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
 CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"

-TANKO_VANTAGE = "192.168.68.15"   # amdpve — Tanko CT 112 via pct exec
+TANKO_VANTAGE = "192.168.68.12"   # minipve — Tanko CT 112 via pct exec
 AGENT_ZERO_HOST = "192.168.68.14"  # kagentz host, Agent Zero docker


@@ -52,7 +52,7 @@ done
 printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
 cmd="${*: -1}"
 case "$host" in
-  192.168.68.15)
+  192.168.68.12)
    case "$cmd" in
      *"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
      *curl*)                  printf '%s' "$TANKO_HTTP" ;;
@@ -28,7 +28,7 @@ session start.
 ## Requires

 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
+- **SSH access** to minipve (192.168.68.12) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `kagentz.sysloggh.net` (C3 public path), `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -228,20 +228,20 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | Crash loop >10/h | Alert user |


-### Step 3: Platform B — Tanko (DSH on amdpve CT 112)
+### Step 3: Platform B — Tanko (DSH on minipve CT 112)

 Mumuni is out of scope for this host (see the note above): she runs on her own
 container and is monitored on her side.

 Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
 there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
-as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
-PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
+as the `dsh-web` systemd unit inside **CT 112**, which resides on the **minipve**
+PVE host (**192.168.68.12**). Direct SSH to 192.168.68.122 is not a dependency
 of this contract — per-worker key availability varies — so CT 112 probes run
-from the amdpve vantage via `pct exec`:
+from the minipve vantage via `pct exec`:

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- <command>"
+ssh root@192.168.68.12 "pct exec 112 -- <command>"
 ```

 > **By design (verified 2026-09-08):** the `dsh-web` gateway binds
@@ -253,7 +253,7 @@ ssh root@192.168.68.15 "pct exec 112 -- <command>"
 **B1: Gateway Service State (Tanko)**

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
+ssh root@192.168.68.12 "pct exec 112 -- systemctl is-active dsh-web"
 ```

 Expected: `active`. Anything else → gateway service down → apply the Tanko heal
@@ -262,7 +262,7 @@ Expected: `active`. Anything else → gateway service down → apply the Tanko h
 **B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
+ssh root@192.168.68.12 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
 ```

 Alive = **ANY** HTTP status response from the endpoint — the expected set is
@@ -273,13 +273,13 @@ process answering `503` is running and self-heal must NOT restart-loop it.
 Down = connection refused (`000`) or timeout only. Statuses outside the
 expected set are logged/reported as a warning — reported, never healed on.

-**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
+**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to minipve)**

 ```bash
 curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
 ```

-Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
+Fallback only — used when the monitoring node has no pct/SSH path to minipve.
 Alive = **ANY** HTTP status response from the endpoint — healthy signals are
 `302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
 status, including `404`/`5xx`, also counts alive: the endpoint is up and
@@ -404,29 +404,29 @@ ExecStartPost=/bin/systemctl --no-block start dsh-web-token.service
 4. Every later request through `/` presents that cookie; the token is not needed
   again until the cookie expires or a new browser is used.

-**Verification** (amdpve vantage):
+**Verification** (minipve vantage):
 ```bash
 # 1. Login endpoint is Authentik-gated: unauthenticated -> 302 (not 200/303).
-ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \
+ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1/dsh-web-login"
 # Expected: 302

 # 2. Legacy :8081 endpoint is gone (connection refused -> 000).
-ssh root@192.168.68.15 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
+ssh root@192.168.68.12 "pct exec 112 -- curl -s --max-time 3 -o /dev/null \
  -w '%{http_code}\n' http://192.168.68.122:8081/"
 # Expected: 000

 # 3. Backend cookie mint + reuse (exactly what /dsh-web-login proxies to).
-TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- cat /etc/dsh-web/launch-token")
-ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
+TOKEN=$(ssh root@192.168.68.12 "pct exec 112 -- cat /etc/dsh-web/launch-token")
+ssh root@192.168.68.12 "pct exec 112 -- curl -s -c /tmp/dsh.jar -o /dev/null \
  -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
-ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
+ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
  -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
 # Expected: 200 — the minted dsh-auth-... cookie (authority
 # tankodhs.sysloggh.net) is replayed on the next request and accepted.

 # 4. Token refresh is non-disruptive and idempotent.
-ssh root@192.168.68.15 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh"
+ssh root@192.168.68.12 "pct exec 112 -- /opt/deepseek-harness/capture-dsh-token.sh"
 # Expected: "token unchanged; nginx not reloaded" when nothing changed
 ```

@@ -437,32 +437,32 @@ fresh cookie. Both verified live 2026-09-11.

 ```bash
 # 5. Cookie survives a dsh-web restart, and the new token mints a new cookie.
-ssh root@192.168.68.15 "pct exec 112 -- systemctl restart dsh-web"
+ssh root@192.168.68.12 "pct exec 112 -- systemctl restart dsh-web"
 # dsh-web is Type=simple: restart returns before :3080 is listening. Bounded-poll
 # until the socket answers (any status but 000) before asserting the cookie.
 for i in $(seq 1 60); do
-  UP=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
+  UP=$(ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
    -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/")
  [ "$UP" != "000" ] && break
  sleep 2
 done
-ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
+ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh.jar -o /dev/null \
  -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
 # Expected: 200 — the pre-restart cookie is still accepted.
 # The restart's ExecStartPost (or the 2-minute timer) refreshes the include. A
 # manual run may no-op on the flock, so poll until the include carries a token
 # the running process accepts (bounded wait) before the mint+reuse check.
 for i in $(seq 1 60); do
-  TOKEN=$(ssh root@192.168.68.15 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
-  CODE=$(ssh root@192.168.68.15 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
+  TOKEN=$(ssh root@192.168.68.12 "pct exec 112 -- sed -n 's/.*token=//p' /etc/dsh-web/nginx-login.conf | tr -d ';\n'")
+  CODE=$(ssh root@192.168.68.12 "pct exec 112 -- curl -s -o /dev/null -w '%{http_code}' \
    -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'")
  [ "$CODE" = "303" ] && break
  sleep 2
 done
 # Expected: 303 — the include now holds the token the running process accepts.
-ssh root@192.168.68.15 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \
+ssh root@192.168.68.12 "pct exec 112 -- curl -s -c /tmp/dsh-new.jar -o /dev/null \
  -H 'Host: tankodhs.sysloggh.net' 'http://127.0.0.1:3080/?token=$TOKEN'"
-ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \
+ssh root@192.168.68.12 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null \
  -w '%{http_code}\n' -H 'Host: tankodhs.sysloggh.net' http://127.0.0.1:3080/"
 # Expected: 200 — the refreshed token minted a fresh cookie.
 ```
Author	SHA1	Message	Date
abiba-bot	3b74ca28d1	Merge branch 'master' into fix/agent-health-root-hardcoding-20260928 PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s Details	2026-09-28 23:03:53 +00:00
root	af9397d672	fix(alignment): accept multiple wrapper shapes in hermes-real check PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 19s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details The fleet's wrappers do not all use a hermes-real indirection. Some (tanko, mumuni) exec the venv module directly. This check now: 1. Tries hermes-real at {home}/.local/bin (koonimo's shape) 2. Tries the venv under {home}/.hermes/hermes-agent/venv (tanko/mumuni shape) 3. Tries /usr/local/lib/hermes-agent/venv (legacy system-wide shape) Each match is reported with which shape it matched, so a genuinely broken wrapper is still a failure while a different-but-valid shape is not.	2026-09-28 22:23:08 +00:00
root	97dc2d772f	fix(alignment): repair f-string quoting in config check, add home to scope PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 8s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 18s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s Details - Fixed line 537: f-string now uses single quotes inside double-quoted shell command to avoid nested quote collision - Added home = get_user_home(user) to check_config_integrity loop scope so the config check can resolve the correct home directory	2026-09-28 21:20:42 +00:00
root	2238777a2f	fix(alignment): resolve /root/ hardcoding and stale tanko-DSH references PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 13m19s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Skipped Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped Details F1: agent-health-check.py now resolves the home directory from the agent's user field via a shared helper (get_user_home) instead of hardcoding /root/. This fixes the false-positive wrapper-missing:tanko report — tanko has a working wrapper at /home/jerome/.local/bin/hermes, but the check was looking in /root/.local/bin/. F2: Updated stale references that described tanko as DSH-only: - hermes-zulip-restore.prose.md: tanko excluded — hybrid (DSH + Hermes) - hermes-zulip-plugin.prose.md: tanko excluded — hybrid (DSH + Hermes) - infrastructure-control.prose.md: tanko is hybrid (DSH + Hermes) agent - docs/probe-drift-round2-evidence.md: marked as historical record with dated note explaining that the DSH-only observations reflected the /root/ hardcoding bug, not the underlying truth Refs: fix/agent-health-root-hardcoding-20260928	2026-09-28 20:48:49 +00:00
abiba-bot	6b2ba1bba5	Merge pull request 'fix(alignment): clarify tanko live LiteLLM proxy status in health-check description' (#145 ) from fix/contract-alignment-f2-hermes-baseline-20260928 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 27s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 33s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 14s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 8s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 4s Details	2026-09-28 12:59:50 +00:00
root	a48b947242	fix(alignment): correct tanko status - hybrid (DSH + Hermes), not DSH-only PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 21s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 21s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 20s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details PR #145 originally claimed 'the other 3 CTs are DSH-only' but all 4 agents run a Hermes gateway. Only tanko additionally runs DSH, making it the hybrid one. Fixed line 300 to say what the contract actually checks for each agent, and fixed line 76 to stop asserting tanko's Hermes config is gone.	2026-09-28 12:57:00 +00:00
abiba-bot	ba9d29b4b9	Merge pull request 'fix(alignment): recognize tanko as hybrid (DSH + Hermes) in agent-health-check' (#146 ) from fix/contract-alignment-f3-tanko-hybrid-20260928 into master	2026-09-28 12:54:42 +00:00
abiba-bot	d6376e5142	Merge pull request 'fix(alignment): use llmuser with sudo for swap-gpu-dense-model.sh' (#144 ) from fix/contract-alignment-f1-swap-gpu-user-20260928 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 18s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 24s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 15s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 2s Details	2026-09-28 12:53:53 +00:00
root	2ea6b4fc17	Merge PR #146 : fix(alignment): recognize tanko as hybrid (DSH + Hermes) in agent-health-check	2026-09-28 12:50:23 +00:00
root	c6fd8eece3	Merge PR #145 : fix(alignment): clarify tanko live LiteLLM proxy status in health-check description	2026-09-28 12:50:23 +00:00
root	4c715526ef	Merge PR #144 : fix(alignment): use llmuser with sudo for swap-gpu-dense-model.sh	2026-09-28 12:50:23 +00:00
root	308265e7ce	fix(alignment): recognize tanko as hybrid (DSH + Hermes) in agent-health-check PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 17s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 32s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s Details Tanko runs both DSH (pnpm dsh web) and Hermes (hermes gateway) concurrently. The false premise was that tanko is DSH-only with no Hermes gateway, which caused the gateway liveness, config.yaml, and wrapper integrity checks to be skipped entirely. Now tanko gets the full Hermes-era checks like koonimo and koby, while dsh/pi-only agents still skip those legs correctly.	2026-09-28 12:44:25 +00:00
root	88e9243ce4	fix(alignment): clarify tanko's live LiteLLM proxy status in health-check description	2026-09-28 12:38:42 +00:00
root	13ac189365	fix(alignment): use llmuser with sudo for swap-gpu-dense-model.sh (root SSH to .8 denied) PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 25s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 16s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 42s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 23s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details	2026-09-28 12:34:32 +00:00
root	2851a0cfc8	Merge PR #143 : fix(agent-health): use llmuser for .8 GPU health probe instead of root PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 17s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 14s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 22s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 3s Details	2026-09-28 07:27:50 +00:00
root	e0c9852de8	no-mistakes(document): Document llmuser SSH user for .8 GPU health probe PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 12s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 24s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 23s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s Details	2026-09-28 07:22:13 +00:00
root	bf3a1ba523	fix(agent-health): use llmuser for .8 GPU health probe instead of root PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Failing after 28s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped Details The rebuilt VM 101 (192.168.68.8) kept only one SSH key in root's authorized_keys, so the health check's root probe returns Permission denied and misreports the healthy host as UNREACHABLE. The llama-server runs as llmuser, so that user can see the :8080 pid via ss -tlnp. Add per-host user to GPU_HOSTS (default root, llmuser for .8) and pass it through check_gpu_ports() into all ssh() calls. Closes the gpu-unreachable:192.168.68.8 leg while leaving the root SSH security decision for the captain.	2026-09-28 07:12:57 +00:00
root	f230812e3a	Merge PR #142 : sync tanko CT 112 PVE mapping to minipve across consumers PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 18s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-09-28 07:01:19 +00:00
root	96769a103f	no-mistakes(review): sync tanko CT 112 mapping to minipve across consumers PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 12s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s Details	2026-09-28 06:47:46 +00:00
root	d697baa7b6	fix(agent-health): update tanko PVE mapping from amdpve to minipve (CT 112 migrated 2026-09-27) PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 13s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 9s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 24s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 26s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details	2026-09-28 06:35:34 +00:00
abiba-bot	fb7f351a2b	Merge pull request 'fix(audit-hermes): handle fallback_providers as list or dict' (#141 ) from fix/audit-hermes-fallback-list-20260927 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-09-27 11:35:07 +00:00
root	e42b970dec	fix(audit-hermes): handle fallback_providers as list or dict PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 17s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details The audit assumed fallback_providers was always a dict (single provider). Two live agents (koby, koonimo) carry it as a LIST of dicts (one entry per fallback), so the script crashed with: File "audit-hermes-config.py", line 211, in audit fb.get("provider") == "deepseek", AttributeError: 'list' object has no attribute 'get' Both are REAL agent configs, so this is not a malformed-input case — the script simply could not audit two of the four agents it exists to audit. Fix: - Normalize fallback_providers to a list of entries (dict → [dict], list → list) - Apply the existing checks to each entry - A malformed entry (not a mapping) produces a reported VIOLATION naming the offending entry, NOT an uncaught exception Adds regression test using the real failing shape (list of dicts) and proves it bites against the pre-fix revision. Real audit results after fix: - mumuni: FAIL — 7 violations - tanko: FAIL — 21 violations - koby: FAIL — 16 violations (previously crashed) - koonimo: FAIL — 10 violations (previously crashed) No agent configs were changed. No existing rules were relaxed.	2026-09-27 11:17:18 +00:00