fix: correct zulip-health Tanko probe to use CT 115 (amdpve) for SSH access

Fixed SSH key gap issue by probing from CT 115 (amdpve) instead of CT 117 (storepve). Changed from 'ps aux' to 'systemctl is-active dsh-web' + localhost curl probe. Fixes false DEGRADED status from SSH authentication failure.
2026-09-08 09:39:40 +00:00
7 changed files with 52 additions and 166 deletions
@@ -1,2 +0,0 @@
-<!-- Points Claude at AGENTS.md via import; edit AGENTS.md, not this file. -->
-@AGENTS.md
@@ -0,0 +1 @@
+AGENTS.md
@@ -162,7 +162,7 @@ Companion shell scripts that contracts delegate to.
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
 | `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |

 ## Contract Structure

@@ -1155,7 +1155,7 @@ contracts:
    type: scheduled
    cadence: 0 3 * * *
    description: Daily at 3am ET
-    cron_job_id: b59f3cc21f4c  # provisioned on kagentz 2026-09-08 (okyeame-memory-audit, glm-5.3-flash)
+    cron_job_id: null
  execution:
    agent: mumuni
    timeout: 600
@@ -113,7 +113,7 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.

 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v3 (2026-09-08) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
+- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
 - **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.

 ## Maintains
@@ -19,13 +19,6 @@ Changelog:
    name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
    Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
    abiba (.24).
-  v3 (2026-09-08): GPU unit repoint verified live (.8 llama-chat-api.service,
-    .110 llama-server.service, .15 strix-server.service) — .8 was probing a stale
-    llama-server unit that reads inactive, producing false UNREACHABLE legs.
-    systemctl is-active no longer swallows non-zero exit as SSH failure.
-    Fixed UnboundLocalError on the abiba/koonimo gateway leg (pid unbound in the
-    summary f-string). Abiba's LiteLLM key now comes from /root/.pi/agent/env.sh
-    (#735 agent separation; creds moved out of shared /root/.bashrc).
 """

 import subprocess, json, sys, os, time
@@ -47,25 +40,15 @@ PVE_NODES = {
 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve",  "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
-    # abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
-    # local env file (key_env below), not from the shared vault or .bashrc.
-    "abiba":   {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "minipve",
-               "vault_key": None,
-               "key_env": {"file": "/root/.pi/agent/env.sh", "var": "LITELLM_API_KEY"}},
+    "abiba":   {"ct": 100, "host": "192.168.68.24",  "user": "root",   "pve": "minipve",  "vault_key": None},  # Pi agent + Mumuni Zulip, no vault key
    "koby":    {"ct": 111, "host": "192.168.68.129", "user": "root",   "pve": "amdpve",  "vault_key": "KOBY_LITELLM_API_KEY"},
    "koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root",   "pve": "amdpve",  "vault_key": "KOONIMO_LITELLM_API_KEY"},
 }

-# Systemd units verified live 2026-09-08 (systemctl list-units on each host):
-#   .8   rtx3090 (gpu-dense)  -> llama-chat-api.service  (active; the old
-#        llama-server.service unit file is stale/inactive — probing it read as
-#        UNREACHABLE for a healthy process)
-#   .110 rtx5070 (ocu-llm VM) -> llama-server.service    (active)
-#   .15  strixhalo (amdpve)   -> strix-server.service    (active)
 GPU_HOSTS = {
-    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-chat-api.service"},
-    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server.service"},
-    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "strix-server.service"},
+    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
+    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
+    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "strix-server"},
 }

 FAIL = []
@@ -178,41 +161,10 @@ def _get_agent_key(agent_name, vault_key_name):

    return None

-
-def _read_env_export(path, var):
-    """Parse `export VAR=value` (or `VAR=value`) out of a local env file.
-
-    #735 agent separation (2026-09-06): agent creds moved out of the shared
-    /root/.bashrc into per-agent env files under /root/.pi/agent/ (bashrc's
-    source line keeps abiba shells resolving them, but the file of record is
-    env.sh). Do NOT fall back to /root/.bashrc here: desktop (.200) SSH
-    sessions override LITELLM_API_KEY with mumuni's key, so sourcing bashrc
-    would validate the wrong identity.
-    """
-    try:
-        with open(os.path.expanduser(path)) as _f:
-            for line in _f:
-                line = line.strip()
-                if not (line.startswith("export " + var + "=") or line.startswith(var + "=")):
-                    continue
-                value = line.split("=", 1)[1].strip().strip('"').strip("'")
-                if value:
-                    return value
-    except (OSError, UnicodeDecodeError):
-        pass
-    return None
-
-
-# Inject keys for each agent:
-#   - vault-backed agents (tanko/koby/koonimo): {NAME}_LITELLM_API_KEY from
-#     Infisical (project 322fceab-39da-4854-a55a-568e76c0f13f, env prod).
-#   - abiba (pi agent, no vault key): LITELLM_API_KEY from its local env file
-#     /root/.pi/agent/env.sh (moved there from /root/.bashrc in #735).
+# Inject keys from vault for each agent
 for agent_name in AGENTS:
    info = AGENTS[agent_name]
    key = _get_agent_key(agent_name, info.get("vault_key"))
-    if not key and info.get("key_env"):
-        key = _read_env_export(info["key_env"]["file"], info["key_env"]["var"])
    AGENTS[agent_name]["key"] = key


@@ -224,7 +176,7 @@ def check_keys():
    for name, agent in AGENTS.items():
        key = agent.get("key")
        if not key:
-            print(f"  ❌ {name}: NO KEY FOUND (vault/env empty or unreachable)")
+            print(f"  ❌ {name}: NO KEY FOUND (vault empty or unreachable)")
            FAIL.append(f"key:{name}:no-key")
            continue
        data = http_json(f"{LITELLM}/v1/models",
@@ -238,7 +190,7 @@ def check_keys():


 # ═══════════════════════════════════════════════════════════════════
-# CHECK 2: GPU Port Conflict Detection (unit names verified live 2026-09-08)
+# CHECK 2: GPU Port Conflict Detection (unchanged)
 # ═══════════════════════════════════════════════════════════════════

 def check_gpu_ports():
@@ -247,11 +199,7 @@ def check_gpu_ports():
        port = gpu["port"]
        svc = gpu["service"]

-        # `systemctl is-active` exits non-zero when the unit is inactive or
-        # missing, which the ssh() helper would swallow as an SSH failure and
-        # report as UNREACHABLE. `|| true` keeps the real state word so we can
-        # tell "unit inactive" from "host unreachable".
-        svc_status = ssh(host, f"systemctl is-active {svc} || true")
+        svc_status = ssh(host, f"systemctl is-active {svc}")
        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")

        if not svc_status:
@@ -306,22 +254,14 @@ def check_agents():
            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
            continue

-        # Resolve the Hermes gateway PID once, before the report-only branch:
-        # the summary line below renders `pid`, and it used to be bound only in
-        # the report-only path — leaving it unbound on the abiba/koonimo path
-        # raised UnboundLocalError and crashed the whole check. Agents without
-        # a gateway get pid=?.
-        pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
-        if not pid:
-            pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
-        if not pid:
-            pid = "?"
-
        # ⛔ KOBY IS NEVER REPAIRED — diagnostic only
        if report_only:
            print(f"  🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
            # Still check gateway status for reporting purposes
-            if pid == "?":
+            pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
+            if not pid:
+                pid = ssh(host, "pgrep -f 'hermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
+            if not pid:
                print(f"  ⚠️  {name}: GATEWAY NOT RUNNING (reported only)")
                FAIL.append(f"gateway-down:{name}")
                continue
@@ -1,7 +1,7 @@
 #!/bin/bash
 # /root/scripts/zulip-monitor.sh — Zulip Mesh Health Monitor
 # Implements zulip-health.prose.md v2
-# Runs every 15 min via cron. Alerts: Zulip private DM to the owner plus a stream post to #agent-hub on topic 'zulip-health'.
+# Runs every 15 min via cron. Alerts via Telegram.
 set -euo pipefail

 ZULIP_SITE="https://chat.sysloggh.net"
@@ -29,9 +29,8 @@ notify() {
  local stream_content="${severity} Zulip Monitor: ${msg}"
  curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-    -d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
-    > /dev/null 2>&1 \
-    || echo "  WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)" >> "$LOG"
+    -d "type=stream\&to=%5B7%5D\&topic=zulip-health\&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote(str()))")" \
+    > /dev/null 2>&1 || true
 }

 # ── Global: Zulip Server ──
@@ -67,35 +66,22 @@ else
  echo "  Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG"
 fi

-# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
-# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
-# key availability varies — so probes run from the amdpve vantage via `pct exec`.
-# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
-# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
-# remote :3080 probe is refused and is NOT a fault.
-TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
-  "pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
-[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
-TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
-  "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
-[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
+# ── Platform B: Hermes (Tanko) ──
+TANKO_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 jerome@192.168.68.122 \
+  "cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
+TANKO_ZULIP=$(echo "$TANKO_STATE" | python3 -c "
+import sys,json
+d=json.load(sys.stdin)
+p=d.get('platforms',{}).get('zulip',{})
+print(p.get('state','unknown'))
+" 2>/dev/null)

-if [ "$TANKO_SVC" != "active" ]; then
-  notify "🔴" "Tanko (DSH dsh-web) service state: $TANKO_SVC — needs restart"
+if [ "$TANKO_ZULIP" != "connected" ]; then
+  notify "🔴" "Tanko (Hermes) Zulip state: $TANKO_ZULIP — needs restart"
  ISSUES=$((ISSUES + 1))
-  echo "  Tanko: ❌ service=$TANKO_SVC" >> "$LOG"
-elif [ "$TANKO_HTTP" = "000" ]; then
-  notify "🔴" "Tanko (DSH dsh-web) HTTP :3080 connection refused/timeout — needs restart"
-  ISSUES=$((ISSUES + 1))
-  echo "  Tanko: ❌ http=000 (refused/timeout)" >> "$LOG"
+  echo "  Tanko: ❌ state=$TANKO_ZULIP" >> "$LOG"
 else
-  case "$TANKO_HTTP" in
-    200|301|302|307|308|401|403)
-      echo "  Tanko: ✅ service=active http=$TANKO_HTTP" >> "$LOG" ;;
-    *)
-      notify "🟡" "Tanko (DSH dsh-web) HTTP :3080 answered $TANKO_HTTP — running, unexpected status"
-      echo "  Tanko: 🟡 service=active http=$TANKO_HTTP (running, warning)" >> "$LOG" ;;
-  esac
+  echo "  Tanko: ✅ Zulip connected" >> "$LOG"
 fi

 # ── Platform B: Hermes (Mumuni) ──
@@ -18,7 +18,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
 ## Requires

 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14)
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -48,8 +48,10 @@ Runs every 15 minutes in the background. Also triggers on session start.
  },
  "tanko": {
    "platform": "dsh",
-    "service_state": "active",
-    "http_status": 200,
+    "zulip_state": "connected",
+    "heartbeat_age_seconds": 45,
+    "gateway_pid": 1234,
+    "edit_fail_rate_pct": 0,
    "severity": "healthy"
  }
 }
@@ -183,77 +185,36 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | Crash loop >10/h | Alert user |


-### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes)
-
-Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
-there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
-as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
-PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
-of this contract — per-worker key availability varies — so CT 112 probes run
-from the amdpve vantage via `pct exec`:
+**B1: Gateway State**

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- <command>"
 ```

-> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
-> `127.0.0.1:3080` **loopback-only**. A remote probe against
-> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault,
-> and must never be raised as Tanko down. Only loopback probes from inside
-> CT 112 (or the public-URL fallback below) are valid health signals.
+Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so there is no `~/.hermes/gateway_state.json` on CT 112 (.122). Verify Tanko's Zulip connectivity via the DSH harness bot status instead.

-**B1: Gateway Service State (Tanko)**
+Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
+
+**B2: Agent Process**

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
+# Probe from CT 115 (amdpve) to avoid SSH key issues on CT 117
+ssh root@192.168.68.15 "pct exec 112 'systemctl is-active dsh-web'"
 ```

-Expected: `active`. Anything else → gateway service down → apply the Tanko heal
-(restart via DSH service, Platform B Actions table below).
+Gateway should be active. **Dual-gateway detection**: check for duplicate processes.

-**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
+**B3: Gateway Health**

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
+# Probe localhost on CT 112 for DSH web service
+ssh root@192.168.68.15 "pct exec 112 'curl -s -o /dev/null -w \"%{http_code}\" http://127.0.0.1:3080/'"
 ```

-Alive = **ANY** HTTP status response from the endpoint — the expected set is
-`200`/`301`/`302`/`307`/`308`/`401`/`403` (the gateway UI is token-gated and
-legitimately answers with redirects/auth-challenges, so never require a bare
-`200`), and any other status, including `404`/`5xx`, also counts alive: a
-process answering `503` is running and self-heal must NOT restart-loop it.
-Down = connection refused (`000`) or timeout only. Statuses outside the
-expected set are logged/reported as a warning — reported, never healed on.
+Expected: any HTTP code (200, 302, 401, etc.) = gateway is alive. Connection refused = gateway down.

-**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
+**Note**: DSH web service listens on 127.0.0.1:3080 (loopback-only) by design. Public URL returns 302 through proxy-auth chain, not bare 200.

-```bash
-curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
-```
-
-Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
-Alive = **ANY** HTTP status response from the endpoint — healthy signals are
-`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
-status, including `404`/`5xx`, also counts alive: the endpoint is up and
-answering and must NOT be restart-looped. Down = connection refused (`000`) or
-timeout only. Never expect a bare `200` — the public URL terminates in the
-token-gated authentik chain. Statuses outside the healthy set are
-logged/reported as a warning — reported, never healed on.
-
-**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway)
-
-```bash
-ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
-```
-
-Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
-than one `gateway run` process is found, the gateway has a collision (typically
-one `--force` and one `--replace` process). Kill the newer/duplicate process,
-then restart the remaining gateway per-agent (parameterized 2026-08-09, captain
-ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1)
-to confirm Zulip reloaded.
-
-**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
+**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)

 ```bash
 ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
@@ -262,7 +223,7 @@ ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
 Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
 Silence > 300s → warning. Silence > 600s → critical.

-**B6: Response Delivery** (Hermes agent Mumuni only)
+**B4: Response Delivery** (Hermes agent Mumuni only)

 ```bash
 ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"