scripts/zulip-monitor.sh

@@ -66,22 +66,35 @@ else
   echo "  Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG"
 fi
 
-# ── Platform B: Hermes (Tanko) ──
-TANKO_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 jerome@192.168.68.122 \
-  "cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
-TANKO_ZULIP=$(echo "$TANKO_STATE" | python3 -c "
-import sys,json
-d=json.load(sys.stdin)
-p=d.get('platforms',{}).get('zulip',{})
-print(p.get('state','unknown'))
-" 2>/dev/null)
+# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
+# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
+# key availability varies — so probes run from the amdpve vantage via `pct exec`.
+# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
+# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
+# remote :3080 probe is refused and is NOT a fault.
+TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
+  "pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
+[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
+TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
+  "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
+[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
 
-if [ "$TANKO_ZULIP" != "connected" ]; then
-  notify "🔴" "Tanko (Hermes) Zulip state: $TANKO_ZULIP — needs restart"
+if [ "$TANKO_SVC" != "active" ]; then
+  notify "🔴" "Tanko (DSH dsh-web) service state: $TANKO_SVC — needs restart"
   ISSUES=$((ISSUES + 1))
-  echo "  Tanko: ❌ state=$TANKO_ZULIP" >> "$LOG"
+  echo "  Tanko: ❌ service=$TANKO_SVC" >> "$LOG"
+elif [ "$TANKO_HTTP" = "000" ]; then
+  notify "🔴" "Tanko (DSH dsh-web) HTTP :3080 connection refused/timeout — needs restart"
+  ISSUES=$((ISSUES + 1))
+  echo "  Tanko: ❌ http=000 (refused/timeout)" >> "$LOG"
 else
-  echo "  Tanko: ✅ Zulip connected" >> "$LOG"
+  case "$TANKO_HTTP" in
+    200|301|302|307|308|401|403)
+      echo "  Tanko: ✅ service=active http=$TANKO_HTTP" >> "$LOG" ;;
+    *)
+      notify "🟡" "Tanko (DSH dsh-web) HTTP :3080 answered $TANKO_HTTP — running, unexpected status"
+      echo "  Tanko: 🟡 service=active http=$TANKO_HTTP (running, warning)" >> "$LOG" ;;
+  esac
 fi
 
 # ── Platform B: Hermes (Mumuni) ──

zulip-health.prose.md

@@ -18,7 +18,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
 ## Requires
 
 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
-- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14)
+- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -48,10 +48,8 @@ Runs every 15 minutes in the background. Also triggers on session start.
   },
   "tanko": {
     "platform": "dsh",
-    "zulip_state": "connected",
-    "heartbeat_age_seconds": 45,
-    "gateway_pid": 1234,
-    "edit_fail_rate_pct": 0,
+    "service_state": "active",
+    "http_status": 200,
     "severity": "healthy"
   }
 }
@@ -185,29 +183,77 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | Crash loop >10/h | Alert user |
 
 
-**B1: Gateway State**
+### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes)
+
+Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
+there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
+as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
+PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
+of this contract — per-worker key availability varies — so CT 112 probes run
+from the amdpve vantage via `pct exec`:
 
 ```bash
+ssh root@192.168.68.15 "pct exec 112 -- <command>"
 ```
 
-Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so there is no `~/.hermes/gateway_state.json` on CT 112 (.122). Verify Tanko's Zulip connectivity via the DSH harness bot status instead.
+> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
+> `127.0.0.1:3080` **loopback-only**. A remote probe against
+> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault,
+> and must never be raised as Tanko down. Only loopback probes from inside
+> CT 112 (or the public-URL fallback below) are valid health signals.
 
-Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
+**B1: Gateway Service State (Tanko)**
 
-**B2: Agent Process**
+```bash
+ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
+```
+
+Expected: `active`. Anything else → gateway service down → apply the Tanko heal
+(restart via DSH service, Platform B Actions table below).
+
+**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
+
+```bash
+ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
+```
+
+Alive = **ANY** HTTP status response from the endpoint — the expected set is
+`200`/`301`/`302`/`307`/`308`/`401`/`403` (the gateway UI is token-gated and
+legitimately answers with redirects/auth-challenges, so never require a bare
+`200`), and any other status, including `404`/`5xx`, also counts alive: a
+process answering `503` is running and self-heal must NOT restart-loop it.
+Down = connection refused (`000`) or timeout only. Statuses outside the
+expected set are logged/reported as a warning — reported, never healed on.
+
+**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
+
+```bash
+curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
+```
+
+Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
+Alive = **ANY** HTTP status response from the endpoint — healthy signals are
+`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
+status, including `404`/`5xx`, also counts alive: the endpoint is up and
+answering and must NOT be restart-looped. Down = connection refused (`000`) or
+timeout only. Never expect a bare `200` — the public URL terminates in the
+token-gated authentik chain. Statuses outside the healthy set are
+logged/reported as a warning — reported, never healed on.
+
+**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway)
 
 ```bash
 ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
 ```
 
-Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling):
+Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
+than one `gateway run` process is found, the gateway has a collision (typically
+one `--force` and one `--replace` process). Kill the newer/duplicate process,
+then restart the remaining gateway per-agent (parameterized 2026-08-09, captain
+ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1)
+to confirm Zulip reloaded.
 
-| Agent | Restart command | Notes |
-|-------|-----------------|-------|
-
-Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
-
-**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
+**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
 
 ```bash
 ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
@@ -216,7 +262,7 @@ ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
 Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
 Silence > 300s → warning. Silence > 600s → critical.
 
-**B4: Response Delivery** (Hermes agent Mumuni only)
+**B6: Response Delivery** (Hermes agent Mumuni only)
 
 ```bash
 ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"