From 5313e6b9ba7d3f64569652866a521ae1e99b0a54 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 8 Sep 2026 10:26:56 +0000 Subject: [PATCH] fix: tanko zulip-health probes via amdpve pct exec (dsh-web loopback :3080 + public-URL fallback) --- zulip-health.prose.md | 66 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 53 insertions(+), 13 deletions(-) diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 1e28d17..883c2f5 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -18,7 +18,7 @@ Runs every 15 minutes in the background. Also triggers on session start. ## Requires - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` -- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14) +- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` only (no direct SSH to .122 after agent migrations); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14) - **PM2** on localhost for pi process management - **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` @@ -185,29 +185,69 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta | Crash loop >10/h | Alert user | -**B1: Gateway State** +### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes) + +Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so +there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs +as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve** +PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is no longer a valid +vantage point after the agent migrations — reach CT 112 only via `pct exec` on +amdpve: ```bash +ssh root@192.168.68.15 "pct exec 112 -- " ``` -Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so there is no `~/.hermes/gateway_state.json` on CT 112 (.122). Verify Tanko's Zulip connectivity via the DSH harness bot status instead. +> **By design (verified 2026-09-08):** the `dsh-web` gateway binds +> `127.0.0.1:3080` **loopback-only**. A remote probe against +> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault, +> and must never be raised as Tanko down. Only loopback probes from inside +> CT 112 (or the public-URL fallback below) are valid health signals. -Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed. +**B1: Gateway Service State (Tanko)** -**B2: Agent Process** +```bash +ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web" +``` + +Expected: `active`. Anything else → gateway service down → apply the Tanko heal +(restart via DSH service, Platform B Actions table below). + +**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)** + +```bash +ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" +``` + +Alive = **ANY** HTTP status of `200`/`301`/`302`/`307`/`308`/`401`/`403` — the +gateway UI is token-gated and legitimately answers with redirects/auth-challenges, +so never require a bare `200`. Down = connection refused (`000`) or timeout only. + +**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)** + +```bash +curl -s --connect-timeout 10 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/ +``` + +Fallback only — used when the monitoring node has no pct/SSH path to amdpve. +Healthy = `302` (authentik proxy-auth redirect) or `401` (auth-gated). Down = +connection refused, timeout, or 5xx. Never expect a bare `200` — the public URL +terminates in the token-gated authentik chain. + +**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway) ```bash ssh root@ "ps aux | grep 'gateway run' | grep -v grep" ``` -Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling): +Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more +than one `gateway run` process is found, the gateway has a collision (typically +one `--force` and one `--replace` process). Kill the newer/duplicate process, +then restart the remaining gateway per-agent (parameterized 2026-08-09, captain +ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1) +to confirm Zulip reloaded. -| Agent | Restart command | Notes | -|-------|-----------------|-------| - -Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded. - -**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway) +**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway) ```bash ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" @@ -216,7 +256,7 @@ ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" Expected: recent heartbeat (within 5 min), `polls=N` incrementing. Silence > 300s → warning. Silence > 600s → critical. -**B4: Response Delivery** (Hermes agent Mumuni only) +**B6: Response Delivery** (Hermes agent Mumuni only) ```bash ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"