diff --git a/scripts/zulip-monitor.sh b/scripts/zulip-monitor.sh index ab4d4cd..e0c815c 100755 --- a/scripts/zulip-monitor.sh +++ b/scripts/zulip-monitor.sh @@ -66,22 +66,35 @@ else echo " Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG" fi -# ── Platform B: Hermes (Tanko) ── -TANKO_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 jerome@192.168.68.122 \ - "cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}") -TANKO_ZULIP=$(echo "$TANKO_STATE" | python3 -c " -import sys,json -d=json.load(sys.stdin) -p=d.get('platforms',{}).get('zulip',{}) -print(p.get('state','unknown')) -" 2>/dev/null) +# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ── +# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker +# key availability varies — so probes run from the amdpve vantage via `pct exec`. +# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve +# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a +# remote :3080 probe is refused and is NOT a fault. +TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \ + "pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true) +[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown" +TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \ + "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true) +[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000" -if [ "$TANKO_ZULIP" != "connected" ]; then - notify "🔴" "Tanko (Hermes) Zulip state: $TANKO_ZULIP — needs restart" +if [ "$TANKO_SVC" != "active" ]; then + notify "🔴" "Tanko (DSH dsh-web) service state: $TANKO_SVC — needs restart" ISSUES=$((ISSUES + 1)) - echo " Tanko: ❌ state=$TANKO_ZULIP" >> "$LOG" + echo " Tanko: ❌ service=$TANKO_SVC" >> "$LOG" +elif [ "$TANKO_HTTP" = "000" ]; then + notify "🔴" "Tanko (DSH dsh-web) HTTP :3080 connection refused/timeout — needs restart" + ISSUES=$((ISSUES + 1)) + echo " Tanko: ❌ http=000 (refused/timeout)" >> "$LOG" else - echo " Tanko: ✅ Zulip connected" >> "$LOG" + case "$TANKO_HTTP" in + 200|301|302|307|308|401|403) + echo " Tanko: ✅ service=active http=$TANKO_HTTP" >> "$LOG" ;; + *) + notify "🟡" "Tanko (DSH dsh-web) HTTP :3080 answered $TANKO_HTTP — running, unexpected status" + echo " Tanko: 🟡 service=active http=$TANKO_HTTP (running, warning)" >> "$LOG" ;; + esac fi # ── Platform B: Hermes (Mumuni) ── diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 1e28d17..3e200ce 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -18,7 +18,7 @@ Runs every 15 minutes in the background. Also triggers on session start. ## Requires - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` -- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14) +- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14) - **PM2** on localhost for pi process management - **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` @@ -48,10 +48,8 @@ Runs every 15 minutes in the background. Also triggers on session start. }, "tanko": { "platform": "dsh", - "zulip_state": "connected", - "heartbeat_age_seconds": 45, - "gateway_pid": 1234, - "edit_fail_rate_pct": 0, + "service_state": "active", + "http_status": 200, "severity": "healthy" } } @@ -185,29 +183,77 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta | Crash loop >10/h | Alert user | -**B1: Gateway State** +### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes) + +Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so +there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs +as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve** +PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency +of this contract — per-worker key availability varies — so CT 112 probes run +from the amdpve vantage via `pct exec`: ```bash +ssh root@192.168.68.15 "pct exec 112 -- " ``` -Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so there is no `~/.hermes/gateway_state.json` on CT 112 (.122). Verify Tanko's Zulip connectivity via the DSH harness bot status instead. +> **By design (verified 2026-09-08):** the `dsh-web` gateway binds +> `127.0.0.1:3080` **loopback-only**. A remote probe against +> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault, +> and must never be raised as Tanko down. Only loopback probes from inside +> CT 112 (or the public-URL fallback below) are valid health signals. -Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed. +**B1: Gateway Service State (Tanko)** -**B2: Agent Process** +```bash +ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web" +``` + +Expected: `active`. Anything else → gateway service down → apply the Tanko heal +(restart via DSH service, Platform B Actions table below). + +**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)** + +```bash +ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" +``` + +Alive = **ANY** HTTP status response from the endpoint — the expected set is +`200`/`301`/`302`/`307`/`308`/`401`/`403` (the gateway UI is token-gated and +legitimately answers with redirects/auth-challenges, so never require a bare +`200`), and any other status, including `404`/`5xx`, also counts alive: a +process answering `503` is running and self-heal must NOT restart-loop it. +Down = connection refused (`000`) or timeout only. Statuses outside the +expected set are logged/reported as a warning — reported, never healed on. + +**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)** + +```bash +curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/ +``` + +Fallback only — used when the monitoring node has no pct/SSH path to amdpve. +Alive = **ANY** HTTP status response from the endpoint — healthy signals are +`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other +status, including `404`/`5xx`, also counts alive: the endpoint is up and +answering and must NOT be restart-looped. Down = connection refused (`000`) or +timeout only. Never expect a bare `200` — the public URL terminates in the +token-gated authentik chain. Statuses outside the healthy set are +logged/reported as a warning — reported, never healed on. + +**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway) ```bash ssh root@ "ps aux | grep 'gateway run' | grep -v grep" ``` -Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling): +Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more +than one `gateway run` process is found, the gateway has a collision (typically +one `--force` and one `--replace` process). Kill the newer/duplicate process, +then restart the remaining gateway per-agent (parameterized 2026-08-09, captain +ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1) +to confirm Zulip reloaded. -| Agent | Restart command | Notes | -|-------|-----------------|-------| - -Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded. - -**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway) +**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway) ```bash ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" @@ -216,7 +262,7 @@ ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" Expected: recent heartbeat (within 5 min), `polls=N` incrementing. Silence > 300s → warning. Silence > 600s → critical. -**B4: Response Delivery** (Hermes agent Mumuni only) +**B6: Response Delivery** (Hermes agent Mumuni only) ```bash ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"