SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix: tanko zulip-health probes via amdpve pct exec (dsh-web loopback :3080 + public-URL fallback)

Browse Source
This commit is contained in:
root
2026-09-08 10:26:56 +00:00
parent 0e4eda0abb
commit 5313e6b9ba
1 changed files with 53 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
zulip-health.prose.md
+53 -13
View File
@@ -18,7 +18,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14)
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` only (no direct SSH to .122 after agent migrations); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -185,29 +185,69 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| Crash loop >10/h | Alert user |
**B1: Gateway State**
### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes)
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is no longer a valid
vantage point after the agent migrations — reach CT 112 only via `pct exec` on
amdpve:
```bash
ssh root@192.168.68.15 "pct exec 112 -- <command>"
```
Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so there is no `~/.hermes/gateway_state.json` on CT 112 (.122). Verify Tanko's Zulip connectivity via the DSH harness bot status instead.
> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
> `127.0.0.1:3080` **loopback-only**. A remote probe against
> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault,
> and must never be raised as Tanko down. Only loopback probes from inside
> CT 112 (or the public-URL fallback below) are valid health signals.
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
**B1: Gateway Service State (Tanko)**
**B2: Agent Process**
```bash
ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
```
Expected: `active`. Anything else → gateway service down → apply the Tanko heal
(restart via DSH service, Platform B Actions table below).
**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
```bash
ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
```
Alive = **ANY** HTTP status of `200`/`301`/`302`/`307`/`308`/`401`/`403` — the
gateway UI is token-gated and legitimately answers with redirects/auth-challenges,
so never require a bare `200`. Down = connection refused (`000`) or timeout only.
**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
```bash
curl -s --connect-timeout 10 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
```
Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
Healthy = `302` (authentik proxy-auth redirect) or `401` (auth-gated). Down =
connection refused, timeout, or 5xx. Never expect a bare `200` — the public URL
terminates in the token-gated authentik chain.
**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway)
```bash
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
```
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling):
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
than one `gateway run` process is found, the gateway has a collision (typically
one `--force` and one `--replace` process). Kill the newer/duplicate process,
then restart the remaining gateway per-agent (parameterized 2026-08-09, captain
ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1)
to confirm Zulip reloaded.
| Agent | Restart command | Notes |
|-------|-----------------|-------|
Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
```bash
ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
@@ -216,7 +256,7 @@ ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
Silence > 300s → warning. Silence > 600s → critical.
**B4: Response Delivery** (Hermes agent Mumuni only)
**B6: Response Delivery** (Hermes agent Mumuni only)
```bash
ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"