fix(zulip-monitor): Abiba leg reads nested zulip.connected; probe failures never restart
The monitor's Abiba leg parsed the :9200/health payload at the top level
(d.get('connected',False)) while the pi Zulip extension serves connection
state NESTED at zulip.connected / zulip.last_error (verified against the
extension's startHealthServer handler and the live endpoint). PI_CONNECTED
was therefore always False and every run took the DISCONNECTED path,
restarting a healthy bot: pm2 restarts=8 with the process created
2026-09-09T09:35:09Z, four ❌ Abiba verdicts today (04:23/05:35/06:55/09:35
UTC) and zero ✅, while the Zulip server answered HTTP 200 and the bot kept
heartbeating. The watchdog was the fault, not the connection.
Changes (Abiba leg only; every other leg byte-identical):
- Read the real nested shape: zulip.connected, zulip.last_error and
zulip.messages_processed. The old retry_count branch is DROPPED — the
payload exposes no retry counter (the extension keeps retryCount internal
and never serialises it), so the branch is fabricated and cannot stay.
- Fail-safe restart decision: a fetch error (HTTP 000), non-2xx response,
empty/unparseable body, or payload missing a boolean zulip.connected is a
clearly-labelled PROBE FAILURE (🟠 alert + ⚠️ log line naming reason and
HTTP code) and NEVER calls pm2 restart. pm2 restart runs only on
affirmative zulip.connected=false (🔴/❌ path unchanged in wording).
connected=true with last_error keeps the degraded 🟡 warn-no-restart path.
Regression test (new tests/zulip-monitor-abiba.sh, 36 checks): extracts the
real Abiba leg from between the # -- abiba-leg-start/-end markers in the
shipped script and executes it verbatim with stubbed curl/notify/pm2 against
fixtures of the real payload shape — asserts connected=true → no restart,
connected=false → restart, and empty/garbage/missing-key/non-boolean/HTTP
000/HTTP 500 bodies → probe-failure alert with zero restarts. The suite
fails loudly on the pre-fix base and on a marker-intact top-level-parse
variant, so this class of bug cannot return silently.
This commit is contained in:
root
+62
-19
@@ -46,26 +46,69 @@ else
|
||||
fi
|
||||
|
||||
# ── Platform A: pi (Abiba) ──
|
||||
PI_HEALTH=$(curl -sf --connect-timeout 5 http://localhost:9200/health 2>/dev/null || echo "{}")
|
||||
PI_CONNECTED=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('connected',False))" 2>/dev/null)
|
||||
PI_ERROR=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('last_error') or '')" 2>/dev/null)
|
||||
PI_RETRIES=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('retry_count',0))" 2>/dev/null)
|
||||
# Probes the pi Zulip extension health endpoint (:9200/health, served by the
|
||||
# extension's startHealthServer; shape documented in zulip-health.prose.md).
|
||||
# FAIL-SAFE contract (pinned by tests/zulip-monitor-abiba.sh): connection state
|
||||
# lives NESTED at zulip.connected / zulip.last_error — there is no top-level
|
||||
# `connected` and no retry counter in the payload. A fetch error, non-2xx
|
||||
# response, empty/unparseable body, or payload missing a boolean
|
||||
# zulip.connected is a PROBE FAILURE: it alerts and NEVER calls pm2 restart.
|
||||
# pm2 restart runs ONLY on affirmative zulip.connected=false.
|
||||
# -- abiba-leg-start (verbatim-extracted by tests/zulip-monitor-abiba.sh)
|
||||
PI_HTTP=$(curl -s -o /dev/null --connect-timeout 5 --max-time 10 -w '%{http_code}' http://localhost:9200/health 2>/dev/null || echo "000")
|
||||
PI_BODY=$(curl -s --connect-timeout 5 --max-time 10 http://localhost:9200/health 2>/dev/null || true)
|
||||
PI_STATE=$(printf '%s' "$PI_BODY" | python3 -c '
|
||||
import sys, json
|
||||
code = sys.argv[1]
|
||||
body = sys.stdin.read()
|
||||
try:
|
||||
d = json.loads(body)
|
||||
except Exception:
|
||||
sys.stdout.write("probe-failed|unparseable body")
|
||||
sys.exit(0)
|
||||
if not code.startswith("2"):
|
||||
sys.stdout.write("probe-failed|HTTP %s" % code)
|
||||
sys.exit(0)
|
||||
if not isinstance(d, dict) or not isinstance(d.get("zulip"), dict):
|
||||
sys.stdout.write("probe-failed|missing zulip.connected")
|
||||
sys.exit(0)
|
||||
z = d["zulip"]
|
||||
if "connected" not in z or not isinstance(z["connected"], bool):
|
||||
sys.stdout.write("probe-failed|missing or non-boolean zulip.connected")
|
||||
sys.exit(0)
|
||||
err = z.get("last_error") or ""
|
||||
if z["connected"]:
|
||||
if err:
|
||||
sys.stdout.write("degraded|%s" % err)
|
||||
else:
|
||||
sys.stdout.write("healthy|%s" % z.get("messages_processed", 0))
|
||||
else:
|
||||
sys.stdout.write("disconnected|")
|
||||
' "$PI_HTTP" 2>/dev/null) || PI_STATE="probe-failed|python error"
|
||||
PI_VERDICT=${PI_STATE%%|*}
|
||||
PI_DETAIL=${PI_STATE#*|}
|
||||
|
||||
if [ "$PI_CONNECTED" != "True" ]; then
|
||||
notify "🔴" "Abiba pi extension DISCONNECTED — restarting"
|
||||
pm2 restart abiba-zulip 2>/dev/null || true
|
||||
ISSUES=$((ISSUES + 1))
|
||||
echo " Abiba: ❌ Disconnected — restarted" >> "$LOG"
|
||||
elif [ -n "$PI_ERROR" ]; then
|
||||
notify "🟡" "Abiba pi extension error: ${PI_ERROR:0:100}"
|
||||
echo " Abiba: 🟡 Error: ${PI_ERROR:0:100}" >> "$LOG"
|
||||
elif [ "$PI_RETRIES" -ge 3 ]; then
|
||||
notify "🟡" "Abiba pi extension: $PI_RETRIES retries — restarting"
|
||||
pm2 restart abiba-zulip 2>/dev/null || true
|
||||
echo " Abiba: 🟡 $PI_RETRIES retries — restarted" >> "$LOG"
|
||||
else
|
||||
echo " Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG"
|
||||
fi
|
||||
case "$PI_VERDICT" in
|
||||
healthy)
|
||||
echo " Abiba: ✅ Connected (processed=$PI_DETAIL)" >> "$LOG" ;;
|
||||
degraded)
|
||||
notify "🟡" "Abiba pi extension error: ${PI_DETAIL:0:100}"
|
||||
echo " Abiba: 🟡 Error: ${PI_DETAIL:0:100}" >> "$LOG" ;;
|
||||
disconnected)
|
||||
notify "🔴" "Abiba pi extension DISCONNECTED — restarting"
|
||||
pm2 restart abiba-zulip 2>/dev/null || true
|
||||
ISSUES=$((ISSUES + 1))
|
||||
echo " Abiba: ❌ Disconnected — restarted" >> "$LOG" ;;
|
||||
probe-failed)
|
||||
notify "🟠" "Abiba pi extension health probe FAILED (${PI_DETAIL}; HTTP $PI_HTTP) — NOT restarting, manual check needed"
|
||||
ISSUES=$((ISSUES + 1))
|
||||
echo " Abiba: ⚠️ Probe failed (${PI_DETAIL}; HTTP $PI_HTTP) — NOT restarted" >> "$LOG" ;;
|
||||
*)
|
||||
notify "🟠" "Abiba pi extension health probe returned unexpected verdict (${PI_STATE}) — NOT restarting, manual check needed"
|
||||
ISSUES=$((ISSUES + 1))
|
||||
echo " Abiba: ⚠️ Unexpected probe verdict (${PI_STATE}) — NOT restarted" >> "$LOG" ;;
|
||||
esac
|
||||
# -- abiba-leg-end
|
||||
|
||||
# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
|
||||
# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
|
||||
|
||||
Reference in New Issue
Block a user