fix(monitoring): retire the Mumuni leg — stop probing her decommissioned .24 deployment
Captain ruling 2026-09-10: Mumuni moved off this host onto her own container (kagentz CT 105 on minipve, 192.168.68.14, dedicated `hermes` user) and is monitored from her side. This host must not monitor anything Mumuni. The stale probes fired false alerts repeatedly: * scripts/zulip-monitor.sh ssh'd to root@192.168.68.24 for the decommissioned deployment's ~/.hermes/gateway_state.json, read "unknown" on every run, and posted a 🔴 "Mumuni (Hermes) Zulip state: unknown" DM + #agent-hub stream alert each cycle. * scripts/daily-infra-report.py published a matching "mumuni:unknown" row in every digest. Changes: * zulip-monitor.sh: delete the "Platform B: Hermes (Mumuni)" leg and its notify; keep the Zulip-server, Platform A pi/Abiba (bridge), Platform B Tanko and Platform C Agent Zero legs. A comment records why the leg is retired so it is not re-added. Also fixes SC2155 so shellcheck is clean. * daily-infra-report.py: delete the .24 ~/.hermes/gateway_state.json agent probe, its hermes --version probe, and the now-dead mumuni render branch. Abiba (CT 100, its own .24 address) and Tanko legs unchanged; the PVE API token name is untouched. * zulip-health.prose.md (v3.1.0): drop the Mumuni-only B4 gateway-process, B5 heartbeat and B6 response-delivery steps and the stale 192.168.68.24 references; state explicitly that Mumuni is not monitored from this host. Tanko/Agent-Zero/bridge steps retained. * agent-health-check.py: correct the v2 changelog roster comment that still placed mumuni at .24/CT100. No behavior change — the mumuni probe was already absent from the AGENTS dict; v5 changelog notes the correction. Tests: tests/test_mumuni_monitor_removal.py pins the removal structurally and behaviorally — the shipped zulip-monitor.sh is run in a sandbox (only its LOG constant rewritten) with stub ssh/curl on PATH; the ssh stub records every target host, so "never reaches .24" and "no Mumuni notify even on the alert path" are asserted from observed behavior. A mutation check (re-inject the old leg) fails the suite, so the guarantee is not vacuous.
This commit is contained in:
abiba
@@ -17,8 +17,9 @@ Changelog:
|
||||
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
|
||||
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
|
||||
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
|
||||
Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
|
||||
abiba (.24).
|
||||
Fleet roster: tanko (.122), koby (.129), koonimo (.114), abiba (.24).
|
||||
(v2 also carried a mumuni probe; see v5 — mumuni is no longer probed: she
|
||||
moved to her own container, kagentz CT 105 / .14, and is monitored there.)
|
||||
v3 (2026-09-08): GPU unit repoint verified live (.8 llama-chat-api.service,
|
||||
.110 llama-server.service, .15 strix-server.service) — .8 was probing a stale
|
||||
llama-server unit that reads inactive, producing false UNREACHABLE legs.
|
||||
@@ -44,6 +45,11 @@ Changelog:
|
||||
separate from `failures`. Every run prints absolute execution provenance
|
||||
(script + cwd) in the header, in the cron ALERT line, and in --json output so
|
||||
a stale-consumer report is distinguishable from a fault at read time.
|
||||
v5 (2026-09-10): roster correction only, no behavior change. mumuni was removed
|
||||
from the AGENTS dict when she moved off this host onto her own container
|
||||
(kagentz CT 105 on minipve, .14, dedicated `hermes` user) and is monitored
|
||||
from her side. This script must not probe mumuni or .24 — the v2 changelog
|
||||
roster line was the last reference still placing her at .24 / CT100.
|
||||
"""
|
||||
|
||||
import subprocess, json, sys, os, time, re, io, contextlib
|
||||
|
||||
@@ -308,26 +308,13 @@ def collect():
|
||||
"updated_at": "",
|
||||
}
|
||||
|
||||
# Mumuni (CT 100, IP 192.168.68.24)
|
||||
mumuni_state = ssh("192.168.68.24", "cat ~/.hermes/gateway_state.json 2>/dev/null")
|
||||
mumuni_data = {}
|
||||
try:
|
||||
mumuni_data = json.loads(mumuni_state) if mumuni_state else {}
|
||||
except:
|
||||
mumuni_data = {}
|
||||
mumuni_platforms = mumuni_data.get("platforms", {})
|
||||
report["agents"]["mumuni"] = {
|
||||
"platform": "hermes", "ct": 100, "ip": "192.168.68.24",
|
||||
"gateway_state": mumuni_data.get("gateway_state", "unknown"),
|
||||
"telegram_state": mumuni_platforms.get("telegram", {}).get("state", "unknown"),
|
||||
"zulip_state": mumuni_platforms.get("zulip", {}).get("state", "not_installed"),
|
||||
"email_state": mumuni_platforms.get("email", {}).get("state", "unknown"),
|
||||
"hermes_version": "",
|
||||
}
|
||||
# Get Hermes version
|
||||
ver = ssh("192.168.68.24", "hermes --version 2>/dev/null | head -1")
|
||||
if ver:
|
||||
report["agents"]["mumuni"]["hermes_version"] = ver.split("·")[0].replace("Hermes Agent ","").strip()
|
||||
# Mumuni is deliberately absent from this digest: captain ruling 2026-09-10.
|
||||
# She moved off this host onto her own container (kagentz CT 105 on minipve,
|
||||
# 192.168.68.14, dedicated `hermes` user) and is monitored from her side. The
|
||||
# former probe ssh'd to 192.168.68.24 for the decommissioned deployment's
|
||||
# ~/.hermes/gateway_state.json, always read "unknown", and published a false
|
||||
# "mumuni:unknown" line in the agent table and the gateway-unknown issue
|
||||
# count of every digest. Do NOT re-add an .24 / gateway_state probe.
|
||||
|
||||
return report
|
||||
|
||||
@@ -539,12 +526,6 @@ th {{ color: #8b949e; font-weight: normal; }}
|
||||
zulip_state = "✅" if agent.get("zulip_state") == "connected" else ("❌" if agent.get("zulip_state") == "disconnected" else "⬜")
|
||||
gateway = agent.get("gateway_state", "?")
|
||||
processed = "DSH"
|
||||
elif name == "mumuni":
|
||||
zulip_state = "⬜" if agent.get("zulip_state") == "not_installed" else ("✅" if agent.get("zulip_state") == "connected" else "⬜")
|
||||
gateway = agent.get("gateway_state", "?")
|
||||
tg = "✅" if agent.get("telegram_state") == "connected" else "❌"
|
||||
ver = agent.get("hermes_version", "")
|
||||
processed = f"TG:{tg} v{ver}"
|
||||
else:
|
||||
zulip_state = "⬜"
|
||||
gateway = agent.get("gateway_state", "?")
|
||||
|
||||
+14
-19
@@ -1,7 +1,10 @@
|
||||
#!/bin/bash
|
||||
# /root/scripts/zulip-monitor.sh — Zulip Mesh Health Monitor
|
||||
# Implements zulip-health.prose.md v2
|
||||
# Implements zulip-health.prose.md v3
|
||||
# Runs every 15 min via cron. Alerts: Zulip private DM to the owner plus a stream post to #agent-hub on topic 'zulip-health'.
|
||||
# Legs: global Zulip server, Platform A pi/Abiba (the Zulip bridge), Platform B
|
||||
# Tanko (DSH), Platform C Agent Zero (kagentz). The former Platform B Hermes
|
||||
# agent leg is retired — see the note after the Tanko leg.
|
||||
set -euo pipefail
|
||||
|
||||
ZULIP_SITE="https://chat.sysloggh.net"
|
||||
@@ -21,7 +24,8 @@ notify() {
|
||||
|
||||
# Zulip DM to owner
|
||||
local content="${severity} Zulip Monitor: ${msg}"
|
||||
local form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
|
||||
local form
|
||||
form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
|
||||
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
|
||||
-u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
|
||||
-d "${form}" > /dev/null 2>&1 || true
|
||||
@@ -141,23 +145,14 @@ else
|
||||
esac
|
||||
fi
|
||||
|
||||
# ── Platform B: Hermes (Mumuni) ──
|
||||
MUMUNI_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.24 \
|
||||
"cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
|
||||
MUMUNI_ZULIP=$(echo "$MUMUNI_STATE" | python3 -c "
|
||||
import sys,json
|
||||
d=json.load(sys.stdin)
|
||||
p=d.get('platforms',{}).get('zulip',{})
|
||||
print(p.get('state','unknown'))
|
||||
" 2>/dev/null)
|
||||
|
||||
if [ "$MUMUNI_ZULIP" != "connected" ]; then
|
||||
notify "🔴" "Mumuni (Hermes) Zulip state: $MUMUNI_ZULIP"
|
||||
ISSUES=$((ISSUES + 1))
|
||||
echo " Mumuni: ❌ state=$MUMUNI_ZULIP" >> "$LOG"
|
||||
else
|
||||
echo " Mumuni: ✅ Zulip connected" >> "$LOG"
|
||||
fi
|
||||
# ── Removed: the former "Platform B: Hermes" agent leg ──
|
||||
# Captain ruling 2026-09-10: that agent moved off this host onto her own
|
||||
# container (kagentz CT 105 on minipve, dedicated `hermes` user) and is now
|
||||
# monitored on her side — see the out-of-scope note in zulip-health.prose.md.
|
||||
# The old leg ssh'd to her former CT 100 deployment and read its Hermes gateway
|
||||
# state, which reported "unknown" on every run and posted a false 🔴 DM plus an
|
||||
# #agent-hub stream alert. Do NOT re-add a probe for her: this monitor must
|
||||
# never contact her former host.
|
||||
|
||||
# ── Platform C: Agent Zero (kagentz) ──
|
||||
AZ_A2A=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \
|
||||
|
||||
Reference in New Issue
Block a user