diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 821eb91..eda223c 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -1,257 +1,313 @@ --- -kind: contract +kind: responsibility name: zulip-health title: Zulip Mesh Health Monitor — Multi-Platform -version: 2.0.0 +version: 3.0.0 +runtime_contract: 2 agent: abiba -triggers: - - on startup - - every 15 minutes while running - - on zulip-status command --- -# Zulip Mesh Health Monitor — Multi-Platform +# Zulip Mesh Health Monitor -Monitors ALL Zulip-connected agents across three platforms. +Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero). Runs every 15 minutes in the background. Also triggers on session start. -## Platform Overview +## Requires -| Platform | Agents | Bot | Adapter | Health Check | -|----------|--------|-----|---------|-------------| -| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` | -| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` | -| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` | +- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` +- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14) +- **PM2** on localhost for pi process management +- **Network access** to `chat.sysloggh.net`, `localhost:9200` +- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` +- **Relay access** via RA-H OS MCP for alert delivery -## Phase 1: Zulip Server Liveness (All Platforms) +## Maintains + +- zulip_server_status: "healthy" | "down" +- agents: map of per-platform agent snapshots (see schema below) +- last_check: timestamp — When the last full diagnostic ran +- overall_severity: "healthy" | "degraded" | "critical" +- restart_debounce: timestamp — Last restart action (enforces 300s minimum) + +### Per-Agent Snapshot Schema + +```json +{ + "abiba": { + "platform": "pi", + "connected": true, + "response_pipeline": "healthy", + "queue_healthy": true, + "pm2_status": "online", + "echo_loop_bot_msgs_15min": 12, + "edit_fail_rate_pct": 3, + "severity": "healthy" + }, + "tanko": { + "platform": "hermes", + "zulip_state": "connected", + "heartbeat_age_seconds": 45, + "gateway_pid": 1234, + "edit_fail_rate_pct": 0, + "severity": "healthy" + } +} +``` + +### Postconditions + +- Every platform is independently checked; one failure doesn't block others +- Restart actions respect 300s debounce window +- Critical conditions generate relay alerts to user +- All checks logged to `/root/zulip-health-monitor.log` with timestamps + +## Strategies + +### When Zulip server is unreachable +Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert. + +### When all agents are silent +Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait. + +### When restart is indicated but debounce window hasn't passed +Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions. + +### When a platform agent is unreachable via SSH +Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks. + +### Severity escalation +- Single agent warning → log only +- Single agent critical → relay message to user +- Two or more agents critical → immediate relay + attempt auto-recovery +- All agents critical + server up → Netbird/DNS likely down + +## Invariants + +- **Never restart more than once per 300s** per agent +- **Never restart if PM2 crashes > 10/h** — alert user instead +- **Never send duplicate alerts** — check last alert timestamp before relaying +- **Health checks are read-only** — diagnostics don't mutate state except for logged restarts +- **Respect Zulip API rate limits** — no more than 200 requests in rapid succession + +## Continuity + +- **On session start**: Run full diagnostic pass +- **Every 15 minutes**: Scheduled background check while Abiba is running +- **On `zulip-status` command**: Run on-demand and report to user +- **On critical alert**: Escalate to relay message immediately, don't wait for schedule + +## Execution + +### Step 1: Zulip Server Liveness ```bash curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \ -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' ``` -Expected: `200`. Anything else → Server issue, alert maintainer. +Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert. ---- +### Step 2: Platform A — pi (Abiba, localhost) -## Platform A: pi (Abiba — CT 100) +**A1: Health Endpoint** -### A1: Health Endpoint - -```js -const health = await fetch("http://localhost:9200/health").then(r => r.json()); -``` +Fetch `http://localhost:9200/health` as JSON. Check: | Field | Healthy | Critical | |-------|---------|----------| | `connected` | `true` | `false` | -| `stuck` | `false` | `true` (Gen 3) | -| `idle_seconds` | < 1800 | >= 1800 | +| `response_pipeline` | `healthy` | `blocked` | +| `queue_healthy` | `true` | `false` | +| `stuck` | `false` | `true` | +| `idle_seconds` | < 1800 | ≥ 1800 | +| `pending_count` | 0 | > 0 with `response_pipeline: blocked` | +| `agent_busy_duration_seconds` | < 300 | ≥ 600 | | `last_error` | `null` | non-null string | -| `retry_count` | 0-2 | 3+ | +| `retry_count` | 0–2 | 3+ | | `queue_id` | non-null string | `null` | -### A2: PM2 Process +**A2: PM2 Process** ```bash pm2 show abiba-zulip --no-color 2>/dev/null ``` -Check: `status=online`, `restarts < 10/h`, `uptime > 60s` +Check: `status=online`, `restarts < 10/h`, `uptime > 60s`. -### A3: Echo Loop Detection +**A3: Echo Loop Detection** ```bash grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5 ``` -> 100 skipped in 15min → 🟢 Info only (echo loop prevention working) +> 100 skipped in 15min → info only (echo loop prevention working). -### A4: Response Delivery +**A4: Response Delivery** ```bash grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20 ``` -> 50% fail rate → 🔴 Critical — check editMessage API +> 50% fail rate → critical — check editMessage API. -### Actions +**Platform A Actions** | Condition | Action | |-----------|--------| +| `response_pipeline: blocked` | `pm2 restart abiba-zulip` | | `connected: false` | `pm2 restart abiba-zulip` | -| `stuck: true` | `pm2 restart abiba-zulip` (Gen 3) | +| `queue_healthy: false` | `pm2 restart abiba-zulip` | +| `stuck: true` | `pm2 restart abiba-zulip` | | `retry_count >= 3` | `pm2 restart abiba-zulip` | +| `response_pipeline: degraded` | Monitor — no action, watchdog handles | | `last_error` set | Log and monitor | | Crash loop >10/h | Alert user | ---- +### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123) -## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123) - -### B1: Gateway State +**B1: Gateway State** ```bash ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" ``` -Check `platforms.zulip.state`: +Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed. -| Value | Meaning | Action | -|-------|---------|--------| -| `"connected"` | ✅ Healthy | None | -| `"disconnected"` | ❌ Disconnected | Check `last_error` | -| `"error"` | ❌ Error | Check `error_message`, restart gateway | -| missing | ❌ Not installed | Run deploy scripts | - -### B2: Agent Process +**B2: Agent Process** ```bash -ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep" +ssh root@ "ps aux | grep 'gateway run' | grep -v grep" ``` -Gateway PID should exist and uptime > 60s. +Gateway PID should exist with uptime > 60s. -### B3: Heartbeat Verification +**B3: Heartbeat Verification** ```bash -ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" +ssh root@ "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" ``` -Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing. -If silence > 300s → 🟡 Warning (queue may be stuck). -If silence > 600s → 🔴 Critical (queue expired, adapter needs restart). +Expected: recent heartbeat (within 5 min), `polls=N` incrementing. +Silence > 300s → warning. Silence > 600s → critical. -### B4: Response Delivery +**B4: Response Delivery** ```bash -ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10" +ssh root@ "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10" ``` -> 50% fail rate → 🔴 Critical +> 50% fail rate → critical. -### Actions +**Platform B Actions** | Condition | Action | |-----------|--------| | `zulip.state != "connected"` | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | -| No heartbeat in 10min | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | +| No heartbeat in 10min | Same as above | | `Failed to finalize` > 50% | Check PATCH API, Zulip server | | Response empty/short | Check A2A endpoint / LiteLLM model | -## Restart Debounce (Gen 3) +### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) + +**C1: A2A Server Health** + +```bash +ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json" +``` + +Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down. + +**C2: Adapter Process** + +```bash +ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep" +``` + +Adapter should be running. Missing → restart inside container. + +**C3: Heartbeat & Queue** + +```bash +ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3" +``` + +Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0. + +**C4: A2A Response Verification** + +```bash +ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \ + -H 'Content-Type: application/json' \ + -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'" +``` + +Expected: task ID with "working" status. Poll for completion with `tasks/get`. + +**Platform C Actions** + +| Condition | Action | +|-----------|--------| +| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` | +| Adapter process missing | Restart adapter inside container with env vars | +| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) | +| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` | + +### Step 5: Global Checks + +**Cross-Agent Echo Loop Detection** + +Check each agent's log for excessive bot-to-bot chatter: +- Abiba: `Skipped.*bot msgs` count +- Tanko/Mumuni: Repeated DM exchanges between bots +- kagentz: Adapter log for bot DMs being processed + +If any bot processes >50 bot-originated messages in 15min → warning. + +### Step 6: Compile and Report + +1. Compile all platform checks and severity +2. Determine `overall_severity` from worst per-agent severity +3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart +4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp +5. If any agent critical or >2 degraded: send relay message to user +6. Update `last_check` timestamp in `### Maintains` snapshot + +### Restart Debounce All restart actions MUST debounce: minimum 300s between restarts. Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart). --- -## Platform C: Agent Zero (kagentz — CT 105) +## History -### C1: A2A Server Health +### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix -```bash -ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json" -``` +**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours. -Expected: `{"name":"kagentz",...}` — JSON response with agent identity. -Connection refused → A2A server is down. +**Fixes — Extension (`index.js`)**: +1. **Rotation extended to 55 min** (from 25) with **±90s jitter** — avoids aligning with cron/monitor cycles +2. **Rate-limit-aware retry** — if connection fails with 429, skip the retry loop entirely, wait 120s, try once -### C2: Adapter Process +**Fixes — Monitor (`zulip-monitor.sh`)**: +3. **Smart Triage** instead of instant restart: + - **Rate-limit detection**: If error log shows recent 429s, wait — don't add more API load + - **Self-healing detection**: If `retry_count` is 1-2, extension is already retrying — don't interrupt + - **Rotation window awareness**: If `queue_age` is 25-35min, disconnection is likely transient rotation — wait + - **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress + - **Debounce gate**: Skip all checks entirely if recently restarted -```bash -ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep" -``` +### Gen 4 (2026-06-29) — Response Pipeline Fix -Adapter should be running. If missing, restart. +**Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence. -### C3: Heartbeat & Queue +**Four-Layer Defense**: +1. **Response Watchdog** — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue +2. **Queue Health Ping** — Every 2min `/api/v1/events?dont_block=true` to detect silent expiry +3. **Proactive Queue Rotation** — New queue every 25min + 600 empty poll reconnection threshold +4. **Health Endpoint v2** — Added `response_pipeline`, `pending_count`, `queue_healthy`, `agent_busy_duration_seconds` -```bash -ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3" -``` +### Gen 3 (2026-06-15) — Stuck Detection -Check: -- `processed=N` incrementing when DMs arrive -- `silence < 600s` (queue expiry timeout) -- `reconnects` — should be 0 under normal operation - -### C4: A2A Response Verification - -```bash -# Test A2A sends a task -ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \ - -H 'Content-Type: application/json' \ - -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'" - -# Expected: task ID returned with "working" status -# Then poll for completion with tasks/get -``` - -### Actions - -| Condition | Action | -|-----------|--------| -| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` | -| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` | -| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) | -| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` | - ---- - -## Global Checks - -### Zulip Server - -```bash -curl -s https://chat.sysloggh.net/api/v1/server_settings \ - -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}" -``` - -Expected: `200` - -### Cross-Agent Echo Loop Detection - -Check each agent's log for excessive bot-to-bot chatter: -- Abiba: `Skipped.*bot msgs` count -- Tanko/Mumuni: Check for repeated DM exchanges between bots -- kagentz: Check adapter log for bot DMs being processed - -If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning. - ---- - -## Consolidated Action Matrix - -| Condition | Severity | Action | -|-----------|----------|--------| -| Zulip server not 200 | 🔴 Critical | Alert maintainer | -| All agents silent | 🔴 Critical | Zulip server likely down | -| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` | -| pi: edit fail >50% | 🔴 Critical | Check editMessage API | -| Hermes: zulip disconnected | 🔴 Critical | Restart gateway | -| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway | -| Az: A2A server down | 🔴 Critical | Restart inside container | -| Az: adapter down | 🔴 Critical | Restart adapter | -| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) | -| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) | - -## Logging - -All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps. -Critical alerts sent as relay messages to user. - -## Gen 3 Changes (2026-06-28) - -### Problem -- Bot shows `connected: true` but silently stops processing messages (Zulip event queue expires without BAD_EVENT_QUEUE_ID error) -- `messages_processed=0` for 45+ min while health endpoint reports healthy -- Multiple overlapping monitors cause restart death spiral - -### Fixes -1. **Stuck detection**: Health endpoint now reports `stuck: bool`, `idle_seconds`, `last_activity_time` -2. **Proactive queue recovery**: Re-register queue after 30min of no events (even without BAD_EVENT_QUEUE_ID) -3. **Monitor stuck awareness**: `zulip-monitor.sh` restarts on `stuck: true` not just `connected: false` -4. **Restart debounce**: 300s minimum between restarts (via `/tmp/zulip-monitor-debounce`) -5. **Broader queue expiry detection**: Matches multiple error message patterns +Added `stuck: bool` and `idle_seconds` to health endpoint. Monitor restarts on `stuck: true`. Added 300s restart debounce. Queue re-registers after 30min of no events.