diff --git a/zulip-health.prose.md b/zulip-health.prose.md new file mode 100644 index 0000000..715f682 --- /dev/null +++ b/zulip-health.prose.md @@ -0,0 +1,235 @@ +--- +kind: contract +name: zulip-health +title: Zulip Mesh Health Monitor — Multi-Platform +version: 2.0.0 +agent: abiba +triggers: + - on startup + - every 15 minutes while running + - on zulip-status command +--- + +# Zulip Mesh Health Monitor — Multi-Platform + +Monitors ALL Zulip-connected agents across three platforms. +Runs every 15 minutes in the background. Also triggers on session start. + +## Platform Overview + +| Platform | Agents | Bot | Adapter | Health Check | +|----------|--------|-----|---------|-------------| +| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` | +| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` | +| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` | + +## Phase 1: Zulip Server Liveness (All Platforms) + +```bash +curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \ + -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' +``` + +Expected: `200`. Anything else → Server issue, alert maintainer. + +--- + +## Platform A: pi (Abiba — CT 100) + +### A1: Health Endpoint + +```js +const health = await fetch("http://localhost:9200/health").then(r => r.json()); +``` + +| Field | Healthy | Critical | +|-------|---------|----------| +| `connected` | `true` | `false` | +| `last_error` | `null` | non-null string | +| `retry_count` | 0-2 | 3+ | +| `queue_id` | non-null string | `null` | + +### A2: PM2 Process + +```bash +pm2 show abiba-zulip --no-color 2>/dev/null +``` + +Check: `status=online`, `restarts < 10/h`, `uptime > 60s` + +### A3: Echo Loop Detection + +```bash +grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5 +``` + +> 100 skipped in 15min → 🟢 Info only (echo loop prevention working) + +### A4: Response Delivery + +```bash +grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20 +``` + +> 50% fail rate → 🔴 Critical — check editMessage API + +### Actions + +| Condition | Action | +|-----------|--------| +| `connected: false` | `pm2 restart abiba-zulip` | +| `retry_count >= 3` | `pm2 restart abiba-zulip` | +| `last_error` set | Log and monitor | +| Crash loop >10/h | Alert user | + +--- + +## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123) + +### B1: Gateway State + +```bash +ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" +ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" +``` + +Check `platforms.zulip.state`: + +| Value | Meaning | Action | +|-------|---------|--------| +| `"connected"` | ✅ Healthy | None | +| `"disconnected"` | ❌ Disconnected | Check `last_error` | +| `"error"` | ❌ Error | Check `error_message`, restart gateway | +| missing | ❌ Not installed | Run deploy scripts | + +### B2: Agent Process + +```bash +ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep" +``` + +Gateway PID should exist and uptime > 60s. + +### B3: Heartbeat Verification + +```bash +ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" +``` + +Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing. +If silence > 300s → 🟡 Warning (queue may be stuck). +If silence > 600s → 🔴 Critical (queue expired, adapter needs restart). + +### B4: Response Delivery + +```bash +ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10" +``` + +> 50% fail rate → 🔴 Critical + +### Actions + +| Condition | Action | +|-----------|--------| +| `zulip.state != "connected"` | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | +| No heartbeat in 10min | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | +| `Failed to finalize` > 50% | Check PATCH API, Zulip server | +| Response empty/short | Check A2A endpoint / LiteLLM model | + +--- + +## Platform C: Agent Zero (kagentz — CT 105) + +### C1: A2A Server Health + +```bash +ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json" +``` + +Expected: `{"name":"kagentz",...}` — JSON response with agent identity. +Connection refused → A2A server is down. + +### C2: Adapter Process + +```bash +ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep" +``` + +Adapter should be running. If missing, restart. + +### C3: Heartbeat & Queue + +```bash +ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3" +``` + +Check: +- `processed=N` incrementing when DMs arrive +- `silence < 600s` (queue expiry timeout) +- `reconnects` — should be 0 under normal operation + +### C4: A2A Response Verification + +```bash +# Test A2A sends a task +ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \ + -H 'Content-Type: application/json' \ + -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'" + +# Expected: task ID returned with "working" status +# Then poll for completion with tasks/get +``` + +### Actions + +| Condition | Action | +|-----------|--------| +| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` | +| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` | +| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) | +| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` | + +--- + +## Global Checks + +### Zulip Server + +```bash +curl -s https://chat.sysloggh.net/api/v1/server_settings \ + -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}" +``` + +Expected: `200` + +### Cross-Agent Echo Loop Detection + +Check each agent's log for excessive bot-to-bot chatter: +- Abiba: `Skipped.*bot msgs` count +- Tanko/Mumuni: Check for repeated DM exchanges between bots +- kagentz: Check adapter log for bot DMs being processed + +If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning. + +--- + +## Consolidated Action Matrix + +| Condition | Severity | Action | +|-----------|----------|--------| +| Zulip server not 200 | 🔴 Critical | Alert maintainer | +| All agents silent | 🔴 Critical | Zulip server likely down | +| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` | +| pi: edit fail >50% | 🔴 Critical | Check editMessage API | +| Hermes: zulip disconnected | 🔴 Critical | Restart gateway | +| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway | +| Az: A2A server down | 🔴 Critical | Restart inside container | +| Az: adapter down | 🔴 Critical | Restart adapter | +| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) | +| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) | + +## Logging + +All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps. +Critical alerts sent as relay messages to user.