prose-contracts/zulip-health.prose.md

kind, name, title, version, agent, triggers

kind	name	title	version	agent	triggers
contract	zulip-health	Zulip Mesh Health Monitor — Multi-Platform	2.0.0	abiba	on startup every 15 minutes while running on zulip-status command

Zulip Mesh Health Monitor — Multi-Platform

Monitors ALL Zulip-connected agents across three platforms. Runs every 15 minutes in the background. Also triggers on session start.

Platform Overview

Platform	Agents	Bot	Adapter	Health Check
pi	Abiba (CT 100)	abiba-bot	/root/.pi/agent/extensions/zulip/index.js	:9200/health
Hermes	Tanko (CT 122), Mumuni (CT 123)	tanko-bot, mumuni-bot	~/.hermes/plugins/platforms/zulip/	gateway_state.json
Agent Zero	kagentz (CT 105)	kagentz-bot	Docker container, /a0/usr/kagentz-zulip/	A2A endpoint :8001

Phase 1: Zulip Server Liveness (All Platforms)

curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'

Expected: 200. Anything else → Server issue, alert maintainer.

---

Platform A: pi (Abiba — CT 100)

A1: Health Endpoint

const health = await fetch("http://localhost:9200/health").then(r => r.json());

Field	Healthy	Critical
connected	true	false
stuck	false	true (Gen 3)
idle_seconds	< 1800	>= 1800
last_error	null	non-null string
retry_count	0-2	3+
queue_id	non-null string	null

A2: PM2 Process

pm2 show abiba-zulip --no-color 2>/dev/null

Check: status=online, restarts < 10/h, uptime > 60s

A3: Echo Loop Detection

grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5

100 skipped in 15min → 🟢 Info only (echo loop prevention working)

A4: Response Delivery

grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20

50% fail rate → 🔴 Critical — check editMessage API

Actions

Condition	Action
connected: false	pm2 restart abiba-zulip
stuck: true	pm2 restart abiba-zulip (Gen 3)
retry_count >= 3	pm2 restart abiba-zulip
last_error set	Log and monitor
Crash loop >10/h	Alert user

---

Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123)

B1: Gateway State

ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"

Check platforms.zulip.state:

Value	Meaning	Action
"connected"	✅ Healthy	None
"disconnected"	❌ Disconnected	Check last_error
"error"	❌ Error	Check error_message, restart gateway
missing	❌ Not installed	Run deploy scripts

B2: Agent Process

ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep"

Gateway PID should exist and uptime > 60s.

B3: Heartbeat Verification

ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"

Expected: recent heartbeat (within 5 min) showing polls=N incrementing. If silence > 300s → 🟡 Warning (queue may be stuck). If silence > 600s → 🔴 Critical (queue expired, adapter needs restart).

B4: Response Delivery

ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"

50% fail rate → 🔴 Critical

Actions

Condition	Action
zulip.state != "connected"	ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
No heartbeat in 10min	ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
Failed to finalize > 50%	Check PATCH API, Zulip server
Response empty/short	Check A2A endpoint / LiteLLM model

Restart Debounce (Gen 3)

All restart actions MUST debounce: minimum 300s between restarts. Track via /tmp/zulip-monitor-debounce (unix timestamp of last restart).

---

Platform C: Agent Zero (kagentz — CT 105)

C1: A2A Server Health

ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"

Expected: {"name":"kagentz",...} — JSON response with agent identity. Connection refused → A2A server is down.

C2: Adapter Process

ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"

Adapter should be running. If missing, restart.

C3: Heartbeat & Queue

ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"

Check:

• processed=N incrementing when DMs arrive
• silence < 600s (queue expiry timeout)
• reconnects — should be 0 under normal operation

C4: A2A Response Verification

# Test A2A sends a task
ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
  -H 'Content-Type: application/json' \
  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"

# Expected: task ID returned with "working" status
# Then poll for completion with tasks/get

Actions

Condition	Action
A2A .well-known/agent.json fails	docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"
Adapter process missing	docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"
Silence > 600s (queue expiry)	Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID)
LiteLLM 401	Check API key in a2a_agent.py LITELLM_KEY

---

Global Checks

Zulip Server

curl -s https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}"

Expected: 200

Cross-Agent Echo Loop Detection

Check each agent's log for excessive bot-to-bot chatter:

• Abiba: Skipped.*bot msgs count
• Tanko/Mumuni: Check for repeated DM exchanges between bots
• kagentz: Check adapter log for bot DMs being processed

If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.

---

Consolidated Action Matrix

Condition	Severity	Action
Zulip server not 200	🔴 Critical	Alert maintainer
All agents silent	🔴 Critical	Zulip server likely down
pi: connected false	🔴 Critical	pm2 restart abiba-zulip
pi: edit fail >50%	🔴 Critical	Check editMessage API
Hermes: zulip disconnected	🔴 Critical	Restart gateway
Hermes: no heartbeat 10min	🔴 Critical	Restart gateway
Az: A2A server down	🔴 Critical	Restart inside container
Az: adapter down	🔴 Critical	Restart adapter
Az: silence >600s	🟡 Warning	Queue expired (auto-recover)
Echo loop detected	🟢 Info	Auto-mitigated (bot filtering)

Logging

All diagnostics logged to /root/zulip-health-monitor.log with timestamps. Critical alerts sent as relay messages to user.

Gen 3 Changes (2026-06-28)

Problem

• Bot shows connected: true but silently stops processing messages (Zulip event queue expires without BAD_EVENT_QUEUE_ID error)
• messages_processed=0 for 45+ min while health endpoint reports healthy
• Multiple overlapping monitors cause restart death spiral

Fixes

1. Stuck detection: Health endpoint now reports stuck: bool, idle_seconds, last_activity_time
2. Proactive queue recovery: Re-register queue after 30min of no events (even without BAD_EVENT_QUEUE_ID)
3. Monitor stuck awareness: zulip-monitor.sh restarts on stuck: true not just connected: false
4. Restart debounce: 300s minimum between restarts (via /tmp/zulip-monitor-debounce)
5. Broader queue expiry detection: Matches multiple error message patterns