prose-contracts/zulip-health.prose.md

kind, name, description, title, version, runtime_contract, agent

kind	name	description	title	version	runtime_contract	agent
responsibility	zulip-health	Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.	Zulip Mesh Health Monitor — Multi-Platform	3.0.0	2	abiba

Zulip Mesh Health Monitor

Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero). Runs every 15 minutes in the background. Also triggers on session start.

Requires

• Zulip API key for abiba-bot@chat.sysloggh.net in $ZULIP_API_KEY
• SSH access to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
• PM2 on localhost for pi process management
• Network access to chat.sysloggh.net, localhost:9200
• Write access to /root/zulip-health-monitor.log and /tmp/zulip-monitor-debounce
• Relay access via RA-H OS MCP for alert delivery

Maintains

• zulip_server_status: "healthy" | "down"
• agents: map of per-platform agent snapshots (see schema below)
• last_check: timestamp — When the last full diagnostic ran
• overall_severity: "healthy" | "degraded" | "critical"
• restart_debounce: timestamp — Last restart action (enforces 300s minimum)

Per-Agent Snapshot Schema

{
  "abiba": {
    "platform": "pi",
    "connected": true,
    "response_pipeline": "healthy",
    "queue_healthy": true,
    "pm2_status": "online",
    "echo_loop_bot_msgs_15min": 12,
    "edit_fail_rate_pct": 3,
    "severity": "healthy"
  },
  "tanko": {
    "platform": "hermes",
    "zulip_state": "connected",
    "heartbeat_age_seconds": 45,
    "gateway_pid": 1234,
    "edit_fail_rate_pct": 0,
    "severity": "healthy"
  }
}

Postconditions

• Every platform is independently checked; one failure doesn't block others
• Restart actions respect 300s debounce window
• Critical conditions generate relay alerts to user
• All checks logged to /root/zulip-health-monitor.log with timestamps

Strategies

When Zulip server is unreachable

Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert.

When all agents are silent

Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait.

When restart is indicated but debounce window hasn't passed

Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions.

When a platform agent is unreachable via SSH

Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.

Severity escalation

Streaming Support (2026-07-05)

Zulip agents now support progressive message editing during agent generation. When a Hermes agent (Tanko, Mumuni) processes a message, the response is streamed in real-time via Zulip's PATCH /api/v1/messages/{id} API:

• Adapter implements edit_message() using _api_patch() helper
• Gateway stream consumer progressively edits the Zulip message
• User sees real-time agent thinking instead of waiting for full response
• Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active

Verification

# Check if agent has streaming:
grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
# Must return 1 — streaming is active

• Single agent warning → log only
• Single agent critical → relay message to user
• Two or more agents critical → immediate relay + attempt auto-recovery
• All agents critical + server up → Netbird/DNS likely down

Invariants

• Never restart more than once per 300s per agent
• Never restart if PM2 crashes > 10/h — alert user instead
• Never send duplicate alerts — check last alert timestamp before relaying
• Health checks are read-only — diagnostics don't mutate state except for logged restarts
• Respect Zulip API rate limits — no more than 200 requests in rapid succession

Continuity

• On session start: Run full diagnostic pass
• Every 15 minutes: Scheduled background check while Abiba is running
• On zulip-status command: Run on-demand and report to user
• On critical alert: Escalate to relay message immediately, don't wait for schedule

Execution

Step 1: Zulip Server Liveness

curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'

Expected: 200. If not → mark zulip_server_status: "down", skip per-platform checks, alert.

Step 2: Platform A — pi (Abiba, localhost)

A1: Health Endpoint

Fetch http://localhost:9200/health as JSON. Check:

Field	Healthy	Critical
connected	true	false
response_pipeline	healthy	blocked
queue_healthy	true	false
stuck	false	true
idle_seconds	< 1800	≥ 1800
pending_count	0	> 0 with response_pipeline: blocked
agent_busy_duration_seconds	< 300	≥ 600
last_error	null	non-null string
retry_count	0–2	3+
queue_id	non-null string	null

A2: PM2 Process

pm2 show abiba-zulip --no-color 2>/dev/null

Check: status=online, restarts < 10/h, uptime > 60s.

A3: Echo Loop Detection

grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5

100 skipped in 15min → info only (echo loop prevention working).

A4: Response Delivery

grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20

50% fail rate → critical — check editMessage API.

Platform A Actions

Condition	Action
response_pipeline: blocked	pm2 restart abiba-zulip
connected: false	pm2 restart abiba-zulip
queue_healthy: false	pm2 restart abiba-zulip
stuck: true	pm2 restart abiba-zulip
retry_count >= 3	pm2 restart abiba-zulip
response_pipeline: degraded	Monitor — no action, watchdog handles
last_error set	Log and monitor
Crash loop >10/h	Alert user

Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)

B1: Gateway State

ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"

Check platforms.zulip.state: connected ✅ | disconnected ❌ | error ❌ | missing → not installed.

B2: Agent Process

ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"

Gateway PID should exist with uptime > 60s. Dual-gateway detection: if more than one gateway run process is found, the gateway has a collision (typically one --force and one --replace process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (pm2 restart mumuni-zulip). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.

B3: Heartbeat Verification

ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"

Expected: recent heartbeat (within 5 min), polls=N incrementing. Silence > 300s → warning. Silence > 600s → critical.

B4: Response Delivery

ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"

50% fail rate → critical.

Platform B Actions

Condition	Action
zulip.state != "connected"	ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
No heartbeat in 10min	Same as above
Failed to finalize > 50%	Check PATCH API, Zulip server
Response empty/short	Check A2A endpoint / LiteLLM model

Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)

C1: A2A Server Health

ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"

Expected: {"name":"kagentz",...}. Connection refused → A2A server down.

C2: Adapter Process

ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"

Adapter should be running. Missing → restart inside container.

C3: Heartbeat & Queue

ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"

Check: processed=N incrementing, silence < 600s, reconnects ≈ 0.

C4: A2A Response Verification

ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
  -H 'Content-Type: application/json' \
  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"

Expected: task ID with "working" status. Poll for completion with tasks/get.

Platform C Actions

Condition	Action
A2A .well-known/agent.json fails	docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"
Adapter process missing	Restart adapter inside container with env vars
Silence > 600s	Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID)
LiteLLM 401	Check API key in a2a_agent.py LITELLM_KEY

Step 5: Global Checks

Cross-Agent Echo Loop Detection

Check each agent's log for excessive bot-to-bot chatter:

• Abiba: Skipped.*bot msgs count
• Tanko/Mumuni: Repeated DM exchanges between bots
• kagentz: Adapter log for bot DMs being processed

If any bot processes >50 bot-originated messages in 15min → warning.

Step 6: Compile and Report

1. Compile all platform checks and severity
2. Determine overall_severity from worst per-agent severity
3. If restart action needed, check /tmp/zulip-monitor-debounce — apply only if >300s since last restart
4. Log full diagnostic to /root/zulip-health-monitor.log with timestamp
5. If any agent critical or >2 degraded: send relay message to user
6. Update last_check timestamp in ### Maintains snapshot

Restart Debounce

All restart actions MUST debounce: minimum 300s between restarts. Track via /tmp/zulip-monitor-debounce (unix timestamp of last restart).

---

History

Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix

Root Cause: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), abiba-bot hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.

Fixes — Extension (index.js):

1. Rotation extended to 55 min (from 25) with ±90s jitter — avoids aligning with cron/monitor cycles
2. Rate-limit-aware retry — if connection fails with 429, skip the retry loop entirely, wait 120s, try once

Fixes — Monitor (zulip-monitor.sh): 3. Smart Triage instead of instant restart:

• Rate-limit detection: If error log shows recent 429s, wait — don't add more API load
• Self-healing detection: If retry_count is 1-2, extension is already retrying — don't interrupt
• Rotation window awareness: If queue_age is 25-35min, disconnection is likely transient rotation — wait
• Persistent failure threshold: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
• Debounce gate: Skip all checks entirely if recently restarted

Gen 4 (2026-06-29) — Response Pipeline Fix

Root Cause: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's agent_end never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed connected: true, stuck: false while user experienced complete silence.

Four-Layer Defense:

1. Response Watchdog — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue
2. Queue Health Ping — Every 2min /api/v1/events?dont_block=true to detect silent expiry
3. Proactive Queue Rotation — New queue every 25min + 600 empty poll reconnection threshold
4. Health Endpoint v2 — Added response_pipeline, pending_count, queue_healthy, agent_busy_duration_seconds

Gen 3 (2026-06-15) — Stuck Detection

Added stuck: bool and idle_seconds to health endpoint. Monitor restarts on stuck: true. Added 300s restart debounce. Queue re-registers after 30min of no events.