SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(zulip-health): v3 rewrite — full OpenProse responsibility structure

Browse Source 
- kind: contract -> responsibility (valid kind fix)
- Add Requires, Maintains (JSON schema), Strategies, Invariants, Continuity
- Merge Platform A/B/C into structured Step 1-6 under Execution
- Add response_pipeline, queue_healthy, agent_busy_duration diagnostics
- Add restart debounce (300s) and PM2 crash-loop safety invariants
- Add runtime_contract: 2 versioning
This commit is contained in:
Abiba
2026-07-02 20:47:17 +00:00
parent b0c0331727
commit 57178e4647
1 changed files with 215 additions and 159 deletions
Download Patch File Download Diff File Expand all files Collapse all files
zulip-health.prose.md
+215 -159
View File
@@ -1,257 +1,313 @@
--- ---
kind: contract kind: responsibility
name: zulip-health name: zulip-health
title: Zulip Mesh Health Monitor — Multi-Platform title: Zulip Mesh Health Monitor — Multi-Platform
version: 2.0.0 version: 3.0.0
runtime_contract: 2
agent: abiba agent: abiba
triggers:
- on startup
- every 15 minutes while running
- on zulip-status command
--- ---
# Zulip Mesh Health Monitor — Multi-Platform # Zulip Mesh Health Monitor
Monitors ALL Zulip-connected agents across three platforms. Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
Runs every 15 minutes in the background. Also triggers on session start. Runs every 15 minutes in the background. Also triggers on session start.
## Platform Overview ## Requires
| Platform | Agents | Bot | Adapter | Health Check | - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|----------|--------|-----|---------|-------------| - **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` | - **PM2** on localhost for pi process management
| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` | - **Network access** to `chat.sysloggh.net`, `localhost:9200`
| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` | - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
- **Relay access** via RA-H OS MCP for alert delivery
## Phase 1: Zulip Server Liveness (All Platforms) ## Maintains
- zulip_server_status: "healthy" | "down"
- agents: map of per-platform agent snapshots (see schema below)
- last_check: timestamp — When the last full diagnostic ran
- overall_severity: "healthy" | "degraded" | "critical"
- restart_debounce: timestamp — Last restart action (enforces 300s minimum)
### Per-Agent Snapshot Schema
```json
{
"abiba": {
"platform": "pi",
"connected": true,
"response_pipeline": "healthy",
"queue_healthy": true,
"pm2_status": "online",
"echo_loop_bot_msgs_15min": 12,
"edit_fail_rate_pct": 3,
"severity": "healthy"
},
"tanko": {
"platform": "hermes",
"zulip_state": "connected",
"heartbeat_age_seconds": 45,
"gateway_pid": 1234,
"edit_fail_rate_pct": 0,
"severity": "healthy"
}
}
```
### Postconditions
- Every platform is independently checked; one failure doesn't block others
- Restart actions respect 300s debounce window
- Critical conditions generate relay alerts to user
- All checks logged to `/root/zulip-health-monitor.log` with timestamps
## Strategies
### When Zulip server is unreachable
Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert.
### When all agents are silent
Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait.
### When restart is indicated but debounce window hasn't passed
Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions.
### When a platform agent is unreachable via SSH
Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.
### Severity escalation
- Single agent warning → log only
- Single agent critical → relay message to user
- Two or more agents critical → immediate relay + attempt auto-recovery
- All agents critical + server up → Netbird/DNS likely down
## Invariants
- **Never restart more than once per 300s** per agent
- **Never restart if PM2 crashes > 10/h** — alert user instead
- **Never send duplicate alerts** — check last alert timestamp before relaying
- **Health checks are read-only** — diagnostics don't mutate state except for logged restarts
- **Respect Zulip API rate limits** — no more than 200 requests in rapid succession
## Continuity
- **On session start**: Run full diagnostic pass
- **Every 15 minutes**: Scheduled background check while Abiba is running
- **On `zulip-status` command**: Run on-demand and report to user
- **On critical alert**: Escalate to relay message immediately, don't wait for schedule
## Execution
### Step 1: Zulip Server Liveness
```bash ```bash
curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \ curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
-u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
``` ```
Expected: `200`. Anything else → Server issue, alert maintainer. Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.
--- ### Step 2: Platform A — pi (Abiba, localhost)
## Platform A: pi (Abiba — CT 100) **A1: Health Endpoint**
### A1: Health Endpoint Fetch `http://localhost:9200/health` as JSON. Check:
```js
const health = await fetch("http://localhost:9200/health").then(r => r.json());
```
| Field | Healthy | Critical | | Field | Healthy | Critical |
|-------|---------|----------| |-------|---------|----------|
| `connected` | `true` | `false` | | `connected` | `true` | `false` |
| `stuck` | `false` | `true` (Gen 3) | | `response_pipeline` | `healthy` | `blocked` |
| `idle_seconds` | < 1800 | >= 1800 | | `queue_healthy` | `true` | `false` |
| `stuck` | `false` | `true` |
| `idle_seconds` | < 1800 | ≥ 1800 |
| `pending_count` | 0 | > 0 with `response_pipeline: blocked` |
| `agent_busy_duration_seconds` | < 300 | ≥ 600 |
| `last_error` | `null` | non-null string | | `last_error` | `null` | non-null string |
| `retry_count` | 0-2 | 3+ | | `retry_count` | 0–2 | 3+ |
| `queue_id` | non-null string | `null` | | `queue_id` | non-null string | `null` |
### A2: PM2 Process **A2: PM2 Process**
```bash ```bash
pm2 show abiba-zulip --no-color 2>/dev/null pm2 show abiba-zulip --no-color 2>/dev/null
``` ```
Check: `status=online`, `restarts < 10/h`, `uptime > 60s` Check: `status=online`, `restarts < 10/h`, `uptime > 60s`.
### A3: Echo Loop Detection **A3: Echo Loop Detection**
```bash ```bash
grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5 grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5
``` ```
> 100 skipped in 15min → 🟢 Info only (echo loop prevention working) > 100 skipped in 15min → info only (echo loop prevention working).
### A4: Response Delivery **A4: Response Delivery**
```bash ```bash
grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20 grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20
``` ```
> 50% fail rate → 🔴 Critical — check editMessage API > 50% fail rate → critical — check editMessage API.
### Actions **Platform A Actions**
| Condition | Action | | Condition | Action |
|-----------|--------| |-----------|--------|
| `response_pipeline: blocked` | `pm2 restart abiba-zulip` |
| `connected: false` | `pm2 restart abiba-zulip` | | `connected: false` | `pm2 restart abiba-zulip` |
| `stuck: true` | `pm2 restart abiba-zulip` (Gen 3) | | `queue_healthy: false` | `pm2 restart abiba-zulip` |
| `stuck: true` | `pm2 restart abiba-zulip` |
| `retry_count >= 3` | `pm2 restart abiba-zulip` | | `retry_count >= 3` | `pm2 restart abiba-zulip` |
| `response_pipeline: degraded` | Monitor — no action, watchdog handles |
| `last_error` set | Log and monitor | | `last_error` set | Log and monitor |
| Crash loop >10/h | Alert user | | Crash loop >10/h | Alert user |
--- ### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123) **B1: Gateway State**
### B1: Gateway State
```bash ```bash
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
``` ```
Check `platforms.zulip.state`: Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
| Value | Meaning | Action | **B2: Agent Process**
|-------|---------|--------|
| `"connected"` | ✅ Healthy | None |
| `"disconnected"` | ❌ Disconnected | Check `last_error` |
| `"error"` | ❌ Error | Check `error_message`, restart gateway |
| missing | ❌ Not installed | Run deploy scripts |
### B2: Agent Process
```bash ```bash
ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep" ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
``` ```
Gateway PID should exist and uptime > 60s. Gateway PID should exist with uptime > 60s.
### B3: Heartbeat Verification **B3: Heartbeat Verification**
```bash ```bash
ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
``` ```
Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing. Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
If silence > 300s → 🟡 Warning (queue may be stuck). Silence > 300s → warning. Silence > 600s → critical.
If silence > 600s → 🔴 Critical (queue expired, adapter needs restart).
### B4: Response Delivery **B4: Response Delivery**
```bash ```bash
ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10" ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
``` ```
> 50% fail rate → 🔴 Critical > 50% fail rate → critical.
### Actions **Platform B Actions**
| Condition | Action | | Condition | Action |
|-----------|--------| |-----------|--------|
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | | `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
| No heartbeat in 10min | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | | No heartbeat in 10min | Same as above |
| `Failed to finalize` > 50% | Check PATCH API, Zulip server | | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
| Response empty/short | Check A2A endpoint / LiteLLM model | | Response empty/short | Check A2A endpoint / LiteLLM model |
## Restart Debounce (Gen 3) ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
**C1: A2A Server Health**
```bash
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
```
Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.
**C2: Adapter Process**
```bash
ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
```
Adapter should be running. Missing → restart inside container.
**C3: Heartbeat & Queue**
```bash
ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
```
Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.
**C4: A2A Response Verification**
```bash
ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
-H 'Content-Type: application/json' \
-d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
```
Expected: task ID with "working" status. Poll for completion with `tasks/get`.
**Platform C Actions**
| Condition | Action |
|-----------|--------|
| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
| Adapter process missing | Restart adapter inside container with env vars |
| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) |
| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
### Step 5: Global Checks
**Cross-Agent Echo Loop Detection**
Check each agent's log for excessive bot-to-bot chatter:
- Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni: Repeated DM exchanges between bots
- kagentz: Adapter log for bot DMs being processed
If any bot processes >50 bot-originated messages in 15min → warning.
### Step 6: Compile and Report
1. Compile all platform checks and severity
2. Determine `overall_severity` from worst per-agent severity
3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
5. If any agent critical or >2 degraded: send relay message to user
6. Update `last_check` timestamp in `### Maintains` snapshot
### Restart Debounce
All restart actions MUST debounce: minimum 300s between restarts. All restart actions MUST debounce: minimum 300s between restarts.
Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart). Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
--- ---
## Platform C: Agent Zero (kagentz — CT 105) ## History
### C1: A2A Server Health ### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
```bash **Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
```
Expected: `{"name":"kagentz",...}` — JSON response with agent identity. **Fixes — Extension (`index.js`)**:
Connection refused → A2A server is down. 1. **Rotation extended to 55 min** (from 25) with **±90s jitter** — avoids aligning with cron/monitor cycles
2. **Rate-limit-aware retry** — if connection fails with 429, skip the retry loop entirely, wait 120s, try once
### C2: Adapter Process **Fixes — Monitor (`zulip-monitor.sh`)**:
3. **Smart Triage** instead of instant restart:
- **Rate-limit detection**: If error log shows recent 429s, wait — don't add more API load
- **Self-healing detection**: If `retry_count` is 1-2, extension is already retrying — don't interrupt
- **Rotation window awareness**: If `queue_age` is 25-35min, disconnection is likely transient rotation — wait
- **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
- **Debounce gate**: Skip all checks entirely if recently restarted
```bash ### Gen 4 (2026-06-29) — Response Pipeline Fix
ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
```
Adapter should be running. If missing, restart. **Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
### C3: Heartbeat & Queue **Four-Layer Defense**:
1. **Response Watchdog** — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue
2. **Queue Health Ping** — Every 2min `/api/v1/events?dont_block=true` to detect silent expiry
3. **Proactive Queue Rotation** — New queue every 25min + 600 empty poll reconnection threshold
4. **Health Endpoint v2** — Added `response_pipeline`, `pending_count`, `queue_healthy`, `agent_busy_duration_seconds`
```bash ### Gen 3 (2026-06-15) — Stuck Detection
ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
```
Check: Added `stuck: bool` and `idle_seconds` to health endpoint. Monitor restarts on `stuck: true`. Added 300s restart debounce. Queue re-registers after 30min of no events.
- `processed=N` incrementing when DMs arrive
- `silence < 600s` (queue expiry timeout)
- `reconnects` — should be 0 under normal operation
### C4: A2A Response Verification
```bash
# Test A2A sends a task
ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
-H 'Content-Type: application/json' \
-d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
# Expected: task ID returned with "working" status
# Then poll for completion with tasks/get
```
### Actions
| Condition | Action |
|-----------|--------|
| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` |
| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) |
| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
---
## Global Checks
### Zulip Server
```bash
curl -s https://chat.sysloggh.net/api/v1/server_settings \
-u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}"
```
Expected: `200`
### Cross-Agent Echo Loop Detection
Check each agent's log for excessive bot-to-bot chatter:
- Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni: Check for repeated DM exchanges between bots
- kagentz: Check adapter log for bot DMs being processed
If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.
---
## Consolidated Action Matrix
| Condition | Severity | Action |
|-----------|----------|--------|
| Zulip server not 200 | 🔴 Critical | Alert maintainer |
| All agents silent | 🔴 Critical | Zulip server likely down |
| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` |
| pi: edit fail >50% | 🔴 Critical | Check editMessage API |
| Hermes: zulip disconnected | 🔴 Critical | Restart gateway |
| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway |
| Az: A2A server down | 🔴 Critical | Restart inside container |
| Az: adapter down | 🔴 Critical | Restart adapter |
| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) |
| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) |
## Logging
All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps.
Critical alerts sent as relay messages to user.
## Gen 3 Changes (2026-06-28)
### Problem
- Bot shows `connected: true` but silently stops processing messages (Zulip event queue expires without BAD_EVENT_QUEUE_ID error)
- `messages_processed=0` for 45+ min while health endpoint reports healthy
- Multiple overlapping monitors cause restart death spiral
### Fixes
1. **Stuck detection**: Health endpoint now reports `stuck: bool`, `idle_seconds`, `last_activity_time`
2. **Proactive queue recovery**: Re-register queue after 30min of no events (even without BAD_EVENT_QUEUE_ID)
3. **Monitor stuck awareness**: `zulip-monitor.sh` restarts on `stuck: true` not just `connected: false`
4. **Restart debounce**: 300s minimum between restarts (via `/tmp/zulip-monitor-debounce`)
5. **Broader queue expiry detection**: Matches multiple error message patterns