prose-contracts/zulip-health.prose.md

---
kind: contract
name: zulip-health
title: Zulip Mesh Health Monitor — Multi-Platform
version: 2.0.0
agent: abiba
triggers:
  - on startup
  - every 15 minutes while running
  - on zulip-status command
---

# Zulip Mesh Health Monitor — Multi-Platform

Monitors ALL Zulip-connected agents across three platforms.
Runs every 15 minutes in the background. Also triggers on session start.

## Platform Overview

| Platform | Agents | Bot | Adapter | Health Check |
|----------|--------|-----|---------|-------------|
| **pi** | Abiba (CT 100) | abiba-bot | `/root/.pi/agent/extensions/zulip/index.js` | `:9200/health` |
| **Hermes** | Tanko (CT 122), Mumuni (CT 123) | tanko-bot, mumuni-bot | `~/.hermes/plugins/platforms/zulip/` | `gateway_state.json` |
| **Agent Zero** | kagentz (CT 105) | kagentz-bot | Docker container, `/a0/usr/kagentz-zulip/` | A2A endpoint `:8001` |

## Phase 1: Zulip Server Liveness (All Platforms)

```bash
curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
```

Expected: `200`. Anything else → Server issue, alert maintainer.

---

## Platform A: pi (Abiba — CT 100)

### A1: Health Endpoint

```js
const health = await fetch("http://localhost:9200/health").then(r => r.json());
```

| Field | Healthy | Critical |
|-------|---------|----------|
| `connected` | `true` | `false` |
| `last_error` | `null` | non-null string |
| `retry_count` | 0-2 | 3+ |
| `queue_id` | non-null string | `null` |

### A2: PM2 Process

```bash
pm2 show abiba-zulip --no-color 2>/dev/null
```

Check: `status=online`, `restarts < 10/h`, `uptime > 60s`

### A3: Echo Loop Detection

```bash
grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5
```

> 100 skipped in 15min → 🟢 Info only (echo loop prevention working)

### A4: Response Delivery

```bash
grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20
```

> 50% fail rate → 🔴 Critical — check editMessage API

### Actions

| Condition | Action |
|-----------|--------|
| `connected: false` | `pm2 restart abiba-zulip` |
| `retry_count >= 3` | `pm2 restart abiba-zulip` |
| `last_error` set | Log and monitor |
| Crash loop >10/h | Alert user |

---

## Platform B: Hermes (Tanko — CT 122, Mumuni — CT 123)

### B1: Gateway State

```bash
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
```

Check `platforms.zulip.state`:

| Value | Meaning | Action |
|-------|---------|--------|
| `"connected"` | ✅ Healthy | None |
| `"disconnected"` | ❌ Disconnected | Check `last_error` |
| `"error"` | ❌ Error | Check `error_message`, restart gateway |
| missing | ❌ Not installed | Run deploy scripts |

### B2: Agent Process

```bash
ssh root@192.168.68.122 "ps aux | grep 'gateway run' | grep -v grep"
```

Gateway PID should exist and uptime > 60s.

### B3: Heartbeat Verification

```bash
ssh root@192.168.68.122 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
```

Expected: recent heartbeat (within 5 min) showing `polls=N` incrementing.
If silence > 300s → 🟡 Warning (queue may be stuck).
If silence > 600s → 🔴 Critical (queue expired, adapter needs restart).

### B4: Response Delivery

```bash
ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
```

> 50% fail rate → 🔴 Critical

### Actions

| Condition | Action |
|-----------|--------|
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
| No heartbeat in 10min | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
| Response empty/short | Check A2A endpoint / LiteLLM model |

---

## Platform C: Agent Zero (kagentz — CT 105)

### C1: A2A Server Health

```bash
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
```

Expected: `{"name":"kagentz",...}` — JSON response with agent identity.
Connection refused → A2A server is down.

### C2: Adapter Process

```bash
ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
```

Adapter should be running. If missing, restart.

### C3: Heartbeat & Queue

```bash
ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
```

Check:
- `processed=N` incrementing when DMs arrive
- `silence < 600s` (queue expiry timeout)
- `reconnects` — should be 0 under normal operation

### C4: A2A Response Verification

```bash
# Test A2A sends a task
ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
  -H 'Content-Type: application/json' \
  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"

# Expected: task ID returned with "working" status
# Then poll for completion with tasks/get
```

### Actions

| Condition | Action |
|-----------|--------|
| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
| Adapter process missing | `docker exec agent-zero bash -c "cd /a0/usr/kagentz-zulip && ZULIP_SITE=... ZULIP_EMAIL=... ZULIP_API_KEY=... A2A_URL=http://localhost:8001/a2a /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &"` |
| Silence > 600s (queue expiry) | Restart adapter (fix deployed: auto-reconnect on BAD_EVENT_QUEUE_ID) |
| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |

---

## Global Checks

### Zulip Server

```bash
curl -s https://chat.sysloggh.net/api/v1/server_settings \
  -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' -o /dev/null -w "%{http_code}"
```

Expected: `200`

### Cross-Agent Echo Loop Detection

Check each agent's log for excessive bot-to-bot chatter:
- Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni: Check for repeated DM exchanges between bots
- kagentz: Check adapter log for bot DMs being processed

If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.

---

## Consolidated Action Matrix

| Condition | Severity | Action |
|-----------|----------|--------|
| Zulip server not 200 | 🔴 Critical | Alert maintainer |
| All agents silent | 🔴 Critical | Zulip server likely down |
| pi: connected false | 🔴 Critical | `pm2 restart abiba-zulip` |
| pi: edit fail >50% | 🔴 Critical | Check editMessage API |
| Hermes: zulip disconnected | 🔴 Critical | Restart gateway |
| Hermes: no heartbeat 10min | 🔴 Critical | Restart gateway |
| Az: A2A server down | 🔴 Critical | Restart inside container |
| Az: adapter down | 🔴 Critical | Restart adapter |
| Az: silence >600s | 🟡 Warning | Queue expired (auto-recover) |
| Echo loop detected | 🟢 Info | Auto-mitigated (bot filtering) |

## Logging

All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps.
Critical alerts sent as relay messages to user.