508 lines
21 KiB
Markdown
508 lines
21 KiB
Markdown
---
|
||
kind: responsibility
|
||
name: zulip-health
|
||
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Abiba pi), Platform B (Hermes agents Tanko/Mumuni/Koonimo/Koby), and Platform C (Agent Zero). Verifies bot registration, DM delivery, cross-platform connectivity, secret injection, and YAML config integrity.
|
||
title: Zulip Mesh Health Monitor — Multi-Platform
|
||
version: 3.1.0
|
||
runtime_contract: 2
|
||
agent: abiba
|
||
---
|
||
|
||
# Zulip Mesh Health Monitor
|
||
|
||
Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
|
||
Runs every 15 minutes in the background. Also triggers on session start.
|
||
v3.1.0 adds Koonimo+Koby to Platform B, Infisical dependency checks, config YAML
|
||
validation, stale PID/lock detection, Telegram adapter health, and the
|
||
cli_agent_setup_mixin patch verification.
|
||
|
||
## Requires
|
||
|
||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
|
||
- **SSH access to amdpve (192.168.68.15)** for `pct exec` access to Koonimo (CT 113) and Koby (CT 111)
|
||
- **PM2** on localhost for pi process management
|
||
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||
- **Relay access** via RA-H OS MCP for alert delivery
|
||
|
||
## Maintains
|
||
|
||
- zulip_server_status: "healthy" | "down"
|
||
- agents: map of per-platform agent snapshots (see schema below)
|
||
- last_check: timestamp — When the last full diagnostic ran
|
||
- overall_severity: "healthy" | "degraded" | "critical"
|
||
- restart_debounce: timestamp — Last restart action (enforces 300s minimum)
|
||
|
||
### Per-Agent Snapshot Schema
|
||
|
||
```json
|
||
{
|
||
"abiba": {
|
||
"platform": "pi",
|
||
"connected": true,
|
||
"response_pipeline": "healthy",
|
||
"queue_healthy": true,
|
||
"pm2_status": "online",
|
||
"echo_loop_bot_msgs_15min": 12,
|
||
"edit_fail_rate_pct": 3,
|
||
"severity": "healthy"
|
||
},
|
||
"tanko": {
|
||
"platform": "hermes",
|
||
"zulip_state": "connected",
|
||
"heartbeat_age_seconds": 45,
|
||
"gateway_pid": 1234,
|
||
"infisical_present": true,
|
||
"config_valid": true,
|
||
"telegram_state": "connected",
|
||
"no_key_required_count": 0,
|
||
"edit_fail_rate_pct": 0,
|
||
"severity": "healthy"
|
||
}
|
||
}
|
||
```
|
||
|
||
New fields in v3.1.0:
|
||
- `infisical_present` — /usr/local/bin/infisical exists on the agent CT
|
||
- `config_valid` — /root/.hermes/config.yaml passes YAML validation
|
||
- `telegram_state` — Telegram adapter status from gateway_state.json
|
||
- `no_key_required_count` — count of `no-key-required` in gateway logs
|
||
|
||
### Postconditions
|
||
|
||
- Every platform is independently checked; one failure doesn't block others
|
||
- Restart actions respect 300s debounce window
|
||
- Critical conditions generate relay alerts to user
|
||
- All checks logged to `/root/zulip-health-monitor.log` with timestamps
|
||
|
||
## Strategies
|
||
|
||
### When Zulip server is unreachable
|
||
Skip all per-platform checks — they'll all fail downstream. Report "Zulip server down" and alert.
|
||
|
||
### When all agents are silent
|
||
Differentiate: if Zulip server returns 200, likely a shared infrastructure issue (Netbird, DNS). If server is down, it's upstream — wait.
|
||
|
||
### When restart is indicated but debounce window hasn't passed
|
||
Log the condition as "pending restart" with the timestamp. If the condition persists after debounce window, apply restart. Never bypass debounce for non-critical conditions.
|
||
|
||
### When a platform agent is unreachable via SSH
|
||
Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.
|
||
|
||
### Severity escalation
|
||
|
||
## Streaming Support (2026-07-05)
|
||
|
||
Zulip agents now support progressive message editing during agent generation.
|
||
When a Hermes agent (Tanko, Mumuni, Koonimo, Koby) processes a message, the
|
||
response is streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
|
||
|
||
- Adapter implements `edit_message()` using `_api_patch()` helper
|
||
- Gateway stream consumer progressively edits the Zulip message
|
||
- User sees real-time agent thinking instead of waiting for full response
|
||
- Verified: Tanko (CT 112), Mumuni (CT 114), Koonimo (CT 113), Koby (CT 111)
|
||
|
||
### Verification
|
||
```bash
|
||
# Check if agent has streaming:
|
||
grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
|
||
# Must return 1 — streaming is active
|
||
```
|
||
- Single agent warning → log only
|
||
- Single agent critical → relay message to user
|
||
- Two or more agents critical → immediate relay + attempt auto-recovery
|
||
- All agents critical + server up → Netbird/DNS likely down
|
||
|
||
## Invariants
|
||
|
||
- **Never restart more than once per 300s** per agent
|
||
- **Never restart if PM2 crashes > 10/h** — alert user instead
|
||
- **Never send duplicate alerts** — check last alert timestamp before relaying
|
||
- **Health checks are read-only** — diagnostics don't mutate state except for logged restarts
|
||
- **Respect Zulip API rate limits** — no more than 200 requests in rapid succession
|
||
|
||
## Continuity
|
||
|
||
- **On session start**: Run full diagnostic pass
|
||
- **Every 15 minutes**: Scheduled background check while Abiba is running
|
||
- **On `zulip-status` command**: Run on-demand and report to user
|
||
- **On critical alert**: Escalate to relay message immediately, don't wait for schedule
|
||
|
||
## Execution
|
||
|
||
### Step 1: Zulip Server Liveness
|
||
|
||
```bash
|
||
curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
|
||
-u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
|
||
```
|
||
|
||
Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.
|
||
|
||
### Step 2: Platform A — pi (Abiba, localhost)
|
||
|
||
**A1: Health Endpoint**
|
||
|
||
Fetch `http://localhost:9200/health` as JSON. Check:
|
||
|
||
| Field | Healthy | Critical |
|
||
|-------|---------|----------|
|
||
| `connected` | `true` | `false` |
|
||
| `response_pipeline` | `healthy` | `blocked` |
|
||
| `queue_healthy` | `true` | `false` |
|
||
| `stuck` | `false` | `true` |
|
||
| `idle_seconds` | < 1800 | ≥ 1800 |
|
||
| `pending_count` | 0 | > 0 with `response_pipeline: blocked` |
|
||
| `agent_busy_duration_seconds` | < 300 | ≥ 600 |
|
||
| `last_error` | `null` | non-null string |
|
||
| `retry_count` | 0–2 | 3+ |
|
||
| `queue_id` | non-null string | `null` |
|
||
|
||
**A2: PM2 Process**
|
||
|
||
```bash
|
||
pm2 show abiba-zulip --no-color 2>/dev/null
|
||
```
|
||
|
||
Check: `status=online`, `restarts < 10/h`, `uptime > 60s`.
|
||
|
||
**A3: Echo Loop Detection**
|
||
|
||
```bash
|
||
grep -a "Skipped.*bot msgs" /root/.pm2/logs/abiba-zulip-out.log | tail -5
|
||
```
|
||
|
||
> 100 skipped in 15min → info only (echo loop prevention working).
|
||
|
||
**A4: Response Delivery**
|
||
|
||
```bash
|
||
grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | tail -20
|
||
```
|
||
|
||
> 50% fail rate → critical — check editMessage API.
|
||
|
||
**Platform A Actions**
|
||
|
||
| Condition | Action |
|
||
|-----------|--------|
|
||
| `response_pipeline: blocked` | `pm2 restart abiba-zulip` |
|
||
| `connected: false` | `pm2 restart abiba-zulip` |
|
||
| `queue_healthy: false` | `pm2 restart abiba-zulip` |
|
||
| `stuck: true` | `pm2 restart abiba-zulip` |
|
||
| `retry_count >= 3` | `pm2 restart abiba-zulip` |
|
||
| `response_pipeline: degraded` | Monitor — no action, watchdog handles |
|
||
| `last_error` set | Log and monitor |
|
||
| Crash loop >10/h | Alert user |
|
||
|
||
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123, Koonimo .114, Koby .111)
|
||
|
||
Platform B now monitors four Hermes agents:
|
||
- Tanko (CT 112, 192.168.68.122) — Zulip + Telegram
|
||
- Mumuni (CT 114, 192.168.68.123) — Zulip + Telegram + Email
|
||
- Koonimo (CT 113, 192.168.68.114, hostname "baggy") — Zulip + Telegram
|
||
- Koby (CT 111, 192.168.68.111, hostname "tdunna") — Zulip + Telegram
|
||
|
||
SSH access: Koonimo is reachable at .114; Koby has no direct SSH. Use `pct exec`
|
||
from amdpve as the primary access method for both:
|
||
```bash
|
||
ssh root@192.168.68.15 "pct exec 113 -- <command>" # Koonimo (or ssh .114)
|
||
ssh root@192.168.68.15 "pct exec 111 -- <command>" # Koby (pct exec only)
|
||
```
|
||
|
||
**B1: Gateway State**
|
||
|
||
```bash
|
||
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
|
||
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
|
||
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
|
||
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
|
||
```
|
||
|
||
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌.
|
||
|
||
**B1.5: Infisical Dependency Check**
|
||
|
||
```bash
|
||
ssh root@<CT> "test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
||
# pct exec variant for Koonimo/Koby:
|
||
ssh root@192.168.68.15 "pct exec 113 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
||
```
|
||
|
||
If MISSING → flag `infisical_present: false`, note as degraded — gateway cannot
|
||
auto-start on reboot without the infisical binary.
|
||
|
||
**B2: Agent Process**
|
||
|
||
```bash
|
||
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- ps aux | grep 'gateway run' | grep -v grep"
|
||
```
|
||
|
||
Gateway PID should exist with uptime > 60s. Check for stale PIDs:
|
||
- `gateway.pid` and `gateway.lock` files that reference a dead process
|
||
- Multiple gateway processes (duplicate PIDs)
|
||
|
||
**B2.5: Config YAML Validation**
|
||
|
||
```bash
|
||
ssh root@<CT> "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
||
```
|
||
|
||
Expected: no output (clean parse). If parse fails → flag `config_valid: false`,
|
||
degraded — gateway is running on stale in-memory config.
|
||
|
||
Check specifically for:
|
||
- Stray `api_key: sk-...` lines indented under `api_key_env` entries in
|
||
`custom_providers` section (hardcoded keys violate hermes-key-enforcement)
|
||
- Indentation errors in `custom_providers`, `auxiliary`, or `compression` blocks
|
||
|
||
**B3: Heartbeat Verification**
|
||
|
||
```bash
|
||
ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
|
||
```
|
||
|
||
Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
|
||
Silence > 300s → warning. Silence > 600s → critical.
|
||
|
||
**B3.5: Stale PID/Lock Detection**
|
||
|
||
Before any restart action, check for stale pid/lock files:
|
||
|
||
```bash
|
||
ssh root@<CT> "ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
||
```
|
||
|
||
If gateway process is dead (no PID) but pid/lock files exist:
|
||
```bash
|
||
ssh root@<CT> "rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
|
||
```
|
||
|
||
Pid/lock files blocking restart → clear them before restart attempt.
|
||
|
||
**B4: Response Delivery**
|
||
|
||
```bash
|
||
ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
|
||
# pct exec variant for Koonimo/Koby:
|
||
ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
|
||
ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
|
||
```
|
||
|
||
> 50% fail rate → critical.
|
||
|
||
**B4.5: LiteLLM Key Injection Verification**
|
||
|
||
Check gateway logs for `no-key-required` failure pattern (indicates the
|
||
cli_agent_setup_mixin.py patch is missing):
|
||
|
||
```bash
|
||
ssh root@<CT> "grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
|
||
```
|
||
|
||
If > 0 → flag `no_key_required_count: <count>`, note as degraded — provider
|
||
requests silently fall back to `no-key-required` when LITELLM_API_KEY env var
|
||
resolves empty.
|
||
|
||
**B5: Telegram Adapter Health**
|
||
|
||
Check Telegram connectivity in gateway state or logs:
|
||
|
||
```bash
|
||
# From gateway_state.json (all agents):
|
||
ssh root@<CT> "cat ~/.hermes/gateway_state.json | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d[\"platforms\"].get(\"telegram\",{}).get(\"state\",\"missing\"))'"
|
||
# pct exec variant for Koonimo/Koby (cat the file; read platforms.telegram.state):
|
||
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
|
||
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
|
||
# From logs (check for stuck DNS resolution):
|
||
ssh root@<CT> "grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
||
ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
||
ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
||
```
|
||
|
||
Telegram states: `connected` ✅ | `disconnected` ❌ | `retrying` ⚠️ | `fatal` ❌ | `paused` ⚠️
|
||
|
||
If stuck on "attempt 1/8" for > 60s → flag Telegram as degraded (Zulip may
|
||
still be fine — do NOT treat as Zulip outage).
|
||
|
||
**B6: cli_agent_setup_mixin.py Patch Verification**
|
||
|
||
Check whether the `no-key-required` fallback string exists without the
|
||
LiteLLM-specific guard (only needed when LiteLLM key injection failures are
|
||
suspected). A bare count of the fallback string cannot distinguish a guarded
|
||
occurrence from an unguarded one, so count both the fallback string and the
|
||
LiteLLM guard token:
|
||
|
||
```bash
|
||
ssh root@<CT> "f=\$(grep -c 'no-key-required' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c 'LITELLM_API_KEY' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
|
||
```
|
||
|
||
If `fallback > 0` and `guard == 0` → the fallback string is present without
|
||
the LiteLLM-specific guard, so the patch is missing. If unsure, inspect each
|
||
occurrence with `grep -n -B2 -A2 'no-key-required'` to confirm the guard
|
||
wraps it.
|
||
|
||
**Platform B Actions**
|
||
|
||
| Condition | Action |
|
||
|-----------|--------|
|
||
| `zulip.state != "connected"` | Restart gateway (see B2 restart commands below) |
|
||
| No heartbeat in 10min | Restart gateway |
|
||
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
|
||
| Response empty/short | Check A2A endpoint / LiteLLM model |
|
||
| `infisical_present: false` | Flag as degraded — log and alert; do NOT auto-restart via the standard command (it cannot inject the key without infisical). Operator may use the Infisical-missing fallback restart below. |
|
||
| `config_valid: false` | Flag as degraded — alert user, gateway running on stale config |
|
||
| Stale pid/lock files detected | Clean files before restart |
|
||
| `no_key_required_count > 0` | Flag as degraded — check LITELLM_API_KEY injection |
|
||
| Telegram stuck on attempt 1/8 | Flag Telegram as degraded, no Zulip action needed |
|
||
|
||
**Restart Commands**
|
||
|
||
Standard restart (Infisical present):
|
||
```bash
|
||
ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
|
||
```
|
||
|
||
The mechanisms differ by design: Tanko/Mumuni launch the gateway through a
|
||
wrapper loop, so the `hermes gateway restart` CLI is the correct entry point
|
||
(it re-arms the wrapper). Koonimo/Koby run the gateway as a systemd unit
|
||
(`hermes-gateway.service`), so `systemctl restart hermes-gateway` is the
|
||
correct entry point under `pct exec`. Do not swap the two — using the CLI on
|
||
Koonimo/Koby would bypass the unit, and using `systemctl` on Tanko/Mumuni
|
||
would miss the wrapper loop.
|
||
|
||
Fallback: Infisical missing → start gateway directly from venv with env vars:
|
||
```bash
|
||
ssh root@<CT> "source /usr/local/lib/hermes-agent/venv/bin/activate && \
|
||
export LITELLM_API_KEY=\$(grep -E '^LITELLM_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
|
||
export ZULIP_API_KEY=\$(grep -E '^ZULIP_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
|
||
cd /root/.hermes && nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &"
|
||
# pct exec variant:
|
||
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
|
||
```
|
||
|
||
### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
|
||
|
||
**C1: A2A Server Health**
|
||
|
||
```bash
|
||
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
|
||
```
|
||
|
||
Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.
|
||
|
||
**C2: Adapter Process**
|
||
|
||
```bash
|
||
ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep"
|
||
```
|
||
|
||
Adapter should be running. Missing → restart inside container.
|
||
|
||
**C3: Heartbeat & Queue**
|
||
|
||
```bash
|
||
ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3"
|
||
```
|
||
|
||
Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.
|
||
|
||
**C4: A2A Response Verification**
|
||
|
||
```bash
|
||
ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
|
||
```
|
||
|
||
Expected: task ID with "working" status. Poll for completion with `tasks/get`.
|
||
|
||
**Platform C Actions**
|
||
|
||
| Condition | Action |
|
||
|-----------|--------|
|
||
| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` |
|
||
| Adapter process missing | Restart adapter inside container with env vars |
|
||
| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) |
|
||
| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
|
||
|
||
### Step 5: Global Checks
|
||
|
||
**Cross-Agent Echo Loop Detection**
|
||
|
||
Check each agent's log for excessive bot-to-bot chatter:
|
||
- Abiba: `Skipped.*bot msgs` count
|
||
- Tanko/Mumuni/Koonimo/Koby: Repeated DM exchanges between bots
|
||
- kagentz: Adapter log for bot DMs being processed
|
||
|
||
If any bot processes >50 bot-originated messages in 15min → warning.
|
||
|
||
### Step 6: Compile and Report
|
||
|
||
1. Compile all platform checks and severity
|
||
2. Determine `overall_severity` from worst per-agent severity
|
||
3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
|
||
4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
|
||
5. If any agent critical or >2 degraded: send relay message to user
|
||
6. Update `last_check` timestamp in `### Maintains` snapshot
|
||
|
||
### Restart Debounce
|
||
|
||
All restart actions MUST debounce: minimum 300s between restarts.
|
||
Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
|
||
|
||
---
|
||
|
||
## History
|
||
|
||
### v3.1.0 (2026-07-23) — Koonimo Outage Lessons
|
||
|
||
Added Koonimo (CT 113) and Koby (CT 111) to Platform B monitoring.
|
||
Added Infisical dependency check, config YAML validation, stale PID/lock
|
||
detection, Telegram adapter health, LiteLLM key injection verification, and
|
||
cli_agent_setup_mixin.py patch verification. Updated restart commands with
|
||
Infisical-missing fallback path.
|
||
|
||
### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
|
||
|
||
**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.
|
||
|
||
**Fixes — Extension (`index.js`)**:
|
||
1. **Rotation extended to 55 min** (from 25) with **±90s jitter** — avoids aligning with cron/monitor cycles
|
||
2. **Rate-limit-aware retry** — if connection fails with 429, skip the retry loop entirely, wait 120s, try once
|
||
|
||
**Fixes — Monitor (`zulip-monitor.sh`)**:
|
||
3. **Smart Triage** instead of instant restart:
|
||
- **Rate-limit detection**: If error log shows recent 429s, wait — don't add more API load
|
||
- **Self-healing detection**: If `retry_count` is 1-2, extension is already retrying — don't interrupt
|
||
- **Rotation window awareness**: If `queue_age` is 25-35min, disconnection is likely transient rotation — wait
|
||
- **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
|
||
- **Debounce gate**: Skip all checks entirely if recently restarted
|
||
|
||
### Gen 4 (2026-06-29) — Response Pipeline Fix
|
||
|
||
**Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
|
||
|
||
**Four-Layer Defense**:
|
||
1. **Response Watchdog** — 30s deadline timer; 5min placeholder edit; 10min error message + dequeue
|
||
2. **Queue Health Ping** — Every 2min `/api/v1/events?dont_block=true` to detect silent expiry
|
||
3. **Proactive Queue Rotation** — New queue every 25min + 600 empty poll reconnection threshold
|
||
4. **Health Endpoint v2** — Added `response_pipeline`, `pending_count`, `queue_healthy`, `agent_busy_duration_seconds`
|
||
|
||
### Gen 3 (2026-06-15) — Stuck Detection
|
||
|
||
Added `stuck: bool` and `idle_seconds` to health endpoint. Monitor restarts on `stuck: true`. Added 300s restart debounce. Queue re-registers after 30min of no events.
|