zulip-health: v3.0.0 -> v3.1.0 — Koonimo outage lessons
- Add Koonimo (CT 113, .114) and Koby (CT 111) to Platform B monitoring - Add Infisical dependency check (/usr/local/bin/infisical presence) - Add config YAML validation (yaml.safe_load check) - Add stale PID/lock detection and cleanup before restart - Add LiteLLM key injection verification (no-key-required pattern) - Add Telegram adapter health check - Add cli_agent_setup_mixin.py patch verification - Update restart commands with Infisical-missing fallback path - Update requires/maintains schema with new fields - Use pct exec from amdpve as primary access for Koonimo/Koby - Verified correct IPs: Koonimo=.114 (not .113), Koby=pct exec only
This commit is contained in:
root
+163
-11
@@ -1,9 +1,9 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: zulip-health
|
||||
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
|
||||
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Abiba pi), Platform B (Hermes agents Tanko/Mumuni/Koonimo/Koby), and Platform C (Agent Zero). Verifies bot registration, DM delivery, cross-platform connectivity, secret injection, and YAML config integrity.
|
||||
title: Zulip Mesh Health Monitor — Multi-Platform
|
||||
version: 3.0.0
|
||||
version: 3.1.0
|
||||
runtime_contract: 2
|
||||
agent: abiba
|
||||
---
|
||||
@@ -12,11 +12,15 @@ agent: abiba
|
||||
|
||||
Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
|
||||
Runs every 15 minutes in the background. Also triggers on session start.
|
||||
v3.1.0 adds Koonimo+Koby to Platform B, Infisical dependency checks, config YAML
|
||||
validation, stale PID/lock detection, Telegram adapter health, and the
|
||||
cli_agent_setup_mixin patch verification.
|
||||
|
||||
## Requires
|
||||
|
||||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
|
||||
- **SSH access to amdpve (192.168.68.15)** for `pct exec` fallback to Koonimo (CT 113) and Koby (CT 111)
|
||||
- **PM2** on localhost for pi process management
|
||||
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
||||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||||
@@ -49,12 +53,22 @@ Runs every 15 minutes in the background. Also triggers on session start.
|
||||
"zulip_state": "connected",
|
||||
"heartbeat_age_seconds": 45,
|
||||
"gateway_pid": 1234,
|
||||
"infisical_present": true,
|
||||
"config_valid": true,
|
||||
"telegram_state": "connected",
|
||||
"no_key_required_count": 0,
|
||||
"edit_fail_rate_pct": 0,
|
||||
"severity": "healthy"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
New fields in v3.1.0:
|
||||
- `infisical_present` — /usr/local/bin/infisical exists on the agent CT
|
||||
- `config_valid` — /root/.hermes/config.yaml passes YAML validation
|
||||
- `telegram_state` — Telegram adapter status from gateway_state.json
|
||||
- `no_key_required_count` — count of `no-key-required` in gateway logs
|
||||
|
||||
### Postconditions
|
||||
|
||||
- Every platform is independently checked; one failure doesn't block others
|
||||
@@ -81,13 +95,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
|
||||
## Streaming Support (2026-07-05)
|
||||
|
||||
Zulip agents now support progressive message editing during agent generation.
|
||||
When a Hermes agent (Tanko, Mumuni) processes a message, the response is
|
||||
streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
|
||||
When a Hermes agent (Tanko, Mumuni, Koonimo, Koby) processes a message, the
|
||||
response is streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
|
||||
|
||||
- Adapter implements `edit_message()` using `_api_patch()` helper
|
||||
- Gateway stream consumer progressively edits the Zulip message
|
||||
- User sees real-time agent thinking instead of waiting for full response
|
||||
- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
|
||||
- Verified: Tanko (CT 112), Mumuni (CT 114), Koonimo (CT 113), Koby (CT 111)
|
||||
|
||||
### Verification
|
||||
```bash
|
||||
@@ -182,34 +196,99 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
|
||||
| `last_error` set | Log and monitor |
|
||||
| Crash loop >10/h | Alert user |
|
||||
|
||||
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
|
||||
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123, Koonimo .113, Koby .111)
|
||||
|
||||
Platform B now monitors four Hermes agents:
|
||||
- Tanko (CT 112, 192.168.68.122) — Zulip + Telegram
|
||||
- Mumuni (CT 114, 192.168.68.123) — Zulip + Telegram + Email
|
||||
- Koonimo (CT 113, 192.168.68.114, hostname "baggy") — Zulip + Telegram
|
||||
- Koby (CT 111, 192.168.68.111, hostname "tdunna") — Zulip + Telegram
|
||||
|
||||
SSH access: Koonimo is reachable at .114; Koby has no direct SSH. Use `pct exec`
|
||||
from amdpve as the primary access method for both:
|
||||
```bash
|
||||
ssh root@192.168.68.15 "pct exec 113 -- <command>" # Koonimo (or ssh .114)
|
||||
ssh root@192.168.68.15 "pct exec 111 -- <command>" # Koby (pct exec only)
|
||||
```
|
||||
|
||||
**B1: Gateway State**
|
||||
|
||||
```bash
|
||||
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
|
||||
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
|
||||
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
|
||||
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
|
||||
```
|
||||
|
||||
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
|
||||
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌.
|
||||
|
||||
**B1.5: Infisical Dependency Check**
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
||||
# pct exec variant for Koonimo/Koby:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
||||
```
|
||||
|
||||
If MISSING → flag `infisical_present: false`, note as degraded — gateway cannot
|
||||
auto-start on reboot without the infisical binary.
|
||||
|
||||
**B2: Agent Process**
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
||||
# pct exec variant:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- ps aux | grep 'gateway run' | grep -v grep"
|
||||
```
|
||||
|
||||
Gateway PID should exist with uptime > 60s.
|
||||
Gateway PID should exist with uptime > 60s. Check for stale PIDs:
|
||||
- `gateway.pid` and `gateway.lock` files that reference a dead process
|
||||
- Multiple gateway processes (duplicate PIDs)
|
||||
|
||||
**B2.5: Config YAML Validation**
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
||||
# pct exec variant:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
||||
```
|
||||
|
||||
Expected: no output (clean parse). If parse fails → flag `config_valid: false`,
|
||||
degraded — gateway is running on stale in-memory config.
|
||||
|
||||
Check specifically for:
|
||||
- Stray `api_key: sk-...` lines indented under `api_key_env` entries in
|
||||
`custom_providers` section (hardcoded keys violate hermes-key-enforcement)
|
||||
- Indentation errors in `custom_providers`, `auxiliary`, or `compression` blocks
|
||||
|
||||
**B3: Heartbeat Verification**
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
|
||||
# pct exec variant:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
|
||||
```
|
||||
|
||||
Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
|
||||
Silence > 300s → warning. Silence > 600s → critical.
|
||||
|
||||
**B3.5: Stale PID/Lock Detection**
|
||||
|
||||
Before any restart action, check for stale pid/lock files:
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
||||
# pct exec variant:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
||||
```
|
||||
|
||||
If gateway process is dead (no PID) but pid/lock files exist:
|
||||
```bash
|
||||
ssh root@<CT> "rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
|
||||
```
|
||||
|
||||
Pid/lock files blocking restart → clear them before restart attempt.
|
||||
|
||||
**B4: Response Delivery**
|
||||
|
||||
```bash
|
||||
@@ -218,14 +297,79 @@ ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/
|
||||
|
||||
> 50% fail rate → critical.
|
||||
|
||||
**B4.5: LiteLLM Key Injection Verification**
|
||||
|
||||
Check gateway logs for `no-key-required` failure pattern (indicates the
|
||||
cli_agent_setup_mixin.py patch is missing):
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "grep -c 'no-k.*ired' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
|
||||
```
|
||||
|
||||
If > 0 → flag `no_key_required_count: <count>`, note as degraded — provider
|
||||
requests silently fall back to `no-key-required` when LITELLM_API_KEY env var
|
||||
resolves empty.
|
||||
|
||||
**B5: Telegram Adapter Health**
|
||||
|
||||
Check Telegram connectivity in gateway state or logs:
|
||||
|
||||
```bash
|
||||
# From gateway_state.json (all agents):
|
||||
ssh root@<CT> "cat ~/.hermes/gateway_state.json | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d[\"platforms\"].get(\"telegram\",{}).get(\"state\",\"missing\"))'"
|
||||
# From logs (check for stuck DNS resolution):
|
||||
ssh root@<CT> "grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
||||
```
|
||||
|
||||
Telegram states: `connected` ✅ | `disconnected` ❌ | `retrying` ⚠️ | `fatal` ❌ | `paused` ⚠️
|
||||
|
||||
If stuck on "attempt 1/8" for > 60s → flag Telegram as degraded (Zulip may
|
||||
still be fine — do NOT treat as Zulip outage).
|
||||
|
||||
**B6: cli_agent_setup_mixin.py Patch Verification**
|
||||
|
||||
Check whether the `no-key-required` fallback string exists without the
|
||||
LiteLLM-specific guard (only needed when LiteLLM key injection failures are
|
||||
suspected):
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "grep -c 'no-key-required' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0"
|
||||
```
|
||||
|
||||
If the fallback string exists without the guard → patch is missing.
|
||||
|
||||
**Platform B Actions**
|
||||
|
||||
| Condition | Action |
|
||||
|-----------|--------|
|
||||
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
|
||||
| No heartbeat in 10min | Same as above |
|
||||
| `zulip.state != "connected"` | Restart gateway (see B2 restart commands below) |
|
||||
| No heartbeat in 10min | Restart gateway |
|
||||
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
|
||||
| Response empty/short | Check A2A endpoint / LiteLLM model |
|
||||
| `infisical_present: false` | Flag as degraded — log and alert, do NOT restart (no infisical = no key on restart) |
|
||||
| `config_valid: false` | Flag as degraded — alert user, gateway running on stale config |
|
||||
| Stale pid/lock files detected | Clean files before restart |
|
||||
| `no_key_required_count > 0` | Flag as degraded — check LITELLM_API_KEY injection |
|
||||
| Telegram stuck on attempt 1/8 | Flag Telegram as degraded, no Zulip action needed |
|
||||
|
||||
**Restart Commands**
|
||||
|
||||
Standard restart (Infisical present):
|
||||
```bash
|
||||
ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
|
||||
# pct exec variant:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
|
||||
```
|
||||
|
||||
Fallback: Infisical missing → start gateway directly from venv with env vars:
|
||||
```bash
|
||||
ssh root@<CT> "source /usr/local/lib/hermes-agent/venv/bin/activate && \
|
||||
export LITELLM_API_KEY=\$(grep LITELLM_API_KEY /root/.hermes/.env | cut -d= -f2) && \
|
||||
export ZULIP_API_KEY=\$(grep ZULIP_API_KEY /root/.hermes/.env | cut -d= -f2) && \
|
||||
cd /root/.hermes && nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &"
|
||||
# pct exec variant:
|
||||
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep LITELLM_API_KEY /root/.hermes/.env | cut -d= -f2); export ZULIP_API_KEY=\$(grep ZULIP_API_KEY /root/.hermes/.env | cut -d= -f2); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
|
||||
```
|
||||
|
||||
### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
|
||||
|
||||
@@ -278,7 +422,7 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`.
|
||||
|
||||
Check each agent's log for excessive bot-to-bot chatter:
|
||||
- Abiba: `Skipped.*bot msgs` count
|
||||
- Tanko/Mumuni: Repeated DM exchanges between bots
|
||||
- Tanko/Mumuni/Koonimo/Koby: Repeated DM exchanges between bots
|
||||
- kagentz: Adapter log for bot DMs being processed
|
||||
|
||||
If any bot processes >50 bot-originated messages in 15min → warning.
|
||||
@@ -301,6 +445,14 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
|
||||
|
||||
## History
|
||||
|
||||
### v3.1.0 (2026-07-23) — Koonimo Outage Lessons
|
||||
|
||||
Added Koonimo (CT 113) and Koby (CT 111) to Platform B monitoring.
|
||||
Added Infisical dependency check, config YAML validation, stale PID/lock
|
||||
detection, Telegram adapter health, LiteLLM key injection verification, and
|
||||
cli_agent_setup_mixin.py patch verification. Updated restart commands with
|
||||
Infisical-missing fallback path.
|
||||
|
||||
### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
|
||||
|
||||
**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.
|
||||
|
||||
Reference in New Issue
Block a user