diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 5dcf8de..800b60a 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -1,9 +1,9 @@ --- kind: responsibility name: zulip-health -description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity. +description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Abiba pi), Platform B (Hermes agents Tanko/Mumuni/Koonimo/Koby), and Platform C (Agent Zero). Verifies bot registration, DM delivery, cross-platform connectivity, secret injection, and YAML config integrity. title: Zulip Mesh Health Monitor — Multi-Platform -version: 3.0.0 +version: 3.1.0 runtime_contract: 2 agent: abiba --- @@ -12,11 +12,15 @@ agent: abiba Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero). Runs every 15 minutes in the background. Also triggers on session start. +v3.1.0 adds Koonimo+Koby to Platform B, Infisical dependency checks, config YAML +validation, stale PID/lock detection, Telegram adapter health, and the +cli_agent_setup_mixin patch verification. ## Requires - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` - **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14) +- **SSH access to amdpve (192.168.68.15)** for `pct exec` fallback to Koonimo (CT 113) and Koby (CT 111) - **PM2** on localhost for pi process management - **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` @@ -49,12 +53,22 @@ Runs every 15 minutes in the background. Also triggers on session start. "zulip_state": "connected", "heartbeat_age_seconds": 45, "gateway_pid": 1234, + "infisical_present": true, + "config_valid": true, + "telegram_state": "connected", + "no_key_required_count": 0, "edit_fail_rate_pct": 0, "severity": "healthy" } } ``` +New fields in v3.1.0: +- `infisical_present` — /usr/local/bin/infisical exists on the agent CT +- `config_valid` — /root/.hermes/config.yaml passes YAML validation +- `telegram_state` — Telegram adapter status from gateway_state.json +- `no_key_required_count` — count of `no-key-required` in gateway logs + ### Postconditions - Every platform is independently checked; one failure doesn't block others @@ -81,13 +95,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse ## Streaming Support (2026-07-05) Zulip agents now support progressive message editing during agent generation. -When a Hermes agent (Tanko, Mumuni) processes a message, the response is -streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API: +When a Hermes agent (Tanko, Mumuni, Koonimo, Koby) processes a message, the +response is streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API: - Adapter implements `edit_message()` using `_api_patch()` helper - Gateway stream consumer progressively edits the Zulip message - User sees real-time agent thinking instead of waiting for full response -- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active +- Verified: Tanko (CT 112), Mumuni (CT 114), Koonimo (CT 113), Koby (CT 111) ### Verification ```bash @@ -182,34 +196,99 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta | `last_error` set | Log and monitor | | Crash loop >10/h | Alert user | -### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123) +### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123, Koonimo .113, Koby .111) + +Platform B now monitors four Hermes agents: +- Tanko (CT 112, 192.168.68.122) — Zulip + Telegram +- Mumuni (CT 114, 192.168.68.123) — Zulip + Telegram + Email +- Koonimo (CT 113, 192.168.68.114, hostname "baggy") — Zulip + Telegram +- Koby (CT 111, 192.168.68.111, hostname "tdunna") — Zulip + Telegram + +SSH access: Koonimo is reachable at .114; Koby has no direct SSH. Use `pct exec` +from amdpve as the primary access method for both: +```bash +ssh root@192.168.68.15 "pct exec 113 -- " # Koonimo (or ssh .114) +ssh root@192.168.68.15 "pct exec 111 -- " # Koby (pct exec only) +``` **B1: Gateway State** ```bash ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" +ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json" +ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json" ``` -Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed. +Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌. + +**B1.5: Infisical Dependency Check** + +```bash +ssh root@ "test -f /usr/local/bin/infisical && echo OK || echo MISSING" +# pct exec variant for Koonimo/Koby: +ssh root@192.168.68.15 "pct exec 113 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING" +``` + +If MISSING → flag `infisical_present: false`, note as degraded — gateway cannot +auto-start on reboot without the infisical binary. **B2: Agent Process** ```bash ssh root@ "ps aux | grep 'gateway run' | grep -v grep" +# pct exec variant: +ssh root@192.168.68.15 "pct exec 113 -- ps aux | grep 'gateway run' | grep -v grep" ``` -Gateway PID should exist with uptime > 60s. +Gateway PID should exist with uptime > 60s. Check for stale PIDs: +- `gateway.pid` and `gateway.lock` files that reference a dead process +- Multiple gateway processes (duplicate PIDs) + +**B2.5: Config YAML Validation** + +```bash +ssh root@ "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1" +# pct exec variant: +ssh root@192.168.68.15 "pct exec 113 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1" +``` + +Expected: no output (clean parse). If parse fails → flag `config_valid: false`, +degraded — gateway is running on stale in-memory config. + +Check specifically for: +- Stray `api_key: sk-...` lines indented under `api_key_env` entries in + `custom_providers` section (hardcoded keys violate hermes-key-enforcement) +- Indentation errors in `custom_providers`, `auxiliary`, or `compression` blocks **B3: Heartbeat Verification** ```bash ssh root@ "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" +# pct exec variant: +ssh root@192.168.68.15 "pct exec 113 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3" ``` Expected: recent heartbeat (within 5 min), `polls=N` incrementing. Silence > 300s → warning. Silence > 600s → critical. +**B3.5: Stale PID/Lock Detection** + +Before any restart action, check for stale pid/lock files: + +```bash +ssh root@ "ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null" +# pct exec variant: +ssh root@192.168.68.15 "pct exec 113 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null" +``` + +If gateway process is dead (no PID) but pid/lock files exist: +```bash +ssh root@ "rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock" +``` + +Pid/lock files blocking restart → clear them before restart attempt. + **B4: Response Delivery** ```bash @@ -218,14 +297,79 @@ ssh root@ "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/ > 50% fail rate → critical. +**B4.5: LiteLLM Key Injection Verification** + +Check gateway logs for `no-key-required` failure pattern (indicates the +cli_agent_setup_mixin.py patch is missing): + +```bash +ssh root@ "grep -c 'no-k.*ired' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0" +``` + +If > 0 → flag `no_key_required_count: `, note as degraded — provider +requests silently fall back to `no-key-required` when LITELLM_API_KEY env var +resolves empty. + +**B5: Telegram Adapter Health** + +Check Telegram connectivity in gateway state or logs: + +```bash +# From gateway_state.json (all agents): +ssh root@ "cat ~/.hermes/gateway_state.json | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d[\"platforms\"].get(\"telegram\",{}).get(\"state\",\"missing\"))'" +# From logs (check for stuck DNS resolution): +ssh root@ "grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5" +``` + +Telegram states: `connected` ✅ | `disconnected` ❌ | `retrying` ⚠️ | `fatal` ❌ | `paused` ⚠️ + +If stuck on "attempt 1/8" for > 60s → flag Telegram as degraded (Zulip may +still be fine — do NOT treat as Zulip outage). + +**B6: cli_agent_setup_mixin.py Patch Verification** + +Check whether the `no-key-required` fallback string exists without the +LiteLLM-specific guard (only needed when LiteLLM key injection failures are +suspected): + +```bash +ssh root@ "grep -c 'no-key-required' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0" +``` + +If the fallback string exists without the guard → patch is missing. + **Platform B Actions** | Condition | Action | |-----------|--------| -| `zulip.state != "connected"` | `ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` | -| No heartbeat in 10min | Same as above | +| `zulip.state != "connected"` | Restart gateway (see B2 restart commands below) | +| No heartbeat in 10min | Restart gateway | | `Failed to finalize` > 50% | Check PATCH API, Zulip server | | Response empty/short | Check A2A endpoint / LiteLLM model | +| `infisical_present: false` | Flag as degraded — log and alert, do NOT restart (no infisical = no key on restart) | +| `config_valid: false` | Flag as degraded — alert user, gateway running on stale config | +| Stale pid/lock files detected | Clean files before restart | +| `no_key_required_count > 0` | Flag as degraded — check LITELLM_API_KEY injection | +| Telegram stuck on attempt 1/8 | Flag Telegram as degraded, no Zulip action needed | + +**Restart Commands** + +Standard restart (Infisical present): +```bash +ssh root@ "pkill -f 'gateway run'; sleep 2; hermes gateway restart" +# pct exec variant: +ssh root@192.168.68.15 "pct exec 113 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'" +``` + +Fallback: Infisical missing → start gateway directly from venv with env vars: +```bash +ssh root@ "source /usr/local/lib/hermes-agent/venv/bin/activate && \ + export LITELLM_API_KEY=\$(grep LITELLM_API_KEY /root/.hermes/.env | cut -d= -f2) && \ + export ZULIP_API_KEY=\$(grep ZULIP_API_KEY /root/.hermes/.env | cut -d= -f2) && \ + cd /root/.hermes && nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &" +# pct exec variant: +ssh root@192.168.68.15 "pct exec 113 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep LITELLM_API_KEY /root/.hermes/.env | cut -d= -f2); export ZULIP_API_KEY=\$(grep ZULIP_API_KEY /root/.hermes/.env | cut -d= -f2); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'" +``` ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) @@ -278,7 +422,7 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`. Check each agent's log for excessive bot-to-bot chatter: - Abiba: `Skipped.*bot msgs` count -- Tanko/Mumuni: Repeated DM exchanges between bots +- Tanko/Mumuni/Koonimo/Koby: Repeated DM exchanges between bots - kagentz: Adapter log for bot DMs being processed If any bot processes >50 bot-originated messages in 15min → warning. @@ -301,6 +445,14 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart). ## History +### v3.1.0 (2026-07-23) — Koonimo Outage Lessons + +Added Koonimo (CT 113) and Koby (CT 111) to Platform B monitoring. +Added Infisical dependency check, config YAML validation, stale PID/lock +detection, Telegram adapter health, LiteLLM key injection verification, and +cli_agent_setup_mixin.py patch verification. Updated restart commands with +Infisical-missing fallback path. + ### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix **Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.