|
|
|
@@ -1,9 +1,9 @@
|
|
|
|
|
---
|
|
|
|
|
kind: responsibility
|
|
|
|
|
name: zulip-health
|
|
|
|
|
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Abiba pi), Platform B (Hermes agents Tanko/Mumuni/Koonimo/Koby), and Platform C (Agent Zero). Verifies bot registration, DM delivery, cross-platform connectivity, secret injection, and YAML config integrity.
|
|
|
|
|
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
|
|
|
|
|
title: Zulip Mesh Health Monitor — Multi-Platform
|
|
|
|
|
version: 3.1.0
|
|
|
|
|
version: 3.0.0
|
|
|
|
|
runtime_contract: 2
|
|
|
|
|
agent: abiba
|
|
|
|
|
---
|
|
|
|
@@ -12,15 +12,11 @@ agent: abiba
|
|
|
|
|
|
|
|
|
|
Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
|
|
|
|
|
Runs every 15 minutes in the background. Also triggers on session start.
|
|
|
|
|
v3.1.0 adds Koonimo+Koby to Platform B, Infisical dependency checks, config YAML
|
|
|
|
|
validation, stale PID/lock detection, Telegram adapter health, and the
|
|
|
|
|
cli_agent_setup_mixin patch verification.
|
|
|
|
|
|
|
|
|
|
## Requires
|
|
|
|
|
|
|
|
|
|
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
|
|
|
|
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
|
|
|
|
|
- **SSH access to amdpve (192.168.68.15)** for `pct exec` access to Koonimo (CT 113) and Koby (CT 111)
|
|
|
|
|
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
|
|
|
|
|
- **PM2** on localhost for pi process management
|
|
|
|
|
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
|
|
|
|
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
|
|
|
@@ -53,22 +49,12 @@ cli_agent_setup_mixin patch verification.
|
|
|
|
|
"zulip_state": "connected",
|
|
|
|
|
"heartbeat_age_seconds": 45,
|
|
|
|
|
"gateway_pid": 1234,
|
|
|
|
|
"infisical_present": true,
|
|
|
|
|
"config_valid": true,
|
|
|
|
|
"telegram_state": "connected",
|
|
|
|
|
"no_key_required_count": 0,
|
|
|
|
|
"edit_fail_rate_pct": 0,
|
|
|
|
|
"severity": "healthy"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
New fields in v3.1.0:
|
|
|
|
|
- `infisical_present` — /usr/local/bin/infisical exists on the agent CT
|
|
|
|
|
- `config_valid` — /root/.hermes/config.yaml passes YAML validation
|
|
|
|
|
- `telegram_state` — Telegram adapter status from gateway_state.json
|
|
|
|
|
- `no_key_required_count` — count of `no-key-required` in gateway logs
|
|
|
|
|
|
|
|
|
|
### Postconditions
|
|
|
|
|
|
|
|
|
|
- Every platform is independently checked; one failure doesn't block others
|
|
|
|
@@ -95,13 +81,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
|
|
|
|
|
## Streaming Support (2026-07-05)
|
|
|
|
|
|
|
|
|
|
Zulip agents now support progressive message editing during agent generation.
|
|
|
|
|
When a Hermes agent (Tanko, Mumuni, Koonimo, Koby) processes a message, the
|
|
|
|
|
response is streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
|
|
|
|
|
When a Hermes agent (Tanko, Mumuni) processes a message, the response is
|
|
|
|
|
streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
|
|
|
|
|
|
|
|
|
|
- Adapter implements `edit_message()` using `_api_patch()` helper
|
|
|
|
|
- Gateway stream consumer progressively edits the Zulip message
|
|
|
|
|
- User sees real-time agent thinking instead of waiting for full response
|
|
|
|
|
- Verified: Tanko (CT 112), Mumuni (CT 114), Koonimo (CT 113), Koby (CT 111)
|
|
|
|
|
- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
|
|
|
|
|
|
|
|
|
|
### Verification
|
|
|
|
|
```bash
|
|
|
|
@@ -196,217 +182,50 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
|
|
|
|
|
| `last_error` set | Log and monitor |
|
|
|
|
|
| Crash loop >10/h | Alert user |
|
|
|
|
|
|
|
|
|
|
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123, Koonimo .114, Koby .129)
|
|
|
|
|
|
|
|
|
|
Platform B now monitors four Hermes agents:
|
|
|
|
|
- Tanko (CT 112, 192.168.68.122) — Zulip + Telegram
|
|
|
|
|
- Mumuni (CT 114, 192.168.68.123) — Zulip + Telegram + Email
|
|
|
|
|
- Koonimo (CT 113, 192.168.68.114, hostname "baggy") — Zulip + Telegram
|
|
|
|
|
- Koby (CT 111, 192.168.68.129, hostname "tdunna") — Zulip + Telegram
|
|
|
|
|
|
|
|
|
|
SSH access: Koonimo is reachable at .114; Koby has no direct SSH. Use `pct exec`
|
|
|
|
|
from amdpve as the primary access method for both:
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- <command>" # Koonimo (or ssh .114)
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- <command>" # Koby (pct exec only)
|
|
|
|
|
```
|
|
|
|
|
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
|
|
|
|
|
|
|
|
|
|
**B1: Gateway State**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
|
|
|
|
|
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌.
|
|
|
|
|
|
|
|
|
|
**B1.5: Infisical Dependency Check**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
|
|
|
|
# pct exec variant for Koonimo/Koby:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If MISSING → flag `infisical_present: false`, note as degraded — gateway cannot
|
|
|
|
|
auto-start on reboot without the infisical binary.
|
|
|
|
|
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
|
|
|
|
|
|
|
|
|
|
**B2: Agent Process**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- ps aux | grep 'gateway run' | grep -v grep"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- ps aux | grep 'gateway run' | grep -v grep"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Gateway PID should exist with uptime > 60s. Check for stale PIDs:
|
|
|
|
|
- `gateway.pid` and `gateway.lock` files that reference a dead process
|
|
|
|
|
- Multiple gateway processes (duplicate PIDs)
|
|
|
|
|
|
|
|
|
|
**B2.5: Config YAML Validation**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Expected: no output (clean parse). If parse fails → flag `config_valid: false`,
|
|
|
|
|
degraded — gateway is running on stale in-memory config.
|
|
|
|
|
|
|
|
|
|
Check specifically for:
|
|
|
|
|
- Stray `api_key: sk-...` lines indented under `api_key_env` entries in
|
|
|
|
|
`custom_providers` section (hardcoded keys violate hermes-key-enforcement)
|
|
|
|
|
- Indentation errors in `custom_providers`, `auxiliary`, or `compression` blocks
|
|
|
|
|
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
|
|
|
|
|
|
|
|
|
|
**B3: Heartbeat Verification**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
|
|
|
|
|
Silence > 300s → warning. Silence > 600s → critical.
|
|
|
|
|
|
|
|
|
|
**B3.5: Stale PID/Lock Detection**
|
|
|
|
|
|
|
|
|
|
Before any restart action, check for stale pid/lock files:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If gateway process is dead (no PID) but pid/lock files exist:
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Pid/lock files blocking restart → clear them before restart attempt.
|
|
|
|
|
|
|
|
|
|
**B4: Response Delivery**
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
|
|
|
|
|
# pct exec variant for Koonimo/Koby:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
> 50% fail rate → critical.
|
|
|
|
|
|
|
|
|
|
**B4.5: LiteLLM Key Injection Verification**
|
|
|
|
|
|
|
|
|
|
Check gateway logs for `no-key-required` failure pattern (indicates the
|
|
|
|
|
cli_agent_setup_mixin.py patch is missing):
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
|
|
|
|
|
# pct exec variant for Koonimo/Koby:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If > 0 → flag `no_key_required_count: <count>`, note as degraded — provider
|
|
|
|
|
requests silently fall back to `no-key-required` when LITELLM_API_KEY env var
|
|
|
|
|
resolves empty.
|
|
|
|
|
|
|
|
|
|
**B5: Telegram Adapter Health**
|
|
|
|
|
|
|
|
|
|
Check Telegram connectivity in gateway state or logs:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
# From gateway_state.json (all agents):
|
|
|
|
|
ssh root@<CT> "cat ~/.hermes/gateway_state.json | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d[\"platforms\"].get(\"telegram\",{}).get(\"state\",\"missing\"))'"
|
|
|
|
|
# pct exec variant for Koonimo/Koby (cat the file; read platforms.telegram.state):
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
|
|
|
|
|
# From logs (check for stuck DNS resolution):
|
|
|
|
|
ssh root@<CT> "grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Telegram states: `connected` ✅ | `disconnected` ❌ | `retrying` ⚠️ | `fatal` ❌ | `paused` ⚠️
|
|
|
|
|
|
|
|
|
|
If stuck on "attempt 1/8" for > 60s → flag Telegram as degraded (Zulip may
|
|
|
|
|
still be fine — do NOT treat as Zulip outage).
|
|
|
|
|
|
|
|
|
|
**B6: cli_agent_setup_mixin.py Patch Verification**
|
|
|
|
|
|
|
|
|
|
Check whether the `no-key-required` fallback string exists without the
|
|
|
|
|
LiteLLM-specific guard (only needed when LiteLLM key injection failures are
|
|
|
|
|
suspected). A bare count of the fallback string cannot distinguish a guarded
|
|
|
|
|
occurrence from an unguarded one, so count both the fallback string and the
|
|
|
|
|
LiteLLM guard token:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "f=\$(grep -c 'no-key-required' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c 'LITELLM_API_KEY' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If `fallback > 0` and `guard == 0` → the fallback string is present without
|
|
|
|
|
the LiteLLM-specific guard, so the patch is missing. If unsure, inspect each
|
|
|
|
|
occurrence with `grep -n -B2 -A2 'no-key-required'` to confirm the guard
|
|
|
|
|
wraps it.
|
|
|
|
|
|
|
|
|
|
**Platform B Actions**
|
|
|
|
|
|
|
|
|
|
| Condition | Action |
|
|
|
|
|
|-----------|--------|
|
|
|
|
|
| `zulip.state != "connected"` | Restart gateway (see B2 restart commands below) |
|
|
|
|
|
| No heartbeat in 10min | Restart gateway |
|
|
|
|
|
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
|
|
|
|
|
| No heartbeat in 10min | Same as above |
|
|
|
|
|
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
|
|
|
|
|
| Response empty/short | Check A2A endpoint / LiteLLM model |
|
|
|
|
|
| `infisical_present: false` | Flag as degraded — log and alert; do NOT auto-restart via the standard command (it cannot inject the key without infisical). Operator may use the Infisical-missing fallback restart below. |
|
|
|
|
|
| `config_valid: false` | Flag as degraded — alert user, gateway running on stale config |
|
|
|
|
|
| Stale pid/lock files detected | Clean files before restart |
|
|
|
|
|
| `no_key_required_count > 0` | Flag as degraded — check LITELLM_API_KEY injection |
|
|
|
|
|
| Telegram stuck on attempt 1/8 | Flag Telegram as degraded, no Zulip action needed |
|
|
|
|
|
|
|
|
|
|
**Restart Commands**
|
|
|
|
|
|
|
|
|
|
Standard restart (Infisical present):
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The mechanisms differ by design: Tanko/Mumuni launch the gateway through a
|
|
|
|
|
wrapper loop, so the `hermes gateway restart` CLI is the correct entry point
|
|
|
|
|
(it re-arms the wrapper). Koonimo/Koby run the gateway as a systemd unit
|
|
|
|
|
(`hermes-gateway.service`), so `systemctl restart hermes-gateway` is the
|
|
|
|
|
correct entry point under `pct exec`. Do not swap the two — using the CLI on
|
|
|
|
|
Koonimo/Koby would bypass the unit, and using `systemctl` on Tanko/Mumuni
|
|
|
|
|
would miss the wrapper loop.
|
|
|
|
|
|
|
|
|
|
Fallback: Infisical missing → start gateway directly from venv with env vars:
|
|
|
|
|
```bash
|
|
|
|
|
ssh root@<CT> "source /usr/local/lib/hermes-agent/venv/bin/activate && \
|
|
|
|
|
export LITELLM_API_KEY=\$(grep -E '^LITELLM_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
|
|
|
|
|
export ZULIP_API_KEY=\$(grep -E '^ZULIP_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
|
|
|
|
|
cd /root/.hermes && nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &"
|
|
|
|
|
# pct exec variant:
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
|
|
|
|
|
ssh root@192.168.68.15 "pct exec 111 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
|
|
|
|
|
|
|
|
|
@@ -459,7 +278,7 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`.
|
|
|
|
|
|
|
|
|
|
Check each agent's log for excessive bot-to-bot chatter:
|
|
|
|
|
- Abiba: `Skipped.*bot msgs` count
|
|
|
|
|
- Tanko/Mumuni/Koonimo/Koby: Repeated DM exchanges between bots
|
|
|
|
|
- Tanko/Mumuni: Repeated DM exchanges between bots
|
|
|
|
|
- kagentz: Adapter log for bot DMs being processed
|
|
|
|
|
|
|
|
|
|
If any bot processes >50 bot-originated messages in 15min → warning.
|
|
|
|
@@ -482,14 +301,6 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
|
|
|
|
|
|
|
|
|
|
## History
|
|
|
|
|
|
|
|
|
|
### v3.1.0 (2026-07-23) — Koonimo Outage Lessons
|
|
|
|
|
|
|
|
|
|
Added Koonimo (CT 113) and Koby (CT 111) to Platform B monitoring.
|
|
|
|
|
Added Infisical dependency check, config YAML validation, stale PID/lock
|
|
|
|
|
detection, Telegram adapter health, LiteLLM key injection verification, and
|
|
|
|
|
cli_agent_setup_mixin.py patch verification. Updated restart commands with
|
|
|
|
|
Infisical-missing fallback path.
|
|
|
|
|
|
|
|
|
|
### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
|
|
|
|
|
|
|
|
|
|
**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.
|
|
|
|
|
root