diff --git a/zulip-resilience-v3.prose.md b/zulip-resilience-v3.prose.md index 875eca7..4825441 100644 --- a/zulip-resilience-v3.prose.md +++ b/zulip-resilience-v3.prose.md @@ -409,3 +409,68 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S | Queue expiry handling | Crash | Auto re-register | | Busy worker deadlock | Router death | Worker SIGKILL + error DM | | PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) | + +--- + +## Incident Log — 2026-07-18 Fleet-Wide Audit + +### Fleet State After Audit + +| Agent | Platform | Zulip State | Issues Found | Fix Applied | +|-------|----------|-------------|--------------|-------------| +| **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s | +| **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) | +| **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed | + +### Key Fixes Applied + +**1. Abiba — Credential Fallback (L4 Pattern)** +- Root cause: `zulip.api_key` in config.yaml is `""` (expected from Infisical). Infisical vault `ABIBA_ZULIP_API_KEY` wasn't being injected into the process environment. +- Fix: Added `.env` file fallback at `/root/.pi/agent/extensions/zulip/.env` with known-working key, sourced before the Infisical `exec`. +- Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local `.env` fallback. + +**2. Abiba — Poll Timeout Handling** +- Root cause: Zulip long-poll uses `AbortSignal.timeout(65000)`. Zulip's default `event_queue_longpoll_timeout_seconds` can exceed 65s. When the signal fires, an `AbortError` is thrown and caught by the circuit breaker as a failure. +- Fix: Caught `AbortError` inside `poll()` and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default. +- Reference: [Zulip Events System — long-poll timeout](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html) + +**3. Tanko — Gateway Restart** +- Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (`infisical-gateway.sh`) restarts on crash but the gateway wasn't re-establishing Zulip on restart. +- Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully. + +### Fleet-Wide Zulip Health Metrics (as of 2026-07-18) + +| Metric | Value | +|--------|-------| +| Zulip server | ✅ HTTP 200 | +| Agents connected | 3/3 (Abiba, Tanko, Mumuni) | +| Abiba circuit breaker | CLOSED (0 failures) | +| Abiba uptime | 2D (post-restart) | +| Tanko gateway uptime | Ongoing | +| Mumuni gateway uptime | Ongoing | +| Watchdog status | ✅ Online (2D uptime) | + +### Hermes Agent Zulip Plugin Improvements + +Based on the audit, improvements that should be ported to all Hermes Zulip adapters: + +1. **Circuit breaker pattern** — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff. +2. **Credential fallback** — All Hermes agents use Infisical for credentials. Add `.env` local fallback per L4 pattern for `ZULIP_API_KEY`. +3. **Queue re-registration** — Handle `BAD_EVENT_QUEUE_ID` with automatic re-registration instead of gateway restart. +4. **Supervisor watchdog** — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks. +5. **Streaming** — All agents have `streaming: true` in their zulip config. Verify `edit_message()` is implemented in each adapter. + +### Abiba pi Zulip Extension v2 — Implemented Resilience Summary + +| Feature | Status | Notes | +|---------|--------|-------| +| Circuit breaker | ✅ | CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout | +| Retry with jitter | ✅ | 2 attempts, 200ms base, 50-100% jitter | +| Queue lifecycle | ✅ | 10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling | +| Crash prevention | ✅ | uncaughtException + unhandledRejection recovery | +| Worker timeout | ✅ | 5min busy timeout → SIGKILL + error DM | +| Health endpoint | ✅ | :9200 with circuit breaker metrics | +| Echo prevention | ✅ | Dynamic bot user resolution | +| Poll timeout (AbortError) | ✅ v2.1 | Normal timeout returns [] instead of error | +| Credential fallback | ✅ v2.1 | .env file before Infisical exec | +| Provider auto-fix | ✅ | Detects reasoning_content models, switches to compatible |