diff --git a/abiba-zulip-restore.prose.md b/abiba-zulip-restore.prose.md new file mode 100644 index 0000000..e9fec3c --- /dev/null +++ b/abiba-zulip-restore.prose.md @@ -0,0 +1,309 @@ +--- +kind: function +name: abiba-zulip-restore +description: > + Restores Zulip connectivity for Abiba (pi agent). Verifies the v2 router-worker + extension code, starts a PM2 process as the Zulip gateway with correct env vars, + validates health endpoint, and confirms DM delivery. Run this whenever Abiba + stops responding on Zulip or after system restart. +agent: abiba +version: 1.0.0 +status: active +runtime_contract: 2 +--- + +# Abiba Zulip Restore — Resume pi Zulip Communication + +Single-shot function that restores full Zulip connectivity for the Abiba pi agent. +Covers extension code validation, PM2 process management, health endpoint +verification, and DM loopback testing. + +## Live-State Fields + +| Field | Value | Trust | +|-------|-------|-------| +| Agent name | abiba | ✅ | +| Bot email | abiba-bot@chat.sysloggh.net | ✅ | +| Zulip server | https://chat.sysloggh.net | ✅ | +| Extension path | /root/.pi/agent/extensions/zulip/index.js | ✅ | +| Config path | /root/.pi/agent/extensions/zulip/config.yaml | ✅ | +| Health port | 9200 | ✅ | +| @all-bots user ID | 20 | ✅ (config, verified by API at runtime) | +| PM2 process name | abiba-zulip | ✅ | +| Provider | syslog-harness (http://192.168.68.116/v1) | ✅ | +| Default model | deepseek-v4-pro | ✅ (settings.json) | + +## Architecture + +The pi Zulip extension uses a **router-worker architecture** (v2): + +- **Router** (PM2 `abiba-zulip`, `ZULIP_ROLE=router`): Polls Zulip for events, + maintains a per-sender pool of pi RPC worker processes. Each sender gets + their own `pi --mode rpc --session-dir` process with persistent sessions. + Handles streaming edits back to Zulip. +- **Worker** (child `pi --mode rpc`): Runs the agent with per-sender persistent + sessions. No Zulip logic in the worker — the router handles all Zulip I/O. + +The extension loads in ALL pi sessions (via `settings.json` extensions array) +but is a **NO-OP** unless `ZULIP_ROLE=router` is set. Only the PM2 router process +carries the env var. + +## Maintains + +- extension_valid: bool — Extension code imports without errors +- pm2_running: bool — PM2 process `abiba-zulip` is online +- health_responding: bool — GET :9200/health returns "ok" +- zulip_connected: bool — Queue registered, bot identity resolved +- loopback_delivered: bool — Test DM sent and received +- model_valid: bool — Configured models match LiteLLM authorized models + +### Postconditions + +- PM2 process `abiba-zulip` online and stable (uptime > 30s) +- Health endpoint returns `{ status: "ok", connected: true }` +- Worker pool creates sessions on demand +- Echo prevention active (BOT_EMAILS includes all known bots) +- PM2 saved for auto-restart on boot + +## Requires + +- Node.js with `yaml` module available +- PM2 installed globally +- Zulip API key in `config.yaml` +- `pi` CLI available on PATH +- Zulip server accessible at https://chat.sysloggh.net +- LiteLLM provider accessible at http://192.168.68.116/v1 + +## Execution + +### Step 1: Validate Extension Code + +```bash +node -e "import('file:///root/.pi/agent/extensions/zulip/index.js').then(() => console.log('OK')).catch(e => {console.error('FAIL:', e.message); process.exit(1)})" +``` + +Expected: `OK`. If FAIL → check for missing dependencies, syntax errors. + +### Step 2: Validate Model IDs + +Compare configured models against LiteLLM authorized models: + +```bash +API_KEY=$(grep -oP 'apiKey:\s*\K.*' /root/.pi/agent/models.json | head -1) +curl -s -H "Authorization: Bearer $API_KEY" http://192.168.68.116/v1/models | \ + python3 -c "import json,sys; d=json.load(sys.stdin); [print(m['id']) for m in d.get('data',[])]" 2>/dev/null +``` + +Check: Every model ID in `models.json` must appear in the authorized list. +If not → fix `models.json` to only include authorized models (prefer `syslog-auto`). + +### Step 3: Verify Config Integrity + +```bash +python3 -c " +import yaml, sys +with open('/root/.pi/agent/extensions/zulip/config.yaml') as f: + cfg = yaml.safe_load(f) +required = ['zulip.site', 'zulip.email', 'zulip.api_key', 'agent.name'] +for k in required: + keys = k.split('.') + v = cfg + for kk in keys: + v = v.get(kk) + if v is None: + print(f'MISSING: {k}') + sys.exit(1) +print('Config valid') +print(f' site={cfg[\"zulip\"][\"site\"]}') +print(f' email={cfg[\"zulip\"][\"email\"]}') +print(f' agent={cfg[\"agent\"][\"name\"]}') +print(f' health_port={cfg.get(\"health_port\", 9200)}') +" +``` + +Expected: Config valid with all fields non-empty. + +### Step 4: Remove Stale Systemd Service + +The old `abiba-zulip.service` points to `/opt/abiba-zulip/dist/index.js` (compiled +TypeScript, not the v2 extension). It's disabled and stale. Remove it: + +```bash +systemctl stop abiba-zulip 2>/dev/null || true +systemctl disable abiba-zulip 2>/dev/null || true +rm -f /etc/systemd/system/abiba-zulip.service +systemctl daemon-reload +``` + +### Step 5: Start PM2 Process + +**Critical:** The extension MUST run via `pi --mode rpc`, NOT `node index.js` directly. +The extension exports a function that requires pi's session lifecycle. Running +`node index.js` loads the module but never calls the export, so nothing happens. +`pi --mode rpc` loads all extensions (including Zulip) and fires `session_start`. + +```bash +# Stop existing if any +pm2 delete abiba-zulip 2>/dev/null || true + +# Start pi in RPC mode with ZULIP_ROLE=router env +ZULIP_ROLE=router pm2 start "$(which pi)" \ + --name abiba-zulip \ + --interpreter none \ + -- --mode rpc --no-session +``` + +Wait 10 seconds for pi to load all extensions, fire session_start, and the Zulip +router to register its event queue. + +### Step 6: Validate PM2 Process + +```bash +pm2 show abiba-zulip --no-color +``` + +Check: `status=online`, `restarts=0`, `uptime > 5s`. + +### Step 7: Check Logs for Connection + +```bash +sleep 3 +tail -20 /root/.pm2/logs/abiba-zulip-out.log +``` + +Look for: +- `[zulip-ext] Connecting to https://chat.sysloggh.net as abiba-bot@chat.sysloggh.net…` +- `[zulip-ext] Bot user_id=N, all-bots user_id=N` +- `[zulip-ext] Connected, queue=N` +- `[zulip-ext] Echo prevention: N bot emails` +- `[zulip-ext] Health endpoint on :9200` + +If error → check API key, network to chat.sysloggh.net. + +### Step 8: Health Endpoint + +```bash +curl -s http://localhost:9200/health | python3 -m json.tool +``` + +Check: +- `status: "ok"` (not "down") +- `zulip.connected: true` +- `zulip.queue_id` is non-null string +- `zulip.bot_user_id` is positive integer + +### Step 9: DM Loopback Test + +```bash +curl -s http://localhost:9200/health | python3 -c " +import json,sys +d = json.load(sys.stdin) +if d.get('zulip',{}).get('connected'): + print(f'✅ Zulip connected. Queue: {d[\"zulip\"][\"queue_id\"]}') + print(f' Bot user_id: {d[\"zulip\"][\"bot_user_id\"]}') + print(f' Messages processed: {d[\"zulip\"][\"messages_processed\"]}') +else: + print('❌ Zulip NOT connected') + sys.exit(1) +" +``` + +### Step 10: Save PM2 for Auto-Start + +```bash +pm2 save +pm2 startup systemd -u root --hp /root 2>/dev/null || true +``` + +### Step 11: Report + +Compile results: + +| Check | Pass? | +|-------|-------| +| Extension code imports | extension_valid | +| Model IDs authorized | model_valid | +| Config integrity | config_valid | +| PM2 process online | pm2_running | +| Health endpoint | health_responding | +| Zulip connected | zulip_connected | + +All pass → ✅ **Abiba Zulip restored.** Relay success to user. + +Partial failure → see recovery matrix below. + +## Recovery Matrix + +| Failure | Recovery | +|---------|----------| +| Extension import fails | Check `node_modules/zulip-js` exists; run `npm install` in extension dir | +| Model ID mismatch | Fix `models.json` to use `syslog-auto` as default model; remove invalid IDs | +| Config missing | Restore from backup or recreate from scratch | +| PM2 won't start | Check `node` version (>=18); check port 9200 not in use | +| Health "down" | Check logs for connection errors; verify Zulip API key; check network | +| "address already in use" | Kill old process: `fuser -k 9200/tcp` | +| Queue registration fails | Check Zulip API key validity; verify bot is active in Zulip admin | +| Rate limit (429) | Extension has built-in retry-after handling — wait, don't restart | + +## Known Failure Modes + +| Symptom | Root Cause | Recovery | +|---------|-----------|----------| +| Extension loads but no events | `ZULIP_ROLE` not set | Ensure PM2 env has `ZULIP_ROLE=router` | +| Worker stays "busy" forever | Model ID not authorized by LiteLLM | Fix models.json (lesson #11) | +| Placeholder sent but no response | editMessage API fails silently | Extension has fallback (sends new msg); check Zulip API | +| Queue expires rapidly | Poll interval too aggressive | v2 uses 3s poll with long-poll — should be fine | +| Bot doesn't respond to @mentions | Not subscribed to stream | Bot auto-subscribes via API | +| Stale error in health | `last_error` not cleared | v2 clears on successful poll (lesson #4) | + +## Appendix: Root Cause & Fix Summary (2026-07-13) + +**Problem:** Zulip extension was offline — no PM2 process running. + +**Root cause:** The PM2 command ran `node index.js` directly (which loads the +extension module but never calls the exported function). The extension requires +pi's session lifecycle — pi loads extensions, fires `session_start`, and the +Zulip extension hooks into that event. + +**Fix:** Run `pi --mode rpc` (not `node index.js`). The `--mode rpc` flag keeps +pi alive listening for RPC commands on stdin while the Zulip extension's router +runs in the background via the `session_start` hook. + +```bash +ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \ + --interpreter none -- --mode rpc --no-session +``` + +**Additional fixes applied:** +- Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM) + with `syslog-auto` + `ornith-1.0-35b` (prevents silent worker failure per + Lesson #11) +- Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code) +- PM2 saved for auto-restart on boot + +## Appendix: PM2 Ecosystem Config (Optional) + +If preferred over manual `pm2 start`, create `/root/ecosystem.config.js` entry: + +```js +module.exports = { + apps: [{ + name: 'abiba-zulip', + script: '/bin/pi', + interpreter: 'none', + args: '--mode rpc --no-session', + cwd: '/root', + env: { + ZULIP_ROLE: 'router', + }, + log_file: '/root/.pm2/logs/abiba-zulip-out.log', + error_file: '/root/.pm2/logs/abiba-zulip-error.log', + max_restarts: 20, + restart_delay: 5000, + }] +}; +``` + +--- + +**Last verified good state**: 2026-07-13 — Extension v2 running via `pi --mode rpc`, health endpoint :9200 returning `{status:"ok",connected:true}`, queue a669f21e. diff --git a/zulip-resilience-v3.prose.md b/zulip-resilience-v3.prose.md new file mode 100644 index 0000000..875eca7 --- /dev/null +++ b/zulip-resilience-v3.prose.md @@ -0,0 +1,411 @@ +--- +kind: responsibility +name: zulip-resilience-v3 +description: > + Rewrite the pi Zulip gateway with production-grade resilience patterns drawn from + Zulip's own event system docs (queue lifecycle, heartbeat monitoring, BAD_EVENT_QUEUE_ID + handling, idle_queue_timeout) and battle-tested Node.js resilience patterns + (circuit breaker, exponential backoff with jitter, bulkhead isolation, supervisor watchdog). +replaces: zulip-self-heal (retired) +agent: abiba +triggers: + - "/zulip self-heal v3" + - "zulip stopped responding" + - "PM2 abiba-zulip crashed" +--- + +# Zulip Gateway v3 — Production Resilience + +## Architecture Overview + +The current v2 gateway (`/root/.pi/agent/extensions/zulip/index.js`) has three structural +weaknesses that cause repeated deaths: + +1. **No crash recovery** — uncaught errors kill the Node process, PM2 exhausts max_restarts +2. **No circuit breaker** — 502/fetch-failed errors escalate to process death with no fallback +3. **No queue lifecycle management** — doesn't use Zulip's documented heartbeat protocol or + idle_queue_timeout, so BAD_EVENT_QUEUE_ID errors cascade into crashes + +The v3 rewrite addresses all three, following patterns from: + +- [Zulip Events System docs](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html) — + queue registration, heartbeat, BAD_EVENT_QUEUE_ID recovery, call_on_each_event loop +- [Zulip API: Get Events](https://zulip.com/api/get-events) — long-poll timeout, dont_block, event ack +- [Circuit Breaker & Retry Patterns in Node.js 2026](https://1xapi.com/blog/resilient-api-circuit-breaker-bulkhead-retry-nodejs-2026) — + Opossum-based circuit breaker with fallback, retry with jitter, bulkhead isolation + +--- + +## Maintains + +- `zulip-gateway`: { status: "healthy" | "degraded" | "down" } +- `circuit-breaker`: { state: "CLOSED" | "OPEN" | "HALF_OPEN", failures, successes } +- `queue-lifecycle`: { queue_id, last_event_id, idle_timeout, heartbeat_age } +- `workers`: { count, busy, idle, stuck } +- `supervisor`: { pid, last_check, health_failures } + +--- + +## Detection Rules + +### Rule 1: Queue Expired (BAD_EVENT_QUEUE_ID) +- **Detect**: Events API returns error with BAD_EVENT_QUEUE_ID in body +- **Fix**: Call `POST /register` to create new queue, update queue_id and last_event_id +- **Debounce**: If 3 re-registrations fail within 60s, escalate (server may be down) +- **Ref**: Zulip docs: "Your software will need to handle that error condition by re-initializing itself" + +### Rule 2: Network Degradation (502/ECONNREFUSED/fetch failed) +- **Detect**: Events API returns 502 or network error +- **Circuit breaker**: Track failure rate over 10s rolling window + - CLOSED → OPEN: 50% failure rate with ≥5 requests + - OPEN → HALF_OPEN: After 30s reset timeout + - HALF_OPEN → CLOSED: Probe succeeds + - HALF_OPEN → OPEN: Probe fails +- **While OPEN**: Log errors, skip events, notify user via DM: "⚠️ Zulip connection degraded — will retry in 30s" + +### Rule 3: Long-Poll Timeout (natural) +- **Detect**: Events API response takes > `event_queue_longpoll_timeout_seconds` +- **Not an error**: Server sends heartbeat events when no real events. Simply re-poll. + +### Rule 4: Worker Busy Timeout (>5 min) +- **Detect**: Worker `busySince` exceeds 5 minutes +- **Fix**: SIGKILL worker, send error DM, clean up pending replies + +### Rule 5: Process Crash (uncaught) +- **Detect**: `uncaughtException` / `unhandledRejection` fires +- **Fix**: Log → clear poll timer → attempt reconnect with backoff → if reconnect fails 3x, exit(1) and let PM2 restart + +### Rule 6: Supervisor Detects Router Stall +- **Detect**: External supervisor (`zulip-watchdog`) polls `/health` every 30s. If 3 consecutive failures: +- **Fix**: `pm2 restart abiba-zulip` gracefully (SIGTERM, drain workers, restart) + +--- + +## Implementation Plan + +### Phase 1: Rewrite Router Core (circuit-breaker + queue lifecycle) + +Replace the poll loop in index.js with a resilience-first event loop: + +```js +// Queue lifecycle (Zulip docs pattern) +async function createOrRefreshQueue() { + // POST /register with event_types=["message"] + // Store: queueId, lastEventId, eventQueueLongpollTimeoutSeconds + // NEW: pass idle_queue_timeout parameter (Zulip 12.0+) +} + +// Circuit breaker (Opossum pattern, implemented inline to avoid dependency) +class ZulipCircuitBreaker { + constructor({ failureThreshold=0.5, resetTimeout=30000, volumeThreshold=5, windowMs=10000 }) { + this.state = "CLOSED"; // CLOSED | OPEN | HALF_OPEN + this.failures = 0; + this.successes = 0; + this.totalRequests = 0; + this.lastFailureTime = null; + this.openedAt = null; + this.failureThreshold = failureThreshold; + this.resetTimeout = resetTimeout; + this.volumeThreshold = volumeThreshold; + this.windowMs = windowMs; + } + + async fire(fn) { + if (this.state === "OPEN") { + if (Date.now() - this.openedAt > this.resetTimeout) { + this.state = "HALF_OPEN"; + } else { + throw new CircuitOpenError("Circuit is OPEN"); + } + } + try { + const result = await fn(); + this.onSuccess(); + return result; + } catch (err) { + this.onFailure(); + throw err; + } + } + + onSuccess() { + this.successes++; + this.totalRequests++; + if (this.state === "HALF_OPEN") { + this.state = "CLOSED"; + this.failures = 0; + } + // Reset counters periodically + if (this.totalRequests > this.volumeThreshold * 2) { + this.failures = Math.floor(this.failures / 2); + this.successes = Math.floor(this.successes / 2); + this.totalRequests = Math.floor(this.totalRequests / 2); + } + } + + onFailure() { + this.failures++; + this.totalRequests++; + this.lastFailureTime = Date.now(); + if (this.totalRequests >= this.volumeThreshold && + this.failures / this.totalRequests >= this.failureThreshold) { + if (this.state !== "OPEN") { + this.state = "OPEN"; + this.openedAt = Date.now(); + console.error(`[zulip-ext] CIRCUIT BREAKER OPEN — ${this.failures}/${this.totalRequests} failures`); + } + } + } +} + +// Retry with exponential backoff + jitter (from resilience patterns) +async function withRetry(fn, { maxAttempts=3, baseDelay=200, maxDelay=10000, shouldRetry=()=>true }={}) { + let lastError; + for (let attempt = 1; attempt <= maxAttempts; attempt++) { + try { + return await fn(); + } catch (err) { + lastError = err; + if (attempt === maxAttempts || !shouldRetry(err)) throw err; + const delay = Math.min(baseDelay * Math.pow(2, attempt - 1), maxDelay); + const jitter = delay * (0.5 + Math.random() * 0.5); // 50-100% of delay + console.warn(`[zulip-ext] Retry ${attempt}/${maxAttempts} after ${Math.round(jitter)}ms: ${err.message.slice(0,80)}`); + await new Promise(r => setTimeout(r, jitter)); + } + } + throw lastError; +} + +// Resilience-first event loop (Zulip call_on_each_event pattern) +async function resilientPollLoop() { + while (connected) { + try { + const events = await circuitBreaker.fire(() => + withRetry(() => zulipQueue.poll(), { + maxAttempts: 2, + baseDelay: 1000, + shouldRetry: (err) => { + const msg = err.message || ""; + return msg.includes("fetch failed") || msg.includes("ECONN") || msg.includes("network"); + } + }) + ); + + lastError = null; + retryCount = 0; + + for (const ev of events) { + await processEvent(ev); + } + + heartbeat(); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + + if (msg.includes("BAD_EVENT_QUEUE_ID") || msg.includes("deregistered")) { + // Queue expired — re-register (Zulip docs pattern) + console.log(`[zulip-ext] Queue expired, re-registering… (${msg.slice(0,80)})`); + try { + zulipQueue = await createZulipQueue(); + console.log(`[zulip-ext] Re-registered, new queue=${zulipQueue.queueId}`); + } catch (reRegErr) { + console.error(`[zulip-ext] Re-registration failed: ${reRegErr.message}`); + connected = false; + retryCount++; + const backoff = Math.min(5000 * Math.pow(2, retryCount), 300000); + console.log(`[zulip-ext] Full reconnect in ${Math.round(backoff/1000)}s`); + await new Promise(r => setTimeout(r, backoff)); + await startPolling(); + return; + } + } else if (err.name === "CircuitOpenError") { + // Circuit is open — skip this cycle, wait for HALF_OPEN + lastError = "circuit_open"; + await new Promise(r => setTimeout(r, POLL_INTERVAL_MS)); + } else { + lastError = msg; + retryCount++; + const backoff = Math.min(POLL_INTERVAL_MS * Math.pow(1.5, Math.min(retryCount, 8)), 60000); + console.error(`[zulip-ext] Poll error (retry ${retryCount}, backoff ${backoff}ms): ${msg}`); + await new Promise(r => setTimeout(r, backoff)); + } + } + } +} +``` + +### Phase 2: PM2 Hardening + +Create `/root/.pm2/ecosystem.config.cjs`: + +```js +module.exports = { + apps: [ + { + name: "abiba-zulip", + script: "/bin/pi", + args: "--mode rpc --session-id zulip-service", + env: { + ZULIP_ROLE: "router", + ZULIP_SITE: "https://chat.sysloggh.net", + ZULIP_EMAIL: "abiba-bot@chat.sysloggh.net", + ZULIP_API_KEY: process.env.ZULIP_API_KEY, + AGENT_NAME: "abiba", + AGENT_OWNER_EMAIL: "jerome@sysloggh.com", + }, + max_restarts: 100, // Up from default 10 — crash loops won't exhaust + min_uptime: "10s", // Must survive 10s to count as "alive" + max_memory_restart: "500M", // OOM protection + restart_delay: 5000, // 5s between restarts + kill_timeout: 15000, // 15s SIGTERM grace before SIGKILL + listen_timeout: 30000, // 30s to bind health port + log_date_format: "YYYY-MM-DD HH:mm:ss Z", + error_file: "/root/.pm2/logs/abiba-zulip-error.log", + out_file: "/root/.pm2/logs/abiba-zulip-out.log", + merge_logs: true, + autorestart: true, + watch: false, + instances: 1, + exec_mode: "fork", + }, + { + name: "zulip-watchdog", + script: "/root/.pi/agent/extensions/zulip/watchdog.js", + max_restarts: 10, + min_uptime: "3s", + restart_delay: 3000, + autorestart: true, + }, + ], +}; +``` + +### Phase 3: Supervisor Watchdog + +Create `/root/.pi/agent/extensions/zulip/watchdog.js`: + +```js +// External supervisor — monitors router health and restarts if stalled. +// This is the pattern Hermes uses: an external process that can recover +// the gateway even if the gateway process itself is hung (not just crashed). + +const HEALTH_URL = "http://127.0.0.1:9200/health"; +const CHECK_INTERVAL_MS = 30_000; +const MAX_FAILURES = 3; + +let failures = 0; + +async function check() { + try { + const res = await fetch(HEALTH_URL, { signal: AbortSignal.timeout(5000) }); + if (res.ok) { + const data = await res.json(); + if (data.status === "ok" && data.zulip?.connected) { + if (failures > 0) { + console.log(`[watchdog] Router recovered after ${failures} failures`); + } + failures = 0; + return; + } + } + failures++; + console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: status not ok`); + } catch (err) { + failures++; + console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: ${err.message}`); + } + + if (failures >= MAX_FAILURES) { + console.error(`[watchdog] ${MAX_FAILURES} consecutive failures — restarting abiba-zulip`); + const { execSync } = require("child_process"); + try { + execSync("pm2 restart abiba-zulip", { timeout: 30000 }); + console.log("[watchdog] Restart command sent"); + } catch (e) { + console.error(`[watchdog] Restart failed: ${e.message}`); + } + failures = 0; + // Wait for restart to complete before checking again + await new Promise(r => setTimeout(r, 15000)); + } +} + +console.log("[watchdog] Zulip gateway supervisor started"); +setInterval(check, CHECK_INTERVAL_MS); +check(); // Immediate first check +``` + +### Phase 4: Health Endpoint Enhancement + +Add circuit breaker stats to the existing health endpoint: + +```js +// In /health response, add: +"circuit_breaker": { + "state": circuitBreaker.state, + "failures": circuitBreaker.failures, + "successes": circuitBreaker.successes, + "total_requests": circuitBreaker.totalRequests, + "failure_rate": circuitBreaker.totalRequests > 0 + ? (circuitBreaker.failures / circuitBreaker.totalRequests).toFixed(2) + : "0.00" +} +``` + +--- + +## Test Plan + +### Test 1: Queue Re-registration +1. Manually delete the Zulip event queue via API +2. Next poll should detect BAD_EVENT_QUEUE_ID +3. Router should auto re-register within 1 poll cycle +4. Verify: `/health` shows new queue_id, connected=true + +### Test 2: Circuit Breaker Trip +1. Block Zulip server with iptables: `iptables -A OUTPUT -d 192.168.68.19 -j DROP` +2. Router should detect failures, trip circuit after 5 failures +3. `/health` should show circuit_breaker.state = "OPEN" +4. Remove iptables rule +5. Circuit should transition to HALF_OPEN → CLOSED within 60s +6. Verify: messages processed after recovery + +### Test 3: Supervisor Recovery +1. Kill the router process: `kill -STOP $(pm2 pid abiba-zulip)` (freeze, don't kill) +2. Watchdog should detect 3 failed health checks in 90s +3. Watchdog should execute `pm2 restart abiba-zulip` +4. Verify: router back online, connected=true + +### Test 4: Worker Busy Timeout +1. Send a message that triggers a long-running operation +2. If worker stays busy >5 minutes, should receive SIGKILL +3. User should receive error DM: "Response timed out" + +### Test 5: End-to-End Message +1. Send DM "What time is it?" from Jerome +2. Should receive response within 30s +3. `/health` should show messages_processed incremented + +--- + +## Rollback Plan + +If v3 causes issues: +1. `pm2 delete abiba-zulip; pm2 delete zulip-watchdog` +2. Restore v2 from git: `cd /root/.pi/agent/extensions/zulip && git checkout index.js` +3. `pm2 resurrect` to reload previous process list +4. Verify: `/health` returns ok + +Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S)` + +--- + +## Success Metrics + +| Metric | Current (v2) | Target (v3) | +|--------|-------------|-------------| +| Uptime between manual interventions | 1-3 days | 30+ days | +| Crash recovery | Manual (PM2 resurrect) | Automatic (circuit breaker + supervisor) | +| Queue expiry handling | Crash | Auto re-register | +| Busy worker deadlock | Router death | Worker SIGKILL + error DM | +| PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |