feat: Zulip v3 resilience contract + restore playbook

- zulip-resilience-v3.prose.md: Production resilience rewrite contract covering circuit breaker, retry with jitter, queue lifecycle management, supervisor watchdog, and PM2 hardening. Research-backed from Zulip event system docs. - abiba-zulip-restore.prose.md: Quick-restore playbook for recovery scenarios.
2026-07-13 21:29:35 +00:00
parent 190ceb9be4
commit 30bf42b841
2 changed files with 720 additions and 0 deletions
@@ -0,0 +1,309 @@
 ---
 kind: function
 name: abiba-zulip-restore
 description: >
  Restores Zulip connectivity for Abiba (pi agent). Verifies the v2 router-worker
  extension code, starts a PM2 process as the Zulip gateway with correct env vars,
  validates health endpoint, and confirms DM delivery. Run this whenever Abiba
  stops responding on Zulip or after system restart.
 agent: abiba
 version: 1.0.0
 status: active
 runtime_contract: 2
 ---
 # Abiba Zulip Restore — Resume pi Zulip Communication
 Single-shot function that restores full Zulip connectivity for the Abiba pi agent.
 Covers extension code validation, PM2 process management, health endpoint
 verification, and DM loopback testing.
 ## Live-State Fields
 | Field | Value | Trust |
 |-------|-------|-------|
 | Agent name | abiba | ✅ |
 | Bot email | abiba-bot@chat.sysloggh.net | ✅ |
 | Zulip server | https://chat.sysloggh.net | ✅ |
 | Extension path | /root/.pi/agent/extensions/zulip/index.js | ✅ |
 | Config path | /root/.pi/agent/extensions/zulip/config.yaml | ✅ |
 | Health port | 9200 | ✅ |
 | @all-bots user ID | 20 | ✅ (config, verified by API at runtime) |
 | PM2 process name | abiba-zulip | ✅ |
 | Provider | syslog-harness (http://192.168.68.116/v1) | ✅ |
 | Default model | deepseek-v4-pro | ✅ (settings.json) |
 ## Architecture
 The pi Zulip extension uses a **router-worker architecture** (v2):
 - **Router** (PM2 `abiba-zulip`, `ZULIP_ROLE=router`): Polls Zulip for events,
  maintains a per-sender pool of pi RPC worker processes. Each sender gets
  their own `pi --mode rpc --session-dir` process with persistent sessions.
  Handles streaming edits back to Zulip.
 - **Worker** (child `pi --mode rpc`): Runs the agent with per-sender persistent
  sessions. No Zulip logic in the worker — the router handles all Zulip I/O.
 The extension loads in ALL pi sessions (via `settings.json` extensions array)
 but is a **NO-OP** unless `ZULIP_ROLE=router` is set. Only the PM2 router process
 carries the env var.
 ## Maintains
 - extension_valid: bool — Extension code imports without errors
 - pm2_running: bool — PM2 process `abiba-zulip` is online
 - health_responding: bool — GET :9200/health returns "ok"
 - zulip_connected: bool — Queue registered, bot identity resolved
 - loopback_delivered: bool — Test DM sent and received
 - model_valid: bool — Configured models match LiteLLM authorized models
 ### Postconditions
 - PM2 process `abiba-zulip` online and stable (uptime > 30s)
 - Health endpoint returns `{ status: "ok", connected: true }`
 - Worker pool creates sessions on demand
 - Echo prevention active (BOT_EMAILS includes all known bots)
 - PM2 saved for auto-restart on boot
 ## Requires
 - Node.js with `yaml` module available
 - PM2 installed globally
 - Zulip API key in `config.yaml`
 - `pi` CLI available on PATH
 - Zulip server accessible at https://chat.sysloggh.net
 - LiteLLM provider accessible at http://192.168.68.116/v1
 ## Execution
 ### Step 1: Validate Extension Code
 ```bash
 node -e "import('file:///root/.pi/agent/extensions/zulip/index.js').then(() => console.log('OK')).catch(e => {console.error('FAIL:', e.message); process.exit(1)})"
 ```
 Expected: `OK`. If FAIL → check for missing dependencies, syntax errors.
 ### Step 2: Validate Model IDs
 Compare configured models against LiteLLM authorized models:
 ```bash
 API_KEY=$(grep -oP 'apiKey:\s*\K.*' /root/.pi/agent/models.json | head -1)
 curl -s -H "Authorization: Bearer $API_KEY" http://192.168.68.116/v1/models | \
  python3 -c "import json,sys; d=json.load(sys.stdin); [print(m['id']) for m in d.get('data',[])]" 2>/dev/null
 ```
 Check: Every model ID in `models.json` must appear in the authorized list.
 If not → fix `models.json` to only include authorized models (prefer `syslog-auto`).
 ### Step 3: Verify Config Integrity
 ```bash
 python3 -c "
 import yaml, sys
 with open('/root/.pi/agent/extensions/zulip/config.yaml') as f:
    cfg = yaml.safe_load(f)
 required = ['zulip.site', 'zulip.email', 'zulip.api_key', 'agent.name']
 for k in required:
    keys = k.split('.')
    v = cfg
    for kk in keys:
        v = v.get(kk)
        if v is None:
            print(f'MISSING: {k}')
            sys.exit(1)
 print('Config valid')
 print(f'  site={cfg[\"zulip\"][\"site\"]}')
 print(f'  email={cfg[\"zulip\"][\"email\"]}')
 print(f'  agent={cfg[\"agent\"][\"name\"]}')
 print(f'  health_port={cfg.get(\"health_port\", 9200)}')
 "
 ```
 Expected: Config valid with all fields non-empty.
 ### Step 4: Remove Stale Systemd Service
 The old `abiba-zulip.service` points to `/opt/abiba-zulip/dist/index.js` (compiled
 TypeScript, not the v2 extension). It's disabled and stale. Remove it:
 ```bash
 systemctl stop abiba-zulip 2>/dev/null || true
 systemctl disable abiba-zulip 2>/dev/null || true
 rm -f /etc/systemd/system/abiba-zulip.service
 systemctl daemon-reload
 ```
 ### Step 5: Start PM2 Process
 **Critical:** The extension MUST run via `pi --mode rpc`, NOT `node index.js` directly.
 The extension exports a function that requires pi's session lifecycle. Running
 `node index.js` loads the module but never calls the export, so nothing happens.
 `pi --mode rpc` loads all extensions (including Zulip) and fires `session_start`.
 ```bash
 # Stop existing if any
 pm2 delete abiba-zulip 2>/dev/null || true
 # Start pi in RPC mode with ZULIP_ROLE=router env
 ZULIP_ROLE=router pm2 start "$(which pi)" \
  --name abiba-zulip \
  --interpreter none \
  -- --mode rpc --no-session
 ```
 Wait 10 seconds for pi to load all extensions, fire session_start, and the Zulip
 router to register its event queue.
 ### Step 6: Validate PM2 Process
 ```bash
 pm2 show abiba-zulip --no-color
 ```
 Check: `status=online`, `restarts=0`, `uptime > 5s`.
 ### Step 7: Check Logs for Connection
 ```bash
 sleep 3
 tail -20 /root/.pm2/logs/abiba-zulip-out.log
 ```
 Look for:
 - `[zulip-ext] Connecting to https://chat.sysloggh.net as abiba-bot@chat.sysloggh.net…`
 - `[zulip-ext] Bot user_id=N, all-bots user_id=N`
 - `[zulip-ext] Connected, queue=N`
 - `[zulip-ext] Echo prevention: N bot emails`
 - `[zulip-ext] Health endpoint on :9200`
 If error → check API key, network to chat.sysloggh.net.
 ### Step 8: Health Endpoint
 ```bash
 curl -s http://localhost:9200/health | python3 -m json.tool
 ```
 Check:
 - `status: "ok"` (not "down")
 - `zulip.connected: true`
 - `zulip.queue_id` is non-null string
 - `zulip.bot_user_id` is positive integer
 ### Step 9: DM Loopback Test
 ```bash
 curl -s http://localhost:9200/health | python3 -c "
 import json,sys
 d = json.load(sys.stdin)
 if d.get('zulip',{}).get('connected'):
    print(f'✅ Zulip connected. Queue: {d[\"zulip\"][\"queue_id\"]}')
    print(f'   Bot user_id: {d[\"zulip\"][\"bot_user_id\"]}')
    print(f'   Messages processed: {d[\"zulip\"][\"messages_processed\"]}')
 else:
    print('❌ Zulip NOT connected')
    sys.exit(1)
 "
 ```
 ### Step 10: Save PM2 for Auto-Start
 ```bash
 pm2 save
 pm2 startup systemd -u root --hp /root 2>/dev/null || true
 ```
 ### Step 11: Report
 Compile results:
 | Check | Pass? |
 |-------|-------|
 | Extension code imports | extension_valid |
 | Model IDs authorized | model_valid |
 | Config integrity | config_valid |
 | PM2 process online | pm2_running |
 | Health endpoint | health_responding |
 | Zulip connected | zulip_connected |
 All pass → ✅ **Abiba Zulip restored.** Relay success to user.
 Partial failure → see recovery matrix below.
 ## Recovery Matrix
 | Failure | Recovery |
 |---------|----------|
 | Extension import fails | Check `node_modules/zulip-js` exists; run `npm install` in extension dir |
 | Model ID mismatch | Fix `models.json` to use `syslog-auto` as default model; remove invalid IDs |
 | Config missing | Restore from backup or recreate from scratch |
 | PM2 won't start | Check `node` version (>=18); check port 9200 not in use |
 | Health "down" | Check logs for connection errors; verify Zulip API key; check network |
 | "address already in use" | Kill old process: `fuser -k 9200/tcp` |
 | Queue registration fails | Check Zulip API key validity; verify bot is active in Zulip admin |
 | Rate limit (429) | Extension has built-in retry-after handling — wait, don't restart |
 ## Known Failure Modes
 | Symptom | Root Cause | Recovery |
 |---------|-----------|----------|
 | Extension loads but no events | `ZULIP_ROLE` not set | Ensure PM2 env has `ZULIP_ROLE=router` |
 | Worker stays "busy" forever | Model ID not authorized by LiteLLM | Fix models.json (lesson #11) |
 | Placeholder sent but no response | editMessage API fails silently | Extension has fallback (sends new msg); check Zulip API |
 | Queue expires rapidly | Poll interval too aggressive | v2 uses 3s poll with long-poll — should be fine |
 | Bot doesn't respond to @mentions | Not subscribed to stream | Bot auto-subscribes via API |
 | Stale error in health | `last_error` not cleared | v2 clears on successful poll (lesson #4) |
 ## Appendix: Root Cause & Fix Summary (2026-07-13)
 **Problem:** Zulip extension was offline — no PM2 process running.
 **Root cause:** The PM2 command ran `node index.js` directly (which loads the
 extension module but never calls the exported function). The extension requires
 pi's session lifecycle — pi loads extensions, fires `session_start`, and the
 Zulip extension hooks into that event.
 **Fix:** Run `pi --mode rpc` (not `node index.js`). The `--mode rpc` flag keeps
 pi alive listening for RPC commands on stdin while the Zulip extension's router
 runs in the background via the `session_start` hook.
 ```bash
 ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \
  --interpreter none -- --mode rpc --no-session
 ```
 **Additional fixes applied:**
 - Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM)
  with `syslog-auto` + `ornith-1.0-35b` (prevents silent worker failure per
  Lesson #11)
 - Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code)
 - PM2 saved for auto-restart on boot
 ## Appendix: PM2 Ecosystem Config (Optional)
 If preferred over manual `pm2 start`, create `/root/ecosystem.config.js` entry:
 ```js
 module.exports = {
  apps: [{
    name: 'abiba-zulip',
    script: '/bin/pi',
    interpreter: 'none',
    args: '--mode rpc --no-session',
    cwd: '/root',
    env: {
      ZULIP_ROLE: 'router',
    },
    log_file: '/root/.pm2/logs/abiba-zulip-out.log',
    error_file: '/root/.pm2/logs/abiba-zulip-error.log',
    max_restarts: 20,
    restart_delay: 5000,
  }]
 };
 ```
 ---
 **Last verified good state**: 2026-07-13 — Extension v2 running via `pi --mode rpc`, health endpoint :9200 returning `{status:"ok",connected:true}`, queue a669f21e.
@@ -0,0 +1,411 @@
 ---
 kind: responsibility
 name: zulip-resilience-v3
 description: >
  Rewrite the pi Zulip gateway with production-grade resilience patterns drawn from
  Zulip's own event system docs (queue lifecycle, heartbeat monitoring, BAD_EVENT_QUEUE_ID
  handling, idle_queue_timeout) and battle-tested Node.js resilience patterns
  (circuit breaker, exponential backoff with jitter, bulkhead isolation, supervisor watchdog).
 replaces: zulip-self-heal (retired)
 agent: abiba
 triggers:
  - "/zulip self-heal v3"
  - "zulip stopped responding"
  - "PM2 abiba-zulip crashed"
 ---
 # Zulip Gateway v3 — Production Resilience
 ## Architecture Overview
 The current v2 gateway (`/root/.pi/agent/extensions/zulip/index.js`) has three structural
 weaknesses that cause repeated deaths:
 1. **No crash recovery** — uncaught errors kill the Node process, PM2 exhausts max_restarts
 2. **No circuit breaker** — 502/fetch-failed errors escalate to process death with no fallback
 3. **No queue lifecycle management** — doesn't use Zulip's documented heartbeat protocol or
   idle_queue_timeout, so BAD_EVENT_QUEUE_ID errors cascade into crashes
 The v3 rewrite addresses all three, following patterns from:
 - [Zulip Events System docs](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html) —
  queue registration, heartbeat, BAD_EVENT_QUEUE_ID recovery, call_on_each_event loop
 - [Zulip API: Get Events](https://zulip.com/api/get-events) — long-poll timeout, dont_block, event ack
 - [Circuit Breaker & Retry Patterns in Node.js 2026](https://1xapi.com/blog/resilient-api-circuit-breaker-bulkhead-retry-nodejs-2026) —
  Opossum-based circuit breaker with fallback, retry with jitter, bulkhead isolation
 ---
 ## Maintains
 - `zulip-gateway`: { status: "healthy" | "degraded" | "down" }
 - `circuit-breaker`: { state: "CLOSED" | "OPEN" | "HALF_OPEN", failures, successes }
 - `queue-lifecycle`: { queue_id, last_event_id, idle_timeout, heartbeat_age }
 - `workers`: { count, busy, idle, stuck }
 - `supervisor`: { pid, last_check, health_failures }
 ---
 ## Detection Rules
 ### Rule 1: Queue Expired (BAD_EVENT_QUEUE_ID)
 - **Detect**: Events API returns error with BAD_EVENT_QUEUE_ID in body
 - **Fix**: Call `POST /register` to create new queue, update queue_id and last_event_id
 - **Debounce**: If 3 re-registrations fail within 60s, escalate (server may be down)
 - **Ref**: Zulip docs: "Your software will need to handle that error condition by re-initializing itself"
 ### Rule 2: Network Degradation (502/ECONNREFUSED/fetch failed)
 - **Detect**: Events API returns 502 or network error
 - **Circuit breaker**: Track failure rate over 10s rolling window
  - CLOSED → OPEN: 50% failure rate with ≥5 requests
  - OPEN → HALF_OPEN: After 30s reset timeout
  - HALF_OPEN → CLOSED: Probe succeeds
  - HALF_OPEN → OPEN: Probe fails
 - **While OPEN**: Log errors, skip events, notify user via DM: "⚠️ Zulip connection degraded — will retry in 30s"
 ### Rule 3: Long-Poll Timeout (natural)
 - **Detect**: Events API response takes > `event_queue_longpoll_timeout_seconds`
 - **Not an error**: Server sends heartbeat events when no real events. Simply re-poll.
 ### Rule 4: Worker Busy Timeout (>5 min)
 - **Detect**: Worker `busySince` exceeds 5 minutes
 - **Fix**: SIGKILL worker, send error DM, clean up pending replies
 ### Rule 5: Process Crash (uncaught)
 - **Detect**: `uncaughtException` / `unhandledRejection` fires
 - **Fix**: Log → clear poll timer → attempt reconnect with backoff → if reconnect fails 3x, exit(1) and let PM2 restart
 ### Rule 6: Supervisor Detects Router Stall
 - **Detect**: External supervisor (`zulip-watchdog`) polls `/health` every 30s. If 3 consecutive failures:
 - **Fix**: `pm2 restart abiba-zulip` gracefully (SIGTERM, drain workers, restart)
 ---
 ## Implementation Plan
 ### Phase 1: Rewrite Router Core (circuit-breaker + queue lifecycle)
 Replace the poll loop in index.js with a resilience-first event loop:
 ```js
 // Queue lifecycle (Zulip docs pattern)
 async function createOrRefreshQueue() {
  // POST /register with event_types=["message"]
  // Store: queueId, lastEventId, eventQueueLongpollTimeoutSeconds
  // NEW: pass idle_queue_timeout parameter (Zulip 12.0+)
 }
 // Circuit breaker (Opossum pattern, implemented inline to avoid dependency)
 class ZulipCircuitBreaker {
  constructor({ failureThreshold=0.5, resetTimeout=30000, volumeThreshold=5, windowMs=10000 }) {
    this.state = "CLOSED"; // CLOSED | OPEN | HALF_OPEN
    this.failures = 0;
    this.successes = 0;
    this.totalRequests = 0;
    this.lastFailureTime = null;
    this.openedAt = null;
    this.failureThreshold = failureThreshold;
    this.resetTimeout = resetTimeout;
    this.volumeThreshold = volumeThreshold;
    this.windowMs = windowMs;
  }
  async fire(fn) {
    if (this.state === "OPEN") {
      if (Date.now() - this.openedAt > this.resetTimeout) {
        this.state = "HALF_OPEN";
      } else {
        throw new CircuitOpenError("Circuit is OPEN");
      }
    }
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }
  onSuccess() {
    this.successes++;
    this.totalRequests++;
    if (this.state === "HALF_OPEN") {
      this.state = "CLOSED";
      this.failures = 0;
    }
    // Reset counters periodically
    if (this.totalRequests > this.volumeThreshold * 2) {
      this.failures = Math.floor(this.failures / 2);
      this.successes = Math.floor(this.successes / 2);
      this.totalRequests = Math.floor(this.totalRequests / 2);
    }
  }
  onFailure() {
    this.failures++;
    this.totalRequests++;
    this.lastFailureTime = Date.now();
    if (this.totalRequests >= this.volumeThreshold &&
        this.failures / this.totalRequests >= this.failureThreshold) {
      if (this.state !== "OPEN") {
        this.state = "OPEN";
        this.openedAt = Date.now();
        console.error(`[zulip-ext] CIRCUIT BREAKER OPEN — ${this.failures}/${this.totalRequests} failures`);
      }
    }
  }
 }
 // Retry with exponential backoff + jitter (from resilience patterns)
 async function withRetry(fn, { maxAttempts=3, baseDelay=200, maxDelay=10000, shouldRetry=()=>true }={}) {
  let lastError;
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      if (attempt === maxAttempts || !shouldRetry(err)) throw err;
      const delay = Math.min(baseDelay * Math.pow(2, attempt - 1), maxDelay);
      const jitter = delay * (0.5 + Math.random() * 0.5); // 50-100% of delay
      console.warn(`[zulip-ext] Retry ${attempt}/${maxAttempts} after ${Math.round(jitter)}ms: ${err.message.slice(0,80)}`);
      await new Promise(r => setTimeout(r, jitter));
    }
  }
  throw lastError;
 }
 // Resilience-first event loop (Zulip call_on_each_event pattern)
 async function resilientPollLoop() {
  while (connected) {
    try {
      const events = await circuitBreaker.fire(() =>
        withRetry(() => zulipQueue.poll(), {
          maxAttempts: 2,
          baseDelay: 1000,
          shouldRetry: (err) => {
            const msg = err.message || "";
            return msg.includes("fetch failed") || msg.includes("ECONN") || msg.includes("network");
          }
        })
      );
      lastError = null;
      retryCount = 0;
      for (const ev of events) {
        await processEvent(ev);
      }
      heartbeat();
    } catch (err) {
      const msg = err instanceof Error ? err.message : String(err);
      if (msg.includes("BAD_EVENT_QUEUE_ID") || msg.includes("deregistered")) {
        // Queue expired — re-register (Zulip docs pattern)
        console.log(`[zulip-ext] Queue expired, re-registering… (${msg.slice(0,80)})`);
        try {
          zulipQueue = await createZulipQueue();
          console.log(`[zulip-ext] Re-registered, new queue=${zulipQueue.queueId}`);
        } catch (reRegErr) {
          console.error(`[zulip-ext] Re-registration failed: ${reRegErr.message}`);
          connected = false;
          retryCount++;
          const backoff = Math.min(5000 * Math.pow(2, retryCount), 300000);
          console.log(`[zulip-ext] Full reconnect in ${Math.round(backoff/1000)}s`);
          await new Promise(r => setTimeout(r, backoff));
          await startPolling();
          return;
        }
      } else if (err.name === "CircuitOpenError") {
        // Circuit is open — skip this cycle, wait for HALF_OPEN
        lastError = "circuit_open";
        await new Promise(r => setTimeout(r, POLL_INTERVAL_MS));
      } else {
        lastError = msg;
        retryCount++;
        const backoff = Math.min(POLL_INTERVAL_MS * Math.pow(1.5, Math.min(retryCount, 8)), 60000);
        console.error(`[zulip-ext] Poll error (retry ${retryCount}, backoff ${backoff}ms): ${msg}`);
        await new Promise(r => setTimeout(r, backoff));
      }
    }
  }
 }
 ```
 ### Phase 2: PM2 Hardening
 Create `/root/.pm2/ecosystem.config.cjs`:
 ```js
 module.exports = {
  apps: [
    {
      name: "abiba-zulip",
      script: "/bin/pi",
      args: "--mode rpc --session-id zulip-service",
      env: {
        ZULIP_ROLE: "router",
        ZULIP_SITE: "https://chat.sysloggh.net",
        ZULIP_EMAIL: "abiba-bot@chat.sysloggh.net",
        ZULIP_API_KEY: process.env.ZULIP_API_KEY,
        AGENT_NAME: "abiba",
        AGENT_OWNER_EMAIL: "jerome@sysloggh.com",
      },
      max_restarts: 100,            // Up from default 10 — crash loops won't exhaust
      min_uptime: "10s",            // Must survive 10s to count as "alive"
      max_memory_restart: "500M",   // OOM protection
      restart_delay: 5000,          // 5s between restarts
      kill_timeout: 15000,          // 15s SIGTERM grace before SIGKILL
      listen_timeout: 30000,        // 30s to bind health port
      log_date_format: "YYYY-MM-DD HH:mm:ss Z",
      error_file: "/root/.pm2/logs/abiba-zulip-error.log",
      out_file: "/root/.pm2/logs/abiba-zulip-out.log",
      merge_logs: true,
      autorestart: true,
      watch: false,
      instances: 1,
      exec_mode: "fork",
    },
    {
      name: "zulip-watchdog",
      script: "/root/.pi/agent/extensions/zulip/watchdog.js",
      max_restarts: 10,
      min_uptime: "3s",
      restart_delay: 3000,
      autorestart: true,
    },
  ],
 };
 ```
 ### Phase 3: Supervisor Watchdog
 Create `/root/.pi/agent/extensions/zulip/watchdog.js`:
 ```js
 // External supervisor — monitors router health and restarts if stalled.
 // This is the pattern Hermes uses: an external process that can recover
 // the gateway even if the gateway process itself is hung (not just crashed).
 const HEALTH_URL = "http://127.0.0.1:9200/health";
 const CHECK_INTERVAL_MS = 30_000;
 const MAX_FAILURES = 3;
 let failures = 0;
 async function check() {
  try {
    const res = await fetch(HEALTH_URL, { signal: AbortSignal.timeout(5000) });
    if (res.ok) {
      const data = await res.json();
      if (data.status === "ok" && data.zulip?.connected) {
        if (failures > 0) {
          console.log(`[watchdog] Router recovered after ${failures} failures`);
        }
        failures = 0;
        return;
      }
    }
    failures++;
    console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: status not ok`);
  } catch (err) {
    failures++;
    console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: ${err.message}`);
  }
  if (failures >= MAX_FAILURES) {
    console.error(`[watchdog] ${MAX_FAILURES} consecutive failures — restarting abiba-zulip`);
    const { execSync } = require("child_process");
    try {
      execSync("pm2 restart abiba-zulip", { timeout: 30000 });
      console.log("[watchdog] Restart command sent");
    } catch (e) {
      console.error(`[watchdog] Restart failed: ${e.message}`);
    }
    failures = 0;
    // Wait for restart to complete before checking again
    await new Promise(r => setTimeout(r, 15000));
  }
 }
 console.log("[watchdog] Zulip gateway supervisor started");
 setInterval(check, CHECK_INTERVAL_MS);
 check(); // Immediate first check
 ```
 ### Phase 4: Health Endpoint Enhancement
 Add circuit breaker stats to the existing health endpoint:
 ```js
 // In /health response, add:
 "circuit_breaker": {
  "state": circuitBreaker.state,
  "failures": circuitBreaker.failures,
  "successes": circuitBreaker.successes,
  "total_requests": circuitBreaker.totalRequests,
  "failure_rate": circuitBreaker.totalRequests > 0
    ? (circuitBreaker.failures / circuitBreaker.totalRequests).toFixed(2)
    : "0.00"
 }
 ```
 ---
 ## Test Plan
 ### Test 1: Queue Re-registration
 1. Manually delete the Zulip event queue via API
 2. Next poll should detect BAD_EVENT_QUEUE_ID
 3. Router should auto re-register within 1 poll cycle
 4. Verify: `/health` shows new queue_id, connected=true
 ### Test 2: Circuit Breaker Trip
 1. Block Zulip server with iptables: `iptables -A OUTPUT -d 192.168.68.19 -j DROP`
 2. Router should detect failures, trip circuit after 5 failures
 3. `/health` should show circuit_breaker.state = "OPEN"
 4. Remove iptables rule
 5. Circuit should transition to HALF_OPEN → CLOSED within 60s
 6. Verify: messages processed after recovery
 ### Test 3: Supervisor Recovery
 1. Kill the router process: `kill -STOP $(pm2 pid abiba-zulip)` (freeze, don't kill)
 2. Watchdog should detect 3 failed health checks in 90s
 3. Watchdog should execute `pm2 restart abiba-zulip`
 4. Verify: router back online, connected=true
 ### Test 4: Worker Busy Timeout
 1. Send a message that triggers a long-running operation
 2. If worker stays busy >5 minutes, should receive SIGKILL
 3. User should receive error DM: "Response timed out"
 ### Test 5: End-to-End Message
 1. Send DM "What time is it?" from Jerome
 2. Should receive response within 30s
 3. `/health` should show messages_processed incremented
 ---
 ## Rollback Plan
 If v3 causes issues:
 1. `pm2 delete abiba-zulip; pm2 delete zulip-watchdog`
 2. Restore v2 from git: `cd /root/.pi/agent/extensions/zulip && git checkout index.js`
 3. `pm2 resurrect` to reload previous process list
 4. Verify: `/health` returns ok
 Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S)`
 ---
 ## Success Metrics
 | Metric | Current (v2) | Target (v3) |
 |--------|-------------|-------------|
 | Uptime between manual interventions | 1-3 days | 30+ days |
 | Crash recovery | Manual (PM2 resurrect) | Automatic (circuit breaker + supervisor) |
 | Queue expiry handling | Crash | Auto re-register |
 | Busy worker deadlock | Router death | Worker SIGKILL + error DM |
 | PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |