prose-contracts/zulip-resilience-v3.prose.md

kind, name, description, replaces, agent, triggers

kind	name	description	replaces	agent	triggers
responsibility	zulip-resilience-v3	Rewrite the pi Zulip gateway with production-grade resilience patterns drawn from Zulip's own event system docs (queue lifecycle, heartbeat monitoring, BAD_EVENT_QUEUE_ID handling, idle_queue_timeout) and battle-tested Node.js resilience patterns (circuit breaker, exponential backoff with jitter, bulkhead isolation, supervisor watchdog).	zulip-self-heal (retired)	abiba	/zulip self-heal v3 zulip stopped responding PM2 abiba-zulip crashed

Zulip Gateway v3 — Production Resilience

Architecture Overview

The current v2 gateway (/root/.pi/agent/extensions/zulip/index.js) has three structural weaknesses that cause repeated deaths:

1. No crash recovery — uncaught errors kill the Node process, PM2 exhausts max_restarts
2. No circuit breaker — 502/fetch-failed errors escalate to process death with no fallback
3. No queue lifecycle management — doesn't use Zulip's documented heartbeat protocol or idle_queue_timeout, so BAD_EVENT_QUEUE_ID errors cascade into crashes

The v3 rewrite addresses all three, following patterns from:

• Zulip Events System docs — queue registration, heartbeat, BAD_EVENT_QUEUE_ID recovery, call_on_each_event loop
• Zulip API: Get Events — long-poll timeout, dont_block, event ack
• Circuit Breaker & Retry Patterns in Node.js 2026 — Opossum-based circuit breaker with fallback, retry with jitter, bulkhead isolation

---

Maintains

• zulip-gateway: { status: "healthy" | "degraded" | "down" }
• circuit-breaker: { state: "CLOSED" | "OPEN" | "HALF_OPEN", failures, successes }
• queue-lifecycle: { queue_id, last_event_id, idle_timeout, heartbeat_age }
• workers: { count, busy, idle, stuck }
• supervisor: { pid, last_check, health_failures }

---

Detection Rules

Rule 1: Queue Expired (BAD_EVENT_QUEUE_ID)

• Detect: Events API returns error with BAD_EVENT_QUEUE_ID in body
• Fix: Call POST /register to create new queue, update queue_id and last_event_id
• Debounce: If 3 re-registrations fail within 60s, escalate (server may be down)
• Ref: Zulip docs: "Your software will need to handle that error condition by re-initializing itself"

Rule 2: Network Degradation (502/ECONNREFUSED/fetch failed)

• Detect: Events API returns 502 or network error
• Circuit breaker: Track failure rate over 10s rolling window
  • CLOSED → OPEN: 50% failure rate with ≥5 requests
  • OPEN → HALF_OPEN: After 30s reset timeout
  • HALF_OPEN → CLOSED: Probe succeeds
  • HALF_OPEN → OPEN: Probe fails
• While OPEN: Log errors, skip events, notify user via DM: "⚠️ Zulip connection degraded — will retry in 30s"

Rule 3: Long-Poll Timeout (natural)

• Detect: Events API response takes > event_queue_longpoll_timeout_seconds
• Not an error: Server sends heartbeat events when no real events. Simply re-poll.

Rule 4: Worker Busy Timeout (>5 min)

• Detect: Worker busySince exceeds 5 minutes
• Fix: SIGKILL worker, send error DM, clean up pending replies

Rule 5: Process Crash (uncaught)

• Detect: uncaughtException / unhandledRejection fires
• Fix: Log → clear poll timer → attempt reconnect with backoff → if reconnect fails 3x, exit(1) and let PM2 restart

Rule 6: Supervisor Detects Router Stall

• Detect: External supervisor (zulip-watchdog) polls /health every 30s. If 3 consecutive failures:
• Fix: pm2 restart abiba-zulip gracefully (SIGTERM, drain workers, restart)

---

Implementation Plan

Phase 1: Rewrite Router Core (circuit-breaker + queue lifecycle)

Replace the poll loop in index.js with a resilience-first event loop:

// Queue lifecycle (Zulip docs pattern)
async function createOrRefreshQueue() {
  // POST /register with event_types=["message"]
  // Store: queueId, lastEventId, eventQueueLongpollTimeoutSeconds
  // NEW: pass idle_queue_timeout parameter (Zulip 12.0+)
}

// Circuit breaker (Opossum pattern, implemented inline to avoid dependency)
class ZulipCircuitBreaker {
  constructor({ failureThreshold=0.5, resetTimeout=30000, volumeThreshold=5, windowMs=10000 }) {
    this.state = "CLOSED"; // CLOSED | OPEN | HALF_OPEN
    this.failures = 0;
    this.successes = 0;
    this.totalRequests = 0;
    this.lastFailureTime = null;
    this.openedAt = null;
    this.failureThreshold = failureThreshold;
    this.resetTimeout = resetTimeout;
    this.volumeThreshold = volumeThreshold;
    this.windowMs = windowMs;
  }

  async fire(fn) {
    if (this.state === "OPEN") {
      if (Date.now() - this.openedAt > this.resetTimeout) {
        this.state = "HALF_OPEN";
      } else {
        throw new CircuitOpenError("Circuit is OPEN");
      }
    }
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  onSuccess() {
    this.successes++;
    this.totalRequests++;
    if (this.state === "HALF_OPEN") {
      this.state = "CLOSED";
      this.failures = 0;
    }
    // Reset counters periodically
    if (this.totalRequests > this.volumeThreshold * 2) {
      this.failures = Math.floor(this.failures / 2);
      this.successes = Math.floor(this.successes / 2);
      this.totalRequests = Math.floor(this.totalRequests / 2);
    }
  }

  onFailure() {
    this.failures++;
    this.totalRequests++;
    this.lastFailureTime = Date.now();
    if (this.totalRequests >= this.volumeThreshold &&
        this.failures / this.totalRequests >= this.failureThreshold) {
      if (this.state !== "OPEN") {
        this.state = "OPEN";
        this.openedAt = Date.now();
        console.error(`[zulip-ext] CIRCUIT BREAKER OPEN — ${this.failures}/${this.totalRequests} failures`);
      }
    }
  }
}

// Retry with exponential backoff + jitter (from resilience patterns)
async function withRetry(fn, { maxAttempts=3, baseDelay=200, maxDelay=10000, shouldRetry=()=>true }={}) {
  let lastError;
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      if (attempt === maxAttempts || !shouldRetry(err)) throw err;
      const delay = Math.min(baseDelay * Math.pow(2, attempt - 1), maxDelay);
      const jitter = delay * (0.5 + Math.random() * 0.5); // 50-100% of delay
      console.warn(`[zulip-ext] Retry ${attempt}/${maxAttempts} after ${Math.round(jitter)}ms: ${err.message.slice(0,80)}`);
      await new Promise(r => setTimeout(r, jitter));
    }
  }
  throw lastError;
}

// Resilience-first event loop (Zulip call_on_each_event pattern)
async function resilientPollLoop() {
  while (connected) {
    try {
      const events = await circuitBreaker.fire(() =>
        withRetry(() => zulipQueue.poll(), {
          maxAttempts: 2,
          baseDelay: 1000,
          shouldRetry: (err) => {
            const msg = err.message || "";
            return msg.includes("fetch failed") || msg.includes("ECONN") || msg.includes("network");
          }
        })
      );

      lastError = null;
      retryCount = 0;

      for (const ev of events) {
        await processEvent(ev);
      }

      heartbeat();
    } catch (err) {
      const msg = err instanceof Error ? err.message : String(err);

      if (msg.includes("BAD_EVENT_QUEUE_ID") || msg.includes("deregistered")) {
        // Queue expired — re-register (Zulip docs pattern)
        console.log(`[zulip-ext] Queue expired, re-registering… (${msg.slice(0,80)})`);
        try {
          zulipQueue = await createZulipQueue();
          console.log(`[zulip-ext] Re-registered, new queue=${zulipQueue.queueId}`);
        } catch (reRegErr) {
          console.error(`[zulip-ext] Re-registration failed: ${reRegErr.message}`);
          connected = false;
          retryCount++;
          const backoff = Math.min(5000 * Math.pow(2, retryCount), 300000);
          console.log(`[zulip-ext] Full reconnect in ${Math.round(backoff/1000)}s`);
          await new Promise(r => setTimeout(r, backoff));
          await startPolling();
          return;
        }
      } else if (err.name === "CircuitOpenError") {
        // Circuit is open — skip this cycle, wait for HALF_OPEN
        lastError = "circuit_open";
        await new Promise(r => setTimeout(r, POLL_INTERVAL_MS));
      } else {
        lastError = msg;
        retryCount++;
        const backoff = Math.min(POLL_INTERVAL_MS * Math.pow(1.5, Math.min(retryCount, 8)), 60000);
        console.error(`[zulip-ext] Poll error (retry ${retryCount}, backoff ${backoff}ms): ${msg}`);
        await new Promise(r => setTimeout(r, backoff));
      }
    }
  }
}

Phase 2: PM2 Hardening

Create /root/.pm2/ecosystem.config.cjs:

module.exports = {
  apps: [
    {
      name: "abiba-zulip",
      script: "/bin/pi",
      args: "--mode rpc --session-id zulip-service",
      env: {
        ZULIP_ROLE: "router",
        ZULIP_SITE: "https://chat.sysloggh.net",
        ZULIP_EMAIL: "abiba-bot@chat.sysloggh.net",
        ZULIP_API_KEY: process.env.ZULIP_API_KEY,
        AGENT_NAME: "abiba",
        AGENT_OWNER_EMAIL: "jerome@sysloggh.com",
      },
      max_restarts: 100,            // Up from default 10 — crash loops won't exhaust
      min_uptime: "10s",            // Must survive 10s to count as "alive"
      max_memory_restart: "500M",   // OOM protection
      restart_delay: 5000,          // 5s between restarts
      kill_timeout: 15000,          // 15s SIGTERM grace before SIGKILL
      listen_timeout: 30000,        // 30s to bind health port
      log_date_format: "YYYY-MM-DD HH:mm:ss Z",
      error_file: "/root/.pm2/logs/abiba-zulip-error.log",
      out_file: "/root/.pm2/logs/abiba-zulip-out.log",
      merge_logs: true,
      autorestart: true,
      watch: false,
      instances: 1,
      exec_mode: "fork",
    },
    {
      name: "zulip-watchdog",
      script: "/root/.pi/agent/extensions/zulip/watchdog.js",
      max_restarts: 10,
      min_uptime: "3s",
      restart_delay: 3000,
      autorestart: true,
    },
  ],
};

Phase 3: Supervisor Watchdog

Create /root/.pi/agent/extensions/zulip/watchdog.js:

// External supervisor — monitors router health and restarts if stalled.
// This is the pattern Hermes uses: an external process that can recover
// the gateway even if the gateway process itself is hung (not just crashed).

const HEALTH_URL = "http://127.0.0.1:9200/health";
const CHECK_INTERVAL_MS = 30_000;
const MAX_FAILURES = 3;

let failures = 0;

async function check() {
  try {
    const res = await fetch(HEALTH_URL, { signal: AbortSignal.timeout(5000) });
    if (res.ok) {
      const data = await res.json();
      if (data.status === "ok" && data.zulip?.connected) {
        if (failures > 0) {
          console.log(`[watchdog] Router recovered after ${failures} failures`);
        }
        failures = 0;
        return;
      }
    }
    failures++;
    console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: status not ok`);
  } catch (err) {
    failures++;
    console.warn(`[watchdog] Health check ${failures}/${MAX_FAILURES}: ${err.message}`);
  }

  if (failures >= MAX_FAILURES) {
    console.error(`[watchdog] ${MAX_FAILURES} consecutive failures — restarting abiba-zulip`);
    const { execSync } = require("child_process");
    try {
      execSync("pm2 restart abiba-zulip", { timeout: 30000 });
      console.log("[watchdog] Restart command sent");
    } catch (e) {
      console.error(`[watchdog] Restart failed: ${e.message}`);
    }
    failures = 0;
    // Wait for restart to complete before checking again
    await new Promise(r => setTimeout(r, 15000));
  }
}

console.log("[watchdog] Zulip gateway supervisor started");
setInterval(check, CHECK_INTERVAL_MS);
check(); // Immediate first check

Phase 4: Health Endpoint Enhancement

Add circuit breaker stats to the existing health endpoint:

// In /health response, add:
"circuit_breaker": {
  "state": circuitBreaker.state,
  "failures": circuitBreaker.failures,
  "successes": circuitBreaker.successes,
  "total_requests": circuitBreaker.totalRequests,
  "failure_rate": circuitBreaker.totalRequests > 0
    ? (circuitBreaker.failures / circuitBreaker.totalRequests).toFixed(2)
    : "0.00"
}

---

Test Plan

Test 1: Queue Re-registration

1. Manually delete the Zulip event queue via API
2. Next poll should detect BAD_EVENT_QUEUE_ID
3. Router should auto re-register within 1 poll cycle
4. Verify: /health shows new queue_id, connected=true

Test 2: Circuit Breaker Trip

1. Block Zulip server with iptables: iptables -A OUTPUT -d 192.168.68.19 -j DROP
2. Router should detect failures, trip circuit after 5 failures
3. /health should show circuit_breaker.state = "OPEN"
4. Remove iptables rule
5. Circuit should transition to HALF_OPEN → CLOSED within 60s
6. Verify: messages processed after recovery

Test 3: Supervisor Recovery

1. Kill the router process: kill -STOP $(pm2 pid abiba-zulip) (freeze, don't kill)
2. Watchdog should detect 3 failed health checks in 90s
3. Watchdog should execute pm2 restart abiba-zulip
4. Verify: router back online, connected=true

Test 4: Worker Busy Timeout

1. Send a message that triggers a long-running operation
2. If worker stays busy >5 minutes, should receive SIGKILL
3. User should receive error DM: "Response timed out"

Test 5: End-to-End Message

1. Send DM "What time is it?" from Jerome
2. Should receive response within 30s
3. /health should show messages_processed incremented

---

Rollback Plan

If v3 causes issues:

1. pm2 delete abiba-zulip; pm2 delete zulip-watchdog
2. Restore v2 from git: cd /root/.pi/agent/extensions/zulip && git checkout index.js
3. pm2 resurrect to reload previous process list
4. Verify: /health returns ok

Backup v2 before starting: cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S)

---

Success Metrics

Metric	Current (v2)	Target (v3)
Uptime between manual interventions	1-3 days	30+ days
Crash recovery	Manual (PM2 resurrect)	Automatic (circuit breaker + supervisor)
Queue expiry handling	Crash	Auto re-register
Busy worker deadlock	Router death	Worker SIGKILL + error DM
PM2 restart exhaustion	Yes (max_restarts=10)	No (max_restarts=100 + watchdog)

---

Incident Log — 2026-07-18 Fleet-Wide Audit

Fleet State After Audit

Agent	Platform	Zulip State	Issues Found	Fix Applied
Abiba	pi (CT 100)	✅ Connected	API key missing from Infisical injection; poll timeout noise	Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s
Tanko	Hermes (CT 112)	✅ Connected	Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip	Full gateway restart (kill wrapper, let infisical-gateway.sh respawn)
Mumuni	Hermes (CT 114)	✅ Connected	No issues found	None needed

Key Fixes Applied

1. Abiba — Credential Fallback (L4 Pattern)

• Root cause: zulip.api_key in config.yaml is "" (expected from Infisical). Infisical vault ABIBA_ZULIP_API_KEY wasn't being injected into the process environment.
• Fix: Added .env file fallback at /root/.pi/agent/extensions/zulip/.env with known-working key, sourced before the Infisical exec.
• Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local .env fallback.

2. Abiba — Poll Timeout Handling

• Root cause: Zulip long-poll uses AbortSignal.timeout(65000). Zulip's default event_queue_longpoll_timeout_seconds can exceed 65s. When the signal fires, an AbortError is thrown and caught by the circuit breaker as a failure.
• Fix: Caught AbortError inside poll() and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default.
• Reference: Zulip Events System — long-poll timeout

3. Tanko — Gateway Restart

• Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (infisical-gateway.sh) restarts on crash but the gateway wasn't re-establishing Zulip on restart.
• Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully.

Fleet-Wide Zulip Health Metrics (as of 2026-07-18)

Metric	Value
Zulip server	✅ HTTP 200
Agents connected	3/3 (Abiba, Tanko, Mumuni)
Abiba circuit breaker	CLOSED (0 failures)
Abiba uptime	2D (post-restart)
Tanko gateway uptime	Ongoing
Mumuni gateway uptime	Ongoing
Watchdog status	✅ Online (2D uptime)

Hermes Agent Zulip Plugin Improvements

Based on the audit, improvements that should be ported to all Hermes Zulip adapters:

1. Circuit breaker pattern — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff.
2. Credential fallback — All Hermes agents use Infisical for credentials. Add .env local fallback per L4 pattern for ZULIP_API_KEY.
3. Queue re-registration — Handle BAD_EVENT_QUEUE_ID with automatic re-registration instead of gateway restart.
4. Supervisor watchdog — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks.
5. Streaming — All agents have streaming: true in their zulip config. Verify edit_message() is implemented in each adapter.

Abiba pi Zulip Extension v2 — Implemented Resilience Summary

Feature	Status	Notes
Circuit breaker	✅	CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout
Retry with jitter	✅	2 attempts, 200ms base, 50-100% jitter
Queue lifecycle	✅	10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling
Crash prevention	✅	uncaughtException + unhandledRejection recovery
Worker timeout	✅	5min busy timeout → SIGKILL + error DM
Health endpoint	✅	:9200 with circuit breaker metrics
Echo prevention	✅	Dynamic bot user resolution
Poll timeout (AbortError)	✅ v2.1	Normal timeout returns [] instead of error
Credential fallback	✅ v2.1	.env file before Infisical exec
Provider auto-fix	✅	Detects reasoning_content models, switches to compatible