feat: Zulip Gateway v3 — Production Resilience Rewrite #34

2026-07-13T21:30:02Z

abiba-bot commented

2026-07-13 21:30:02 +00:00

Summary

Complete rewrite of the pi Zulip gateway from crash-prone v2 to production-resilient v3. Research-backed patterns from Zulip event system docs + Node.js resilience best practices.

Root Cause

Pi workers used DeepSeek v4-pro as default model — a reasoning model that produces only reasoning_content with normal token limits. Workers waited forever for content that never arrived, hung, and were killed by the 5-minute stuck-worker timeout. Every "Processing..." placeholder was a dead worker.

Changes

extension-src/zulip-extension.js (v2→v3)

Circuit breaker (ZulipCircuitBreaker): CLOSED→OPEN→HALF_OPEN states, 50% failure threshold, volumeThreshold=2 for slow-polling, 30s reset timeout
Retry with jitter (withRetry): Exponential backoff + 50-100% jitter, max 2 attempts for transient errors only
Queue lifecycle: idle_queue_timeout=600 (Zulip 12.0+), auto re-register on BAD_EVENT_QUEUE_ID
Busy worker timeout: SIGKILL after 5 minutes + error DM to user
Crash handlers: uncaughtException + unhandledRejection → log + reconnect instead of dying
Enhanced health endpoint: Circuit breaker stats, idle_queue_timeout
Poll loop: while(connected) instead of recursive setTimeout — cleaner lifecycle

extension-src/watchdog.js (NEW)

External supervisor process (Hermes pattern)
Polls /health every 30s, restarts router after 3 consecutive failures
Grace period after restart before re-checking

config/ecosystem.abiba.config.cjs

PM2 hardening: max_restarts=100 (was 10), max_memory_restart=500M, kill_timeout=15s
Separate configs for router and watchdog processes

config/settings.json (NEW)

Fixed default model: deepseek-v4-pro → syslog-harness/syslog-auto
This was the actual root cause of all the worker hangs

Verification

Circuit breaker: CLOSED→HALF_OPEN→OPEN on simulated outage, recovered gracefully ✅
End-to-end: DM processed in 4 seconds ✅
Watchdog: 20+ min uptime, 0 unnecessary restarts ✅
Health: {status:"ok", connected:true, circuit_breaker:"CLOSED"} ✅

## Summary Complete rewrite of the pi Zulip gateway from crash-prone v2 to production-resilient v3. Research-backed patterns from Zulip event system docs + Node.js resilience best practices. ## Root Cause Pi workers used DeepSeek v4-pro as default model — a reasoning model that produces only `reasoning_content` with normal token limits. Workers waited forever for content that never arrived, hung, and were killed by the 5-minute stuck-worker timeout. Every "Processing..." placeholder was a dead worker. ## Changes ### extension-src/zulip-extension.js (v2→v3) - **Circuit breaker** (`ZulipCircuitBreaker`): CLOSED→OPEN→HALF_OPEN states, 50% failure threshold, volumeThreshold=2 for slow-polling, 30s reset timeout - **Retry with jitter** (`withRetry`): Exponential backoff + 50-100% jitter, max 2 attempts for transient errors only - **Queue lifecycle**: `idle_queue_timeout=600` (Zulip 12.0+), auto re-register on BAD_EVENT_QUEUE_ID - **Busy worker timeout**: SIGKILL after 5 minutes + error DM to user - **Crash handlers**: `uncaughtException` + `unhandledRejection` → log + reconnect instead of dying - **Enhanced health endpoint**: Circuit breaker stats, idle_queue_timeout - **Poll loop**: `while(connected)` instead of recursive `setTimeout` — cleaner lifecycle ### extension-src/watchdog.js (NEW) - External supervisor process (Hermes pattern) - Polls `/health` every 30s, restarts router after 3 consecutive failures - Grace period after restart before re-checking ### config/ecosystem.abiba.config.cjs - PM2 hardening: `max_restarts=100` (was 10), `max_memory_restart=500M`, `kill_timeout=15s` - Separate configs for router and watchdog processes ### config/settings.json (NEW) - Fixed default model: `deepseek-v4-pro` → `syslog-harness/syslog-auto` - This was the actual root cause of all the worker hangs ## Verification - Circuit breaker: CLOSED→HALF_OPEN→OPEN on simulated outage, recovered gracefully ✅ - End-to-end: DM processed in 4 seconds ✅ - Watchdog: 20+ min uptime, 0 unnecessary restarts ✅ - Health: `{status:"ok", connected:true, circuit_breaker:"CLOSED"}` ✅

abiba-bot added 1 commit 2026-07-13 21:30:03 +00:00

feat(zulip): v3 resilience rewrite — circuit breaker, retry, watchdog, model fix 33453b8204

Root cause: Workers hung because default model was DeepSeek v4-pro (reasoning
model that produces empty content with normal token limits). pi RPC workers
waited forever for content that never arrived.

Changes:
- Circuit breaker (CLOSED→OPEN→HALF_OPEN) around Zulip API calls
- Retry with exponential backoff + jitter for transient errors
- Queue lifecycle management (idle_queue_timeout, auto re-register on BAD_EVENT_QUEUE_ID)
- External supervisor watchdog (restarts router after 3 health check failures)
- Busy worker timeout (SIGKILL after 5 min + error DM)
- PM2 hardening (max_restarts=100, max_memory_restart=500M)
- Crash handlers (uncaughtException + unhandledRejection → reconnect, not die)
- Fixed default model: deepseek-v4-pro → syslog-harness/syslog-auto
- Enhanced health endpoint with circuit breaker stats

Architecture: Research-backed from Zulip event system docs + Node.js resilience
patterns. Inline circuit breaker (no dependency). Separate watchdog process (Hermes pattern).

Verified: Circuit breaker trips on outage, recovers gracefully. End-to-end DM
processed in 4 seconds. Watchdog monitoring every 30s.

abiba-bot merged commit 9f72da5b8d into main

2026-07-15 16:09:56 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/zulip-platform-plugins#34