Root cause: Workers hung because default model was DeepSeek v4-pro (reasoning
model that produces empty content with normal token limits). pi RPC workers
waited forever for content that never arrived.
Changes:
- Circuit breaker (CLOSED→OPEN→HALF_OPEN) around Zulip API calls
- Retry with exponential backoff + jitter for transient errors
- Queue lifecycle management (idle_queue_timeout, auto re-register on BAD_EVENT_QUEUE_ID)
- External supervisor watchdog (restarts router after 3 health check failures)
- Busy worker timeout (SIGKILL after 5 min + error DM)
- PM2 hardening (max_restarts=100, max_memory_restart=500M)
- Crash handlers (uncaughtException + unhandledRejection → reconnect, not die)
- Fixed default model: deepseek-v4-pro → syslog-harness/syslog-auto
- Enhanced health endpoint with circuit breaker stats
Architecture: Research-backed from Zulip event system docs + Node.js resilience
patterns. Inline circuit breaker (no dependency). Separate watchdog process (Hermes pattern).
Verified: Circuit breaker trips on outage, recovers gracefully. End-to-end DM
processed in 4 seconds. Watchdog monitoring every 30s.
Abiba (pi)