feat(zulip): v3 resilience rewrite — circuit breaker, retry, watchdog, model fix

Root cause: Workers hung because default model was DeepSeek v4-pro (reasoning
model that produces empty content with normal token limits). pi RPC workers
waited forever for content that never arrived.

Changes:
- Circuit breaker (CLOSED→OPEN→HALF_OPEN) around Zulip API calls
- Retry with exponential backoff + jitter for transient errors
- Queue lifecycle management (idle_queue_timeout, auto re-register on BAD_EVENT_QUEUE_ID)
- External supervisor watchdog (restarts router after 3 health check failures)
- Busy worker timeout (SIGKILL after 5 min + error DM)
- PM2 hardening (max_restarts=100, max_memory_restart=500M)
- Crash handlers (uncaughtException + unhandledRejection → reconnect, not die)
- Fixed default model: deepseek-v4-pro → syslog-harness/syslog-auto
- Enhanced health endpoint with circuit breaker stats

Architecture: Research-backed from Zulip event system docs + Node.js resilience
patterns. Inline circuit breaker (no dependency). Separate watchdog process (Hermes pattern).

Verified: Circuit breaker trips on outage, recovers gracefully. End-to-end DM
processed in 4 seconds. Watchdog monitoring every 30s.
This commit is contained in:
Abiba (pi)
2026-07-13 21:29:25 +00:00
parent 441255a07c
commit 33453b8204
4 changed files with 1654 additions and 1341 deletions
+11
View File
@@ -0,0 +1,11 @@
{
"lastChangelogVersion": "0.80.6",
"defaultProvider": "syslog-harness",
"defaultModel": "syslog-auto",
"defaultThinkingLevel": "high",
"extensions": [
"+extensions/mcp/index.ts",
"+extensions/zulip/index.js"
],
"theme": "dark"
}