Root cause: Workers hung because default model was DeepSeek v4-pro (reasoning
model that produces empty content with normal token limits). pi RPC workers
waited forever for content that never arrived.
Changes:
- Circuit breaker (CLOSED→OPEN→HALF_OPEN) around Zulip API calls
- Retry with exponential backoff + jitter for transient errors
- Queue lifecycle management (idle_queue_timeout, auto re-register on BAD_EVENT_QUEUE_ID)
- External supervisor watchdog (restarts router after 3 health check failures)
- Busy worker timeout (SIGKILL after 5 min + error DM)
- PM2 hardening (max_restarts=100, max_memory_restart=500M)
- Crash handlers (uncaughtException + unhandledRejection → reconnect, not die)
- Fixed default model: deepseek-v4-pro → syslog-harness/syslog-auto
- Enhanced health endpoint with circuit breaker stats
Architecture: Research-backed from Zulip event system docs + Node.js resilience
patterns. Inline circuit breaker (no dependency). Separate watchdog process (Hermes pattern).
Verified: Circuit breaker trips on outage, recovers gracefully. End-to-end DM
processed in 4 seconds. Watchdog monitoring every 30s.
Zulip delivers message content as HTML (<p>/approve</p>).
The gateway slash command parser expects plain text, so HTML
tags prevent command matching. This helper strips HTML tags
and decodes common entities (&, <, etc.).
Per contract: zulip-approval-fix.prose.md
Applied to: Mumuni (CT114), Tanko (CT112), Koby (CT111)
- Added Host column (Abiba: 192.168.68.24, Hermes agents: 192.168.68.123)
- Changed Health Port → Health URL with full http://host:port/health URLs
- Updated Check 1 to reference {{health_url}} instead of {{health_port}}
- Added .gitignore for .agents/ (OpenProse run state)
Adds two improvements from pi Zulip extension lessons:
1. _ensure_stream_subscriptions() — checks at connect time if the bot
is subscribed to the primary stream and general-chat; auto-subscribes
if missing. Without this, bots with 0 subscriptions can't receive
stream events (DMs only).
2. selftest check #9 — verifies stream subscriptions and reports
count + names. Catches the 'zero subscriptions' failure mode that
was a major debugging bottleneck in the pi extension.
Same fix as applied to the pi Zulip extension: last_error_msg and
last_error_at are now cleared on every successful poll cycle, not
just on reconnect. Health monitors no longer show stale errors.
- Full contract verification for pi (TypeScript) and Hermes (Python) Zulip plugins
- Fixed @all-bots user_id (1→20) — now dynamically resolved from Zulip API
- Fixed last_event_id stale display in health endpoint (B2)
- Fixed display_recipient logging (B3)
- Added bot_user_id and all_bots_user_id to health endpoint
- Created OpenProse cross-platform verification contract (docs/contracts/)
- Created Hermes plugin deployment script (scripts/deploy-hermes-zulip.py)
Pi extension: 8/9 checks pass (LLM relay untested — no external DMs)
Hermes adapter: 9/10 checks pass (deployment pending)
Bots with no incoming messages were reconnecting every ~60 seconds
(20 polls × 3s interval). Raised to 500 polls (~25min) so idle bots
don't cycle. Connection recovers immediately on BAD_EVENT_QUEUE_ID
regardless of this counter.
Zulip event queues expire after ~600s of inactivity. The adapter now
detects 400/BAD_EVENT_QUEUE_ID responses and auto-reconnects instead of
silently returning empty results.
Complete rewrite of the pi Zulip extension as a standalone Node.js service
with direct harness API calls. No longer depends on pi's session — survives
pi shutdown and restart.
Changes:
- src/index.ts: Complete rewrite — standalone process, not a pi extension
- Direct harness API calls (POST /v1/chat/completions) with conversation memory
- Per-sender conversation history (up to 50 messages)
- Placeholder->edit streaming for Zulip DMs
- Typing indicators via Zulip API
- Health endpoint on :9200
- Exponential backoff reconnection
- Graceful shutdown on SIGINT/SIGTERM
- abiba-zulip.service: systemd service unit file
- README.md: Updated deployment instructions
- package.json/tsconfig.json: Updated for standalone app
Deploy: npm install -> npx tsc -> systemctl enable abiba-zulip
Closes issue #24 (Hermes parity for pi)
A2A server uses Agent Zero's AgentContext.communicate() directly
to process Zulip messages inside the container.
Full chain verified: Zulip DM -> adapter -> A2A -> Agent Zero -> response -> Zulip
Zulip API edits messages via PATCH /api/v1/messages/{message_id}
(POST returns 405 on this endpoint). Also:
- edit_message() now returns SendResult (not bool) for Gateway compatibility
- Clean up duplicate gateway instances before restart
The Gateway streaming interface calls edit_message() with finalize=True
and expects SendResult. Fixed:
1. Added **kwargs to edit_message() signature — silently ignores finalize
2. Changed return type from bool to SendResult for Gateway compatibility
3. Switched from PATCH to POST /api/v1/messages/{id} — Zulip docs use
POST for message editing (PATCH returns 405 Method Not Allowed)
4. Same fixes applied to delete_message() and send()
Discovered during Tanko Gen 4 testing.
Fixes 4 critical reliability issues discovered during Zulip server outage recovery:
1. CLOSE-WAIT socket leak: _reconnect() now creates a fresh httpx.AsyncClient()
when the queue expires, preventing stuck connection pool. Old client is
explicitly aclose()'d.
2. Silent poll loop death: After sustained 502 errors, the poll loop was
returning [] without logging — transport errors are swallowed by _api_call
returning (None, 0). Added heartbeat logging every 5 minutes and silence
detection warnings after 60s of no events.
3. HTTP client stuck pool: Added proactive client recreation after 20 consecutive
empty polls (connection pool exhaustion detection) and periodic refresh every
500 polls (connection reuse limit).
4. Log spam: 502 error responses from Netbird contain full HTML pages — now
truncated to 80 chars with newlines stripped.
Also:
- Support ZULIP_URL env var (in addition to ZULIP_SITE)
- expose silence_seconds, consecutive_empty_polls, client_pool_resets in health stats
- Reduced health callback interval from 600s to 300s
SendResult only accepts: success, message_id, error, raw_response,
retryable. The adapter was passing platform=, chat_id=, and metadata=
which are not valid fields, causing TypeError on every send attempt.
This was the error Tanko was showing in Zulip:
'SendResult.__init__() got an unexpected keyword argument "platform"'
The /register endpoint doesn't always return user_id on all Zulip
server versions. Added _resolve_bot_user_id() that queries the
/users/me endpoint as a fallback. This ensures @mention detection
(is_direct_mention check) works even when register doesn't provide it.
Also bumped to v1.0.3 — cumulative fixes since v1.0.1:
- v1.0.2: BAD_EVENT_QUEUE_ID detection (silent queue expiry)
- v1.0.3: bot user_id resolution (@mention detection)
Root cause of Tanko not responding to DMs after initial connection:
Zulip event queues expire after ~10min of dont_block polling. When the
queue expired, _api_call() returned None for all 400-level errors, which
_fetch_events() treated as 'no events' and returned []. The poll loop
never knew the queue died, so it never reconnected.
Fix:
1. _api_call() now returns (Optional[Dict], status_code) tuple
2. _fetch_events() explicitly checks for 400 status and raises
RuntimeError('BAD_EVENT_QUEUE_ID') for the poll loop to catch
3. All callers updated to unpack the tuple
Now when a queue expires, the poll loop catches the error and
calls _reconnect() to register a fresh queue.
Root cause of 'str' object has no attribute 'value' error during connect():
the adapter passed a raw string 'zulip' to the base class's platform
parameter, which expects a Platform enum instance. When self.name was
accessed (calling self.platform.value.title()), the string had no .value
attribute.
Fix: import Platform from gateway.config and wrap with Platform('zulip').
Gen 3 improvements from build-zulip-plugin contract run:
1. Malformed message resilience
- Per-event try/except in poll loop — one bad event never kills adapter
- malformed_events counter tracked in health stats
- Fulfills Success Criterion: handles_malformed_messages == true
2. Periodic @all-bots refresh
- _all_bots_refresh_forever() background task re-resolves every hour
- Cancelled gracefully on disconnect()
- all_bots_refreshes counter in health stats
3. Health stats callback mechanism
- set_health_callback(callback, interval=600) — register external consumer
- _report_health_if_callback() triggers on dedup cleanup cycle
- Designed for Hermes Gateway to wire to RA-H OS knowledge graph logging
- Cross-contract integration: get_all_bots_user_id() and get_bot_user_id()
exposed for zulip-mention-reliability contract
abiba-bot
Abiba (pi)
jeromeandAbiba (pi)