prose-contracts/zulip-adapter-lessons.prose.md

kind, name, status, note, description

kind	name	status	note	description
pattern	zulip-adapter-lessons	historical	pi Zulip extension retired 2026-07-04. These lessons remain relevant for Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.	Lessons learned from building and debugging the pi Zulip extension and Hermes Zulip plugin. Documents failure modes, fixes, and patterns that apply across all Zulip adapters.

Known Failure Modes & Fixes

1. Queue Registration Content-Type

• Symptom: Event queue registered but delivers no stream events
• Root cause: multipart/form-data (from isomorphic-form-data / form-data npm pkgs) vs application/x-www-form-urlencoded
• Fix: Always use URLSearchParams / form-urlencoded for /api/v1/register
• Applies to: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)

2. Missing bot_user_id for @mention Detection

• Symptom: Bot receives stream messages but doesn't respond to @mentions
• Root cause: /api/v1/register doesn't return user_id; mentioned_users check fails when bot ID is null
• Fix: Call GET /api/v1/users/me after registration to get user_id
• Applies to: pi extension (fixed), Hermes adapter (fixed via _resolve_bot_user_id() ✅)

3. Zero Stream Subscriptions

• Symptom: Bot can send to streams but never receives stream events
• Root cause: Bot user created with 0 stream subscriptions — only DMs arrive
• Fix: POST /api/v1/users/me/subscriptions with required stream names
• Applies to: pi extension (fixed), Hermes adapter (⚠️ needs check)

4. Stale Errors Never Cleared

• Symptom: Health monitor shows persistent error long after recovery
• Root cause: last_error set on failure but never cleared on successful poll
• Fix: Clear lastError on every successful poll (not just on reconnect)
• Applies to: pi extension (fixed), Hermes adapter (⚠️ needs check)

5. Stuck Detection Too Aggressive

• Symptom: Monitor restarts bot during quiet periods (nights/weekends)
• Root cause: 30-min stuck threshold doesn't account for low traffic
• Fix: Raise threshold to 4 hours — or remove stuck detection entirely
• Applies to: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)

6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)

• Symptom: Plugin not loading because check_fn returns False
• Root cause: Some deploy configs use ZULIP_URL, adapter code expects ZULIP_SITE
• Fix: Support both names in check_fn, validate_config, and env_enablement_fn
• Applies to: Hermes adapter (fixed ✅)

7. Poll Interval Too Aggressive

• Symptom: Queue expires faster than expected, reconnection cycling
• Root cause: 3s poll interval with dont_block=true creates many requests
• Fix: Use long-poll (no dont_block) with 30s+ server timeout
• Applies to: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)

8. A2A Server Address Already In Use

• Symptom: A2A server fails to start with "[Errno 98] address already in use"
• Root cause: Previous process holds port after container restart
• Fix: Kill old process before starting, or use docker restart to clear state
• Applies to: kagentz Agent Zero deployment

9. Zulip API Rate Limiting During Reconnection

• Symptom: Extension gets 429 errors and can't reconnect — responses vanish
• Root cause: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
• Fix: Read retry-after header and wait before retrying; increase backoff between retries
• Applies to: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)

10. Placeholder Edit Fails Silently (Streaming Vanishes)

• Symptom: User sees "Thinking..." placeholder but never gets the actual response
• Root cause: editMessage API call fails (e.g., message too old, rate limited) but no fallback
• Fix: If edit fails, send the response as a new message instead
• Applies to: pi extension (fixed), Hermes adapter (verify edit fallback exists)

11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)

• Symptom: User sends DM, sees placeholder, but never gets response. Worker stays "busy" forever, accumulating pending replies (10+). Health endpoint shows green but messages_processed stalls.
• Root cause: Agent's model config (models.json or settings.json) references a model ID that doesn't exist in LiteLLM's authorized model list. Example: qwen3.6-35B-A3B configured but LiteLLM only exposes ornith-1.0-35b under that key. pi's session workers emit 403 on first prompt, then never recover because the error doesn't trigger agent_end — worker stays busy and all subsequent messages pile up in the steer queue.
• Detection: Compare ~/.pi/agent/models.json model IDs against curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models output. A stuck worker shows workers=[<id>:busy:N] with growing N in extension logs.
• Fix: (1) Update models.json to only include models from the authorized list. (2) Set defaultModel to syslog-auto (safe routing model). (3) Delete stale session JSONL files from ~/.pi/agent/sessions/zulip/. (4) Restart PM2 process.
• Prevention: Use syslog-auto as default model for all agents — it handles model routing and fallback automatically. Direct model IDs (ornith-1.0-35b, etc.) should only be used when explicitly requested. Validate model IDs at agent setup time.
• Applies to: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using syslog-harness provider

Deployment Checklist

When deploying a new Zulip adapter, verify:

• Queue registered with form-urlencoded content type
• Bot user_id fetched from /api/v1/users/me
• Subscribed to all expected streams (check with /api/v1/users/me/subscriptions)
• last_error cleared on successful poll
• Env vars support both ZULIP_URL and ZULIP_SITE
• Stream @mentions detected via mentioned_users array
• @all-bots detected via configurable user_id
• Poll uses long-poll (not dont_block=true polling)
• Stuck/idle detection accounts for quiet periods
• Model IDs in config validated against GET /v1/models with actual API key