prose-contracts/zulip-adapter-lessons.prose.md

---
kind: pattern
name: zulip-adapter-lessons
status: historical
note: >
  pi Zulip extension retired 2026-07-04. These lessons remain relevant for
  Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
description: >
  Lessons learned from building and debugging the pi Zulip extension and
  Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
  apply across all Zulip adapters.
---

## Known Failure Modes & Fixes

### 1. Queue Registration Content-Type
- **Symptom**: Event queue registered but delivers no stream events
- **Root cause**: `multipart/form-data` (from `isomorphic-form-data` / `form-data` npm pkgs) vs `application/x-www-form-urlencoded`
- **Fix**: Always use URLSearchParams / form-urlencoded for `/api/v1/register`
- **Applies to**: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)

### 2. Missing bot_user_id for @mention Detection
- **Symptom**: Bot receives stream messages but doesn't respond to @mentions
- **Root cause**: `/api/v1/register` doesn't return `user_id`; `mentioned_users` check fails when bot ID is null
- **Fix**: Call `GET /api/v1/users/me` after registration to get `user_id`
- **Applies to**: pi extension (fixed), Hermes adapter (fixed via `_resolve_bot_user_id()` ✅)

### 3. Zero Stream Subscriptions
- **Symptom**: Bot can send to streams but never receives stream events
- **Root cause**: Bot user created with 0 stream subscriptions — only DMs arrive
- **Fix**: `POST /api/v1/users/me/subscriptions` with required stream names
- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)

### 4. Stale Errors Never Cleared
- **Symptom**: Health monitor shows persistent error long after recovery
- **Root cause**: `last_error` set on failure but never cleared on successful poll
- **Fix**: Clear `lastError` on every successful poll (not just on reconnect)
- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)

### 5. Stuck Detection Too Aggressive
- **Symptom**: Monitor restarts bot during quiet periods (nights/weekends)
- **Root cause**: 30-min stuck threshold doesn't account for low traffic
- **Fix**: Raise threshold to 4 hours — or remove stuck detection entirely
- **Applies to**: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)

### 6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)
- **Symptom**: Plugin not loading because check_fn returns False
- **Root cause**: Some deploy configs use `ZULIP_URL`, adapter code expects `ZULIP_SITE`
- **Fix**: Support both names in check_fn, validate_config, and env_enablement_fn
- **Applies to**: Hermes adapter (fixed ✅)

### 7. Poll Interval Too Aggressive
- **Symptom**: Queue expires faster than expected, reconnection cycling
- **Root cause**: 3s poll interval with `dont_block=true` creates many requests
- **Fix**: Use long-poll (no `dont_block`) with 30s+ server timeout
- **Applies to**: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)

### 8. A2A Server Address Already In Use
- **Symptom**: A2A server fails to start with "[Errno 98] address already in use"
- **Root cause**: Previous process holds port after container restart
- **Fix**: Kill old process before starting, or use docker restart to clear state
- **Applies to**: kagentz Agent Zero deployment

### 9. Zulip API Rate Limiting During Reconnection
- **Symptom**: Extension gets 429 errors and can't reconnect — responses vanish
- **Root cause**: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
- **Fix**: Read `retry-after` header and wait before retrying; increase backoff between retries
- **Applies to**: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)

### 10. Placeholder Edit Fails Silently (Streaming Vanishes)
- **Symptom**: User sees "Thinking..." placeholder but never gets the actual response
- **Root cause**: `editMessage` API call fails (e.g., message too old, rate limited) but no fallback
- **Fix**: If edit fails, send the response as a new message instead
- **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)

### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
- **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
  `messages_processed` stalls.
- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
  configured but LiteLLM only exposes `strix-moe` (alias for qwen3.6-35B-udq4) under that key. pi's session
  workers emit 403 on first prompt, then never recover because the error doesn't trigger
  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
  `workers=[<id>:busy:N]` with growing N in extension logs.
- **Fix**: (1) Update `models.json` to only include models from the authorized list.
  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
  routing and fallback automatically. Direct model IDs (`strix-moe`, etc.) should
  only be used when explicitly requested. Validate model IDs at agent setup time.
- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider

## Deployment Checklist

When deploying a new Zulip adapter, verify:

- [ ] Queue registered with form-urlencoded content type
- [ ] Bot user_id fetched from `/api/v1/users/me`
- [ ] Subscribed to all expected streams (check with `/api/v1/users/me/subscriptions`)
- [ ] `last_error` cleared on successful poll
- [ ] Env vars support both `ZULIP_URL` and `ZULIP_SITE`
- [ ] Stream @mentions detected via `mentioned_users` array
- [ ] `@all-bots` detected via configurable user_id
- [ ] Poll uses long-poll (not `dont_block=true` polling)
- [ ] Stuck/idle detection accounts for quiet periods
- [ ] Model IDs in config validated against `GET /v1/models` with actual API key