root
776df418ca
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3) - .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache - .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV - .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm - LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s - Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma - Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
109 lines
6.3 KiB
Markdown
109 lines
6.3 KiB
Markdown
---
|
|
kind: pattern
|
|
name: zulip-adapter-lessons
|
|
status: historical
|
|
note: >
|
|
pi Zulip extension retired 2026-07-04. These lessons remain relevant for
|
|
Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
|
|
description: >
|
|
Lessons learned from building and debugging the pi Zulip extension and
|
|
Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
|
|
apply across all Zulip adapters.
|
|
---
|
|
|
|
## Known Failure Modes & Fixes
|
|
|
|
### 1. Queue Registration Content-Type
|
|
- **Symptom**: Event queue registered but delivers no stream events
|
|
- **Root cause**: `multipart/form-data` (from `isomorphic-form-data` / `form-data` npm pkgs) vs `application/x-www-form-urlencoded`
|
|
- **Fix**: Always use URLSearchParams / form-urlencoded for `/api/v1/register`
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)
|
|
|
|
### 2. Missing bot_user_id for @mention Detection
|
|
- **Symptom**: Bot receives stream messages but doesn't respond to @mentions
|
|
- **Root cause**: `/api/v1/register` doesn't return `user_id`; `mentioned_users` check fails when bot ID is null
|
|
- **Fix**: Call `GET /api/v1/users/me` after registration to get `user_id`
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (fixed via `_resolve_bot_user_id()` ✅)
|
|
|
|
### 3. Zero Stream Subscriptions
|
|
- **Symptom**: Bot can send to streams but never receives stream events
|
|
- **Root cause**: Bot user created with 0 stream subscriptions — only DMs arrive
|
|
- **Fix**: `POST /api/v1/users/me/subscriptions` with required stream names
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
|
|
|
|
### 4. Stale Errors Never Cleared
|
|
- **Symptom**: Health monitor shows persistent error long after recovery
|
|
- **Root cause**: `last_error` set on failure but never cleared on successful poll
|
|
- **Fix**: Clear `lastError` on every successful poll (not just on reconnect)
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
|
|
|
|
### 5. Stuck Detection Too Aggressive
|
|
- **Symptom**: Monitor restarts bot during quiet periods (nights/weekends)
|
|
- **Root cause**: 30-min stuck threshold doesn't account for low traffic
|
|
- **Fix**: Raise threshold to 4 hours — or remove stuck detection entirely
|
|
- **Applies to**: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)
|
|
|
|
### 6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)
|
|
- **Symptom**: Plugin not loading because check_fn returns False
|
|
- **Root cause**: Some deploy configs use `ZULIP_URL`, adapter code expects `ZULIP_SITE`
|
|
- **Fix**: Support both names in check_fn, validate_config, and env_enablement_fn
|
|
- **Applies to**: Hermes adapter (fixed ✅)
|
|
|
|
### 7. Poll Interval Too Aggressive
|
|
- **Symptom**: Queue expires faster than expected, reconnection cycling
|
|
- **Root cause**: 3s poll interval with `dont_block=true` creates many requests
|
|
- **Fix**: Use long-poll (no `dont_block`) with 30s+ server timeout
|
|
- **Applies to**: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)
|
|
|
|
### 8. A2A Server Address Already In Use
|
|
- **Symptom**: A2A server fails to start with "[Errno 98] address already in use"
|
|
- **Root cause**: Previous process holds port after container restart
|
|
- **Fix**: Kill old process before starting, or use docker restart to clear state
|
|
- **Applies to**: kagentz Agent Zero deployment
|
|
|
|
### 9. Zulip API Rate Limiting During Reconnection
|
|
- **Symptom**: Extension gets 429 errors and can't reconnect — responses vanish
|
|
- **Root cause**: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
|
|
- **Fix**: Read `retry-after` header and wait before retrying; increase backoff between retries
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)
|
|
|
|
### 10. Placeholder Edit Fails Silently (Streaming Vanishes)
|
|
- **Symptom**: User sees "Thinking..." placeholder but never gets the actual response
|
|
- **Root cause**: `editMessage` API call fails (e.g., message too old, rate limited) but no fallback
|
|
- **Fix**: If edit fails, send the response as a new message instead
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)
|
|
|
|
### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
|
|
- **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
|
|
"busy" forever, accumulating pending replies (10+). Health endpoint shows green but
|
|
`messages_processed` stalls.
|
|
- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
|
|
model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
|
|
configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session
|
|
workers emit 403 on first prompt, then never recover because the error doesn't trigger
|
|
`agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
|
|
- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
|
|
`workers=[<id>:busy:N]` with growing N in extension logs.
|
|
- **Fix**: (1) Update `models.json` to only include models from the authorized list.
|
|
(2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
|
|
JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
|
|
- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
|
|
routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should
|
|
only be used when explicitly requested. Validate model IDs at agent setup time.
|
|
- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
|
|
|
|
## Deployment Checklist
|
|
|
|
When deploying a new Zulip adapter, verify:
|
|
|
|
- [ ] Queue registered with form-urlencoded content type
|
|
- [ ] Bot user_id fetched from `/api/v1/users/me`
|
|
- [ ] Subscribed to all expected streams (check with `/api/v1/users/me/subscriptions`)
|
|
- [ ] `last_error` cleared on successful poll
|
|
- [ ] Env vars support both `ZULIP_URL` and `ZULIP_SITE`
|
|
- [ ] Stream @mentions detected via `mentioned_users` array
|
|
- [ ] `@all-bots` detected via configurable user_id
|
|
- [ ] Poll uses long-poll (not `dont_block=true` polling)
|
|
- [ ] Stuck/idle detection accounts for quiet periods
|
|
- [ ] Model IDs in config validated against `GET /v1/models` with actual API key
|