root
253e19f680
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
89 lines
4.7 KiB
Markdown
89 lines
4.7 KiB
Markdown
---
|
|
kind: pattern
|
|
name: zulip-adapter-lessons
|
|
status: historical
|
|
note: >
|
|
pi Zulip extension retired 2026-07-04. These lessons remain relevant for
|
|
Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge.
|
|
description: >
|
|
Lessons learned from building and debugging the pi Zulip extension and
|
|
Hermes Zulip plugin. Documents failure modes, fixes, and patterns that
|
|
apply across all Zulip adapters.
|
|
---
|
|
|
|
## Known Failure Modes & Fixes
|
|
|
|
### 1. Queue Registration Content-Type
|
|
- **Symptom**: Event queue registered but delivers no stream events
|
|
- **Root cause**: `multipart/form-data` (from `isomorphic-form-data` / `form-data` npm pkgs) vs `application/x-www-form-urlencoded`
|
|
- **Fix**: Always use URLSearchParams / form-urlencoded for `/api/v1/register`
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default ✅)
|
|
|
|
### 2. Missing bot_user_id for @mention Detection
|
|
- **Symptom**: Bot receives stream messages but doesn't respond to @mentions
|
|
- **Root cause**: `/api/v1/register` doesn't return `user_id`; `mentioned_users` check fails when bot ID is null
|
|
- **Fix**: Call `GET /api/v1/users/me` after registration to get `user_id`
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (fixed via `_resolve_bot_user_id()` ✅)
|
|
|
|
### 3. Zero Stream Subscriptions
|
|
- **Symptom**: Bot can send to streams but never receives stream events
|
|
- **Root cause**: Bot user created with 0 stream subscriptions — only DMs arrive
|
|
- **Fix**: `POST /api/v1/users/me/subscriptions` with required stream names
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
|
|
|
|
### 4. Stale Errors Never Cleared
|
|
- **Symptom**: Health monitor shows persistent error long after recovery
|
|
- **Root cause**: `last_error` set on failure but never cleared on successful poll
|
|
- **Fix**: Clear `lastError` on every successful poll (not just on reconnect)
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (⚠️ needs check)
|
|
|
|
### 5. Stuck Detection Too Aggressive
|
|
- **Symptom**: Monitor restarts bot during quiet periods (nights/weekends)
|
|
- **Root cause**: 30-min stuck threshold doesn't account for low traffic
|
|
- **Fix**: Raise threshold to 4 hours — or remove stuck detection entirely
|
|
- **Applies to**: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)
|
|
|
|
### 6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)
|
|
- **Symptom**: Plugin not loading because check_fn returns False
|
|
- **Root cause**: Some deploy configs use `ZULIP_URL`, adapter code expects `ZULIP_SITE`
|
|
- **Fix**: Support both names in check_fn, validate_config, and env_enablement_fn
|
|
- **Applies to**: Hermes adapter (fixed ✅)
|
|
|
|
### 7. Poll Interval Too Aggressive
|
|
- **Symptom**: Queue expires faster than expected, reconnection cycling
|
|
- **Root cause**: 3s poll interval with `dont_block=true` creates many requests
|
|
- **Fix**: Use long-poll (no `dont_block`) with 30s+ server timeout
|
|
- **Applies to**: pi extension (uses long-poll ✅), Hermes adapter (uses long-poll ✅)
|
|
|
|
### 8. A2A Server Address Already In Use
|
|
- **Symptom**: A2A server fails to start with "[Errno 98] address already in use"
|
|
- **Root cause**: Previous process holds port after container restart
|
|
- **Fix**: Kill old process before starting, or use docker restart to clear state
|
|
- **Applies to**: kagentz Agent Zero deployment
|
|
|
|
### 9. Zulip API Rate Limiting During Reconnection
|
|
- **Symptom**: Extension gets 429 errors and can't reconnect — responses vanish
|
|
- **Root cause**: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
|
|
- **Fix**: Read `retry-after` header and wait before retrying; increase backoff between retries
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)
|
|
|
|
### 10. Placeholder Edit Fails Silently (Streaming Vanishes)
|
|
- **Symptom**: User sees "Thinking..." placeholder but never gets the actual response
|
|
- **Root cause**: `editMessage` API call fails (e.g., message too old, rate limited) but no fallback
|
|
- **Fix**: If edit fails, send the response as a new message instead
|
|
- **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)
|
|
|
|
## Deployment Checklist
|
|
|
|
When deploying a new Zulip adapter, verify:
|
|
|
|
- [ ] Queue registered with form-urlencoded content type
|
|
- [ ] Bot user_id fetched from `/api/v1/users/me`
|
|
- [ ] Subscribed to all expected streams (check with `/api/v1/users/me/subscriptions`)
|
|
- [ ] `last_error` cleared on successful poll
|
|
- [ ] Env vars support both `ZULIP_URL` and `ZULIP_SITE`
|
|
- [ ] Stream @mentions detected via `mentioned_users` array
|
|
- [ ] `@all-bots` detected via configurable user_id
|
|
- [ ] Poll uses long-poll (not `dont_block=true` polling)
|
|
- [ ] Stuck/idle detection accounts for quiet periods
|