Files
prose-contracts/zulip-adapter-lessons.prose.md
root 253e19f680 prose: review, fix, and consolidate infrastructure contracts
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this)
- FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet)
- FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip)
- FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only)
- FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup)
- ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz)
- ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md
- UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts
- ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts)

All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00

4.7 KiB

kind, name, status, note, description
kind name status note description
pattern zulip-adapter-lessons historical pi Zulip extension retired 2026-07-04. These lessons remain relevant for Hermes and Agent Zero Zulip adapters. Kept as institutional knowledge. Lessons learned from building and debugging the pi Zulip extension and Hermes Zulip plugin. Documents failure modes, fixes, and patterns that apply across all Zulip adapters.

Known Failure Modes & Fixes

1. Queue Registration Content-Type

  • Symptom: Event queue registered but delivers no stream events
  • Root cause: multipart/form-data (from isomorphic-form-data / form-data npm pkgs) vs application/x-www-form-urlencoded
  • Fix: Always use URLSearchParams / form-urlencoded for /api/v1/register
  • Applies to: pi extension (fixed), Hermes adapter (httpx sends form-urlencoded by default )

2. Missing bot_user_id for @mention Detection

  • Symptom: Bot receives stream messages but doesn't respond to @mentions
  • Root cause: /api/v1/register doesn't return user_id; mentioned_users check fails when bot ID is null
  • Fix: Call GET /api/v1/users/me after registration to get user_id
  • Applies to: pi extension (fixed), Hermes adapter (fixed via _resolve_bot_user_id() )

3. Zero Stream Subscriptions

  • Symptom: Bot can send to streams but never receives stream events
  • Root cause: Bot user created with 0 stream subscriptions — only DMs arrive
  • Fix: POST /api/v1/users/me/subscriptions with required stream names
  • Applies to: pi extension (fixed), Hermes adapter (⚠️ needs check)

4. Stale Errors Never Cleared

  • Symptom: Health monitor shows persistent error long after recovery
  • Root cause: last_error set on failure but never cleared on successful poll
  • Fix: Clear lastError on every successful poll (not just on reconnect)
  • Applies to: pi extension (fixed), Hermes adapter (⚠️ needs check)

5. Stuck Detection Too Aggressive

  • Symptom: Monitor restarts bot during quiet periods (nights/weekends)
  • Root cause: 30-min stuck threshold doesn't account for low traffic
  • Fix: Raise threshold to 4 hours — or remove stuck detection entirely
  • Applies to: pi extension (fixed to 4h), Hermes gateway (uses own idle detection)

6. Env Var Name Mismatch (ZULIP_URL vs ZULIP_SITE)

  • Symptom: Plugin not loading because check_fn returns False
  • Root cause: Some deploy configs use ZULIP_URL, adapter code expects ZULIP_SITE
  • Fix: Support both names in check_fn, validate_config, and env_enablement_fn
  • Applies to: Hermes adapter (fixed )

7. Poll Interval Too Aggressive

  • Symptom: Queue expires faster than expected, reconnection cycling
  • Root cause: 3s poll interval with dont_block=true creates many requests
  • Fix: Use long-poll (no dont_block) with 30s+ server timeout
  • Applies to: pi extension (uses long-poll ), Hermes adapter (uses long-poll )

8. A2A Server Address Already In Use

  • Symptom: A2A server fails to start with "[Errno 98] address already in use"
  • Root cause: Previous process holds port after container restart
  • Fix: Kill old process before starting, or use docker restart to clear state
  • Applies to: kagentz Agent Zero deployment

9. Zulip API Rate Limiting During Reconnection

  • Symptom: Extension gets 429 errors and can't reconnect — responses vanish
  • Root cause: Rapid PM2 restarts cause multiple queue registrations, hitting Zulip's per-IP rate limit
  • Fix: Read retry-after header and wait before retrying; increase backoff between retries
  • Applies to: pi extension (fixed), Hermes adapter (has retry logic but should check retry-after)

10. Placeholder Edit Fails Silently (Streaming Vanishes)

  • Symptom: User sees "Thinking..." placeholder but never gets the actual response
  • Root cause: editMessage API call fails (e.g., message too old, rate limited) but no fallback
  • Fix: If edit fails, send the response as a new message instead
  • Applies to: pi extension (fixed), Hermes adapter (verify edit fallback exists)

Deployment Checklist

When deploying a new Zulip adapter, verify:

  • Queue registered with form-urlencoded content type
  • Bot user_id fetched from /api/v1/users/me
  • Subscribed to all expected streams (check with /api/v1/users/me/subscriptions)
  • last_error cleared on successful poll
  • Env vars support both ZULIP_URL and ZULIP_SITE
  • Stream @mentions detected via mentioned_users array
  • @all-bots detected via configurable user_id
  • Poll uses long-poll (not dont_block=true polling)
  • Stuck/idle detection accounts for quiet periods