zulip-platform-plugins/CONTRACT_VERIFICATION_2026-06-29.md

Zulip Extension Contract Verification Report

Date: 2026-06-29 | Tester: Abiba (pi agent) | Scope: pi + Hermes Zulip plugins

---

Pi Zulip Extension (~/.pi/agent/extensions/zulip/)

Test Matrix

#	Test	Result	Detail
1	Queue registration	✅ PASS	Registers with Zulip, gets queue_id
2	Event polling	✅ PASS	Polls every 3s, receives events
3	Own-message filtering	✅ PASS	Correctly skips own sent messages
4	Health endpoint (:9200)	✅ PASS	Responds with connected state, stats
5	Event reception (DM)	✅ PASS	idle_seconds resets on event arrival
6	@mention detection	✅ PASS	Bot user_id 21 resolved correctly
7	Streaming placeholder	⚠️ UNTESTED*	Code path verified, no live DM to trigger
8	LLM response relay	⚠️ UNTESTED*	Requires external user DM
9	Zulip commands (/status, etc)	⚠️ UNTESTED*	Requires external user DM

*Untested: No external user has DMed the bot since last restart.

Bugs Found

#	Severity	Issue	Fix
B1	HIGH	all_bots_user_id: 1 in config.yaml — actual Zulip value is 20	✅ FIXED — updated to 20
B2	MEDIUM	last_event_id always shows -1 in health endpoint — object property captured by value, not reference	Fix: use getter or update property in poll()
B3	LOW	display_recipient logged as [object Object] — should use JSON.stringify	Fix: change log line
B4	MEDIUM	PM2 restarts=4 in <30min — indicates crash/restart loop	Investigate cause
B5	LOW	No /zulip-test self-test command exists	Add loopback test

---

Hermes Zulip Plugin (plugins/platforms/zulip/)

Test Matrix

#	Test	Result	Detail
1	Import & structure	✅ PASS	ZulipAdapter(BasePlatformAdapter) OK
2	Requirements check	✅ PASS	Env detection works
3	Adapter instantiation	✅ PASS	Config parsed correctly
4	Self-test (pre-connect)	✅ PASS	Reports degraded (expected, pre-connect)
5	Connect to Zulip	✅ PASS	Queue registered, bot_id=21 resolved
6	Self-test (post-connect)	✅ PASS	7/8 checks pass (connected flag mock issue)
7	Health stats	✅ PASS	Poll stats, dedup, silence tracking all work
8	@all-bots resolution	✅ PASS	Dynamically resolves to 20 (correct!)
9	Disconnect	✅ PASS	Clean shutdown
10	Deployed as service	❌ FAIL	Not deployed, no systemd service

Bugs Found

#	Severity	Issue	Fix
H1	CRITICAL	Hermes plugin NOT deployed — no systemd service, no running instance	Create systemd service + deploy
H2	MEDIUM	httpx not available globally — needs pip install --break-system-packages	Add to requirements or venv
H3	LOW	selftest connected flag relies on _mark_connected() mock issue	Minor, doesn't affect real Hermes Gateway

---

Cross-Platform Contract Gaps

#	Gap	Severity	Recommendation
G1	No unified contract test — pi and Hermes tested independently	HIGH	Create cross-platform .prose.md contract
G2	@all-bots ID inconsistency — pi hardcoded 1, Hermes dynamic-resolved to 20	HIGH (fixed B1)	Make pi extension also dynamic-resolve
G3	No agent identity verification — no way to confirm "this bot is Abiba"	MEDIUM	Add /whoami or identity stamp
G4	No cross-bot communication test — can't verify @all-bots works end-to-end	MEDIUM	Schedule coordinated test when all bots online
G5	Health check format differs — pi returns flat JSON, Hermes returns nested stats	LOW	Standardize health endpoint schema

---

Immediate Actions

1. ✅ DONE — Fixed all_bots_user_id: 1 → 20 in pi config
2. TODO — Restart pi Zulip service to pick up config change: pm2 restart abiba-zulip
3. TODO — Deploy Hermes Zulip plugin as systemd service
4. TODO — Fix last_event_id health display bug (B2)
5. TODO — Create OpenProse cross-platform contract for automated validation

---

Contract Upgrade Recommendations

1. Dynamic @all-bots Resolution (both platforms)

Pi extension should resolve all_bots_user_id from Zulip API (like Hermes does) instead of hardcoding.

2. Standardized Health Schema

{
  "status": "ok",
  "platform": "pi",
  "agent": "abiba",
  "zulip": {
    "connected": true,
    "queue_id": "...",
    "last_event_id": 2,
    "bot_user_id": 21,
    "all_bots_user_id": 20,
    "messages_processed": 5,
    "silence_seconds": 12
  },
  "uptime_seconds": 3600,
  "checks": {
    "queue": "ok",
    "poll_loop": "ok",
    "self_test": "ok"
  }
}

3. Built-in Self-Test Command

Add /zulip-test that:

• Sends a test DM to owner
• Verifies event reception
• Reports round-trip time
• Validates all code paths

4. Cross-Platform Contract (OpenProse)

Create a .prose.md contract that:

• Verifies both pi and Hermes plugins connect to Zulip
• Validates @mention and @all-bots routing
• Runs on schedule (daily) or on-demand
• Reports to knowledge graph

5. Queue Health Monitoring

The last_event_id=-1 issue is actually a Zulip server behavior (fresh instance). Add monitoring that alerts if last_event_id doesn't advance after external messages are sent.