prose-contracts/abiba-zulip-restore.prose.md

kind, name, description, agent, version, status, runtime_contract

kind	name	description	agent	version	status	runtime_contract
function	abiba-zulip-restore	Restores Zulip connectivity for Abiba (pi agent). Verifies the v2 router-worker extension code, starts a PM2 process as the Zulip gateway with correct env vars, validates health endpoint, and confirms DM delivery. Run this whenever Abiba stops responding on Zulip or after system restart.	abiba	1.0.0	active	2

Abiba Zulip Restore — Resume pi Zulip Communication

Single-shot function that restores full Zulip connectivity for the Abiba pi agent. Covers extension code validation, PM2 process management, health endpoint verification, and DM loopback testing.

Live-State Fields

Field	Value	Trust
Agent name	abiba	✅
Bot email	abiba-bot@chat.sysloggh.net	✅
Zulip server	https://chat.sysloggh.net	✅
Extension path	/root/.pi/agent/extensions/zulip/index.js	✅
Config path	/root/.pi/agent/extensions/zulip/config.yaml	✅
Health port	9200	✅
@all-bots user ID	20	✅ (config, verified by API at runtime)
PM2 process name	abiba-zulip	✅
Provider	syslog-harness (http://192.168.68.116/v1)	✅
Default model	deepseek-v4-pro	✅ (settings.json)

Architecture

The pi Zulip extension uses a router-worker architecture (v2):

• Router (PM2 abiba-zulip, ZULIP_ROLE=router): Polls Zulip for events, maintains a per-sender pool of pi RPC worker processes. Each sender gets their own pi --mode rpc --session-dir process with persistent sessions. Handles streaming edits back to Zulip.
• Worker (child pi --mode rpc): Runs the agent with per-sender persistent sessions. No Zulip logic in the worker — the router handles all Zulip I/O.

The extension loads in ALL pi sessions (via settings.json extensions array) but is a NO-OP unless ZULIP_ROLE=router is set. Only the PM2 router process carries the env var.

Maintains

• extension_valid: bool — Extension code imports without errors
• pm2_running: bool — PM2 process abiba-zulip is online
• health_responding: bool — GET :9200/health returns "ok"
• zulip_connected: bool — Queue registered, bot identity resolved
• loopback_delivered: bool — Test DM sent and received
• model_valid: bool — Configured models match LiteLLM authorized models

Postconditions

• PM2 process abiba-zulip online and stable (uptime > 30s)
• Health endpoint returns { status: "ok", connected: true }
• Worker pool creates sessions on demand
• Echo prevention active (BOT_EMAILS includes all known bots)
• PM2 saved for auto-restart on boot

Requires

• Node.js with yaml module available
• PM2 installed globally
• Zulip API key in config.yaml
• pi CLI available on PATH
• Zulip server accessible at https://chat.sysloggh.net
• LiteLLM provider accessible at http://192.168.68.116/v1

Execution

Step 1: Validate Extension Code

node -e "import('file:///root/.pi/agent/extensions/zulip/index.js').then(() => console.log('OK')).catch(e => {console.error('FAIL:', e.message); process.exit(1)})"

Expected: OK. If FAIL → check for missing dependencies, syntax errors.

Step 2: Validate Model IDs

Compare configured models against LiteLLM authorized models:

API_KEY=$(grep -oP 'apiKey:\s*\K.*' /root/.pi/agent/models.json | head -1)
curl -s -H "Authorization: Bearer $API_KEY" http://192.168.68.116/v1/models | \
  python3 -c "import json,sys; d=json.load(sys.stdin); [print(m['id']) for m in d.get('data',[])]" 2>/dev/null

Check: Every model ID in models.json must appear in the authorized list. If not → fix models.json to only include authorized models (prefer syslog-auto).

Step 3: Verify Config Integrity

python3 -c "
import yaml, sys
with open('/root/.pi/agent/extensions/zulip/config.yaml') as f:
    cfg = yaml.safe_load(f)
required = ['zulip.site', 'zulip.email', 'zulip.api_key', 'agent.name']
for k in required:
    keys = k.split('.')
    v = cfg
    for kk in keys:
        v = v.get(kk)
        if v is None:
            print(f'MISSING: {k}')
            sys.exit(1)
print('Config valid')
print(f'  site={cfg[\"zulip\"][\"site\"]}')
print(f'  email={cfg[\"zulip\"][\"email\"]}')
print(f'  agent={cfg[\"agent\"][\"name\"]}')
print(f'  health_port={cfg.get(\"health_port\", 9200)}')
"

Expected: Config valid with all fields non-empty.

Step 4: Remove Stale Systemd Service

The old abiba-zulip.service points to /opt/abiba-zulip/dist/index.js (compiled TypeScript, not the v2 extension). It's disabled and stale. Remove it:

systemctl stop abiba-zulip 2>/dev/null || true
systemctl disable abiba-zulip 2>/dev/null || true
rm -f /etc/systemd/system/abiba-zulip.service
systemctl daemon-reload

Step 5: Start PM2 Process

Critical: The extension MUST run via pi --mode rpc, NOT node index.js directly. The extension exports a function that requires pi's session lifecycle. Running node index.js loads the module but never calls the export, so nothing happens. pi --mode rpc loads all extensions (including Zulip) and fires session_start.

# Stop existing if any
pm2 delete abiba-zulip 2>/dev/null || true

# Start pi in RPC mode with ZULIP_ROLE=router env
ZULIP_ROLE=router pm2 start "$(which pi)" \
  --name abiba-zulip \
  --interpreter none \
  -- --mode rpc --no-session

Wait 10 seconds for pi to load all extensions, fire session_start, and the Zulip router to register its event queue.

Step 6: Validate PM2 Process

pm2 show abiba-zulip --no-color

Check: status=online, restarts=0, uptime > 5s.

Step 7: Check Logs for Connection

sleep 3
tail -20 /root/.pm2/logs/abiba-zulip-out.log

Look for:

• [zulip-ext] Connecting to https://chat.sysloggh.net as abiba-bot@chat.sysloggh.net…
• [zulip-ext] Bot user_id=N, all-bots user_id=N
• [zulip-ext] Connected, queue=N
• [zulip-ext] Echo prevention: N bot emails
• [zulip-ext] Health endpoint on :9200

If error → check API key, network to chat.sysloggh.net.

Step 8: Health Endpoint

curl -s http://localhost:9200/health | python3 -m json.tool

Check:

• status: "ok" (not "down")
• zulip.connected: true
• zulip.queue_id is non-null string
• zulip.bot_user_id is positive integer

Step 9: DM Loopback Test

curl -s http://localhost:9200/health | python3 -c "
import json,sys
d = json.load(sys.stdin)
if d.get('zulip',{}).get('connected'):
    print(f'✅ Zulip connected. Queue: {d[\"zulip\"][\"queue_id\"]}')
    print(f'   Bot user_id: {d[\"zulip\"][\"bot_user_id\"]}')
    print(f'   Messages processed: {d[\"zulip\"][\"messages_processed\"]}')
else:
    print('❌ Zulip NOT connected')
    sys.exit(1)
"

Step 10: Save PM2 for Auto-Start

pm2 save
pm2 startup systemd -u root --hp /root 2>/dev/null || true

Step 11: Report

Compile results:

Check	Pass?
Extension code imports	extension_valid
Model IDs authorized	model_valid
Config integrity	config_valid
PM2 process online	pm2_running
Health endpoint	health_responding
Zulip connected	zulip_connected

All pass → ✅ Abiba Zulip restored. Relay success to user.

Partial failure → see recovery matrix below.

Recovery Matrix

Failure	Recovery
Extension import fails	Check node_modules/zulip-js exists; run npm install in extension dir
Model ID mismatch	Fix models.json to use syslog-auto as default model; remove invalid IDs
Config missing	Restore from backup or recreate from scratch
PM2 won't start	Check node version (>=18); check port 9200 not in use
Health "down"	Check logs for connection errors; verify Zulip API key; check network
"address already in use"	Kill old process: fuser -k 9200/tcp
Queue registration fails	Check Zulip API key validity; verify bot is active in Zulip admin
Rate limit (429)	Extension has built-in retry-after handling — wait, don't restart

Known Failure Modes

Symptom	Root Cause	Recovery
Extension loads but no events	ZULIP_ROLE not set	Ensure PM2 env has ZULIP_ROLE=router
Worker stays "busy" forever	Model ID not authorized by LiteLLM	Fix models.json (lesson #11)
Placeholder sent but no response	editMessage API fails silently	Extension has fallback (sends new msg); check Zulip API
Queue expires rapidly	Poll interval too aggressive	v2 uses 3s poll with long-poll — should be fine
Bot doesn't respond to @mentions	Not subscribed to stream	Bot auto-subscribes via API
Stale error in health	last_error not cleared	v2 clears on successful poll (lesson #4)

Appendix: Root Cause & Fix Summary (2026-07-13)

Problem: Zulip extension was offline — no PM2 process running.

Root cause: The PM2 command ran node index.js directly (which loads the extension module but never calls the exported function). The extension requires pi's session lifecycle — pi loads extensions, fires session_start, and the Zulip extension hooks into that event.

Fix: Run pi --mode rpc (not node index.js). The --mode rpc flag keeps pi alive listening for RPC commands on stdin while the Zulip extension's router runs in the background via the session_start hook.

ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \
  --interpreter none -- --mode rpc --no-session

Additional fixes applied:

• Fixed models.json: replaced qwen3.6-35B-A3B (not authorized by LiteLLM) with syslog-auto + strix-moe (prevents silent worker failure per Lesson #11)
• Removed stale systemd unit abiba-zulip.service (pointed to old TS code)
• PM2 saved for auto-restart on boot

Appendix: PM2 Ecosystem Config (Optional)

If preferred over manual pm2 start, create /root/ecosystem.config.js entry:

module.exports = {
  apps: [{
    name: 'abiba-zulip',
    script: '/bin/pi',
    interpreter: 'none',
    args: '--mode rpc --no-session',
    cwd: '/root',
    env: {
      ZULIP_ROLE: 'router',
    },
    log_file: '/root/.pm2/logs/abiba-zulip-out.log',
    error_file: '/root/.pm2/logs/abiba-zulip-error.log',
    max_restarts: 20,
    restart_delay: 5000,
  }]
};

---

Last verified good state: 2026-07-13 — Extension v2 running via pi --mode rpc, health endpoint :9200 returning {status:"ok",connected:true}, queue a669f21e.