prose-contracts/abiba-zulip-restore.prose.md

---
kind: function
name: abiba-zulip-restore
description: >
  Restores Zulip connectivity for Abiba (pi agent). Verifies the v2 router-worker
  extension code, starts a PM2 process as the Zulip gateway with correct env vars,
  validates health endpoint, and confirms DM delivery. Run this whenever Abiba
  stops responding on Zulip or after system restart.
agent: abiba
version: 1.0.0
status: active
runtime_contract: 2
---

# Abiba Zulip Restore — Resume pi Zulip Communication

Single-shot function that restores full Zulip connectivity for the Abiba pi agent.
Covers extension code validation, PM2 process management, health endpoint
verification, and DM loopback testing.

## Live-State Fields

| Field | Value | Trust |
|-------|-------|-------|
| Agent name | abiba | ✅ |
| Bot email | abiba-bot@chat.sysloggh.net | ✅ |
| Zulip server | https://chat.sysloggh.net | ✅ |
| Extension path | /root/.pi/agent/extensions/zulip/index.js | ✅ |
| Config path | /root/.pi/agent/extensions/zulip/config.yaml | ✅ |
| Health port | 9200 | ✅ |
| @all-bots user ID | 20 | ✅ (config, verified by API at runtime) |
| PM2 process name | abiba-zulip | ✅ |
| Provider | syslog-harness (http://192.168.68.116/v1) | ✅ |
| Default model | deepseek-v4-pro | ✅ (settings.json) |

## Architecture

The pi Zulip extension uses a **router-worker architecture** (v2):

- **Router** (PM2 `abiba-zulip`, `ZULIP_ROLE=router`): Polls Zulip for events,
  maintains a per-sender pool of pi RPC worker processes. Each sender gets
  their own `pi --mode rpc --session-dir` process with persistent sessions.
  Handles streaming edits back to Zulip.
- **Worker** (child `pi --mode rpc`): Runs the agent with per-sender persistent
  sessions. No Zulip logic in the worker — the router handles all Zulip I/O.

The extension loads in ALL pi sessions (via `settings.json` extensions array)
but is a **NO-OP** unless `ZULIP_ROLE=router` is set. Only the PM2 router process
carries the env var.

## Maintains

- extension_valid: bool — Extension code imports without errors
- pm2_running: bool — PM2 process `abiba-zulip` is online
- health_responding: bool — GET :9200/health returns "ok"
- zulip_connected: bool — Queue registered, bot identity resolved
- loopback_delivered: bool — Test DM sent and received
- model_valid: bool — Configured models match LiteLLM authorized models

### Postconditions

- PM2 process `abiba-zulip` online and stable (uptime > 30s)
- Health endpoint returns `{ status: "ok", connected: true }`
- Worker pool creates sessions on demand
- Echo prevention active (BOT_EMAILS includes all known bots)
- PM2 saved for auto-restart on boot

## Requires

- Node.js with `yaml` module available
- PM2 installed globally
- Zulip API key in `config.yaml`
- `pi` CLI available on PATH
- Zulip server accessible at https://chat.sysloggh.net
- LiteLLM provider accessible at http://192.168.68.116/v1

## Execution

### Step 1: Validate Extension Code

```bash
node -e "import('file:///root/.pi/agent/extensions/zulip/index.js').then(() => console.log('OK')).catch(e => {console.error('FAIL:', e.message); process.exit(1)})"
```

Expected: `OK`. If FAIL → check for missing dependencies, syntax errors.

### Step 2: Validate Model IDs

Compare configured models against LiteLLM authorized models:

```bash
API_KEY=$(grep -oP 'apiKey:\s*\K.*' /root/.pi/agent/models.json | head -1)
curl -s -H "Authorization: Bearer $API_KEY" http://192.168.68.116/v1/models | \
  python3 -c "import json,sys; d=json.load(sys.stdin); [print(m['id']) for m in d.get('data',[])]" 2>/dev/null
```

Check: Every model ID in `models.json` must appear in the authorized list.
If not → fix `models.json` to only include authorized models (prefer `syslog-auto`).

### Step 3: Verify Config Integrity

```bash
python3 -c "
import yaml, sys
with open('/root/.pi/agent/extensions/zulip/config.yaml') as f:
    cfg = yaml.safe_load(f)
required = ['zulip.site', 'zulip.email', 'zulip.api_key', 'agent.name']
for k in required:
    keys = k.split('.')
    v = cfg
    for kk in keys:
        v = v.get(kk)
        if v is None:
            print(f'MISSING: {k}')
            sys.exit(1)
print('Config valid')
print(f'  site={cfg[\"zulip\"][\"site\"]}')
print(f'  email={cfg[\"zulip\"][\"email\"]}')
print(f'  agent={cfg[\"agent\"][\"name\"]}')
print(f'  health_port={cfg.get(\"health_port\", 9200)}')
"
```

Expected: Config valid with all fields non-empty.

### Step 4: Remove Stale Systemd Service

The old `abiba-zulip.service` points to `/opt/abiba-zulip/dist/index.js` (compiled
TypeScript, not the v2 extension). It's disabled and stale. Remove it:

```bash
systemctl stop abiba-zulip 2>/dev/null || true
systemctl disable abiba-zulip 2>/dev/null || true
rm -f /etc/systemd/system/abiba-zulip.service
systemctl daemon-reload
```

### Step 5: Start PM2 Process

**Critical:** The extension MUST run via `pi --mode rpc`, NOT `node index.js` directly.
The extension exports a function that requires pi's session lifecycle. Running
`node index.js` loads the module but never calls the export, so nothing happens.
`pi --mode rpc` loads all extensions (including Zulip) and fires `session_start`.

```bash
# Stop existing if any
pm2 delete abiba-zulip 2>/dev/null || true

# Start pi in RPC mode with ZULIP_ROLE=router env
ZULIP_ROLE=router pm2 start "$(which pi)" \
  --name abiba-zulip \
  --interpreter none \
  -- --mode rpc --no-session
```

Wait 10 seconds for pi to load all extensions, fire session_start, and the Zulip
router to register its event queue.

### Step 6: Validate PM2 Process

```bash
pm2 show abiba-zulip --no-color
```

Check: `status=online`, `restarts=0`, `uptime > 5s`.

### Step 7: Check Logs for Connection

```bash
sleep 3
tail -20 /root/.pm2/logs/abiba-zulip-out.log
```

Look for:
- `[zulip-ext] Connecting to https://chat.sysloggh.net as abiba-bot@chat.sysloggh.net…`
- `[zulip-ext] Bot user_id=N, all-bots user_id=N`
- `[zulip-ext] Connected, queue=N`
- `[zulip-ext] Echo prevention: N bot emails`
- `[zulip-ext] Health endpoint on :9200`

If error → check API key, network to chat.sysloggh.net.

### Step 8: Health Endpoint

```bash
curl -s http://localhost:9200/health | python3 -m json.tool
```

Check:
- `status: "ok"` (not "down")
- `zulip.connected: true`
- `zulip.queue_id` is non-null string
- `zulip.bot_user_id` is positive integer

### Step 9: DM Loopback Test

```bash
curl -s http://localhost:9200/health | python3 -c "
import json,sys
d = json.load(sys.stdin)
if d.get('zulip',{}).get('connected'):
    print(f'✅ Zulip connected. Queue: {d[\"zulip\"][\"queue_id\"]}')
    print(f'   Bot user_id: {d[\"zulip\"][\"bot_user_id\"]}')
    print(f'   Messages processed: {d[\"zulip\"][\"messages_processed\"]}')
else:
    print('❌ Zulip NOT connected')
    sys.exit(1)
"
```

### Step 10: Save PM2 for Auto-Start

```bash
pm2 save
pm2 startup systemd -u root --hp /root 2>/dev/null || true
```

### Step 11: Report

Compile results:

| Check | Pass? |
|-------|-------|
| Extension code imports | extension_valid |
| Model IDs authorized | model_valid |
| Config integrity | config_valid |
| PM2 process online | pm2_running |
| Health endpoint | health_responding |
| Zulip connected | zulip_connected |

All pass → ✅ **Abiba Zulip restored.** Relay success to user.

Partial failure → see recovery matrix below.

## Recovery Matrix

| Failure | Recovery |
|---------|----------|
| Extension import fails | Check `node_modules/zulip-js` exists; run `npm install` in extension dir |
| Model ID mismatch | Fix `models.json` to use `syslog-auto` as default model; remove invalid IDs |
| Config missing | Restore from backup or recreate from scratch |
| PM2 won't start | Check `node` version (>=18); check port 9200 not in use |
| Health "down" | Check logs for connection errors; verify Zulip API key; check network |
| "address already in use" | Kill old process: `fuser -k 9200/tcp` |
| Queue registration fails | Check Zulip API key validity; verify bot is active in Zulip admin |
| Rate limit (429) | Extension has built-in retry-after handling — wait, don't restart |

## Known Failure Modes

| Symptom | Root Cause | Recovery |
|---------|-----------|----------|
| Extension loads but no events | `ZULIP_ROLE` not set | Ensure PM2 env has `ZULIP_ROLE=router` |
| Worker stays "busy" forever | Model ID not authorized by LiteLLM | Fix models.json (lesson #11) |
| Placeholder sent but no response | editMessage API fails silently | Extension has fallback (sends new msg); check Zulip API |
| Queue expires rapidly | Poll interval too aggressive | v2 uses 3s poll with long-poll — should be fine |
| Bot doesn't respond to @mentions | Not subscribed to stream | Bot auto-subscribes via API |
| Stale error in health | `last_error` not cleared | v2 clears on successful poll (lesson #4) |

## Appendix: Root Cause & Fix Summary (2026-07-13)

**Problem:** Zulip extension was offline — no PM2 process running.

**Root cause:** The PM2 command ran `node index.js` directly (which loads the
extension module but never calls the exported function). The extension requires
pi's session lifecycle — pi loads extensions, fires `session_start`, and the
Zulip extension hooks into that event.

**Fix:** Run `pi --mode rpc` (not `node index.js`). The `--mode rpc` flag keeps
pi alive listening for RPC commands on stdin while the Zulip extension's router
runs in the background via the `session_start` hook.

```bash
ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \
  --interpreter none -- --mode rpc --no-session
```

**Additional fixes applied:**
- Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM)
  with `syslog-auto` + `strix-moe` (prevents silent worker failure per
  Lesson #11)
- Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code)
- PM2 saved for auto-restart on boot

## Appendix: PM2 Ecosystem Config (Optional)

If preferred over manual `pm2 start`, create `/root/ecosystem.config.js` entry:

```js
module.exports = {
  apps: [{
    name: 'abiba-zulip',
    script: '/bin/pi',
    interpreter: 'none',
    args: '--mode rpc --no-session',
    cwd: '/root',
    env: {
      ZULIP_ROLE: 'router',
    },
    log_file: '/root/.pm2/logs/abiba-zulip-out.log',
    error_file: '/root/.pm2/logs/abiba-zulip-error.log',
    max_restarts: 20,
    restart_delay: 5000,
  }]
};
```

---

**Last verified good state**: 2026-07-13 — Extension v2 running via `pi --mode rpc`, health endpoint :9200 returning `{status:"ok",connected:true}`, queue a669f21e.