---
kind: responsibility
name: zulip-self-heal
status: retired
description: >
  RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
  Self-healing for Zulip infrastructure continues through Hermes agents.
  Abiba no longer monitors or manages Zulip.
agent: abiba
triggers:
  - none (retired)
---

> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.

## Maintains

- zulip-server: { status: "healthy" | "degraded" | "down" }
- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
- zulip-config: { syntax_valid: bool, proxy_configured: bool }
- last_action: { timestamp, action, result }

## Detection Rules

### Rule 1: Zulip Server Down (502/000)
- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
- **Fix**: `docker restart zulip-zulip-1` if container crashed
- **Verify**: Re-test server_settings, wait for "healthy"

### Rule 2: Zulip Config Corruption (400/500 non-proxy)
- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s

### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
- **Verify**: Re-test via NetBird URL

### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
- **Fix (pi)**: `pm2 restart abiba-zulip`
- **Fix (Hermes)**: `hermes gateway restart` on agent host
- **Verify**: Check health endpoint for `queue_healthy: true`

### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
- **Detect**: No heartbeat in agent log for >600s
- **Fix**: Restart gateway on affected host
- **Verify**: Check for new heartbeat in logs within 60s

### Rule 6: NetBird Tunnel Down
- **Detect**: Server returns 502 with NetBird HTML in response
- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
- **Fix**: Alert user — NetBird outage requires manual intervention
- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"

## Hosts & Credentials

| Host | IP | User | Service |
|------|----|------|---------|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |

## Debounce

All restart actions must debounce: minimum 300s between restarts per agent.
Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).

## Reporting

Every cycle produces a knowledge graph node:
- Title: `[LEARN] zulip-self-heal: <timestamp>`
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"

## Configuration

Zulip server config at 192.168.68.19:
- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
- Fixed values:
  - `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
  - `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
  - `LOAD_BALANCER_IPS = ["192.168.68.10"]`