prose-contracts/zulip-self-heal.prose.md

kind, name, status, description, agent, triggers

kind	name	status	description	agent	triggers
responsibility	zulip-self-heal	retired	RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned). Self-healing for Zulip infrastructure continues through Hermes agents. Abiba no longer monitors or manages Zulip.	abiba	none (retired)

⚠️ RETIRED — This contract was embedded in the pi Zulip extension code (performHealthCheck()). That code has been removed. Zulip self-healing for Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.

Maintains

• zulip-server: { status: "healthy" | "degraded" | "down" }
• zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
• zulip-config: { syntax_valid: bool, proxy_configured: bool }
• last_action: { timestamp, action, result }

Detection Rules

Rule 1: Zulip Server Down (502/000)

• Detect: curl https://chat.sysloggh.net/api/v1/server_settings returns 502/000
• Diagnose: SSH to 192.168.68.19, check docker ps for zulip container status
• Fix: docker restart zulip-zulip-1 if container crashed
• Verify: Re-test server_settings, wait for "healthy"

Rule 2: Zulip Config Corruption (400/500 non-proxy)

• Detect: Server returns 400 or 500 with "SyntaxError" in Django error logs
• Diagnose: docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log
• Fix: Identify syntax error in /home/zulip/deployments/*/zproject/prod_settings.py, fix with sed
• Verify: docker restart zulip-zulip-1, re-test after 60s

Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)

• Detect: Server returns 500 with "ProxyMisconfigurationError" in logs
• Diagnose: Check error for proxy IP (e.g., "detected from 192.168.68.10")
• Fix: Add [loadbalancer] ips = <detected_ip> to /etc/zulip/zulip.conf + restart
• Verify: Re-test via NetBird URL

Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)

• Detect: Agent health shows queue_healthy: false with BAD_EVENT_QUEUE_ID
• Fix (pi): pm2 restart abiba-zulip
• Fix (Hermes): hermes gateway restart on agent host
• Verify: Check health endpoint for queue_healthy: true

Rule 5: Agent Adapter Crashed (no heartbeat >10min)

• Detect: No heartbeat in agent log for >600s
• Fix: Restart gateway on affected host
• Verify: Check for new heartbeat in logs within 60s

Rule 6: NetBird Tunnel Down

• Detect: Server returns 502 with NetBird HTML in response
• Diagnose: Access server directly via 192.168.68.19 to check if Zulip is up
• Fix: Alert user — NetBird outage requires manual intervention
• Escalate: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"

Hosts & Credentials

Host	IP	User	Service
Zulip server	192.168.68.19	root	Docker: zulip-zulip-1
Abiba (pi)	localhost	root	PM2: abiba-zulip
Mumuni	192.168.68.24 (CT100 abiba)	root	hermes gateway restart
Tanko	192.168.68.122	jerome	PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart

Debounce

All restart actions must debounce: minimum 300s between restarts per agent. Track via /tmp/zulip-heal-debounce-<agent> (unix timestamp of last restart).

Reporting

Every cycle produces a knowledge graph node:

• Title: [LEARN] zulip-self-heal: <timestamp>
• metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
• Issues fixed → DM: "🛠 Zulip Self-Heal: fixed "
• Issues escalated → DM: "⚠️ Zulip Self-Heal: needs attention"

Configuration

Zulip server config at 192.168.68.19:

• /etc/zulip/zulip.conf — [loadbalancer] ips for NetBird trust
• /home/zulip/deployments/<date>/zproject/prod_settings.py — Django settings
• Fixed values:
  • CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]
  • SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
  • LOAD_BALANCER_IPS = ["192.168.68.10"]