root
3b6cf44a30
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Mumuni CT114 destroyed. Mumuni now runs inside Abiba CT100 at 192.168.68.24. Updated all contract files and agent-health-check.py.
93 lines
4.0 KiB
Markdown
93 lines
4.0 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: zulip-self-heal
|
|
status: retired
|
|
description: >
|
|
RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
|
|
Self-healing for Zulip infrastructure continues through Hermes agents.
|
|
Abiba no longer monitors or manages Zulip.
|
|
agent: abiba
|
|
triggers:
|
|
- none (retired)
|
|
---
|
|
|
|
> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
|
|
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
|
|
> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
|
|
|
|
## Maintains
|
|
|
|
- zulip-server: { status: "healthy" | "degraded" | "down" }
|
|
- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
|
|
- zulip-config: { syntax_valid: bool, proxy_configured: bool }
|
|
- last_action: { timestamp, action, result }
|
|
|
|
## Detection Rules
|
|
|
|
### Rule 1: Zulip Server Down (502/000)
|
|
- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
|
|
- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
|
|
- **Fix**: `docker restart zulip-zulip-1` if container crashed
|
|
- **Verify**: Re-test server_settings, wait for "healthy"
|
|
|
|
### Rule 2: Zulip Config Corruption (400/500 non-proxy)
|
|
- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
|
|
- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
|
|
- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
|
|
- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
|
|
|
|
### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
|
|
- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
|
|
- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
|
|
- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
|
|
- **Verify**: Re-test via NetBird URL
|
|
|
|
### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
|
|
- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
|
|
- **Fix (pi)**: `pm2 restart abiba-zulip`
|
|
- **Fix (Hermes)**: `hermes gateway restart` on agent host
|
|
- **Verify**: Check health endpoint for `queue_healthy: true`
|
|
|
|
### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
|
|
- **Detect**: No heartbeat in agent log for >600s
|
|
- **Fix**: Restart gateway on affected host
|
|
- **Verify**: Check for new heartbeat in logs within 60s
|
|
|
|
### Rule 6: NetBird Tunnel Down
|
|
- **Detect**: Server returns 502 with NetBird HTML in response
|
|
- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
|
|
- **Fix**: Alert user — NetBird outage requires manual intervention
|
|
- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
|
|
|
|
## Hosts & Credentials
|
|
|
|
| Host | IP | User | Service |
|
|
|------|----|------|---------|
|
|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
|
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
|
| Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` |
|
|
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
|
|
|
## Debounce
|
|
|
|
All restart actions must debounce: minimum 300s between restarts per agent.
|
|
Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
|
|
|
|
## Reporting
|
|
|
|
Every cycle produces a knowledge graph node:
|
|
- Title: `[LEARN] zulip-self-heal: <timestamp>`
|
|
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
|
|
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
|
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
|
|
|
## Configuration
|
|
|
|
Zulip server config at 192.168.68.19:
|
|
- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
|
|
- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
|
|
- Fixed values:
|
|
- `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
|
|
- `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
|
|
- `LOAD_BALANCER_IPS = ["192.168.68.10"]`
|