mumuni-bot
ca39fead74
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Post-merge verification of PR #54 found: 1. Lifecycle commands in zulip-health/zulip-self-heal used 'sudo systemctl' for hermes-gateway - sudo is NOT installed on kagentz (no sudoers, no polkit user rules). Correct path: root SSH invocation (root@192.168.68.14), matching how the Proxmox host actually manages the unit. 2. hermes-zulip-restore.prose.md line 5 + zulip-resilience-v3 line 423 still said 'Mumuni CT100' in prose - repointed. Found via independent review + live runtime check (whoami, which sudo, journalctl, systemctl show). Refs PR #54, relay #738/#739.
92 lines
4.0 KiB
Markdown
92 lines
4.0 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: zulip-self-heal
|
|
status: retired
|
|
description: >
|
|
RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
|
|
Self-healing for Zulip infrastructure continues through Hermes agents.
|
|
Abiba no longer monitors or manages Zulip.
|
|
agent: abiba
|
|
triggers:
|
|
- none (retired)
|
|
---
|
|
|
|
> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
|
|
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
|
|
> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
|
|
|
|
## Maintains
|
|
|
|
- zulip-server: { status: "healthy" | "degraded" | "down" }
|
|
- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
|
|
- zulip-config: { syntax_valid: bool, proxy_configured: bool }
|
|
- last_action: { timestamp, action, result }
|
|
|
|
## Detection Rules
|
|
|
|
### Rule 1: Zulip Server Down (502/000)
|
|
- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
|
|
- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
|
|
- **Fix**: `docker restart zulip-zulip-1` if container crashed
|
|
- **Verify**: Re-test server_settings, wait for "healthy"
|
|
|
|
### Rule 2: Zulip Config Corruption (400/500 non-proxy)
|
|
- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
|
|
- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
|
|
- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
|
|
- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
|
|
|
|
### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
|
|
- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
|
|
- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
|
|
- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
|
|
- **Verify**: Re-test via NetBird URL
|
|
|
|
### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
|
|
- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
|
|
- **Fix (pi)**: `pm2 restart abiba-zulip`
|
|
- **Fix (Hermes)**: `hermes gateway restart` on agent host
|
|
- **Verify**: Check health endpoint for `queue_healthy: true`
|
|
|
|
### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
|
|
- **Detect**: No heartbeat in agent log for >600s
|
|
- **Fix**: Restart gateway on affected host
|
|
- **Verify**: Check for new heartbeat in logs within 60s
|
|
|
|
### Rule 6: NetBird Tunnel Down
|
|
- **Detect**: Server returns 502 with NetBird HTML in response
|
|
- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
|
|
- **Fix**: Alert user — NetBird outage requires manual intervention
|
|
- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
|
|
|
|
## Hosts & Credentials
|
|
|
|
| Host | IP | User | Service |
|
|
|------|----|------|---------|
|
|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
|
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
|
| Mumuni | 192.168.68.14 (kagentz CT105) | root (SSH) | `ssh root@192.168.68.14 "systemctl restart hermes-gateway"` |
|
|
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
|
|
|
## Debounce
|
|
|
|
All restart actions must debounce: minimum 300s between restarts per agent.
|
|
Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
|
|
|
|
## Reporting
|
|
|
|
This contract is RETIRED — health-check logs are NOT knowledge graph content.
|
|
No graph nodes are created. Logs go to Gitea (SyslogSolution/health-logs).
|
|
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
|
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
|
|
|
## Configuration
|
|
|
|
Zulip server config at 192.168.68.19:
|
|
- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
|
|
- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
|
|
- Fixed values:
|
|
- `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
|
|
- `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
|
|
- `LOAD_BALANCER_IPS = ["192.168.68.10"]`
|