mumuni-bot
0aa0ea4906
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2dfc3e1 (PR #50's out-of-band squash) overwrote agent-health-check.py with
a version whose AGENTS dict was empty — the health check silently skipped
every agent since. Restore the pre-stomp roster verbatim: tanko (runtime:
dsh), abiba, koby, koonimo.
Also land two Tanko-DSH rows from PR #52 that the earlier merge missed:
- hermes-zulip-plugin live-state table (plugin retired on CT112)
- zulip-self-heal restart-services table (restart via DSH service, not
hermes gateway restart)
Mumuni rows from the branch were NOT restored: they carry pre-migration
CT100/.24 data, superseded by PRs #54-56 (Mumuni now on kagentz CT105/.14).
Closes the reland of PR #52's substance; the stale branch head stays closed.
92 lines
3.9 KiB
Markdown
92 lines
3.9 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: zulip-self-heal
|
|
status: retired
|
|
description: >
|
|
RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
|
|
Self-healing for Zulip infrastructure continues through Hermes agents.
|
|
Abiba no longer monitors or manages Zulip.
|
|
agent: abiba
|
|
triggers:
|
|
- none (retired)
|
|
---
|
|
|
|
> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
|
|
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
|
|
> agents (Mumuni on Hermes, Tanko on DSH) continues through their own platform
|
|
> monitoring.
|
|
|
|
## Maintains
|
|
|
|
- zulip-server: { status: "healthy" | "degraded" | "down" }
|
|
- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
|
|
- zulip-config: { syntax_valid: bool, proxy_configured: bool }
|
|
- last_action: { timestamp, action, result }
|
|
|
|
## Detection Rules
|
|
|
|
### Rule 1: Zulip Server Down (502/000)
|
|
- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
|
|
- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
|
|
- **Fix**: `docker restart zulip-zulip-1` if container crashed
|
|
- **Verify**: Re-test server_settings, wait for "healthy"
|
|
|
|
### Rule 2: Zulip Config Corruption (400/500 non-proxy)
|
|
- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
|
|
- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
|
|
- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
|
|
- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
|
|
|
|
### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
|
|
- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
|
|
- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
|
|
- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
|
|
- **Verify**: Re-test via NetBird URL
|
|
|
|
### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
|
|
- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
|
|
- **Fix (pi)**: `pm2 restart abiba-zulip`
|
|
- **Fix (Hermes)**: `hermes gateway restart` on agent host
|
|
- **Verify**: Check health endpoint for `queue_healthy: true`
|
|
|
|
### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
|
|
- **Detect**: No heartbeat in agent log for >600s
|
|
- **Fix**: Restart gateway on affected host
|
|
- **Verify**: Check for new heartbeat in logs within 60s
|
|
|
|
### Rule 6: NetBird Tunnel Down
|
|
- **Detect**: Server returns 502 with NetBird HTML in response
|
|
- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
|
|
- **Fix**: Alert user — NetBird outage requires manual intervention
|
|
- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
|
|
|
|
## Hosts & Credentials
|
|
|
|
| Host | IP | User | Service |
|
|
|------|----|------|---------|
|
|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
|
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
|
| Tanko | 192.168.68.122 (CT 112) | jerome | DSH (DeepSeek Harness) — restart via DSH service, not `hermes gateway restart` (no longer a Hermes agent since 2026-08-27) |
|
|
|
|
## Debounce
|
|
|
|
All restart actions must debounce: minimum 300s between restarts per agent.
|
|
Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
|
|
|
|
## Reporting
|
|
|
|
This contract is RETIRED — health-check logs are NOT knowledge graph content.
|
|
No graph nodes are created. Logs go to Gitea (SyslogSolution/health-logs).
|
|
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
|
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
|
|
|
## Configuration
|
|
|
|
Zulip server config at 192.168.68.19:
|
|
- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
|
|
- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
|
|
- Fixed values:
|
|
- `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
|
|
- `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
|
|
- `LOAD_BALANCER_IPS = ["192.168.68.10"]`
|