root
253e19f680
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
93 lines
3.9 KiB
Markdown
93 lines
3.9 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: zulip-self-heal
|
|
status: retired
|
|
description: >
|
|
RETIRED 2026-07-04 — Was part of the pi Zulip extension (now decommissioned).
|
|
Self-healing for Zulip infrastructure continues through Hermes agents.
|
|
Abiba no longer monitors or manages Zulip.
|
|
agent: abiba
|
|
triggers:
|
|
- none (retired)
|
|
---
|
|
|
|
> **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
|
|
> (`performHealthCheck()`). That code has been removed. Zulip self-healing for
|
|
> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.
|
|
|
|
## Maintains
|
|
|
|
- zulip-server: { status: "healthy" | "degraded" | "down" }
|
|
- zulip-agents: { abiba: agent_state, mumuni: agent_state, tanko: agent_state }
|
|
- zulip-config: { syntax_valid: bool, proxy_configured: bool }
|
|
- last_action: { timestamp, action, result }
|
|
|
|
## Detection Rules
|
|
|
|
### Rule 1: Zulip Server Down (502/000)
|
|
- **Detect**: `curl https://chat.sysloggh.net/api/v1/server_settings` returns 502/000
|
|
- **Diagnose**: SSH to 192.168.68.19, check `docker ps` for zulip container status
|
|
- **Fix**: `docker restart zulip-zulip-1` if container crashed
|
|
- **Verify**: Re-test server_settings, wait for "healthy"
|
|
|
|
### Rule 2: Zulip Config Corruption (400/500 non-proxy)
|
|
- **Detect**: Server returns 400 or 500 with "SyntaxError" in Django error logs
|
|
- **Diagnose**: `docker exec zulip-zulip-1 tail -20 /var/log/zulip/errors.log`
|
|
- **Fix**: Identify syntax error in `/home/zulip/deployments/*/zproject/prod_settings.py`, fix with sed
|
|
- **Verify**: `docker restart zulip-zulip-1`, re-test after 60s
|
|
|
|
### Rule 3: Proxy Misconfiguration (500 with ProxyMisconfigurationError)
|
|
- **Detect**: Server returns 500 with "ProxyMisconfigurationError" in logs
|
|
- **Diagnose**: Check error for proxy IP (e.g., "detected from 192.168.68.10")
|
|
- **Fix**: Add `[loadbalancer] ips = <detected_ip>` to `/etc/zulip/zulip.conf` + restart
|
|
- **Verify**: Re-test via NetBird URL
|
|
|
|
### Rule 4: Agent Queue Expired (BAD_EVENT_QUEUE_ID)
|
|
- **Detect**: Agent health shows `queue_healthy: false` with BAD_EVENT_QUEUE_ID
|
|
- **Fix (pi)**: `pm2 restart abiba-zulip`
|
|
- **Fix (Hermes)**: `hermes gateway restart` on agent host
|
|
- **Verify**: Check health endpoint for `queue_healthy: true`
|
|
|
|
### Rule 5: Agent Adapter Crashed (no heartbeat >10min)
|
|
- **Detect**: No heartbeat in agent log for >600s
|
|
- **Fix**: Restart gateway on affected host
|
|
- **Verify**: Check for new heartbeat in logs within 60s
|
|
|
|
### Rule 6: NetBird Tunnel Down
|
|
- **Detect**: Server returns 502 with NetBird HTML in response
|
|
- **Diagnose**: Access server directly via 192.168.68.19 to check if Zulip is up
|
|
- **Fix**: Alert user — NetBird outage requires manual intervention
|
|
- **Escalate**: Send Zulip DM "NetBird tunnel down — Zulip unreachable via chat.sysloggh.net"
|
|
|
|
## Hosts & Credentials
|
|
|
|
| Host | IP | User | Service |
|
|
|------|----|------|---------|
|
|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
|
|
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
|
|
| Mumuni | 192.168.68.123 | root | `hermes gateway restart` |
|
|
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
|
|
|
|
## Debounce
|
|
|
|
All restart actions must debounce: minimum 300s between restarts per agent.
|
|
Track via `/tmp/zulip-heal-debounce-<agent>` (unix timestamp of last restart).
|
|
|
|
## Reporting
|
|
|
|
Every cycle produces a knowledge graph node:
|
|
- Title: `[LEARN] zulip-self-heal: <timestamp>`
|
|
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
|
|
- Issues fixed → DM: "🛠 Zulip Self-Heal: fixed <issue>"
|
|
- Issues escalated → DM: "⚠️ Zulip Self-Heal: <issue> needs attention"
|
|
|
|
## Configuration
|
|
|
|
Zulip server config at 192.168.68.19:
|
|
- `/etc/zulip/zulip.conf` — `[loadbalancer] ips` for NetBird trust
|
|
- `/home/zulip/deployments/<date>/zproject/prod_settings.py` — Django settings
|
|
- Fixed values:
|
|
- `CSRF_TRUSTED_ORIGINS = ["https://chat.sysloggh.net"]`
|
|
- `SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")`
|
|
- `LOAD_BALANCER_IPS = ["192.168.68.10"]`
|