prose-contracts/pm2-self-heal.prose.md

kind, name, description

kind	name	description
responsibility	pm2-self-heal	Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner, spoton-service, zulip-watchdog) and auto-restarts any that are stopped or errored. Logs every action to Gitea (SyslogSolution/health-logs — not knowledge graph, hard rule) and alerts the owner via Zulip DM on failures. CRITICAL: Never restart abiba-zulip — it runs this contract. AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative): gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2; gpu-watchdog decommissioned (function folded into gpu-monitor.service); gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the 2026-07-04 'removed/decommissioned' note was stale and is removed).

Maintains

• abiba-telegram: { status: "online", uptime: string, restarts: number }
• abiba-zulip: { status: "online", uptime: string, restarts: number }
• gitea-runner: { status: "online", uptime: string, restarts: number }
• spoton-service: { status: "online", uptime: string, restarts: number }
• zulip-watchdog: { status: "online", uptime: string, restarts: number }
• last_check: timestamp

Note (2026-07-04, SUPERSEDED 2026-08-09): abiba-zulip remains ONLINE and is monitored — the decommission note was stale (process re-added; do not treat it as removed).

Continuity

• Self-driven: check every 300 seconds (5 min)
• Also wakes on user request (prose run pm2-self-heal)

Remediation Rules

Rule 1: Process Stopped or Errored

• Detect: pm2 status shows "stopped" or "errored" for a named process
• Fix: pm2 restart <name>
• Verify: Re-check status after 5 seconds
• Escalate: If still failed after 2 retries, send Zulip DM to owner

Rule 2: Process Restarting Too Often (crash-loop guard)

• Detect: pm2 status shows restarts > 30 (cumulative lifetime counter) or a process reporting a restart count > 1000 while showing "online" (a crash-loop mask)
• Note: PM2 counter never decrements; only full delete+re-add resets it
• Fix: pm2 delete <name> && pm2 start <ecosystem> --only <name>
• Script guard (2026-08-16): scripts/pm2-self-heal.sh now restarts abiba-telegram when TEL_RESTARTS > 1000 even if the process reports "online" — catches a quiet crash-loop that never toggles status to "stopped"/"errored" (e.g. the 10k-restarts spoton incident). Alerts include the restart count.
• Escalate: Only when restarts > 30 — alerts to Zulip DM
• Historical fix: Previous cycles were caused by abiba-zulip extension's stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2). This was the root cause of the 18-restart accumulation. Threshold raised and PM2 counter reset on 2026-06-28.

Execution

1. Check PM2 status — Run pm2 status --no-color and parse the table (5th data column = PID, 8th = restarts, 9th = status)
2. Check abiba-telegram:
   • If status is "online" → pass
   • If status is "stopped" or "errored" → apply Rule 1
   • If restarts > 5 → alert owner
3. Check abiba-zulip (self-process, read-only):
   • If status is "online" → pass, log restarts count
   • If status is "stopped" or "errored" → DO NOT RESTART — alert owner immediately
   • If restarts > 5 in last hour → alert owner with full diagnostics
4. Log results — Append to SyslogSolution/health-logs/pm2/{timestamp}.md in Gitea (not knowledge graph — hard rule)
5. Alert — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
6. Wait 5 min → repeat from step 1

Example Output (when healthy)

{
  "run_id": "pm2-heal-001",
  "checked_at": "2026-06-26T15:00:00Z",
  "processes": {
    "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 },
    "abiba-telegram": { "status": "online", "uptime": "48h", "restarts": 0 }
  },
  "actions_taken": [],
  "overall": "healthy"
}

Example Output (when fixed)

{
  "processes": {
    "abiba-telegram": { "status": "errored → restarted → online" },
    "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 }
  },
  "actions_taken": [
    { "target": "abiba-telegram", "action": "pm2 restart", "result": "online" }
  ],
  "overall": "fixed"
}