feat: pm2-self-heal — monitors and restarts critical PM2 processes

Responsibility contract that checks abiba-zulip and abiba-telegram every 5 minutes. Restarts stopped/errored processes, escalates if restart count exceeds 5/hour. Logs to knowledge graph and alerts owner via Zulip DM.
2026-06-26 17:44:27 +00:00
parent 41bd3c5748
commit f340c6301d
1 changed files with 72 additions and 0 deletions
@@ -0,0 +1,72 @@
+---
+kind: responsibility
+name: pm2-self-heal
+description: >
+  Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and
+  auto-restarts any that are stopped or errored. Logs every action to
+  the knowledge graph and alerts the owner via Zulip DM on failures.
+---
+
+## Maintains
+
+- abiba-zulip: { status: "online", uptime: string, restarts: number }
+- abiba-telegram: { status: "online", uptime: string, restarts: number }
+- last_check: timestamp
+
+## Continuity
+
+- Self-driven: check every 300 seconds (5 min)
+- Also wakes on user request (`prose run pm2-self-heal`)
+
+## Remediation Rules
+
+### Rule 1: Process Stopped or Errored
+- **Detect**: `pm2 status` shows "stopped" or "errored" for a named process
+- **Fix**: `pm2 restart <name>`
+- **Verify**: Re-check status after 5 seconds
+- **Escalate**: If still failed after 2 retries, send Zulip DM to owner
+
+### Rule 2: Process Restarting Too Often
+- **Detect**: `pm2 status` shows restarts > 5 in the last hour
+- **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
+- **Escalate**: Always — alert owner with restart count
+
+## Execution
+
+1. **Check PM2 status** — Run `pm2 status` and parse the table
+2. **For each critical process** (abiba-zulip, abiba-telegram):
+   - If status is "online" → pass
+   - If status is "stopped" or "errored" → apply Rule 1
+   - If restarts > 5 → apply Rule 2
+3. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken
+4. **Alert owner** — Send Zulip DM if escalation needed
+5. **Wait 5 min** → repeat from step 1
+
+## Example Output (when healthy)
+
+```json
+{
+  "run_id": "pm2-heal-001",
+  "checked_at": "2026-06-26T15:00:00Z",
+  "processes": {
+    "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 },
+    "abiba-telegram": { "status": "online", "uptime": "48h", "restarts": 0 }
+  },
+  "actions_taken": [],
+  "overall": "healthy"
+}
+```
+
+## Example Output (when fixed)
+
+```json
+{
+  "processes": {
+    "abiba-zulip": { "status": "errored → restarted → online" }
+  },
+  "actions_taken": [
+    { "target": "abiba-zulip", "action": "pm2 restart", "result": "online" }
+  ],
+  "overall": "fixed"
+}
+```