From f340c6301df123da0e3da22cc5102a1dd97bba4f Mon Sep 17 00:00:00 2001 From: root Date: Fri, 26 Jun 2026 17:44:27 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20pm2-self-heal=20=E2=80=94=20monitors=20?= =?UTF-8?q?and=20restarts=20critical=20PM2=20processes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Responsibility contract that checks abiba-zulip and abiba-telegram every 5 minutes. Restarts stopped/errored processes, escalates if restart count exceeds 5/hour. Logs to knowledge graph and alerts owner via Zulip DM. --- pm2-self-heal.prose.md | 72 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 pm2-self-heal.prose.md diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md new file mode 100644 index 0000000..ae986c2 --- /dev/null +++ b/pm2-self-heal.prose.md @@ -0,0 +1,72 @@ +--- +kind: responsibility +name: pm2-self-heal +description: > + Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and + auto-restarts any that are stopped or errored. Logs every action to + the knowledge graph and alerts the owner via Zulip DM on failures. +--- + +## Maintains + +- abiba-zulip: { status: "online", uptime: string, restarts: number } +- abiba-telegram: { status: "online", uptime: string, restarts: number } +- last_check: timestamp + +## Continuity + +- Self-driven: check every 300 seconds (5 min) +- Also wakes on user request (`prose run pm2-self-heal`) + +## Remediation Rules + +### Rule 1: Process Stopped or Errored +- **Detect**: `pm2 status` shows "stopped" or "errored" for a named process +- **Fix**: `pm2 restart ` +- **Verify**: Re-check status after 5 seconds +- **Escalate**: If still failed after 2 retries, send Zulip DM to owner + +### Rule 2: Process Restarting Too Often +- **Detect**: `pm2 status` shows restarts > 5 in the last hour +- **Fix**: `pm2 delete && pm2 start --only ` +- **Escalate**: Always — alert owner with restart count + +## Execution + +1. **Check PM2 status** — Run `pm2 status` and parse the table +2. **For each critical process** (abiba-zulip, abiba-telegram): + - If status is "online" → pass + - If status is "stopped" or "errored" → apply Rule 1 + - If restarts > 5 → apply Rule 2 +3. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken +4. **Alert owner** — Send Zulip DM if escalation needed +5. **Wait 5 min** → repeat from step 1 + +## Example Output (when healthy) + +```json +{ + "run_id": "pm2-heal-001", + "checked_at": "2026-06-26T15:00:00Z", + "processes": { + "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 }, + "abiba-telegram": { "status": "online", "uptime": "48h", "restarts": 0 } + }, + "actions_taken": [], + "overall": "healthy" +} +``` + +## Example Output (when fixed) + +```json +{ + "processes": { + "abiba-zulip": { "status": "errored → restarted → online" } + }, + "actions_taken": [ + { "target": "abiba-zulip", "action": "pm2 restart", "result": "online" } + ], + "overall": "fixed" +} +```