feat: pm2-self-heal — monitors and restarts critical PM2 processes
Responsibility contract that checks abiba-zulip and abiba-telegram every 5 minutes. Restarts stopped/errored processes, escalates if restart count exceeds 5/hour. Logs to knowledge graph and alerts owner via Zulip DM.
This commit is contained in:
@@ -0,0 +1,72 @@
|
|||||||
|
---
|
||||||
|
kind: responsibility
|
||||||
|
name: pm2-self-heal
|
||||||
|
description: >
|
||||||
|
Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and
|
||||||
|
auto-restarts any that are stopped or errored. Logs every action to
|
||||||
|
the knowledge graph and alerts the owner via Zulip DM on failures.
|
||||||
|
---
|
||||||
|
|
||||||
|
## Maintains
|
||||||
|
|
||||||
|
- abiba-zulip: { status: "online", uptime: string, restarts: number }
|
||||||
|
- abiba-telegram: { status: "online", uptime: string, restarts: number }
|
||||||
|
- last_check: timestamp
|
||||||
|
|
||||||
|
## Continuity
|
||||||
|
|
||||||
|
- Self-driven: check every 300 seconds (5 min)
|
||||||
|
- Also wakes on user request (`prose run pm2-self-heal`)
|
||||||
|
|
||||||
|
## Remediation Rules
|
||||||
|
|
||||||
|
### Rule 1: Process Stopped or Errored
|
||||||
|
- **Detect**: `pm2 status` shows "stopped" or "errored" for a named process
|
||||||
|
- **Fix**: `pm2 restart <name>`
|
||||||
|
- **Verify**: Re-check status after 5 seconds
|
||||||
|
- **Escalate**: If still failed after 2 retries, send Zulip DM to owner
|
||||||
|
|
||||||
|
### Rule 2: Process Restarting Too Often
|
||||||
|
- **Detect**: `pm2 status` shows restarts > 5 in the last hour
|
||||||
|
- **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
|
||||||
|
- **Escalate**: Always — alert owner with restart count
|
||||||
|
|
||||||
|
## Execution
|
||||||
|
|
||||||
|
1. **Check PM2 status** — Run `pm2 status` and parse the table
|
||||||
|
2. **For each critical process** (abiba-zulip, abiba-telegram):
|
||||||
|
- If status is "online" → pass
|
||||||
|
- If status is "stopped" or "errored" → apply Rule 1
|
||||||
|
- If restarts > 5 → apply Rule 2
|
||||||
|
3. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken
|
||||||
|
4. **Alert owner** — Send Zulip DM if escalation needed
|
||||||
|
5. **Wait 5 min** → repeat from step 1
|
||||||
|
|
||||||
|
## Example Output (when healthy)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"run_id": "pm2-heal-001",
|
||||||
|
"checked_at": "2026-06-26T15:00:00Z",
|
||||||
|
"processes": {
|
||||||
|
"abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 },
|
||||||
|
"abiba-telegram": { "status": "online", "uptime": "48h", "restarts": 0 }
|
||||||
|
},
|
||||||
|
"actions_taken": [],
|
||||||
|
"overall": "healthy"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Output (when fixed)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"processes": {
|
||||||
|
"abiba-zulip": { "status": "errored → restarted → online" }
|
||||||
|
},
|
||||||
|
"actions_taken": [
|
||||||
|
{ "target": "abiba-zulip", "action": "pm2 restart", "result": "online" }
|
||||||
|
],
|
||||||
|
"overall": "fixed"
|
||||||
|
}
|
||||||
|
```
|
||||||
Reference in New Issue
Block a user