fix: pm2-self-heal — self-process guard, never restart abiba-zulip
- Added --no-color flag to all PM2 commands - Self-process guard: abiba-zulip is READ-ONLY, never restarted - Corrected column indices (PID=6, restarts=8, status=9) - Added pm2-self-heal.sh shell script implementation
This commit is contained in:
+14
-8
@@ -5,6 +5,7 @@ description: >
|
||||
Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and
|
||||
auto-restarts any that are stopped or errored. Logs every action to
|
||||
the knowledge graph and alerts the owner via Zulip DM on failures.
|
||||
CRITICAL: Never restart abiba-zulip — it runs this contract.
|
||||
---
|
||||
|
||||
## Maintains
|
||||
@@ -33,14 +34,18 @@ description: >
|
||||
|
||||
## Execution
|
||||
|
||||
1. **Check PM2 status** — Run `pm2 status` and parse the table
|
||||
2. **For each critical process** (abiba-zulip, abiba-telegram):
|
||||
1. **Check PM2 status** — Run `pm2 status --no-color` and parse the table (5th data column = PID, 8th = restarts, 9th = status)
|
||||
2. **Check abiba-telegram**:
|
||||
- If status is "online" → pass
|
||||
- If status is "stopped" or "errored" → apply Rule 1
|
||||
- If restarts > 5 → apply Rule 2
|
||||
3. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken
|
||||
4. **Alert owner** — Send Zulip DM if escalation needed
|
||||
5. **Wait 5 min** → repeat from step 1
|
||||
- If restarts > 5 → alert owner
|
||||
3. **Check abiba-zulip** (self-process, read-only):
|
||||
- If status is "online" → pass, log restarts count
|
||||
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
|
||||
- If restarts > 5 in last hour → alert owner with full diagnostics
|
||||
4. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken
|
||||
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
|
||||
6. **Wait 5 min** → repeat from step 1
|
||||
|
||||
## Example Output (when healthy)
|
||||
|
||||
@@ -62,10 +67,11 @@ description: >
|
||||
```json
|
||||
{
|
||||
"processes": {
|
||||
"abiba-zulip": { "status": "errored → restarted → online" }
|
||||
"abiba-telegram": { "status": "errored → restarted → online" },
|
||||
"abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 }
|
||||
},
|
||||
"actions_taken": [
|
||||
{ "target": "abiba-zulip", "action": "pm2 restart", "result": "online" }
|
||||
{ "target": "abiba-telegram", "action": "pm2 restart", "result": "online" }
|
||||
],
|
||||
"overall": "fixed"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user