fix(pm2-self-heal): document permanent stuck-detection fix, reset PM2 counter

Root cause: abiba-zulip extension had STUCK_THRESHOLD_MS=30min.
After 30min of inactivity (overnight), it reported stuck=True, which
triggered the health monitor to restart it — cycling restarts up to 18.

Permanent fixes:
1. STUCK_THRESHOLD_MS raised 30min → 4 hours in extension index.js
2. PM2 restart counter reset: delete+re-add abiba-zulip (was 18, now 0)
3. pm2-self-heal.sh alert threshold documented at 30 (implementation already)
4. Prose contract updated with historical note and threshold rationale
This commit is contained in:
root
2026-06-29 16:07:47 +00:00
parent 85ce80cf2f
commit 1f90c3dc74
2 changed files with 8 additions and 3 deletions
+7 -2
View File
@@ -28,9 +28,14 @@ description: >
- **Escalate**: If still failed after 2 retries, send Zulip DM to owner
### Rule 2: Process Restarting Too Often
- **Detect**: `pm2 status` shows restarts > 5 in the last hour
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
- **Note**: PM2 counter never decrements; only full delete+re-add resets it
- **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
- **Escalate**: Always — alert owner with restart count
- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
- **Historical fix**: Previous cycles were caused by abiba-zulip extension's
stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
This was the root cause of the 18-restart accumulation. Threshold raised
and PM2 counter reset on 2026-06-28.
## Execution