fix(pm2-self-heal): document permanent stuck-detection fix, reset PM2 counter

Root cause: abiba-zulip extension had STUCK_THRESHOLD_MS=30min.
After 30min of inactivity (overnight), it reported stuck=True, which
triggered the health monitor to restart it — cycling restarts up to 18.

Permanent fixes:
1. STUCK_THRESHOLD_MS raised 30min → 4 hours in extension index.js
2. PM2 restart counter reset: delete+re-add abiba-zulip (was 18, now 0)
3. pm2-self-heal.sh alert threshold documented at 30 (implementation already)
4. Prose contract updated with historical note and threshold rationale
This commit is contained in:
root
2026-06-29 16:07:47 +00:00
parent 85ce80cf2f
commit 1f90c3dc74
2 changed files with 8 additions and 3 deletions
+7 -2
View File
@@ -28,9 +28,14 @@ description: >
- **Escalate**: If still failed after 2 retries, send Zulip DM to owner
### Rule 2: Process Restarting Too Often
- **Detect**: `pm2 status` shows restarts > 5 in the last hour
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
- **Note**: PM2 counter never decrements; only full delete+re-add resets it
- **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
- **Escalate**: Always — alert owner with restart count
- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
- **Historical fix**: Previous cycles were caused by abiba-zulip extension's
stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
This was the root cause of the 18-restart accumulation. Threshold raised
and PM2 counter reset on 2026-06-28.
## Execution
+1 -1
View File
@@ -55,7 +55,7 @@ ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs)
if [ "$ZUL_STATUS" != "online" ]; then
ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n"
ALERTS="${ALERTS} NOT auto-restarting (runs this contract)\n"
elif [ "$ZUL_RESTARTS" -gt 8 ] 2>/dev/null; then
elif [ "$ZUL_RESTARTS" -gt 30 ] 2>/dev/null; then
ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n"
fi