diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index 1878a2a..f18f9e1 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -28,9 +28,14 @@ description: > - **Escalate**: If still failed after 2 retries, send Zulip DM to owner ### Rule 2: Process Restarting Too Often -- **Detect**: `pm2 status` shows restarts > 5 in the last hour +- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter) +- **Note**: PM2 counter never decrements; only full delete+re-add resets it - **Fix**: `pm2 delete && pm2 start --only ` -- **Escalate**: Always — alert owner with restart count +- **Escalate**: Only when restarts > 30 — alerts to Zulip DM +- **Historical fix**: Previous cycles were caused by abiba-zulip extension's + stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2). + This was the root cause of the 18-restart accumulation. Threshold raised + and PM2 counter reset on 2026-06-28. ## Execution diff --git a/pm2-self-heal.sh b/pm2-self-heal.sh index f56638f..49b7806 100755 --- a/pm2-self-heal.sh +++ b/pm2-self-heal.sh @@ -55,7 +55,7 @@ ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs) if [ "$ZUL_STATUS" != "online" ]; then ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n" ALERTS="${ALERTS} NOT auto-restarting (runs this contract)\n" -elif [ "$ZUL_RESTARTS" -gt 8 ] 2>/dev/null; then +elif [ "$ZUL_RESTARTS" -gt 30 ] 2>/dev/null; then ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n" fi