fix: ship pm2-self-heal crash-loop guard + align gpu-dense docs to Qwen3.8-27B
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- pm2-self-heal.sh: restart abiba-telegram when TEL_RESTARTS > 1000 even if 'online' (catches quiet crash-loops like the 10k-restarts spoton incident); alert includes count. - pm2-self-heal.prose.md: document the crash-loop guard under Rule 2. - gpu-fleet.prose.md / gpu-self-heal.prose.md: gpu-dense is Qwen3.8-27B-Uncensored-Q4_K_M (~16.8GB, alias qwen3.6-27B-code for LiteLLM routing), not SmartCode-Fable-5 (verified live on .8:8080 Aug 16).
This commit is contained in:
@@ -41,10 +41,15 @@ description: >
|
||||
- **Verify**: Re-check status after 5 seconds
|
||||
- **Escalate**: If still failed after 2 retries, send Zulip DM to owner
|
||||
|
||||
### Rule 2: Process Restarting Too Often
|
||||
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
|
||||
### Rule 2: Process Restarting Too Often (crash-loop guard)
|
||||
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter) **or** a
|
||||
process reporting a restart count > 1000 while showing "online" (a crash-loop mask)
|
||||
- **Note**: PM2 counter never decrements; only full delete+re-add resets it
|
||||
- **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
|
||||
- **Script guard (2026-08-16)**: `scripts/pm2-self-heal.sh` now restarts `abiba-telegram`
|
||||
when `TEL_RESTARTS > 1000` even if the process reports "online" — catches a quiet
|
||||
crash-loop that never toggles status to "stopped"/"errored" (e.g. the 10k-restarts
|
||||
spoton incident). Alerts include the restart count.
|
||||
- **Escalate**: Only when restarts > 30 — alerts to Zulip DM
|
||||
- **Historical fix**: Previous cycles were caused by abiba-zulip extension's
|
||||
stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
|
||||
|
||||
Reference in New Issue
Block a user