prose-contracts/pm2-self-heal.prose.md at fix/gpu-dense-docs

Files

T

root 6195b59317 Item 1: Koby report-only — encode HARD RULE in contracts (2026-08-17)

- hermes-config-template.prose.md: Add Rule 17 — Koby is never repaired, full stop
- hermes-agent-baseline.prose.md: Document Koby report-only posture
- contract-registry.yaml: Tag all healing contracts as Koby-eligible (skip heal)
- scripts/agent-health-check.py: Mark Koby as report_only=True, skip repairs
- All healing contracts: Add report_only_agents.koby marker

Captain-approved ship via no-mistakes. PR auto-merges green.

2026-08-20 07:42:22 +00:00

3.5 KiB

Raw Permalink Blame History

kind, name, description, report_only_agents

kind

name

description

report_only_agents

responsibility

pm2-self-heal

Monitors critical PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. Logs every action to the knowledge graph and alerts the owner via Zulip DM on failures. Abiba-zulip is the live Zulip bridge and may be restarted; alert owner on failure.

koby

Maintains

abiba-telegram: { status: "online", uptime: string, restarts: number }
abiba-zulip: { status: "online", uptime: string, restarts: number }
gpu-monitor: { status: "online", uptime: string, restarts: number } (systemd-managed, PM2-tracked)
gitea-runner: { status: "online", uptime: string, restarts: number }
last_check: timestamp

Status (2026-08-03): abiba-zulip fully restored — live Zulip bridge, heartbeating, monitored. gpu-monitor runs via systemd (stable); PM2 tracks it for status reporting only. gpu-watchdog retired from PM2.

Continuity

Self-driven: check every 300 seconds (5 min)
Also wakes on user request (prose run pm2-self-heal)

Remediation Rules

Rule 1: Process Stopped or Errored

Detect: pm2 status shows "stopped" or "errored" for a named process
Fix: pm2 restart <name>
Verify: Re-check status after 5 seconds
Escalate: If still failed after 2 retries, send Zulip DM to owner

Rule 2: Process Restarting Too Often

Detect: pm2 status shows restarts > 30 (cumulative lifetime counter)
Note: PM2 counter never decrements; only full delete+re-add resets it
Fix: pm2 delete <name> && pm2 start <ecosystem> --only <name>
Escalate: Only when restarts > 30 — alerts to Zulip DM
Historical fix: Previous cycles were caused by abiba-zulip extension's stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2). This was the root cause of the 18-restart accumulation. Threshold raised and PM2 counter reset on 2026-06-28.

Execution

Check PM2 status — Run pm2 status --no-color and parse the table (5th data column = PID, 8th = restarts, 9th = status)
Check abiba-telegram:
- If status is "online" → pass
- If status is "stopped" or "errored" → apply Rule 1
- If restarts > 5 → alert owner
Check abiba-zulip (live Zulip bridge, heartbeating):
- If status is "online" → pass, log restarts count
- If status is "stopped" or "errored" → restart (pm2 restart abiba-zulip — fully restored)
- If restarts > 5 in last hour → alert owner with full diagnostics
Log results — Append to SyslogSolution/health-logs/pm2/{timestamp}.md in Gitea (not knowledge graph — hard rule)
Alert — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
Wait 5 min → repeat from step 1

Example Output (when healthy)

{
  "run_id": "pm2-heal-001",
  "checked_at": "2026-06-26T15:00:00Z",
  "processes": {
    "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 },
    "abiba-telegram": { "status": "online", "uptime": "48h", "restarts": 0 }
  },
  "actions_taken": [],
  "overall": "healthy"
}

Example Output (when fixed)

{
  "processes": {
    "abiba-telegram": { "status": "errored → restarted → online" },
    "abiba-zulip": { "status": "online", "uptime": "2h", "restarts": 0 }
  },
  "actions_taken": [
    { "target": "abiba-telegram", "action": "pm2 restart", "result": "online" }
  ],
  "overall": "fixed"
}

3.5 KiB Raw Permalink Blame History