From 546ca7984913371f7e2e18229d88bb044ab26b19 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 9 Jul 2026 06:27:48 +0000 Subject: [PATCH] =?UTF-8?q?fix:=20cron=20alignment=20+=20VM=E2=86=92bare?= =?UTF-8?q?=20metal=20drift?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract) - pm2-self-heal: added gpu-monitor to tracked PM2 processes - infra-monitor+control: removed daytime-only restriction (now 24×7) - infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110 - Added @reboot pm2 resurrect for persistence across reboots - README: zulip-monitor.sh → agent-health-check.py - Cleaned up commented-out zulip-monitor cron line Live actions: gpu-monitor-server added to PM2 (persistent across reboots) --- README.md | 4 ++-- infrastructure-update.prose.md | 9 +++++---- pm2-self-heal.prose.md | 1 + 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 647ecb9..1936e9b 100644 --- a/README.md +++ b/README.md @@ -160,8 +160,8 @@ Companion shell scripts that contracts delegate to. | File | Description | |---|---| | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). | -| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. | -| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. | +| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). | +| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. | ## Contract Structure diff --git a/infrastructure-update.prose.md b/infrastructure-update.prose.md index 156b84c..de44c37 100644 --- a/infrastructure-update.prose.md +++ b/infrastructure-update.prose.md @@ -24,10 +24,11 @@ version: 1.1.0 Before ANY update wave: 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`) -2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os) -3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116) -4. ✅ LiteLLM health check passing -5. ✅ Zulip server reachable +2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os) +3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo) +4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116) +5. ✅ LiteLLM health check passing +6. ✅ Zulip server reachable 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600) 7. ✅ Disk >20% free on all nodes 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index 23026b6..1eb1fb5 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -12,6 +12,7 @@ description: > - abiba-telegram: { status: "online", uptime: string, restarts: number } - gpu-watchdog: { status: "online", uptime: string, restarts: number } +- gpu-monitor: { status: "online", uptime: string, restarts: number } - gitea-runner: { status: "online", uptime: string, restarts: number } - last_check: timestamp