abiba-bot/prose-contracts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: cron alignment + VM→bare metal drift

Browse Source 
- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line

Live actions: gpu-monitor-server added to PM2 (persistent across reboots)
This commit is contained in:
root
2026-07-09 06:27:48 +00:00
parent c31dbad95f
commit 546ca79849
3 changed files with 8 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -2
View File
@@ -160,8 +160,8 @@ Companion shell scripts that contracts delegate to.
| File | Description | | File | Description |
|---|---| |---|---|
| `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). | | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. | | `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. | | `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
## Contract Structure ## Contract Structure
infrastructure-update.prose.md
+5 -4
View File
@@ -24,10 +24,11 @@ version: 1.1.0
Before ANY update wave: Before ANY update wave:
1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`) 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os) 2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116) 3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
4. ✅ LiteLLM health check passing 4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
5. ✅ Zulip server reachable 5. ✅ LiteLLM health check passing
6. ✅ Zulip server reachable
6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600) 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
7. ✅ Disk >20% free on all nodes 7. ✅ Disk >20% free on all nodes
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
pm2-self-heal.prose.md
+1
View File
@@ -12,6 +12,7 @@ description: >
- abiba-telegram: { status: "online", uptime: string, restarts: number } - abiba-telegram: { status: "online", uptime: string, restarts: number }
- gpu-watchdog: { status: "online", uptime: string, restarts: number } - gpu-watchdog: { status: "online", uptime: string, restarts: number }
- gpu-monitor: { status: "online", uptime: string, restarts: number }
- gitea-runner: { status: "online", uptime: string, restarts: number } - gitea-runner: { status: "online", uptime: string, restarts: number }
- last_check: timestamp - last_check: timestamp