Merge pull request 'fix: cron alignment + VM→bare metal drift' (#11) from fix/cron-alignment-jul2026 into master
This commit is contained in:
@@ -160,8 +160,8 @@ Companion shell scripts that contracts delegate to.
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
|
||||
| `zulip-monitor.sh` | Monitors Zulip bot health and message flow. |
|
||||
| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes. |
|
||||
| `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
|
||||
| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |
|
||||
|
||||
## Contract Structure
|
||||
|
||||
|
||||
@@ -24,10 +24,11 @@ version: 1.1.0
|
||||
|
||||
Before ANY update wave:
|
||||
1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
|
||||
2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
|
||||
3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
|
||||
4. ✅ LiteLLM health check passing
|
||||
5. ✅ Zulip server reachable
|
||||
2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
|
||||
3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
|
||||
4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
|
||||
5. ✅ LiteLLM health check passing
|
||||
6. ✅ Zulip server reachable
|
||||
6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
|
||||
7. ✅ Disk >20% free on all nodes
|
||||
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
|
||||
|
||||
@@ -12,6 +12,7 @@ description: >
|
||||
|
||||
- abiba-telegram: { status: "online", uptime: string, restarts: number }
|
||||
- gpu-watchdog: { status: "online", uptime: string, restarts: number }
|
||||
- gpu-monitor: { status: "online", uptime: string, restarts: number }
|
||||
- gitea-runner: { status: "online", uptime: string, restarts: number }
|
||||
- last_check: timestamp
|
||||
|
||||
|
||||
Reference in New Issue
Block a user