SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fix/cron-alignment-jul2026
prose-contracts/infrastructure-update.prose.md
T
root 546ca79849
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
Details
fix: cron alignment + VM→bare metal drift 
- pm2-self-heal: cron 30min/daytime → 5min/24×7 (matches contract)
- pm2-self-heal: added gpu-monitor to tracked PM2 processes
- infra-monitor+control: removed daytime-only restriction (now 24×7)
- infrastructure-update: pre-flight VM 101/103 → bare metal .8/.110
- Added @reboot pm2 resurrect for persistence across reboots
- README: zulip-monitor.sh → agent-health-check.py
- Cleaned up commented-out zulip-monitor cron line

Live actions: gpu-monitor-server added to PM2 (persistent across reboots)
2026-07-09 06:27:48 +00:00

6.9 KiB
Raw Permalink Blame History

kind, name, description, agent, triggers, version
kind name description agent triggers version
responsibility infrastructure-update Autonomous system-wide update contract covering all 5 Proxmox nodes, 15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages, Docker images, and container stacks in safe waves with health checks and automatic rollback on failure. abiba
on "infra update" command
weekly (Sunday 03:00 EDT) via cron
on security advisory relay from Mumuni
1.1.0

Maintains

  • update-status: { phase, node, action, result, timestamp }
  • update-history: array of past update runs with results
  • security-state: { cve_count, last_patched, pending_updates }

Pre-Flight Checklist

Before ANY update wave:

  1. ✅ All Proxmox nodes online (GET /api2/json/nodes)
  2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
  3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
  4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
  5. ✅ LiteLLM health check passing
  6. ✅ Zulip server reachable
  7. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
  8. ✅ Disk >20% free on all nodes
  9. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

Wave 1: Storage & Infra Nodes (lowest impact)

Target Type Command Timeout
storepve (.6) Proxmox node apt update && apt upgrade -y 5 min
minipve (.12) Proxmox node apt update && apt upgrade -y 5 min
VM 109 (.7) Docker host VM apt update && apt upgrade -y 5 min
CT 106 (.65) RA-H OS apt update && apt upgrade -y 3 min
CT 117 (zulip, storepve) Zulip apt update && apt upgrade -y 3 min

Verify after Wave 1:

  • Docker healthy on .7: docker ps
  • RA-H OS MCP responding: curl 192.168.68.65:3100/mcp
  • Zulip responding: curl https://chat.sysloggh.net/api/v1/server_settings

Wave 2: Compute & Agent Nodes

Target Type Command Timeout
amdpve (.15) Proxmox node apt update && apt upgrade -y 5 min
acerpve (.9) Proxmox node apt update && apt upgrade -y 5 min
ocupve (.5) Proxmox node apt update && apt upgrade -y 5 min
CT 100 (.24) Abiba (pi) apt update && apt upgrade -y 3 min
CT 116 (.116) syslog-api (LiteLLM host) apt update && apt upgrade -y 3 min
CT 112 (tanko, amdpve) Tanko apt update && apt upgrade -y 3 min
CT 114 (mumuni, minipve) Mumuni apt update && apt upgrade -y 3 min
VM 101 (.8) llm-gpu (RTX 3090) apt update && apt upgrade -y 3 min
VM 103 (.110) ocu-llm (RTX 5070) apt update && apt upgrade -y 3 min

Verify after Wave 2:

  • All VMs/CTs running: check via Proxmox API
  • LiteLLM healthy: curl localhost:4000/health/liveliness (via CT 116)
  • GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
  • Zulip agents connected: check Mumuni/Tanko gateway state
  • Abiba PM2 processes online: pm2 status

Wave 3: Docker Image Updates

Host Stack Command
VM 109 (.7) Firecrawl cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d
VM 109 (.7) SearXNG cd /opt/search-stack/searxng && docker compose pull && docker compose up -d
VM 109 (.7) Home stack (Pulse, Stirling PDF, JDownloader 2) cd /opt/home_stack && docker compose pull && docker compose up -d
VM 109 (.7) Audiobookshelf cd /opt/audiobookshelf && docker compose pull && docker compose up -d
CT 116 (.116) Inference Harness (LiteLLM, Prometheus, Grafana) cd /opt/inference-harness && docker compose pull && docker compose up -d
CT 117 (zulip, storepve) Zulip docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1

Verify after Wave 3:

  • All containers healthy: docker ps on each host
  • End-to-end inference test: curl localhost:4000/v1/chat/completions (via CT 116) with syslog-auto
  • Zulip test: send test message to #agent-hub
  • Dashboard loading: curl localhost:3001/ (via CT 116)
  • Firecrawl test: curl :3002/
  • SearXNG test: curl :8888

Wave 4: Proxmox Kernel Reboot (if needed)

Only if [ -f /var/run/reboot-required ] on any node.

Target Action
Affected PVE node Verify all CTs/VMs migrated or stopped
reboot via PVE API
Wait 120s for node to come back
Start any stopped CTs

Rollback Protocol

If ANY verification fails:

  1. Apt rollback: Restore from Proxmox snapshot if taken, or apt install <pkg>=<old_version>
  2. Docker rollback: docker compose down && docker compose up -d (uses cached images)
  3. Config rollback: Restore from /tmp/infra-update-backup-<date>/ snapshots
  4. Escalate: Send Zulip DM with failure details if auto-rollback fails

Config Backup

Before Wave 1, snapshot these files:

/opt/inference-harness/docker-compose.yml               (CT 116 .116)
/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
/opt/monitoring/prometheus.yml                          (CT 116 .116)
/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
/opt/home_stack/docker-compose.yml                      (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
/etc/systemd/system/ornith-server.service               (amdpve .15)
/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)

Run: mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...

Security-Specific Updates

Check Command Action
CVE count apt list --upgradable 2>/dev/null | grep -i security | wc -l Report in update summary
Kernel vulns uname -r vs latest available Flag if >2 versions behind
Docker CVEs docker scout quickview or trivy image Flag critical CVEs
SSL certs openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null | openssl x509 -noout -dates Alert if <30 days

Success Criteria

  • All 5 PVE nodes updated, no reboot-loop
  • All VMs/CTs running post-update
  • All Docker containers healthy (VM 109 + CT 116 + CT 117)
  • LiteLLM inference passing (syslog-auto test)
  • Zulip server + all 3 agents connected
  • GPU fleet at full capacity (3/3)
  • Zero security CVEs remaining
  • <10 min total downtime per service

Report

After completion, send Zulip DM:

📋 Infrastructure Update — YYYY-MM-DD

Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
Security fixes: N CVEs patched
Downtime: <service> <duration>
Failures: none / <details>
Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
Reference in New Issue View Git Blame Copy Permalink