SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths

Browse Source
This commit is contained in:
Agent Zero
2026-07-06 18:05:16 -04:00
parent 5aa93117de
commit 09ffbf10aa
1 changed files with 41 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-update.prose.md
+41 -31
View File
@@ -3,7 +3,7 @@ kind: responsibility
name: infrastructure-update name: infrastructure-update
description: > description: >
Autonomous system-wide update contract covering all 5 Proxmox nodes, Autonomous system-wide update contract covering all 5 Proxmox nodes,
15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages, 15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
Docker images, and container stacks in safe waves with health checks Docker images, and container stacks in safe waves with health checks
and automatic rollback on failure. and automatic rollback on failure.
agent: abiba agent: abiba
@@ -11,7 +11,7 @@ triggers:
- on "infra update" command - on "infra update" command
- weekly (Sunday 03:00 EDT) via cron - weekly (Sunday 03:00 EDT) via cron
- on security advisory relay from Mumuni - on security advisory relay from Mumuni
version: 1.0.0 version: 1.1.0
--- ---
## Maintains ## Maintains
@@ -24,11 +24,11 @@ version: 1.0.0
Before ANY update wave: Before ANY update wave:
1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`) 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123) 2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
3. ✅ Docker healthy on .7, .116 3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
4. ✅ LiteLLM health check passing 4. ✅ LiteLLM health check passing
5. ✅ Zulip server reachable 5. ✅ Zulip server reachable
6. ✅ GPU fleet healthy (all 3 GPUs) 6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
7. ✅ Disk >20% free on all nodes 7. ✅ Disk >20% free on all nodes
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
@@ -38,7 +38,7 @@ Before ANY update wave:
|--------|------|---------|---------| |--------|------|---------|---------|
| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min | | VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min | | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min | | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
@@ -55,16 +55,16 @@ Before ANY update wave:
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min | | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min | | CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min | | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min | | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min | | VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min | | VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
**Verify after Wave 2:** **Verify after Wave 2:**
- All CTs running: check via Proxmox API - All VMs/CTs running: check via Proxmox API
- LiteLLM healthy: `curl :4001/health/liveliness` - LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only) - GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router
- Zulip agents connected: check Mumuni/Tanko gateway state - Zulip agents connected: check Mumuni/Tanko gateway state
- Abiba PM2 processes online: `pm2 status` - Abiba PM2 processes online: `pm2 status`
@@ -72,20 +72,24 @@ Before ANY update wave:
| Host | Stack | Command | | Host | Stack | Command |
|------|-------|---------| |------|-------|---------|
| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` | | VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` | | VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` | | VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | | VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` | | VM 109 (.7) | Komodo | `cd /opt/komodo && docker compose -f mongo.compose.yaml pull && docker compose -f mongo.compose.yaml up -d` |
| VM 109 (.7) | Dockhand | `cd /opt/dockhand && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Kokoro TTS | `cd /opt/kokoro-tts && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Monitoring stack | `cd /opt/monitoring && docker compose pull && docker compose up -d` |
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` | | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
**Verify after Wave 3:** **Verify after Wave 3:**
- All containers healthy: `docker ps` on each host - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto - End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
- Zulip test: send test message to #agent-hub - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl :3001/` - Dashboard loading: `curl localhost:3001/` (via CT 116)
- Firecrawl test: `curl :3002/health` - Firecrawl test: `curl :3002/health`
- SearXNG test: `curl storepve:8888` - SearXNG test: `curl :8888`
## Wave 4: Proxmox Kernel Reboot (if needed) ## Wave 4: Proxmox Kernel Reboot (if needed)
@@ -111,16 +115,22 @@ If ANY verification fails:
Before Wave 1, snapshot these files: Before Wave 1, snapshot these files:
``` ```
/opt/inference-harness/litellm_config.yaml (.116) /opt/inference-harness/docker-compose.yml (CT 116 .116)
/opt/inference-harness/docker-compose.yml (.116) /opt/inference-harness/litellm_config.yaml (CT 116 .116)
/etc/nginx/nginx.conf (.116 container) /opt/monitoring/prometheus.yml (CT 116 .116)
/opt/search-stack/*/docker-compose.yaml (.7) /etc/nginx/nginx.conf (harness-nginx on CT 116)
/root/.pi/agent/extensions/config.yaml (.24) /opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109 .7)
/etc/systemd/system/ornith-server.service (.15) /opt/search-stack/searxng/docker-compose.yml (VM 109 .7)
/etc/systemd/system/llama-server.service (.8, .110) /opt/home_stack/docker-compose.yml (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml (VM 109 .7)
/opt/komodo/compose.env (VM 109 .7)
/opt/komodo/mongo.compose.yaml (VM 109 .7)
/opt/dockhand/docker-compose.yaml (VM 109 .7)
/opt/kokoro-tts/docker-compose.yaml (VM 109 .7)
/root/.pi/agent/extensions/config.yaml (CT 100 .24)
``` ```
Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...` Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
## Security-Specific Updates ## Security-Specific Updates
@@ -134,8 +144,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
## Success Criteria ## Success Criteria
- [ ] All 5 PVE nodes updated, no reboot-loop - [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All CTs running post-update - [ ] All VMs/CTs running post-update
- [ ] All 22 Docker containers healthy - [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
- [ ] LiteLLM inference passing (syslog-auto test) - [ ] LiteLLM inference passing (syslog-auto test)
- [ ] Zulip server + all 3 agents connected - [ ] Zulip server + all 3 agents connected
- [ ] GPU fleet at full capacity (3/3) - [ ] GPU fleet at full capacity (3/3)
@@ -148,7 +158,7 @@ After completion, send Zulip DM:
``` ```
📋 Infrastructure Update — YYYY-MM-DD 📋 Infrastructure Update — YYYY-MM-DD
Updated: 5 PVE nodes, 12 CTs, 22 containers Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
Security fixes: N CVEs patched Security fixes: N CVEs patched
Downtime: <service> <duration> Downtime: <service> <duration>
Failures: none / <details> Failures: none / <details>