SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Revert "infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths"
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m28s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
Details

Browse Source 
This reverts commit 09ffbf10aa.
This commit is contained in:
Agent Zero
2026-07-06 18:57:23 -04:00
parent c5cf6f1f16
commit 4cdecbc502
1 changed files with 31 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-update.prose.md
+31 -41
View File
@@ -3,7 +3,7 @@ kind: responsibility
name: infrastructure-update name: infrastructure-update
description: > description: >
Autonomous system-wide update contract covering all 5 Proxmox nodes, Autonomous system-wide update contract covering all 5 Proxmox nodes,
15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages, 15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
Docker images, and container stacks in safe waves with health checks Docker images, and container stacks in safe waves with health checks
and automatic rollback on failure. and automatic rollback on failure.
agent: abiba agent: abiba
@@ -11,7 +11,7 @@ triggers:
- on "infra update" command - on "infra update" command
- weekly (Sunday 03:00 EDT) via cron - weekly (Sunday 03:00 EDT) via cron
- on security advisory relay from Mumuni - on security advisory relay from Mumuni
version: 1.1.0 version: 1.0.0
--- ---
## Maintains ## Maintains
@@ -24,11 +24,11 @@ version: 1.1.0
Before ANY update wave: Before ANY update wave:
1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`) 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os) 2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116) 3. ✅ Docker healthy on .7, .116
4. ✅ LiteLLM health check passing 4. ✅ LiteLLM health check passing
5. ✅ Zulip server reachable 5. ✅ Zulip server reachable
6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600) 6. ✅ GPU fleet healthy (all 3 GPUs)
7. ✅ Disk >20% free on all nodes 7. ✅ Disk >20% free on all nodes
8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
@@ -38,7 +38,7 @@ Before ANY update wave:
|--------|------|---------|---------| |--------|------|---------|---------|
| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min | | CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min | | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min | | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
@@ -55,16 +55,16 @@ Before ANY update wave:
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min | | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min | | CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min | | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min | | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min | | CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min | | CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
**Verify after Wave 2:** **Verify after Wave 2:**
- All VMs/CTs running: check via Proxmox API - All CTs running: check via Proxmox API
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116) - LiteLLM healthy: `curl :4001/health/liveliness`
- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router - GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
- Zulip agents connected: check Mumuni/Tanko gateway state - Zulip agents connected: check Mumuni/Tanko gateway state
- Abiba PM2 processes online: `pm2 status` - Abiba PM2 processes online: `pm2 status`
@@ -72,24 +72,20 @@ Before ANY update wave:
| Host | Stack | Command | | Host | Stack | Command |
|------|-------|---------| |------|-------|---------|
| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` | | CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` | | CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` | | CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | | CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Komodo | `cd /opt/komodo && docker compose -f mongo.compose.yaml pull && docker compose -f mongo.compose.yaml up -d` | | CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Dockhand | `cd /opt/dockhand && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Kokoro TTS | `cd /opt/kokoro-tts && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Monitoring stack | `cd /opt/monitoring && docker compose pull && docker compose up -d` |
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` | | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
**Verify after Wave 3:** **Verify after Wave 3:**
- All containers healthy: `docker ps` on each host - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto - End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
- Zulip test: send test message to #agent-hub - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116) - Dashboard loading: `curl :3001/`
- Firecrawl test: `curl :3002/health` - Firecrawl test: `curl :3002/health`
- SearXNG test: `curl :8888` - SearXNG test: `curl storepve:8888`
## Wave 4: Proxmox Kernel Reboot (if needed) ## Wave 4: Proxmox Kernel Reboot (if needed)
@@ -115,22 +111,16 @@ If ANY verification fails:
Before Wave 1, snapshot these files: Before Wave 1, snapshot these files:
``` ```
/opt/inference-harness/docker-compose.yml (CT 116 .116) /opt/inference-harness/litellm_config.yaml (.116)
/opt/inference-harness/litellm_config.yaml (CT 116 .116) /opt/inference-harness/docker-compose.yml (.116)
/opt/monitoring/prometheus.yml (CT 116 .116) /etc/nginx/nginx.conf (.116 container)
/etc/nginx/nginx.conf (harness-nginx on CT 116) /opt/search-stack/*/docker-compose.yaml (.7)
/opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109 .7) /root/.pi/agent/extensions/config.yaml (.24)
/opt/search-stack/searxng/docker-compose.yml (VM 109 .7) /etc/systemd/system/ornith-server.service (.15)
/opt/home_stack/docker-compose.yml (VM 109 .7) /etc/systemd/system/llama-server.service (.8, .110)
/opt/audiobookshelf/docker-compose.yml (VM 109 .7)
/opt/komodo/compose.env (VM 109 .7)
/opt/komodo/mongo.compose.yaml (VM 109 .7)
/opt/dockhand/docker-compose.yaml (VM 109 .7)
/opt/kokoro-tts/docker-compose.yaml (VM 109 .7)
/root/.pi/agent/extensions/config.yaml (CT 100 .24)
``` ```
Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...` Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
## Security-Specific Updates ## Security-Specific Updates
@@ -144,8 +134,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
## Success Criteria ## Success Criteria
- [ ] All 5 PVE nodes updated, no reboot-loop - [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update - [ ] All CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117) - [ ] All 22 Docker containers healthy
- [ ] LiteLLM inference passing (syslog-auto test) - [ ] LiteLLM inference passing (syslog-auto test)
- [ ] Zulip server + all 3 agents connected - [ ] Zulip server + all 3 agents connected
- [ ] GPU fleet at full capacity (3/3) - [ ] GPU fleet at full capacity (3/3)
@@ -158,7 +148,7 @@ After completion, send Zulip DM:
``` ```
📋 Infrastructure Update — YYYY-MM-DD 📋 Infrastructure Update — YYYY-MM-DD
Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers Updated: 5 PVE nodes, 12 CTs, 22 containers
Security fixes: N CVEs patched Security fixes: N CVEs patched
Downtime: <service> <duration> Downtime: <service> <duration>
Failures: none / <details> Failures: none / <details>