diff --git a/infrastructure-update.prose.md b/infrastructure-update.prose.md index 98b7f5a..1922544 100644 --- a/infrastructure-update.prose.md +++ b/infrastructure-update.prose.md @@ -3,7 +3,7 @@ kind: responsibility name: infrastructure-update description: > Autonomous system-wide update contract covering all 5 Proxmox nodes, - 15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages, + 15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages, Docker images, and container stacks in safe waves with health checks and automatic rollback on failure. agent: abiba @@ -11,7 +11,7 @@ triggers: - on "infra update" command - weekly (Sunday 03:00 EDT) via cron - on security advisory relay from Mumuni -version: 1.1.0 +version: 1.0.0 --- ## Maintains @@ -24,11 +24,11 @@ version: 1.1.0 Before ANY update wave: 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`) -2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os) -3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116) +2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123) +3. ✅ Docker healthy on .7, .116 4. ✅ LiteLLM health check passing 5. ✅ Zulip server reachable -6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600) +6. ✅ GPU fleet healthy (all 3 GPUs) 7. ✅ Disk >20% free on all nodes 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files) @@ -38,7 +38,7 @@ Before ANY update wave: |--------|------|---------|---------| | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min | -| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min | +| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min | | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min | | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min | @@ -55,16 +55,16 @@ Before ANY update wave: | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min | -| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min | +| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min | | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min | | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min | -| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min | -| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min | +| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min | +| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min | **Verify after Wave 2:** -- All VMs/CTs running: check via Proxmox API -- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116) -- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router +- All CTs running: check via Proxmox API +- LiteLLM healthy: `curl :4001/health/liveliness` +- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only) - Zulip agents connected: check Mumuni/Tanko gateway state - Abiba PM2 processes online: `pm2 status` @@ -72,24 +72,20 @@ Before ANY update wave: | Host | Stack | Command | |------|-------|---------| -| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` | -| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` | -| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` | -| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | -| VM 109 (.7) | Komodo | `cd /opt/komodo && docker compose -f mongo.compose.yaml pull && docker compose -f mongo.compose.yaml up -d` | -| VM 109 (.7) | Dockhand | `cd /opt/dockhand && docker compose pull && docker compose up -d` | -| VM 109 (.7) | Kokoro TTS | `cd /opt/kokoro-tts && docker compose pull && docker compose up -d` | -| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` | -| CT 116 (.116) | Monitoring stack | `cd /opt/monitoring && docker compose pull && docker compose up -d` | +| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` | +| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` | +| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` | +| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | +| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` | | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` | **Verify after Wave 3:** - All containers healthy: `docker ps` on each host -- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto +- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto - Zulip test: send test message to #agent-hub -- Dashboard loading: `curl localhost:3001/` (via CT 116) +- Dashboard loading: `curl :3001/` - Firecrawl test: `curl :3002/health` -- SearXNG test: `curl :8888` +- SearXNG test: `curl storepve:8888` ## Wave 4: Proxmox Kernel Reboot (if needed) @@ -115,22 +111,16 @@ If ANY verification fails: Before Wave 1, snapshot these files: ``` -/opt/inference-harness/docker-compose.yml (CT 116 .116) -/opt/inference-harness/litellm_config.yaml (CT 116 .116) -/opt/monitoring/prometheus.yml (CT 116 .116) -/etc/nginx/nginx.conf (harness-nginx on CT 116) -/opt/search-stack/firecrawl-source/docker-compose.yaml (VM 109 .7) -/opt/search-stack/searxng/docker-compose.yml (VM 109 .7) -/opt/home_stack/docker-compose.yml (VM 109 .7) -/opt/audiobookshelf/docker-compose.yml (VM 109 .7) -/opt/komodo/compose.env (VM 109 .7) -/opt/komodo/mongo.compose.yaml (VM 109 .7) -/opt/dockhand/docker-compose.yaml (VM 109 .7) -/opt/kokoro-tts/docker-compose.yaml (VM 109 .7) -/root/.pi/agent/extensions/config.yaml (CT 100 .24) +/opt/inference-harness/litellm_config.yaml (.116) +/opt/inference-harness/docker-compose.yml (.116) +/etc/nginx/nginx.conf (.116 container) +/opt/search-stack/*/docker-compose.yaml (.7) +/root/.pi/agent/extensions/config.yaml (.24) +/etc/systemd/system/ornith-server.service (.15) +/etc/systemd/system/llama-server.service (.8, .110) ``` -Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...` +Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...` ## Security-Specific Updates @@ -144,8 +134,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...` ## Success Criteria - [ ] All 5 PVE nodes updated, no reboot-loop -- [ ] All VMs/CTs running post-update -- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117) +- [ ] All CTs running post-update +- [ ] All 22 Docker containers healthy - [ ] LiteLLM inference passing (syslog-auto test) - [ ] Zulip server + all 3 agents connected - [ ] GPU fleet at full capacity (3/3) @@ -158,7 +148,7 @@ After completion, send Zulip DM: ``` 📋 Infrastructure Update — YYYY-MM-DD -Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers +Updated: 5 PVE nodes, 12 CTs, 22 containers Security fixes: N CVEs patched Downtime: Failures: none /