diff --git a/infrastructure-update.prose.md b/infrastructure-update.prose.md index 01667fe..a262ae6 100644 --- a/infrastructure-update.prose.md +++ b/infrastructure-update.prose.md @@ -3,7 +3,8 @@ kind: responsibility name: infrastructure-update description: > Autonomous system-wide update contract covering all 5 Proxmox nodes, - 15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages, + 15+ containers/VMs, and 5 Docker ecosystems (docker-vm .7, CT 116 .116, + CT 117, hwpve .11, NetBird VPS 72.61.0.17). Updates apt packages, Docker images, and container stacks in safe waves with health checks and automatic rollback on failure. agent: abiba @@ -11,7 +12,7 @@ triggers: - on "infra update" command - weekly (Sunday 03:00 EDT) via cron - on security advisory relay from Mumuni -version: 1.2.0 +version: 1.3.0 --- ## Maintains @@ -80,7 +81,13 @@ Before ANY update wave: | VM 109 (.7) | Home stack (Pulse, Stirling PDF) — JDownloader moved to CT 118 LXC 2026-08-01 | `cd /opt/home_stack && docker compose pull && docker compose up -d` | | VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` | | CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` | -| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` | +| CT 117 (storepve) | Zulip | `pct exec 117 -- bash -c 'cd /opt/zulip && docker compose pull && docker compose up -d'` (from storepve; compose recreates on zulip_default network) | +| CT 117 (storepve) | Jitsi | `pct exec 117 -- bash -c 'cd /opt/jitsi && docker compose pull && docker compose up -d'` (from storepve) | +| hwpve (.11) | Authentik (server, worker, postgres) | `ssh root@192.168.68.11 'cd /root && docker compose pull && docker compose up -d'` | +| NetBird VPS (72.61.0.17) | NetBird (server, dashboard, proxy, traefik, crowdsec) | `ssh root@72.61.0.17 'cd /root && docker compose pull && docker compose up -d'` | +| VM 109 (.7) | Trove test | `cd /opt/trove-test && docker compose pull && docker compose up -d` | +| VM 109 (.7) | docker-stats | `cd /opt/docker-stats && docker compose pull && docker compose up -d` | +| CT 116 (.116) | Monitoring (Grafana, Prometheus, Alertmanager, PVE exporter) | `cd /opt/monitoring && docker compose pull && docker compose up -d` | **Verify after Wave 3:** - All containers healthy: `docker ps` on each host @@ -88,7 +95,10 @@ Before ANY update wave: - MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub) - Zulip test: send test message to #agent-hub - Dashboard loading: `curl localhost:3001/` (via CT 116) -- Firecrawl test: `curl :3002/` +- Firecrawl test: `curl -X POST http://192.168.68.7:3002/v1/search -H 'Content-Type: application/json' -d '{"query":"health","limit":1}'` → `"success":true` (GET `/` returns 404 by design) +- Authentik test: `curl http://192.168.68.11:9000/` → 302 redirect to login +- NetBird test: `curl -s -o /dev/null -w '%{http_code}' https://netbird.sysloggh.net/` → 200 +- harness-litellm cold start: allow 3-5 min after recreate — reports unhealthy and :4000 refuses connections while loading config/DB, then recovers to 200 on its own (verified 2026-09-08) - SearXNG test: `curl :8888` ## Wave 4: Proxmox Kernel Reboot @@ -158,6 +168,8 @@ Before Wave 1, snapshot these files: /opt/search-stack/searxng/docker-compose.yml (VM 109 .7) /opt/home_stack/docker-compose.yml (VM 109 .7) /opt/audiobookshelf/docker-compose.yml (VM 109 .7) +/root/compose.yml (hwpve .11 — Authentik server/worker/postgres) +/root/docker-compose.yml (NetBird VPS — netbird server/dashboard/proxy, traefik, crowdsec) /root/.pi/agent/extensions/config.yaml (CT 100 .24) /etc/systemd/system/strix-server.service (amdpve .15 — strix-moe) /etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110) @@ -220,7 +232,7 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants: - [ ] All 5 PVE nodes updated, no reboot-loop - [ ] All VMs/CTs running post-update -- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117) +- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117 + hwpve .11 + NetBird VPS) - [ ] LiteLLM inference passing (syslog-auto test) - [ ] Zulip server + all 3 agents connected - [ ] GPU fleet at full capacity (3/3)