prose: review, fix, and consolidate infrastructure contracts

- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00
parent 3077fda0b7
commit 253e19f680
14 changed files with 628 additions and 213 deletions
@@ -0,0 +1,156 @@
+---
+kind: responsibility
+name: infrastructure-update
+description: >
+  Autonomous system-wide update contract covering all 5 Proxmox nodes,
+  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
+  Docker images, and container stacks in safe waves with health checks
+  and automatic rollback on failure.
+agent: abiba
+triggers:
+  - on "infra update" command
+  - weekly (Sunday 03:00 EDT) via cron
+  - on security advisory relay from Mumuni
+version: 1.0.0
+---
+
+## Maintains
+
+- update-status: { phase, node, action, result, timestamp }
+- update-history: array of past update runs with results
+- security-state: { cve_count, last_patched, pending_updates }
+
+## Pre-Flight Checklist
+
+Before ANY update wave:
+1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
+2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
+3. ✅ Docker healthy on .7, .116
+4. ✅ LiteLLM health check passing
+5. ✅ Zulip server reachable
+6. ✅ GPU fleet healthy (all 3 GPUs)
+7. ✅ Disk >20% free on all nodes
+8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)
+
+## Wave 1: Storage & Infra Nodes (lowest impact)
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
+| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
+| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 1:**
+- Docker healthy on .7: `docker ps`
+- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
+- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`
+
+## Wave 2: Compute & Agent Nodes
+
+| Target | Type | Command | Timeout |
+|--------|------|---------|---------|
+| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
+| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
+| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
+| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
+| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
+
+**Verify after Wave 2:**
+- All CTs running: check via Proxmox API
+- LiteLLM healthy: `curl :4001/health/liveliness`
+- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- Zulip agents connected: check Mumuni/Tanko gateway state
+- Abiba PM2 processes online: `pm2 status`
+
+## Wave 3: Docker Image Updates
+
+| Host | Stack | Command |
+|------|-------|---------|
+| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |
+
+**Verify after Wave 3:**
+- All containers healthy: `docker ps` on each host
+- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
+- Zulip test: send test message to #agent-hub
+- Dashboard loading: `curl :3001/`
+- Firecrawl test: `curl :3002/health`
+- SearXNG test: `curl storepve:8888`
+
+## Wave 4: Proxmox Kernel Reboot (if needed)
+
+Only if `[ -f /var/run/reboot-required ]` on any node.
+
+| Target | Action |
+|--------|--------|
+| Affected PVE node | Verify all CTs/VMs migrated or stopped |
+| | `reboot` via PVE API |
+| | Wait 120s for node to come back |
+| | Start any stopped CTs |
+
+## Rollback Protocol
+
+If ANY verification fails:
+
+1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
+2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
+3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
+4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails
+
+## Config Backup
+
+Before Wave 1, snapshot these files:
+```
+/opt/inference-harness/litellm_config.yaml        (.116)
+/opt/inference-harness/docker-compose.yml          (.116)
+/etc/nginx/nginx.conf                              (.116 container)
+/opt/search-stack/*/docker-compose.yaml            (.7)
+/root/.pi/agent/extensions/config.yaml             (.24)
+/etc/systemd/system/ornith-server.service          (.15)
+/etc/systemd/system/llama-server.service           (.8, .110)
+```
+
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
+
+## Security-Specific Updates
+
+| Check | Command | Action |
+|-------|---------|--------|
+| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
+| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
+| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
+| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |
+
+## Success Criteria
+
+- [ ] All 5 PVE nodes updated, no reboot-loop
+- [ ] All CTs running post-update
+- [ ] All 22 Docker containers healthy
+- [ ] LiteLLM inference passing (syslog-auto test)
+- [ ] Zulip server + all 3 agents connected
+- [ ] GPU fleet at full capacity (3/3)
+- [ ] Zero security CVEs remaining
+- [ ] <10 min total downtime per service
+
+## Report
+
+After completion, send Zulip DM:
+```
+📋 Infrastructure Update — YYYY-MM-DD
+
+Updated: 5 PVE nodes, 12 CTs, 22 containers
+Security fixes: N CVEs patched
+Downtime: <service> <duration>
+Failures: none / <details>
+Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
+```