prose-contracts/infrastructure-update.prose.md

kind, name, description, agent, triggers, version

kind	name	description	agent	triggers	version
responsibility	infrastructure-update	Autonomous system-wide update contract covering all 5 Proxmox nodes, 15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages, Docker images, and container stacks in safe waves with health checks and automatic rollback on failure.	abiba	on "infra update" command weekly (Sunday 03:00 EDT) via cron on security advisory relay from Mumuni	1.2.0

Maintains

• update-status: { phase, node, action, result, timestamp }
• update-history: array of past update runs with results
• security-state: { cve_count, last_patched, pending_updates }

Pre-Flight Checklist

Before ANY update wave:

1. ✅ All Proxmox nodes online (GET /api2/json/nodes)
2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
7. ✅ Zulip server reachable
8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
9. ✅ Disk >20% free on all nodes
10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

Wave 1: Storage & Infra Nodes (lowest impact)

Target	Type	Command	Timeout
storepve (.6)	Proxmox node	apt update && apt upgrade -y	5 min
minipve (.12)	Proxmox node	apt update && apt upgrade -y	5 min
VM 109 (.7)	Docker host VM	apt update && apt upgrade -y	5 min
CT 106 (.65)	RA-H OS	apt update && apt upgrade -y	3 min
CT 117 (zulip, storepve)	Zulip	apt update && apt upgrade -y	3 min

Verify after Wave 1:

• Docker healthy on .7: docker ps
• RA-H OS MCP responding: curl 192.168.68.65:3100/mcp
• Zulip responding: curl https://chat.sysloggh.net/api/v1/server_settings

Wave 2: Compute & Agent Nodes

Target	Type	Command	Timeout
amdpve (.15)	Proxmox node	apt update && apt upgrade -y	5 min
acerpve (.9)	Proxmox node	apt update && apt upgrade -y	5 min
ocupve (.5)	Proxmox node	apt update && apt upgrade -y	5 min
CT 100 (.24)	Abiba (pi)	apt update && apt upgrade -y	3 min
CT 116 (.116)	syslog-api (LiteLLM host)	apt update && apt upgrade -y	3 min
CT 112 (tanko, amdpve)	Tanko	apt update && apt upgrade -y	3 min
CT 114 (mumuni, minipve)	Mumuni	apt update && apt upgrade -y	3 min
VM 101 (.8)	llm-gpu (RTX 3090)	apt update && apt upgrade -y	3 min
VM 103 (.110)	ocu-llm (RTX 5070)	apt update && apt upgrade -y	3 min

Verify after Wave 2:

• All VMs/CTs running: check via Proxmox API
• LiteLLM healthy: curl localhost:4000/health/liveliness (via CT 116)
• LiteLLM MCP tools: curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY" → 90 tools
• GPU servers responding: check :8080 on VM 101, VM 103; check strix-moe via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
• Zulip agents connected: check Mumuni/Tanko gateway state
• Abiba PM2 processes online: pm2 status

Wave 3: Docker Image Updates

Host	Stack	Command
VM 109 (.7)	Firecrawl	cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d
VM 109 (.7)	SearXNG	cd /opt/search-stack/searxng && docker compose pull && docker compose up -d
VM 109 (.7)	Home stack (Pulse, Stirling PDF, JDownloader 2)	cd /opt/home_stack && docker compose pull && docker compose up -d
VM 109 (.7)	Audiobookshelf	cd /opt/audiobookshelf && docker compose pull && docker compose up -d
CT 116 (.116)	Inference Harness (LiteLLM, Prometheus, Grafana)	cd /opt/inference-harness && docker compose pull && docker compose up -d
CT 117 (zulip, storepve)	Zulip	docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1

Verify after Wave 3:

• All containers healthy: docker ps on each host
• End-to-end inference test: curl localhost:4000/v1/chat/completions (via CT 116) with syslog-auto
• MCP integration test: curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY" → 90 tools (23 RA-H OS + 67 GitHub)
• Zulip test: send test message to #agent-hub
• Dashboard loading: curl localhost:3001/ (via CT 116)
• Firecrawl test: curl :3002/
• SearXNG test: curl :8888

Wave 4: Proxmox Kernel Reboot

Only if [ -f /var/run/reboot-required ] on any node.

Target	Action
Affected PVE node	Verify all CTs/VMs migrated or stopped
	reboot via PVE API (or systemctl reboot -f if dbus fails)
	Wait 120s for node to come back
	Start any stopped CTs

Post-reboot sweep (known gaps)

After every node reboot, run these checks:

1. CT auto-start sweep — LXC containers sometimes don't start despite onboot: 1. Check every CT on the rebooted node and start any left stopped:

   pct list | awk '/stopped/{print $1}' | xargs -I{} pct start {}
   

   Known cases: scottdenya (CT 115 on amdpve), authentik (CT 104 on minipve).

2. Zulip recovery — When docker-vm or storepve reboots, the Zulip main container loses its Docker network assignment (SIGKILL during storage outage detaches it from zulip_default network). Run:

   ssh root@192.168.68.19 'docker rm -f zulip-zulip-1 && cd /opt/zulip && docker compose up -d'
   

   The compose restart recreates the container on the correct network.

3. docker-vm Docker daemon — After reboot, Docker can take 3-4 minutes to become active. The docker-proxy for Pulse (port 7655) starts early, so Pulse is accessible before docker ps reports ready. Wait for Docker before checking other stacks.

VPS ↔ docker-vm tunnel

After any VPS or docker-vm reboot, verify the dedicated WireGuard tunnel:

ssh root@72.61.0.17 'wg show wg1' | grep "latest handshake"
# If no handshake in >60s:
ssh root@72.61.0.17 'wg-quick up wg1'

The tunnel uses PersistentKeepalive=25 and is systemd-enabled, but should be verified after a reboot.

Rollback Protocol

If ANY verification fails:

1. Apt rollback: Restore from Proxmox snapshot if taken, or apt install <pkg>=<old_version>
2. Docker rollback: docker compose down && docker compose up -d (uses cached images)
3. Config rollback: Restore from /tmp/infra-update-backup-<date>/ snapshots
4. Escalate: Send Zulip DM with failure details if auto-rollback fails

Config Backup

Before Wave 1, snapshot these files:

/opt/inference-harness/docker-compose.yml               (CT 116 .116) ⚡ contains MCP_SERVER env vars
/opt/inference-harness/litellm_config.yaml              (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
/opt/monitoring/prometheus.yml                          (CT 116 .116)
/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
/opt/home_stack/docker-compose.yml                      (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
/etc/systemd/system/ornith-server.service               (amdpve .15 — strix-moe)
/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
# Hermes agent configs (key enforcement — 2026-07-10)
/root/.hermes/config.yaml                               (Mumuni CT 114, Tanko CT 112, etc.)
/root/.config/systemd/user/hermes-gateway.service       (Mumuni CT 114 — EnvironmentFile fixed)
/etc/environment                                        (Mumuni CT 114 — LITELLM_API_KEY)

MCP Gateway (2026-07-10)

LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.

Configuration

litellm_config.yaml (/opt/inference-harness/litellm_config.yaml):

mcp_servers:
  ra_h_os:
    url: "http://192.168.68.65:3100/mcp"
    transport: "http"
    auth_type: "none"

docker-compose.yml env vars:

- MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
- MCP_SERVER_RAHOS_TRANSPORT=http

Access

Key	MCP Access
Master key	✅ Full — 90 tools (vault-injected)
Agent keys (mumuni, tanko, etc.)	❌ Per-key grants not supported in v1.90.0-rc.1

Known Limitations

• Per-key MCP server grants not functional — only master key has access
• Responses API (/v1/responses) with MCP tools broken on llama.cpp backends
• HTTP 307 redirect on /mcp → use /mcp/ (trailing slash) or /mcp-rest/ endpoints
• api_mode: responses in Hermes appends /v1/responses to base_url → base_url must end at /v1, never /responses (double-path bug)

Migration Path

When LiteLLM is upgraded to a version supporting per-key MCP grants:

1. Grant agent keys mcp_servers: ["ra_h_os"]
2. Update Hermes mcp_servers.ra-h-os.url from http://192.168.68.65:3100/mcp → http://192.168.68.116:4000/mcp/
3. Add headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"} to MCP config

Security-Specific Updates

Check	Command	Action
CVE count	apt list --upgradable 2>/dev/null | grep -i security | wc -l	Report in update summary
Kernel vulns	uname -r vs latest available	Flag if >2 versions behind
Docker CVEs	docker scout quickview or trivy image	Flag critical CVEs
SSL certs	openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null | openssl x509 -noout -dates	Alert if <30 days

Success Criteria

• All 5 PVE nodes updated, no reboot-loop
• All VMs/CTs running post-update
• All Docker containers healthy (VM 109 + CT 116 + CT 117)
• LiteLLM inference passing (syslog-auto test)
• Zulip server + all 3 agents connected
• GPU fleet at full capacity (3/3)
• LiteLLM MCP gateway healthy (90 tools via master key)
• Zero security CVEs remaining
• <10 min total downtime per service

Report

After completion, send Zulip DM:

📋 Infrastructure Update — YYYY-MM-DD

Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
Security fixes: N CVEs patched
Downtime: <service> <duration>
Failures: none / <details>
Configs backed up: /tmp/infra-update-backup-YYYYMMDD/