prose-contracts/infrastructure-update.prose.md

---
kind: responsibility
name: infrastructure-update
description: >
  Autonomous system-wide update contract covering all 5 Proxmox nodes,
  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
agent: abiba
triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
version: 1.2.0
---

## Maintains

- update-status: { phase, node, action, result, timestamp }
- update-history: array of past update runs with results
- security-state: { cve_count, last_patched, pending_updates }

## Pre-Flight Checklist

Before ANY update wave:
1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
2. ✅ All critical VMs/CTs running (VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
3. ✅ GPU bare-metal hosts reachable: .8 (RTX 3090), .110 (RTX 5070), .15 (Strix Halo)
4. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
5. ✅ LiteLLM health check passing (port 4000, /mcp-rest/tools/list with master key)
6. ✅ LiteLLM MCP gateway serving RA-H OS tools (90 tools)
7. ✅ Zulip server reachable
8. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
9. ✅ Disk >20% free on all nodes
10. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

## Wave 1: Storage & Infra Nodes (lowest impact)

| Target | Type | Command | Timeout |
|--------|------|---------|---------|
| storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
| CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
| CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |

**Verify after Wave 1:**
- Docker healthy on .7: `docker ps`
- RA-H OS MCP responding: `curl 192.168.68.65:3100/mcp`
- Zulip responding: `curl https://chat.sysloggh.net/api/v1/server_settings`

## Wave 2: Compute & Agent Nodes

| Target | Type | Command | Timeout |
|--------|------|---------|---------|
| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |

**Verify after Wave 2:**
- All VMs/CTs running: check via Proxmox API
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
- LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools
- GPU servers responding: check :8080 on VM 101, VM 103; check strix-moe via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
- Zulip agents connected: check Mumuni/Tanko gateway state
- Abiba PM2 processes online: `pm2 status`

## Wave 3: Docker Image Updates

| Host | Stack | Command |
|------|-------|---------|
| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
| CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |

**Verify after Wave 3:**
- All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
- MCP integration test: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools (23 RA-H OS + 67 GitHub)
- Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116)
- Firecrawl test: `curl :3002/`
- SearXNG test: `curl :8888`

## Wave 4: Proxmox Kernel Reboot

Only if `[ -f /var/run/reboot-required ]` on any node.

| Target | Action |
|--------|--------|
| Affected PVE node | Verify all CTs/VMs migrated or stopped |
| | `reboot` via PVE API (or `systemctl reboot -f` if dbus fails) |
| | Wait 120s for node to come back |
| | Start any stopped CTs |

### Post-reboot sweep (known gaps)

After every node reboot, run these checks:

1. **CT auto-start sweep** — LXC containers sometimes don't start despite
   `onboot: 1`. Check every CT on the rebooted node and start any left stopped:
   ```bash
   pct list | awk '/stopped/{print $1}' | xargs -I{} pct start {}
   ```
   Known cases: scottdenya (CT 115 on amdpve), authentik (CT 104 on minipve).

2. **Zulip recovery** — When docker-vm or storepve reboots, the Zulip main
   container loses its Docker network assignment (SIGKILL during storage
   outage detaches it from `zulip_default` network). Run:
   ```bash
   ssh root@192.168.68.19 'docker rm -f zulip-zulip-1 && cd /opt/zulip && docker compose up -d'
   ```
   The compose restart recreates the container on the correct network.

3. **docker-vm Docker daemon** — After reboot, Docker can take 3-4 minutes
   to become `active`. The docker-proxy for Pulse (port 7655) starts early,
   so Pulse is accessible before `docker ps` reports ready. Wait for Docker
   before checking other stacks.

### VPS ↔ docker-vm tunnel

After any VPS or docker-vm reboot, verify the dedicated WireGuard tunnel:
```bash
ssh root@72.61.0.17 'wg show wg1' | grep "latest handshake"
# If no handshake in >60s:
ssh root@72.61.0.17 'wg-quick up wg1'
```
The tunnel uses PersistentKeepalive=25 and is systemd-enabled, but should
be verified after a reboot.

## Rollback Protocol

If ANY verification fails:

1. **Apt rollback**: Restore from Proxmox snapshot if taken, or `apt install <pkg>=<old_version>`
2. **Docker rollback**: `docker compose down && docker compose up -d` (uses cached images)
3. **Config rollback**: Restore from `/tmp/infra-update-backup-<date>/` snapshots
4. **Escalate**: Send Zulip DM with failure details if auto-rollback fails

## Config Backup

Before Wave 1, snapshot these files:
```
/opt/inference-harness/docker-compose.yml               (CT 116 .116) ⚡ contains MCP_SERVER env vars
/opt/inference-harness/litellm_config.yaml              (CT 116 .116) ⚡ contains mcp_servers.ra_h_os
/opt/monitoring/prometheus.yml                          (CT 116 .116)
/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
/opt/home_stack/docker-compose.yml                      (VM 109 .7)
/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
/etc/systemd/system/ornith-server.service               (amdpve .15 — strix-moe)
/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
# Hermes agent configs (key enforcement — 2026-07-10)
/root/.hermes/config.yaml                               (Mumuni CT 114, Tanko CT 112, etc.)
/root/.config/systemd/user/hermes-gateway.service       (Mumuni CT 114 — EnvironmentFile fixed)
/etc/environment                                        (Mumuni CT 114 — LITELLM_API_KEY)
```

## MCP Gateway (2026-07-10)

LiteLLM CT 116 now serves as an authenticated MCP gateway for RA-H OS tools.

### Configuration

**litellm_config.yaml** (`/opt/inference-harness/litellm_config.yaml`):
```yaml
mcp_servers:
  ra_h_os:
    url: "http://192.168.68.65:3100/mcp"
    transport: "http"
    auth_type: "none"
```

**docker-compose.yml** env vars:
```yaml
- MCP_SERVER_RAHOS_URL=http://192.168.68.65:3100/mcp
- MCP_SERVER_RAHOS_TRANSPORT=http
```

### Access

| Key | MCP Access |
|-----|-----------|
| Master key | ✅ Full — 90 tools (vault-injected) |
| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.90.0-rc.1 |

### Known Limitations
- Per-key MCP server grants not functional — only master key has access
- Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
- HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
- `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)

### Migration Path
When LiteLLM is upgraded to a version supporting per-key MCP grants:
1. Grant agent keys `mcp_servers: ["ra_h_os"]`
2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config

## Security-Specific Updates

| Check | Command | Action |
|-------|---------|--------|
| CVE count | `apt list --upgradable 2>/dev/null \| grep -i security \| wc -l` | Report in update summary |
| Kernel vulns | `uname -r` vs latest available | Flag if >2 versions behind |
| Docker CVEs | `docker scout quickview` or `trivy image` | Flag critical CVEs |
| SSL certs | `openssl s_client -connect chat.sysloggh.net:443 </dev/null 2>/dev/null \| openssl x509 -noout -dates` | Alert if <30 days |

## Success Criteria

- [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
- [ ] LiteLLM inference passing (syslog-auto test)
- [ ] Zulip server + all 3 agents connected
- [ ] GPU fleet at full capacity (3/3)
- [ ] LiteLLM MCP gateway healthy (90 tools via master key)
- [ ] Zero security CVEs remaining
- [ ] <10 min total downtime per service

## Report

After completion, send Zulip DM:
```
📋 Infrastructure Update — YYYY-MM-DD

Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
Security fixes: N CVEs patched
Downtime: <service> <duration>
Failures: none / <details>
Configs backed up: /tmp/infra-update-backup-YYYYMMDD/
```