prose-contracts/infrastructure-control.prose.md

---
kind: pattern
name: infrastructure-control
description: >
  Full infrastructure monitoring and control pattern covering the 5-node
  Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric,
  and network services. Declares what agents can observe, what they can
  enforce, and where human escalation is required.
---

## Architecture Overview

```
                    ┌──────────────────────────────────────────┐
                    │          OpenProse Contracts              │
                    │  (monitor → diagnose → remediate → log)   │
                    └─────┬──────────┬──────────┬──────────────┘
                          │          │          │
             ┌────────────┼──────────┼──────────┼────────────┐
             ▼            ▼          ▼          ▼            ▼
       ┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐
       │ Proxmox  │ │  Docker  │ │Storage│ │Network│ │  Agents  │
       │ Cluster  │ │Ecosystems│ │Fabric │ │Services│ │(health)  │
       │ 5 nodes  │ │ 3 hosts  │ │NFS+ZFS│ │DNS+Auth│ │6 agents  │
       └──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘
```

## Layer 1: Proxmox Cluster

### Cluster Topology

```
minipve  (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110,
│                                 mumuni CT 114, syslog-api CT 116,
│                                 jitsi CT 118
│
amdpve   (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105,
│                                 tanko CT 112, tdunna CT 111,
│                                 baggy CT 113, scottdenya CT 115
│
storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106,
│                                 PBS CT 107, media CT 108,
│                                 zulip CT 117
│
acerpve  (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102
│
ocupve   (.5) ──── 12C/14GB ─── ocu-llm VM 103
```

### Maintains (Monitoring)

```yaml
## Maintains
- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime
- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage
- cluster.storage: each storage target's usage %, active/inactive
- cluster.ha: high-availability state and fencing status
```

### Rules (Monitoring)

```
Rule PM-1: Node Offline
  Detect → PVE API shows node status != "online"
  Action → Alert via Zulip DM with node name and last known uptime
  Escalate → If >2 nodes offline, page owner
  Requires: PVE API token

Rule PM-2: Node Resource Pressure
  Detect → CPU > 80% or RAM > 85% for 5 consecutive checks
  Action → Log to KG, alert via Zulip DM with resource snapshot
  Escalate → If sustained >90% for 1h, page owner
  Requires: PVE API token

Rule PM-3: VM/CT Unexpectedly Stopped
  Detect → PVE API shows running VM now "stopped"
  Action → `pct start <vmid>` or `qm start <vmid>` via PVE API
  Verify → Re-check status after 30s
  Escalate → If still stopped after 2 retries, alert owner
  Requires: PVE API token with appropriate permissions

Rule PM-4: Storage Usage Warning
  Detect → Storage pool usage > 80%
  Action → Log to KG, alert via Zulip DM
  Escalate → If >90%, immediate alert (risk of VM freeze)
  Requires: PVE API
```

### Control (Remediation)

```yaml
## Enforce
- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped
  - Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116),
              docker-vm (109), gitea (110), authentik (104), zulip (117)
- PM-4 storage: if mediastore > 85%, alert with biggest consumers

## Prohibited
- Never migrate VMs/CTs between nodes automatically
- Never delete storage volumes
- Never modify cluster HA configuration
```

## Layer 2: Docker Ecosystems

### Docker Hosts

```yaml
Host: docker-vm (192.168.68.7, storepve VM 109)
  Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used)
  Stacks:
    firecrawl:      api, rabbitmq, postgres, playwright, redis
    searxng:        searxng (:8888), valkey
    home_stack:     jdownloader (:5800), bentopdf (:8989),
                    pulse (:7655), audiobookshelf (:13378)
  Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS)

Host: syslog-api (192.168.68.116, minipve CT 116)
  Resources: 6GB RAM, 2 vCPU, 39GB disk
  Stacks:
    harness:        litellm (:4001), nginx (:80), router (:9000),
                    postgres, redis, dashboard (:3000)

Host: netbird (72.61.0.17, external VM)
  Stacks:
    netbird:        server, dashboard, proxy, crowdsec, traefik
```

### Maintains (Monitoring)

```yaml
## Maintains
- docker.hosts: each host's status, container count, resource usage
- docker.containers: each container's status (running/restarting/exited),
                     uptime, restart count, image version
- docker.stacks: each compose stack's service count and health
- docker.resources: per-host disk usage, Docker system df
```

### Rules (Monitoring)

```
Rule DC-1: Container Not Running
  Detect → `docker ps` shows "exited" or "restarting" for critical container
  Action → `docker compose up -d <service>` on the affected stack
  Verify → Re-check after 10s
  Escalate → If still failed after 3 retries, alert owner
  Requires: SSH to docker-vm or CT 116

Rule DC-2: Container Restart Looping
  Detect → Container restart count > 5 in last hour
  Action → `docker logs --tail 50 <container>` → diagnose → KG log
  Escalate → Always alert owner with diagnostic excerpt
  Requires: SSH to docker-vm or CT 116

Rule DC-3: Docker Host Disk Pressure
  Detect → `docker system df` shows < 10% reclaimable or disk > 85%
  Action → `docker system prune -af` (safe: removes unused images/containers)
  Verify → Re-check disk usage
  Escalate → If still >85% after prune, alert owner
  Requires: SSH to docker-vm

Rule DC-4: Stack Health Check Failed
  Detect → Container HEALTHCHECK status shows "unhealthy"
  Action → `docker compose restart <service>`
  Verify → Re-check health after 15s
  Escalate → After 2 retries
  Requires: SSH to affected host

Rule DC-5: Firecrawl API Down
  Detect → HTTP GET :3002/health returns non-200
  Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1`
  Escalate → If still down after 2 retries
  Requires: SSH to docker-vm

Rule DC-6: SearXNG Down
  Detect → HTTP GET :8888 returns non-200
  Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng`
  Escalate → If still down after 2 retries
  Requires: SSH to docker-vm
```

### Control (Remediation)

```yaml
## Enforce
- DC-1 auto-restart: restart critical containers when exited
- DC-3 auto-prune: prune unused Docker artifacts weekly
- DC-4/5/6: restart unhealthy stacks with verification

## Prohibited
- Never remove volumes (data loss risk)
- Never modify docker-compose files remotely
- Never update container images without owner approval
```

## Layer 3: Storage Fabric

### Storage Topology

```
storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used)
                            └── audiobookshelf, JDownloader downloads
storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used)
                            └── media library, configs
storepve ── PBS CT 107 ──► backup target for all VMs/CTs
```

### Maintains

```yaml
## Maintains
- storage.nfs_mounts: each mount's availability, usage %, inode count
- storage.pbs: last successful backup timestamp, backup schedule compliance
- storage.mediastore_growth: weekly growth rate (high-use volume)
```

### Rules

```
Rule ST-1: NFS Mount Unreachable
  Detect → `df` or `mountpoint -q` fails for NFS path
  Action → Check storepve NFS server status via PVE API
  Escalate → Immediate alert — containers may be read-only or crashing
  Requires: PVE API + SSH to docker-vm

Rule ST-2: Mediadisk Growth Alert
  Detect → mediastore (7.3TB, 75%) growing > 50GB/week
  Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10`
  Log → KG node with growth trend
  Escalate → If projected to exceed 90% within 30 days

Rule ST-3: PBS Backup Compliance
  Detect → Any CT/VM without a backup in the last 48h
  Action → Log to KG, alert via Zulip DM
  Requires: PVE API to check PBS tasks
```

## Layer 4: Network Services

### Service Map

| Service | Host | IP | Port | Purpose |
|---------|------|----|------|---------|
| adguard | CT 102 | .102 | 53, 80, 443 | DNS + DHCP |
| authentik | CT 104 | .104 | 443 | OIDC auth provider |
| gitea | CT 110 | .110 | 3000, 22 | Git hosting |
| zulip | CT 117 | .117 | 443 | Team chat |
| netbird | 72.61.0.17 | .17 | 443, 8006 | VPN controller |
| minipve | .12 | .12 | 443, 8006 | PVE API endpoint |
| PBS | CT 107 | .107 | 8007 | Proxmox Backup Server |

### Maintains

```yaml
## Maintains
- network.dns_resolution: critical hostnames resolve correctly
- network.auth_endpoints: authentik OIDC login flow works
- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse)
```

### Rules

```
Rule NW-1: DNS Failure
  Detect → `host` or `nslookup` fails for known hostnames
  Action → Check adguard CT 102 status via PVE API
  Escalate → If authentik/gitea unreachable due to DNS → page owner
  Requires: PVE API + dig/nslookup

Rule NW-2: Auth Endpoint Down
  Detect → `GET https://auth.sysloggh.net` returns non-200
  Action → Check authentik CT 104 via PVE API → start if stopped
  Escalate → Can't fix: relay to Mumuni (has authentik SSH access)
  Requires: PVE API

Rule NW-3: Public Endpoint Unreachable
  Detect → Any public URL fails (litellm, git, chat, pulse)
  Action → Check container + nginx health on relevant host
  Requires: SSH to affected host + curl
```

## Layer 5: Agent Health

### Agent Registry

| Agent | Platform | CT | Status Endpoint | Last Seen |
|-------|----------|----|----------------|-----------|
| abiba | pi | 100 | :9200/health | live |
| tanko | Hermes | 122 | (via Gateway) | live |
| mumuni | Hermes | 114 | (via Gateway) | live |
| kagentz | Agent Zero | 105 | (via PVE) | running |
| koby | Hermes | 111 | (via PVE) | stopped |
| koonimo | Hermes | 113 | (via PVE) | stopped |

### Rules

```
Rule AG-1: Agent VM/CT Stopped
  Detect → PVE API shows agent CT as "stopped"
  Action → `pct start <vmid>` via PVE API
  Verify → Re-check after 15s
  Escalate → If persistent, alert owner
  Requires: PVE API

Rule AG-2: Agent Not Responding
  Detect → Health endpoint non-200 or Gateway disconnected
  Action → Check CT is running, then check service status
  Requires: PVE API + SSH
```

## Access Matrix

| Resource | Abiba (CT 100) | Tanko (CT 122) | Mumuni (CT 114) |
|----------|---------------|----------------|-----------------|
| **PVE API** (minipve:443) | ✅ API token | ❌ | ❌ |
| **docker-vm** (SSH .7) | ✅ root SSH | ❌ | ❌ |
| **CT 116** (SSH) | ✅ root SSH | ❌ | ❌ |
| **Gitea** | ✅ API token | Need to add | Need to add |
| **Zulip API** | ✅ bot token | ✅ bot token | ✅ bot token |
| **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge |
| **Netbird** (SSH .17) | ✅ root SSH | ❌ | ❌ |
| **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH |

## Priority Matrix for Auto-Remediation

| Priority | Condition | Action | SLA |
|----------|-----------|--------|-----|
| **P0** | Critical CT/VM stopped | Auto-start via PVE API | 2 min |
| **P0** | LiteLLM/router down | Auto-restart container | 3 min |
| **P1** | Firecrawl/SearXNG down | Auto-restart container | 5 min |
| **P1** | Docker disk > 85% | Auto-prune | 5 min |
| **P2** | Storage usage > 80% | Alert + log | 15 min |
| **P2** | CT resource pressure | Alert + log | 15 min |
| **P3** | Non-critical container down | Alert + log | 30 min |
| **P3** | PBS backup missed | Alert + log | 1h |

## Escalation Path

```
1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access
                              │                              │
                              ▼                              ▼
                        Kwame (Zulip DM)              Mumuni (for authentik)
                                                      Tanko (for hermes agents)
```

## Known Gaps

| Gap | What's Needed |
|-----|--------------|
| Authentik SSH | No agent has SSH — Mumuni needs key added |
| Koby/Koonimo stopped | CTs 111/113 are offline — need human intervention |
| Mumuni PVE access | No PVE token — needs `monitoring@pve!mumuni` |
| docker-vm backups | VM 109 isn't running PBS agent — backup status unknown |
| Netbird redundancy | Single point of failure for VPN |