prose-contracts/infrastructure-control.prose.md

kind, name, description

kind	name	description
pattern	infrastructure-control	Full infrastructure monitoring and control pattern covering the 5-node Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric, and network services. Declares what agents can observe, what they can enforce, and where human escalation is required.

Architecture Overview

                    ┌──────────────────────────────────────────┐
                    │          OpenProse Contracts              │
                    │  (monitor → diagnose → remediate → log)   │
                    └─────┬──────────┬──────────┬──────────────┘
                          │          │          │
             ┌────────────┼──────────┼──────────┼────────────┐
             ▼            ▼          ▼          ▼            ▼
       ┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐
       │ Proxmox  │ │  Docker  │ │Storage│ │Network│ │  Agents  │
       │ Cluster  │ │Ecosystems│ │Fabric │ │Services│ │(health)  │
       │ 5 nodes  │ │ 3 hosts  │ │NFS+ZFS│ │DNS+Auth│ │6 agents  │
       └──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘

Layer 1: Proxmox Cluster

Cluster Topology

minipve  (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110,
│                                 mumuni CT 114, syslog-api CT 116,
│                                 jitsi CT 118
│
amdpve   (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105,
│                                 tanko CT 112, tdunna CT 111,
│                                 baggy CT 113, scottdenya CT 115
│
storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106,
│                                 PBS CT 107, media CT 108,
│                                 zulip CT 117
│
acerpve  (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102
│
ocupve   (.5) ──── 12C/14GB ─── ocu-llm VM 103

Maintains (Monitoring)

## Maintains
- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime
- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage
- cluster.storage: each storage target's usage %, active/inactive
- cluster.ha: high-availability state and fencing status

Rules (Monitoring)

Rule PM-1: Node Offline
  Detect → PVE API shows node status != "online"
  Action → Alert via Zulip DM with node name and last known uptime
  Escalate → If >2 nodes offline, page owner
  Requires: PVE API token

Rule PM-2: Node Resource Pressure
  Detect → CPU > 80% or RAM > 85% for 5 consecutive checks
  Action → Log to KG, alert via Zulip DM with resource snapshot
  Escalate → If sustained >90% for 1h, page owner
  Requires: PVE API token

Rule PM-3: VM/CT Unexpectedly Stopped
  Detect → PVE API shows running VM now "stopped"
  Action → `pct start <vmid>` or `qm start <vmid>` via PVE API
  Verify → Re-check status after 30s
  Escalate → If still stopped after 2 retries, alert owner
  Requires: PVE API token with appropriate permissions

Rule PM-4: Storage Usage Warning
  Detect → Storage pool usage > 80%
  Action → Log to KG, alert via Zulip DM
  Escalate → If >90%, immediate alert (risk of VM freeze)
  Requires: PVE API

Control (Remediation)

## Enforce
- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped
  - Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116),
              docker-vm (109), gitea (110), authentik (104), zulip (117)
- PM-4 storage: if mediastore > 85%, alert with biggest consumers

## Prohibited
- Never migrate VMs/CTs between nodes automatically
- Never delete storage volumes
- Never modify cluster HA configuration

Layer 2: Docker Ecosystems

Docker Hosts

Host: docker-vm (192.168.68.7, storepve VM 109)
  Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used)
  Stacks:
    firecrawl:      api, rabbitmq, postgres, playwright, redis
    searxng:        searxng (:8888), valkey
    home_stack:     jdownloader (:5800), bentopdf (:8989),
                    pulse (:7655), audiobookshelf (:13378)
  Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS)

Host: syslog-api (192.168.68.116, minipve CT 116)
  Resources: 6GB RAM, 2 vCPU, 39GB disk
  Stacks:
    harness:        litellm (:4001), nginx (:80), router (:9000),
                    postgres, redis, dashboard (:3000)

Host: netbird (72.61.0.17, external VM)
  Stacks:
    netbird:        server, dashboard, proxy, crowdsec, traefik

Maintains (Monitoring)

## Maintains
- docker.hosts: each host's status, container count, resource usage
- docker.containers: each container's status (running/restarting/exited),
                     uptime, restart count, image version
- docker.stacks: each compose stack's service count and health
- docker.resources: per-host disk usage, Docker system df

Rules (Monitoring)

Rule DC-1: Container Not Running
  Detect → `docker ps` shows "exited" or "restarting" for critical container
  Action → `docker compose up -d <service>` on the affected stack
  Verify → Re-check after 10s
  Escalate → If still failed after 3 retries, alert owner
  Requires: SSH to docker-vm or CT 116

Rule DC-2: Container Restart Looping
  Detect → Container restart count > 5 in last hour
  Action → `docker logs --tail 50 <container>` → diagnose → KG log
  Escalate → Always alert owner with diagnostic excerpt
  Requires: SSH to docker-vm or CT 116

Rule DC-3: Docker Host Disk Pressure
  Detect → `docker system df` shows < 10% reclaimable or disk > 85%
  Action → `docker system prune -af` (safe: removes unused images/containers)
  Verify → Re-check disk usage
  Escalate → If still >85% after prune, alert owner
  Requires: SSH to docker-vm

Rule DC-4: Stack Health Check Failed
  Detect → Container HEALTHCHECK status shows "unhealthy"
  Action → `docker compose restart <service>`
  Verify → Re-check health after 15s
  Escalate → After 2 retries
  Requires: SSH to affected host

Rule DC-5: Firecrawl API Down
  Detect → HTTP GET :3002/health returns non-200
  Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1`
  Escalate → If still down after 2 retries
  Requires: SSH to docker-vm

Rule DC-6: SearXNG Down
  Detect → HTTP GET :8888 returns non-200
  Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng`
  Escalate → If still down after 2 retries
  Requires: SSH to docker-vm

Control (Remediation)

## Enforce
- DC-1 auto-restart: restart critical containers when exited
- DC-3 auto-prune: prune unused Docker artifacts weekly
- DC-4/5/6: restart unhealthy stacks with verification

## Prohibited
- Never remove volumes (data loss risk)
- Never modify docker-compose files remotely
- Never update container images without owner approval

Layer 3: Storage Fabric

Storage Topology

storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used)
                            └── audiobookshelf, JDownloader downloads
storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used)
                            └── media library, configs
storepve ── PBS CT 107 ──► backup target for all VMs/CTs

Maintains

## Maintains
- storage.nfs_mounts: each mount's availability, usage %, inode count
- storage.pbs: last successful backup timestamp, backup schedule compliance
- storage.mediastore_growth: weekly growth rate (high-use volume)

Rules

Rule ST-1: NFS Mount Unreachable
  Detect → `df` or `mountpoint -q` fails for NFS path
  Action → Check storepve NFS server status via PVE API
  Escalate → Immediate alert — containers may be read-only or crashing
  Requires: PVE API + SSH to docker-vm

Rule ST-2: Mediadisk Growth Alert
  Detect → mediastore (7.3TB, 75%) growing > 50GB/week
  Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10`
  Log → KG node with growth trend
  Escalate → If projected to exceed 90% within 30 days

Rule ST-3: PBS Backup Compliance
  Detect → Any CT/VM without a backup in the last 48h
  Action → Log to KG, alert via Zulip DM
  Requires: PVE API to check PBS tasks

Layer 4: Network Services

Service Map

Service	Host	IP	Port	Purpose
adguard	CT 102	.102	53, 80, 443	DNS + DHCP
authentik	CT 104	.104	443	OIDC auth provider
gitea	CT 110	.110	3000, 22	Git hosting
zulip	CT 117	.117	443	Team chat
netbird	72.61.0.17	.17	443, 8006	VPN controller
minipve	.12	.12	443, 8006	PVE API endpoint
PBS	CT 107	.107	8007	Proxmox Backup Server

Maintains

## Maintains
- network.dns_resolution: critical hostnames resolve correctly
- network.auth_endpoints: authentik OIDC login flow works
- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse)

Rules

Rule NW-1: DNS Failure
  Detect → `host` or `nslookup` fails for known hostnames
  Action → Check adguard CT 102 status via PVE API
  Escalate → If authentik/gitea unreachable due to DNS → page owner
  Requires: PVE API + dig/nslookup

Rule NW-2: Auth Endpoint Down
  Detect → `GET https://auth.sysloggh.net` returns non-200
  Action → Check authentik CT 104 via PVE API → start if stopped
  Escalate → Can't fix: relay to Mumuni (has authentik SSH access)
  Requires: PVE API

Rule NW-3: Public Endpoint Unreachable
  Detect → Any public URL fails (litellm, git, chat, pulse)
  Action → Check container + nginx health on relevant host
  Requires: SSH to affected host + curl

Layer 5: Agent Health

Agent Registry

Agent	Platform	CT	Status Endpoint	Last Seen
abiba	pi	100	:9200/health	live
tanko	Hermes	122	(via Gateway)	live
mumuni	Hermes	114	(via Gateway)	live
kagentz	Agent Zero	105	(via PVE)	running
koby	Hermes	111	(via PVE)	stopped
koonimo	Hermes	113	(via PVE)	stopped

Rules

Rule AG-1: Agent VM/CT Stopped
  Detect → PVE API shows agent CT as "stopped"
  Action → `pct start <vmid>` via PVE API
  Verify → Re-check after 15s
  Escalate → If persistent, alert owner
  Requires: PVE API

Rule AG-2: Agent Not Responding
  Detect → Health endpoint non-200 or Gateway disconnected
  Action → Check CT is running, then check service status
  Requires: PVE API + SSH

Access Matrix

Resource	Abiba (CT 100)	Tanko (CT 122)	Mumuni (CT 114)
PVE API (minipve:443)	✅ API token	❌	❌
docker-vm (SSH .7)	✅ root SSH	❌	❌
CT 116 (SSH)	✅ root SSH	❌	❌
Gitea	✅ API token	Need to add	Need to add
Zulip API	✅ bot token	✅ bot token	✅ bot token
RA-H OS	✅ MCP bridge	✅ MCP bridge	✅ MCP bridge
Netbird (SSH .17)	✅ root SSH	❌	❌
Authentik	❌ No SSH	❌ No SSH	❌ No SSH

Priority Matrix for Auto-Remediation

Priority	Condition	Action	SLA
P0	Critical CT/VM stopped	Auto-start via PVE API	2 min
P0	LiteLLM/router down	Auto-restart container	3 min
P1	Firecrawl/SearXNG down	Auto-restart container	5 min
P1	Docker disk > 85%	Auto-prune	5 min
P2	Storage usage > 80%	Alert + log	15 min
P2	CT resource pressure	Alert + log	15 min
P3	Non-critical container down	Alert + log	30 min
P3	PBS backup missed	Alert + log	1h

Escalation Path

1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access
                              │                              │
                              ▼                              ▼
                        Kwame (Zulip DM)              Mumuni (for authentik)
                                                      Tanko (for hermes agents)

Known Gaps

Gap	What's Needed
Authentik SSH	No agent has SSH — Mumuni needs key added
Koby/Koonimo stopped	CTs 111/113 are offline — need human intervention
Mumuni PVE access	No PVE token — needs monitoring@pve!mumuni
docker-vm backups	VM 109 isn't running PBS agent — backup status unknown
Netbird redundancy	Single point of failure for VPN