prose-contracts/infrastructure-control.prose.md

---
kind: pattern
name: infrastructure-control
description: >
  Full infrastructure monitoring and control pattern covering the
  5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
  NFS storage, and network services. Defines monitors, remediations,
  and the access matrix for all environments.
---

# Infrastructure Control Pattern

## Topology

```
                    ┌─────────────────────────┐
                    │   OpenProse Contract    │
                    │  (declares what's true) │
                    └──────────┬──────────────┘
                               │
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
   │   CT 100    │    │   CT 122     │    │   CT 114     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
                             ▼
          ┌──────────────────────────────────────┐
          │         Proxmox Cluster API          │
          │    minipve.sysloggh.net:443          │
          │   (monitoring@pve!mumuni token)      │
          └────┬──────┬──────┬──────┬──────┬─────┘
               │      │      │      │      │
          ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
          ▼      ▼      ▼      ▼      ▼
       minipve amdpve storepve acerpve ocupve
        (.12)   (.15)   (.6)    (.9)    (.5)

          ▼
   ┌─────────────────────────────────────────────┐
   │            Docker Ecosystems                │
   │  ┌────────────┬────────────┬──────────────┐ │
   │  │ docker-vm  │ CT 116     │ Netbird      │ │
   │  │ (.7)       │ (.116)     │ (72.61.0.17) │ │
   │  │ 11 ctrs    │ 6 ctrs     │ 5 ctrs       │ │
   │  └────────────┴────────────┴──────────────┘ │
   └─────────────────────────────────────────────┘
```

## Section 1: Access Matrix

### API Tokens & Credentials

| Resource | Auth Method | Credential Source | Status |
|----------|------------|-------------------|--------|
| Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ |
| Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ |
| docker-vm (.7) | SSH root | SSH key | ✅ |
| CT 116 (syslog-api) | SSH root | SSH key | ✅ |
| Tanko CT (.122) | SSH jerome | SSH key | ✅ |
| Netbird (.17) | SSH root | SSH key | ✅ |
| Gitea | API token | abiba-bot token | ✅ |
| Zulip | Bot API key | per-bot tokens | ✅ |
| RA-H OS | MCP bridge | port 3100 | ✅ |

### Reachability Matrix

| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) |
|-----------|---------|----------------|--------|-------------|---------------|
| **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird |
| **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ |
| **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ |

**Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba.

## Section 2: Proxmox Cluster — Monitoring

### Nodes (5)

| Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |

### Checks (every 5 min)

```
## Maintains

- cluster-status: { online_nodes: int, offline_nodes: int, timestamp }
- node-cpu-usage: { node: pct, warnings: [] }
- node-memory-usage: { node: free_pct, warnings: [] }
- node-uptime: { node: seconds, just_rebooted: bool }

## Checks

- For each node in [minipve, amdpve, storepve, acerpve, ocupve]:
  - GET /api2/json/nodes/{node}/status → check status == "online"
  - GET /api2/json/nodes/{node}/status → cpu < 0.80
  - GET /api2/json/nodes/{node}/status → free_mem > 10%
  - GET /api2/json/nodes/{node}/status → uptime > 300 (warn if just rebooted)

- For each CT in inventory (19 total):
  - GET /api2/json/cluster/resources → filter by type=lxc
  - Warn if status != "running"

- Storage pools:
  - GET /api2/json/nodes/storepve/storage/mediastore → used < 80%
  - GET /api2/json/nodes/storepve/storage/pbs-backup → last backup < 48h

## Remediations

- Node offline → alert via Zulip DM, escalate after 3x
- CT stopped → `pct start <vmid>` via PVE API, verify after 30s
- VM stopped → `qm start <vmid>` via PVE API, verify after 60s
- Storage > 80% → warn via Zulip DM
- Storage > 95% → crit via Zulip + relay to maintainer
```

## Section 3: Docker Ecosystems — Monitoring

### Ecosystem A: docker-vm (192.168.68.7)

11 containers across 4 compose stacks:

| Stack | Path | Containers |
|-------|------|-----------|
| **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis |
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
| **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse |
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |

### Ecosystem B: CT 116 syslog-api (192.168.68.116)

| Container | Image | Port |
|-----------|-------|------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 |
| harness-nginx | nginx:alpine | :80 |
| harness-router | inference-harness-router | :9000 |
| harness-postgres | postgres:16-alpine | :5432 |
| harness-redis | redis:7-alpine | :6379 |
| harness-dashboard | inference-harness-dashboard | :3000 |

### Ecosystem C: Netbird (72.61.0.17)

| Container | Image | Role |
|-----------|-------|------|
| netbird-server | netbird | VPN controller |
| netbird-dashboard | netbird UI | Web management |
| netbird-proxy | nginx | TLS termination |
| netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF |
| netbird-traefik | traefik | Reverse proxy |

### Checks (every 60s)

```
## Maintains

- docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int }
- container-status: { name: string, status: string, restarts: int, image: string }

## Checks

For each Docker host:
  - docker ps --format "{{.Names}}" → every required container is running
  - docker inspect <name> --format "{{.State.Health.Status}}" → "healthy"
  - docker inspect <name> --format "{{.RestartCount}}" → < 3/hour
  - df -h / | awk '{print $5}' → usage < 80%

For docker-vm specifically:
  - mountpoint -q /media/storage → NFS mounted
  - mountpoint -q /media/mediastore → NFS mounted
  - docker compose ls → expected stacks present

## Remediations

- Container unhealthy → `docker compose up -d <name>`, wait 10s, re-check
- Container missing → `docker compose up -d` in its stack directory
- Docker daemon down → `systemctl restart docker`
- NFS mount lost → `mount -a`, if fails → alert (storepve issue)
- Disk > 80% → alert via Zulip DM
- Disk > 95% → crit + relay to maintainer
```

## Section 4: Storage — Monitoring

### NFS Mounts (docker-vm → storepve)

| Mount | Export | Capacity | Used | Alert |
|-------|--------|----------|------|-------|
| /media/storage | storepve:/media/storage | 3.6TB | 264GB (8%) | None |
| /media/mediastore | storepve:/media/mediastore | 7.3TB | 5.2TB (75%) | ⚠️ warn at 80% |

### Proxmox Storage Backends

| Storage | Type | Content | Active |
|---------|------|---------|--------|
| local | dir | ISO, vztmpl, backup | ✅ |
| local-lvm | lvmthin | images, rootdir | ✅ |
| zfs-vm | nfs | rootdir, images | ✅ |
| mediastore | dir | images, backup, rootdir | ✅ |
| zfs-iso | nfs | vztmpl, iso | ✅ |
| pbs-backup | pbs | backup | ✅ |

### Checks (every 10 min)

```
## Maintains

- storage-usage: { mount: string, used_pct: float, growth_rate: float }
- backup-status: { last_success: timestamp, age_hours: int }
- snapshot-status: { pool: string, newest_age_hours: int }

## Specific Alerts

- mediastore growth rate > 10GB/day → warn (check what's writing)
- mediastore > 90% → crit (only 730GB remaining)
- No PBS backup in 48h → fail
```

## Section 5: Network Services — Monitoring

| Service | Domain | Status |
|---------|--------|--------|
| Proxmox API | minipve.sysloggh.net:443 | ✅ |
| Authentik | auth.sysloggh.net:443 | ✅ |
| Gitea | git.sysloggh.net:443 | ✅ |
| Zulip | chat.sysloggh.net:443 | ✅ |
| LiteLLM | litellm.sysloggh.net:443 | ✅ |
| Pulse | pulse.sysloggh.net:443 | ✅ |
| SearXNG | searxng.sysloggh.net:8888 | ✅ |
| Firecrawl | firecrawl.sysloggh.net:3002 | ✅ |

### Checks (every 2 min)

```
## Maintains

- dns-resolution: { services: [{name, resolves}] }
- ssl-expiry: { services: [{name, days_remaining}] }
- endpoint-reachability: { services: [{name, http_code}] }
```

## Section 6: Alert Routing & Escalation

### Severity Levels

| Severity | Channel | Format | Rate Limit |
|----------|---------|--------|-----------|
| **warn** | Zulip DM to owner | "⚠️ {check}: {detail}" | 1x/check/hour |
| **crit** | Zulip DM + #agent-hub | "🚨 {check}: {detail}" | Immediate |
| **escalated** | Zulip DM + relay to maintainer | "🔥 {check}: {detail}" | Immediate |

### Escalation Chain

```
Container unhealthy (3 consecutive failures)
  → Alert owner
  → 15 min no response → Relay to maintainer

Node offline
  → Alert owner
  → 5 min → Attempt restart via PVE API
  → 2 failures → Escalate

Storage > 95%
  → Alert owner + maintainer relay
  → Immediate action required

SSL cert < 7 days
  → Alert owner daily
  → 3 days out → Escalate

ZFS pool degraded
  → Cannot auto-fix → Escalate immediately
```

## Appendix A: Quick Health Commands

```bash
# Full cluster status
PVE="https://minipve.sysloggh.net"
AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3"
curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH"

# Docker health from Abiba
ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'"
ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'"

# Storage check
ssh root@192.168.68.7 "df -h /media/storage /media/mediastore"
```

## Appendix B: CT Inventory

| CT | Name | Node | IP | Role | Agent |
|----|------|------|----|------|-------|
| 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi |
| 101 | llm-gpu | acerpve | — | GPU VM | ❌ |
| 102 | adguard | acerpve | — | DNS | ❌ |
| 103 | ocu-llm | ocupve | — | GPU VLM | ❌ |
| 104 | authentik | minipve | .11 | OIDC | ❌ |
| 105 | kagentz | amdpve | — | Agent Zero | ✅ |
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
| 107 | pbs | storepve | — | Backups | ❌ |
| 108 | media | storepve | — | Media | ❌ |
| 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | — | Git | ❌ |
| 111 | tdunna | amdpve | — | ? | ❌ |
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
| 113 | baggy | amdpve | — | ? | ❌ |
| 114 | mumuni | minipve | — | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | — | ? | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ |
| 117 | zulip | storepve | — | Chat | ❌ |
| 118 | jitsi | minipve | — | Video | ❌ |

## Appendix C: Docker Compose Files Location

| Host | Stack | Compose File |
|------|-------|-------------|
| docker-vm (.7) | Firecrawl | `/opt/search-stack/firecrawl-source/docker-compose.yaml` |
| docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` |
| docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` |
| docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
| CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` |
| Netbird (.17) | Netbird | Docker run (not compose) |