--- kind: pattern name: infrastructure-control description: > Full infrastructure monitoring and control pattern covering the 5-node Proxmox cluster, 3 Docker ecosystems (22 containers), NFS storage, and network services. Defines monitors, remediations, and the access matrix for all environments. --- # Infrastructure Control Pattern ## Topology ``` ┌─────────────────────────┐ │ OpenProse Contract │ │ (declares what's true) │ └──────────┬──────────────┘ │ ┌────────────────────┼────────────────────┐ ▼ ▼ ▼ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ Abiba │ │ Tanko │ │ Mumuni │ │ (pi) │ │ (Hermes) │ │ (Hermes) │ │ CT 100 │ │ CT 122 │ │ CT 114 │ └──────┬──────┘ └──────┬───────┘ └──────┬───────┘ │ │ │ └──────────────────┼────────────────────┘ ▼ ┌──────────────────────────────────────┐ │ Proxmox Cluster API │ │ minipve.sysloggh.net:443 │ │ (monitoring@pve!mumuni token) │ └────┬──────┬──────┬──────┬──────┬─────┘ │ │ │ │ │ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ▼ ▼ ▼ ▼ ▼ minipve amdpve storepve acerpve ocupve (.12) (.15) (.6) (.9) (.5) ▼ ┌─────────────────────────────────────────────┐ │ Docker Ecosystems │ │ ┌────────────┬────────────┬──────────────┐ │ │ │ docker-vm │ CT 116 │ Netbird │ │ │ │ (.7) │ (.116) │ (72.61.0.17) │ │ │ │ 11 ctrs │ 6 ctrs │ 5 ctrs │ │ │ └────────────┴────────────┴──────────────┘ │ └─────────────────────────────────────────────┘ ``` ## Section 1: Access Matrix ### API Tokens & Credentials | Resource | Auth Method | Credential Source | Status | |----------|------------|-------------------|--------| | Proxmox Cluster | PVE API Token | `monitoring@pve!mumuni=...` | ✅ | | Proxmox Root | Password via API ticket | `root@pam:kakashi19` | ✅ | | docker-vm (.7) | SSH root | SSH key | ✅ | | CT 116 (syslog-api) | SSH root | SSH key | ✅ | | Tanko CT (.122) | SSH jerome | SSH key | ✅ | | Netbird (.17) | SSH root | SSH key | ✅ | | Gitea | API token | abiba-bot token | ✅ | | Zulip | Bot API key | per-bot tokens | ✅ | | RA-H OS | MCP bridge | port 3100 | ✅ | ### Reachability Matrix | From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Netbird (.17) | |-----------|---------|----------------|--------|-------------|---------------| | **Abiba** (CT 100) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH | ❌ no Netbird | | **Tanko** (CT 122) | ❌ | ❌ | ❌ | ✅ | ❌ | | **docker-vm** (.7) | ❌ | ✅ | ❌ | ❌ | ❌ | **Conclusion:** Only Abiba has cross-infrastructure access. All monitoring contracts run from Abiba. ## Section 2: Proxmox Cluster — Monitoring ### Nodes (5) | Node | IP | CPU | RAM | VMs/CTs | Role | |------|----|-----|-----|---------|------| | minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging | | amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute | | storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat | | acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | ### Checks (every 5 min) ``` ## Maintains - cluster-status: { online_nodes: int, offline_nodes: int, timestamp } - node-cpu-usage: { node: pct, warnings: [] } - node-memory-usage: { node: free_pct, warnings: [] } - node-uptime: { node: seconds, just_rebooted: bool } ## Checks - For each node in [minipve, amdpve, storepve, acerpve, ocupve]: - GET /api2/json/nodes/{node}/status → check status == "online" - GET /api2/json/nodes/{node}/status → cpu < 0.80 - GET /api2/json/nodes/{node}/status → free_mem > 10% - GET /api2/json/nodes/{node}/status → uptime > 300 (warn if just rebooted) - For each CT in inventory (19 total): - GET /api2/json/cluster/resources → filter by type=lxc - Warn if status != "running" - Storage pools: - GET /api2/json/nodes/storepve/storage/mediastore → used < 80% - GET /api2/json/nodes/storepve/storage/pbs-backup → last backup < 48h ## Remediations - Node offline → alert via Zulip DM, escalate after 3x - CT stopped → `pct start ` via PVE API, verify after 30s - VM stopped → `qm start ` via PVE API, verify after 60s - Storage > 80% → warn via Zulip DM - Storage > 95% → crit via Zulip + relay to maintainer ``` ## Section 3: Docker Ecosystems — Monitoring ### Ecosystem A: docker-vm (192.168.68.7) 11 containers across 4 compose stacks: | Stack | Path | Containers | |-------|------|-----------| | **Firecrawl** | `/opt/search-stack/firecrawl-source/` | api, rabbitmq, postgres, playwright, redis | | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey | | **Home stack** | `/opt/home_stack/` | jdownloader, bentopdf, pulse | | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf | ### Ecosystem B: CT 116 syslog-api (192.168.68.116) | Container | Image | Port | |-----------|-------|------| | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4001 | | harness-nginx | nginx:alpine | :80 | | harness-router | inference-harness-router | :9000 | | harness-postgres | postgres:16-alpine | :5432 | | harness-redis | redis:7-alpine | :6379 | | harness-dashboard | inference-harness-dashboard | :3000 | ### Ecosystem C: Netbird (72.61.0.17) | Container | Image | Role | |-----------|-------|------| | netbird-server | netbird | VPN controller | | netbird-dashboard | netbird UI | Web management | | netbird-proxy | nginx | TLS termination | | netbird-crowdsec | crowdsecurity/crowdsec:v1.7.7 | WAF | | netbird-traefik | traefik | Reverse proxy | ### Checks (every 60s) ``` ## Maintains - docker-health: { ecosystem: string, healthy: int, unhealthy: int, total: int } - container-status: { name: string, status: string, restarts: int, image: string } ## Checks For each Docker host: - docker ps --format "{{.Names}}" → every required container is running - docker inspect --format "{{.State.Health.Status}}" → "healthy" - docker inspect --format "{{.RestartCount}}" → < 3/hour - df -h / | awk '{print $5}' → usage < 80% For docker-vm specifically: - mountpoint -q /media/storage → NFS mounted - mountpoint -q /media/mediastore → NFS mounted - docker compose ls → expected stacks present ## Remediations - Container unhealthy → `docker compose up -d `, wait 10s, re-check - Container missing → `docker compose up -d` in its stack directory - Docker daemon down → `systemctl restart docker` - NFS mount lost → `mount -a`, if fails → alert (storepve issue) - Disk > 80% → alert via Zulip DM - Disk > 95% → crit + relay to maintainer ``` ## Section 4: Storage — Monitoring ### NFS Mounts (docker-vm → storepve) | Mount | Export | Capacity | Used | Alert | |-------|--------|----------|------|-------| | /media/storage | storepve:/media/storage | 3.6TB | 264GB (8%) | None | | /media/mediastore | storepve:/media/mediastore | 7.3TB | 5.2TB (75%) | ⚠️ warn at 80% | ### Proxmox Storage Backends | Storage | Type | Content | Active | |---------|------|---------|--------| | local | dir | ISO, vztmpl, backup | ✅ | | local-lvm | lvmthin | images, rootdir | ✅ | | zfs-vm | nfs | rootdir, images | ✅ | | mediastore | dir | images, backup, rootdir | ✅ | | zfs-iso | nfs | vztmpl, iso | ✅ | | pbs-backup | pbs | backup | ✅ | ### Checks (every 10 min) ``` ## Maintains - storage-usage: { mount: string, used_pct: float, growth_rate: float } - backup-status: { last_success: timestamp, age_hours: int } - snapshot-status: { pool: string, newest_age_hours: int } ## Specific Alerts - mediastore growth rate > 10GB/day → warn (check what's writing) - mediastore > 90% → crit (only 730GB remaining) - No PBS backup in 48h → fail ``` ## Section 5: Network Services — Monitoring | Service | Domain | Status | |---------|--------|--------| | Proxmox API | minipve.sysloggh.net:443 | ✅ | | Authentik | auth.sysloggh.net:443 | ✅ | | Gitea | git.sysloggh.net:443 | ✅ | | Zulip | chat.sysloggh.net:443 | ✅ | | LiteLLM | litellm.sysloggh.net:443 | ✅ | | Pulse | pulse.sysloggh.net:443 | ✅ | | SearXNG | searxng.sysloggh.net:8888 | ✅ | | Firecrawl | firecrawl.sysloggh.net:3002 | ✅ | ### Checks (every 2 min) ``` ## Maintains - dns-resolution: { services: [{name, resolves}] } - ssl-expiry: { services: [{name, days_remaining}] } - endpoint-reachability: { services: [{name, http_code}] } ``` ## Section 6: Alert Routing & Escalation ### Severity Levels | Severity | Channel | Format | Rate Limit | |----------|---------|--------|-----------| | **warn** | Zulip DM to owner | "⚠️ {check}: {detail}" | 1x/check/hour | | **crit** | Zulip DM + #agent-hub | "🚨 {check}: {detail}" | Immediate | | **escalated** | Zulip DM + relay to maintainer | "🔥 {check}: {detail}" | Immediate | ### Escalation Chain ``` Container unhealthy (3 consecutive failures) → Alert owner → 15 min no response → Relay to maintainer Node offline → Alert owner → 5 min → Attempt restart via PVE API → 2 failures → Escalate Storage > 95% → Alert owner + maintainer relay → Immediate action required SSL cert < 7 days → Alert owner daily → 3 days out → Escalate ZFS pool degraded → Cannot auto-fix → Escalate immediately ``` ## Appendix A: Quick Health Commands ```bash # Full cluster status PVE="https://minipve.sysloggh.net" AUTH="Authorization: PVEAPIToken=monitoring@pve!mumuni=eafd56c5-93d4-4d40-a41d-e688be0987f3" curl -sfk "$PVE/api2/json/cluster/resources" -H "$AUTH" # Docker health from Abiba ssh root@192.168.68.7 "docker ps --format '{{.Names}} {{.Status}}'" ssh root@192.168.68.116 "docker ps --format '{{.Names}} {{.Status}}'" # Storage check ssh root@192.168.68.7 "df -h /media/storage /media/mediastore" ``` ## Appendix B: CT Inventory | CT | Name | Node | IP | Role | Agent | |----|------|------|----|------|-------| | 100 | abiba | amdpve | .24 | Pi agent (this host) | ✅ pi | | 101 | llm-gpu | acerpve | — | GPU VM | ❌ | | 102 | adguard | acerpve | — | DNS | ❌ | | 103 | ocu-llm | ocupve | — | GPU VLM | ❌ | | 104 | authentik | minipve | .11 | OIDC | ❌ | | 105 | kagentz | amdpve | — | Agent Zero | ✅ | | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | | 107 | pbs | storepve | — | Backups | ❌ | | 108 | media | storepve | — | Media | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 110 | gitea | minipve | — | Git | ❌ | | 111 | tdunna | amdpve | — | ? | ❌ | | 112 | tanko | amdpve | .122 | Hermes agent | ✅ | | 113 | baggy | amdpve | — | ? | ❌ | | 114 | mumuni | minipve | — | Hermes agent | ✅ | | 115 | scottdenya | amdpve | — | ? | ❌ | | 116 | syslog-api | minipve | .116 | LiteLLM stack | ❌ | | 117 | zulip | storepve | — | Chat | ❌ | | 118 | jitsi | minipve | — | Video | ❌ | ## Appendix C: Docker Compose Files Location | Host | Stack | Compose File | |------|-------|-------------| | docker-vm (.7) | Firecrawl | `/opt/search-stack/firecrawl-source/docker-compose.yaml` | | docker-vm (.7) | SearXNG | `/opt/search-stack/searxng/docker-compose.yml` | | docker-vm (.7) | Home stack | `/opt/home_stack/docker-compose.yml` | | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` | | CT 116 (.116) | LiteLLM | `/root/docker-compose-litellm.yml` | | Netbird (.17) | Netbird | Docker run (not compose) |