From e090b27e204863c8285482ec8b3feba015259264 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 27 Jun 2026 13:11:15 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20expand=20infrastructure-control=20patte?= =?UTF-8?q?rn=20=E2=80=94=20Proxmox,=20Docker,=20storage,=20network?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full expansion of the infrastructure-control pattern covering: Layer 1 — Proxmox Cluster (5 nodes) - 4 monitoring rules (offline, resource pressure, stopped VMs/CTs, storage) - Auto-start for 8 critical CTs/VMs on unexpected stop - Node resource threshold alerting Layer 2 — Docker Ecosystems (3 hosts, 22 containers) - 6 monitoring rules across docker-vm, syslog-api, netbird - Auto-restart for critical containers and stacks - Disk pressure auto-prune Layer 3 — Storage Fabric (3.6TB + 7.3TB NFS) - Mount availability monitoring - Growth rate tracking for high-use volumes - PBS backup compliance checking Layer 4 — Network Services (DNS, auth, git, chat, VPN) - Service endpoint monitoring - Escalation to Mumuni for authentik issues Layer 5 — Agent Health (6 agents across 3 platforms) - CT auto-start for stopped agents - Health endpoint verification Full access matrix and priority-based SLA table. --- infrastructure-control.prose.md | 426 ++++++++++++++++++++++++-------- 1 file changed, 320 insertions(+), 106 deletions(-) diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index 9da55d3..5a698bd 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -2,141 +2,355 @@ kind: pattern name: infrastructure-control description: > - Architectural pattern showing how OpenProse contracts connect to - Gitea, containers, and remote agents to enforce conditions across - all environments. Not an executable contract — a reference model. + Full infrastructure monitoring and control pattern covering the 5-node + Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric, + and network services. Declares what agents can observe, what they can + enforce, and where human escalation is required. --- -## The Full Control Stack +## Architecture Overview ``` - ┌─────────────────────────┐ - │ OpenProse Contract │ - │ (declares what's true) │ - └──────────┬──────────────┘ - │ - ┌────────────────────┼────────────────────┐ - ▼ ▼ ▼ - ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ - │ Agent │ │ Agent │ │ Agent │ - │ (pi) │ │ (Tanko) │ │ (Mumuni) │ - │ CT 100 │ │ CT 112 │ │ CT 114 │ - └──────┬──────┘ └──────┬───────┘ └──────┬───────┘ - │ │ │ - └──────────────────┼────────────────────┘ - ▼ - ┌─────────────────┐ - │ Gitea Repos │ - │ (source of │ - │ truth for │ - │ code + │ - │ contracts) │ - └────────┬────────┘ - │ - ┌──────────────────┼────────────────────┐ - ▼ ▼ ▼ - ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ - │ CT 116 │ │ Authentik │ │ Zulip │ - │ (LiteLLM, │ │ CT .11 │ │ CT .117 │ - │ nginx, │ │ (OIDC auth) │ │ (messaging) │ - │ router) │ │ │ │ │ - └─────────────┘ └──────────────┘ └──────────────┘ + ┌──────────────────────────────────────────┐ + │ OpenProse Contracts │ + │ (monitor → diagnose → remediate → log) │ + └─────┬──────────┬──────────┬──────────────┘ + │ │ │ + ┌────────────┼──────────┼──────────┼────────────┐ + ▼ ▼ ▼ ▼ ▼ + ┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐ + │ Proxmox │ │ Docker │ │Storage│ │Network│ │ Agents │ + │ Cluster │ │Ecosystems│ │Fabric │ │Services│ │(health) │ + │ 5 nodes │ │ 3 hosts │ │NFS+ZFS│ │DNS+Auth│ │6 agents │ + └──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘ ``` -## How Each Layer Connects +## Layer 1: Proxmox Cluster -### 1. Code → Gitea -The contract enforces code state via git operations: +### Cluster Topology -```yaml -## Enforce -- git pull origin main on CT 116 — ensures latest config is deployed -- git diff HEAD...origin/main — warns if drift is detected -- git commit + push after auto-fix — changes are versioned +``` +minipve (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110, +│ mumuni CT 114, syslog-api CT 116, +│ jitsi CT 118 +│ +amdpve (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105, +│ tanko CT 112, tdunna CT 111, +│ baggy CT 113, scottdenya CT 115 +│ +storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106, +│ PBS CT 107, media CT 108, +│ zulip CT 117 +│ +acerpve (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102 +│ +ocupve (.5) ──── 12C/14GB ─── ocu-llm VM 103 ``` -**What the agent does:** Runs `git` commands via SSH to check and update repos. -**What enforces it:** Agent has SSH keys and Gitea credentials. - -### 2. Contracts → Agents -Each agent runs OpenProse contracts in its own environment: - -| Agent | Runs contracts via | Can reach | -|---|---|---| -| **Abiba (pi)** | `prose run` in-session | CT 116 (SSH), Zulip API, Gitea API | -| **Tanko (Hermes)** | Hermes Gateway with OpenProse skill | CT 112, Zulip API, Gitea API | -| **Mumuni (Hermes)** | Hermes Gateway with OpenProse skill | CT 114, Zulip API, Gitea API | - -**What enforces it:** Each agent has SSH keys to the environments it manages. - -### 3. Contracts → Containers -The contract enforces container state via Docker commands: +### Maintains (Monitoring) ```yaml ## Maintains -- harness-litellm.status == "healthy" -- harness-nginx.status == "healthy" - -## Enforce -- If container unhealthy → `docker compose up -d ` - - Requires: SSH access to host - - Verifies: `docker ps` after restart +- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime +- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage +- cluster.storage: each storage target's usage %, active/inactive +- cluster.ha: high-availability state and fencing status ``` -**What the agent does:** SSH to the host, run Docker commands. -**What enforces it:** SSH key + Docker socket access on the target CT. +### Rules (Monitoring) -### 4. Contracts → Remote Agents (Tanko, Mumuni) -When a contract needs another agent to act: +``` +Rule PM-1: Node Offline + Detect → PVE API shows node status != "online" + Action → Alert via Zulip DM with node name and last known uptime + Escalate → If >2 nodes offline, page owner + Requires: PVE API token + +Rule PM-2: Node Resource Pressure + Detect → CPU > 80% or RAM > 85% for 5 consecutive checks + Action → Log to KG, alert via Zulip DM with resource snapshot + Escalate → If sustained >90% for 1h, page owner + Requires: PVE API token + +Rule PM-3: VM/CT Unexpectedly Stopped + Detect → PVE API shows running VM now "stopped" + Action → `pct start ` or `qm start ` via PVE API + Verify → Re-check status after 30s + Escalate → If still stopped after 2 retries, alert owner + Requires: PVE API token with appropriate permissions + +Rule PM-4: Storage Usage Warning + Detect → Storage pool usage > 80% + Action → Log to KG, alert via Zulip DM + Escalate → If >90%, immediate alert (risk of VM freeze) + Requires: PVE API +``` + +### Control (Remediation) ```yaml -## If out of scope -- If Authentik server down → relay to Mumuni (has access) - - Creates relay message with diagnostic data - - Mumuni's contract picks it up and acts +## Enforce +- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped + - Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116), + docker-vm (109), gitea (110), authentik (104), zulip (117) +- PM-4 storage: if mediastore > 85%, alert with biggest consumers + +## Prohibited +- Never migrate VMs/CTs between nodes automatically +- Never delete storage volumes +- Never modify cluster HA configuration ``` -**What the agent does:** Creates a relay message via RA-H OS. -**What enforces it:** Mumuni's own contracts check relays on wake. +## Layer 2: Docker Ecosystems + +### Docker Hosts + +```yaml +Host: docker-vm (192.168.68.7, storepve VM 109) + Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used) + Stacks: + firecrawl: api, rabbitmq, postgres, playwright, redis + searxng: searxng (:8888), valkey + home_stack: jdownloader (:5800), bentopdf (:8989), + pulse (:7655), audiobookshelf (:13378) + Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS) + +Host: syslog-api (192.168.68.116, minipve CT 116) + Resources: 6GB RAM, 2 vCPU, 39GB disk + Stacks: + harness: litellm (:4001), nginx (:80), router (:9000), + postgres, redis, dashboard (:3000) + +Host: netbird (72.61.0.17, external VM) + Stacks: + netbird: server, dashboard, proxy, crowdsec, traefik +``` + +### Maintains (Monitoring) + +```yaml +## Maintains +- docker.hosts: each host's status, container count, resource usage +- docker.containers: each container's status (running/restarting/exited), + uptime, restart count, image version +- docker.stacks: each compose stack's service count and health +- docker.resources: per-host disk usage, Docker system df +``` + +### Rules (Monitoring) + +``` +Rule DC-1: Container Not Running + Detect → `docker ps` shows "exited" or "restarting" for critical container + Action → `docker compose up -d ` on the affected stack + Verify → Re-check after 10s + Escalate → If still failed after 3 retries, alert owner + Requires: SSH to docker-vm or CT 116 + +Rule DC-2: Container Restart Looping + Detect → Container restart count > 5 in last hour + Action → `docker logs --tail 50 ` → diagnose → KG log + Escalate → Always alert owner with diagnostic excerpt + Requires: SSH to docker-vm or CT 116 + +Rule DC-3: Docker Host Disk Pressure + Detect → `docker system df` shows < 10% reclaimable or disk > 85% + Action → `docker system prune -af` (safe: removes unused images/containers) + Verify → Re-check disk usage + Escalate → If still >85% after prune, alert owner + Requires: SSH to docker-vm + +Rule DC-4: Stack Health Check Failed + Detect → Container HEALTHCHECK status shows "unhealthy" + Action → `docker compose restart ` + Verify → Re-check health after 15s + Escalate → After 2 retries + Requires: SSH to affected host + +Rule DC-5: Firecrawl API Down + Detect → HTTP GET :3002/health returns non-200 + Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1` + Escalate → If still down after 2 retries + Requires: SSH to docker-vm + +Rule DC-6: SearXNG Down + Detect → HTTP GET :8888 returns non-200 + Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng` + Escalate → If still down after 2 retries + Requires: SSH to docker-vm +``` + +### Control (Remediation) + +```yaml +## Enforce +- DC-1 auto-restart: restart critical containers when exited +- DC-3 auto-prune: prune unused Docker artifacts weekly +- DC-4/5/6: restart unhealthy stacks with verification + +## Prohibited +- Never remove volumes (data loss risk) +- Never modify docker-compose files remotely +- Never update container images without owner approval +``` + +## Layer 3: Storage Fabric + +### Storage Topology + +``` +storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used) + └── audiobookshelf, JDownloader downloads +storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used) + └── media library, configs +storepve ── PBS CT 107 ──► backup target for all VMs/CTs +``` + +### Maintains + +```yaml +## Maintains +- storage.nfs_mounts: each mount's availability, usage %, inode count +- storage.pbs: last successful backup timestamp, backup schedule compliance +- storage.mediastore_growth: weekly growth rate (high-use volume) +``` + +### Rules + +``` +Rule ST-1: NFS Mount Unreachable + Detect → `df` or `mountpoint -q` fails for NFS path + Action → Check storepve NFS server status via PVE API + Escalate → Immediate alert — containers may be read-only or crashing + Requires: PVE API + SSH to docker-vm + +Rule ST-2: Mediadisk Growth Alert + Detect → mediastore (7.3TB, 75%) growing > 50GB/week + Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10` + Log → KG node with growth trend + Escalate → If projected to exceed 90% within 30 days + +Rule ST-3: PBS Backup Compliance + Detect → Any CT/VM without a backup in the last 48h + Action → Log to KG, alert via Zulip DM + Requires: PVE API to check PBS tasks +``` + +## Layer 4: Network Services + +### Service Map + +| Service | Host | IP | Port | Purpose | +|---------|------|----|------|---------| +| adguard | CT 102 | .102 | 53, 80, 443 | DNS + DHCP | +| authentik | CT 104 | .104 | 443 | OIDC auth provider | +| gitea | CT 110 | .110 | 3000, 22 | Git hosting | +| zulip | CT 117 | .117 | 443 | Team chat | +| netbird | 72.61.0.17 | .17 | 443, 8006 | VPN controller | +| minipve | .12 | .12 | 443, 8006 | PVE API endpoint | +| PBS | CT 107 | .107 | 8007 | Proxmox Backup Server | + +### Maintains + +```yaml +## Maintains +- network.dns_resolution: critical hostnames resolve correctly +- network.auth_endpoints: authentik OIDC login flow works +- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse) +``` + +### Rules + +``` +Rule NW-1: DNS Failure + Detect → `host` or `nslookup` fails for known hostnames + Action → Check adguard CT 102 status via PVE API + Escalate → If authentik/gitea unreachable due to DNS → page owner + Requires: PVE API + dig/nslookup + +Rule NW-2: Auth Endpoint Down + Detect → `GET https://auth.sysloggh.net` returns non-200 + Action → Check authentik CT 104 via PVE API → start if stopped + Escalate → Can't fix: relay to Mumuni (has authentik SSH access) + Requires: PVE API + +Rule NW-3: Public Endpoint Unreachable + Detect → Any public URL fails (litellm, git, chat, pulse) + Action → Check container + nginx health on relevant host + Requires: SSH to affected host + curl +``` + +## Layer 5: Agent Health + +### Agent Registry + +| Agent | Platform | CT | Status Endpoint | Last Seen | +|-------|----------|----|----------------|-----------| +| abiba | pi | 100 | :9200/health | live | +| tanko | Hermes | 122 | (via Gateway) | live | +| mumuni | Hermes | 114 | (via Gateway) | live | +| kagentz | Agent Zero | 105 | (via PVE) | running | +| koby | Hermes | 111 | (via PVE) | stopped | +| koonimo | Hermes | 113 | (via PVE) | stopped | + +### Rules + +``` +Rule AG-1: Agent VM/CT Stopped + Detect → PVE API shows agent CT as "stopped" + Action → `pct start ` via PVE API + Verify → Re-check after 15s + Escalate → If persistent, alert owner + Requires: PVE API + +Rule AG-2: Agent Not Responding + Detect → Health endpoint non-200 or Gateway disconnected + Action → Check CT is running, then check service status + Requires: PVE API + SSH +``` ## Access Matrix -Each agent needs explicit credentials to each environment: - -| Environment | Abiba (pi) | Tanko (Hermes) | Mumuni (Hermes) | -|---|---|---|---| +| Resource | Abiba (CT 100) | Tanko (CT 122) | Mumuni (CT 114) | +|----------|---------------|----------------|-----------------| +| **PVE API** (minipve:443) | ✅ API token | ❌ | ❌ | +| **docker-vm** (SSH .7) | ✅ root SSH | ❌ | ❌ | +| **CT 116** (SSH) | ✅ root SSH | ❌ | ❌ | | **Gitea** | ✅ API token | Need to add | Need to add | -| **CT 116 (LiteLLM)** | ✅ SSH key | Need to add | ❌ (has own CTs) | -| **Zulip API** | ✅ Bot token | ✅ Bot token | ✅ Bot token | +| **Zulip API** | ✅ bot token | ✅ bot token | ✅ bot token | | **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge | +| **Netbird** (SSH .17) | ✅ root SSH | ❌ | ❌ | | **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH | -## The Gap (What's Missing) +## Priority Matrix for Auto-Remediation -Currently, the contracts CAN: +| Priority | Condition | Action | SLA | +|----------|-----------|--------|-----| +| **P0** | Critical CT/VM stopped | Auto-start via PVE API | 2 min | +| **P0** | LiteLLM/router down | Auto-restart container | 3 min | +| **P1** | Firecrawl/SearXNG down | Auto-restart container | 5 min | +| **P1** | Docker disk > 85% | Auto-prune | 5 min | +| **P2** | Storage usage > 80% | Alert + log | 15 min | +| **P2** | CT resource pressure | Alert + log | 15 min | +| **P3** | Non-critical container down | Alert + log | 30 min | +| **P3** | PBS backup missed | Alert + log | 1h | -| Capability | Works? | How | -|---|---|---| -| Read container state | ✅ SSH to CT 116 | -| Restart containers | ✅ SSH + docker compose | -| Commit to Gitea | ✅ API token | -| Alert via Zulip | ✅ Bot API | -| Log to KG | ✅ MCP bridge | +## Escalation Path -But they CANNOT yet: +``` +1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access + │ │ + ▼ ▼ + Kwame (Zulip DM) Mumuni (for authentik) + Tanko (for hermes agents) +``` -| Gap | Why | Fix | -|---|---|---| -| Push code to Tanko's CT | No SSH key for CT 112 | Add Abiba's key to CT 112 | -| Restart Hermes gateway | No remote command | Add `hermes gateway restart` via SSH | -| Auto-merge PRs | Needs Gitea write token | Already have it, need to wire | +## Known Gaps -## Reference Implementation - -The `litellm-self-heal` contract already implements most of this pattern: -1. It checks conditions (container health, endpoints) -2. It enforces fixes (docker compose up) -3. It logs results (knowledge graph) -4. It alerts via Zulip DM - -The same pattern extends to any environment where the agent has credentials. +| Gap | What's Needed | +|-----|--------------| +| Authentik SSH | No agent has SSH — Mumuni needs key added | +| Koby/Koonimo stopped | CTs 111/113 are offline — need human intervention | +| Mumuni PVE access | No PVE token — needs `monitoring@pve!mumuni` | +| docker-vm backups | VM 109 isn't running PBS agent — backup status unknown | +| Netbird redundancy | Single point of failure for VPN |