SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: expand infrastructure-control pattern — Proxmox, Docker, storage, network

Browse Source 
Full expansion of the infrastructure-control pattern covering:

Layer 1 — Proxmox Cluster (5 nodes)
  - 4 monitoring rules (offline, resource pressure, stopped VMs/CTs, storage)
  - Auto-start for 8 critical CTs/VMs on unexpected stop
  - Node resource threshold alerting

Layer 2 — Docker Ecosystems (3 hosts, 22 containers)
  - 6 monitoring rules across docker-vm, syslog-api, netbird
  - Auto-restart for critical containers and stacks
  - Disk pressure auto-prune

Layer 3 — Storage Fabric (3.6TB + 7.3TB NFS)
  - Mount availability monitoring
  - Growth rate tracking for high-use volumes
  - PBS backup compliance checking

Layer 4 — Network Services (DNS, auth, git, chat, VPN)
  - Service endpoint monitoring
  - Escalation to Mumuni for authentik issues

Layer 5 — Agent Health (6 agents across 3 platforms)
  - CT auto-start for stopped agents
  - Health endpoint verification

Full access matrix and priority-based SLA table.
This commit is contained in:
root
2026-06-27 13:11:15 +00:00
parent 4c6b0410ea
commit e090b27e20
1 changed files with 320 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-control.prose.md
+322 -108
View File
@@ -2,141 +2,355 @@
kind: pattern
name: infrastructure-control
description: >
Architectural pattern showing how OpenProse contracts connect to
Gitea, containers, and remote agents to enforce conditions across
all environments. Not an executable contract — a reference model.
Full infrastructure monitoring and control pattern covering the 5-node
Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric,
and network services. Declares what agents can observe, what they can
enforce, and where human escalation is required.
---
## The Full Control Stack
## Architecture Overview
```
┌─────────────────────────┐
│ OpenProse Contract │
│ (declares what's true) │
└──────────┬──────────────┘
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Agent │ │ Agent │ │ Agent │
│ (pi) │ │ (Tanko) │ │ (Mumuni) │
│ CT 100 │ │ CT 112 │ │ CT 114 │
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘
┌──────────────────────────────────────────┐
│ OpenProse Contracts │
│ (monitor → diagnose → remediate → log) │
└─────┬──────────┬──────────┬──────────────┘
│ │ │
└──────────────────┼────────────────────┘
▼
┌─────────────────┐
│ Gitea Repos │
│ (source of │
│ truth for │
│ code + │
│ contracts) │
└────────┬────────┘
┌────────────┼──────────┼──────────┼────────────┐
▼ ▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐
│ Proxmox │ │ Docker │ │Storage│ │Network│ │ Agents │
│ Cluster │ │Ecosystems│ │Fabric │ │Services│ │(health) │
│ 5 nodes │ │ 3 hosts │ │NFS+ZFS│ │DNS+Auth│ │6 agents │
└──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘
```
## Layer 1: Proxmox Cluster
### Cluster Topology
```
minipve (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110,
│ mumuni CT 114, syslog-api CT 116,
│ jitsi CT 118
│
┌──────────────────┼────────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ CT 116 │ │ Authentik │ │ Zulip │
│ (LiteLLM, │ │ CT .11 │ │ CT .117 │
│ nginx, │ │ (OIDC auth) │ │ (messaging) │
│ router) │ │ │ │ │
└─────────────┘ └──────────────┘ └──────────────┘
amdpve (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105,
│ tanko CT 112, tdunna CT 111,
│ baggy CT 113, scottdenya CT 115
│
storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106,
│ PBS CT 107, media CT 108,
│ zulip CT 117
│
acerpve (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102
│
ocupve (.5) ──── 12C/14GB ─── ocu-llm VM 103
```
## How Each Layer Connects
### 1. Code → Gitea
The contract enforces code state via git operations:
```yaml
## Enforce
- git pull origin main on CT 116 — ensures latest config is deployed
- git diff HEAD...origin/main — warns if drift is detected
- git commit + push after auto-fix — changes are versioned
```
**What the agent does:** Runs `git` commands via SSH to check and update repos.
**What enforces it:** Agent has SSH keys and Gitea credentials.
### 2. Contracts → Agents
Each agent runs OpenProse contracts in its own environment:
| Agent | Runs contracts via | Can reach |
|---|---|---|
| **Abiba (pi)** | `prose run` in-session | CT 116 (SSH), Zulip API, Gitea API |
| **Tanko (Hermes)** | Hermes Gateway with OpenProse skill | CT 112, Zulip API, Gitea API |
| **Mumuni (Hermes)** | Hermes Gateway with OpenProse skill | CT 114, Zulip API, Gitea API |
**What enforces it:** Each agent has SSH keys to the environments it manages.
### 3. Contracts → Containers
The contract enforces container state via Docker commands:
### Maintains (Monitoring)
```yaml
## Maintains
- harness-litellm.status == "healthy"
- harness-nginx.status == "healthy"
## Enforce
- If container unhealthy → `docker compose up -d <name>`
- Requires: SSH access to host
- Verifies: `docker ps` after restart
- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime
- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage
- cluster.storage: each storage target's usage %, active/inactive
- cluster.ha: high-availability state and fencing status
```
**What the agent does:** SSH to the host, run Docker commands.
**What enforces it:** SSH key + Docker socket access on the target CT.
### Rules (Monitoring)
### 4. Contracts → Remote Agents (Tanko, Mumuni)
When a contract needs another agent to act:
```
Rule PM-1: Node Offline
Detect → PVE API shows node status != "online"
Action → Alert via Zulip DM with node name and last known uptime
Escalate → If >2 nodes offline, page owner
Requires: PVE API token
Rule PM-2: Node Resource Pressure
Detect → CPU > 80% or RAM > 85% for 5 consecutive checks
Action → Log to KG, alert via Zulip DM with resource snapshot
Escalate → If sustained >90% for 1h, page owner
Requires: PVE API token
Rule PM-3: VM/CT Unexpectedly Stopped
Detect → PVE API shows running VM now "stopped"
Action → `pct start <vmid>` or `qm start <vmid>` via PVE API
Verify → Re-check status after 30s
Escalate → If still stopped after 2 retries, alert owner
Requires: PVE API token with appropriate permissions
Rule PM-4: Storage Usage Warning
Detect → Storage pool usage > 80%
Action → Log to KG, alert via Zulip DM
Escalate → If >90%, immediate alert (risk of VM freeze)
Requires: PVE API
```
### Control (Remediation)
```yaml
## If out of scope
- If Authentik server down → relay to Mumuni (has access)
- Creates relay message with diagnostic data
- Mumuni's contract picks it up and acts
## Enforce
- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped
- Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116),
docker-vm (109), gitea (110), authentik (104), zulip (117)
- PM-4 storage: if mediastore > 85%, alert with biggest consumers
## Prohibited
- Never migrate VMs/CTs between nodes automatically
- Never delete storage volumes
- Never modify cluster HA configuration
```
**What the agent does:** Creates a relay message via RA-H OS.
**What enforces it:** Mumuni's own contracts check relays on wake.
## Layer 2: Docker Ecosystems
### Docker Hosts
```yaml
Host: docker-vm (192.168.68.7, storepve VM 109)
Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used)
Stacks:
firecrawl: api, rabbitmq, postgres, playwright, redis
searxng: searxng (:8888), valkey
home_stack: jdownloader (:5800), bentopdf (:8989),
pulse (:7655), audiobookshelf (:13378)
Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS)
Host: syslog-api (192.168.68.116, minipve CT 116)
Resources: 6GB RAM, 2 vCPU, 39GB disk
Stacks:
harness: litellm (:4001), nginx (:80), router (:9000),
postgres, redis, dashboard (:3000)
Host: netbird (72.61.0.17, external VM)
Stacks:
netbird: server, dashboard, proxy, crowdsec, traefik
```
### Maintains (Monitoring)
```yaml
## Maintains
- docker.hosts: each host's status, container count, resource usage
- docker.containers: each container's status (running/restarting/exited),
uptime, restart count, image version
- docker.stacks: each compose stack's service count and health
- docker.resources: per-host disk usage, Docker system df
```
### Rules (Monitoring)
```
Rule DC-1: Container Not Running
Detect → `docker ps` shows "exited" or "restarting" for critical container
Action → `docker compose up -d <service>` on the affected stack
Verify → Re-check after 10s
Escalate → If still failed after 3 retries, alert owner
Requires: SSH to docker-vm or CT 116
Rule DC-2: Container Restart Looping
Detect → Container restart count > 5 in last hour
Action → `docker logs --tail 50 <container>` → diagnose → KG log
Escalate → Always alert owner with diagnostic excerpt
Requires: SSH to docker-vm or CT 116
Rule DC-3: Docker Host Disk Pressure
Detect → `docker system df` shows < 10% reclaimable or disk > 85%
Action → `docker system prune -af` (safe: removes unused images/containers)
Verify → Re-check disk usage
Escalate → If still >85% after prune, alert owner
Requires: SSH to docker-vm
Rule DC-4: Stack Health Check Failed
Detect → Container HEALTHCHECK status shows "unhealthy"
Action → `docker compose restart <service>`
Verify → Re-check health after 15s
Escalate → After 2 retries
Requires: SSH to affected host
Rule DC-5: Firecrawl API Down
Detect → HTTP GET :3002/health returns non-200
Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1`
Escalate → If still down after 2 retries
Requires: SSH to docker-vm
Rule DC-6: SearXNG Down
Detect → HTTP GET :8888 returns non-200
Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng`
Escalate → If still down after 2 retries
Requires: SSH to docker-vm
```
### Control (Remediation)
```yaml
## Enforce
- DC-1 auto-restart: restart critical containers when exited
- DC-3 auto-prune: prune unused Docker artifacts weekly
- DC-4/5/6: restart unhealthy stacks with verification
## Prohibited
- Never remove volumes (data loss risk)
- Never modify docker-compose files remotely
- Never update container images without owner approval
```
## Layer 3: Storage Fabric
### Storage Topology
```
storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used)
└── audiobookshelf, JDownloader downloads
storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used)
└── media library, configs
storepve ── PBS CT 107 ──► backup target for all VMs/CTs
```
### Maintains
```yaml
## Maintains
- storage.nfs_mounts: each mount's availability, usage %, inode count
- storage.pbs: last successful backup timestamp, backup schedule compliance
- storage.mediastore_growth: weekly growth rate (high-use volume)
```
### Rules
```
Rule ST-1: NFS Mount Unreachable
Detect → `df` or `mountpoint -q` fails for NFS path
Action → Check storepve NFS server status via PVE API
Escalate → Immediate alert — containers may be read-only or crashing
Requires: PVE API + SSH to docker-vm
Rule ST-2: Mediadisk Growth Alert
Detect → mediastore (7.3TB, 75%) growing > 50GB/week
Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10`
Log → KG node with growth trend
Escalate → If projected to exceed 90% within 30 days
Rule ST-3: PBS Backup Compliance
Detect → Any CT/VM without a backup in the last 48h
Action → Log to KG, alert via Zulip DM
Requires: PVE API to check PBS tasks
```
## Layer 4: Network Services
### Service Map
| Service | Host | IP | Port | Purpose |
|---------|------|----|------|---------|
| adguard | CT 102 | .102 | 53, 80, 443 | DNS + DHCP |
| authentik | CT 104 | .104 | 443 | OIDC auth provider |
| gitea | CT 110 | .110 | 3000, 22 | Git hosting |
| zulip | CT 117 | .117 | 443 | Team chat |
| netbird | 72.61.0.17 | .17 | 443, 8006 | VPN controller |
| minipve | .12 | .12 | 443, 8006 | PVE API endpoint |
| PBS | CT 107 | .107 | 8007 | Proxmox Backup Server |
### Maintains
```yaml
## Maintains
- network.dns_resolution: critical hostnames resolve correctly
- network.auth_endpoints: authentik OIDC login flow works
- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse)
```
### Rules
```
Rule NW-1: DNS Failure
Detect → `host` or `nslookup` fails for known hostnames
Action → Check adguard CT 102 status via PVE API
Escalate → If authentik/gitea unreachable due to DNS → page owner
Requires: PVE API + dig/nslookup
Rule NW-2: Auth Endpoint Down
Detect → `GET https://auth.sysloggh.net` returns non-200
Action → Check authentik CT 104 via PVE API → start if stopped
Escalate → Can't fix: relay to Mumuni (has authentik SSH access)
Requires: PVE API
Rule NW-3: Public Endpoint Unreachable
Detect → Any public URL fails (litellm, git, chat, pulse)
Action → Check container + nginx health on relevant host
Requires: SSH to affected host + curl
```
## Layer 5: Agent Health
### Agent Registry
| Agent | Platform | CT | Status Endpoint | Last Seen |
|-------|----------|----|----------------|-----------|
| abiba | pi | 100 | :9200/health | live |
| tanko | Hermes | 122 | (via Gateway) | live |
| mumuni | Hermes | 114 | (via Gateway) | live |
| kagentz | Agent Zero | 105 | (via PVE) | running |
| koby | Hermes | 111 | (via PVE) | stopped |
| koonimo | Hermes | 113 | (via PVE) | stopped |
### Rules
```
Rule AG-1: Agent VM/CT Stopped
Detect → PVE API shows agent CT as "stopped"
Action → `pct start <vmid>` via PVE API
Verify → Re-check after 15s
Escalate → If persistent, alert owner
Requires: PVE API
Rule AG-2: Agent Not Responding
Detect → Health endpoint non-200 or Gateway disconnected
Action → Check CT is running, then check service status
Requires: PVE API + SSH
```
## Access Matrix
Each agent needs explicit credentials to each environment:
| Environment | Abiba (pi) | Tanko (Hermes) | Mumuni (Hermes) |
|---|---|---|---|
| Resource | Abiba (CT 100) | Tanko (CT 122) | Mumuni (CT 114) |
|----------|---------------|----------------|-----------------|
| **PVE API** (minipve:443) | ✅ API token | ❌ | ❌ |
| **docker-vm** (SSH .7) | ✅ root SSH | ❌ | ❌ |
| **CT 116** (SSH) | ✅ root SSH | ❌ | ❌ |
| **Gitea** | ✅ API token | Need to add | Need to add |
| **CT 116 (LiteLLM)** | ✅ SSH key | Need to add | ❌ (has own CTs) |
| **Zulip API** | ✅ Bot token | ✅ Bot token | ✅ Bot token |
| **Zulip API** | ✅ bot token | ✅ bot token | ✅ bot token |
| **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge |
| **Netbird** (SSH .17) | ✅ root SSH | ❌ | ❌ |
| **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH |
## The Gap (What's Missing)
## Priority Matrix for Auto-Remediation
Currently, the contracts CAN:
| Priority | Condition | Action | SLA |
|----------|-----------|--------|-----|
| **P0** | Critical CT/VM stopped | Auto-start via PVE API | 2 min |
| **P0** | LiteLLM/router down | Auto-restart container | 3 min |
| **P1** | Firecrawl/SearXNG down | Auto-restart container | 5 min |
| **P1** | Docker disk > 85% | Auto-prune | 5 min |
| **P2** | Storage usage > 80% | Alert + log | 15 min |
| **P2** | CT resource pressure | Alert + log | 15 min |
| **P3** | Non-critical container down | Alert + log | 30 min |
| **P3** | PBS backup missed | Alert + log | 1h |
| Capability | Works? | How |
|---|---|---|
| Read container state | ✅ SSH to CT 116 |
| Restart containers | ✅ SSH + docker compose |
| Commit to Gitea | ✅ API token |
| Alert via Zulip | ✅ Bot API |
| Log to KG | ✅ MCP bridge |
## Escalation Path
But they CANNOT yet:
```
1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access
│ │
▼ ▼
Kwame (Zulip DM) Mumuni (for authentik)
Tanko (for hermes agents)
```
| Gap | Why | Fix |
|---|---|---|
| Push code to Tanko's CT | No SSH key for CT 112 | Add Abiba's key to CT 112 |
| Restart Hermes gateway | No remote command | Add `hermes gateway restart` via SSH |
| Auto-merge PRs | Needs Gitea write token | Already have it, need to wire |
## Known Gaps
## Reference Implementation
The `litellm-self-heal` contract already implements most of this pattern:
1. It checks conditions (container health, endpoints)
2. It enforces fixes (docker compose up)
3. It logs results (knowledge graph)
4. It alerts via Zulip DM
The same pattern extends to any environment where the agent has credentials.
| Gap | What's Needed |
|-----|--------------|
| Authentik SSH | No agent has SSH — Mumuni needs key added |
| Koby/Koonimo stopped | CTs 111/113 are offline — need human intervention |
| Mumuni PVE access | No PVE token — needs `monitoring@pve!mumuni` |
| docker-vm backups | VM 109 isn't running PBS agent — backup status unknown |
| Netbird redundancy | Single point of failure for VPN |