feat: expand infrastructure-control pattern — Proxmox, Docker, storage, network
Full expansion of the infrastructure-control pattern covering: Layer 1 — Proxmox Cluster (5 nodes) - 4 monitoring rules (offline, resource pressure, stopped VMs/CTs, storage) - Auto-start for 8 critical CTs/VMs on unexpected stop - Node resource threshold alerting Layer 2 — Docker Ecosystems (3 hosts, 22 containers) - 6 monitoring rules across docker-vm, syslog-api, netbird - Auto-restart for critical containers and stacks - Disk pressure auto-prune Layer 3 — Storage Fabric (3.6TB + 7.3TB NFS) - Mount availability monitoring - Growth rate tracking for high-use volumes - PBS backup compliance checking Layer 4 — Network Services (DNS, auth, git, chat, VPN) - Service endpoint monitoring - Escalation to Mumuni for authentik issues Layer 5 — Agent Health (6 agents across 3 platforms) - CT auto-start for stopped agents - Health endpoint verification Full access matrix and priority-based SLA table.
This commit is contained in:
root
+319
-105
@@ -2,141 +2,355 @@
|
|||||||
kind: pattern
|
kind: pattern
|
||||||
name: infrastructure-control
|
name: infrastructure-control
|
||||||
description: >
|
description: >
|
||||||
Architectural pattern showing how OpenProse contracts connect to
|
Full infrastructure monitoring and control pattern covering the 5-node
|
||||||
Gitea, containers, and remote agents to enforce conditions across
|
Proxmox cluster, 3 Docker ecosystems (22 containers), storage fabric,
|
||||||
all environments. Not an executable contract — a reference model.
|
and network services. Declares what agents can observe, what they can
|
||||||
|
enforce, and where human escalation is required.
|
||||||
---
|
---
|
||||||
|
|
||||||
## The Full Control Stack
|
## Architecture Overview
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────────────────┐
|
┌──────────────────────────────────────────┐
|
||||||
│ OpenProse Contract │
|
│ OpenProse Contracts │
|
||||||
│ (declares what's true) │
|
│ (monitor → diagnose → remediate → log) │
|
||||||
└──────────┬──────────────┘
|
└─────┬──────────┬──────────┬──────────────┘
|
||||||
│
|
|
||||||
┌────────────────────┼────────────────────┐
|
|
||||||
▼ ▼ ▼
|
|
||||||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
||||||
│ Agent │ │ Agent │ │ Agent │
|
|
||||||
│ (pi) │ │ (Tanko) │ │ (Mumuni) │
|
|
||||||
│ CT 100 │ │ CT 112 │ │ CT 114 │
|
|
||||||
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘
|
|
||||||
│ │ │
|
│ │ │
|
||||||
└──────────────────┼────────────────────┘
|
┌────────────┼──────────┼──────────┼────────────┐
|
||||||
▼
|
▼ ▼ ▼ ▼ ▼
|
||||||
┌─────────────────┐
|
┌──────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌──────────┐
|
||||||
│ Gitea Repos │
|
│ Proxmox │ │ Docker │ │Storage│ │Network│ │ Agents │
|
||||||
│ (source of │
|
│ Cluster │ │Ecosystems│ │Fabric │ │Services│ │(health) │
|
||||||
│ truth for │
|
│ 5 nodes │ │ 3 hosts │ │NFS+ZFS│ │DNS+Auth│ │6 agents │
|
||||||
│ code + │
|
└──────────┘ └──────────┘ └──────┘ └──────┘ └──────────┘
|
||||||
│ contracts) │
|
|
||||||
└────────┬────────┘
|
|
||||||
│
|
|
||||||
┌──────────────────┼────────────────────┐
|
|
||||||
▼ ▼ ▼
|
|
||||||
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
|
|
||||||
│ CT 116 │ │ Authentik │ │ Zulip │
|
|
||||||
│ (LiteLLM, │ │ CT .11 │ │ CT .117 │
|
|
||||||
│ nginx, │ │ (OIDC auth) │ │ (messaging) │
|
|
||||||
│ router) │ │ │ │ │
|
|
||||||
└─────────────┘ └──────────────┘ └──────────────┘
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## How Each Layer Connects
|
## Layer 1: Proxmox Cluster
|
||||||
|
|
||||||
### 1. Code → Gitea
|
### Cluster Topology
|
||||||
The contract enforces code state via git operations:
|
|
||||||
|
|
||||||
```yaml
|
```
|
||||||
## Enforce
|
minipve (.12) ─── 16C/30GB ─── authentik CT 104, gitea CT 110,
|
||||||
- git pull origin main on CT 116 — ensures latest config is deployed
|
│ mumuni CT 114, syslog-api CT 116,
|
||||||
- git diff HEAD...origin/main — warns if drift is detected
|
│ jitsi CT 118
|
||||||
- git commit + push after auto-fix — changes are versioned
|
│
|
||||||
|
amdpve (.15) ─── 32C/62GB ─── abiba CT 100, kagentz CT 105,
|
||||||
|
│ tanko CT 112, tdunna CT 111,
|
||||||
|
│ baggy CT 113, scottdenya CT 115
|
||||||
|
│
|
||||||
|
storepve (.6) ──── 28C/31GB ─── docker-vm VM 109, ra-h-os CT 106,
|
||||||
|
│ PBS CT 107, media CT 108,
|
||||||
|
│ zulip CT 117
|
||||||
|
│
|
||||||
|
acerpve (.9) ──── 28C/31GB ─── llm-gpu VM 101, adguard CT 102
|
||||||
|
│
|
||||||
|
ocupve (.5) ──── 12C/14GB ─── ocu-llm VM 103
|
||||||
```
|
```
|
||||||
|
|
||||||
**What the agent does:** Runs `git` commands via SSH to check and update repos.
|
### Maintains (Monitoring)
|
||||||
**What enforces it:** Agent has SSH keys and Gitea credentials.
|
|
||||||
|
|
||||||
### 2. Contracts → Agents
|
|
||||||
Each agent runs OpenProse contracts in its own environment:
|
|
||||||
|
|
||||||
| Agent | Runs contracts via | Can reach |
|
|
||||||
|---|---|---|
|
|
||||||
| **Abiba (pi)** | `prose run` in-session | CT 116 (SSH), Zulip API, Gitea API |
|
|
||||||
| **Tanko (Hermes)** | Hermes Gateway with OpenProse skill | CT 112, Zulip API, Gitea API |
|
|
||||||
| **Mumuni (Hermes)** | Hermes Gateway with OpenProse skill | CT 114, Zulip API, Gitea API |
|
|
||||||
|
|
||||||
**What enforces it:** Each agent has SSH keys to the environments it manages.
|
|
||||||
|
|
||||||
### 3. Contracts → Containers
|
|
||||||
The contract enforces container state via Docker commands:
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
## Maintains
|
## Maintains
|
||||||
- harness-litellm.status == "healthy"
|
- cluster.nodes: each node's status (online/offline), CPU%, RAM%, uptime
|
||||||
- harness-nginx.status == "healthy"
|
- cluster.vms: each VM/CT's status (running/stopped), CPU, RAM, disk usage
|
||||||
|
- cluster.storage: each storage target's usage %, active/inactive
|
||||||
## Enforce
|
- cluster.ha: high-availability state and fencing status
|
||||||
- If container unhealthy → `docker compose up -d <name>`
|
|
||||||
- Requires: SSH access to host
|
|
||||||
- Verifies: `docker ps` after restart
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**What the agent does:** SSH to the host, run Docker commands.
|
### Rules (Monitoring)
|
||||||
**What enforces it:** SSH key + Docker socket access on the target CT.
|
|
||||||
|
|
||||||
### 4. Contracts → Remote Agents (Tanko, Mumuni)
|
```
|
||||||
When a contract needs another agent to act:
|
Rule PM-1: Node Offline
|
||||||
|
Detect → PVE API shows node status != "online"
|
||||||
|
Action → Alert via Zulip DM with node name and last known uptime
|
||||||
|
Escalate → If >2 nodes offline, page owner
|
||||||
|
Requires: PVE API token
|
||||||
|
|
||||||
|
Rule PM-2: Node Resource Pressure
|
||||||
|
Detect → CPU > 80% or RAM > 85% for 5 consecutive checks
|
||||||
|
Action → Log to KG, alert via Zulip DM with resource snapshot
|
||||||
|
Escalate → If sustained >90% for 1h, page owner
|
||||||
|
Requires: PVE API token
|
||||||
|
|
||||||
|
Rule PM-3: VM/CT Unexpectedly Stopped
|
||||||
|
Detect → PVE API shows running VM now "stopped"
|
||||||
|
Action → `pct start <vmid>` or `qm start <vmid>` via PVE API
|
||||||
|
Verify → Re-check status after 30s
|
||||||
|
Escalate → If still stopped after 2 retries, alert owner
|
||||||
|
Requires: PVE API token with appropriate permissions
|
||||||
|
|
||||||
|
Rule PM-4: Storage Usage Warning
|
||||||
|
Detect → Storage pool usage > 80%
|
||||||
|
Action → Log to KG, alert via Zulip DM
|
||||||
|
Escalate → If >90%, immediate alert (risk of VM freeze)
|
||||||
|
Requires: PVE API
|
||||||
|
```
|
||||||
|
|
||||||
|
### Control (Remediation)
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
## If out of scope
|
## Enforce
|
||||||
- If Authentik server down → relay to Mumuni (has access)
|
- PM-3 auto-start: start critical VMs/CTs when unexpectedly stopped
|
||||||
- Creates relay message with diagnostic data
|
- Critical: abiba (100), tanko (112), mumuni (114), syslog-api (116),
|
||||||
- Mumuni's contract picks it up and acts
|
docker-vm (109), gitea (110), authentik (104), zulip (117)
|
||||||
|
- PM-4 storage: if mediastore > 85%, alert with biggest consumers
|
||||||
|
|
||||||
|
## Prohibited
|
||||||
|
- Never migrate VMs/CTs between nodes automatically
|
||||||
|
- Never delete storage volumes
|
||||||
|
- Never modify cluster HA configuration
|
||||||
```
|
```
|
||||||
|
|
||||||
**What the agent does:** Creates a relay message via RA-H OS.
|
## Layer 2: Docker Ecosystems
|
||||||
**What enforces it:** Mumuni's own contracts check relays on wake.
|
|
||||||
|
### Docker Hosts
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
Host: docker-vm (192.168.68.7, storepve VM 109)
|
||||||
|
Resources: 10GB RAM, 4 vCPU, 158GB disk (33% used)
|
||||||
|
Stacks:
|
||||||
|
firecrawl: api, rabbitmq, postgres, playwright, redis
|
||||||
|
searxng: searxng (:8888), valkey
|
||||||
|
home_stack: jdownloader (:5800), bentopdf (:8989),
|
||||||
|
pulse (:7655), audiobookshelf (:13378)
|
||||||
|
Mounts: /media/storage (3.6TB NFS), /media/mediastore (7.3TB NFS)
|
||||||
|
|
||||||
|
Host: syslog-api (192.168.68.116, minipve CT 116)
|
||||||
|
Resources: 6GB RAM, 2 vCPU, 39GB disk
|
||||||
|
Stacks:
|
||||||
|
harness: litellm (:4001), nginx (:80), router (:9000),
|
||||||
|
postgres, redis, dashboard (:3000)
|
||||||
|
|
||||||
|
Host: netbird (72.61.0.17, external VM)
|
||||||
|
Stacks:
|
||||||
|
netbird: server, dashboard, proxy, crowdsec, traefik
|
||||||
|
```
|
||||||
|
|
||||||
|
### Maintains (Monitoring)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
## Maintains
|
||||||
|
- docker.hosts: each host's status, container count, resource usage
|
||||||
|
- docker.containers: each container's status (running/restarting/exited),
|
||||||
|
uptime, restart count, image version
|
||||||
|
- docker.stacks: each compose stack's service count and health
|
||||||
|
- docker.resources: per-host disk usage, Docker system df
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rules (Monitoring)
|
||||||
|
|
||||||
|
```
|
||||||
|
Rule DC-1: Container Not Running
|
||||||
|
Detect → `docker ps` shows "exited" or "restarting" for critical container
|
||||||
|
Action → `docker compose up -d <service>` on the affected stack
|
||||||
|
Verify → Re-check after 10s
|
||||||
|
Escalate → If still failed after 3 retries, alert owner
|
||||||
|
Requires: SSH to docker-vm or CT 116
|
||||||
|
|
||||||
|
Rule DC-2: Container Restart Looping
|
||||||
|
Detect → Container restart count > 5 in last hour
|
||||||
|
Action → `docker logs --tail 50 <container>` → diagnose → KG log
|
||||||
|
Escalate → Always alert owner with diagnostic excerpt
|
||||||
|
Requires: SSH to docker-vm or CT 116
|
||||||
|
|
||||||
|
Rule DC-3: Docker Host Disk Pressure
|
||||||
|
Detect → `docker system df` shows < 10% reclaimable or disk > 85%
|
||||||
|
Action → `docker system prune -af` (safe: removes unused images/containers)
|
||||||
|
Verify → Re-check disk usage
|
||||||
|
Escalate → If still >85% after prune, alert owner
|
||||||
|
Requires: SSH to docker-vm
|
||||||
|
|
||||||
|
Rule DC-4: Stack Health Check Failed
|
||||||
|
Detect → Container HEALTHCHECK status shows "unhealthy"
|
||||||
|
Action → `docker compose restart <service>`
|
||||||
|
Verify → Re-check health after 15s
|
||||||
|
Escalate → After 2 retries
|
||||||
|
Requires: SSH to affected host
|
||||||
|
|
||||||
|
Rule DC-5: Firecrawl API Down
|
||||||
|
Detect → HTTP GET :3002/health returns non-200
|
||||||
|
Action → `docker compose -f /opt/search-stack/firecrawl-source/docker-compose.yaml restart firecrawl-api-1`
|
||||||
|
Escalate → If still down after 2 retries
|
||||||
|
Requires: SSH to docker-vm
|
||||||
|
|
||||||
|
Rule DC-6: SearXNG Down
|
||||||
|
Detect → HTTP GET :8888 returns non-200
|
||||||
|
Action → `docker compose -f /opt/search-stack/searxng/docker-compose.yml restart searxng`
|
||||||
|
Escalate → If still down after 2 retries
|
||||||
|
Requires: SSH to docker-vm
|
||||||
|
```
|
||||||
|
|
||||||
|
### Control (Remediation)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
## Enforce
|
||||||
|
- DC-1 auto-restart: restart critical containers when exited
|
||||||
|
- DC-3 auto-prune: prune unused Docker artifacts weekly
|
||||||
|
- DC-4/5/6: restart unhealthy stacks with verification
|
||||||
|
|
||||||
|
## Prohibited
|
||||||
|
- Never remove volumes (data loss risk)
|
||||||
|
- Never modify docker-compose files remotely
|
||||||
|
- Never update container images without owner approval
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layer 3: Storage Fabric
|
||||||
|
|
||||||
|
### Storage Topology
|
||||||
|
|
||||||
|
```
|
||||||
|
storepve (.6) ── NFS ──► docker-vm: /media/storage (3.6TB, 8% used)
|
||||||
|
└── audiobookshelf, JDownloader downloads
|
||||||
|
storepve (.6) ── NFS ──► docker-vm: /media/mediastore (7.3TB, 75% used)
|
||||||
|
└── media library, configs
|
||||||
|
storepve ── PBS CT 107 ──► backup target for all VMs/CTs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Maintains
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
## Maintains
|
||||||
|
- storage.nfs_mounts: each mount's availability, usage %, inode count
|
||||||
|
- storage.pbs: last successful backup timestamp, backup schedule compliance
|
||||||
|
- storage.mediastore_growth: weekly growth rate (high-use volume)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rules
|
||||||
|
|
||||||
|
```
|
||||||
|
Rule ST-1: NFS Mount Unreachable
|
||||||
|
Detect → `df` or `mountpoint -q` fails for NFS path
|
||||||
|
Action → Check storepve NFS server status via PVE API
|
||||||
|
Escalate → Immediate alert — containers may be read-only or crashing
|
||||||
|
Requires: PVE API + SSH to docker-vm
|
||||||
|
|
||||||
|
Rule ST-2: Mediadisk Growth Alert
|
||||||
|
Detect → mediastore (7.3TB, 75%) growing > 50GB/week
|
||||||
|
Action → Identify top consumers: `du -sh /media/mediastore/* | sort -rh | head 10`
|
||||||
|
Log → KG node with growth trend
|
||||||
|
Escalate → If projected to exceed 90% within 30 days
|
||||||
|
|
||||||
|
Rule ST-3: PBS Backup Compliance
|
||||||
|
Detect → Any CT/VM without a backup in the last 48h
|
||||||
|
Action → Log to KG, alert via Zulip DM
|
||||||
|
Requires: PVE API to check PBS tasks
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layer 4: Network Services
|
||||||
|
|
||||||
|
### Service Map
|
||||||
|
|
||||||
|
| Service | Host | IP | Port | Purpose |
|
||||||
|
|---------|------|----|------|---------|
|
||||||
|
| adguard | CT 102 | .102 | 53, 80, 443 | DNS + DHCP |
|
||||||
|
| authentik | CT 104 | .104 | 443 | OIDC auth provider |
|
||||||
|
| gitea | CT 110 | .110 | 3000, 22 | Git hosting |
|
||||||
|
| zulip | CT 117 | .117 | 443 | Team chat |
|
||||||
|
| netbird | 72.61.0.17 | .17 | 443, 8006 | VPN controller |
|
||||||
|
| minipve | .12 | .12 | 443, 8006 | PVE API endpoint |
|
||||||
|
| PBS | CT 107 | .107 | 8007 | Proxmox Backup Server |
|
||||||
|
|
||||||
|
### Maintains
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
## Maintains
|
||||||
|
- network.dns_resolution: critical hostnames resolve correctly
|
||||||
|
- network.auth_endpoints: authentik OIDC login flow works
|
||||||
|
- network.external_access: public endpoints respond (litellm, git, chat, auth, pulse)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rules
|
||||||
|
|
||||||
|
```
|
||||||
|
Rule NW-1: DNS Failure
|
||||||
|
Detect → `host` or `nslookup` fails for known hostnames
|
||||||
|
Action → Check adguard CT 102 status via PVE API
|
||||||
|
Escalate → If authentik/gitea unreachable due to DNS → page owner
|
||||||
|
Requires: PVE API + dig/nslookup
|
||||||
|
|
||||||
|
Rule NW-2: Auth Endpoint Down
|
||||||
|
Detect → `GET https://auth.sysloggh.net` returns non-200
|
||||||
|
Action → Check authentik CT 104 via PVE API → start if stopped
|
||||||
|
Escalate → Can't fix: relay to Mumuni (has authentik SSH access)
|
||||||
|
Requires: PVE API
|
||||||
|
|
||||||
|
Rule NW-3: Public Endpoint Unreachable
|
||||||
|
Detect → Any public URL fails (litellm, git, chat, pulse)
|
||||||
|
Action → Check container + nginx health on relevant host
|
||||||
|
Requires: SSH to affected host + curl
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layer 5: Agent Health
|
||||||
|
|
||||||
|
### Agent Registry
|
||||||
|
|
||||||
|
| Agent | Platform | CT | Status Endpoint | Last Seen |
|
||||||
|
|-------|----------|----|----------------|-----------|
|
||||||
|
| abiba | pi | 100 | :9200/health | live |
|
||||||
|
| tanko | Hermes | 122 | (via Gateway) | live |
|
||||||
|
| mumuni | Hermes | 114 | (via Gateway) | live |
|
||||||
|
| kagentz | Agent Zero | 105 | (via PVE) | running |
|
||||||
|
| koby | Hermes | 111 | (via PVE) | stopped |
|
||||||
|
| koonimo | Hermes | 113 | (via PVE) | stopped |
|
||||||
|
|
||||||
|
### Rules
|
||||||
|
|
||||||
|
```
|
||||||
|
Rule AG-1: Agent VM/CT Stopped
|
||||||
|
Detect → PVE API shows agent CT as "stopped"
|
||||||
|
Action → `pct start <vmid>` via PVE API
|
||||||
|
Verify → Re-check after 15s
|
||||||
|
Escalate → If persistent, alert owner
|
||||||
|
Requires: PVE API
|
||||||
|
|
||||||
|
Rule AG-2: Agent Not Responding
|
||||||
|
Detect → Health endpoint non-200 or Gateway disconnected
|
||||||
|
Action → Check CT is running, then check service status
|
||||||
|
Requires: PVE API + SSH
|
||||||
|
```
|
||||||
|
|
||||||
## Access Matrix
|
## Access Matrix
|
||||||
|
|
||||||
Each agent needs explicit credentials to each environment:
|
| Resource | Abiba (CT 100) | Tanko (CT 122) | Mumuni (CT 114) |
|
||||||
|
|----------|---------------|----------------|-----------------|
|
||||||
| Environment | Abiba (pi) | Tanko (Hermes) | Mumuni (Hermes) |
|
| **PVE API** (minipve:443) | ✅ API token | ❌ | ❌ |
|
||||||
|---|---|---|---|
|
| **docker-vm** (SSH .7) | ✅ root SSH | ❌ | ❌ |
|
||||||
|
| **CT 116** (SSH) | ✅ root SSH | ❌ | ❌ |
|
||||||
| **Gitea** | ✅ API token | Need to add | Need to add |
|
| **Gitea** | ✅ API token | Need to add | Need to add |
|
||||||
| **CT 116 (LiteLLM)** | ✅ SSH key | Need to add | ❌ (has own CTs) |
|
| **Zulip API** | ✅ bot token | ✅ bot token | ✅ bot token |
|
||||||
| **Zulip API** | ✅ Bot token | ✅ Bot token | ✅ Bot token |
|
|
||||||
| **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge |
|
| **RA-H OS** | ✅ MCP bridge | ✅ MCP bridge | ✅ MCP bridge |
|
||||||
|
| **Netbird** (SSH .17) | ✅ root SSH | ❌ | ❌ |
|
||||||
| **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH |
|
| **Authentik** | ❌ No SSH | ❌ No SSH | ❌ No SSH |
|
||||||
|
|
||||||
## The Gap (What's Missing)
|
## Priority Matrix for Auto-Remediation
|
||||||
|
|
||||||
Currently, the contracts CAN:
|
| Priority | Condition | Action | SLA |
|
||||||
|
|----------|-----------|--------|-----|
|
||||||
|
| **P0** | Critical CT/VM stopped | Auto-start via PVE API | 2 min |
|
||||||
|
| **P0** | LiteLLM/router down | Auto-restart container | 3 min |
|
||||||
|
| **P1** | Firecrawl/SearXNG down | Auto-restart container | 5 min |
|
||||||
|
| **P1** | Docker disk > 85% | Auto-prune | 5 min |
|
||||||
|
| **P2** | Storage usage > 80% | Alert + log | 15 min |
|
||||||
|
| **P2** | CT resource pressure | Alert + log | 15 min |
|
||||||
|
| **P3** | Non-critical container down | Alert + log | 30 min |
|
||||||
|
| **P3** | PBS backup missed | Alert + log | 1h |
|
||||||
|
|
||||||
| Capability | Works? | How |
|
## Escalation Path
|
||||||
|---|---|---|
|
|
||||||
| Read container state | ✅ SSH to CT 116 |
|
|
||||||
| Restart containers | ✅ SSH + docker compose |
|
|
||||||
| Commit to Gitea | ✅ API token |
|
|
||||||
| Alert via Zulip | ✅ Bot API |
|
|
||||||
| Log to KG | ✅ MCP bridge |
|
|
||||||
|
|
||||||
But they CANNOT yet:
|
```
|
||||||
|
1. Auto-remediation ──► 2. Zulip DM to owner ──► 3. RA-H OS relay to agent with access
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
Kwame (Zulip DM) Mumuni (for authentik)
|
||||||
|
Tanko (for hermes agents)
|
||||||
|
```
|
||||||
|
|
||||||
| Gap | Why | Fix |
|
## Known Gaps
|
||||||
|---|---|---|
|
|
||||||
| Push code to Tanko's CT | No SSH key for CT 112 | Add Abiba's key to CT 112 |
|
|
||||||
| Restart Hermes gateway | No remote command | Add `hermes gateway restart` via SSH |
|
|
||||||
| Auto-merge PRs | Needs Gitea write token | Already have it, need to wire |
|
|
||||||
|
|
||||||
## Reference Implementation
|
| Gap | What's Needed |
|
||||||
|
|-----|--------------|
|
||||||
The `litellm-self-heal` contract already implements most of this pattern:
|
| Authentik SSH | No agent has SSH — Mumuni needs key added |
|
||||||
1. It checks conditions (container health, endpoints)
|
| Koby/Koonimo stopped | CTs 111/113 are offline — need human intervention |
|
||||||
2. It enforces fixes (docker compose up)
|
| Mumuni PVE access | No PVE token — needs `monitoring@pve!mumuni` |
|
||||||
3. It logs results (knowledge graph)
|
| docker-vm backups | VM 109 isn't running PBS agent — backup status unknown |
|
||||||
4. It alerts via Zulip DM
|
| Netbird redundancy | Single point of failure for VPN |
|
||||||
|
|
||||||
The same pattern extends to any environment where the agent has credentials.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user