prose: review, fix, and consolidate infrastructure contracts

- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
2026-07-04 21:47:33 +00:00
parent 3077fda0b7
commit 253e19f680
14 changed files with 628 additions and 213 deletions
@@ -0,0 +1,255 @@
+---
+kind: responsibility
+name: disk-gc-threat-response
+description: >
+  Recurring disk health scan, garbage collection, and threat response
+  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
+  Docker image bloat — 5 dangling images, 15 build cache layers.
+  Recovered 35.67GB via `docker system prune -a --force`.
+id: 067NV8KJ03ZG71S44N41F31022
+version: 1.0.0
+---
+
+# Disk GC & Threat Response
+
+## Goal
+
+Every container in the fleet stays below 80% disk usage through regular inspection
+and automated garbage collection. Critical breaches are detected, remediated, and
+logged within 5 minutes of discovery.
+
+## Scope
+
+All CTs in the Proxmox cluster, with special attention to Docker hosts:
+
+| Host | CT | Disk Risk | GC Strategy |
+|------|----|-----------|-------------|
+| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
+| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
+| All others | — | LOW — no Docker | apt clean, log rotate |
+
+## Threat Levels
+
+| Level | Threshold | Response | Escalation |
+|-------|-----------|----------|------------|
+| **GREEN** | < 75% | Log only | None |
+| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
+| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
+| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
+| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |
+
+## Requires
+
+- SSH access to all Docker hosts (matrix from infrastructure-control pattern)
+- Proxmox API access from abiba (token already provisioned)
+- Zulip bot credentials for alerting
+
+## Maintains
+
+Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
+the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
+bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
+documented reason.
+
+### disk-health
+Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
+timestamp, and active threat level.
+
+### gc-history
+Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
+and whether threat was resolved.
+
+### threat-log
+Active and resolved threat entries with severity, timestamps, remediation applied,
+and escalation trail.
+
+## Continuity
+
+- self-driven: full fleet scan every 6 hours
+- May also be invoked manually: `prose run disk-gc-threat-response`
+- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
+
+## Shape
+
+- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
+- `delegates`:
+  - `disk-scanner`: per-CT disk check (pct exec df or SSH)
+  - `gc-docker`: docker system prune execution
+  - `gc-system`: apt clean, log rotate, tmp cleanup
+  - `alerter`: Zulip notification dispatch
+- `prohibited`: deleting user data, removing running containers, force-killing
+  production services
+
+## Runtime
+
+- `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
+- `retry`: 2 attempts for SSH failures before marking a CT unreachable
+
+## Execution
+
+```prose
+let fleet = call disk-scanner
+  scope: all
+
+let threats = []
+for ct in fleet:
+  if ct.usage_pct >= 95:
+    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
+  else if ct.usage_pct >= 85:
+    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
+  else if ct.usage_pct >= 75:
+    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
+
+-- sort by severity descending
+sort threats by pct desc
+
+for threat in threats:
+  let result = call gc-executor
+    ct: threat.ct
+    level: threat.level
+    strategy: lookup-gc-strategy(threat.ct)
+  
+  call alerter
+    threat: threat
+    result: result
+
+call summary-reporter
+  fleet: fleet
+  threats: threats
+```
+
+## GC Strategies by Host Type
+
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+
+```bash
+# Phase 1: Safe prune (won't touch running containers' images)
+docker system prune -f
+
+# Phase 2: Aggressive (if still > 85% after Phase 1)
+docker system prune -a --force
+
+# Phase 3: Emergency (if still > 95%)
+docker system prune -a --force --volumes
+docker builder prune --all --force
+
+# Verification after each phase
+df -h /
+docker system df
+```
+
+### Non-Docker CTs
+
+```bash
+# Package cache
+apt-get clean
+apt-get autoremove --yes
+
+# Log rotation
+journalctl --vacuum-size=100M
+find /var/log -type f -name "*.log" -mtime +30 -delete
+
+# Temp files
+find /tmp -type f -mtime +7 -delete
+find /var/tmp -type f -mtime +30 -delete
+
+# Snap (if installed)
+snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
+  snap remove "$snap" --revision="$rev"
+done
+```
+
+### Special Cases
+
+| CT | Special GC |
+|----|-----------|
+| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
+| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
+| 106 (ra-h-os) | Check relay DB size, enforce TTL |
+| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |
+
+## Alert Templates
+
+### AMBER (75-84%)
+```
+⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Next scheduled GC will attempt cleanup. No immediate action needed.
+```
+
+### RED (85-94%)
+```
+🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
+Status: {resolved|still elevated — {reason}}
+```
+
+### CRITICAL (≥95%)
+```
+🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
+Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
+{service_status} — {manual_action if needed}
+@Kwame — container nearly full, resolved: {yes_no}
+```
+
+## Incident Log: 2026-07-04 — kagentz docker bloat
+
+### Discovery
+Kwame noticed kagentz was full, asked Abiba to investigate.
+
+### Diagnosis
+- CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
+- `/var/lib` = 55G (93% of disk)
+- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
+- 1 stopped container, 1 unused network, 15 build cache layers
+
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 35.67GB
+→ Post: 16G / 59G (27%), 41G free
+→ 5 dangling images removed (kagentz-bridge + old builds)
+→ 1 stopped container removed
+→ 1 unused network removed (kagentz-bridge_default)
+→ 15 build cache layers removed
+```
+
+### Root Cause
+Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
+dangling images and orphaned build cache. No automated GC was in place.
+
+### Preventive Measures
+- This contract now runs disk GC fleet-wide every 6 hours
+- Docker hosts get `docker system prune` on amber, `-a --force` on red
+- kagentz is flagged as HIGH risk due to development activity
+```
+
+Done. Here's what we have:
+
+## Storage Summary
+
+| System | Used | Free | Health |
+|--------|------|------|--------|
+| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
+| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
+| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
+| **/root/go** (abiba) | 773M | — | Fine |
+
+## New Contract: `disk-gc-threat-response.prose.md`
+
+Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+
+### Three-in-One
+
+1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+
+2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+
+3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
+
+### The incident root cause
+Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
+
+Want me to push this to the prose-contracts repo?