prose-contracts/disk-gc-threat-response.prose.md

kind, name, description, id, version

kind	name	description	id	version
responsibility	disk-gc-threat-response	Recurring disk health scan, garbage collection, and threat response across all 19 Proxmox CTs and Docker hosts. Triggered by incident 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from Docker image bloat — 5 dangling images, 15 build cache layers. Recovered 35.67GB via `docker system prune -a --force`.	067NV8KJ03ZG71S44N41F31022	1.0.0

Disk GC & Threat Response

Goal

Every container in the fleet stays below 80% disk usage through regular inspection and automated garbage collection. Critical breaches are detected, remediated, and logged within 5 minutes of discovery.

Scope

All CTs in the Proxmox cluster, with special attention to Docker hosts:

Host	CT	Disk Risk	GC Strategy
kagentz	105	HIGH — Agent Zero builds images	docker system prune -a
syslog-api	116	HIGH — Prometheus data, 8 containers	docker system prune, log rotate
docker-vm	109	MED — 11 containers, NFS mounts	docker system prune, check mounts
abiba	100	LOW — local docker, go cache	apt/docker/log prune
All others	—	LOW — no Docker	apt clean, log rotate

Threat Levels

Level	Threshold	Response	Escalation
GREEN	< 75%	Log only	None
AMBER	75-84%	Warn via Zulip DM	GC scheduled for next run
RED	85-94%	Immediate GC attempt	Zulip DM + channel alert
CRITICAL	≥ 95%	Aggressive GC + emergency cleanup	Zulip + relay to Kwame
FULL	100% (df shows 100%)	Stop writes, manual intervention	Call/Telegram Kwame

Requires

• SSH access to all Docker hosts (matrix from infrastructure-control pattern)
• Proxmox API access from abiba (token already provisioned)
• Zulip bot credentials for alerting

Maintains

Per-CT disk state, GC history, and threat resolution ledger. Each entry carries the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed bytes. Postcondition: no CT runs above 85% for more than one scan cycle without documented reason.

disk-health

Current disk state for every CT: pct_used, pct_avail, rootfs_size, last GC timestamp, and active threat level.

gc-history

Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed, and whether threat was resolved.

threat-log

Active and resolved threat entries with severity, timestamps, remediation applied, and escalation trail.

Continuity

• self-driven: full fleet scan every 6 hours
• May also be invoked manually: prose run disk-gc-threat-response
• Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates

Shape

• self: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
• delegates:
  • disk-scanner: per-CT disk check (pct exec df or SSH)
  • gc-docker: docker system prune execution
  • gc-system: apt clean, log rotate, tmp cleanup
  • alerter: Zulip notification dispatch
• prohibited: deleting user data, removing running containers, force-killing production services

Runtime

• timeout: 300 seconds per CT (GC may take time on large docker hosts)
• retry: 2 attempts for SSH failures before marking a CT unreachable

Execution

let fleet = call disk-scanner
  scope: all

let threats = []
for ct in fleet:
  if ct.usage_pct >= 95:
    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
  else if ct.usage_pct >= 85:
    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
  else if ct.usage_pct >= 75:
    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }

-- sort by severity descending
sort threats by pct desc

for threat in threats:
  let result = call gc-executor
    ct: threat.ct
    level: threat.level
    strategy: lookup-gc-strategy(threat.ct)
  
  call alerter
    threat: threat
    result: result

call summary-reporter
  fleet: fleet
  threats: threats

GC Strategies by Host Type

Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)

# Phase 1: Safe prune (won't touch running containers' images)
docker system prune -f

# Phase 2: Aggressive (if still > 85% after Phase 1)
docker system prune -a --force

# Phase 3: Emergency (if still > 95%)
docker system prune -a --force --volumes
docker builder prune --all --force

# Verification after each phase
df -h /
docker system df

Non-Docker CTs

# Package cache
apt-get clean
apt-get autoremove --yes

# Log rotation
journalctl --vacuum-size=100M
find /var/log -type f -name "*.log" -mtime +30 -delete

# Temp files
find /tmp -type f -mtime +7 -delete
find /var/tmp -type f -mtime +30 -delete

# Snap (if installed)
snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
  snap remove "$snap" --revision="$rev"
done

Special Cases

CT	Special GC
116 (syslog-api)	Prometheus retention: check --storage.tsdb.retention.time
100 (abiba)	Go module cache: go clean -cache -modcache if > 500MB
106 (ra-h-os)	Check relay DB size, enforce TTL
109 (docker-vm)	Check /media/storage and /media/mediastore mounts first

Alert Templates

AMBER (75-84%)

⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
Next scheduled GC will attempt cleanup. No immediate action needed.

RED (85-94%)

🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
Status: {resolved|still elevated — {reason}}

CRITICAL (≥95%)

🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
{service_status} — {manual_action if needed}
@Kwame — container nearly full, resolved: {yes_no}

Incident Log: 2026-07-04 — kagentz docker bloat

Discovery

Kwame noticed kagentz was full, asked Abiba to investigate.

Diagnosis

• CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
• /var/lib = 55G (93% of disk)
• Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
• 1 stopped container, 1 unused network, 15 build cache layers

Resolution

docker system prune -a --force
→ Reclaimed 35.67GB
→ Post: 16G / 59G (27%), 41G free
→ 5 dangling images removed (kagentz-bridge + old builds)
→ 1 stopped container removed
→ 1 unused network removed (kagentz-bridge_default)
→ 15 build cache layers removed

Root Cause

Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates dangling images and orphaned build cache. No automated GC was in place.

Preventive Measures

• This contract now runs disk GC fleet-wide every 6 hours
• Docker hosts get docker system prune on amber, -a --force on red
• kagentz is flagged as HIGH risk due to development activity

Done. Here's what we have:

## Storage Summary

| System | Used | Free | Health |
|--------|------|------|--------|
| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
| **/root/go** (abiba) | 773M | — | Fine |

## New Contract: `disk-gc-threat-response.prose.md`

Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:

### Three-in-One

1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.

2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.

3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.

### The incident root cause
Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.

Want me to push this to the prose-contracts repo?