prose-contracts/disk-gc-threat-response.prose.md

---
kind: responsibility
name: disk-gc-threat-response
description: >
  Recurring disk health scan, garbage collection, and threat response
  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
  2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
  Docker image bloat — 5 dangling images, 15 build cache layers.
  Recovered 35.67GB via `docker system prune -a --force`.
id: 067NV8KJ03ZG71S44N41F31022
version: 1.0.0
---

# Disk GC & Threat Response

## Goal

Every container in the fleet stays below 80% disk usage through regular inspection
and automated garbage collection. Critical breaches are detected, remediated, and
logged within 5 minutes of discovery.

## Scope

All CTs in the Proxmox cluster, with special attention to Docker hosts:

| Host | CT | Disk Risk | GC Strategy |
|------|----|-----------|-------------|
| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
| All others | — | LOW — no Docker | apt clean, log rotate |

## Threat Levels

| Level | Threshold | Response | Escalation |
|-------|-----------|----------|------------|
| **GREEN** | < 75% | Log only | None |
| **AMBER** | 75-84% | Warn via Zulip DM | GC scheduled for next run |
| **RED** | 85-94% | Immediate GC attempt | Zulip DM + channel alert |
| **CRITICAL** | ≥ 95% | Aggressive GC + emergency cleanup | Zulip + relay to Kwame |
| **FULL** | 100% (df shows 100%) | Stop writes, manual intervention | Call/Telegram Kwame |

## Requires

- SSH access to all Docker hosts (matrix from infrastructure-control pattern)
- Proxmox API access from abiba (token already provisioned)
- Zulip bot credentials for alerting

## Maintains

Per-CT disk state, GC history, and threat resolution ledger. Each entry carries
the CT ID, hostname, node, disk usage snapshot, GC action taken, and reclaimed
bytes. Postcondition: no CT runs above 85% for more than one scan cycle without
documented reason.

### disk-health
Current disk state for every CT: `pct_used`, `pct_avail`, `rootfs_size`, last GC
timestamp, and active threat level.

### gc-history
Append-only log of every GC action: timestamp, CT, action taken, bytes reclaimed,
and whether threat was resolved.

### threat-log
Active and resolved threat entries with severity, timestamps, remediation applied,
and escalation trail.

## Continuity

- self-driven: full fleet scan every 6 hours
- May also be invoked manually: `prose run disk-gc-threat-response`
- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates

## Shape

- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
- `delegates`:
  - `disk-scanner`: per-CT disk check (pct exec df or SSH)
  - `gc-docker`: docker system prune execution
  - `gc-system`: apt clean, log rotate, tmp cleanup
  - `alerter`: Zulip notification dispatch
- `prohibited`: deleting user data, removing running containers, force-killing
  production services

## Runtime

- `timeout`: 300 seconds per CT (GC may take time on large docker hosts)
- `retry`: 2 attempts for SSH failures before marking a CT unreachable

## Execution

```prose
let fleet = call disk-scanner
  scope: all

let threats = []
for ct in fleet:
  if ct.usage_pct >= 95:
    push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct }
  else if ct.usage_pct >= 85:
    push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
  else if ct.usage_pct >= 75:
    push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }

-- sort by severity descending
sort threats by pct desc

for threat in threats:
  let result = call gc-executor
    ct: threat.ct
    level: threat.level
    strategy: lookup-gc-strategy(threat.ct)
  
  call alerter
    threat: threat
    result: result

call summary-reporter
  fleet: fleet
  threats: threats
```

## GC Strategies by Host Type

### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)

```bash
# Phase 1: Safe prune (won't touch running containers' images)
docker system prune -f

# Phase 2: Aggressive (if still > 85% after Phase 1)
docker system prune -a --force

# Phase 3: Emergency (if still > 95%)
docker system prune -a --force --volumes
docker builder prune --all --force

# Verification after each phase
df -h /
docker system df
```

### Non-Docker CTs

```bash
# Package cache
apt-get clean
apt-get autoremove --yes

# Log rotation
journalctl --vacuum-size=100M
find /var/log -type f -name "*.log" -mtime +30 -delete

# Temp files
find /tmp -type f -mtime +7 -delete
find /var/tmp -type f -mtime +30 -delete

# Snap (if installed)
snap list --all | awk '/disabled/ {print $1, $3}' | while read snap rev; do
  snap remove "$snap" --revision="$rev"
done
```

### Special Cases

| CT | Special GC |
|----|-----------|
| 116 (syslog-api) | Prometheus retention: check `--storage.tsdb.retention.time` |
| 100 (abiba) | Go module cache: `go clean -cache -modcache` if > 500MB |
| 106 (ra-h-os) | Check relay DB size, enforce TTL |
| 109 (docker-vm) | Check `/media/storage` and `/media/mediastore` mounts first |

## Alert Templates

### AMBER (75-84%)
```
⚠️ Disk GC — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
Next scheduled GC will attempt cleanup. No immediate action needed.
```

### RED (85-94%)
```
🚨 Disk Threat — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
GC executed: reclaimed {reclaimed}G. New usage: {new_pct}%.
Status: {resolved|still elevated — {reason}}
```

### CRITICAL (≥95%)
```
🔥 CRITICAL Disk — {hostname} (CT {id}) at {pct}% ({used}G/{total}G)
Emergency GC: reclaimed {reclaimed}G. New usage: {new_pct}%.
{service_status} — {manual_action if needed}
@Kwame — container nearly full, resolved: {yes_no}
```

## Incident Log: 2026-07-04 — kagentz docker bloat

### Discovery
Kwame noticed kagentz was full, asked Abiba to investigate.

### Diagnosis
- CT 105 (kagentz, amdpve): 49G used / 59G total (87%)
- `/var/lib` = 55G (93% of disk)
- Docker images: 7 total, 5 dangling, 47.83GB disk, 34.98GB reclaimable
- 1 stopped container, 1 unused network, 15 build cache layers

### Resolution
```
docker system prune -a --force
→ Reclaimed 35.67GB
→ Post: 16G / 59G (27%), 41G free
→ 5 dangling images removed (kagentz-bridge + old builds)
→ 1 stopped container removed
→ 1 unused network removed (kagentz-bridge_default)
→ 15 build cache layers removed
```

### Root Cause
Agent Zero's iterative development pattern (rebuild kagentz-bridge) creates
dangling images and orphaned build cache. No automated GC was in place.

### Preventive Measures
- This contract now runs disk GC fleet-wide every 6 hours
- Docker hosts get `docker system prune` on amber, `-a --force` on red
- kagentz is flagged as HIGH risk due to development activity
- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
```

## Incident Log: 2026-07-09 — amdpve docker bloat

### Discovery
Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
amdpve (.15) flagged at 78% (AMBER threshold: 75%).

### Diagnosis
- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
- Not a running service — zero containers, zero active volumes

### Resolution
```
docker system prune -a --force
→ Reclaimed 11.56GB
→ Post: 55G / 94G (62%), 35G free
→ 1 image removed (llama-strix-hip:latest, 15.5GB)
→ 8 build cache layers removed
→ amdpve now GREEN
```

### Root Cause
Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
artifacts. Docker on amdpve serves no running purpose — it's only used for
one-off GPU builds. No automated post-migration cleanup was in place.

### Preventive Measures
- amdpve added to Docker GC scan list
- Post-migration cleanup step added: after any GPU backend migration, prune
  the old backend's Docker images within 24 hours
- Contract now scans GPU bare-metal hosts alongside CTs
- Access via `pct-run` script for all CTs (no hardcoded IPs)

## Access Matrix (documented 2026-07-09)

### CT Access (via pct-run)
| CT | Name | Node | Status |
|----|------|------|--------|
| 100 | abiba | amdpve | local |
| 102 | adguard | acerpve | ✅ reachable |
| 104 | authentik | minipve | ✅ reachable |
| 105 | kagentz | amdpve | ✅ reachable |
| 106 | ra-h-os | storepve | ✅ reachable |
| 107 | pbs | storepve | ✅ reachable |
| 108 | media | storepve | ✅ reachable |
| 110 | gitea | minipve | ✅ reachable |
| 111 | tdunna | amdpve | ✅ reachable |
| 112 | tanko | amdpve | ✅ reachable |
| 113 | baggy | amdpve | ✅ reachable |
| 114 | mumuni | minipve | ✅ reachable |
| 115 | scottdenya | amdpve | ✅ reachable |
| 116 | syslog-api | minipve | ✅ reachable |
| 117 | zulip | storepve | ✅ reachable |

### GPU Bare Metal (via direct SSH)
| Host | IP | GPU | Status |
|------|-----|-----|--------|
| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |