From 25c6c76fb347ca10fa5fe356134548b20098cfdb Mon Sep 17 00:00:00 2001
From: root <root@abiba.sysloggh.net>
Date: Thu, 9 Jul 2026 05:17:05 +0000
Subject: [PATCH] =?UTF-8?q?disk-gc:=202026-07-09=20run=20=E2=80=94=20amdpv?=
 =?UTF-8?q?e=20Docker=20bloat=20resolved=20(11.56GB=20reclaimed),=20access?=
 =?UTF-8?q?=20matrix=20documented,=20pct-run=20verified=20for=20all=2015?=
 =?UTF-8?q?=20CTs?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

---
 disk-gc-threat-response.prose.md | 75 ++++++++++++++++++++++++--------
 1 file changed, 56 insertions(+), 19 deletions(-)

diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md
index ad41204..ffec016 100644
--- a/disk-gc-threat-response.prose.md
+++ b/disk-gc-threat-response.prose.md
@@ -224,32 +224,69 @@ dangling images and orphaned build cache. No automated GC was in place.
 - This contract now runs disk GC fleet-wide every 6 hours
 - Docker hosts get `docker system prune` on amber, `-a --force` on red
 - kagentz is flagged as HIGH risk due to development activity
+- amdpve is flagged for Docker bloat monitoring — abandoned build images accumulate
 ```
 
-Done. Here's what we have:
+## Incident Log: 2026-07-09 — amdpve docker bloat
 
-## Storage Summary
+### Discovery
+Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts.
+amdpve (.15) flagged at 78% (AMBER threshold: 75%).
 
-| System | Used | Free | Health |
-|--------|------|------|--------|
-| **Abiba** (CT 100) | 12G / 59G (21%) | 45G | ✅ Green |
-| **kagentz** (CT 105) | 16G / 59G (27%) | 41G | ✅ Resolved (was 87%) |
-| **/var/lib/docker** (abiba) | 2.8G | — | Fine |
-| **/root/go** (abiba) | 773M | — | Fine |
+### Diagnosis
+- amdpve (.15, Strix Halo host): 70G used / 94G total (78%)
+- Docker images: 1 image, 0 containers running, 15.47GB (100% reclaimable)
+- Image: `llama-strix-hip:latest` — abandoned ROCm/HIP Docker build from 7 days ago
+- Root cause: Strix Halo migrated from Docker-based HIP path to bare-metal Vulkan
+  (`/root/llama.cpp/build-vk/`) but the old Docker image was never cleaned up
+- Not a running service — zero containers, zero active volumes
 
-## New Contract: `disk-gc-threat-response.prose.md`
+### Resolution
+```
+docker system prune -a --force
+→ Reclaimed 11.56GB
+→ Post: 55G / 94G (62%), 35G free
+→ 1 image removed (llama-strix-hip:latest, 15.5GB)
+→ 8 build cache layers removed
+→ amdpve now GREEN
+```
 
-Created at `/root/prose-contracts/disk-gc-threat-response.prose.md`. Here's what it does:
+### Root Cause
+Technology migration (Docker HIP → bare-metal Vulkan) left orphaned build
+artifacts. Docker on amdpve serves no running purpose — it's only used for
+one-off GPU builds. No automated post-migration cleanup was in place.
 
-### Three-in-One
+### Preventive Measures
+- amdpve added to Docker GC scan list
+- Post-migration cleanup step added: after any GPU backend migration, prune
+  the old backend's Docker images within 24 hours
+- Contract now scans GPU bare-metal hosts alongside CTs
+- Access via `pct-run` script for all CTs (no hardcoded IPs)
 
-1. **Infra Update** — Fleet-wide disk scan every 6 hours across all 19 CTs, with per-host GC strategy (Docker hosts vs standard LXC). Today's incident is logged as the baseline.
+## Access Matrix (documented 2026-07-09)
 
-2. **Garbage Collection** — Tiered response: Phase 1 (`docker system prune -f`) for amber, Phase 2 (`-a --force`) for red, Phase 3 (`--volumes` + `builder prune --all`) for critical. Non-Docker CTs get `apt clean`, log rotation, journalctl vacuum.
+### CT Access (via pct-run)
+| CT | Name | Node | Status |
+|----|------|------|--------|
+| 100 | abiba | amdpve | local |
+| 102 | adguard | acerpve | ✅ reachable |
+| 104 | authentik | minipve | ✅ reachable |
+| 105 | kagentz | amdpve | ✅ reachable |
+| 106 | ra-h-os | storepve | ✅ reachable |
+| 107 | pbs | storepve | ✅ reachable |
+| 108 | media | storepve | ✅ reachable |
+| 110 | gitea | minipve | ✅ reachable |
+| 111 | tdunna | amdpve | ✅ reachable |
+| 112 | tanko | amdpve | ✅ reachable |
+| 113 | baggy | amdpve | ✅ reachable |
+| 114 | mumuni | minipve | ✅ reachable |
+| 115 | scottdenya | amdpve | ✅ reachable |
+| 116 | syslog-api | minipve | ✅ reachable |
+| 117 | zulip | storepve | ✅ reachable |
 
-3. **Threat Resolution** — Five severity levels (Green → Amber → Red → Critical → Full), with escalating alerts via Zulip DM, channel, and relay to you for critical breaches. kagentz is flagged HIGH risk due to Agent Zero dev patterns.
-
-### The incident root cause
-Agent Zero was repeatedly rebuilding `kagentz-bridge`, generating 5 dangling images and 15 build cache layers. No automated GC existed. Recovery: **35.67GB freed** in one `docker system prune -a --force`.
-
-Want me to push this to the prose-contracts repo?
\ No newline at end of file
+### GPU Bare Metal (via direct SSH)
+| Host | IP | GPU | Status |
+|------|-----|-----|--------|
+| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
+| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
+| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
\ No newline at end of file