fix: contract improvements from 2026-07-09 run log review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m43s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped

- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
  remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
This commit is contained in:
root
2026-07-09 05:32:00 +00:00
parent a3786ab289
commit 97977f320c
4 changed files with 40 additions and 13 deletions
+22 -8
View File
@@ -3,10 +3,11 @@ kind: responsibility
name: disk-gc-threat-response
description: >
Recurring disk health scan, garbage collection, and threat response
across all 19 Proxmox CTs and Docker hosts. Triggered by incident
across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
Docker image bloat — 5 dangling images, 15 build cache layers.
Recovered 35.67GB via `docker system prune -a --force`.
Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
id: 067NV8KJ03ZG71S44N41F31022
version: 1.0.0
---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
## Scope
All CTs in the Proxmox cluster, with special attention to Docker hosts:
All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
Docker hosts get special attention:
| Host | CT | Disk Risk | GC Strategy |
|------|----|-----------|-------------|
| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
| All others | — | LOW — no Docker | apt clean, log rotate |
| All other CTs | — | LOW — no Docker | apt clean, log rotate |
| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
## Threat Levels
@@ -123,7 +130,7 @@ call summary-reporter
## GC Strategies by Host Type
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
```bash
# Phase 1: Safe prune (won't touch running containers' images)
@@ -289,4 +296,11 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|------|-----|-----|--------|
| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
### KVM VM (via direct SSH)
| Host | IP | Role | Status |
|------|-----|------|--------|
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.