fix: contract improvements from 2026-07-09 run log review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m43s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 14m43s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs - disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope, remove jitsi (decommissioned), update access matrix with KVM VM section - infrastructure-control: docker-vm container count 11→16, add Trove agents stack - infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
This commit is contained in:
@@ -3,10 +3,11 @@ kind: responsibility
|
||||
name: disk-gc-threat-response
|
||||
description: >
|
||||
Recurring disk health scan, garbage collection, and threat response
|
||||
across all 19 Proxmox CTs and Docker hosts. Triggered by incident
|
||||
across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
|
||||
2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
|
||||
Docker image bloat — 5 dangling images, 15 build cache layers.
|
||||
Recovered 35.67GB via `docker system prune -a --force`.
|
||||
Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
|
||||
bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
|
||||
id: 067NV8KJ03ZG71S44N41F31022
|
||||
version: 1.0.0
|
||||
---
|
||||
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
|
||||
|
||||
## Scope
|
||||
|
||||
All CTs in the Proxmox cluster, with special attention to Docker hosts:
|
||||
All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
|
||||
Docker hosts get special attention:
|
||||
|
||||
| Host | CT | Disk Risk | GC Strategy |
|
||||
|------|----|-----------|-------------|
|
||||
| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
|
||||
| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
|
||||
| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
|
||||
| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
|
||||
| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
|
||||
| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
|
||||
| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
|
||||
| All others | — | LOW — no Docker | apt clean, log rotate |
|
||||
| All other CTs | — | LOW — no Docker | apt clean, log rotate |
|
||||
| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
|
||||
|
||||
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
|
||||
> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
|
||||
|
||||
## Threat Levels
|
||||
|
||||
@@ -123,7 +130,7 @@ call summary-reporter
|
||||
|
||||
## GC Strategies by Host Type
|
||||
|
||||
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
|
||||
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
|
||||
|
||||
```bash
|
||||
# Phase 1: Safe prune (won't touch running containers' images)
|
||||
@@ -289,4 +296,11 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|
||||
|------|-----|-----|--------|
|
||||
| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
|
||||
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
|
||||
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
|
||||
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
|
||||
|
||||
### KVM VM (via direct SSH)
|
||||
| Host | IP | Role | Status |
|
||||
|------|-----|------|--------|
|
||||
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
|
||||
|
||||
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
|
||||
Reference in New Issue
Block a user