From 97977f320c7eec386b9d8f339ae3a7f09002114c Mon Sep 17 00:00:00 2001 From: root Date: Thu, 9 Jul 2026 05:32:00 +0000 Subject: [PATCH] fix: contract improvements from 2026-07-09 run log review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs - disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope, remove jitsi (decommissioned), update access matrix with KVM VM section - infrastructure-control: docker-vm container count 11→16, add Trove agents stack - infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed) --- disk-gc-threat-response.prose.md | 30 ++++++++++++++++++++++-------- infrastructure-control.prose.md | 5 ++++- infrastructure-monitoring.prose.md | 12 ++++++++---- scripts/pct-run.sh | 6 ++++++ 4 files changed, 40 insertions(+), 13 deletions(-) diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md index ffec016..51cdab4 100644 --- a/disk-gc-threat-response.prose.md +++ b/disk-gc-threat-response.prose.md @@ -3,10 +3,11 @@ kind: responsibility name: disk-gc-threat-response description: > Recurring disk health scan, garbage collection, and threat response - across all 19 Proxmox CTs and Docker hosts. Triggered by incident + across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from Docker image bloat — 5 dangling images, 15 build cache layers. - Recovered 35.67GB via `docker system prune -a --force`. + Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker + bloat (15.5GB abandoned HIP image), reclaimed 11.56GB. id: 067NV8KJ03ZG71S44N41F31022 version: 1.0.0 --- @@ -21,15 +22,21 @@ logged within 5 minutes of discovery. ## Scope -All CTs in the Proxmox cluster, with special attention to Docker hosts: +All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH. +Docker hosts get special attention: | Host | CT | Disk Risk | GC Strategy | |------|----|-----------|-------------| | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` | -| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate | -| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts | +| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate | +| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts | +| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` | | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune | -| All others | — | LOW — no Docker | apt clean, log rotate | +| All other CTs | — | LOW — no Docker | apt clean, log rotate | +| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate | + +> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned. +> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110. ## Threat Levels @@ -123,7 +130,7 @@ call summary-reporter ## GC Strategies by Host Type -### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109) +### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15) ```bash # Phase 1: Safe prune (won't touch running containers' images) @@ -289,4 +296,11 @@ one-off GPU builds. No automated post-migration cleanup was in place. |------|-----|-----|--------| | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable | | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable | -| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable | \ No newline at end of file +| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable | + +### KVM VM (via direct SSH) +| Host | IP | Role | Status | +|------|-----|------|--------| +| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable | + +> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7. \ No newline at end of file diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index df956cf..4df936e 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -144,7 +144,7 @@ description: > ### Ecosystem A: docker-vm (192.168.68.7) -11 containers across 4 compose stacks: +16 containers across 4 compose stacks + trove agents: | Stack | Path | Containers | |-------|------|-----------| @@ -152,6 +152,9 @@ description: > | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey | | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse | | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf | +| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats | + +**Last verified:** 2026-07-09 — 16/16 containers running healthy. ### Home Stack Services diff --git a/infrastructure-monitoring.prose.md b/infrastructure-monitoring.prose.md index 5db12f3..c55f98e 100644 --- a/infrastructure-monitoring.prose.md +++ b/infrastructure-monitoring.prose.md @@ -7,10 +7,14 @@ description: > from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics via existing /metrics Prometheus endpoint. - STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters) - already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15) - NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed. - This contract defines the target state; proxmox-monitor is the as-built. + DEPLOYMENT STATUS (2026-07-09): + ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters + (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active. + ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15, + NVIDIA sidecar exporters (.8/.110:9400) never installed. + Router falls back to direct GPU /health probes. + ⚠️ This contract is target-state aspirational — not as-built. + As-built GPU monitoring is via gpu-monitor contract (port 9100 poll). version: 1.0.0 --- diff --git a/scripts/pct-run.sh b/scripts/pct-run.sh index 2b2d37e..e5fbf09 100755 --- a/scripts/pct-run.sh +++ b/scripts/pct-run.sh @@ -30,6 +30,12 @@ declare -A CT_NODES=( # acerpve (192.168.68.9) [102]=acerpve # adguard # ocupve (192.168.68.5) — no CTs (bare metal GPU .110) + # + # REMOVED CTs (migrated to bare metal, decommissioned, or VMs): + # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) + # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) + # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) + # 118 jitsi → stopped, not in service ) # Each node must be root-accessible via SSH hostname -- 2.54.0