Merge pull request 'fix: contract improvements from 2026-07-09 run log review' (#7) from fix/contract-improvements-jul2026 into master

disk-gc-threat-response.prose.md

@@ -3,10 +3,11 @@ kind: responsibility
 name: disk-gc-threat-response
 description: >
   Recurring disk health scan, garbage collection, and threat response
-  across all 19 Proxmox CTs and Docker hosts. Triggered by incident
+  across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
   2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
   Docker image bloat — 5 dangling images, 15 build cache layers.
-  Recovered 35.67GB via `docker system prune -a --force`.
+  Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
+  bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
 
 ## Scope
 
-All CTs in the Proxmox cluster, with special attention to Docker hosts:
+All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
+Docker hosts get special attention:
 
 | Host | CT | Disk Risk | GC Strategy |
 |------|----|-----------|-------------|
 | kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
-| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
+| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
-| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
+| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
+| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
 | abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
-| All others | — | LOW — no Docker | apt clean, log rotate |
+| All other CTs | — | LOW — no Docker | apt clean, log rotate |
+| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
+> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
 
 ## Threat Levels
 
@@ -123,7 +130,7 @@ call summary-reporter
 
 ## GC Strategies by Host Type
 
-### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
+### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
 
 ```bash
 # Phase 1: Safe prune (won't touch running containers' images)
@@ -289,4 +296,11 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 |------|-----|-----|--------|
 | llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
 | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
-| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
+
+### KVM VM (via direct SSH)
+| Host | IP | Role | Status |
+|------|-----|------|--------|
+| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
+
+> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.

infrastructure-control.prose.md

@@ -144,7 +144,7 @@ description: >
 
 ### Ecosystem A: docker-vm (192.168.68.7)
 
-11 containers across 4 compose stacks:
+16 containers across 4 compose stacks + trove agents:
 
 | Stack | Path | Containers |
 |-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
 | **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
 | **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
 | **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
+| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
+
+**Last verified:** 2026-07-09 — 16/16 containers running healthy.
 
 ### Home Stack Services
 

infrastructure-monitoring.prose.md

@@ -7,10 +7,14 @@ description: >
   from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
   via existing /metrics Prometheus endpoint.
   
-  STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
+  DEPLOYMENT STATUS (2026-07-09):
-  already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
+  ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
-  NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
+     (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
-  This contract defines the target state; proxmox-monitor is the as-built.
+  ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
+     NVIDIA sidecar exporters (.8/.110:9400) never installed.
+     Router falls back to direct GPU /health probes.
+  ⚠️ This contract is target-state aspirational — not as-built.
+     As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
 version: 1.0.0
 ---
 

scripts/pct-run.sh

@@ -30,6 +30,12 @@ declare -A CT_NODES=(
     # acerpve (192.168.68.9)
     [102]=acerpve   # adguard
     # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
+    #
+    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
+    #   101 llm-gpu   → bare metal 192.168.68.8   (RTX 3090)
+    #   103 ocu-llm   → bare metal 192.168.68.110 (RTX 5070)
+    #   109 docker-vm → KVM VM 192.168.68.7       (use direct SSH)
+    #   118 jitsi     → stopped, not in service
 )
 
 # Each node must be root-accessible via SSH hostname