abiba-bot/prose-contracts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: contract improvements from 2026-07-09 run log review

Browse Source 
- pct-run.sh: add CT 109 note (KVM VM), document decommissioned/migrated CTs
- disk-gc-threat-response: add docker-vm (SSH .7), add amdpve Docker scope,
  remove jitsi (decommissioned), update access matrix with KVM VM section
- infrastructure-control: docker-vm container count 11→16, add Trove agents stack
- infrastructure-monitoring: deployment status updated (core live, GPU exporters not deployed)
This commit is contained in:
root
2026-07-09 05:32:00 +00:00
parent a3786ab289
commit 97977f320c
4 changed files with 40 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
disk-gc-threat-response.prose.md
+22 -8
View File
@@ -3,10 +3,11 @@ kind: responsibility
name: disk-gc-threat-response
description: >
Recurring disk health scan, garbage collection, and threat response
across all 19 Proxmox CTs and Docker hosts. Triggered by incident
across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident
2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
Docker image bloat — 5 dangling images, 15 build cache layers.
Recovered 35.67GB via `docker system prune -a --force`.
Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
bloat (15.5GB abandoned HIP image), reclaimed 11.56GB.
id: 067NV8KJ03ZG71S44N41F31022
version: 1.0.0
---
@@ -21,15 +22,21 @@ logged within 5 minutes of discovery.
## Scope
All CTs in the Proxmox cluster, with special attention to Docker hosts:
All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH.
Docker hosts get special attention:
| Host | CT | Disk Risk | GC Strategy |
|------|----|-----------|-------------|
| kagentz | 105 | HIGH — Agent Zero builds images | `docker system prune -a` |
| syslog-api | 116 | HIGH — Prometheus data, 8 containers | `docker system prune`, log rotate |
| docker-vm | 109 | MED — 11 containers, NFS mounts | `docker system prune`, check mounts |
| syslog-api | 116 | HIGH — Prometheus data, 10 containers | `docker system prune`, log rotate |
| docker-vm | 109 | HIGH — 16 containers across 4 stacks, NFS mounts | `docker system prune`, check mounts |
| amdpve | — | MED — GPU bare metal, Docker for one-off builds | `docker system prune -a` |
| abiba | 100 | LOW — local docker, go cache | apt/docker/log prune |
| All others | — | LOW — no Docker | apt clean, log rotate |
| All other CTs | — | LOW — no Docker | apt clean, log rotate |
| GPU bare metal (.8, .110) | — | LOW — no Docker on GPU hosts | log rotate |
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped, not scanned.
> **Migrated:** CT 101 (llm-gpu) → bare metal .8, CT 103 (ocu-llm) → bare metal .110.
## Threat Levels
@@ -123,7 +130,7 @@ call summary-reporter
## GC Strategies by Host Type
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109)
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
```bash
# Phase 1: Safe prune (won't touch running containers' images)
@@ -289,4 +296,11 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|------|-----|-----|--------|
| llm-gpu | 192.168.68.8 | RTX 3090 | ✅ reachable |
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
### KVM VM (via direct SSH)
| Host | IP | Role | Status |
|------|-----|------|--------|
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |
> **Decommissioned:** CT 118 (jitsi) — intentionally stopped.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
infrastructure-control.prose.md
+4 -1
View File
@@ -144,7 +144,7 @@ description: >
### Ecosystem A: docker-vm (192.168.68.7)
11 containers across 4 compose stacks:
16 containers across 4 compose stacks + trove agents:
| Stack | Path | Containers |
|-------|------|-----------|
@@ -152,6 +152,9 @@ description: >
| **SearXNG** | `/opt/search-stack/searxng/` | searxng, valkey |
| **Home stack** | `/opt/home_stack/` | jdownloader, stirling-pdf, pulse |
| **Audiobookshelf** | `/opt/audiobookshelf/` | audiobookshelf |
| **Trove agents** | docker run (standalone) | trove-agent-proxmox, trove-test-agent-1, trove-test-server-1, docker-stats |
**Last verified:** 2026-07-09 — 16/16 containers running healthy.
### Home Stack Services
infrastructure-monitoring.prose.md
+8 -4
View File
@@ -7,10 +7,14 @@ description: >
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
via existing /metrics Prometheus endpoint.
STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
This contract defines the target state; proxmox-monitor is the as-built.
DEPLOYMENT STATUS (2026-07-09):
✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
(via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active.
❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15,
NVIDIA sidecar exporters (.8/.110:9400) never installed.
Router falls back to direct GPU /health probes.
⚠️ This contract is target-state aspirational — not as-built.
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
version: 1.0.0
---
scripts/pct-run.sh
+6
View File
@@ -30,6 +30,12 @@ declare -A CT_NODES=(
# acerpve (192.168.68.9)
[102]=acerpve # adguard
# ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
#
# REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
# 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090)
# 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070)
# 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH)
# 118 jitsi → stopped, not in service
)
# Each node must be root-accessible via SSH hostname