Merge pull request 'fix(disk-gc): hard guest-level report-only gate for CT 111/.129 + correct stale fleet map' (#81) from fix/disk-gc-report-only-129 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s

This commit was merged in pull request #81.
This commit is contained in:
2026-09-12 19:13:45 +00:00
12 changed files with 578 additions and 79 deletions
+1 -1
View File
@@ -55,7 +55,7 @@ Two incidents taught us this:
### Stage 3 — AI Review ### Stage 3 — AI Review
- Diff is sent to `syslog-auto` model via LiteLLM - Diff is sent to `syslog-auto` model via LiteLLM
- Review checks against infrastructure-control ground truth: - Review checks against infrastructure-control ground truth:
- CT IDs match PVE cluster (100-117, no 122/123) - CT IDs match the PVE cluster inventory in `infrastructure-control.prose.md` Appendix B (100-120 with gaps; no 122/123)
- Grafana is direct LAN :3001, NOT behind nginx - Grafana is direct LAN :3001, NOT behind nginx
- Zulip is CT 117 on storepve (bridge IP .19) - Zulip is CT 117 on storepve (bridge IP .19)
- Strix Halo :8080 is firewalled to .116 only - Strix Halo :8080 is firewalled to .116 only
+96 -44
View File
@@ -1,11 +1,14 @@
--- ---
report_only_agents: report_only_agents:
- koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129 - koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
# ⛔ The guest/host-keyed report-only gate the GC executor MUST honour lives in the body
# "Hard gate" YAML block below — that block is authoritative and is the only copy.
kind: responsibility kind: responsibility
name: disk-gc-threat-response name: disk-gc-threat-response
description: > description: >
Recurring disk health scan, garbage collection, and threat response Recurring disk health scan, garbage collection, and threat response
across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident across 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts
(fleet verified against `pvesh get /cluster/resources` 2026-09-12). Triggered by incident
2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from
Docker image bloat — 5 dangling images, 15 build cache layers. Docker image bloat — 5 dangling images, 15 build cache layers.
Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker
@@ -25,7 +28,7 @@ logged within 5 minutes of discovery.
## Scope ## Scope
All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH. All 20 Proxmox guests (17 LXC via `pct-run` + 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts via direct SSH.
Docker hosts get special attention: Docker hosts get special attention:
| Host | CT | Disk Risk | GC Strategy | | Host | CT | Disk Risk | GC Strategy |
@@ -99,35 +102,67 @@ and escalation trail.
## Execution ## Execution
### Hard gate: report-only guests (READ THIS BEFORE RUNNING GC)
**CT 111 / hostname `tdunna` / 192.168.68.129 is DETECT-AND-REPORT-ONLY.** It belongs to Theo.
The captain ruled 2026-08-17 and re-confirmed 2026-09-10 that Theo handles CT 111 himself.
At **every** threat level — AMBER, RED, or CRITICAL — the executor must:
- push the threat row and alert the owner, and
- **never** call `gc-executor`, and **never** run any GC command against that guest: no
`apt-get clean/autoremove`, no `journalctl --vacuum-*`, no `find /var/log -delete`, no
`/tmp`/`/var/tmp` deletion, no snap removal, no `docker system prune`.
This gate is keyed on **guest id / hostname / IP**, not on an agent name. The frontmatter
`report_only_agents` marker (e.g. `koby`) names an AGENT while the scan unit is a GUEST, so an
agent-name marker can silently miss the guest it lives on — it must never be the only gate.
**The authoritative machine-readable exclusion list is the YAML block below.** The executor
reads it at run time; `scripts/disk-gc-plan.py` turns a fleet scan into the action plan using it.
Extend the list here, never by hand-maintaining a second copy. The Execution loop below MUST
call that planner and MUST NOT reimplement the gate.
```yaml
# disk-gc report-only guests — authoritative. Keyed on guest/host, not agent.
report_only_guests:
- guest: 111
hostname: tdunna
ip: 192.168.68.129
node: storepve
reason: "Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10"
```
### Loop
```prose ```prose
let fleet = call disk-scanner let fleet = call disk-scanner
scope: all scope: all
let threats = [] -- The report-only gate is IMPLEMENTED IN scripts/disk-gc-plan.py and MUST NOT be
for ct in fleet: -- reimplemented here. That planner reads the contract's `report_only_guests` YAML block
if ct.usage_pct >= 95: -- and matches on guest id OR hostname OR IP, so the tested gate is the executed gate.
push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct } let plan = call disk-gc-plan
else if ct.usage_pct >= 85: fleet: fleet
push threats { ct: ct.id, level: "RED", pct: ct.usage_pct }
else if ct.usage_pct >= 75:
push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct }
-- sort by severity descending for row in plan:
sort threats by pct desc if row.action == "report-only":
-- Excluded guest: alert only. No gc-executor call is constructed for it, at any level.
for threat in threats: call alerter
threat: row
result: { action: "report-only", reason: row.reason }
else:
let result = call gc-executor let result = call gc-executor
ct: threat.ct ct: row.target
level: threat.level level: row.level
strategy: lookup-gc-strategy(threat.ct) strategy: lookup-gc-strategy(row.target)
call alerter call alerter
threat: threat threat: row
result: result result: result
call summary-reporter call summary-reporter
fleet: fleet fleet: fleet
threats: threats plan: plan
``` ```
## GC Strategies by Host Type ## GC Strategies by Host Type
@@ -239,7 +274,9 @@ dangling images and orphaned build cache. No automated GC was in place.
## Incident Log: 2026-07-09 — amdpve docker bloat ## Incident Log: 2026-07-09 — amdpve docker bloat
### Discovery ### Discovery
Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts. Scheduled fleet disk scan across all 20 Proxmox guests (17 LXC via `pct-run`, 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts.
> **Report-only gate applies to this scan:** CT 111 (`tdunna`, 192.168.68.129) is alerted but never
> garbage-collected at any level.
amdpve (.15) flagged at 78% (AMBER threshold: 75%). amdpve (.15) flagged at 78% (AMBER threshold: 75%).
### Diagnosis ### Diagnosis
@@ -272,25 +309,35 @@ one-off GPU builds. No automated post-migration cleanup was in place.
- Contract now scans GPU bare-metal hosts alongside CTs - Contract now scans GPU bare-metal hosts alongside CTs
- Access via `pct-run` script for all CTs (no hardcoded IPs) - Access via `pct-run` script for all CTs (no hardcoded IPs)
## Access Matrix (documented 2026-07-09) ## Access Matrix (verified against `pvesh get /cluster/resources` 2026-09-12)
### CT Access (via pct-run) ### Guest Access (via `pct-run` — CT id only, node resolved by `scripts/pct-run.sh`)
| CT | Name | Node | Status | | Guest | Name | Node | Type | Status |
|----|------|------|--------| |------|------|------|------|--------|
| 100 | abiba | minipve | local | | 100 | abiba | minipve | lxc | ✅ reachable (probed via pct-run like any other guest; no local shortcut) |
| 102 | adguard | minipve | ✅ reachable | | 102 | adguard | minipve | lxc | ✅ reachable |
| 104 | authentik | minipve | ✅ reachable | | 104 | authentik | minipve | lxc | ✅ reachable |
| 105 | kagentz | minipve | ✅ reachable | | 105 | kagentz | **amdpve** | lxc | ✅ reachable (was documented as minipve — corrected) |
| 106 | ra-h-os | storepve | ✅ reachable | | 106 | ra-h-os | storepve | lxc | ✅ reachable |
| 107 | pbs | storepve | ✅ reachable | | 107 | pbs | storepve | lxc | ✅ reachable |
| 108 | media | storepve | ✅ reachable | | 108 | media | storepve | lxc | ✅ reachable |
| 110 | gitea | minipve | ✅ reachable | | 110 | gitea | minipve | lxc | ✅ reachable |
| 111 | tdunna | amdpve | ✅ reachable | | 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) |
| 112 | tanko | amdpve | ✅ reachable | | 112 | tanko | amdpve | lxc | ✅ reachable |
| 113 | baggy | amdpve | ✅ reachable | | 113 | baggy | amdpve | lxc | ✅ reachable |
| 115 | scottdenya | amdpve | ✅ reachable | | 115 | scottdenya | amdpve | lxc | ✅ reachable |
| 116 | syslog-api | minipve | ✅ reachable | | 116 | syslog-api | minipve | lxc | ✅ reachable |
| 117 | zulip | storepve | ✅ reachable | | 117 | zulip | storepve | lxc | ✅ reachable |
| 118 | jdownloader | storepve | lxc | ✅ reachable |
| 119 | infisical-vault | minipve | lxc | ✅ reachable |
| 120 | adguard2 | amdpve | lxc | ✅ reachable |
### QEMU VMs (via direct SSH)
| VM | Name | Node | IP | Status |
|----|------|------|-----|--------|
| 101 | llm-gpu (workload now bare metal .8) | acerpve | — | ✅ reachable |
| 103 | ocu-llm (workload now bare metal .110) | ocupve | — | ✅ reachable |
| 109 | docker-vm | storepve | 192.168.68.7 | ✅ reachable |
### GPU Bare Metal (via direct SSH) ### GPU Bare Metal (via direct SSH)
| Host | IP | GPU | Status | | Host | IP | GPU | Status |
@@ -299,9 +346,14 @@ one-off GPU builds. No automated post-migration cleanup was in place.
| ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable | | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable |
| amdpve | 192.168.68.15 | Strix Halo | ✅ reachable | | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable |
### KVM VM (via direct SSH) > **Fleet count:** 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts. Corrected
| Host | IP | Role | Status | > 2026-09-12: CT 105 → amdpve, CT 111 → storepve, and guests 118/119/120 were missing.
|------|-----|------|--------| >
| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable | > **CT 100 probe gap (folded in):** CT 100 previously reported "unreachable (not reported)" every
> run. Root cause is the same stale access layer: `pct-run` resolves the guest's node from its map,
> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7. > and the map/contract must reflect `pvesh /cluster/resources`. Verified working from inside CT 100:
> `scripts/pct-run.sh 100 "df -P / | tail -1"` → `23% /`. Probe CT 100 through `pct-run` like any
> other guest — never through a local-only path, since the scanner itself runs inside CT 100 and a
> container has no `pct` binary.
>
> **KVM VM:** CT 109 (docker-vm) is a QEMU VM, not LXC — access via SSH .7.
+4
View File
@@ -261,6 +261,10 @@ repaired.
not exist` on .15. `agent-health-check.py` now carries the live-verified not exist` on .15. `agent-health-check.py` now carries the live-verified
`storepve` mapping (the script is not the topology source of truth); the `storepve` mapping (the script is not the topology source of truth); the
CRITICAL contract itself needs an authorized correction. CRITICAL contract itself needs an authorized correction.
**✅ Resolved 2026-09-12:** the topology was corrected in its owner,
`infrastructure-control.prose.md` (CT 111 → storepve, CT 105 → amdpve), and
`scripts/pct-run.sh` now matches. This snapshot is left as observed; treat
those owner documents as authoritative.
2. **Strix Halo `:8080` firewall claim is stale.** `prose-ai-review.sh` 2. **Strix Halo `:8080` firewall claim is stale.** `prose-ai-review.sh`
ground-truth rule #4 and `gpu-monitor.prose.md` say `:8080` is firewalled to ground-truth rule #4 and `gpu-monitor.prose.md` say `:8080` is firewalled to
`.116` only and `.24` cannot probe it. Live on .15: `.116` only and `.24` cannot probe it. Live on .15:
+2 -1
View File
@@ -24,11 +24,12 @@ done
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------| |-------|-----|------|-----|---------------|------------|----------|
| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** | | Koby | 111 | storepve | .129 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes | | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo). > **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
> CT 111 (tdunna, 192.168.68.129, storepve) is report-only — Theo's box; alert only, never garbage-collect.
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
Keys are stored in Infisical vault (project=agents, env=production) and injected at Keys are stored in Infisical vault (project=agents, env=production) and injected at
+3 -3
View File
@@ -47,7 +47,7 @@ connectivity recovery including end-to-end DM validation.
## Requires ## Requires
- SSH access to target host (direct or via amdpve for CTs) - SSH access to target host (direct, or via the guest's Proxmox node for CTs)
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
- Python 3 with `httpx` installed on target - Python 3 with `httpx` installed on target
@@ -56,7 +56,7 @@ connectivity recovery including end-to-end DM validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User | | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------| |------|-----|---------|-------------|-------------|------|
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
| Field | Value | Trust | | Field | Value | Trust |
@@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation.
### Step 1: Resolve Target ### Step 1: Resolve Target
Map `target` to host, CT ID, hermes_home, and user from the live-state table. Map `target` to host, CT ID, hermes_home, and user from the live-state table.
For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`. For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
### Step 2: Pull Latest Plugin Source ### Step 2: Pull Latest Plugin Source
+3 -3
View File
@@ -43,7 +43,7 @@ gateway restart, and connection validation.
## Requires ## Requires
- SSH access to target host (direct or via amdpve for CTs) - SSH access to target host (direct, or via the guest's Proxmox node for CTs)
- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
- Python 3 with `httpx` installed on target - Python 3 with `httpx` installed on target
- Zulip server accessible at `https://chat.sysloggh.net` - Zulip server accessible at `https://chat.sysloggh.net`
@@ -52,7 +52,7 @@ gateway restart, and connection validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User | | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------| |------|-----|---------|-------------|-------------|------|
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
| Field | Value | Trust | | Field | Value | Trust |
@@ -67,7 +67,7 @@ gateway restart, and connection validation.
### Step 1: Locate Target ### Step 1: Locate Target
Map `target` to connectivity parameters from the live-state table above. Map `target` to connectivity parameters from the live-state table above.
For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`. For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec <id>`.
### Step 2: Deploy Zulip Adapter ### Step 2: Deploy Zulip Adapter
+17 -13
View File
@@ -16,8 +16,8 @@ description: >
**Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster **Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster
(now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve (now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve
(192.168.68.4) is a standalone PVE node + NetBird routing peer; (192.168.68.4) is a standalone PVE node + NetBird routing peer;
London relocation pending. CTs 100 (abiba) and 105 (kagentz) moved London relocation pending. CT 100 (abiba) is on minipve; CT 105
to minipve. (kagentz) is on amdpve.
--- ---
# Infrastructure Control Pattern # Infrastructure Control Pattern
@@ -105,16 +105,16 @@ description: >
| Node | IP | CPU | RAM | VMs/CTs | Role | | Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------| |------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | abiba, kagentz, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | | minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute | | amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat | | storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs | | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on > **Note:** CTs on storepve include jdownloader (CT 118) and tdunna (CT 111).
> minipve at .10, not acerpve. Abiba (CT 100) and kagentz (CT 105) are on > AdGuard (CT 102) is on minipve at .10, not acerpve. Abiba (CT 100) is on
> minipve (moved from hwepve 2026-08-15). Mumuni runs inside Abiba CT100 > minipve (moved from hwepve 2026-08-15); kagentz (CT 105) is on amdpve.
> (.24); CT 114 (mumuni) no longer exists in the cluster. > Mumuni runs inside Abiba CT100 (.24); CT 114 (mumuni) no longer exists in the cluster.
> >
> **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):** > **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):**
> Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve. > Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve.
@@ -602,13 +602,13 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 102 | adguard | **minipve** | **.10** | DNS | ❌ | | 102 | adguard | **minipve** | **.10** | DNS | ❌ |
| 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ | | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
| 104 | authentik | minipve | .11 | OIDC | ❌ | | 104 | authentik | minipve | .11 | OIDC | ❌ |
| 105 | kagentz | minipve | — | Agent Zero | ✅ | | 105 | kagentz | amdpve | — | Agent Zero | ✅ |
| 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
| 107 | pbs | storepve | — | Backups | ❌ | | 107 | pbs | storepve | — | Backups | ❌ |
| 108 | media | storepve | — | Media | ❌ | | 108 | media | storepve | — | Media | ❌ |
| 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
| 110 | gitea | minipve | **.17** | Git | ❌ | | 110 | gitea | minipve | **.17** | Git | ❌ |
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ | | 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ |
| 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ | | 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ |
| 113 | baggy | amdpve | .114 | Hermes agent | ✅ | | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ | | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
@@ -616,6 +616,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 117 | zulip | storepve | .19 | Chat | ❌ | | 117 | zulip | storepve | .19 | Chat | ❌ |
| 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ | | 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ |
| 119 | infisical-vault | minipve | — | Vault | ❌ | | 119 | infisical-vault | minipve | — | Vault | ❌ |
| 120 | adguard2 | amdpve | — | DNS (secondary AdGuard) | ❌ |
## Appendix C: Docker Compose Files Location ## Appendix C: Docker Compose Files Location
@@ -636,8 +637,8 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
| CT | Name | Node | pct-run | | CT | Name | Node | pct-run |
|-----|------|------|---------| |-----|------|------|---------|
| 100 | abiba | minipve | `pct-run 100` | | 100 | abiba | minipve | `pct-run 100` |
| 105 | kagentz | minipve | `pct-run 105` | | 105 | kagentz | amdpve | `pct-run 105` |
| 111 | tdunna | amdpve | `pct-run 111` | | 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) |
| 112 | tanko | amdpve | `pct-run 112` | | 112 | tanko | amdpve | `pct-run 112` |
| 113 | baggy | amdpve | `pct-run 113` | | 113 | baggy | amdpve | `pct-run 113` |
| 115 | scottdenya | amdpve | `pct-run 115` | | 115 | scottdenya | amdpve | `pct-run 115` |
@@ -648,6 +649,9 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
| 107 | proxmox-backup | storepve | `pct-run 107` | | 107 | proxmox-backup | storepve | `pct-run 107` |
| 108 | media | storepve | `pct-run 108` | | 108 | media | storepve | `pct-run 108` |
| 117 | zulip | storepve | `pct-run 117` | | 117 | zulip | storepve | `pct-run 117` |
| 118 | jdownloader | storepve | `pct-run 118` |
| 119 | infisical-vault | minipve | `pct-run 119` |
| 120 | adguard2 | amdpve | `pct-run 120` |
| 102 | adguard | **minipve** | `pct-run 102` | | 102 | adguard | **minipve** | `pct-run 102` |
GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly: GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
@@ -0,0 +1,44 @@
disk-gc report-only verification — CT 111 / tdunna / 192.168.68.129
date: 2026-09-12T19:11:36Z
host: abiba (this scanner runs INSIDE CT 100 / abiba)
branch head: 6e612ce37b1b9a9688b04e7f816848e7a4185fca
command: scripts/disk-gc-plan.py --scan <live fleet scan>
PURPOSE: prove that on a REAL fleet scan, CT 111 is alerted and NO gc-executor action
is emitted for it at any level. No GC command was executed against .129.
--- live fleet scan (df -P / via scripts/pct-run.sh for LXC, direct SSH for hosts) ---
tdunna 84%
acerpve 192.168.68.9 77%
amdpve 192.168.68.15 76%
ocu-llm 192.168.68.110 69%
storepve 192.168.68.6 65%
kagentz 61%
tanko 56%
minipve 192.168.68.12 49%
authentik 45%
infisical-vault 39%
adguard 38%
ocupve 192.168.68.5 38%
scottdenya 35%
syslog-api 34%
llm-gpu 192.168.68.8 25%
abiba 23%
baggy 22%
jdownloader 21%
gitea 16%
ra-h-os 13%
zulip 12%
docker-vm 192.168.68.7 11%
adguard2 10%
media 9%
proxmox-backup-server 4%
--- planner output (action plan) ---
111 AMBER 84.0% -> REPORT-ONLY (no GC) — Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10
acerpve AMBER 77.0% -> gc-executor
amdpve AMBER 76.0% -> gc-executor
--- verdict ---
CT 111 (tdunna) 84% AMBER -> report-only; no gc-executor row emitted; no GC run on .129.
Owned hosts acerpve .9 (77%) and amdpve .15 (76%) -> gc-executor (ours).
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env python3
"""disk-gc-plan — turn a fleet disk scan into the GC action plan.
This is the executable side of `disk-gc-threat-response.prose.md`. It exists so the
report-only gate is enforced by code that can be tested, rather than by prose the
executor might misread.
THE HARD GATE: guests listed in the contract's `report_only_guests` block are
DETECT-AND-REPORT-ONLY at EVERY level (AMBER, RED, CRITICAL). This tool will never
emit a `gc-executor` action for one, so no GC command can be constructed for it.
The gate is keyed on GUEST identity — guest id, hostname, or IP — never on an agent
name. An agent-name marker can silently miss the guest it lives on; a guest marker
cannot.
The authoritative exclusion list lives in the contract itself (the fenced ```yaml
block containing `report_only_guests:`). This tool reads it from there so there is
only ever one copy.
Usage:
disk-gc-plan.py --scan scan.json # [{"id":111,"usage_pct":84}, ...]
cat scan.json | disk-gc-plan.py # same, via stdin
disk-gc-plan.py --scan scan.json --json # machine-readable plan
Scan entries may carry any of: id / guest / vmid / ct / ctid, hostname / name, ip.
A threshold-crossing entry with no recognizable identity is reported, never GC'd.
Exit codes: 0 ok, 1 usage/parse error.
"""
from __future__ import annotations
import argparse
import json
import pathlib
import re
import sys
import yaml
REPO = pathlib.Path(__file__).resolve().parent.parent
DEFAULT_CONTRACT = REPO / "disk-gc-threat-response.prose.md"
AMBER, RED, CRITICAL = 75, 85, 95
IDENTITY_FIELDS = ("id", "guest", "vmid", "ct", "ctid", "hostname", "name", "ip")
IDENTITY_TYPE_PREFIX = re.compile(r"^(?:lxc|qemu)/")
UNIDENTIFIED_REASON = "unidentified target - refusing to schedule GC"
def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]:
"""Read the authoritative report_only_guests block out of the contract.
The contract carries it as a fenced ```yaml block. Parsing the declared,
machine-readable block is the intended interface — the contract owns the list.
"""
text = contract_path.read_text(encoding="utf-8")
for block in re.findall(r"```yaml\n(.*?)```", text, re.S):
if "report_only_guests:" in block:
data = yaml.safe_load(block)
guests = data.get("report_only_guests") or []
if not isinstance(guests, list):
raise SystemExit("report_only_guests must be a list")
if not guests:
raise SystemExit(
"report_only_guests is empty or missing - refusing to plan GC "
"without the report-only gate"
)
for guest in guests:
if not isinstance(guest, dict) or not _keys(guest):
raise SystemExit(
"report_only_guests entry has no recognizable identity key "
f"(expected one of: {', '.join(IDENTITY_FIELDS)}): {guest!r}"
)
return guests
raise SystemExit(
f"no authoritative report_only_guests block found in {contract_path}"
)
def _canonical_number(number: float) -> str:
if float(number).is_integer():
return str(int(number))
return str(number).strip().lower()
def _normalize_identity(value: object) -> str:
"""Canonicalise a guest identity so differently-encoded ids compare equal:
numeric and numeric-string ids collapse to an integer string, Proxmox
type prefixes and leading zeros are stripped, and hostnames/IPs are only
trimmed and lowercased."""
if isinstance(value, bool):
return str(value).strip().lower()
if isinstance(value, (int, float)):
return _canonical_number(float(value))
text = str(value).strip().lower()
text = IDENTITY_TYPE_PREFIX.sub("", text)
try:
return _canonical_number(float(text))
except ValueError:
return text
def _keys(entry: dict) -> set[str]:
"""Guest/host identity keys, shared by exclusions and scan entries so the two
sides of the gate can never key on different fields."""
out: set[str] = set()
for field in IDENTITY_FIELDS:
value = entry.get(field)
if value is None:
continue
key = _normalize_identity(value)
if key:
out.add(key)
return out
def level_for(pct: float) -> str | None:
if pct >= CRITICAL:
return "CRITICAL"
if pct >= RED:
return "RED"
if pct >= AMBER:
return "AMBER"
return None
def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]:
excluded = [(e, _keys(e)) for e in report_only]
plan: list[dict] = []
for entry in scan:
pct = entry.get("usage_pct")
if pct is None:
continue
level = level_for(float(pct))
if level is None:
continue # GREEN: log only, no action
scan_keys = _keys(entry)
target = next(
(entry.get(k) for k in IDENTITY_FIELDS if entry.get(k) not in (None, "")),
"?",
)
if not scan_keys:
plan.append({
"target": target,
"level": level,
"pct": float(pct),
"action": "report-only",
"reason": UNIDENTIFIED_REASON,
})
continue
match = next((e for e, keys in excluded if keys & scan_keys), None)
if match is not None:
plan.append({
"target": target,
"level": level,
"pct": float(pct),
"action": "report-only",
"reason": match.get("reason", "").strip(),
})
else:
plan.append({
"target": target,
"level": level,
"pct": float(pct),
"action": "gc-executor",
})
plan.sort(key=lambda row: row["pct"], reverse=True)
return plan
def main() -> int:
ap = argparse.ArgumentParser(description="Plan disk GC actions with the report-only gate.")
ap.add_argument("--scan", help="JSON file: list of {id|ct|hostname|ip, usage_pct}")
ap.add_argument("--contract", default=str(DEFAULT_CONTRACT))
ap.add_argument("--json", action="store_true", help="emit the plan as JSON")
args = ap.parse_args()
raw = pathlib.Path(args.scan).read_text() if args.scan else sys.stdin.read()
try:
scan = json.loads(raw)
except json.JSONDecodeError as exc:
print(f"invalid scan JSON: {exc}", file=sys.stderr)
return 1
if not isinstance(scan, list):
print("scan must be a JSON list", file=sys.stderr)
return 1
report_only = load_report_only_guests(pathlib.Path(args.contract))
plan = build_plan(scan, report_only)
if args.json:
print(json.dumps(plan, indent=2))
return 0
if not plan:
print("no threats (nothing at or above 75%)")
return 0
for row in plan:
if row["action"] == "report-only":
print(f" {row['target']} {row['level']} {row['pct']}% -> REPORT-ONLY (no GC) — {row['reason']}")
else:
print(f" {row['target']} {row['level']} {row['pct']}% -> gc-executor")
return 0
if __name__ == "__main__":
sys.exit(main())
+7 -6
View File
@@ -11,11 +11,13 @@ set -euo pipefail
# ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ── # ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
declare -A CT_NODES=( declare -A CT_NODES=(
# amdpve (192.168.68.15) # amdpve (192.168.68.15)
[111]=amdpve # tdunna [105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh)
[112]=amdpve # tanko [112]=amdpve # tanko
[113]=amdpve # baggy [113]=amdpve # baggy
[115]=amdpve # scottdenya [115]=amdpve # scottdenya
[120]=amdpve # adguard2 (added 2026-09-12)
# minipve (192.168.68.12) # minipve (192.168.68.12)
[100]=minipve # abiba (was hwepve)
[102]=minipve # adguard (was acerpve) [102]=minipve # adguard (was acerpve)
[104]=minipve # authentik [104]=minipve # authentik
[110]=minipve # gitea [110]=minipve # gitea
@@ -25,17 +27,16 @@ declare -A CT_NODES=(
[106]=storepve # ra-h-os [106]=storepve # ra-h-os
[107]=storepve # proxmox-backup [107]=storepve # proxmox-backup
[108]=storepve # media [108]=storepve # media
[111]=storepve # tdunna (was amdpve — corrected 2026-09-12; live per pvesh)
[117]=storepve # zulip [117]=storepve # zulip
[118]=storepve # jdownloader [118]=storepve # jdownloader
# acerpve (192.168.68.9) — no CTs (bare metal GPU .8) # acerpve (192.168.68.9) — no CTs (bare metal GPU .8)
[100]=minipve # abiba (was hwepve)
[105]=minipve # kagentz (was hwepve)
# ocupve (192.168.68.5) — no CTs (bare metal GPU .110) # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
# #
# REMOVED CTs (migrated to bare metal, decommissioned, or VMs): # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
# 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) [QEMU VM on acerpve]
# 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) [QEMU VM on ocupve]
# 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) [QEMU VM on storepve]
) )
# Each node must be root-accessible via SSH hostname # Each node must be root-accessible via SSH hostname
+7 -5
View File
@@ -47,17 +47,19 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology: The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
**Proxmox Cluster "Tabiri" (5 nodes):** **Proxmox Cluster "Tabiri" (5 nodes):**
- amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya - amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2
- minipve (192.168.68.12): abiba, kagentz, adguard, authentik, gitea, syslog-api, infisical-vault - minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip - storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna
- acerpve (192.168.68.9): llm-gpu - acerpve (192.168.68.9): llm-gpu
- ocupve (192.168.68.5): ocu-llm - ocupve (192.168.68.5): ocu-llm
**CT IDs (verified 2026-07-24 against PVE API):** **CT IDs (verified 2026-09-12 against PVE API):**
100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os 100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os
107:pbs 108:media 110:gitea 111:tdunna 112:tanko 107:pbs 108:media 110:gitea 111:tdunna 112:tanko
113:baggy 115:scottdenya 116:syslog-api 117:zulip 113:baggy 115:scottdenya 116:syslog-api 117:zulip
118:jdownloader 119:infisical-vault 118:jdownloader 119:infisical-vault 120:adguard2
**CT 111 (tdunna, 192.168.68.129) is REPORT-ONLY — Theo's box; alert only, never garbage-collect.**
**NO CT 122, CT 123, or .19 exist in the cluster.** **NO CT 122, CT 123, or .19 exist in the cluster.**
+185
View File
@@ -0,0 +1,185 @@
"""Regression tests for the disk-gc report-only gate (CT 111 / tdunna / .129).
WHY THIS FILE EXISTS: `disk-gc-threat-response.prose.md` defined AMBER as "GC scheduled
for next run" and its Execution loop called `gc-executor` for EVERY threat, with no
guest-level exclusion. CT 111 (tdunna, 192.168.68.129) belongs to Theo and is
report-only per the captain (2026-08-17, re-confirmed 2026-09-10) — so a single AMBER
reading on that guest would have scheduled GC commands (apt clean, journal vacuum,
log/tmp deletion, snap removal) against someone else's box. The only marker was
frontmatter `report_only_agents`, which names an AGENT while the scan unit is a GUEST.
These tests execute the real planner (`scripts/disk-gc-plan.py`) and assert observable
behaviour: an excluded guest never produces a `gc-executor` action at any level, while
our own guests still do.
"""
from __future__ import annotations
import json
import pathlib
import subprocess
import sys
ROOT = pathlib.Path(__file__).resolve().parent.parent
PLAN = ROOT / "scripts" / "disk-gc-plan.py"
def _plan(scan, tmp_path):
scan_file = tmp_path / "scan.json"
scan_file.write_text(json.dumps(scan))
proc = subprocess.run(
[sys.executable, str(PLAN), "--scan", str(scan_file), "--json"],
capture_output=True, text=True,
)
assert proc.returncode == 0, proc.stderr
return json.loads(proc.stdout)
def _actions_for(plan, target):
return [row for row in plan if str(row["target"]) == str(target)]
def test_excluded_guest_never_gets_gc_at_any_level(tmp_path):
"""CT 111 at AMBER, RED and CRITICAL — always report-only, never gc-executor."""
for pct, level in ((84, "AMBER"), (90, "RED"), (97, "CRITICAL")):
plan = _plan([{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129",
"usage_pct": pct}], tmp_path)
rows = _actions_for(plan, 111)
assert rows, f"CT 111 must still be reported at {level}"
assert rows[0]["level"] == level
assert rows[0]["action"] == "report-only", rows
assert not any(r["action"] == "gc-executor" for r in rows)
def test_exclusion_matches_on_any_identity_key(tmp_path):
"""The gate is keyed on guest/host, so id, ct, ctid, hostname or IP all match."""
for entry in ({"id": 111, "usage_pct": 95},
{"ct": 111, "usage_pct": 95},
{"ctid": 111, "usage_pct": 95},
{"hostname": "tdunna", "usage_pct": 95},
{"ip": "192.168.68.129", "usage_pct": 95}):
plan = _plan([entry], tmp_path)
assert all(r["action"] == "report-only" for r in plan), (entry, plan)
def test_exclusion_matches_encoded_identities(tmp_path):
"""A differently-encoded CT 111 id must not slip past the gate to gc-executor."""
encodings = ({"id": 111.0},
{"id": "111.0"},
{"id": "lxc/111"},
{"id": "qemu/111"},
{"id": "0111"},
{"id": " 111 "})
for alias in encodings:
plan = _plan([{**alias, "usage_pct": 97}], tmp_path)
assert plan, alias
assert plan[0]["action"] == "report-only", (alias, plan)
def test_our_own_guests_still_get_gc(tmp_path):
"""acerpve .9 and amdpve .15 are ours — they must still be acted on."""
plan = _plan([{"hostname": "acerpve", "ip": "192.168.68.9", "usage_pct": 77},
{"hostname": "amdpve", "ip": "192.168.68.15", "usage_pct": 76}], tmp_path)
assert len(plan) == 2
assert all(r["action"] == "gc-executor" for r in plan), plan
def test_below_threshold_emits_nothing(tmp_path):
"""GREEN guests produce no action at all."""
assert _plan([{"id": 111, "usage_pct": 40}], tmp_path) == []
def test_agent_name_alone_does_not_gate_a_guest(tmp_path):
"""An agent-name marker must not be the gate: an unrelated guest still gets GC."""
plan = _plan([{"id": 999, "hostname": "koby", "usage_pct": 95}], tmp_path)
assert plan and plan[0]["action"] == "gc-executor"
def test_unidentified_threat_fails_closed(tmp_path):
"""A threshold-crossing entry with no recognized identity must not schedule GC."""
plan = _plan([{"usage_pct": 97}], tmp_path)
assert plan, "an unidentified threat must still be reported"
assert plan[0]["action"] == "report-only", plan
assert plan[0]["reason"], plan
def test_exclusion_entry_without_identity_fails_closed(tmp_path):
"""A mis-typed exclusion entry must break the run, never silently disable the gate."""
contract = tmp_path / "broken.prose.md"
contract.write_text(
"```yaml\n"
"report_only_guests:\n"
" - node: storepve\n"
" reason: \"typo - no guest identity\"\n"
"```\n"
)
scan_file = tmp_path / "scan.json"
scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}]))
proc = subprocess.run(
[sys.executable, str(PLAN), "--scan", str(scan_file),
"--contract", str(contract), "--json"],
capture_output=True, text=True,
)
assert proc.returncode != 0, proc.stdout
assert "gc-executor" not in proc.stdout
assert "identity" in proc.stderr.lower(), proc.stderr
def test_empty_exclusion_block_fails_closed(tmp_path):
"""An emptied report_only_guests list must break the run, not disable the gate."""
contract = tmp_path / "empty.prose.md"
contract.write_text("```yaml\nreport_only_guests: []\n```\n")
scan_file = tmp_path / "scan.json"
scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}]))
proc = subprocess.run(
[sys.executable, str(PLAN), "--scan", str(scan_file),
"--contract", str(contract), "--json"],
capture_output=True, text=True,
)
assert proc.returncode != 0, proc.stdout
assert "gc-executor" not in proc.stdout
assert "report-only gate" in proc.stderr.lower(), proc.stderr
def _plan_with_contract(scan, contract_text, tmp_path, name):
contract = tmp_path / name
contract.write_text(contract_text)
scan_file = tmp_path / f"scan-{name}.json"
scan_file.write_text(json.dumps(scan))
proc = subprocess.run(
[sys.executable, str(PLAN), "--scan", str(scan_file),
"--contract", str(contract), "--json"],
capture_output=True, text=True,
)
assert proc.returncode == 0, proc.stderr
return json.loads(proc.stdout)
def test_gate_is_read_from_the_contract_block(tmp_path):
"""The gate is data-driven by the contract block: the planner excludes the guest
when the block names it and acts on it when the block does not. Executes the real
planner interface against both fixtures so the behaviour change is observable."""
scan = [{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", "usage_pct": 95}]
with_gate = (
"```yaml\n"
"report_only_guests:\n"
" - guest: 111\n"
" hostname: tdunna\n"
" ip: 192.168.68.129\n"
" reason: \"fixture reason\"\n"
"```\n"
)
without_gate = (
"```yaml\n"
"report_only_guests:\n"
" - guest: 999\n"
" reason: \"fixture excludes a different guest\"\n"
"```\n"
)
gated = _plan_with_contract(scan, with_gate, tmp_path, "gated.prose.md")
assert gated[0]["action"] == "report-only", gated
assert gated[0]["reason"] == "fixture reason", gated
ungated = _plan_with_contract(scan, without_gate, tmp_path, "ungated.prose.md")
assert ungated[0]["action"] == "gc-executor", ungated
assert ungated[0]["action"] != gated[0]["action"]