diff --git a/AGENTS.md b/AGENTS.md index b9a840c..95fb2a4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,7 @@ Two incidents taught us this: ### Stage 3 — AI Review - Diff is sent to `syslog-auto` model via LiteLLM - Review checks against infrastructure-control ground truth: - - CT IDs match PVE cluster (100-117, no 122/123) + - CT IDs match the PVE cluster inventory in `infrastructure-control.prose.md` Appendix B (100-120 with gaps; no 122/123) - Grafana is direct LAN :3001, NOT behind nginx - Zulip is CT 117 on storepve (bridge IP .19) - Strix Halo :8080 is firewalled to .116 only diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md index 34243d6..4824a2f 100644 --- a/disk-gc-threat-response.prose.md +++ b/disk-gc-threat-response.prose.md @@ -1,11 +1,14 @@ --- report_only_agents: - koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129 +# ⛔ The guest/host-keyed report-only gate the GC executor MUST honour lives in the body +# "Hard gate" YAML block below — that block is authoritative and is the only copy. kind: responsibility name: disk-gc-threat-response description: > Recurring disk health scan, garbage collection, and threat response - across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident + across 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts + (fleet verified against `pvesh get /cluster/resources` 2026-09-12). Triggered by incident 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from Docker image bloat — 5 dangling images, 15 build cache layers. Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker @@ -25,7 +28,7 @@ logged within 5 minutes of discovery. ## Scope -All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH. +All 20 Proxmox guests (17 LXC via `pct-run` + 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts via direct SSH. Docker hosts get special attention: | Host | CT | Disk Risk | GC Strategy | @@ -99,35 +102,67 @@ and escalation trail. ## Execution +### Hard gate: report-only guests (READ THIS BEFORE RUNNING GC) + +**CT 111 / hostname `tdunna` / 192.168.68.129 is DETECT-AND-REPORT-ONLY.** It belongs to Theo. +The captain ruled 2026-08-17 and re-confirmed 2026-09-10 that Theo handles CT 111 himself. +At **every** threat level — AMBER, RED, or CRITICAL — the executor must: + +- push the threat row and alert the owner, and +- **never** call `gc-executor`, and **never** run any GC command against that guest: no + `apt-get clean/autoremove`, no `journalctl --vacuum-*`, no `find /var/log -delete`, no + `/tmp`/`/var/tmp` deletion, no snap removal, no `docker system prune`. + +This gate is keyed on **guest id / hostname / IP**, not on an agent name. The frontmatter +`report_only_agents` marker (e.g. `koby`) names an AGENT while the scan unit is a GUEST, so an +agent-name marker can silently miss the guest it lives on — it must never be the only gate. + +**The authoritative machine-readable exclusion list is the YAML block below.** The executor +reads it at run time; `scripts/disk-gc-plan.py` turns a fleet scan into the action plan using it. +Extend the list here, never by hand-maintaining a second copy. The Execution loop below MUST +call that planner and MUST NOT reimplement the gate. + +```yaml +# disk-gc report-only guests — authoritative. Keyed on guest/host, not agent. +report_only_guests: + - guest: 111 + hostname: tdunna + ip: 192.168.68.129 + node: storepve + reason: "Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10" +``` + +### Loop + ```prose let fleet = call disk-scanner scope: all -let threats = [] -for ct in fleet: - if ct.usage_pct >= 95: - push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct } - else if ct.usage_pct >= 85: - push threats { ct: ct.id, level: "RED", pct: ct.usage_pct } - else if ct.usage_pct >= 75: - push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct } +-- The report-only gate is IMPLEMENTED IN scripts/disk-gc-plan.py and MUST NOT be +-- reimplemented here. That planner reads the contract's `report_only_guests` YAML block +-- and matches on guest id OR hostname OR IP, so the tested gate is the executed gate. +let plan = call disk-gc-plan + fleet: fleet --- sort by severity descending -sort threats by pct desc +for row in plan: + if row.action == "report-only": + -- Excluded guest: alert only. No gc-executor call is constructed for it, at any level. + call alerter + threat: row + result: { action: "report-only", reason: row.reason } + else: + let result = call gc-executor + ct: row.target + level: row.level + strategy: lookup-gc-strategy(row.target) -for threat in threats: - let result = call gc-executor - ct: threat.ct - level: threat.level - strategy: lookup-gc-strategy(threat.ct) - - call alerter - threat: threat - result: result + call alerter + threat: row + result: result call summary-reporter fleet: fleet - threats: threats + plan: plan ``` ## GC Strategies by Host Type @@ -239,7 +274,9 @@ dangling images and orphaned build cache. No automated GC was in place. ## Incident Log: 2026-07-09 — amdpve docker bloat ### Discovery -Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts. +Scheduled fleet disk scan across all 20 Proxmox guests (17 LXC via `pct-run`, 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts. +> **Report-only gate applies to this scan:** CT 111 (`tdunna`, 192.168.68.129) is alerted but never +> garbage-collected at any level. amdpve (.15) flagged at 78% (AMBER threshold: 75%). ### Diagnosis @@ -272,25 +309,35 @@ one-off GPU builds. No automated post-migration cleanup was in place. - Contract now scans GPU bare-metal hosts alongside CTs - Access via `pct-run` script for all CTs (no hardcoded IPs) -## Access Matrix (documented 2026-07-09) +## Access Matrix (verified against `pvesh get /cluster/resources` 2026-09-12) -### CT Access (via pct-run) -| CT | Name | Node | Status | -|----|------|------|--------| -| 100 | abiba | minipve | local | -| 102 | adguard | minipve | ✅ reachable | -| 104 | authentik | minipve | ✅ reachable | -| 105 | kagentz | minipve | ✅ reachable | -| 106 | ra-h-os | storepve | ✅ reachable | -| 107 | pbs | storepve | ✅ reachable | -| 108 | media | storepve | ✅ reachable | -| 110 | gitea | minipve | ✅ reachable | -| 111 | tdunna | amdpve | ✅ reachable | -| 112 | tanko | amdpve | ✅ reachable | -| 113 | baggy | amdpve | ✅ reachable | -| 115 | scottdenya | amdpve | ✅ reachable | -| 116 | syslog-api | minipve | ✅ reachable | -| 117 | zulip | storepve | ✅ reachable | +### Guest Access (via `pct-run` — CT id only, node resolved by `scripts/pct-run.sh`) +| Guest | Name | Node | Type | Status | +|------|------|------|------|--------| +| 100 | abiba | minipve | lxc | ✅ reachable (probed via pct-run like any other guest; no local shortcut) | +| 102 | adguard | minipve | lxc | ✅ reachable | +| 104 | authentik | minipve | lxc | ✅ reachable | +| 105 | kagentz | **amdpve** | lxc | ✅ reachable (was documented as minipve — corrected) | +| 106 | ra-h-os | storepve | lxc | ✅ reachable | +| 107 | pbs | storepve | lxc | ✅ reachable | +| 108 | media | storepve | lxc | ✅ reachable | +| 110 | gitea | minipve | lxc | ✅ reachable | +| 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) | +| 112 | tanko | amdpve | lxc | ✅ reachable | +| 113 | baggy | amdpve | lxc | ✅ reachable | +| 115 | scottdenya | amdpve | lxc | ✅ reachable | +| 116 | syslog-api | minipve | lxc | ✅ reachable | +| 117 | zulip | storepve | lxc | ✅ reachable | +| 118 | jdownloader | storepve | lxc | ✅ reachable | +| 119 | infisical-vault | minipve | lxc | ✅ reachable | +| 120 | adguard2 | amdpve | lxc | ✅ reachable | + +### QEMU VMs (via direct SSH) +| VM | Name | Node | IP | Status | +|----|------|------|-----|--------| +| 101 | llm-gpu (workload now bare metal .8) | acerpve | — | ✅ reachable | +| 103 | ocu-llm (workload now bare metal .110) | ocupve | — | ✅ reachable | +| 109 | docker-vm | storepve | 192.168.68.7 | ✅ reachable | ### GPU Bare Metal (via direct SSH) | Host | IP | GPU | Status | @@ -299,9 +346,14 @@ one-off GPU builds. No automated post-migration cleanup was in place. | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable | | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable | -### KVM VM (via direct SSH) -| Host | IP | Role | Status | -|------|-----|------|--------| -| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable | - -> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7. +> **Fleet count:** 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts. Corrected +> 2026-09-12: CT 105 → amdpve, CT 111 → storepve, and guests 118/119/120 were missing. +> +> **CT 100 probe gap (folded in):** CT 100 previously reported "unreachable (not reported)" every +> run. Root cause is the same stale access layer: `pct-run` resolves the guest's node from its map, +> and the map/contract must reflect `pvesh /cluster/resources`. Verified working from inside CT 100: +> `scripts/pct-run.sh 100 "df -P / | tail -1"` → `23% /`. Probe CT 100 through `pct-run` like any +> other guest — never through a local-only path, since the scanner itself runs inside CT 100 and a +> container has no `pct` binary. +> +> **KVM VM:** CT 109 (docker-vm) is a QEMU VM, not LXC — access via SSH .7. diff --git a/docs/probe-drift-round2-evidence.md b/docs/probe-drift-round2-evidence.md index c6a5b2d..827abbb 100644 --- a/docs/probe-drift-round2-evidence.md +++ b/docs/probe-drift-round2-evidence.md @@ -261,6 +261,10 @@ repaired. not exist` on .15. `agent-health-check.py` now carries the live-verified `storepve` mapping (the script is not the topology source of truth); the CRITICAL contract itself needs an authorized correction. + **✅ Resolved 2026-09-12:** the topology was corrected in its owner, + `infrastructure-control.prose.md` (CT 111 → storepve, CT 105 → amdpve), and + `scripts/pct-run.sh` now matches. This snapshot is left as observed; treat + those owner documents as authoritative. 2. **Strix Halo `:8080` firewall claim is stale.** `prose-ai-review.sh` ground-truth rule #4 and `gpu-monitor.prose.md` say `:8080` is firewalled to `.116` only and `.24` cannot probe it. Live on .15: diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 71ab79f..9158303 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -24,11 +24,12 @@ done | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | |-------|-----|------|-----|---------------|------------|----------| -| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** | +| Koby | 111 | storepve | .129 | `koby` | Infisical vault | **Hermes** | | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | > **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo). +> CT 111 (tdunna, 192.168.68.129, storepve) is report-only — Theo's box; alert only, never garbage-collect. Access: `pct-run ` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. Keys are stored in Infisical vault (project=agents, env=production) and injected at diff --git a/hermes-zulip-plugin.prose.md b/hermes-zulip-plugin.prose.md index 8dae1cf..26c8458 100644 --- a/hermes-zulip-plugin.prose.md +++ b/hermes-zulip-plugin.prose.md @@ -47,7 +47,7 @@ connectivity recovery including end-to-end DM validation. ## Requires -- SSH access to target host (direct or via amdpve for CTs) +- SSH access to target host (direct, or via the guest's Proxmox node for CTs) - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Python 3 with `httpx` installed on target @@ -56,7 +56,7 @@ connectivity recovery including end-to-end DM validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* | -| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | +| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Field | Value | Trust | @@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation. ### Step 1: Resolve Target Map `target` to host, CT ID, hermes_home, and user from the live-state table. -For CT112 and CT111, route through `ssh root@amdpve` then `pct exec `. +For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec `. ### Step 2: Pull Latest Plugin Source diff --git a/hermes-zulip-restore.prose.md b/hermes-zulip-restore.prose.md index bafb5a3..fb6169d 100644 --- a/hermes-zulip-restore.prose.md +++ b/hermes-zulip-restore.prose.md @@ -43,7 +43,7 @@ gateway restart, and connection validation. ## Requires -- SSH access to target host (direct or via amdpve for CTs) +- SSH access to target host (direct, or via the guest's Proxmox node for CTs) - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Python 3 with `httpx` installed on target - Zulip server accessible at `https://chat.sysloggh.net` @@ -52,7 +52,7 @@ gateway restart, and connection validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| -| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | +| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Field | Value | Trust | @@ -67,7 +67,7 @@ gateway restart, and connection validation. ### Step 1: Locate Target Map `target` to connectivity parameters from the live-state table above. -For CT112 and CT111, route through `ssh root@amdpve` then `pct exec `. +For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec `. ### Step 2: Deploy Zulip Adapter diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index d7a4a36..5c40667 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -16,8 +16,8 @@ description: > **Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster (now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve (192.168.68.4) is a standalone PVE node + NetBird routing peer; - London relocation pending. CTs 100 (abiba) and 105 (kagentz) moved - to minipve. + London relocation pending. CT 100 (abiba) is on minipve; CT 105 + (kagentz) is on amdpve. --- # Infrastructure Control Pattern @@ -105,16 +105,16 @@ description: > | Node | IP | CPU | RAM | VMs/CTs | Role | |------|----|-----|-----|---------|------| -| minipve | .12 | 16C | 30GB | abiba, kagentz, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | -| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute | -| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat | +| minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | +| amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute | +| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat | | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | -> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on -> minipve at .10, not acerpve. Abiba (CT 100) and kagentz (CT 105) are on -> minipve (moved from hwepve 2026-08-15). Mumuni runs inside Abiba CT100 -> (.24); CT 114 (mumuni) no longer exists in the cluster. +> **Note:** CTs on storepve include jdownloader (CT 118) and tdunna (CT 111). +> AdGuard (CT 102) is on minipve at .10, not acerpve. Abiba (CT 100) is on +> minipve (moved from hwepve 2026-08-15); kagentz (CT 105) is on amdpve. +> Mumuni runs inside Abiba CT100 (.24); CT 114 (mumuni) no longer exists in the cluster. > > **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):** > Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve. @@ -602,13 +602,13 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 102 | adguard | **minipve** | **.10** | DNS | ❌ | | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ | | 104 | authentik | minipve | .11 | OIDC | ❌ | -| 105 | kagentz | minipve | — | Agent Zero | ✅ | +| 105 | kagentz | amdpve | — | Agent Zero | ✅ | | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | | 107 | pbs | storepve | — | Backups | ❌ | | 108 | media | storepve | — | Media | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 110 | gitea | minipve | **.17** | Git | ❌ | -| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ | +| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ | | 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ | | 113 | baggy | amdpve | .114 | Hermes agent | ✅ | | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ | @@ -616,6 +616,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 117 | zulip | storepve | .19 | Chat | ❌ | | 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ | | 119 | infisical-vault | minipve | — | Vault | ❌ | +| 120 | adguard2 | amdpve | — | DNS (secondary AdGuard) | ❌ | ## Appendix C: Docker Compose Files Location @@ -636,8 +637,8 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run. | CT | Name | Node | pct-run | |-----|------|------|---------| | 100 | abiba | minipve | `pct-run 100` | -| 105 | kagentz | minipve | `pct-run 105` | -| 111 | tdunna | amdpve | `pct-run 111` | +| 105 | kagentz | amdpve | `pct-run 105` | +| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) | | 112 | tanko | amdpve | `pct-run 112` | | 113 | baggy | amdpve | `pct-run 113` | | 115 | scottdenya | amdpve | `pct-run 115` | @@ -648,6 +649,9 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run. | 107 | proxmox-backup | storepve | `pct-run 107` | | 108 | media | storepve | `pct-run 108` | | 117 | zulip | storepve | `pct-run 117` | +| 118 | jdownloader | storepve | `pct-run 118` | +| 119 | infisical-vault | minipve | `pct-run 119` | +| 120 | adguard2 | amdpve | `pct-run 120` | | 102 | adguard | **minipve** | `pct-run 102` | GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly: diff --git a/runs/disk-gc-ct111-report-only-verification-20260912.log b/runs/disk-gc-ct111-report-only-verification-20260912.log new file mode 100644 index 0000000..641d21d --- /dev/null +++ b/runs/disk-gc-ct111-report-only-verification-20260912.log @@ -0,0 +1,44 @@ +disk-gc report-only verification — CT 111 / tdunna / 192.168.68.129 +date: 2026-09-12T19:11:36Z +host: abiba (this scanner runs INSIDE CT 100 / abiba) +branch head: 6e612ce37b1b9a9688b04e7f816848e7a4185fca +command: scripts/disk-gc-plan.py --scan + +PURPOSE: prove that on a REAL fleet scan, CT 111 is alerted and NO gc-executor action +is emitted for it at any level. No GC command was executed against .129. + +--- live fleet scan (df -P / via scripts/pct-run.sh for LXC, direct SSH for hosts) --- + tdunna 84% + acerpve 192.168.68.9 77% + amdpve 192.168.68.15 76% + ocu-llm 192.168.68.110 69% + storepve 192.168.68.6 65% + kagentz 61% + tanko 56% + minipve 192.168.68.12 49% + authentik 45% + infisical-vault 39% + adguard 38% + ocupve 192.168.68.5 38% + scottdenya 35% + syslog-api 34% + llm-gpu 192.168.68.8 25% + abiba 23% + baggy 22% + jdownloader 21% + gitea 16% + ra-h-os 13% + zulip 12% + docker-vm 192.168.68.7 11% + adguard2 10% + media 9% + proxmox-backup-server 4% + +--- planner output (action plan) --- + 111 AMBER 84.0% -> REPORT-ONLY (no GC) — Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10 + acerpve AMBER 77.0% -> gc-executor + amdpve AMBER 76.0% -> gc-executor + +--- verdict --- + CT 111 (tdunna) 84% AMBER -> report-only; no gc-executor row emitted; no GC run on .129. + Owned hosts acerpve .9 (77%) and amdpve .15 (76%) -> gc-executor (ours). diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py new file mode 100755 index 0000000..ca1e867 --- /dev/null +++ b/scripts/disk-gc-plan.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""disk-gc-plan — turn a fleet disk scan into the GC action plan. + +This is the executable side of `disk-gc-threat-response.prose.md`. It exists so the +report-only gate is enforced by code that can be tested, rather than by prose the +executor might misread. + +THE HARD GATE: guests listed in the contract's `report_only_guests` block are +DETECT-AND-REPORT-ONLY at EVERY level (AMBER, RED, CRITICAL). This tool will never +emit a `gc-executor` action for one, so no GC command can be constructed for it. + +The gate is keyed on GUEST identity — guest id, hostname, or IP — never on an agent +name. An agent-name marker can silently miss the guest it lives on; a guest marker +cannot. + +The authoritative exclusion list lives in the contract itself (the fenced ```yaml +block containing `report_only_guests:`). This tool reads it from there so there is +only ever one copy. + +Usage: + disk-gc-plan.py --scan scan.json # [{"id":111,"usage_pct":84}, ...] + cat scan.json | disk-gc-plan.py # same, via stdin + disk-gc-plan.py --scan scan.json --json # machine-readable plan + +Scan entries may carry any of: id / guest / vmid / ct / ctid, hostname / name, ip. +A threshold-crossing entry with no recognizable identity is reported, never GC'd. +Exit codes: 0 ok, 1 usage/parse error. +""" +from __future__ import annotations + +import argparse +import json +import pathlib +import re +import sys + +import yaml + +REPO = pathlib.Path(__file__).resolve().parent.parent +DEFAULT_CONTRACT = REPO / "disk-gc-threat-response.prose.md" + +AMBER, RED, CRITICAL = 75, 85, 95 + +IDENTITY_FIELDS = ("id", "guest", "vmid", "ct", "ctid", "hostname", "name", "ip") +IDENTITY_TYPE_PREFIX = re.compile(r"^(?:lxc|qemu)/") +UNIDENTIFIED_REASON = "unidentified target - refusing to schedule GC" + + +def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: + """Read the authoritative report_only_guests block out of the contract. + + The contract carries it as a fenced ```yaml block. Parsing the declared, + machine-readable block is the intended interface — the contract owns the list. + """ + text = contract_path.read_text(encoding="utf-8") + for block in re.findall(r"```yaml\n(.*?)```", text, re.S): + if "report_only_guests:" in block: + data = yaml.safe_load(block) + guests = data.get("report_only_guests") or [] + if not isinstance(guests, list): + raise SystemExit("report_only_guests must be a list") + if not guests: + raise SystemExit( + "report_only_guests is empty or missing - refusing to plan GC " + "without the report-only gate" + ) + for guest in guests: + if not isinstance(guest, dict) or not _keys(guest): + raise SystemExit( + "report_only_guests entry has no recognizable identity key " + f"(expected one of: {', '.join(IDENTITY_FIELDS)}): {guest!r}" + ) + return guests + raise SystemExit( + f"no authoritative report_only_guests block found in {contract_path}" + ) + + +def _canonical_number(number: float) -> str: + if float(number).is_integer(): + return str(int(number)) + return str(number).strip().lower() + + +def _normalize_identity(value: object) -> str: + """Canonicalise a guest identity so differently-encoded ids compare equal: + numeric and numeric-string ids collapse to an integer string, Proxmox + type prefixes and leading zeros are stripped, and hostnames/IPs are only + trimmed and lowercased.""" + if isinstance(value, bool): + return str(value).strip().lower() + if isinstance(value, (int, float)): + return _canonical_number(float(value)) + text = str(value).strip().lower() + text = IDENTITY_TYPE_PREFIX.sub("", text) + try: + return _canonical_number(float(text)) + except ValueError: + return text + + +def _keys(entry: dict) -> set[str]: + """Guest/host identity keys, shared by exclusions and scan entries so the two + sides of the gate can never key on different fields.""" + out: set[str] = set() + for field in IDENTITY_FIELDS: + value = entry.get(field) + if value is None: + continue + key = _normalize_identity(value) + if key: + out.add(key) + return out + + +def level_for(pct: float) -> str | None: + if pct >= CRITICAL: + return "CRITICAL" + if pct >= RED: + return "RED" + if pct >= AMBER: + return "AMBER" + return None + + +def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]: + excluded = [(e, _keys(e)) for e in report_only] + plan: list[dict] = [] + for entry in scan: + pct = entry.get("usage_pct") + if pct is None: + continue + level = level_for(float(pct)) + if level is None: + continue # GREEN: log only, no action + scan_keys = _keys(entry) + target = next( + (entry.get(k) for k in IDENTITY_FIELDS if entry.get(k) not in (None, "")), + "?", + ) + if not scan_keys: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "report-only", + "reason": UNIDENTIFIED_REASON, + }) + continue + match = next((e for e, keys in excluded if keys & scan_keys), None) + if match is not None: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "report-only", + "reason": match.get("reason", "").strip(), + }) + else: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "gc-executor", + }) + plan.sort(key=lambda row: row["pct"], reverse=True) + return plan + + +def main() -> int: + ap = argparse.ArgumentParser(description="Plan disk GC actions with the report-only gate.") + ap.add_argument("--scan", help="JSON file: list of {id|ct|hostname|ip, usage_pct}") + ap.add_argument("--contract", default=str(DEFAULT_CONTRACT)) + ap.add_argument("--json", action="store_true", help="emit the plan as JSON") + args = ap.parse_args() + + raw = pathlib.Path(args.scan).read_text() if args.scan else sys.stdin.read() + try: + scan = json.loads(raw) + except json.JSONDecodeError as exc: + print(f"invalid scan JSON: {exc}", file=sys.stderr) + return 1 + if not isinstance(scan, list): + print("scan must be a JSON list", file=sys.stderr) + return 1 + + report_only = load_report_only_guests(pathlib.Path(args.contract)) + plan = build_plan(scan, report_only) + + if args.json: + print(json.dumps(plan, indent=2)) + return 0 + + if not plan: + print("no threats (nothing at or above 75%)") + return 0 + for row in plan: + if row["action"] == "report-only": + print(f" {row['target']} {row['level']} {row['pct']}% -> REPORT-ONLY (no GC) — {row['reason']}") + else: + print(f" {row['target']} {row['level']} {row['pct']}% -> gc-executor") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/pct-run.sh b/scripts/pct-run.sh index afbf56a..d4239d5 100755 --- a/scripts/pct-run.sh +++ b/scripts/pct-run.sh @@ -11,11 +11,13 @@ set -euo pipefail # ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ── declare -A CT_NODES=( # amdpve (192.168.68.15) - [111]=amdpve # tdunna + [105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh) [112]=amdpve # tanko [113]=amdpve # baggy [115]=amdpve # scottdenya + [120]=amdpve # adguard2 (added 2026-09-12) # minipve (192.168.68.12) + [100]=minipve # abiba (was hwepve) [102]=minipve # adguard (was acerpve) [104]=minipve # authentik [110]=minipve # gitea @@ -25,17 +27,16 @@ declare -A CT_NODES=( [106]=storepve # ra-h-os [107]=storepve # proxmox-backup [108]=storepve # media + [111]=storepve # tdunna (was amdpve — corrected 2026-09-12; live per pvesh) [117]=storepve # zulip [118]=storepve # jdownloader # acerpve (192.168.68.9) — no CTs (bare metal GPU .8) - [100]=minipve # abiba (was hwepve) - [105]=minipve # kagentz (was hwepve) # ocupve (192.168.68.5) — no CTs (bare metal GPU .110) # # REMOVED CTs (migrated to bare metal, decommissioned, or VMs): - # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) - # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) - # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) + # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) [QEMU VM on acerpve] + # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) [QEMU VM on ocupve] + # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) [QEMU VM on storepve] ) # Each node must be root-accessible via SSH hostname diff --git a/scripts/prose-ai-review.sh b/scripts/prose-ai-review.sh index cefc089..76c394a 100755 --- a/scripts/prose-ai-review.sh +++ b/scripts/prose-ai-review.sh @@ -47,17 +47,19 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol The infrastructure-control.prose.md contract is the canonical reference for the cluster topology: **Proxmox Cluster "Tabiri" (5 nodes):** -- amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya -- minipve (192.168.68.12): abiba, kagentz, adguard, authentik, gitea, syslog-api, infisical-vault -- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip +- amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2 +- minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault +- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna - acerpve (192.168.68.9): llm-gpu - ocupve (192.168.68.5): ocu-llm -**CT IDs (verified 2026-07-24 against PVE API):** +**CT IDs (verified 2026-09-12 against PVE API):** 100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os 107:pbs 108:media 110:gitea 111:tdunna 112:tanko 113:baggy 115:scottdenya 116:syslog-api 117:zulip -118:jdownloader 119:infisical-vault +118:jdownloader 119:infisical-vault 120:adguard2 + +**CT 111 (tdunna, 192.168.68.129) is REPORT-ONLY — Theo's box; alert only, never garbage-collect.** **NO CT 122, CT 123, or .19 exist in the cluster.** diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py new file mode 100644 index 0000000..cd19d82 --- /dev/null +++ b/tests/test_disk_gc_report_only.py @@ -0,0 +1,185 @@ +"""Regression tests for the disk-gc report-only gate (CT 111 / tdunna / .129). + +WHY THIS FILE EXISTS: `disk-gc-threat-response.prose.md` defined AMBER as "GC scheduled +for next run" and its Execution loop called `gc-executor` for EVERY threat, with no +guest-level exclusion. CT 111 (tdunna, 192.168.68.129) belongs to Theo and is +report-only per the captain (2026-08-17, re-confirmed 2026-09-10) — so a single AMBER +reading on that guest would have scheduled GC commands (apt clean, journal vacuum, +log/tmp deletion, snap removal) against someone else's box. The only marker was +frontmatter `report_only_agents`, which names an AGENT while the scan unit is a GUEST. + +These tests execute the real planner (`scripts/disk-gc-plan.py`) and assert observable +behaviour: an excluded guest never produces a `gc-executor` action at any level, while +our own guests still do. +""" +from __future__ import annotations + +import json +import pathlib +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +PLAN = ROOT / "scripts" / "disk-gc-plan.py" + + +def _plan(scan, tmp_path): + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps(scan)) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def _actions_for(plan, target): + return [row for row in plan if str(row["target"]) == str(target)] + + +def test_excluded_guest_never_gets_gc_at_any_level(tmp_path): + """CT 111 at AMBER, RED and CRITICAL — always report-only, never gc-executor.""" + for pct, level in ((84, "AMBER"), (90, "RED"), (97, "CRITICAL")): + plan = _plan([{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", + "usage_pct": pct}], tmp_path) + rows = _actions_for(plan, 111) + assert rows, f"CT 111 must still be reported at {level}" + assert rows[0]["level"] == level + assert rows[0]["action"] == "report-only", rows + assert not any(r["action"] == "gc-executor" for r in rows) + + +def test_exclusion_matches_on_any_identity_key(tmp_path): + """The gate is keyed on guest/host, so id, ct, ctid, hostname or IP all match.""" + for entry in ({"id": 111, "usage_pct": 95}, + {"ct": 111, "usage_pct": 95}, + {"ctid": 111, "usage_pct": 95}, + {"hostname": "tdunna", "usage_pct": 95}, + {"ip": "192.168.68.129", "usage_pct": 95}): + plan = _plan([entry], tmp_path) + assert all(r["action"] == "report-only" for r in plan), (entry, plan) + + +def test_exclusion_matches_encoded_identities(tmp_path): + """A differently-encoded CT 111 id must not slip past the gate to gc-executor.""" + encodings = ({"id": 111.0}, + {"id": "111.0"}, + {"id": "lxc/111"}, + {"id": "qemu/111"}, + {"id": "0111"}, + {"id": " 111 "}) + for alias in encodings: + plan = _plan([{**alias, "usage_pct": 97}], tmp_path) + assert plan, alias + assert plan[0]["action"] == "report-only", (alias, plan) + + +def test_our_own_guests_still_get_gc(tmp_path): + """acerpve .9 and amdpve .15 are ours — they must still be acted on.""" + plan = _plan([{"hostname": "acerpve", "ip": "192.168.68.9", "usage_pct": 77}, + {"hostname": "amdpve", "ip": "192.168.68.15", "usage_pct": 76}], tmp_path) + assert len(plan) == 2 + assert all(r["action"] == "gc-executor" for r in plan), plan + + +def test_below_threshold_emits_nothing(tmp_path): + """GREEN guests produce no action at all.""" + assert _plan([{"id": 111, "usage_pct": 40}], tmp_path) == [] + + +def test_agent_name_alone_does_not_gate_a_guest(tmp_path): + """An agent-name marker must not be the gate: an unrelated guest still gets GC.""" + plan = _plan([{"id": 999, "hostname": "koby", "usage_pct": 95}], tmp_path) + assert plan and plan[0]["action"] == "gc-executor" + + +def test_unidentified_threat_fails_closed(tmp_path): + """A threshold-crossing entry with no recognized identity must not schedule GC.""" + plan = _plan([{"usage_pct": 97}], tmp_path) + assert plan, "an unidentified threat must still be reported" + assert plan[0]["action"] == "report-only", plan + assert plan[0]["reason"], plan + + +def test_exclusion_entry_without_identity_fails_closed(tmp_path): + """A mis-typed exclusion entry must break the run, never silently disable the gate.""" + contract = tmp_path / "broken.prose.md" + contract.write_text( + "```yaml\n" + "report_only_guests:\n" + " - node: storepve\n" + " reason: \"typo - no guest identity\"\n" + "```\n" + ) + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}])) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode != 0, proc.stdout + assert "gc-executor" not in proc.stdout + assert "identity" in proc.stderr.lower(), proc.stderr + + +def test_empty_exclusion_block_fails_closed(tmp_path): + """An emptied report_only_guests list must break the run, not disable the gate.""" + contract = tmp_path / "empty.prose.md" + contract.write_text("```yaml\nreport_only_guests: []\n```\n") + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}])) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode != 0, proc.stdout + assert "gc-executor" not in proc.stdout + assert "report-only gate" in proc.stderr.lower(), proc.stderr + + +def _plan_with_contract(scan, contract_text, tmp_path, name): + contract = tmp_path / name + contract.write_text(contract_text) + scan_file = tmp_path / f"scan-{name}.json" + scan_file.write_text(json.dumps(scan)) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def test_gate_is_read_from_the_contract_block(tmp_path): + """The gate is data-driven by the contract block: the planner excludes the guest + when the block names it and acts on it when the block does not. Executes the real + planner interface against both fixtures so the behaviour change is observable.""" + scan = [{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", "usage_pct": 95}] + with_gate = ( + "```yaml\n" + "report_only_guests:\n" + " - guest: 111\n" + " hostname: tdunna\n" + " ip: 192.168.68.129\n" + " reason: \"fixture reason\"\n" + "```\n" + ) + without_gate = ( + "```yaml\n" + "report_only_guests:\n" + " - guest: 999\n" + " reason: \"fixture excludes a different guest\"\n" + "```\n" + ) + + gated = _plan_with_contract(scan, with_gate, tmp_path, "gated.prose.md") + assert gated[0]["action"] == "report-only", gated + assert gated[0]["reason"] == "fixture reason", gated + + ungated = _plan_with_contract(scan, without_gate, tmp_path, "ungated.prose.md") + assert ungated[0]["action"] == "gc-executor", ungated + assert ungated[0]["action"] != gated[0]["action"]