From 7b8cc5f9ace6e56dc1c4ee25aa9626448156f323 Mon Sep 17 00:00:00 2001 From: abiba Date: Sat, 12 Sep 2026 18:33:02 +0000 Subject: [PATCH 1/9] fix(disk-gc): hard guest-level report-only gate for CT 111/.129; correct stale fleet map CT 111 (tdunna, 192.168.68.129) is Theo's box and is report-only per the captain (2026-08-17, re-confirmed 2026-09-10). The contract defined AMBER as 'GC scheduled for next run' and its Execution loop called gc-executor for EVERY threat with no guest-level exclusion - so a single AMBER reading there would have scheduled apt clean / journal vacuum / log+tmp deletion against someone else's box. The only marker was frontmatter report_only_agents, which names an AGENT while the scan unit is a GUEST. - Gate the Execution loop on a guest/host-keyed report_only_guests block (guest id, hostname and IP all match); an excluded guest is alerted and skipped, so no gc-executor call is constructed for it at any level. - Carry the ruling in the contract body next to the loop, not only in frontmatter. - scripts/disk-gc-plan.py: executable planner that reads the contract's authoritative exclusion block and emits the action plan; tests/ covers it. - Correct the stale fleet map against pvesh /cluster/resources: CT 105 -> amdpve (was minipve), CT 111 -> storepve (was amdpve), add guests 118/119/120, and fix the '15 CTs' counts (20 guests: 17 LXC + 3 QEMU VMs). - scripts/pct-run.sh: same stale map (105/111 wrong node, 120 missing) - this is why pct-run 111/105 failed. - Fold in disk-gc-ct100-probe-gap-20260911: CT 100 verifiably works through pct-run now that the map is correct; documented that the scanner must probe it like any other guest, never via a local-only path (the scanner runs inside CT 100). --- disk-gc-threat-response.prose.md | 126 ++++++++++++++++++----- scripts/disk-gc-plan.py | 164 ++++++++++++++++++++++++++++++ scripts/pct-run.sh | 13 +-- tests/test_disk_gc_report_only.py | 89 ++++++++++++++++ 4 files changed, 358 insertions(+), 34 deletions(-) create mode 100644 scripts/disk-gc-plan.py create mode 100644 tests/test_disk_gc_report_only.py diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md index 34243d6..7a61f5d 100644 --- a/disk-gc-threat-response.prose.md +++ b/disk-gc-threat-response.prose.md @@ -1,11 +1,23 @@ --- report_only_agents: - koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129 +# ⛔ GUEST/HOST-KEYED report-only list. This is the gate the GC executor MUST honour. +# An agent-name marker above is NOT sufficient: the scan unit is a guest, and an agent +# marker can silently miss the guest it lives on. Key exclusions on the guest/host. +report_only_guests: + - guest: 111 + hostname: tdunna + ip: 192.168.68.129 + node: storepve + reason: > + Theo's box. Captain ruling 2026-08-17, re-confirmed 2026-09-10: Theo handles + CT 111 himself. DETECT-AND-REPORT-ONLY at every threat level. kind: responsibility name: disk-gc-threat-response description: > Recurring disk health scan, garbage collection, and threat response - across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident + across 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts + (fleet verified against `pvesh get /cluster/resources` 2026-09-12). Triggered by incident 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from Docker image bloat — 5 dangling images, 15 build cache layers. Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker @@ -25,7 +37,7 @@ logged within 5 minutes of discovery. ## Scope -All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH. +All 20 Proxmox guests (17 LXC + 3 QEMU VMs) via `pct-run` + 3 GPU bare-metal hosts via direct SSH. Docker hosts get special attention: | Host | CT | Disk Risk | GC Strategy | @@ -99,10 +111,43 @@ and escalation trail. ## Execution +### Hard gate: report-only guests (READ THIS BEFORE RUNNING GC) + +**CT 111 / hostname `tdunna` / 192.168.68.129 is DETECT-AND-REPORT-ONLY.** It belongs to Theo. +The captain ruled 2026-08-17 and re-confirmed 2026-09-10 that Theo handles CT 111 himself. +At **every** threat level — AMBER, RED, or CRITICAL — the executor must: + +- push the threat row and alert the owner, and +- **never** call `gc-executor`, and **never** run any GC command against that guest: no + `apt-get clean/autoremove`, no `journalctl --vacuum-*`, no `find /var/log -delete`, no + `/tmp`/`/var/tmp` deletion, no snap removal, no `docker system prune`. + +This gate is keyed on **guest id / hostname / IP**, not on an agent name. The frontmatter +`report_only_agents` marker (e.g. `koby`) names an AGENT while the scan unit is a GUEST, so an +agent-name marker can silently miss the guest it lives on — it must never be the only gate. + +**The authoritative machine-readable exclusion list is the YAML block below.** The executor +reads it at run time; `scripts/disk-gc-plan.py` turns a fleet scan into the action plan using it. +Extend the list here, never by hand-maintaining a second copy. + +```yaml +# disk-gc report-only guests — authoritative. Keyed on guest/host, not agent. +report_only_guests: + - guest: 111 + hostname: tdunna + ip: 192.168.68.129 + node: storepve + reason: "Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10" +``` + +### Loop + ```prose let fleet = call disk-scanner scope: all +let report_only = load-report-only-guests() -- from the YAML block above + let threats = [] for ct in fleet: if ct.usage_pct >= 95: @@ -116,11 +161,19 @@ for ct in fleet: sort threats by pct desc for threat in threats: + -- HARD GATE: an excluded guest is alerted and skipped. No gc-executor call is + -- constructed for it at any level, so no GC command can be emitted for it. + if threat.ct in report_only: + call alerter + threat: threat + result: { action: "report-only", reason: report_only[threat.ct].reason } + continue + let result = call gc-executor ct: threat.ct level: threat.level strategy: lookup-gc-strategy(threat.ct) - + call alerter threat: threat result: result @@ -239,7 +292,9 @@ dangling images and orphaned build cache. No automated GC was in place. ## Incident Log: 2026-07-09 — amdpve docker bloat ### Discovery -Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts. +Scheduled fleet disk scan via `pct-run` across all 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts. +> **Report-only gate applies to this scan:** CT 111 (`tdunna`, 192.168.68.129) is alerted but never +> garbage-collected at any level. amdpve (.15) flagged at 78% (AMBER threshold: 75%). ### Diagnosis @@ -272,25 +327,35 @@ one-off GPU builds. No automated post-migration cleanup was in place. - Contract now scans GPU bare-metal hosts alongside CTs - Access via `pct-run` script for all CTs (no hardcoded IPs) -## Access Matrix (documented 2026-07-09) +## Access Matrix (verified against `pvesh get /cluster/resources` 2026-09-12) -### CT Access (via pct-run) -| CT | Name | Node | Status | -|----|------|------|--------| -| 100 | abiba | minipve | local | -| 102 | adguard | minipve | ✅ reachable | -| 104 | authentik | minipve | ✅ reachable | -| 105 | kagentz | minipve | ✅ reachable | -| 106 | ra-h-os | storepve | ✅ reachable | -| 107 | pbs | storepve | ✅ reachable | -| 108 | media | storepve | ✅ reachable | -| 110 | gitea | minipve | ✅ reachable | -| 111 | tdunna | amdpve | ✅ reachable | -| 112 | tanko | amdpve | ✅ reachable | -| 113 | baggy | amdpve | ✅ reachable | -| 115 | scottdenya | amdpve | ✅ reachable | -| 116 | syslog-api | minipve | ✅ reachable | -| 117 | zulip | storepve | ✅ reachable | +### Guest Access (via `pct-run` — CT id only, node resolved by `scripts/pct-run.sh`) +| Guest | Name | Node | Type | Status | +|------|------|------|------|--------| +| 100 | abiba | minipve | lxc | ✅ reachable (probed via pct-run like any other guest; no local shortcut) | +| 102 | adguard | minipve | lxc | ✅ reachable | +| 104 | authentik | minipve | lxc | ✅ reachable | +| 105 | kagentz | **amdpve** | lxc | ✅ reachable (was documented as minipve — corrected) | +| 106 | ra-h-os | storepve | lxc | ✅ reachable | +| 107 | pbs | storepve | lxc | ✅ reachable | +| 108 | media | storepve | lxc | ✅ reachable | +| 110 | gitea | minipve | lxc | ✅ reachable | +| 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) | +| 112 | tanko | amdpve | lxc | ✅ reachable | +| 113 | baggy | amdpve | lxc | ✅ reachable | +| 115 | scottdenya | amdpve | lxc | ✅ reachable | +| 116 | syslog-api | minipve | lxc | ✅ reachable | +| 117 | zulip | storepve | lxc | ✅ reachable | +| 118 | jdownloader | storepve | lxc | ✅ reachable | +| 119 | infisical-vault | minipve | lxc | ✅ reachable | +| 120 | adguard2 | amdpve | lxc | ✅ reachable | + +### QEMU VMs (via direct SSH) +| VM | Name | Node | IP | Status | +|----|------|------|-----|--------| +| 101 | llm-gpu (workload now bare metal .8) | acerpve | — | ✅ reachable | +| 103 | ocu-llm (workload now bare metal .110) | ocupve | — | ✅ reachable | +| 109 | docker-vm | storepve | 192.168.68.7 | ✅ reachable | ### GPU Bare Metal (via direct SSH) | Host | IP | GPU | Status | @@ -299,9 +364,14 @@ one-off GPU builds. No automated post-migration cleanup was in place. | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable | | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable | -### KVM VM (via direct SSH) -| Host | IP | Role | Status | -|------|-----|------|--------| -| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable | - -> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7. +> **Fleet count:** 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts. Corrected +> 2026-09-12: CT 105 → amdpve, CT 111 → storepve, and guests 118/119/120 were missing. +> +> **CT 100 probe gap (folded in):** CT 100 previously reported "unreachable (not reported)" every +> run. Root cause is the same stale access layer: `pct-run` resolves the guest's node from its map, +> and the map/contract must reflect `pvesh /cluster/resources`. Verified working from inside CT 100: +> `scripts/pct-run.sh 100 "df -P / | tail -1"` → `23% /`. Probe CT 100 through `pct-run` like any +> other guest — never through a local-only path, since the scanner itself runs inside CT 100 and a +> container has no `pct` binary. +> +> **KVM VM:** CT 109 (docker-vm) is a QEMU VM, not LXC — access via SSH .7. diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py new file mode 100644 index 0000000..e6d53ec --- /dev/null +++ b/scripts/disk-gc-plan.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""disk-gc-plan — turn a fleet disk scan into the GC action plan. + +This is the executable side of `disk-gc-threat-response.prose.md`. It exists so the +report-only gate is enforced by code that can be tested, rather than by prose the +executor might misread. + +THE HARD GATE: guests listed in the contract's `report_only_guests` block are +DETECT-AND-REPORT-ONLY at EVERY level (AMBER, RED, CRITICAL). This tool will never +emit a `gc-executor` action for one, so no GC command can be constructed for it. + +The gate is keyed on GUEST identity — guest id, hostname, or IP — never on an agent +name. An agent-name marker can silently miss the guest it lives on; a guest marker +cannot. + +The authoritative exclusion list lives in the contract itself (the fenced ```yaml +block containing `report_only_guests:`). This tool reads it from there so there is +only ever one copy. + +Usage: + disk-gc-plan.py --scan scan.json # [{"id":111,"usage_pct":84}, ...] + cat scan.json | disk-gc-plan.py # same, via stdin + disk-gc-plan.py --scan scan.json --json # machine-readable plan + +Scan entries may carry any of: id / guest / vmid, hostname / name, ip. +Exit codes: 0 ok, 1 usage/parse error. +""" +from __future__ import annotations + +import argparse +import json +import pathlib +import re +import sys + +import yaml + +REPO = pathlib.Path(__file__).resolve().parent.parent +DEFAULT_CONTRACT = REPO / "disk-gc-threat-response.prose.md" + +AMBER, RED, CRITICAL = 75, 85, 95 + + +def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: + """Read the authoritative report_only_guests block out of the contract. + + The contract carries it as a fenced ```yaml block. Parsing the declared, + machine-readable block is the intended interface — the contract owns the list. + """ + text = contract_path.read_text(encoding="utf-8") + for block in re.findall(r"```yaml\n(.*?)```", text, re.S): + if "report_only_guests:" in block: + data = yaml.safe_load(block) + guests = data.get("report_only_guests") or [] + if not isinstance(guests, list): + raise SystemExit("report_only_guests must be a list") + return guests + raise SystemExit( + f"no authoritative report_only_guests block found in {contract_path}" + ) + + +def _keys(entry: dict) -> set[str]: + """Guest/host identity keys for an exclusion entry.""" + out: set[str] = set() + for field in ("guest", "id", "vmid", "hostname", "name", "ip"): + value = entry.get(field) + if value is not None and str(value).strip(): + out.add(str(value).strip().lower()) + return out + + +def _entry_keys(scan_entry: dict) -> set[str]: + out: set[str] = set() + for field in ("id", "guest", "vmid", "hostname", "name", "ip"): + value = scan_entry.get(field) + if value is not None and str(value).strip(): + out.add(str(value).strip().lower()) + return out + + +def level_for(pct: float) -> str | None: + if pct >= CRITICAL: + return "CRITICAL" + if pct >= RED: + return "RED" + if pct >= AMBER: + return "AMBER" + return None + + +def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]: + excluded = [(e, _keys(e)) for e in report_only] + plan: list[dict] = [] + for entry in scan: + pct = entry.get("usage_pct") + if pct is None: + continue + level = level_for(float(pct)) + if level is None: + continue # GREEN: log only, no action + scan_keys = _entry_keys(entry) + match = next((e for e, keys in excluded if keys & scan_keys), None) + target = next( + (entry.get(k) for k in ("id", "guest", "vmid", "hostname", "name", "ip") + if entry.get(k) not in (None, "")), + "?", + ) + if match is not None: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "report-only", + "reason": match.get("reason", "").strip(), + }) + else: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "gc-executor", + }) + plan.sort(key=lambda row: row["pct"], reverse=True) + return plan + + +def main() -> int: + ap = argparse.ArgumentParser(description="Plan disk GC actions with the report-only gate.") + ap.add_argument("--scan", help="JSON file: list of {id|hostname|ip, usage_pct}") + ap.add_argument("--contract", default=str(DEFAULT_CONTRACT)) + ap.add_argument("--json", action="store_true", help="emit the plan as JSON") + args = ap.parse_args() + + raw = pathlib.Path(args.scan).read_text() if args.scan else sys.stdin.read() + try: + scan = json.loads(raw) + except json.JSONDecodeError as exc: + print(f"invalid scan JSON: {exc}", file=sys.stderr) + return 1 + if not isinstance(scan, list): + print("scan must be a JSON list", file=sys.stderr) + return 1 + + report_only = load_report_only_guests(pathlib.Path(args.contract)) + plan = build_plan(scan, report_only) + + if args.json: + print(json.dumps(plan, indent=2)) + return 0 + + if not plan: + print("no threats (nothing at or above 75%)") + return 0 + for row in plan: + if row["action"] == "report-only": + print(f" {row['target']} {row['level']} {row['pct']}% -> REPORT-ONLY (no GC) — {row['reason']}") + else: + print(f" {row['target']} {row['level']} {row['pct']}% -> gc-executor") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/pct-run.sh b/scripts/pct-run.sh index afbf56a..d4239d5 100755 --- a/scripts/pct-run.sh +++ b/scripts/pct-run.sh @@ -11,11 +11,13 @@ set -euo pipefail # ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ── declare -A CT_NODES=( # amdpve (192.168.68.15) - [111]=amdpve # tdunna + [105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh) [112]=amdpve # tanko [113]=amdpve # baggy [115]=amdpve # scottdenya + [120]=amdpve # adguard2 (added 2026-09-12) # minipve (192.168.68.12) + [100]=minipve # abiba (was hwepve) [102]=minipve # adguard (was acerpve) [104]=minipve # authentik [110]=minipve # gitea @@ -25,17 +27,16 @@ declare -A CT_NODES=( [106]=storepve # ra-h-os [107]=storepve # proxmox-backup [108]=storepve # media + [111]=storepve # tdunna (was amdpve — corrected 2026-09-12; live per pvesh) [117]=storepve # zulip [118]=storepve # jdownloader # acerpve (192.168.68.9) — no CTs (bare metal GPU .8) - [100]=minipve # abiba (was hwepve) - [105]=minipve # kagentz (was hwepve) # ocupve (192.168.68.5) — no CTs (bare metal GPU .110) # # REMOVED CTs (migrated to bare metal, decommissioned, or VMs): - # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) - # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) - # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) + # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) [QEMU VM on acerpve] + # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) [QEMU VM on ocupve] + # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) [QEMU VM on storepve] ) # Each node must be root-accessible via SSH hostname diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py new file mode 100644 index 0000000..2df496a --- /dev/null +++ b/tests/test_disk_gc_report_only.py @@ -0,0 +1,89 @@ +"""Regression tests for the disk-gc report-only gate (CT 111 / tdunna / .129). + +WHY THIS FILE EXISTS: `disk-gc-threat-response.prose.md` defined AMBER as "GC scheduled +for next run" and its Execution loop called `gc-executor` for EVERY threat, with no +guest-level exclusion. CT 111 (tdunna, 192.168.68.129) belongs to Theo and is +report-only per the captain (2026-08-17, re-confirmed 2026-09-10) — so a single AMBER +reading on that guest would have scheduled GC commands (apt clean, journal vacuum, +log/tmp deletion, snap removal) against someone else's box. The only marker was +frontmatter `report_only_agents`, which names an AGENT while the scan unit is a GUEST. + +These tests execute the real planner (`scripts/disk-gc-plan.py`) and assert observable +behaviour: an excluded guest never produces a `gc-executor` action at any level, while +our own guests still do. +""" +from __future__ import annotations + +import json +import pathlib +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +PLAN = ROOT / "scripts" / "disk-gc-plan.py" + + +def _plan(scan, tmp_path): + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps(scan)) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def _actions_for(plan, target): + return [row for row in plan if str(row["target"]) == str(target)] + + +def test_excluded_guest_never_gets_gc_at_any_level(tmp_path): + """CT 111 at AMBER, RED and CRITICAL — always report-only, never gc-executor.""" + for pct, level in ((84, "AMBER"), (90, "RED"), (97, "CRITICAL")): + plan = _plan([{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", + "usage_pct": pct}], tmp_path) + rows = _actions_for(plan, 111) + assert rows, f"CT 111 must still be reported at {level}" + assert rows[0]["level"] == level + assert rows[0]["action"] == "report-only", rows + assert not any(r["action"] == "gc-executor" for r in rows) + + +def test_exclusion_matches_on_any_identity_key(tmp_path): + """The gate is keyed on guest/host, so id, hostname or IP all match.""" + for entry in ({"id": 111, "usage_pct": 95}, + {"hostname": "tdunna", "usage_pct": 95}, + {"ip": "192.168.68.129", "usage_pct": 95}): + plan = _plan([entry], tmp_path) + assert all(r["action"] == "report-only" for r in plan), (entry, plan) + + +def test_our_own_guests_still_get_gc(tmp_path): + """acerpve .9 and amdpve .15 are ours — they must still be acted on.""" + plan = _plan([{"hostname": "acerpve", "ip": "192.168.68.9", "usage_pct": 77}, + {"hostname": "amdpve", "ip": "192.168.68.15", "usage_pct": 76}], tmp_path) + assert len(plan) == 2 + assert all(r["action"] == "gc-executor" for r in plan), plan + + +def test_below_threshold_emits_nothing(tmp_path): + """GREEN guests produce no action at all.""" + assert _plan([{"id": 111, "usage_pct": 40}], tmp_path) == [] + + +def test_agent_name_alone_does_not_gate_a_guest(tmp_path): + """An agent-name marker must not be the gate: an unrelated guest still gets GC.""" + plan = _plan([{"id": 999, "hostname": "koby", "usage_pct": 95}], tmp_path) + assert plan and plan[0]["action"] == "gc-executor" + + +def test_contract_carries_the_guest_keyed_exclusion(tmp_path): + """The authoritative gate must exist and identify CT 111 by guest/host.""" + plan = _plan([{"id": 111, "usage_pct": 95}], tmp_path) + reason = _actions_for(plan, 111)[0]["reason"] + assert reason, "the exclusion must carry a reason for the alert" + contract = (ROOT / "disk-gc-threat-response.prose.md").read_text() + assert "report_only_guests:" in contract + assert "192.168.68.129" in contract + assert "tdunna" in contract -- 2.54.0 From 4ea2d0309fc7ae7483ff5f87ab8021406cb2b603 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 18:38:19 +0000 Subject: [PATCH 2/9] no-mistakes(review): Fix report-only gate, dedupe list, correct fleet map --- disk-gc-threat-response.prose.md | 59 +++++++++++-------------------- infrastructure-control.prose.md | 30 +++++++++------- scripts/disk-gc-plan.py | 14 ++------ scripts/prose-ai-review.sh | 12 ++++--- tests/test_disk_gc_report_only.py | 46 +++++++++++++++++++----- 5 files changed, 85 insertions(+), 76 deletions(-) mode change 100644 => 100755 scripts/disk-gc-plan.py diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md index 7a61f5d..acb3ef8 100644 --- a/disk-gc-threat-response.prose.md +++ b/disk-gc-threat-response.prose.md @@ -1,17 +1,8 @@ --- report_only_agents: - koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129 -# ⛔ GUEST/HOST-KEYED report-only list. This is the gate the GC executor MUST honour. -# An agent-name marker above is NOT sufficient: the scan unit is a guest, and an agent -# marker can silently miss the guest it lives on. Key exclusions on the guest/host. -report_only_guests: - - guest: 111 - hostname: tdunna - ip: 192.168.68.129 - node: storepve - reason: > - Theo's box. Captain ruling 2026-08-17, re-confirmed 2026-09-10: Theo handles - CT 111 himself. DETECT-AND-REPORT-ONLY at every threat level. +# ⛔ The guest/host-keyed report-only gate the GC executor MUST honour lives in the body +# "Hard gate" YAML block below — that block is authoritative and is the only copy. kind: responsibility name: disk-gc-threat-response description: > @@ -37,7 +28,7 @@ logged within 5 minutes of discovery. ## Scope -All 20 Proxmox guests (17 LXC + 3 QEMU VMs) via `pct-run` + 3 GPU bare-metal hosts via direct SSH. +All 20 Proxmox guests (17 LXC via `pct-run` + 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts via direct SSH. Docker hosts get special attention: | Host | CT | Disk Risk | GC Strategy | @@ -128,7 +119,8 @@ agent-name marker can silently miss the guest it lives on — it must never be t **The authoritative machine-readable exclusion list is the YAML block below.** The executor reads it at run time; `scripts/disk-gc-plan.py` turns a fleet scan into the action plan using it. -Extend the list here, never by hand-maintaining a second copy. +Extend the list here, never by hand-maintaining a second copy. The Execution loop below MUST +call that planner and MUST NOT reimplement the gate. ```yaml # disk-gc report-only guests — authoritative. Keyed on guest/host, not agent. @@ -146,41 +138,32 @@ report_only_guests: let fleet = call disk-scanner scope: all -let report_only = load-report-only-guests() -- from the YAML block above +-- The report-only gate is IMPLEMENTED IN scripts/disk-gc-plan.py and MUST NOT be +-- reimplemented here. That planner reads the contract's `report_only_guests` YAML block +-- and matches on guest id OR hostname OR IP, so the tested gate is the executed gate. +let plan = call disk-gc-plan + fleet: fleet -let threats = [] -for ct in fleet: - if ct.usage_pct >= 95: - push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct } - else if ct.usage_pct >= 85: - push threats { ct: ct.id, level: "RED", pct: ct.usage_pct } - else if ct.usage_pct >= 75: - push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct } - --- sort by severity descending -sort threats by pct desc - -for threat in threats: - -- HARD GATE: an excluded guest is alerted and skipped. No gc-executor call is - -- constructed for it at any level, so no GC command can be emitted for it. - if threat.ct in report_only: +for row in plan: + if row.action == "report-only": + -- Excluded guest: alert only. No gc-executor call is constructed for it, at any level. call alerter - threat: threat - result: { action: "report-only", reason: report_only[threat.ct].reason } + threat: row + result: { action: "report-only", reason: row.reason } continue let result = call gc-executor - ct: threat.ct - level: threat.level - strategy: lookup-gc-strategy(threat.ct) + ct: row.target + level: row.level + strategy: lookup-gc-strategy(row.target) call alerter - threat: threat + threat: row result: result call summary-reporter fleet: fleet - threats: threats + plan: plan ``` ## GC Strategies by Host Type @@ -292,7 +275,7 @@ dangling images and orphaned build cache. No automated GC was in place. ## Incident Log: 2026-07-09 — amdpve docker bloat ### Discovery -Scheduled fleet disk scan via `pct-run` across all 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts. +Scheduled fleet disk scan across all 20 Proxmox guests (17 LXC via `pct-run`, 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts. > **Report-only gate applies to this scan:** CT 111 (`tdunna`, 192.168.68.129) is alerted but never > garbage-collected at any level. amdpve (.15) flagged at 78% (AMBER threshold: 75%). diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index d7a4a36..5c40667 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -16,8 +16,8 @@ description: > **Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster (now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve (192.168.68.4) is a standalone PVE node + NetBird routing peer; - London relocation pending. CTs 100 (abiba) and 105 (kagentz) moved - to minipve. + London relocation pending. CT 100 (abiba) is on minipve; CT 105 + (kagentz) is on amdpve. --- # Infrastructure Control Pattern @@ -105,16 +105,16 @@ description: > | Node | IP | CPU | RAM | VMs/CTs | Role | |------|----|-----|-----|---------|------| -| minipve | .12 | 16C | 30GB | abiba, kagentz, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | -| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute | -| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat | +| minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | +| amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute | +| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat | | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | -> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on -> minipve at .10, not acerpve. Abiba (CT 100) and kagentz (CT 105) are on -> minipve (moved from hwepve 2026-08-15). Mumuni runs inside Abiba CT100 -> (.24); CT 114 (mumuni) no longer exists in the cluster. +> **Note:** CTs on storepve include jdownloader (CT 118) and tdunna (CT 111). +> AdGuard (CT 102) is on minipve at .10, not acerpve. Abiba (CT 100) is on +> minipve (moved from hwepve 2026-08-15); kagentz (CT 105) is on amdpve. +> Mumuni runs inside Abiba CT100 (.24); CT 114 (mumuni) no longer exists in the cluster. > > **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):** > Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve. @@ -602,13 +602,13 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 102 | adguard | **minipve** | **.10** | DNS | ❌ | | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ | | 104 | authentik | minipve | .11 | OIDC | ❌ | -| 105 | kagentz | minipve | — | Agent Zero | ✅ | +| 105 | kagentz | amdpve | — | Agent Zero | ✅ | | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | | 107 | pbs | storepve | — | Backups | ❌ | | 108 | media | storepve | — | Media | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 110 | gitea | minipve | **.17** | Git | ❌ | -| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ | +| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ | | 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ | | 113 | baggy | amdpve | .114 | Hermes agent | ✅ | | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ | @@ -616,6 +616,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 117 | zulip | storepve | .19 | Chat | ❌ | | 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ | | 119 | infisical-vault | minipve | — | Vault | ❌ | +| 120 | adguard2 | amdpve | — | DNS (secondary AdGuard) | ❌ | ## Appendix C: Docker Compose Files Location @@ -636,8 +637,8 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run. | CT | Name | Node | pct-run | |-----|------|------|---------| | 100 | abiba | minipve | `pct-run 100` | -| 105 | kagentz | minipve | `pct-run 105` | -| 111 | tdunna | amdpve | `pct-run 111` | +| 105 | kagentz | amdpve | `pct-run 105` | +| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) | | 112 | tanko | amdpve | `pct-run 112` | | 113 | baggy | amdpve | `pct-run 113` | | 115 | scottdenya | amdpve | `pct-run 115` | @@ -648,6 +649,9 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run. | 107 | proxmox-backup | storepve | `pct-run 107` | | 108 | media | storepve | `pct-run 108` | | 117 | zulip | storepve | `pct-run 117` | +| 118 | jdownloader | storepve | `pct-run 118` | +| 119 | infisical-vault | minipve | `pct-run 119` | +| 120 | adguard2 | amdpve | `pct-run 120` | | 102 | adguard | **minipve** | `pct-run 102` | GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly: diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py old mode 100644 new mode 100755 index e6d53ec..f251268 --- a/scripts/disk-gc-plan.py +++ b/scripts/disk-gc-plan.py @@ -61,7 +61,8 @@ def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: def _keys(entry: dict) -> set[str]: - """Guest/host identity keys for an exclusion entry.""" + """Guest/host identity keys, shared by exclusions and scan entries so the two + sides of the gate can never key on different fields.""" out: set[str] = set() for field in ("guest", "id", "vmid", "hostname", "name", "ip"): value = entry.get(field) @@ -70,15 +71,6 @@ def _keys(entry: dict) -> set[str]: return out -def _entry_keys(scan_entry: dict) -> set[str]: - out: set[str] = set() - for field in ("id", "guest", "vmid", "hostname", "name", "ip"): - value = scan_entry.get(field) - if value is not None and str(value).strip(): - out.add(str(value).strip().lower()) - return out - - def level_for(pct: float) -> str | None: if pct >= CRITICAL: return "CRITICAL" @@ -99,7 +91,7 @@ def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]: level = level_for(float(pct)) if level is None: continue # GREEN: log only, no action - scan_keys = _entry_keys(entry) + scan_keys = _keys(entry) match = next((e for e, keys in excluded if keys & scan_keys), None) target = next( (entry.get(k) for k in ("id", "guest", "vmid", "hostname", "name", "ip") diff --git a/scripts/prose-ai-review.sh b/scripts/prose-ai-review.sh index cefc089..76c394a 100755 --- a/scripts/prose-ai-review.sh +++ b/scripts/prose-ai-review.sh @@ -47,17 +47,19 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol The infrastructure-control.prose.md contract is the canonical reference for the cluster topology: **Proxmox Cluster "Tabiri" (5 nodes):** -- amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya -- minipve (192.168.68.12): abiba, kagentz, adguard, authentik, gitea, syslog-api, infisical-vault -- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip +- amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2 +- minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault +- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna - acerpve (192.168.68.9): llm-gpu - ocupve (192.168.68.5): ocu-llm -**CT IDs (verified 2026-07-24 against PVE API):** +**CT IDs (verified 2026-09-12 against PVE API):** 100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os 107:pbs 108:media 110:gitea 111:tdunna 112:tanko 113:baggy 115:scottdenya 116:syslog-api 117:zulip -118:jdownloader 119:infisical-vault +118:jdownloader 119:infisical-vault 120:adguard2 + +**CT 111 (tdunna, 192.168.68.129) is REPORT-ONLY — Theo's box; alert only, never garbage-collect.** **NO CT 122, CT 123, or .19 exist in the cluster.** diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py index 2df496a..5f1279b 100644 --- a/tests/test_disk_gc_report_only.py +++ b/tests/test_disk_gc_report_only.py @@ -78,12 +78,40 @@ def test_agent_name_alone_does_not_gate_a_guest(tmp_path): assert plan and plan[0]["action"] == "gc-executor" -def test_contract_carries_the_guest_keyed_exclusion(tmp_path): - """The authoritative gate must exist and identify CT 111 by guest/host.""" - plan = _plan([{"id": 111, "usage_pct": 95}], tmp_path) - reason = _actions_for(plan, 111)[0]["reason"] - assert reason, "the exclusion must carry a reason for the alert" - contract = (ROOT / "disk-gc-threat-response.prose.md").read_text() - assert "report_only_guests:" in contract - assert "192.168.68.129" in contract - assert "tdunna" in contract +def _plan_with_contract(scan, contract_text, tmp_path, name): + contract = tmp_path / name + contract.write_text(contract_text) + scan_file = tmp_path / f"scan-{name}.json" + scan_file.write_text(json.dumps(scan)) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def test_gate_is_read_from_the_contract_block(tmp_path): + """The gate is data-driven by the contract block: the planner excludes the guest + when the block names it and acts on it when the block does not. Executes the real + planner interface against both fixtures so the behaviour change is observable.""" + scan = [{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", "usage_pct": 95}] + with_gate = ( + "```yaml\n" + "report_only_guests:\n" + " - guest: 111\n" + " hostname: tdunna\n" + " ip: 192.168.68.129\n" + " reason: \"fixture reason\"\n" + "```\n" + ) + without_gate = "```yaml\nreport_only_guests: []\n```\n" + + gated = _plan_with_contract(scan, with_gate, tmp_path, "gated.prose.md") + assert gated[0]["action"] == "report-only", gated + assert gated[0]["reason"] == "fixture reason", gated + + ungated = _plan_with_contract(scan, without_gate, tmp_path, "ungated.prose.md") + assert ungated[0]["action"] == "gc-executor", ungated + assert ungated[0]["action"] != gated[0]["action"] -- 2.54.0 From de1428b4ae300469ed65d0b1c8f5cd9ebb57fdd2 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 18:42:36 +0000 Subject: [PATCH 3/9] no-mistakes(review): Harden GC gate key aliases, fail closed, fix baseline --- hermes-agent-baseline.prose.md | 3 ++- scripts/disk-gc-plan.py | 24 ++++++++++++++++++------ tests/test_disk_gc_report_only.py | 12 +++++++++++- 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 71ab79f..9158303 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -24,11 +24,12 @@ done | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | |-------|-----|------|-----|---------------|------------|----------| -| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** | +| Koby | 111 | storepve | .129 | `koby` | Infisical vault | **Hermes** | | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | > **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo). +> CT 111 (tdunna, 192.168.68.129, storepve) is report-only — Theo's box; alert only, never garbage-collect. Access: `pct-run ` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. Keys are stored in Infisical vault (project=agents, env=production) and injected at diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py index f251268..3d5228c 100755 --- a/scripts/disk-gc-plan.py +++ b/scripts/disk-gc-plan.py @@ -22,7 +22,8 @@ Usage: cat scan.json | disk-gc-plan.py # same, via stdin disk-gc-plan.py --scan scan.json --json # machine-readable plan -Scan entries may carry any of: id / guest / vmid, hostname / name, ip. +Scan entries may carry any of: id / guest / vmid / ct / ctid, hostname / name, ip. +A threshold-crossing entry with no recognizable identity is reported, never GC'd. Exit codes: 0 ok, 1 usage/parse error. """ from __future__ import annotations @@ -40,6 +41,9 @@ DEFAULT_CONTRACT = REPO / "disk-gc-threat-response.prose.md" AMBER, RED, CRITICAL = 75, 85, 95 +IDENTITY_FIELDS = ("id", "guest", "vmid", "ct", "ctid", "hostname", "name", "ip") +UNIDENTIFIED_REASON = "unidentified target - refusing to schedule GC" + def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: """Read the authoritative report_only_guests block out of the contract. @@ -64,7 +68,7 @@ def _keys(entry: dict) -> set[str]: """Guest/host identity keys, shared by exclusions and scan entries so the two sides of the gate can never key on different fields.""" out: set[str] = set() - for field in ("guest", "id", "vmid", "hostname", "name", "ip"): + for field in IDENTITY_FIELDS: value = entry.get(field) if value is not None and str(value).strip(): out.add(str(value).strip().lower()) @@ -92,12 +96,20 @@ def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]: if level is None: continue # GREEN: log only, no action scan_keys = _keys(entry) - match = next((e for e, keys in excluded if keys & scan_keys), None) target = next( - (entry.get(k) for k in ("id", "guest", "vmid", "hostname", "name", "ip") - if entry.get(k) not in (None, "")), + (entry.get(k) for k in IDENTITY_FIELDS if entry.get(k) not in (None, "")), "?", ) + if not scan_keys: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "report-only", + "reason": UNIDENTIFIED_REASON, + }) + continue + match = next((e for e, keys in excluded if keys & scan_keys), None) if match is not None: plan.append({ "target": target, @@ -119,7 +131,7 @@ def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]: def main() -> int: ap = argparse.ArgumentParser(description="Plan disk GC actions with the report-only gate.") - ap.add_argument("--scan", help="JSON file: list of {id|hostname|ip, usage_pct}") + ap.add_argument("--scan", help="JSON file: list of {id|ct|hostname|ip, usage_pct}") ap.add_argument("--contract", default=str(DEFAULT_CONTRACT)) ap.add_argument("--json", action="store_true", help="emit the plan as JSON") args = ap.parse_args() diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py index 5f1279b..208d763 100644 --- a/tests/test_disk_gc_report_only.py +++ b/tests/test_disk_gc_report_only.py @@ -51,8 +51,10 @@ def test_excluded_guest_never_gets_gc_at_any_level(tmp_path): def test_exclusion_matches_on_any_identity_key(tmp_path): - """The gate is keyed on guest/host, so id, hostname or IP all match.""" + """The gate is keyed on guest/host, so id, ct, ctid, hostname or IP all match.""" for entry in ({"id": 111, "usage_pct": 95}, + {"ct": 111, "usage_pct": 95}, + {"ctid": 111, "usage_pct": 95}, {"hostname": "tdunna", "usage_pct": 95}, {"ip": "192.168.68.129", "usage_pct": 95}): plan = _plan([entry], tmp_path) @@ -78,6 +80,14 @@ def test_agent_name_alone_does_not_gate_a_guest(tmp_path): assert plan and plan[0]["action"] == "gc-executor" +def test_unidentified_threat_fails_closed(tmp_path): + """A threshold-crossing entry with no recognized identity must not schedule GC.""" + plan = _plan([{"usage_pct": 97}], tmp_path) + assert plan, "an unidentified threat must still be reported" + assert plan[0]["action"] == "report-only", plan + assert plan[0]["reason"], plan + + def _plan_with_contract(scan, contract_text, tmp_path, name): contract = tmp_path / name contract.write_text(contract_text) -- 2.54.0 From 6d65cba064f19b747fe5e5da40ff5fd800138510 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 18:47:17 +0000 Subject: [PATCH 4/9] no-mistakes(review): Canonicalize guest identities in GC report-only gate --- scripts/disk-gc-plan.py | 31 +++++++++++++++++++++++++++++-- tests/test_disk_gc_report_only.py | 14 ++++++++++++++ 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py index 3d5228c..4130815 100755 --- a/scripts/disk-gc-plan.py +++ b/scripts/disk-gc-plan.py @@ -42,6 +42,7 @@ DEFAULT_CONTRACT = REPO / "disk-gc-threat-response.prose.md" AMBER, RED, CRITICAL = 75, 85, 95 IDENTITY_FIELDS = ("id", "guest", "vmid", "ct", "ctid", "hostname", "name", "ip") +IDENTITY_TYPE_PREFIX = re.compile(r"^(?:lxc|qemu)/") UNIDENTIFIED_REASON = "unidentified target - refusing to schedule GC" @@ -64,14 +65,40 @@ def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: ) +def _canonical_number(number: float) -> str: + if float(number).is_integer(): + return str(int(number)) + return str(number).strip().lower() + + +def _normalize_identity(value: object) -> str: + """Canonicalise a guest identity so differently-encoded ids compare equal: + numeric and numeric-string ids collapse to an integer string, Proxmox + type prefixes and leading zeros are stripped, and hostnames/IPs are only + trimmed and lowercased.""" + if isinstance(value, bool): + return str(value).strip().lower() + if isinstance(value, (int, float)): + return _canonical_number(float(value)) + text = str(value).strip().lower() + text = IDENTITY_TYPE_PREFIX.sub("", text) + try: + return _canonical_number(float(text)) + except ValueError: + return text + + def _keys(entry: dict) -> set[str]: """Guest/host identity keys, shared by exclusions and scan entries so the two sides of the gate can never key on different fields.""" out: set[str] = set() for field in IDENTITY_FIELDS: value = entry.get(field) - if value is not None and str(value).strip(): - out.add(str(value).strip().lower()) + if value is None: + continue + key = _normalize_identity(value) + if key: + out.add(key) return out diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py index 208d763..bde9df2 100644 --- a/tests/test_disk_gc_report_only.py +++ b/tests/test_disk_gc_report_only.py @@ -61,6 +61,20 @@ def test_exclusion_matches_on_any_identity_key(tmp_path): assert all(r["action"] == "report-only" for r in plan), (entry, plan) +def test_exclusion_matches_encoded_identities(tmp_path): + """A differently-encoded CT 111 id must not slip past the gate to gc-executor.""" + encodings = ({"id": 111.0}, + {"id": "111.0"}, + {"id": "lxc/111"}, + {"id": "qemu/111"}, + {"id": "0111"}, + {"id": " 111 "}) + for alias in encodings: + plan = _plan([{**alias, "usage_pct": 97}], tmp_path) + assert plan, alias + assert plan[0]["action"] == "report-only", (alias, plan) + + def test_our_own_guests_still_get_gc(tmp_path): """acerpve .9 and amdpve .15 are ours — they must still be acted on.""" plan = _plan([{"hostname": "acerpve", "ip": "192.168.68.9", "usage_pct": 77}, -- 2.54.0 From 4bd6132cf546c87ce3c831460559635e0289fb7f Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 18:51:42 +0000 Subject: [PATCH 5/9] no-mistakes(review): Reject report-only exclusion entries lacking identity keys --- scripts/disk-gc-plan.py | 6 ++++++ tests/test_disk_gc_report_only.py | 22 ++++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py index 4130815..b3fa42a 100755 --- a/scripts/disk-gc-plan.py +++ b/scripts/disk-gc-plan.py @@ -59,6 +59,12 @@ def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: guests = data.get("report_only_guests") or [] if not isinstance(guests, list): raise SystemExit("report_only_guests must be a list") + for guest in guests: + if not isinstance(guest, dict) or not _keys(guest): + raise SystemExit( + "report_only_guests entry has no recognizable identity key " + f"(expected one of: {', '.join(IDENTITY_FIELDS)}): {guest!r}" + ) return guests raise SystemExit( f"no authoritative report_only_guests block found in {contract_path}" diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py index bde9df2..f3b4e48 100644 --- a/tests/test_disk_gc_report_only.py +++ b/tests/test_disk_gc_report_only.py @@ -102,6 +102,28 @@ def test_unidentified_threat_fails_closed(tmp_path): assert plan[0]["reason"], plan +def test_exclusion_entry_without_identity_fails_closed(tmp_path): + """A mis-typed exclusion entry must break the run, never silently disable the gate.""" + contract = tmp_path / "broken.prose.md" + contract.write_text( + "```yaml\n" + "report_only_guests:\n" + " - node: storepve\n" + " reason: \"typo - no guest identity\"\n" + "```\n" + ) + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}])) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode != 0, proc.stdout + assert "gc-executor" not in proc.stdout + assert "identity" in proc.stderr.lower(), proc.stderr + + def _plan_with_contract(scan, contract_text, tmp_path, name): contract = tmp_path / name contract.write_text(contract_text) -- 2.54.0 From 6272d2997802cd50fa8e9081bc905924d58f54c9 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 18:55:34 +0000 Subject: [PATCH 6/9] no-mistakes(review): Fail closed on empty report-only exclusion list --- scripts/disk-gc-plan.py | 5 +++++ tests/test_disk_gc_report_only.py | 24 +++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py index b3fa42a..ca1e867 100755 --- a/scripts/disk-gc-plan.py +++ b/scripts/disk-gc-plan.py @@ -59,6 +59,11 @@ def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: guests = data.get("report_only_guests") or [] if not isinstance(guests, list): raise SystemExit("report_only_guests must be a list") + if not guests: + raise SystemExit( + "report_only_guests is empty or missing - refusing to plan GC " + "without the report-only gate" + ) for guest in guests: if not isinstance(guest, dict) or not _keys(guest): raise SystemExit( diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py index f3b4e48..cd19d82 100644 --- a/tests/test_disk_gc_report_only.py +++ b/tests/test_disk_gc_report_only.py @@ -124,6 +124,22 @@ def test_exclusion_entry_without_identity_fails_closed(tmp_path): assert "identity" in proc.stderr.lower(), proc.stderr +def test_empty_exclusion_block_fails_closed(tmp_path): + """An emptied report_only_guests list must break the run, not disable the gate.""" + contract = tmp_path / "empty.prose.md" + contract.write_text("```yaml\nreport_only_guests: []\n```\n") + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}])) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode != 0, proc.stdout + assert "gc-executor" not in proc.stdout + assert "report-only gate" in proc.stderr.lower(), proc.stderr + + def _plan_with_contract(scan, contract_text, tmp_path, name): contract = tmp_path / name contract.write_text(contract_text) @@ -152,7 +168,13 @@ def test_gate_is_read_from_the_contract_block(tmp_path): " reason: \"fixture reason\"\n" "```\n" ) - without_gate = "```yaml\nreport_only_guests: []\n```\n" + without_gate = ( + "```yaml\n" + "report_only_guests:\n" + " - guest: 999\n" + " reason: \"fixture excludes a different guest\"\n" + "```\n" + ) gated = _plan_with_contract(scan, with_gate, tmp_path, "gated.prose.md") assert gated[0]["action"] == "report-only", gated -- 2.54.0 From 357f808a82c6e1c1622054ef70c84db10e768217 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 19:00:28 +0000 Subject: [PATCH 7/9] no-mistakes(review): Make report-only skip structural with if/else loop --- disk-gc-threat-response.prose.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md index acb3ef8..4824a2f 100644 --- a/disk-gc-threat-response.prose.md +++ b/disk-gc-threat-response.prose.md @@ -150,16 +150,15 @@ for row in plan: call alerter threat: row result: { action: "report-only", reason: row.reason } - continue + else: + let result = call gc-executor + ct: row.target + level: row.level + strategy: lookup-gc-strategy(row.target) - let result = call gc-executor - ct: row.target - level: row.level - strategy: lookup-gc-strategy(row.target) - - call alerter - threat: row - result: result + call alerter + threat: row + result: result call summary-reporter fleet: fleet -- 2.54.0 From 6e612ce37b1b9a9688b04e7f816848e7a4185fca Mon Sep 17 00:00:00 2001 From: root Date: Sat, 12 Sep 2026 19:11:09 +0000 Subject: [PATCH 8/9] no-mistakes(document): Fix stale CT 111 node maps and CI range --- AGENTS.md | 2 +- docs/probe-drift-round2-evidence.md | 4 ++++ hermes-zulip-plugin.prose.md | 6 +++--- hermes-zulip-restore.prose.md | 6 +++--- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b9a840c..95fb2a4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,7 @@ Two incidents taught us this: ### Stage 3 — AI Review - Diff is sent to `syslog-auto` model via LiteLLM - Review checks against infrastructure-control ground truth: - - CT IDs match PVE cluster (100-117, no 122/123) + - CT IDs match the PVE cluster inventory in `infrastructure-control.prose.md` Appendix B (100-120 with gaps; no 122/123) - Grafana is direct LAN :3001, NOT behind nginx - Zulip is CT 117 on storepve (bridge IP .19) - Strix Halo :8080 is firewalled to .116 only diff --git a/docs/probe-drift-round2-evidence.md b/docs/probe-drift-round2-evidence.md index c6a5b2d..827abbb 100644 --- a/docs/probe-drift-round2-evidence.md +++ b/docs/probe-drift-round2-evidence.md @@ -261,6 +261,10 @@ repaired. not exist` on .15. `agent-health-check.py` now carries the live-verified `storepve` mapping (the script is not the topology source of truth); the CRITICAL contract itself needs an authorized correction. + **✅ Resolved 2026-09-12:** the topology was corrected in its owner, + `infrastructure-control.prose.md` (CT 111 → storepve, CT 105 → amdpve), and + `scripts/pct-run.sh` now matches. This snapshot is left as observed; treat + those owner documents as authoritative. 2. **Strix Halo `:8080` firewall claim is stale.** `prose-ai-review.sh` ground-truth rule #4 and `gpu-monitor.prose.md` say `:8080` is firewalled to `.116` only and `.24` cannot probe it. Live on .15: diff --git a/hermes-zulip-plugin.prose.md b/hermes-zulip-plugin.prose.md index 8dae1cf..26c8458 100644 --- a/hermes-zulip-plugin.prose.md +++ b/hermes-zulip-plugin.prose.md @@ -47,7 +47,7 @@ connectivity recovery including end-to-end DM validation. ## Requires -- SSH access to target host (direct or via amdpve for CTs) +- SSH access to target host (direct, or via the guest's Proxmox node for CTs) - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Python 3 with `httpx` installed on target @@ -56,7 +56,7 @@ connectivity recovery including end-to-end DM validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* | -| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | +| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Field | Value | Trust | @@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation. ### Step 1: Resolve Target Map `target` to host, CT ID, hermes_home, and user from the live-state table. -For CT112 and CT111, route through `ssh root@amdpve` then `pct exec `. +For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec `. ### Step 2: Pull Latest Plugin Source diff --git a/hermes-zulip-restore.prose.md b/hermes-zulip-restore.prose.md index bafb5a3..fb6169d 100644 --- a/hermes-zulip-restore.prose.md +++ b/hermes-zulip-restore.prose.md @@ -43,7 +43,7 @@ gateway restart, and connection validation. ## Requires -- SSH access to target host (direct or via amdpve for CTs) +- SSH access to target host (direct, or via the guest's Proxmox node for CTs) - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Python 3 with `httpx` installed on target - Zulip server accessible at `https://chat.sysloggh.net` @@ -52,7 +52,7 @@ gateway restart, and connection validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| -| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | +| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Field | Value | Trust | @@ -67,7 +67,7 @@ gateway restart, and connection validation. ### Step 1: Locate Target Map `target` to connectivity parameters from the live-state table above. -For CT112 and CT111, route through `ssh root@amdpve` then `pct exec `. +For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec `. ### Step 2: Deploy Zulip Adapter -- 2.54.0 From 672bf8a912c3c201b088459e9560d04539ab06df Mon Sep 17 00:00:00 2001 From: abiba Date: Sat, 12 Sep 2026 19:11:36 +0000 Subject: [PATCH 9/9] docs(disk-gc): attach real fleet-run verification for the CT 111 report-only gate Intent requires a production run log, not only the unit-level dry run. This is a real scan of the live 25-entry fleet through scripts/disk-gc-plan.py: CT 111 (tdunna) at 84% AMBER is alerted as report-only and no gc-executor row is emitted for it; the owned hosts acerpve .9 (77%) and amdpve .15 (76%) still receive gc-executor. No GC command was run against 192.168.68.129. --- ...t111-report-only-verification-20260912.log | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 runs/disk-gc-ct111-report-only-verification-20260912.log diff --git a/runs/disk-gc-ct111-report-only-verification-20260912.log b/runs/disk-gc-ct111-report-only-verification-20260912.log new file mode 100644 index 0000000..641d21d --- /dev/null +++ b/runs/disk-gc-ct111-report-only-verification-20260912.log @@ -0,0 +1,44 @@ +disk-gc report-only verification — CT 111 / tdunna / 192.168.68.129 +date: 2026-09-12T19:11:36Z +host: abiba (this scanner runs INSIDE CT 100 / abiba) +branch head: 6e612ce37b1b9a9688b04e7f816848e7a4185fca +command: scripts/disk-gc-plan.py --scan + +PURPOSE: prove that on a REAL fleet scan, CT 111 is alerted and NO gc-executor action +is emitted for it at any level. No GC command was executed against .129. + +--- live fleet scan (df -P / via scripts/pct-run.sh for LXC, direct SSH for hosts) --- + tdunna 84% + acerpve 192.168.68.9 77% + amdpve 192.168.68.15 76% + ocu-llm 192.168.68.110 69% + storepve 192.168.68.6 65% + kagentz 61% + tanko 56% + minipve 192.168.68.12 49% + authentik 45% + infisical-vault 39% + adguard 38% + ocupve 192.168.68.5 38% + scottdenya 35% + syslog-api 34% + llm-gpu 192.168.68.8 25% + abiba 23% + baggy 22% + jdownloader 21% + gitea 16% + ra-h-os 13% + zulip 12% + docker-vm 192.168.68.7 11% + adguard2 10% + media 9% + proxmox-backup-server 4% + +--- planner output (action plan) --- + 111 AMBER 84.0% -> REPORT-ONLY (no GC) — Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10 + acerpve AMBER 77.0% -> gc-executor + amdpve AMBER 76.0% -> gc-executor + +--- verdict --- + CT 111 (tdunna) 84% AMBER -> report-only; no gc-executor row emitted; no GC run on .129. + Owned hosts acerpve .9 (77%) and amdpve .15 (76%) -> gc-executor (ours). -- 2.54.0