diff --git a/docs/probe-drift-round2-evidence.md b/docs/probe-drift-round2-evidence.md index 169261a..db44d57 100644 --- a/docs/probe-drift-round2-evidence.md +++ b/docs/probe-drift-round2-evidence.md @@ -72,8 +72,8 @@ Root causes (all stale expectations; no live fault): |---|---| | `ct-unreachable:koby:192.168.68.15` | CT 111 (tdunna/koby) runs on **storepve (.6)**, not amdpve (.15). | | `wrapper-*:abiba` | Abiba is pi-only since the harness purge. `/root/.local/bin/hermes` is a dangling symlink; no `hermes-real`, no `~/.hermes/.env`. | -| `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. | -| `wrapper-infisical-path:koonimo` | Koonimo's wrapper injects `KOONIMO_LITELLM_API_KEY` from `~/.hermes/.env` and never invokes infisical. The check required `/usr/bin/infisical`, which is only one valid mechanism. | +| `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. Koby's wrapper is also the genuine no-infisical case: it sources `~/.hermes/.env` rather than `/usr/bin/infisical`, which the check now accepts. | +| `wrapper-infisical-path:koonimo` | Koonimo's wrapper **does** reference `/usr/bin/infisical` — but past the old check's `head -20` window, so the check looked for the path in the wrong slice and false-failed. The fix that mattered was reading the full wrapper body (and then verifying any absolute infisical path it finds actually exists). | **After** — same absolute path, `python3 scripts/agent-health-check.py --no-deploy` (v4): diff --git a/infrastructure-monitoring.prose.md b/infrastructure-monitoring.prose.md index 0860c01..2969ac6 100644 --- a/infrastructure-monitoring.prose.md +++ b/infrastructure-monitoring.prose.md @@ -103,14 +103,22 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo) ## Execution -### Liveness rule (any-HTTP-response) +### Liveness rule (scoped) -A probe is **ALIVE** if the endpoint returns **ANY** HTTP status — including -`401`/`403` auth challenges and `3xx` redirects. A bare `200` is not required and -must never be a pass condition for an auth-gated endpoint. **DOWN = connection -refused (`000`) or timeout only.** Same rule as zulip-health (Tanko) and -gpu-monitor. The PVE API (`:8006/api2/json`) legitimately answers `401` to an -unauthenticated probe — that is the healthy signal, not a failure. +The any-HTTP-response rule applies ONLY to unauthenticated/auth-gated endpoints, +where any HTTP answer proves a listener is up: the PVE API +(`https://:8006/api2/json/version`) and LiteLLM health +(`/litellm/health`, `301` → `/litellm/health/liveliness`). For those endpoints a +probe is **ALIVE** on **ANY** HTTP status — `401`/`403` auth challenges and `3xx` +redirects included — and **DOWN = connection refused (`000`) or timeout only**. +The PVE API legitimately answers `401` to an unauthenticated probe — that is the +healthy signal, not a failure. Same scoped rule as zulip-health (Tanko) and +gpu-monitor. + +Probes whose success condition is specifically a bare `200` are NOT covered by +the any-HTTP rule. On those — the authenticated Zulip POST and the router +`/health` — an unexpected status (`401`/`403` from a bad or missing credential, +`5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive". ### check-health @@ -172,10 +180,13 @@ curl -s http://192.168.68.116:4001/metrics | head -20 **Report format**: Begin every report with the **absolute path the probe executed from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is distinguishable from a real fault at read time. Summarize actual results from -each probe. Apply the any-HTTP-response liveness rule above: only -connection-refused (`000`) or timeout is DOWN; empty output is a warning. A probe -that requires a bare `200` on an auth-gated endpoint (PVE API → `401`, LiteLLM -health → `301` redirect) is a stale expectation, not a fault. +each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE +API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is +DOWN; empty output is a warning. For probes whose expected result is a bare `200` +(the authenticated Zulip POST, router `/health`), flag an alert on any unexpected +status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200` +expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect) +is a stale expectation, not a fault. ### Phase 1: GPU Exporters diff --git a/scripts/agent-health-check.py b/scripts/agent-health-check.py index 50600f9..5003c32 100755 --- a/scripts/agent-health-check.py +++ b/scripts/agent-health-check.py @@ -32,14 +32,21 @@ Changelog: 2026-08-17 ruling: every koby leg is detected and reported, never counted as a fleet failure and never repaired. koby's PVE mapping corrected to storepve (CT 111 tdunna lives on .6 — the old amdpve mapping produced a false - ct-unreachable). The wrapper infisical-path check no longer FAILs .env-based - wrappers that legitimately never invoke infisical (koonimo/baggy). Every run - now prints absolute execution provenance (script + cwd) in the header and in - --json output so a stale-consumer report is distinguishable from a fault at - read time. + ct-unreachable). The wrapper infisical-path check had two stale-expectation + bugs: it read only the first 20 lines of the wrapper, so koonimo (whose + wrapper does reference /usr/bin/infisical, just past line 20) was falsely + FAILed as "path may be wrong"; and it treated the absence of any infisical + reference as a fault, though koby's wrapper sources the key from + ~/.hermes/.env and never invokes infisical. The check now reads the full + wrapper body, accepts a no-infisical wrapper, and verifies that any absolute + infisical path the wrapper references actually exists. Report-only findings + are surfaced in a machine-readable `report_only` array in --json output, + separate from `failures`. Every run prints absolute execution provenance + (script + cwd) in the header, in the cron ALERT line, and in --json output so + a stale-consumer report is distinguishable from a fault at read time. """ -import subprocess, json, sys, os, time +import subprocess, json, sys, os, time, re from datetime import datetime LITELLM = "http://192.168.68.116:80" @@ -88,6 +95,7 @@ GPU_HOSTS = { } FAIL = [] +REPORT_ONLY = [] def _fail(key, agent_name=None): @@ -95,11 +103,13 @@ def _fail(key, agent_name=None): Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs are detected and reported, never repaired and never counted as fleet - failures. A red fleet alert on a known report-only leg is a false alarm. Any - non-report-only agent (or a leg with no agent, e.g. GPU hosts) records - normally. + failures. A red fleet alert on a known report-only leg is a false alarm. + Report-only findings are tracked separately so --json consumers can still + see them without them counting as fleet failures. Any non-report-only agent + (or a leg with no agent, e.g. GPU hosts) records normally. """ if agent_name and AGENTS.get(agent_name, {}).get("report_only"): + REPORT_ONLY.append(key) print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired") return FAIL.append(key) @@ -509,23 +519,44 @@ def check_wrapper_integrity(): print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)") # Credential-injection mechanism. The Hermes-era wrapper injected creds - # with `/usr/bin/infisical run`; newer wrappers source the agent key from - # ~/.hermes/.env and never mention infisical (koonimo/baggy does exactly - # this). A wrapper with NO infisical reference is therefore a valid - # alternate mechanism, not a fault — only flag a wrapper that DOES call - # infisical when the binary cannot resolve. The old check FAILed every - # .env-based wrapper as "infisical path may be wrong": a stale - # expectation, not a fault. + # with `/usr/bin/infisical run`, but the mechanism is not required to be + # infisical at all: koby's wrapper sources the key from ~/.hermes/.env + # and never mentions infisical, which is valid. The old check read only + # the first 20 lines, so koonimo's wrapper — which DOES reference + # /usr/bin/infisical, just past line 20 — false-failed as "path may be + # wrong". Read the full body, accept a no-infisical wrapper, and verify + # that any absolute infisical path the wrapper hardcodes actually exists + # (PATH resolution alone is not enough — a dangling /usr/bin/infisical is + # a broken wrapper even when a different infisical is on PATH). wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or "" if "infisical" in wrapper_body: - inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user) - if not inf_actual: - print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING") - _fail(f"wrapper-no-infisical:{name}", name) - elif "/usr/bin/infisical" not in wrapper_body: - print(f" ⚠️ {name}: wrapper infisical path differs (infisical at {inf_actual}) — informational") + inf_paths = [] + for _m in re.finditer(r"(/[A-Za-z0-9._/-]*infisical)", wrapper_body): + if _m.group(1) not in inf_paths: + inf_paths.append(_m.group(1)) + dangling = [] + for _p in inf_paths: + _exists = ssh(host, f"test -x {_p} && echo OK || echo MISS", user=user) + if not _exists or _exists.strip().splitlines()[-1] != "OK": + dangling.append(_p) + if inf_paths: + if dangling: + inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user) + suffix = f" (infisical at {inf_actual})" if inf_actual else "" + print(f" ❌ {name}: wrapper hardcodes missing infisical path(s) " + f"{', '.join(dangling)}{suffix}") + _fail(f"wrapper-infisical-path:{name}", name) + elif "/usr/bin/infisical" not in wrapper_body: + print(f" ⚠️ {name}: wrapper infisical path differs — informational") + else: + print(f" ✅ {name}: wrapper infisical path OK") else: - print(f" ✅ {name}: wrapper infisical path OK") + inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user) + if not inf_actual: + print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING") + _fail(f"wrapper-no-infisical:{name}", name) + else: + print(f" ✅ {name}: wrapper infisical resolves via PATH ({inf_actual})") else: print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK") @@ -614,14 +645,16 @@ def main(): # Provenance: a report is only actionable if the reader can tell WHICH copy # of this script produced it. Emit the absolute execution path (script + cwd) - # in both human and --json output so a stale-consumer report is - # distinguishable from a real fault at read time. + # in human, cron-alert, and --json output so a stale-consumer report is + # distinguishable from a real fault at read time. The production cron path + # runs --quiet, so provenance must NOT be behind the quiet guard. script_path = os.path.abspath(__file__) cwd = os.getcwd() if not quiet: print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}") - print(f"📍 executed from: script={script_path} cwd={cwd}") + print(f"📍 executed from: script={script_path} cwd={cwd}") + if not quiet: print() load_agent_keys() @@ -656,14 +689,15 @@ def main(): if FAIL: print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}") if quiet: - print(f"ALERT agent-health:{','.join(FAIL)}") + print(f"ALERT agent-health:{','.join(FAIL)} script={script_path} cwd={cwd}") elif not quiet: print("\n✅ All checks passed") if as_json: print(json.dumps({"timestamp": datetime.now().isoformat(), "execution_path": script_path, "cwd": cwd, - "failures": FAIL, "healthy": len(FAIL) == 0})) + "failures": FAIL, "report_only": REPORT_ONLY, + "healthy": len(FAIL) == 0})) sys.exit(1 if FAIL else 0) diff --git a/scripts/prose-lint.sh b/scripts/prose-lint.sh index f9ef02f..1756856 100755 --- a/scripts/prose-lint.sh +++ b/scripts/prose-lint.sh @@ -87,11 +87,21 @@ echo " ✅ IP consistency verified (.19=.122=.123 all reachable)" # Report provenance — every contract report must state the absolute path it # executed from, so a stale-consumer report is distinguishable from a real fault # at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds). -PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true) -if [ -n "$PROV_FILES" ]; then +# Enforced only inside the **Report format** paragraph, and a check-health +# contract with no Report format paragraph FAILs rather than being skipped. +PROV_FILES=$(grep -rlE '^### check-health|\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true) +if [ -z "$PROV_FILES" ]; then + echo " ❌ No check-health/report-format contracts found — provenance not enforced" + FAILED=1 +else PROV_BAD=0 while IFS= read -r f; do - if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then + [ -n "$f" ] || continue + REPORT_PARA=$(awk '/\*\*Report format\*\*/{found=1} found{print} found && /^[[:space:]]*$/{exit}' "$f") + if [ -z "$REPORT_PARA" ]; then + echo " ❌ $f: check-health contract has no **Report format** paragraph" + PROV_BAD=1 + elif ! printf '%s\n' "$REPORT_PARA" | grep -qE 'absolute path|pwd -P|executed from'; then echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)" PROV_BAD=1 fi diff --git a/tests/test_probe_drift.py b/tests/test_probe_drift.py index 316231c..37591fc 100644 --- a/tests/test_probe_drift.py +++ b/tests/test_probe_drift.py @@ -7,30 +7,46 @@ stale expectations rather than live faults. abiba (pi-only since the harness purge) was tested as a Hermes host, koby (report-only per the captain's 2026-08-17 ruling) was counted as repairable, koby's CT 111 was probed on amdpve where it does not exist (it runs on - storepve .6), and a .env-based hermes wrapper was FAILed for not mentioning - infisical. + storepve .6), and the wrapper infisical check had two bugs — it read only + the first 20 lines, so koonimo's wrapper (which references /usr/bin/infisical + past line 20) false-failed, and it treated koby's genuine no-infisical + (~/.hermes/.env) wrapper as broken. * gpu-monitor emitted "DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000" three times from probing bare port 80 on GPU hosts while :8080 answered 200. * infrastructure-monitoring probed CT 116 for the PVE API (no pveproxy -> 000) instead of the five real cluster nodes, which answer 401 = alive. -These tests pin the corrections so the false alarms cannot return silently. -They are static/structural over the contracts plus an inert import of the health -script — no live network, vault, or SSH access is required. +These tests execute the health script (with SSH/vault stubbed) and the real +provenance consumer (scripts/prose-lint.sh), and parse the contracts' executable +check-health probe blocks into normalized probe sets. No live network, vault, or +SSH access is required. """ from __future__ import annotations import importlib.util +import json +import os import pathlib import re +import subprocess +import sys +import textwrap import pytest ROOT = pathlib.Path(__file__).resolve().parents[1] AHC = ROOT / "scripts" / "agent-health-check.py" +LINT = ROOT / "scripts" / "prose-lint.sh" GPU = ROOT / "gpu-monitor.prose.md" INFRA = ROOT / "infrastructure-monitoring.prose.md" -PROXMOX = ROOT / "proxmox-monitor.prose.md" + +PVE_NODE_IPS = { + "192.168.68.9", + "192.168.68.5", + "192.168.68.15", + "192.168.68.6", + "192.168.68.12", +} @pytest.fixture(scope="module") @@ -43,6 +59,26 @@ def ahc(): return module +# ── helpers: execute the health script with SSH/vault stubbed ───────── + +def _run_main(ahc, monkeypatch, capsys, argv, ssh_result=None): + ahc.FAIL.clear() + ahc.REPORT_ONLY.clear() + monkeypatch.setattr(ahc, "load_agent_keys", lambda: None) + monkeypatch.setattr(ahc, "ssh", lambda *a, **k: ssh_result) + monkeypatch.setattr(sys, "argv", ["agent-health-check.py", "--no-deploy", *argv]) + with pytest.raises(SystemExit) as exc: + ahc.main() + return exc.value.code, capsys.readouterr().out + + +def _json_payload(out): + for line in reversed(out.splitlines()): + if line.startswith('{"timestamp"'): + return json.loads(line) + raise AssertionError(f"no JSON payload in output:\n{out}") + + # ── agent-health-check: stale-expectation legs ─────────────────────── def test_import_does_not_contact_vault(ahc): @@ -68,17 +104,19 @@ def test_koby_ct111_is_on_storepve(ahc): assert ahc.AGENTS["koby"]["pve"] == "storepve" -@pytest.mark.parametrize("agent", ["koby", "koonimo", "tanko"]) -def test_report_only_legs_never_count_as_failures(ahc, agent): - ahc.FAIL.clear() - try: +def test_report_only_legs_never_count_as_failures(ahc): + for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)): + ahc.FAIL.clear() + ahc.REPORT_ONLY.clear() ahc._fail(f"probe:{agent}", agent) - if ahc.AGENTS[agent].get("report_only"): + if report_only: assert ahc.FAIL == [] + assert ahc.REPORT_ONLY == [f"probe:{agent}"] else: assert ahc.FAIL == [f"probe:{agent}"] - finally: - ahc.FAIL.clear() + assert ahc.REPORT_ONLY == [] + ahc.FAIL.clear() + ahc.REPORT_ONLY.clear() def test_failure_recording_accepts_agentless_keys(ahc): @@ -90,63 +128,237 @@ def test_failure_recording_accepts_agentless_keys(ahc): ahc.FAIL.clear() -def test_script_reports_absolute_execution_provenance(ahc): - src = AHC.read_text() - assert "execution_path" in src - assert "os.getcwd()" in src +def test_json_reports_absolute_execution_provenance(ahc, monkeypatch, capsys): + code, out = _run_main(ahc, monkeypatch, capsys, ["--json"]) + payload = _json_payload(out) + assert payload["execution_path"] == os.path.abspath(str(AHC)) + assert payload["cwd"] == os.getcwd() + assert code == 1 # stubbed SSH fails every leg, but provenance is still emitted -def test_wrapper_check_has_no_bare_infisical_expectation(ahc): - # A wrapper that never mentions infisical (koonimo sources ~/.hermes/.env) - # must not be FAILed merely for lacking an infisical reference. - assert "wrapper resolves creds without infisical" in AHC.read_text() +def test_quiet_run_still_carries_provenance_on_the_alert_path(ahc, monkeypatch, capsys): + # The production cron runs --quiet; a report without provenance is + # unactionable (item 4). Both the header line and the ALERT line must carry it. + code, out = _run_main(ahc, monkeypatch, capsys, ["--quiet"]) + assert code == 1 + assert "📍 executed from: script=" in out + alerts = [ln for ln in out.splitlines() if ln.startswith("ALERT agent-health:")] + assert alerts, out + assert f"script={os.path.abspath(str(AHC))}" in alerts[0] + assert f"cwd={os.getcwd()}" in alerts[0] -# ── item 4: every contract report carries its execution path ────────── +def test_json_surfaces_report_only_findings_separately(ahc, monkeypatch, capsys): + # Koby's down legs are reported but must not count as fleet failures; the + # --json payload exposes them in their own array (item 1 + f8). + _, out = _run_main(ahc, monkeypatch, capsys, ["--json"]) + payload = _json_payload(out) + assert isinstance(payload["report_only"], list) + assert any(key.startswith(("gateway-down:koby", "ct-unreachable:koby")) + for key in payload["report_only"]) + assert not any("koby" in key for key in payload["failures"]) -@pytest.mark.parametrize("contract", [GPU, INFRA, PROXMOX], ids=lambda p: p.name) -def test_report_format_requires_execution_provenance(contract): + +# ── agent-health-check: wrapper infisical behavior (f3) ─────────────── + +def _stub_wrapper_ssh(ahc, monkeypatch, wrapper_body, test_x_result="OK", command_v="/usr/local/bin/infisical"): + def fake_ssh(host, cmd, user="root"): + if cmd.startswith("cat /root/.local/bin/hermes"): + return wrapper_body + if cmd.startswith("ls -la /root/.local/bin/hermes "): + return "-rwxr-xr-x 1 root root 0 Jan 1 00:00 /root/.local/bin/hermes" + if cmd.startswith("ls -la /root/.local/bin/hermes-real") or "venv/bin/hermes" in cmd: + return "-rwxr-xr-x 1 root root 0 Jan 1 00:00 /root/.local/bin/hermes-real" + if cmd.startswith("grep -c 'LITELLM_API_KEY'"): + return "1" + if cmd.startswith("test -x "): + return test_x_result + if cmd.startswith("command -v infisical"): + return command_v + return None + + monkeypatch.setattr(ahc, "ssh", fake_ssh) + monkeypatch.setattr(ahc, "AGENTS", {"koonimo": dict(ahc.AGENTS["koonimo"])}) + ahc.FAIL.clear() + ahc.REPORT_ONLY.clear() + + +def test_env_based_wrapper_without_infisical_is_not_failed(ahc, monkeypatch, capsys): + _stub_wrapper_ssh(ahc, monkeypatch, + "#!/bin/bash\nsource ~/.hermes/.env\nexec hermes-real \"$@\"\n") + ahc.check_wrapper_integrity() + out = capsys.readouterr().out + assert ahc.FAIL == [] + assert "wrapper resolves creds without infisical" in out + + +def test_dangling_absolute_infisical_path_is_failed(ahc, monkeypatch, capsys): + # Wrapper hardcodes /usr/bin/infisical, which is absent, while PATH resolves + # infisical to /usr/local/bin/infisical. The literal path must be verified, + # not inferred from PATH resolution. + _stub_wrapper_ssh(ahc, monkeypatch, + "#!/bin/bash\n/usr/bin/infisical run -- hermes-real \"$@\"\n", + test_x_result="MISS", command_v="/usr/local/bin/infisical") + ahc.check_wrapper_integrity() + assert "wrapper-infisical-path:koonimo" in ahc.FAIL + + +def test_existing_absolute_infisical_path_passes(ahc, monkeypatch, capsys): + _stub_wrapper_ssh(ahc, monkeypatch, + "#!/bin/bash\n/usr/bin/infisical run -- hermes-real \"$@\"\n", + test_x_result="OK") + ahc.check_wrapper_integrity() + out = capsys.readouterr().out + assert ahc.FAIL == [] + assert "wrapper infisical path OK" in out + + +# ── item 4: prose-lint enforces report provenance (real consumer) ───── + +GOOD_CONTRACT = textwrap.dedent("""\ + --- + kind: function + name: good + description: fixture with provenance + --- + + ## Parameters + + - x: y + + ## Returns + + ok + + ### check-health + + ```bash + pwd -P + ``` + + **Report format**: Begin with the absolute path the probe executed from. + """) + +DECOY_CONTRACT = textwrap.dedent("""\ + --- + kind: function + name: decoy + description: fixture with provenance only outside the report format + --- + + ## Parameters + + - x: y + + ## Returns + + ok + + The absolute path of the config is /etc/foo. + + ### check-health + + ```bash + true + ``` + + **Report format**: Summarize actual results from each probe. + """) + +MISSING_CONTRACT = textwrap.dedent("""\ + --- + kind: function + name: missing + description: check-health contract with no report format + --- + + ## Parameters + + - x: y + + ## Returns + + ok + + ### check-health + + ```bash + pwd -P + ``` + """) + + +def _run_lint(tmp_path, text, name): + (tmp_path / name).write_text(text) + return subprocess.run(["bash", str(LINT)], cwd=tmp_path, + capture_output=True, text=True) + + +def test_prose_lint_accepts_report_format_with_provenance(tmp_path): + result = _run_lint(tmp_path, GOOD_CONTRACT, "good.prose.md") + assert result.returncode == 0, result.stdout + result.stderr + + +def test_prose_lint_rejects_report_format_without_provenance(tmp_path): + result = _run_lint(tmp_path, DECOY_CONTRACT, "decoy.prose.md") + assert result.returncode == 1, result.stdout + assert "lacks execution provenance" in result.stdout + + +def test_prose_lint_requires_report_format_on_check_health_contract(tmp_path): + result = _run_lint(tmp_path, MISSING_CONTRACT, "missing.prose.md") + assert result.returncode == 1, result.stdout + assert "no **Report format** paragraph" in result.stdout + + +# ── contracts: parse the executable check-health probe block ───────── + +def _check_health_block(contract): + """Extract the bash probe block under ### check-health (the probe interface).""" text = contract.read_text() - assert "**Report format**" in text - assert re.search(r"absolute path|pwd -P|executed from", text) + marker = "### check-health" + assert marker in text, f"{contract.name} has no {marker}" + after = text.split(marker, 1)[1] + match = re.search(r"```bash\n(.*?)```", after, re.S) + assert match, f"{contract.name} check-health has no bash probe block" + return match.group(1) -# ── item 3: gpu-monitor probes the real GPU endpoints ──────────────── - -def test_gpu_monitor_probes_gpu_health_on_8080(): - text = GPU.read_text() - assert "http://192.168.68.8:8080/health" in text - assert "http://192.168.68.110:8080/health" in text +def _urls(block): + # Comments document the forbidden/deprecated probes; only count real commands. + code = "\n".join(ln for ln in block.splitlines() + if not ln.lstrip().startswith("#")) + return set(re.findall(r"https?://[^\s\"')]+", code)) -def test_gpu_monitor_forbids_bare_port_80_on_gpu_hosts(): - text = GPU.read_text() - assert re.search(r"never .{0,20}bare port 80", text, re.I) - # The false-DEGRADED symptom must be documented, not just implied. - assert "DEGRADED" in text +def test_gpu_monitor_probes_every_gpu_health_on_8080(): + block = _check_health_block(GPU) + gpu_health = {u for u in _urls(block) if ":8080/health" in u} + assert {"http://192.168.68.8:8080/health", + "http://192.168.68.110:8080/health"} <= gpu_health + + +def test_gpu_monitor_never_probes_bare_port_80_on_gpu_hosts(): + block = _check_health_block(GPU) + bare = {u for u in _urls(block) + if re.match(r"https?://192\.168\.68\.(8|110|15)/", u)} + assert bare == set() + assert re.search(r"never .{0,40}bare port 80", block, re.I) def test_gpu_monitor_documents_router_301_as_alive(): - text = GPU.read_text() - assert "/gpu/gpu-data" in text - assert "301" in text + block = _check_health_block(GPU) + assert "/health/unified" in block + assert "301" in block -# ── item 2: infrastructure-monitoring PVE API vantage ──────────────── +def test_infra_monitoring_probes_every_real_pve_node(): + block = _check_health_block(INFRA) + match = re.search(r"for node in ([^\n;]+)", block) + assert match, "PVE liveness loop not found in check-health" + assert set(match.group(1).split()) == PVE_NODE_IPS + assert "https://$node:8006/api2/json/version" in block + def test_infra_monitoring_does_not_probe_ct116_for_pve_api(): - assert "https://192.168.68.116:8006" not in INFRA.read_text() - - -@pytest.mark.parametrize( - "node", - ["192.168.68.9", "192.168.68.5", "192.168.68.15", "192.168.68.6", "192.168.68.12"], -) -def test_infra_monitoring_probes_every_real_pve_node(node): - assert node in INFRA.read_text() - - -def test_infra_monitoring_uses_any_http_liveness_rule(): - text = INFRA.read_text() - assert "api2/json/version" in text - assert "ANY HTTP status" in text or "any-HTTP" in text + assert "192.168.68.116:8006" not in _check_health_block(INFRA)