no-mistakes(review): Harden health-check provenance, infisical verification, report-only JSON, tests

This commit is contained in:
2026-09-10 01:35:44 +00:00
parent c66d9c1e20
commit 194e256ac5
5 changed files with 367 additions and 100 deletions
+2 -2
View File
@@ -72,8 +72,8 @@ Root causes (all stale expectations; no live fault):
|---|---| |---|---|
| `ct-unreachable:koby:192.168.68.15` | CT 111 (tdunna/koby) runs on **storepve (.6)**, not amdpve (.15). | | `ct-unreachable:koby:192.168.68.15` | CT 111 (tdunna/koby) runs on **storepve (.6)**, not amdpve (.15). |
| `wrapper-*:abiba` | Abiba is pi-only since the harness purge. `/root/.local/bin/hermes` is a dangling symlink; no `hermes-real`, no `~/.hermes/.env`. | | `wrapper-*:abiba` | Abiba is pi-only since the harness purge. `/root/.local/bin/hermes` is a dangling symlink; no `hermes-real`, no `~/.hermes/.env`. |
| `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. | | `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. Koby's wrapper is also the genuine no-infisical case: it sources `~/.hermes/.env` rather than `/usr/bin/infisical`, which the check now accepts. |
| `wrapper-infisical-path:koonimo` | Koonimo's wrapper injects `KOONIMO_LITELLM_API_KEY` from `~/.hermes/.env` and never invokes infisical. The check required `/usr/bin/infisical`, which is only one valid mechanism. | | `wrapper-infisical-path:koonimo` | Koonimo's wrapper **does** reference `/usr/bin/infisical` — but past the old check's `head -20` window, so the check looked for the path in the wrong slice and false-failed. The fix that mattered was reading the full wrapper body (and then verifying any absolute infisical path it finds actually exists). |
**After** — same absolute path, `python3 scripts/agent-health-check.py --no-deploy` (v4): **After** — same absolute path, `python3 scripts/agent-health-check.py --no-deploy` (v4):
+22 -11
View File
@@ -103,14 +103,22 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
## Execution ## Execution
### Liveness rule (any-HTTP-response) ### Liveness rule (scoped)
A probe is **ALIVE** if the endpoint returns **ANY** HTTP status — including The any-HTTP-response rule applies ONLY to unauthenticated/auth-gated endpoints,
`401`/`403` auth challenges and `3xx` redirects. A bare `200` is not required and where any HTTP answer proves a listener is up: the PVE API
must never be a pass condition for an auth-gated endpoint. **DOWN = connection (`https://<node>:8006/api2/json/version`) and LiteLLM health
refused (`000`) or timeout only.** Same rule as zulip-health (Tanko) and (`/litellm/health`, `301` → `/litellm/health/liveliness`). For those endpoints a
gpu-monitor. The PVE API (`:8006/api2/json`) legitimately answers `401` to an probe is **ALIVE** on **ANY** HTTP status — `401`/`403` auth challenges and `3xx`
unauthenticated probe — that is the healthy signal, not a failure. redirects included — and **DOWN = connection refused (`000`) or timeout only**.
The PVE API legitimately answers `401` to an unauthenticated probe — that is the
healthy signal, not a failure. Same scoped rule as zulip-health (Tanko) and
gpu-monitor.
Probes whose success condition is specifically a bare `200` are NOT covered by
the any-HTTP rule. On those — the authenticated Zulip POST and the router
`/health` — an unexpected status (`401`/`403` from a bad or missing credential,
`5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive".
### check-health ### check-health
@@ -172,10 +180,13 @@ curl -s http://192.168.68.116:4001/metrics | head -20
**Report format**: Begin every report with the **absolute path the probe executed **Report format**: Begin every report with the **absolute path the probe executed
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
distinguishable from a real fault at read time. Summarize actual results from distinguishable from a real fault at read time. Summarize actual results from
each probe. Apply the any-HTTP-response liveness rule above: only each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE
connection-refused (`000`) or timeout is DOWN; empty output is a warning. A probe API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is
that requires a bare `200` on an auth-gated endpoint (PVE API → `401`, LiteLLM DOWN; empty output is a warning. For probes whose expected result is a bare `200`
health → `301` redirect) is a stale expectation, not a fault. (the authenticated Zulip POST, router `/health`), flag an alert on any unexpected
status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200`
expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect)
is a stale expectation, not a fault.
### Phase 1: GPU Exporters ### Phase 1: GPU Exporters
+62 -28
View File
@@ -32,14 +32,21 @@ Changelog:
2026-08-17 ruling: every koby leg is detected and reported, never counted as a 2026-08-17 ruling: every koby leg is detected and reported, never counted as a
fleet failure and never repaired. koby's PVE mapping corrected to storepve fleet failure and never repaired. koby's PVE mapping corrected to storepve
(CT 111 tdunna lives on .6 — the old amdpve mapping produced a false (CT 111 tdunna lives on .6 — the old amdpve mapping produced a false
ct-unreachable). The wrapper infisical-path check no longer FAILs .env-based ct-unreachable). The wrapper infisical-path check had two stale-expectation
wrappers that legitimately never invoke infisical (koonimo/baggy). Every run bugs: it read only the first 20 lines of the wrapper, so koonimo (whose
now prints absolute execution provenance (script + cwd) in the header and in wrapper does reference /usr/bin/infisical, just past line 20) was falsely
--json output so a stale-consumer report is distinguishable from a fault at FAILed as "path may be wrong"; and it treated the absence of any infisical
read time. reference as a fault, though koby's wrapper sources the key from
~/.hermes/.env and never invokes infisical. The check now reads the full
wrapper body, accepts a no-infisical wrapper, and verifies that any absolute
infisical path the wrapper references actually exists. Report-only findings
are surfaced in a machine-readable `report_only` array in --json output,
separate from `failures`. Every run prints absolute execution provenance
(script + cwd) in the header, in the cron ALERT line, and in --json output so
a stale-consumer report is distinguishable from a fault at read time.
""" """
import subprocess, json, sys, os, time import subprocess, json, sys, os, time, re
from datetime import datetime from datetime import datetime
LITELLM = "http://192.168.68.116:80" LITELLM = "http://192.168.68.116:80"
@@ -88,6 +95,7 @@ GPU_HOSTS = {
} }
FAIL = [] FAIL = []
REPORT_ONLY = []
def _fail(key, agent_name=None): def _fail(key, agent_name=None):
@@ -95,11 +103,13 @@ def _fail(key, agent_name=None):
Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs
are detected and reported, never repaired and never counted as fleet are detected and reported, never repaired and never counted as fleet
failures. A red fleet alert on a known report-only leg is a false alarm. Any failures. A red fleet alert on a known report-only leg is a false alarm.
non-report-only agent (or a leg with no agent, e.g. GPU hosts) records Report-only findings are tracked separately so --json consumers can still
normally. see them without them counting as fleet failures. Any non-report-only agent
(or a leg with no agent, e.g. GPU hosts) records normally.
""" """
if agent_name and AGENTS.get(agent_name, {}).get("report_only"): if agent_name and AGENTS.get(agent_name, {}).get("report_only"):
REPORT_ONLY.append(key)
print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired") print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired")
return return
FAIL.append(key) FAIL.append(key)
@@ -509,23 +519,44 @@ def check_wrapper_integrity():
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)") print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
# Credential-injection mechanism. The Hermes-era wrapper injected creds # Credential-injection mechanism. The Hermes-era wrapper injected creds
# with `/usr/bin/infisical run`; newer wrappers source the agent key from # with `/usr/bin/infisical run`, but the mechanism is not required to be
# ~/.hermes/.env and never mention infisical (koonimo/baggy does exactly # infisical at all: koby's wrapper sources the key from ~/.hermes/.env
# this). A wrapper with NO infisical reference is therefore a valid # and never mentions infisical, which is valid. The old check read only
# alternate mechanism, not a fault — only flag a wrapper that DOES call # the first 20 lines, so koonimo's wrapper — which DOES reference
# infisical when the binary cannot resolve. The old check FAILed every # /usr/bin/infisical, just past line 20 — false-failed as "path may be
# .env-based wrapper as "infisical path may be wrong": a stale # wrong". Read the full body, accept a no-infisical wrapper, and verify
# expectation, not a fault. # that any absolute infisical path the wrapper hardcodes actually exists
# (PATH resolution alone is not enough — a dangling /usr/bin/infisical is
# a broken wrapper even when a different infisical is on PATH).
wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or "" wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
if "infisical" in wrapper_body: if "infisical" in wrapper_body:
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user) inf_paths = []
if not inf_actual: for _m in re.finditer(r"(/[A-Za-z0-9._/-]*infisical)", wrapper_body):
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING") if _m.group(1) not in inf_paths:
_fail(f"wrapper-no-infisical:{name}", name) inf_paths.append(_m.group(1))
elif "/usr/bin/infisical" not in wrapper_body: dangling = []
print(f" ⚠️ {name}: wrapper infisical path differs (infisical at {inf_actual}) — informational") for _p in inf_paths:
_exists = ssh(host, f"test -x {_p} && echo OK || echo MISS", user=user)
if not _exists or _exists.strip().splitlines()[-1] != "OK":
dangling.append(_p)
if inf_paths:
if dangling:
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
suffix = f" (infisical at {inf_actual})" if inf_actual else ""
print(f" ❌ {name}: wrapper hardcodes missing infisical path(s) "
f"{', '.join(dangling)}{suffix}")
_fail(f"wrapper-infisical-path:{name}", name)
elif "/usr/bin/infisical" not in wrapper_body:
print(f" ⚠️ {name}: wrapper infisical path differs — informational")
else:
print(f" ✅ {name}: wrapper infisical path OK")
else: else:
print(f" ✅ {name}: wrapper infisical path OK") inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
_fail(f"wrapper-no-infisical:{name}", name)
else:
print(f" ✅ {name}: wrapper infisical resolves via PATH ({inf_actual})")
else: else:
print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK") print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
@@ -614,14 +645,16 @@ def main():
# Provenance: a report is only actionable if the reader can tell WHICH copy # Provenance: a report is only actionable if the reader can tell WHICH copy
# of this script produced it. Emit the absolute execution path (script + cwd) # of this script produced it. Emit the absolute execution path (script + cwd)
# in both human and --json output so a stale-consumer report is # in human, cron-alert, and --json output so a stale-consumer report is
# distinguishable from a real fault at read time. # distinguishable from a real fault at read time. The production cron path
# runs --quiet, so provenance must NOT be behind the quiet guard.
script_path = os.path.abspath(__file__) script_path = os.path.abspath(__file__)
cwd = os.getcwd() cwd = os.getcwd()
if not quiet: if not quiet:
print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}") print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print(f"📍 executed from: script={script_path} cwd={cwd}") print(f"📍 executed from: script={script_path} cwd={cwd}")
if not quiet:
print() print()
load_agent_keys() load_agent_keys()
@@ -656,14 +689,15 @@ def main():
if FAIL: if FAIL:
print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}") print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
if quiet: if quiet:
print(f"ALERT agent-health:{','.join(FAIL)}") print(f"ALERT agent-health:{','.join(FAIL)} script={script_path} cwd={cwd}")
elif not quiet: elif not quiet:
print("\n✅ All checks passed") print("\n✅ All checks passed")
if as_json: if as_json:
print(json.dumps({"timestamp": datetime.now().isoformat(), print(json.dumps({"timestamp": datetime.now().isoformat(),
"execution_path": script_path, "cwd": cwd, "execution_path": script_path, "cwd": cwd,
"failures": FAIL, "healthy": len(FAIL) == 0})) "failures": FAIL, "report_only": REPORT_ONLY,
"healthy": len(FAIL) == 0}))
sys.exit(1 if FAIL else 0) sys.exit(1 if FAIL else 0)
+13 -3
View File
@@ -87,11 +87,21 @@ echo " ✅ IP consistency verified (.19=.122=.123 all reachable)"
# Report provenance — every contract report must state the absolute path it # Report provenance — every contract report must state the absolute path it
# executed from, so a stale-consumer report is distinguishable from a real fault # executed from, so a stale-consumer report is distinguishable from a real fault
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds). # at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true) # Enforced only inside the **Report format** paragraph, and a check-health
if [ -n "$PROV_FILES" ]; then # contract with no Report format paragraph FAILs rather than being skipped.
PROV_FILES=$(grep -rlE '^### check-health|\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
if [ -z "$PROV_FILES" ]; then
echo " ❌ No check-health/report-format contracts found — provenance not enforced"
FAILED=1
else
PROV_BAD=0 PROV_BAD=0
while IFS= read -r f; do while IFS= read -r f; do
if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then [ -n "$f" ] || continue
REPORT_PARA=$(awk '/\*\*Report format\*\*/{found=1} found{print} found && /^[[:space:]]*$/{exit}' "$f")
if [ -z "$REPORT_PARA" ]; then
echo " ❌ $f: check-health contract has no **Report format** paragraph"
PROV_BAD=1
elif ! printf '%s\n' "$REPORT_PARA" | grep -qE 'absolute path|pwd -P|executed from'; then
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)" echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
PROV_BAD=1 PROV_BAD=1
fi fi
+268 -56
View File
@@ -7,30 +7,46 @@ stale expectations rather than live faults.
abiba (pi-only since the harness purge) was tested as a Hermes host, koby abiba (pi-only since the harness purge) was tested as a Hermes host, koby
(report-only per the captain's 2026-08-17 ruling) was counted as repairable, (report-only per the captain's 2026-08-17 ruling) was counted as repairable,
koby's CT 111 was probed on amdpve where it does not exist (it runs on koby's CT 111 was probed on amdpve where it does not exist (it runs on
storepve .6), and a .env-based hermes wrapper was FAILed for not mentioning storepve .6), and the wrapper infisical check had two bugs — it read only
infisical. the first 20 lines, so koonimo's wrapper (which references /usr/bin/infisical
past line 20) false-failed, and it treated koby's genuine no-infisical
(~/.hermes/.env) wrapper as broken.
* gpu-monitor emitted "DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000" three * gpu-monitor emitted "DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000" three
times from probing bare port 80 on GPU hosts while :8080 answered 200. times from probing bare port 80 on GPU hosts while :8080 answered 200.
* infrastructure-monitoring probed CT 116 for the PVE API (no pveproxy -> * infrastructure-monitoring probed CT 116 for the PVE API (no pveproxy ->
000) instead of the five real cluster nodes, which answer 401 = alive. 000) instead of the five real cluster nodes, which answer 401 = alive.
These tests pin the corrections so the false alarms cannot return silently. These tests execute the health script (with SSH/vault stubbed) and the real
They are static/structural over the contracts plus an inert import of the health provenance consumer (scripts/prose-lint.sh), and parse the contracts' executable
script — no live network, vault, or SSH access is required. check-health probe blocks into normalized probe sets. No live network, vault, or
SSH access is required.
""" """
from __future__ import annotations from __future__ import annotations
import importlib.util import importlib.util
import json
import os
import pathlib import pathlib
import re import re
import subprocess
import sys
import textwrap
import pytest import pytest
ROOT = pathlib.Path(__file__).resolve().parents[1] ROOT = pathlib.Path(__file__).resolve().parents[1]
AHC = ROOT / "scripts" / "agent-health-check.py" AHC = ROOT / "scripts" / "agent-health-check.py"
LINT = ROOT / "scripts" / "prose-lint.sh"
GPU = ROOT / "gpu-monitor.prose.md" GPU = ROOT / "gpu-monitor.prose.md"
INFRA = ROOT / "infrastructure-monitoring.prose.md" INFRA = ROOT / "infrastructure-monitoring.prose.md"
PROXMOX = ROOT / "proxmox-monitor.prose.md"
PVE_NODE_IPS = {
"192.168.68.9",
"192.168.68.5",
"192.168.68.15",
"192.168.68.6",
"192.168.68.12",
}
@pytest.fixture(scope="module") @pytest.fixture(scope="module")
@@ -43,6 +59,26 @@ def ahc():
return module return module
# ── helpers: execute the health script with SSH/vault stubbed ─────────
def _run_main(ahc, monkeypatch, capsys, argv, ssh_result=None):
ahc.FAIL.clear()
ahc.REPORT_ONLY.clear()
monkeypatch.setattr(ahc, "load_agent_keys", lambda: None)
monkeypatch.setattr(ahc, "ssh", lambda *a, **k: ssh_result)
monkeypatch.setattr(sys, "argv", ["agent-health-check.py", "--no-deploy", *argv])
with pytest.raises(SystemExit) as exc:
ahc.main()
return exc.value.code, capsys.readouterr().out
def _json_payload(out):
for line in reversed(out.splitlines()):
if line.startswith('{"timestamp"'):
return json.loads(line)
raise AssertionError(f"no JSON payload in output:\n{out}")
# ── agent-health-check: stale-expectation legs ─────────────────────── # ── agent-health-check: stale-expectation legs ───────────────────────
def test_import_does_not_contact_vault(ahc): def test_import_does_not_contact_vault(ahc):
@@ -68,17 +104,19 @@ def test_koby_ct111_is_on_storepve(ahc):
assert ahc.AGENTS["koby"]["pve"] == "storepve" assert ahc.AGENTS["koby"]["pve"] == "storepve"
@pytest.mark.parametrize("agent", ["koby", "koonimo", "tanko"]) def test_report_only_legs_never_count_as_failures(ahc):
def test_report_only_legs_never_count_as_failures(ahc, agent): for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
ahc.FAIL.clear() ahc.FAIL.clear()
try: ahc.REPORT_ONLY.clear()
ahc._fail(f"probe:{agent}", agent) ahc._fail(f"probe:{agent}", agent)
if ahc.AGENTS[agent].get("report_only"): if report_only:
assert ahc.FAIL == [] assert ahc.FAIL == []
assert ahc.REPORT_ONLY == [f"probe:{agent}"]
else: else:
assert ahc.FAIL == [f"probe:{agent}"] assert ahc.FAIL == [f"probe:{agent}"]
finally: assert ahc.REPORT_ONLY == []
ahc.FAIL.clear() ahc.FAIL.clear()
ahc.REPORT_ONLY.clear()
def test_failure_recording_accepts_agentless_keys(ahc): def test_failure_recording_accepts_agentless_keys(ahc):
@@ -90,63 +128,237 @@ def test_failure_recording_accepts_agentless_keys(ahc):
ahc.FAIL.clear() ahc.FAIL.clear()
def test_script_reports_absolute_execution_provenance(ahc): def test_json_reports_absolute_execution_provenance(ahc, monkeypatch, capsys):
src = AHC.read_text() code, out = _run_main(ahc, monkeypatch, capsys, ["--json"])
assert "execution_path" in src payload = _json_payload(out)
assert "os.getcwd()" in src assert payload["execution_path"] == os.path.abspath(str(AHC))
assert payload["cwd"] == os.getcwd()
assert code == 1 # stubbed SSH fails every leg, but provenance is still emitted
def test_wrapper_check_has_no_bare_infisical_expectation(ahc): def test_quiet_run_still_carries_provenance_on_the_alert_path(ahc, monkeypatch, capsys):
# A wrapper that never mentions infisical (koonimo sources ~/.hermes/.env) # The production cron runs --quiet; a report without provenance is
# must not be FAILed merely for lacking an infisical reference. # unactionable (item 4). Both the header line and the ALERT line must carry it.
assert "wrapper resolves creds without infisical" in AHC.read_text() code, out = _run_main(ahc, monkeypatch, capsys, ["--quiet"])
assert code == 1
assert "📍 executed from: script=" in out
alerts = [ln for ln in out.splitlines() if ln.startswith("ALERT agent-health:")]
assert alerts, out
assert f"script={os.path.abspath(str(AHC))}" in alerts[0]
assert f"cwd={os.getcwd()}" in alerts[0]
# ── item 4: every contract report carries its execution path ────────── def test_json_surfaces_report_only_findings_separately(ahc, monkeypatch, capsys):
# Koby's down legs are reported but must not count as fleet failures; the
# --json payload exposes them in their own array (item 1 + f8).
_, out = _run_main(ahc, monkeypatch, capsys, ["--json"])
payload = _json_payload(out)
assert isinstance(payload["report_only"], list)
assert any(key.startswith(("gateway-down:koby", "ct-unreachable:koby"))
for key in payload["report_only"])
assert not any("koby" in key for key in payload["failures"])
@pytest.mark.parametrize("contract", [GPU, INFRA, PROXMOX], ids=lambda p: p.name)
def test_report_format_requires_execution_provenance(contract): # ── agent-health-check: wrapper infisical behavior (f3) ───────────────
def _stub_wrapper_ssh(ahc, monkeypatch, wrapper_body, test_x_result="OK", command_v="/usr/local/bin/infisical"):
def fake_ssh(host, cmd, user="root"):
if cmd.startswith("cat /root/.local/bin/hermes"):
return wrapper_body
if cmd.startswith("ls -la /root/.local/bin/hermes "):
return "-rwxr-xr-x 1 root root 0 Jan 1 00:00 /root/.local/bin/hermes"
if cmd.startswith("ls -la /root/.local/bin/hermes-real") or "venv/bin/hermes" in cmd:
return "-rwxr-xr-x 1 root root 0 Jan 1 00:00 /root/.local/bin/hermes-real"
if cmd.startswith("grep -c 'LITELLM_API_KEY'"):
return "1"
if cmd.startswith("test -x "):
return test_x_result
if cmd.startswith("command -v infisical"):
return command_v
return None
monkeypatch.setattr(ahc, "ssh", fake_ssh)
monkeypatch.setattr(ahc, "AGENTS", {"koonimo": dict(ahc.AGENTS["koonimo"])})
ahc.FAIL.clear()
ahc.REPORT_ONLY.clear()
def test_env_based_wrapper_without_infisical_is_not_failed(ahc, monkeypatch, capsys):
_stub_wrapper_ssh(ahc, monkeypatch,
"#!/bin/bash\nsource ~/.hermes/.env\nexec hermes-real \"$@\"\n")
ahc.check_wrapper_integrity()
out = capsys.readouterr().out
assert ahc.FAIL == []
assert "wrapper resolves creds without infisical" in out
def test_dangling_absolute_infisical_path_is_failed(ahc, monkeypatch, capsys):
# Wrapper hardcodes /usr/bin/infisical, which is absent, while PATH resolves
# infisical to /usr/local/bin/infisical. The literal path must be verified,
# not inferred from PATH resolution.
_stub_wrapper_ssh(ahc, monkeypatch,
"#!/bin/bash\n/usr/bin/infisical run -- hermes-real \"$@\"\n",
test_x_result="MISS", command_v="/usr/local/bin/infisical")
ahc.check_wrapper_integrity()
assert "wrapper-infisical-path:koonimo" in ahc.FAIL
def test_existing_absolute_infisical_path_passes(ahc, monkeypatch, capsys):
_stub_wrapper_ssh(ahc, monkeypatch,
"#!/bin/bash\n/usr/bin/infisical run -- hermes-real \"$@\"\n",
test_x_result="OK")
ahc.check_wrapper_integrity()
out = capsys.readouterr().out
assert ahc.FAIL == []
assert "wrapper infisical path OK" in out
# ── item 4: prose-lint enforces report provenance (real consumer) ─────
GOOD_CONTRACT = textwrap.dedent("""\
---
kind: function
name: good
description: fixture with provenance
---
## Parameters
- x: y
## Returns
ok
### check-health
```bash
pwd -P
```
**Report format**: Begin with the absolute path the probe executed from.
""")
DECOY_CONTRACT = textwrap.dedent("""\
---
kind: function
name: decoy
description: fixture with provenance only outside the report format
---
## Parameters
- x: y
## Returns
ok
The absolute path of the config is /etc/foo.
### check-health
```bash
true
```
**Report format**: Summarize actual results from each probe.
""")
MISSING_CONTRACT = textwrap.dedent("""\
---
kind: function
name: missing
description: check-health contract with no report format
---
## Parameters
- x: y
## Returns
ok
### check-health
```bash
pwd -P
```
""")
def _run_lint(tmp_path, text, name):
(tmp_path / name).write_text(text)
return subprocess.run(["bash", str(LINT)], cwd=tmp_path,
capture_output=True, text=True)
def test_prose_lint_accepts_report_format_with_provenance(tmp_path):
result = _run_lint(tmp_path, GOOD_CONTRACT, "good.prose.md")
assert result.returncode == 0, result.stdout + result.stderr
def test_prose_lint_rejects_report_format_without_provenance(tmp_path):
result = _run_lint(tmp_path, DECOY_CONTRACT, "decoy.prose.md")
assert result.returncode == 1, result.stdout
assert "lacks execution provenance" in result.stdout
def test_prose_lint_requires_report_format_on_check_health_contract(tmp_path):
result = _run_lint(tmp_path, MISSING_CONTRACT, "missing.prose.md")
assert result.returncode == 1, result.stdout
assert "no **Report format** paragraph" in result.stdout
# ── contracts: parse the executable check-health probe block ─────────
def _check_health_block(contract):
"""Extract the bash probe block under ### check-health (the probe interface)."""
text = contract.read_text() text = contract.read_text()
assert "**Report format**" in text marker = "### check-health"
assert re.search(r"absolute path|pwd -P|executed from", text) assert marker in text, f"{contract.name} has no {marker}"
after = text.split(marker, 1)[1]
match = re.search(r"```bash\n(.*?)```", after, re.S)
assert match, f"{contract.name} check-health has no bash probe block"
return match.group(1)
# ── item 3: gpu-monitor probes the real GPU endpoints ──────────────── def _urls(block):
# Comments document the forbidden/deprecated probes; only count real commands.
def test_gpu_monitor_probes_gpu_health_on_8080(): code = "\n".join(ln for ln in block.splitlines()
text = GPU.read_text() if not ln.lstrip().startswith("#"))
assert "http://192.168.68.8:8080/health" in text return set(re.findall(r"https?://[^\s\"')]+", code))
assert "http://192.168.68.110:8080/health" in text
def test_gpu_monitor_forbids_bare_port_80_on_gpu_hosts(): def test_gpu_monitor_probes_every_gpu_health_on_8080():
text = GPU.read_text() block = _check_health_block(GPU)
assert re.search(r"never .{0,20}bare port 80", text, re.I) gpu_health = {u for u in _urls(block) if ":8080/health" in u}
# The false-DEGRADED symptom must be documented, not just implied. assert {"http://192.168.68.8:8080/health",
assert "DEGRADED" in text "http://192.168.68.110:8080/health"} <= gpu_health
def test_gpu_monitor_never_probes_bare_port_80_on_gpu_hosts():
block = _check_health_block(GPU)
bare = {u for u in _urls(block)
if re.match(r"https?://192\.168\.68\.(8|110|15)/", u)}
assert bare == set()
assert re.search(r"never .{0,40}bare port 80", block, re.I)
def test_gpu_monitor_documents_router_301_as_alive(): def test_gpu_monitor_documents_router_301_as_alive():
text = GPU.read_text() block = _check_health_block(GPU)
assert "/gpu/gpu-data" in text assert "/health/unified" in block
assert "301" in text assert "301" in block
# ── item 2: infrastructure-monitoring PVE API vantage ──────────────── def test_infra_monitoring_probes_every_real_pve_node():
block = _check_health_block(INFRA)
match = re.search(r"for node in ([^\n;]+)", block)
assert match, "PVE liveness loop not found in check-health"
assert set(match.group(1).split()) == PVE_NODE_IPS
assert "https://$node:8006/api2/json/version" in block
def test_infra_monitoring_does_not_probe_ct116_for_pve_api(): def test_infra_monitoring_does_not_probe_ct116_for_pve_api():
assert "https://192.168.68.116:8006" not in INFRA.read_text() assert "192.168.68.116:8006" not in _check_health_block(INFRA)
@pytest.mark.parametrize(
"node",
["192.168.68.9", "192.168.68.5", "192.168.68.15", "192.168.68.6", "192.168.68.12"],
)
def test_infra_monitoring_probes_every_real_pve_node(node):
assert node in INFRA.read_text()
def test_infra_monitoring_uses_any_http_liveness_rule():
text = INFRA.read_text()
assert "api2/json/version" in text
assert "ANY HTTP status" in text or "any-HTTP" in text