no-mistakes(review): Harden health-check provenance, infisical verification, report-only JSON, tests
This commit is contained in:
@@ -72,8 +72,8 @@ Root causes (all stale expectations; no live fault):
|
|||||||
|---|---|
|
|---|---|
|
||||||
| `ct-unreachable:koby:192.168.68.15` | CT 111 (tdunna/koby) runs on **storepve (.6)**, not amdpve (.15). |
|
| `ct-unreachable:koby:192.168.68.15` | CT 111 (tdunna/koby) runs on **storepve (.6)**, not amdpve (.15). |
|
||||||
| `wrapper-*:abiba` | Abiba is pi-only since the harness purge. `/root/.local/bin/hermes` is a dangling symlink; no `hermes-real`, no `~/.hermes/.env`. |
|
| `wrapper-*:abiba` | Abiba is pi-only since the harness purge. `/root/.local/bin/hermes` is a dangling symlink; no `hermes-real`, no `~/.hermes/.env`. |
|
||||||
| `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. |
|
| `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. Koby's wrapper is also the genuine no-infisical case: it sources `~/.hermes/.env` rather than `/usr/bin/infisical`, which the check now accepts. |
|
||||||
| `wrapper-infisical-path:koonimo` | Koonimo's wrapper injects `KOONIMO_LITELLM_API_KEY` from `~/.hermes/.env` and never invokes infisical. The check required `/usr/bin/infisical`, which is only one valid mechanism. |
|
| `wrapper-infisical-path:koonimo` | Koonimo's wrapper **does** reference `/usr/bin/infisical` — but past the old check's `head -20` window, so the check looked for the path in the wrong slice and false-failed. The fix that mattered was reading the full wrapper body (and then verifying any absolute infisical path it finds actually exists). |
|
||||||
|
|
||||||
**After** — same absolute path, `python3 scripts/agent-health-check.py --no-deploy` (v4):
|
**After** — same absolute path, `python3 scripts/agent-health-check.py --no-deploy` (v4):
|
||||||
|
|
||||||
|
|||||||
@@ -103,14 +103,22 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
|
|||||||
|
|
||||||
## Execution
|
## Execution
|
||||||
|
|
||||||
### Liveness rule (any-HTTP-response)
|
### Liveness rule (scoped)
|
||||||
|
|
||||||
A probe is **ALIVE** if the endpoint returns **ANY** HTTP status — including
|
The any-HTTP-response rule applies ONLY to unauthenticated/auth-gated endpoints,
|
||||||
`401`/`403` auth challenges and `3xx` redirects. A bare `200` is not required and
|
where any HTTP answer proves a listener is up: the PVE API
|
||||||
must never be a pass condition for an auth-gated endpoint. **DOWN = connection
|
(`https://<node>:8006/api2/json/version`) and LiteLLM health
|
||||||
refused (`000`) or timeout only.** Same rule as zulip-health (Tanko) and
|
(`/litellm/health`, `301` → `/litellm/health/liveliness`). For those endpoints a
|
||||||
gpu-monitor. The PVE API (`:8006/api2/json`) legitimately answers `401` to an
|
probe is **ALIVE** on **ANY** HTTP status — `401`/`403` auth challenges and `3xx`
|
||||||
unauthenticated probe — that is the healthy signal, not a failure.
|
redirects included — and **DOWN = connection refused (`000`) or timeout only**.
|
||||||
|
The PVE API legitimately answers `401` to an unauthenticated probe — that is the
|
||||||
|
healthy signal, not a failure. Same scoped rule as zulip-health (Tanko) and
|
||||||
|
gpu-monitor.
|
||||||
|
|
||||||
|
Probes whose success condition is specifically a bare `200` are NOT covered by
|
||||||
|
the any-HTTP rule. On those — the authenticated Zulip POST and the router
|
||||||
|
`/health` — an unexpected status (`401`/`403` from a bad or missing credential,
|
||||||
|
`5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive".
|
||||||
|
|
||||||
### check-health
|
### check-health
|
||||||
|
|
||||||
@@ -172,10 +180,13 @@ curl -s http://192.168.68.116:4001/metrics | head -20
|
|||||||
**Report format**: Begin every report with the **absolute path the probe executed
|
**Report format**: Begin every report with the **absolute path the probe executed
|
||||||
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
|
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
|
||||||
distinguishable from a real fault at read time. Summarize actual results from
|
distinguishable from a real fault at read time. Summarize actual results from
|
||||||
each probe. Apply the any-HTTP-response liveness rule above: only
|
each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE
|
||||||
connection-refused (`000`) or timeout is DOWN; empty output is a warning. A probe
|
API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is
|
||||||
that requires a bare `200` on an auth-gated endpoint (PVE API → `401`, LiteLLM
|
DOWN; empty output is a warning. For probes whose expected result is a bare `200`
|
||||||
health → `301` redirect) is a stale expectation, not a fault.
|
(the authenticated Zulip POST, router `/health`), flag an alert on any unexpected
|
||||||
|
status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200`
|
||||||
|
expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect)
|
||||||
|
is a stale expectation, not a fault.
|
||||||
|
|
||||||
|
|
||||||
### Phase 1: GPU Exporters
|
### Phase 1: GPU Exporters
|
||||||
|
|||||||
@@ -32,14 +32,21 @@ Changelog:
|
|||||||
2026-08-17 ruling: every koby leg is detected and reported, never counted as a
|
2026-08-17 ruling: every koby leg is detected and reported, never counted as a
|
||||||
fleet failure and never repaired. koby's PVE mapping corrected to storepve
|
fleet failure and never repaired. koby's PVE mapping corrected to storepve
|
||||||
(CT 111 tdunna lives on .6 — the old amdpve mapping produced a false
|
(CT 111 tdunna lives on .6 — the old amdpve mapping produced a false
|
||||||
ct-unreachable). The wrapper infisical-path check no longer FAILs .env-based
|
ct-unreachable). The wrapper infisical-path check had two stale-expectation
|
||||||
wrappers that legitimately never invoke infisical (koonimo/baggy). Every run
|
bugs: it read only the first 20 lines of the wrapper, so koonimo (whose
|
||||||
now prints absolute execution provenance (script + cwd) in the header and in
|
wrapper does reference /usr/bin/infisical, just past line 20) was falsely
|
||||||
--json output so a stale-consumer report is distinguishable from a fault at
|
FAILed as "path may be wrong"; and it treated the absence of any infisical
|
||||||
read time.
|
reference as a fault, though koby's wrapper sources the key from
|
||||||
|
~/.hermes/.env and never invokes infisical. The check now reads the full
|
||||||
|
wrapper body, accepts a no-infisical wrapper, and verifies that any absolute
|
||||||
|
infisical path the wrapper references actually exists. Report-only findings
|
||||||
|
are surfaced in a machine-readable `report_only` array in --json output,
|
||||||
|
separate from `failures`. Every run prints absolute execution provenance
|
||||||
|
(script + cwd) in the header, in the cron ALERT line, and in --json output so
|
||||||
|
a stale-consumer report is distinguishable from a fault at read time.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import subprocess, json, sys, os, time
|
import subprocess, json, sys, os, time, re
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
LITELLM = "http://192.168.68.116:80"
|
LITELLM = "http://192.168.68.116:80"
|
||||||
@@ -88,6 +95,7 @@ GPU_HOSTS = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
FAIL = []
|
FAIL = []
|
||||||
|
REPORT_ONLY = []
|
||||||
|
|
||||||
|
|
||||||
def _fail(key, agent_name=None):
|
def _fail(key, agent_name=None):
|
||||||
@@ -95,11 +103,13 @@ def _fail(key, agent_name=None):
|
|||||||
|
|
||||||
Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs
|
Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs
|
||||||
are detected and reported, never repaired and never counted as fleet
|
are detected and reported, never repaired and never counted as fleet
|
||||||
failures. A red fleet alert on a known report-only leg is a false alarm. Any
|
failures. A red fleet alert on a known report-only leg is a false alarm.
|
||||||
non-report-only agent (or a leg with no agent, e.g. GPU hosts) records
|
Report-only findings are tracked separately so --json consumers can still
|
||||||
normally.
|
see them without them counting as fleet failures. Any non-report-only agent
|
||||||
|
(or a leg with no agent, e.g. GPU hosts) records normally.
|
||||||
"""
|
"""
|
||||||
if agent_name and AGENTS.get(agent_name, {}).get("report_only"):
|
if agent_name and AGENTS.get(agent_name, {}).get("report_only"):
|
||||||
|
REPORT_ONLY.append(key)
|
||||||
print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired")
|
print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired")
|
||||||
return
|
return
|
||||||
FAIL.append(key)
|
FAIL.append(key)
|
||||||
@@ -509,23 +519,44 @@ def check_wrapper_integrity():
|
|||||||
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
|
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
|
||||||
|
|
||||||
# Credential-injection mechanism. The Hermes-era wrapper injected creds
|
# Credential-injection mechanism. The Hermes-era wrapper injected creds
|
||||||
# with `/usr/bin/infisical run`; newer wrappers source the agent key from
|
# with `/usr/bin/infisical run`, but the mechanism is not required to be
|
||||||
# ~/.hermes/.env and never mention infisical (koonimo/baggy does exactly
|
# infisical at all: koby's wrapper sources the key from ~/.hermes/.env
|
||||||
# this). A wrapper with NO infisical reference is therefore a valid
|
# and never mentions infisical, which is valid. The old check read only
|
||||||
# alternate mechanism, not a fault — only flag a wrapper that DOES call
|
# the first 20 lines, so koonimo's wrapper — which DOES reference
|
||||||
# infisical when the binary cannot resolve. The old check FAILed every
|
# /usr/bin/infisical, just past line 20 — false-failed as "path may be
|
||||||
# .env-based wrapper as "infisical path may be wrong": a stale
|
# wrong". Read the full body, accept a no-infisical wrapper, and verify
|
||||||
# expectation, not a fault.
|
# that any absolute infisical path the wrapper hardcodes actually exists
|
||||||
|
# (PATH resolution alone is not enough — a dangling /usr/bin/infisical is
|
||||||
|
# a broken wrapper even when a different infisical is on PATH).
|
||||||
wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
|
wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
|
||||||
if "infisical" in wrapper_body:
|
if "infisical" in wrapper_body:
|
||||||
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
|
inf_paths = []
|
||||||
if not inf_actual:
|
for _m in re.finditer(r"(/[A-Za-z0-9._/-]*infisical)", wrapper_body):
|
||||||
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
|
if _m.group(1) not in inf_paths:
|
||||||
_fail(f"wrapper-no-infisical:{name}", name)
|
inf_paths.append(_m.group(1))
|
||||||
elif "/usr/bin/infisical" not in wrapper_body:
|
dangling = []
|
||||||
print(f" ⚠️ {name}: wrapper infisical path differs (infisical at {inf_actual}) — informational")
|
for _p in inf_paths:
|
||||||
|
_exists = ssh(host, f"test -x {_p} && echo OK || echo MISS", user=user)
|
||||||
|
if not _exists or _exists.strip().splitlines()[-1] != "OK":
|
||||||
|
dangling.append(_p)
|
||||||
|
if inf_paths:
|
||||||
|
if dangling:
|
||||||
|
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
|
||||||
|
suffix = f" (infisical at {inf_actual})" if inf_actual else ""
|
||||||
|
print(f" ❌ {name}: wrapper hardcodes missing infisical path(s) "
|
||||||
|
f"{', '.join(dangling)}{suffix}")
|
||||||
|
_fail(f"wrapper-infisical-path:{name}", name)
|
||||||
|
elif "/usr/bin/infisical" not in wrapper_body:
|
||||||
|
print(f" ⚠️ {name}: wrapper infisical path differs — informational")
|
||||||
|
else:
|
||||||
|
print(f" ✅ {name}: wrapper infisical path OK")
|
||||||
else:
|
else:
|
||||||
print(f" ✅ {name}: wrapper infisical path OK")
|
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
|
||||||
|
if not inf_actual:
|
||||||
|
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
|
||||||
|
_fail(f"wrapper-no-infisical:{name}", name)
|
||||||
|
else:
|
||||||
|
print(f" ✅ {name}: wrapper infisical resolves via PATH ({inf_actual})")
|
||||||
else:
|
else:
|
||||||
print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
|
print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
|
||||||
|
|
||||||
@@ -614,14 +645,16 @@ def main():
|
|||||||
|
|
||||||
# Provenance: a report is only actionable if the reader can tell WHICH copy
|
# Provenance: a report is only actionable if the reader can tell WHICH copy
|
||||||
# of this script produced it. Emit the absolute execution path (script + cwd)
|
# of this script produced it. Emit the absolute execution path (script + cwd)
|
||||||
# in both human and --json output so a stale-consumer report is
|
# in human, cron-alert, and --json output so a stale-consumer report is
|
||||||
# distinguishable from a real fault at read time.
|
# distinguishable from a real fault at read time. The production cron path
|
||||||
|
# runs --quiet, so provenance must NOT be behind the quiet guard.
|
||||||
script_path = os.path.abspath(__file__)
|
script_path = os.path.abspath(__file__)
|
||||||
cwd = os.getcwd()
|
cwd = os.getcwd()
|
||||||
|
|
||||||
if not quiet:
|
if not quiet:
|
||||||
print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
|
print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
|
||||||
print(f"📍 executed from: script={script_path} cwd={cwd}")
|
print(f"📍 executed from: script={script_path} cwd={cwd}")
|
||||||
|
if not quiet:
|
||||||
print()
|
print()
|
||||||
|
|
||||||
load_agent_keys()
|
load_agent_keys()
|
||||||
@@ -656,14 +689,15 @@ def main():
|
|||||||
if FAIL:
|
if FAIL:
|
||||||
print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
|
print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
|
||||||
if quiet:
|
if quiet:
|
||||||
print(f"ALERT agent-health:{','.join(FAIL)}")
|
print(f"ALERT agent-health:{','.join(FAIL)} script={script_path} cwd={cwd}")
|
||||||
elif not quiet:
|
elif not quiet:
|
||||||
print("\n✅ All checks passed")
|
print("\n✅ All checks passed")
|
||||||
|
|
||||||
if as_json:
|
if as_json:
|
||||||
print(json.dumps({"timestamp": datetime.now().isoformat(),
|
print(json.dumps({"timestamp": datetime.now().isoformat(),
|
||||||
"execution_path": script_path, "cwd": cwd,
|
"execution_path": script_path, "cwd": cwd,
|
||||||
"failures": FAIL, "healthy": len(FAIL) == 0}))
|
"failures": FAIL, "report_only": REPORT_ONLY,
|
||||||
|
"healthy": len(FAIL) == 0}))
|
||||||
|
|
||||||
sys.exit(1 if FAIL else 0)
|
sys.exit(1 if FAIL else 0)
|
||||||
|
|
||||||
|
|||||||
+13
-3
@@ -87,11 +87,21 @@ echo " ✅ IP consistency verified (.19=.122=.123 all reachable)"
|
|||||||
# Report provenance — every contract report must state the absolute path it
|
# Report provenance — every contract report must state the absolute path it
|
||||||
# executed from, so a stale-consumer report is distinguishable from a real fault
|
# executed from, so a stale-consumer report is distinguishable from a real fault
|
||||||
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
|
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
|
||||||
PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
|
# Enforced only inside the **Report format** paragraph, and a check-health
|
||||||
if [ -n "$PROV_FILES" ]; then
|
# contract with no Report format paragraph FAILs rather than being skipped.
|
||||||
|
PROV_FILES=$(grep -rlE '^### check-health|\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
|
||||||
|
if [ -z "$PROV_FILES" ]; then
|
||||||
|
echo " ❌ No check-health/report-format contracts found — provenance not enforced"
|
||||||
|
FAILED=1
|
||||||
|
else
|
||||||
PROV_BAD=0
|
PROV_BAD=0
|
||||||
while IFS= read -r f; do
|
while IFS= read -r f; do
|
||||||
if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then
|
[ -n "$f" ] || continue
|
||||||
|
REPORT_PARA=$(awk '/\*\*Report format\*\*/{found=1} found{print} found && /^[[:space:]]*$/{exit}' "$f")
|
||||||
|
if [ -z "$REPORT_PARA" ]; then
|
||||||
|
echo " ❌ $f: check-health contract has no **Report format** paragraph"
|
||||||
|
PROV_BAD=1
|
||||||
|
elif ! printf '%s\n' "$REPORT_PARA" | grep -qE 'absolute path|pwd -P|executed from'; then
|
||||||
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
|
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
|
||||||
PROV_BAD=1
|
PROV_BAD=1
|
||||||
fi
|
fi
|
||||||
|
|||||||
+268
-56
@@ -7,30 +7,46 @@ stale expectations rather than live faults.
|
|||||||
abiba (pi-only since the harness purge) was tested as a Hermes host, koby
|
abiba (pi-only since the harness purge) was tested as a Hermes host, koby
|
||||||
(report-only per the captain's 2026-08-17 ruling) was counted as repairable,
|
(report-only per the captain's 2026-08-17 ruling) was counted as repairable,
|
||||||
koby's CT 111 was probed on amdpve where it does not exist (it runs on
|
koby's CT 111 was probed on amdpve where it does not exist (it runs on
|
||||||
storepve .6), and a .env-based hermes wrapper was FAILed for not mentioning
|
storepve .6), and the wrapper infisical check had two bugs — it read only
|
||||||
infisical.
|
the first 20 lines, so koonimo's wrapper (which references /usr/bin/infisical
|
||||||
|
past line 20) false-failed, and it treated koby's genuine no-infisical
|
||||||
|
(~/.hermes/.env) wrapper as broken.
|
||||||
* gpu-monitor emitted "DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000" three
|
* gpu-monitor emitted "DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000" three
|
||||||
times from probing bare port 80 on GPU hosts while :8080 answered 200.
|
times from probing bare port 80 on GPU hosts while :8080 answered 200.
|
||||||
* infrastructure-monitoring probed CT 116 for the PVE API (no pveproxy ->
|
* infrastructure-monitoring probed CT 116 for the PVE API (no pveproxy ->
|
||||||
000) instead of the five real cluster nodes, which answer 401 = alive.
|
000) instead of the five real cluster nodes, which answer 401 = alive.
|
||||||
|
|
||||||
These tests pin the corrections so the false alarms cannot return silently.
|
These tests execute the health script (with SSH/vault stubbed) and the real
|
||||||
They are static/structural over the contracts plus an inert import of the health
|
provenance consumer (scripts/prose-lint.sh), and parse the contracts' executable
|
||||||
script — no live network, vault, or SSH access is required.
|
check-health probe blocks into normalized probe sets. No live network, vault, or
|
||||||
|
SSH access is required.
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import importlib.util
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import os
|
||||||
import pathlib
|
import pathlib
|
||||||
import re
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import textwrap
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||||
AHC = ROOT / "scripts" / "agent-health-check.py"
|
AHC = ROOT / "scripts" / "agent-health-check.py"
|
||||||
|
LINT = ROOT / "scripts" / "prose-lint.sh"
|
||||||
GPU = ROOT / "gpu-monitor.prose.md"
|
GPU = ROOT / "gpu-monitor.prose.md"
|
||||||
INFRA = ROOT / "infrastructure-monitoring.prose.md"
|
INFRA = ROOT / "infrastructure-monitoring.prose.md"
|
||||||
PROXMOX = ROOT / "proxmox-monitor.prose.md"
|
|
||||||
|
PVE_NODE_IPS = {
|
||||||
|
"192.168.68.9",
|
||||||
|
"192.168.68.5",
|
||||||
|
"192.168.68.15",
|
||||||
|
"192.168.68.6",
|
||||||
|
"192.168.68.12",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="module")
|
@pytest.fixture(scope="module")
|
||||||
@@ -43,6 +59,26 @@ def ahc():
|
|||||||
return module
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
# ── helpers: execute the health script with SSH/vault stubbed ─────────
|
||||||
|
|
||||||
|
def _run_main(ahc, monkeypatch, capsys, argv, ssh_result=None):
|
||||||
|
ahc.FAIL.clear()
|
||||||
|
ahc.REPORT_ONLY.clear()
|
||||||
|
monkeypatch.setattr(ahc, "load_agent_keys", lambda: None)
|
||||||
|
monkeypatch.setattr(ahc, "ssh", lambda *a, **k: ssh_result)
|
||||||
|
monkeypatch.setattr(sys, "argv", ["agent-health-check.py", "--no-deploy", *argv])
|
||||||
|
with pytest.raises(SystemExit) as exc:
|
||||||
|
ahc.main()
|
||||||
|
return exc.value.code, capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def _json_payload(out):
|
||||||
|
for line in reversed(out.splitlines()):
|
||||||
|
if line.startswith('{"timestamp"'):
|
||||||
|
return json.loads(line)
|
||||||
|
raise AssertionError(f"no JSON payload in output:\n{out}")
|
||||||
|
|
||||||
|
|
||||||
# ── agent-health-check: stale-expectation legs ───────────────────────
|
# ── agent-health-check: stale-expectation legs ───────────────────────
|
||||||
|
|
||||||
def test_import_does_not_contact_vault(ahc):
|
def test_import_does_not_contact_vault(ahc):
|
||||||
@@ -68,17 +104,19 @@ def test_koby_ct111_is_on_storepve(ahc):
|
|||||||
assert ahc.AGENTS["koby"]["pve"] == "storepve"
|
assert ahc.AGENTS["koby"]["pve"] == "storepve"
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("agent", ["koby", "koonimo", "tanko"])
|
def test_report_only_legs_never_count_as_failures(ahc):
|
||||||
def test_report_only_legs_never_count_as_failures(ahc, agent):
|
for agent, report_only in (("koby", True), ("koonimo", False), ("tanko", False)):
|
||||||
ahc.FAIL.clear()
|
ahc.FAIL.clear()
|
||||||
try:
|
ahc.REPORT_ONLY.clear()
|
||||||
ahc._fail(f"probe:{agent}", agent)
|
ahc._fail(f"probe:{agent}", agent)
|
||||||
if ahc.AGENTS[agent].get("report_only"):
|
if report_only:
|
||||||
assert ahc.FAIL == []
|
assert ahc.FAIL == []
|
||||||
|
assert ahc.REPORT_ONLY == [f"probe:{agent}"]
|
||||||
else:
|
else:
|
||||||
assert ahc.FAIL == [f"probe:{agent}"]
|
assert ahc.FAIL == [f"probe:{agent}"]
|
||||||
finally:
|
assert ahc.REPORT_ONLY == []
|
||||||
ahc.FAIL.clear()
|
ahc.FAIL.clear()
|
||||||
|
ahc.REPORT_ONLY.clear()
|
||||||
|
|
||||||
|
|
||||||
def test_failure_recording_accepts_agentless_keys(ahc):
|
def test_failure_recording_accepts_agentless_keys(ahc):
|
||||||
@@ -90,63 +128,237 @@ def test_failure_recording_accepts_agentless_keys(ahc):
|
|||||||
ahc.FAIL.clear()
|
ahc.FAIL.clear()
|
||||||
|
|
||||||
|
|
||||||
def test_script_reports_absolute_execution_provenance(ahc):
|
def test_json_reports_absolute_execution_provenance(ahc, monkeypatch, capsys):
|
||||||
src = AHC.read_text()
|
code, out = _run_main(ahc, monkeypatch, capsys, ["--json"])
|
||||||
assert "execution_path" in src
|
payload = _json_payload(out)
|
||||||
assert "os.getcwd()" in src
|
assert payload["execution_path"] == os.path.abspath(str(AHC))
|
||||||
|
assert payload["cwd"] == os.getcwd()
|
||||||
|
assert code == 1 # stubbed SSH fails every leg, but provenance is still emitted
|
||||||
|
|
||||||
|
|
||||||
def test_wrapper_check_has_no_bare_infisical_expectation(ahc):
|
def test_quiet_run_still_carries_provenance_on_the_alert_path(ahc, monkeypatch, capsys):
|
||||||
# A wrapper that never mentions infisical (koonimo sources ~/.hermes/.env)
|
# The production cron runs --quiet; a report without provenance is
|
||||||
# must not be FAILed merely for lacking an infisical reference.
|
# unactionable (item 4). Both the header line and the ALERT line must carry it.
|
||||||
assert "wrapper resolves creds without infisical" in AHC.read_text()
|
code, out = _run_main(ahc, monkeypatch, capsys, ["--quiet"])
|
||||||
|
assert code == 1
|
||||||
|
assert "📍 executed from: script=" in out
|
||||||
|
alerts = [ln for ln in out.splitlines() if ln.startswith("ALERT agent-health:")]
|
||||||
|
assert alerts, out
|
||||||
|
assert f"script={os.path.abspath(str(AHC))}" in alerts[0]
|
||||||
|
assert f"cwd={os.getcwd()}" in alerts[0]
|
||||||
|
|
||||||
|
|
||||||
# ── item 4: every contract report carries its execution path ──────────
|
def test_json_surfaces_report_only_findings_separately(ahc, monkeypatch, capsys):
|
||||||
|
# Koby's down legs are reported but must not count as fleet failures; the
|
||||||
|
# --json payload exposes them in their own array (item 1 + f8).
|
||||||
|
_, out = _run_main(ahc, monkeypatch, capsys, ["--json"])
|
||||||
|
payload = _json_payload(out)
|
||||||
|
assert isinstance(payload["report_only"], list)
|
||||||
|
assert any(key.startswith(("gateway-down:koby", "ct-unreachable:koby"))
|
||||||
|
for key in payload["report_only"])
|
||||||
|
assert not any("koby" in key for key in payload["failures"])
|
||||||
|
|
||||||
@pytest.mark.parametrize("contract", [GPU, INFRA, PROXMOX], ids=lambda p: p.name)
|
|
||||||
def test_report_format_requires_execution_provenance(contract):
|
# ── agent-health-check: wrapper infisical behavior (f3) ───────────────
|
||||||
|
|
||||||
|
def _stub_wrapper_ssh(ahc, monkeypatch, wrapper_body, test_x_result="OK", command_v="/usr/local/bin/infisical"):
|
||||||
|
def fake_ssh(host, cmd, user="root"):
|
||||||
|
if cmd.startswith("cat /root/.local/bin/hermes"):
|
||||||
|
return wrapper_body
|
||||||
|
if cmd.startswith("ls -la /root/.local/bin/hermes "):
|
||||||
|
return "-rwxr-xr-x 1 root root 0 Jan 1 00:00 /root/.local/bin/hermes"
|
||||||
|
if cmd.startswith("ls -la /root/.local/bin/hermes-real") or "venv/bin/hermes" in cmd:
|
||||||
|
return "-rwxr-xr-x 1 root root 0 Jan 1 00:00 /root/.local/bin/hermes-real"
|
||||||
|
if cmd.startswith("grep -c 'LITELLM_API_KEY'"):
|
||||||
|
return "1"
|
||||||
|
if cmd.startswith("test -x "):
|
||||||
|
return test_x_result
|
||||||
|
if cmd.startswith("command -v infisical"):
|
||||||
|
return command_v
|
||||||
|
return None
|
||||||
|
|
||||||
|
monkeypatch.setattr(ahc, "ssh", fake_ssh)
|
||||||
|
monkeypatch.setattr(ahc, "AGENTS", {"koonimo": dict(ahc.AGENTS["koonimo"])})
|
||||||
|
ahc.FAIL.clear()
|
||||||
|
ahc.REPORT_ONLY.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_based_wrapper_without_infisical_is_not_failed(ahc, monkeypatch, capsys):
|
||||||
|
_stub_wrapper_ssh(ahc, monkeypatch,
|
||||||
|
"#!/bin/bash\nsource ~/.hermes/.env\nexec hermes-real \"$@\"\n")
|
||||||
|
ahc.check_wrapper_integrity()
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert ahc.FAIL == []
|
||||||
|
assert "wrapper resolves creds without infisical" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_dangling_absolute_infisical_path_is_failed(ahc, monkeypatch, capsys):
|
||||||
|
# Wrapper hardcodes /usr/bin/infisical, which is absent, while PATH resolves
|
||||||
|
# infisical to /usr/local/bin/infisical. The literal path must be verified,
|
||||||
|
# not inferred from PATH resolution.
|
||||||
|
_stub_wrapper_ssh(ahc, monkeypatch,
|
||||||
|
"#!/bin/bash\n/usr/bin/infisical run -- hermes-real \"$@\"\n",
|
||||||
|
test_x_result="MISS", command_v="/usr/local/bin/infisical")
|
||||||
|
ahc.check_wrapper_integrity()
|
||||||
|
assert "wrapper-infisical-path:koonimo" in ahc.FAIL
|
||||||
|
|
||||||
|
|
||||||
|
def test_existing_absolute_infisical_path_passes(ahc, monkeypatch, capsys):
|
||||||
|
_stub_wrapper_ssh(ahc, monkeypatch,
|
||||||
|
"#!/bin/bash\n/usr/bin/infisical run -- hermes-real \"$@\"\n",
|
||||||
|
test_x_result="OK")
|
||||||
|
ahc.check_wrapper_integrity()
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert ahc.FAIL == []
|
||||||
|
assert "wrapper infisical path OK" in out
|
||||||
|
|
||||||
|
|
||||||
|
# ── item 4: prose-lint enforces report provenance (real consumer) ─────
|
||||||
|
|
||||||
|
GOOD_CONTRACT = textwrap.dedent("""\
|
||||||
|
---
|
||||||
|
kind: function
|
||||||
|
name: good
|
||||||
|
description: fixture with provenance
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parameters
|
||||||
|
|
||||||
|
- x: y
|
||||||
|
|
||||||
|
## Returns
|
||||||
|
|
||||||
|
ok
|
||||||
|
|
||||||
|
### check-health
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pwd -P
|
||||||
|
```
|
||||||
|
|
||||||
|
**Report format**: Begin with the absolute path the probe executed from.
|
||||||
|
""")
|
||||||
|
|
||||||
|
DECOY_CONTRACT = textwrap.dedent("""\
|
||||||
|
---
|
||||||
|
kind: function
|
||||||
|
name: decoy
|
||||||
|
description: fixture with provenance only outside the report format
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parameters
|
||||||
|
|
||||||
|
- x: y
|
||||||
|
|
||||||
|
## Returns
|
||||||
|
|
||||||
|
ok
|
||||||
|
|
||||||
|
The absolute path of the config is /etc/foo.
|
||||||
|
|
||||||
|
### check-health
|
||||||
|
|
||||||
|
```bash
|
||||||
|
true
|
||||||
|
```
|
||||||
|
|
||||||
|
**Report format**: Summarize actual results from each probe.
|
||||||
|
""")
|
||||||
|
|
||||||
|
MISSING_CONTRACT = textwrap.dedent("""\
|
||||||
|
---
|
||||||
|
kind: function
|
||||||
|
name: missing
|
||||||
|
description: check-health contract with no report format
|
||||||
|
---
|
||||||
|
|
||||||
|
## Parameters
|
||||||
|
|
||||||
|
- x: y
|
||||||
|
|
||||||
|
## Returns
|
||||||
|
|
||||||
|
ok
|
||||||
|
|
||||||
|
### check-health
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pwd -P
|
||||||
|
```
|
||||||
|
""")
|
||||||
|
|
||||||
|
|
||||||
|
def _run_lint(tmp_path, text, name):
|
||||||
|
(tmp_path / name).write_text(text)
|
||||||
|
return subprocess.run(["bash", str(LINT)], cwd=tmp_path,
|
||||||
|
capture_output=True, text=True)
|
||||||
|
|
||||||
|
|
||||||
|
def test_prose_lint_accepts_report_format_with_provenance(tmp_path):
|
||||||
|
result = _run_lint(tmp_path, GOOD_CONTRACT, "good.prose.md")
|
||||||
|
assert result.returncode == 0, result.stdout + result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
def test_prose_lint_rejects_report_format_without_provenance(tmp_path):
|
||||||
|
result = _run_lint(tmp_path, DECOY_CONTRACT, "decoy.prose.md")
|
||||||
|
assert result.returncode == 1, result.stdout
|
||||||
|
assert "lacks execution provenance" in result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def test_prose_lint_requires_report_format_on_check_health_contract(tmp_path):
|
||||||
|
result = _run_lint(tmp_path, MISSING_CONTRACT, "missing.prose.md")
|
||||||
|
assert result.returncode == 1, result.stdout
|
||||||
|
assert "no **Report format** paragraph" in result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
# ── contracts: parse the executable check-health probe block ─────────
|
||||||
|
|
||||||
|
def _check_health_block(contract):
|
||||||
|
"""Extract the bash probe block under ### check-health (the probe interface)."""
|
||||||
text = contract.read_text()
|
text = contract.read_text()
|
||||||
assert "**Report format**" in text
|
marker = "### check-health"
|
||||||
assert re.search(r"absolute path|pwd -P|executed from", text)
|
assert marker in text, f"{contract.name} has no {marker}"
|
||||||
|
after = text.split(marker, 1)[1]
|
||||||
|
match = re.search(r"```bash\n(.*?)```", after, re.S)
|
||||||
|
assert match, f"{contract.name} check-health has no bash probe block"
|
||||||
|
return match.group(1)
|
||||||
|
|
||||||
|
|
||||||
# ── item 3: gpu-monitor probes the real GPU endpoints ────────────────
|
def _urls(block):
|
||||||
|
# Comments document the forbidden/deprecated probes; only count real commands.
|
||||||
def test_gpu_monitor_probes_gpu_health_on_8080():
|
code = "\n".join(ln for ln in block.splitlines()
|
||||||
text = GPU.read_text()
|
if not ln.lstrip().startswith("#"))
|
||||||
assert "http://192.168.68.8:8080/health" in text
|
return set(re.findall(r"https?://[^\s\"')]+", code))
|
||||||
assert "http://192.168.68.110:8080/health" in text
|
|
||||||
|
|
||||||
|
|
||||||
def test_gpu_monitor_forbids_bare_port_80_on_gpu_hosts():
|
def test_gpu_monitor_probes_every_gpu_health_on_8080():
|
||||||
text = GPU.read_text()
|
block = _check_health_block(GPU)
|
||||||
assert re.search(r"never .{0,20}bare port 80", text, re.I)
|
gpu_health = {u for u in _urls(block) if ":8080/health" in u}
|
||||||
# The false-DEGRADED symptom must be documented, not just implied.
|
assert {"http://192.168.68.8:8080/health",
|
||||||
assert "DEGRADED" in text
|
"http://192.168.68.110:8080/health"} <= gpu_health
|
||||||
|
|
||||||
|
|
||||||
|
def test_gpu_monitor_never_probes_bare_port_80_on_gpu_hosts():
|
||||||
|
block = _check_health_block(GPU)
|
||||||
|
bare = {u for u in _urls(block)
|
||||||
|
if re.match(r"https?://192\.168\.68\.(8|110|15)/", u)}
|
||||||
|
assert bare == set()
|
||||||
|
assert re.search(r"never .{0,40}bare port 80", block, re.I)
|
||||||
|
|
||||||
|
|
||||||
def test_gpu_monitor_documents_router_301_as_alive():
|
def test_gpu_monitor_documents_router_301_as_alive():
|
||||||
text = GPU.read_text()
|
block = _check_health_block(GPU)
|
||||||
assert "/gpu/gpu-data" in text
|
assert "/health/unified" in block
|
||||||
assert "301" in text
|
assert "301" in block
|
||||||
|
|
||||||
|
|
||||||
# ── item 2: infrastructure-monitoring PVE API vantage ────────────────
|
def test_infra_monitoring_probes_every_real_pve_node():
|
||||||
|
block = _check_health_block(INFRA)
|
||||||
|
match = re.search(r"for node in ([^\n;]+)", block)
|
||||||
|
assert match, "PVE liveness loop not found in check-health"
|
||||||
|
assert set(match.group(1).split()) == PVE_NODE_IPS
|
||||||
|
assert "https://$node:8006/api2/json/version" in block
|
||||||
|
|
||||||
|
|
||||||
def test_infra_monitoring_does_not_probe_ct116_for_pve_api():
|
def test_infra_monitoring_does_not_probe_ct116_for_pve_api():
|
||||||
assert "https://192.168.68.116:8006" not in INFRA.read_text()
|
assert "192.168.68.116:8006" not in _check_health_block(INFRA)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
"node",
|
|
||||||
["192.168.68.9", "192.168.68.5", "192.168.68.15", "192.168.68.6", "192.168.68.12"],
|
|
||||||
)
|
|
||||||
def test_infra_monitoring_probes_every_real_pve_node(node):
|
|
||||||
assert node in INFRA.read_text()
|
|
||||||
|
|
||||||
|
|
||||||
def test_infra_monitoring_uses_any_http_liveness_rule():
|
|
||||||
text = INFRA.read_text()
|
|
||||||
assert "api2/json/version" in text
|
|
||||||
assert "ANY HTTP status" in text or "any-HTTP" in text
|
|
||||||
|
|||||||
Reference in New Issue
Block a user