SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

no-mistakes(review): Harden health-check provenance, infisical verification, report-only JSON, tests

Browse Source
This commit is contained in:
abiba-bot
2026-09-10 01:35:44 +00:00
parent c66d9c1e20
commit 194e256ac5
5 changed files with 367 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/agent-health-check.py
+62 -28
View File
@@ -32,14 +32,21 @@ Changelog:
2026-08-17 ruling: every koby leg is detected and reported, never counted as a
fleet failure and never repaired. koby's PVE mapping corrected to storepve
(CT 111 tdunna lives on .6 — the old amdpve mapping produced a false
ct-unreachable). The wrapper infisical-path check no longer FAILs .env-based
wrappers that legitimately never invoke infisical (koonimo/baggy). Every run
now prints absolute execution provenance (script + cwd) in the header and in
--json output so a stale-consumer report is distinguishable from a fault at
read time.
ct-unreachable). The wrapper infisical-path check had two stale-expectation
bugs: it read only the first 20 lines of the wrapper, so koonimo (whose
wrapper does reference /usr/bin/infisical, just past line 20) was falsely
FAILed as "path may be wrong"; and it treated the absence of any infisical
reference as a fault, though koby's wrapper sources the key from
~/.hermes/.env and never invokes infisical. The check now reads the full
wrapper body, accepts a no-infisical wrapper, and verifies that any absolute
infisical path the wrapper references actually exists. Report-only findings
are surfaced in a machine-readable `report_only` array in --json output,
separate from `failures`. Every run prints absolute execution provenance
(script + cwd) in the header, in the cron ALERT line, and in --json output so
a stale-consumer report is distinguishable from a fault at read time.
"""
import subprocess, json, sys, os, time
import subprocess, json, sys, os, time, re
from datetime import datetime
LITELLM = "http://192.168.68.116:80"
@@ -88,6 +95,7 @@ GPU_HOSTS = {
}
FAIL = []
REPORT_ONLY = []
def _fail(key, agent_name=None):
@@ -95,11 +103,13 @@ def _fail(key, agent_name=None):
Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs
are detected and reported, never repaired and never counted as fleet
failures. A red fleet alert on a known report-only leg is a false alarm. Any
non-report-only agent (or a leg with no agent, e.g. GPU hosts) records
normally.
failures. A red fleet alert on a known report-only leg is a false alarm.
Report-only findings are tracked separately so --json consumers can still
see them without them counting as fleet failures. Any non-report-only agent
(or a leg with no agent, e.g. GPU hosts) records normally.
"""
if agent_name and AGENTS.get(agent_name, {}).get("report_only"):
REPORT_ONLY.append(key)
print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired")
return
FAIL.append(key)
@@ -509,23 +519,44 @@ def check_wrapper_integrity():
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
# Credential-injection mechanism. The Hermes-era wrapper injected creds
# with `/usr/bin/infisical run`; newer wrappers source the agent key from
# ~/.hermes/.env and never mention infisical (koonimo/baggy does exactly
# this). A wrapper with NO infisical reference is therefore a valid
# alternate mechanism, not a fault — only flag a wrapper that DOES call
# infisical when the binary cannot resolve. The old check FAILed every
# .env-based wrapper as "infisical path may be wrong": a stale
# expectation, not a fault.
# with `/usr/bin/infisical run`, but the mechanism is not required to be
# infisical at all: koby's wrapper sources the key from ~/.hermes/.env
# and never mentions infisical, which is valid. The old check read only
# the first 20 lines, so koonimo's wrapper — which DOES reference
# /usr/bin/infisical, just past line 20 — false-failed as "path may be
# wrong". Read the full body, accept a no-infisical wrapper, and verify
# that any absolute infisical path the wrapper hardcodes actually exists
# (PATH resolution alone is not enough — a dangling /usr/bin/infisical is
# a broken wrapper even when a different infisical is on PATH).
wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
if "infisical" in wrapper_body:
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
_fail(f"wrapper-no-infisical:{name}", name)
elif "/usr/bin/infisical" not in wrapper_body:
print(f" ⚠️ {name}: wrapper infisical path differs (infisical at {inf_actual}) — informational")
inf_paths = []
for _m in re.finditer(r"(/[A-Za-z0-9._/-]*infisical)", wrapper_body):
if _m.group(1) not in inf_paths:
inf_paths.append(_m.group(1))
dangling = []
for _p in inf_paths:
_exists = ssh(host, f"test -x {_p} && echo OK || echo MISS", user=user)
if not _exists or _exists.strip().splitlines()[-1] != "OK":
dangling.append(_p)
if inf_paths:
if dangling:
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
suffix = f" (infisical at {inf_actual})" if inf_actual else ""
print(f" ❌ {name}: wrapper hardcodes missing infisical path(s) "
f"{', '.join(dangling)}{suffix}")
_fail(f"wrapper-infisical-path:{name}", name)
elif "/usr/bin/infisical" not in wrapper_body:
print(f" ⚠️ {name}: wrapper infisical path differs — informational")
else:
print(f" ✅ {name}: wrapper infisical path OK")
else:
print(f" ✅ {name}: wrapper infisical path OK")
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
_fail(f"wrapper-no-infisical:{name}", name)
else:
print(f" ✅ {name}: wrapper infisical resolves via PATH ({inf_actual})")
else:
print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
@@ -614,14 +645,16 @@ def main():
# Provenance: a report is only actionable if the reader can tell WHICH copy
# of this script produced it. Emit the absolute execution path (script + cwd)
# in both human and --json output so a stale-consumer report is
# distinguishable from a real fault at read time.
# in human, cron-alert, and --json output so a stale-consumer report is
# distinguishable from a real fault at read time. The production cron path
# runs --quiet, so provenance must NOT be behind the quiet guard.
script_path = os.path.abspath(__file__)
cwd = os.getcwd()
if not quiet:
print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print(f"📍 executed from: script={script_path} cwd={cwd}")
print(f"📍 executed from: script={script_path} cwd={cwd}")
if not quiet:
print()
load_agent_keys()
@@ -656,14 +689,15 @@ def main():
if FAIL:
print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
if quiet:
print(f"ALERT agent-health:{','.join(FAIL)}")
print(f"ALERT agent-health:{','.join(FAIL)} script={script_path} cwd={cwd}")
elif not quiet:
print("\n✅ All checks passed")
if as_json:
print(json.dumps({"timestamp": datetime.now().isoformat(),
"execution_path": script_path, "cwd": cwd,
"failures": FAIL, "healthy": len(FAIL) == 0}))
"failures": FAIL, "report_only": REPORT_ONLY,
"healthy": len(FAIL) == 0}))
sys.exit(1 if FAIL else 0)
scripts/prose-lint.sh
+13 -3
View File
@@ -87,11 +87,21 @@ echo " ✅ IP consistency verified (.19=.122=.123 all reachable)"
# Report provenance — every contract report must state the absolute path it
# executed from, so a stale-consumer report is distinguishable from a real fault
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
if [ -n "$PROV_FILES" ]; then
# Enforced only inside the **Report format** paragraph, and a check-health
# contract with no Report format paragraph FAILs rather than being skipped.
PROV_FILES=$(grep -rlE '^### check-health|\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
if [ -z "$PROV_FILES" ]; then
echo " ❌ No check-health/report-format contracts found — provenance not enforced"
FAILED=1
else
PROV_BAD=0
while IFS= read -r f; do
if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then
[ -n "$f" ] || continue
REPORT_PARA=$(awk '/\*\*Report format\*\*/{found=1} found{print} found && /^[[:space:]]*$/{exit}' "$f")
if [ -z "$REPORT_PARA" ]; then
echo " ❌ $f: check-health contract has no **Report format** paragraph"
PROV_BAD=1
elif ! printf '%s\n' "$REPORT_PARA" | grep -qE 'absolute path|pwd -P|executed from'; then
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
PROV_BAD=1
fi