SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix: probe-drift round 2 — stale monitoring expectations (health check, PVE API, GPU port 80, report provenance)

Browse Source 
Second probe-drift correction pass after #65/#66/#68. All four legs were stale
consumer expectations, not live faults.

1. scripts/agent-health-check.py (v4)
   - abiba declared pi-only runtime (harness purge): Hermes-era gateway, config
     and wrapper legs are skipped instead of failing.
   - koby declared report_only (captain ruling 2026-08-17, Rule 17): every koby
     leg is detected and reported, never counted as a fleet failure or repaired.
   - koby's CT 111 mapping corrected to storepve (.6); the old amdpve mapping
     made `pct status 111` fail and read as ct-unreachable.
   - wrapper check no longer FAILs .env-based wrappers that legitimately never
     invoke infisical (koonimo).
   - keys load in main() (load_agent_keys) so the module is importable/testable.
   - every run prints absolute execution provenance (script + cwd), in the header
     and in --json.
   Before: 6 FAILURE(S). After: 0 failures, koby reported read-only.

2. infrastructure-monitoring.prose.md
   - PVE API probe repointed from CT 116 (no pveproxy, 000) to the five real
     nodes on https://<node>:8006/api2/json/version, all 401 = alive.
   - any-HTTP-response liveness rule added (401/3xx alive; 000/timeout = DOWN).
   - LiteLLM health documented as 301 -> /litellm/health/liveliness, not bare 200.

3. gpu-monitor.prose.md
   - GPU health probes on :8080 (or router /health/unified); bare port 80 on a
     GPU host is forbidden (no listener -> false DEGRADED).
   - router /health/unified 301 -> /gpu/gpu-data documented as alive.
   - port-discipline + liveness rule + direct-fallback execution step.

4. Report provenance (all contracts)
   - docs/AUTHORING-GUIDE.md documents the rule; scripts/prose-lint.sh enforces
     that any **Report format** contract states an absolute path (pwd -P).
   - provenance added to gpu-monitor, infrastructure-monitoring, proxmox-monitor.

Tests: tests/test_probe_drift.py (23 passed); prose-lint.sh clean + shellcheck
clean; CI frontmatter validation passes. Evidence with per-leg before/after and
absolute paths: docs/probe-drift-round2-evidence.md.
This commit is contained in:
abiba-bot
2026-09-10 01:23:54 +00:00
parent 532250b017
commit c66d9c1e20
8 changed files with 689 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/agent-health-check.py
+107 -39
View File
@@ -26,6 +26,17 @@ Changelog:
Fixed UnboundLocalError on the abiba/koonimo gateway leg (pid unbound in the
summary f-string). Abiba's LiteLLM key now comes from /root/.pi/agent/env.sh
(#735 agent separation; creds moved out of shared /root/.bashrc).
v4 (2026-09-10): probe-drift round 2 (prose-contracts follow-up to #65/#66/#68).
abiba declared pi-only runtime — Hermes-era config/wrapper/gateway checks are
skipped (harness purge). koby declared report_only per the captain's
2026-08-17 ruling: every koby leg is detected and reported, never counted as a
fleet failure and never repaired. koby's PVE mapping corrected to storepve
(CT 111 tdunna lives on .6 — the old amdpve mapping produced a false
ct-unreachable). The wrapper infisical-path check no longer FAILs .env-based
wrappers that legitimately never invoke infisical (koonimo/baggy). Every run
now prints absolute execution provenance (script + cwd) in the header and in
--json output so a stale-consumer report is distinguishable from a fault at
read time.
"""
import subprocess, json, sys, os, time
@@ -49,10 +60,18 @@ AGENTS = {
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
# abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
# local env file (key_env below), not from the shared vault or .bashrc.
# runtime=pi: abiba has run pi-only since the harness purge. There is no
# Hermes gateway, no ~/.hermes/config.yaml and no hermes CLI wrapper on .24
# (the /root/.local/bin/hermes symlink is dangling), so the Hermes-era
# config/wrapper/gateway legs are skipped rather than reported as faults.
"abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "minipve",
"vault_key": None,
"vault_key": None, "runtime": "pi",
"key_env": {"file": "/root/.pi/agent/env.sh", "var": "LITELLM_API_KEY"}},
"koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "amdpve", "vault_key": "KOBY_LITELLM_API_KEY"},
# koby = report-only (captain's 2026-08-17 ruling, Rule 17): detect and
# report, NEVER repair, and never count against fleet failures. CT 111
# (tdunna) lives on storepve (.6) — verified live 2026-09-10; the previous
# amdpve mapping made `pct status 111` fail and read as ct-unreachable.
"koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "storepve", "vault_key": "KOBY_LITELLM_API_KEY", "report_only": True},
"koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root", "pve": "amdpve", "vault_key": "KOONIMO_LITELLM_API_KEY"},
}
@@ -70,6 +89,22 @@ GPU_HOSTS = {
FAIL = []
def _fail(key, agent_name=None):
"""Record a failure, except for report-only agents.
Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs
are detected and reported, never repaired and never counted as fleet
failures. A red fleet alert on a known report-only leg is a false alarm. Any
non-report-only agent (or a leg with no agent, e.g. GPU hosts) records
normally.
"""
if agent_name and AGENTS.get(agent_name, {}).get("report_only"):
print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired")
return
FAIL.append(key)
INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
@@ -203,17 +238,21 @@ def _read_env_export(path, var):
return None
# Inject keys for each agent:
# - vault-backed agents (tanko/koby/koonimo): {NAME}_LITELLM_API_KEY from
# Infisical (project 322fceab-39da-4854-a55a-568e76c0f13f, env prod).
# - abiba (pi agent, no vault key): LITELLM_API_KEY from its local env file
# /root/.pi/agent/env.sh (moved there from /root/.bashrc in #735).
for agent_name in AGENTS:
info = AGENTS[agent_name]
key = _get_agent_key(agent_name, info.get("vault_key"))
if not key and info.get("key_env"):
key = _read_env_export(info["key_env"]["file"], info["key_env"]["var"])
AGENTS[agent_name]["key"] = key
def load_agent_keys():
"""Populate AGENTS[*]["key"] from the vault or the agent's local env file.
Called from main(), not at import: keeping this out of module scope lets the
module be imported (and unit tested) without live vault/SSH access. Vault
format is {NAME}_LITELLM_API_KEY (project 322fceab-39da-4854-a55a-568e76c0f13f,
env prod); abiba has no vault key and reads LITELLM_API_KEY from its local
/root/.pi/agent/env.sh (moved there from /root/.bashrc in #735).
"""
for agent_name in AGENTS:
info = AGENTS[agent_name]
key = _get_agent_key(agent_name, info.get("vault_key"))
if not key and info.get("key_env"):
key = _read_env_export(info["key_env"]["file"], info["key_env"]["var"])
AGENTS[agent_name]["key"] = key
# ═══════════════════════════════════════════════════════════════════
@@ -225,7 +264,7 @@ def check_keys():
key = agent.get("key")
if not key:
print(f" ❌ {name}: NO KEY FOUND (vault/env empty or unreachable)")
FAIL.append(f"key:{name}:no-key")
_fail(f"key:{name}:no-key", name)
continue
data = http_json(f"{LITELLM}/v1/models",
headers={"Authorization": f"Bearer {key}"})
@@ -234,7 +273,7 @@ def check_keys():
print(f" ✅ {name}: key valid → {model}")
else:
print(f" ❌ {name}: KEY FAILURE — auth rejected or unreachable")
FAIL.append(f"key:{name}")
_fail(f"key:{name}", name)
# ═══════════════════════════════════════════════════════════════════
@@ -294,12 +333,17 @@ def check_agents():
# Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
# Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
if agent.get("runtime") == "dsh":
# Non-Hermes runtimes have no gateway to probe. dsh = Tanko since
# 2026-08-27; pi = abiba since the harness purge (.24 is pi-only).
if agent.get("runtime") in ("dsh", "pi"):
is_dsh = agent.get("runtime") == "dsh"
label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
since = "since 2026-08-27" if is_dsh else "since the harness purge"
live = ssh(host, "true", user=user)
print(f" {'✅' if live is not None else '❌'} {name}: DSH (DeepSeek Harness) — "
f"no Hermes gateway since 2026-08-27 (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
print(f" {'✅' if live is not None else '❌'} {name}: {label} — "
f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
if live is None:
FAIL.append(f"unreachable:{name}")
_fail(f"unreachable:{name}", name)
continue
if not host or not user:
@@ -323,7 +367,7 @@ def check_agents():
# Still check gateway status for reporting purposes
if pid == "?":
print(f" ⚠️ {name}: GATEWAY NOT RUNNING (reported only)")
FAIL.append(f"gateway-down:{name}")
_fail(f"gateway-down:{name}", name)
continue
else:
print(f" ✅ {name}: gateway running (pid={pid}, report-only mode)")
@@ -386,12 +430,12 @@ def check_ct_liveness():
status = ssh(pve_ip, f"pct status {ct} 2>/dev/null", user="root")
if not status:
print(f" ❌ {name} (CT {ct} on {pve_node}): PVE UNREACHABLE")
FAIL.append(f"ct-unreachable:{name}:{pve_ip}")
_fail(f"ct-unreachable:{name}:{pve_ip}", name)
elif "running" in status:
print(f" ✅ {name} (CT {ct} on {pve_node}): running")
elif "stopped" in status:
print(f" ❌ {name} (CT {ct} on {pve_node}): STOPPED")
FAIL.append(f"ct-stopped:{name}")
_fail(f"ct-stopped:{name}", name)
else:
print(f" ⚠️ {name} (CT {ct} on {pve_node}): {status.strip()}")
@@ -407,6 +451,9 @@ def check_config_integrity():
if agent.get("runtime") == "dsh":
print(f" ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
continue
if agent.get("runtime") == "pi":
print(f" ⏭️ {name}: pi-only runtime — no Hermes config.yaml since the harness purge")
continue
host = agent.get("host")
user = agent.get("user")
if not host or not user:
@@ -421,12 +468,12 @@ def check_config_integrity():
user=user)
if not yaml_ok:
print(f" ❌ {name}: SSH UNREACHABLE (config check skipped)")
FAIL.append(f"config-unreachable:{name}")
_fail(f"config-unreachable:{name}", name)
elif "OK" in yaml_ok:
print(f" ✅ {name}: config.yaml valid YAML")
else:
print(f" ❌ {name}: config.yaml YAML ERROR — {yaml_ok[:120]}")
FAIL.append(f"config-yaml-error:{name}")
_fail(f"config-yaml-error:{name}", name)
# ═══════════════════════════════════════════════════════════════════
@@ -440,6 +487,9 @@ def check_wrapper_integrity():
if agent.get("runtime") == "dsh":
print(f" ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
continue
if agent.get("runtime") == "pi":
print(f" ⏭️ {name}: pi-only runtime — no hermes CLI wrapper since the harness purge")
continue
host = agent.get("host")
user = agent.get("user")
if not host or not user:
@@ -453,24 +503,31 @@ def check_wrapper_integrity():
wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
if not wrapper:
print(f" ❌ {name}: NO HERMES CLI WRAPPER FOUND")
FAIL.append(f"wrapper-missing:{name}")
_fail(f"wrapper-missing:{name}", name)
continue
else:
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
# Check wrapper has correct infisical path
infisical_path_valid = ssh(host,
"head -20 /root/.local/bin/hermes 2>/dev/null | grep -q '/usr/bin/infisical' && echo OK || echo MISS",
user=user)
if infisical_path_valid == "MISS":
# Check if infisical exists on path
# Credential-injection mechanism. The Hermes-era wrapper injected creds
# with `/usr/bin/infisical run`; newer wrappers source the agent key from
# ~/.hermes/.env and never mention infisical (koonimo/baggy does exactly
# this). A wrapper with NO infisical reference is therefore a valid
# alternate mechanism, not a fault — only flag a wrapper that DOES call
# infisical when the binary cannot resolve. The old check FAILed every
# .env-based wrapper as "infisical path may be wrong": a stale
# expectation, not a fault.
wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
if "infisical" in wrapper_body:
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f" ❌ {name}: INFISICAL NOT INSTALLED (wrapper broken)")
FAIL.append(f"wrapper-no-infisical:{name}")
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
_fail(f"wrapper-no-infisical:{name}", name)
elif "/usr/bin/infisical" not in wrapper_body:
print(f" ⚠️ {name}: wrapper infisical path differs (infisical at {inf_actual}) — informational")
else:
print(f" ⚠️ {name}: wrapper infisical path may be wrong (infisical at {inf_actual})")
FAIL.append(f"wrapper-infisical-path:{name}")
print(f" ✅ {name}: wrapper infisical path OK")
else:
print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
# Check hermes-real exists
hermes_real = ssh(host,
@@ -483,7 +540,7 @@ def check_wrapper_integrity():
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
print(f" ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
FAIL.append(f"wrapper-no-hermes-real:{name}")
_fail(f"wrapper-no-hermes-real:{name}", name)
else:
print(f" ✅ {name}: hermes-real at alt path")
@@ -511,10 +568,10 @@ def check_vault_secrets():
key = agent.get("key")
if not key:
print(f" ❌ {name}: vault secret {vault_key_name} MISSING or EMPTY")
FAIL.append(f"vault-empty:{name}:{vault_key_name}")
_fail(f"vault-empty:{name}:{vault_key_name}", name)
elif not key.startswith("sk-"):
print(f" ❌ {name}: vault secret {vault_key_name} WRONG FORMAT (starts '{key[:8]}...')")
FAIL.append(f"vault-bad-format:{name}:{vault_key_name}")
_fail(f"vault-bad-format:{name}:{vault_key_name}", name)
else:
print(f" ✅ {name}: vault {vault_key_name}=sk-...{key[-4:]}")
@@ -555,10 +612,20 @@ def main():
if not quiet and "--no-deploy" not in sys.argv:
deploy_self()
# Provenance: a report is only actionable if the reader can tell WHICH copy
# of this script produced it. Emit the absolute execution path (script + cwd)
# in both human and --json output so a stale-consumer report is
# distinguishable from a real fault at read time.
script_path = os.path.abspath(__file__)
cwd = os.getcwd()
if not quiet:
print(f"🏥 Agent Health Check v2 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print(f"📍 executed from: script={script_path} cwd={cwd}")
print()
load_agent_keys()
print("🔑 LiteLLM Keys:")
check_keys()
print()
@@ -595,6 +662,7 @@ def main():
if as_json:
print(json.dumps({"timestamp": datetime.now().isoformat(),
"execution_path": script_path, "cwd": cwd,
"failures": FAIL, "healthy": len(FAIL) == 0}))
sys.exit(1 if FAIL else 0)
scripts/prose-lint.sh
+21 -2
View File
@@ -57,7 +57,7 @@ echo "── 2. Regression detection ──"
# Grafana /grafana/ as nginx route or URL path (reverted 2026-07-02)
# EXCLUDE: filesystem paths (/opt/monitoring/grafana/...), directory creation, revert docs
GRAFANA_HITS=$(grep -rn '/grafana/' *.prose.md 2>/dev/null \
GRAFANA_HITS=$(grep -rn '/grafana/' ./*.prose.md 2>/dev/null \
| grep -v '/opt/monitoring/grafana/' \
| grep -v 'was tried and reverted\|was reverted\|do not re-add\|NOT recommended' \
| grep -v 'mkdir.*grafana\|Create.*grafana' \
@@ -71,7 +71,7 @@ else
fi
# Stale CT IDs (CT 122, CT 123 as CT IDs — not IPs .122, .123)
CT_STALE=$(grep -rn '\bCT 122\b' *.prose.md 2>/dev/null || true)
CT_STALE=$(grep -rn '\bCT 122\b' ./*.prose.md 2>/dev/null || true)
if [ -n "$CT_STALE" ]; then
echo " ❌ REGRESSION: CT 122 used as CT ID — Tanko is CT 112"
echo "$CT_STALE"
@@ -84,6 +84,25 @@ fi
# .122/.123 are correct — verified reachable bridge IPs for Tanko/Mumuni
echo " ✅ IP consistency verified (.19=.122=.123 all reachable)"
# Report provenance — every contract report must state the absolute path it
# executed from, so a stale-consumer report is distinguishable from a real fault
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
if [ -n "$PROV_FILES" ]; then
PROV_BAD=0
while IFS= read -r f; do
if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
PROV_BAD=1
fi
done <<< "$PROV_FILES"
if [ "$PROV_BAD" -eq 1 ]; then
FAILED=1
else
echo " ✅ Report provenance present in all report-format contracts"
fi
fi
# ── 3. Cross-contract consistency ──
echo ""
echo "── 3. Cross-contract consistency ──"