fix: probe-drift round 2 — stale monitoring expectations (health check, PVE API, GPU port 80, report provenance)
Second probe-drift correction pass after #65/#66/#68. All four legs were stale
consumer expectations, not live faults.
1. scripts/agent-health-check.py (v4)
- abiba declared pi-only runtime (harness purge): Hermes-era gateway, config
and wrapper legs are skipped instead of failing.
- koby declared report_only (captain ruling 2026-08-17, Rule 17): every koby
leg is detected and reported, never counted as a fleet failure or repaired.
- koby's CT 111 mapping corrected to storepve (.6); the old amdpve mapping
made `pct status 111` fail and read as ct-unreachable.
- wrapper check no longer FAILs .env-based wrappers that legitimately never
invoke infisical (koonimo).
- keys load in main() (load_agent_keys) so the module is importable/testable.
- every run prints absolute execution provenance (script + cwd), in the header
and in --json.
Before: 6 FAILURE(S). After: 0 failures, koby reported read-only.
2. infrastructure-monitoring.prose.md
- PVE API probe repointed from CT 116 (no pveproxy, 000) to the five real
nodes on https://<node>:8006/api2/json/version, all 401 = alive.
- any-HTTP-response liveness rule added (401/3xx alive; 000/timeout = DOWN).
- LiteLLM health documented as 301 -> /litellm/health/liveliness, not bare 200.
3. gpu-monitor.prose.md
- GPU health probes on :8080 (or router /health/unified); bare port 80 on a
GPU host is forbidden (no listener -> false DEGRADED).
- router /health/unified 301 -> /gpu/gpu-data documented as alive.
- port-discipline + liveness rule + direct-fallback execution step.
4. Report provenance (all contracts)
- docs/AUTHORING-GUIDE.md documents the rule; scripts/prose-lint.sh enforces
that any **Report format** contract states an absolute path (pwd -P).
- provenance added to gpu-monitor, infrastructure-monitoring, proxmox-monitor.
Tests: tests/test_probe_drift.py (23 passed); prose-lint.sh clean + shellcheck
clean; CI frontmatter validation passes. Evidence with per-leg before/after and
absolute paths: docs/probe-drift-round2-evidence.md.
This commit is contained in:
+21
-2
@@ -57,7 +57,7 @@ echo "── 2. Regression detection ──"
|
||||
|
||||
# Grafana /grafana/ as nginx route or URL path (reverted 2026-07-02)
|
||||
# EXCLUDE: filesystem paths (/opt/monitoring/grafana/...), directory creation, revert docs
|
||||
GRAFANA_HITS=$(grep -rn '/grafana/' *.prose.md 2>/dev/null \
|
||||
GRAFANA_HITS=$(grep -rn '/grafana/' ./*.prose.md 2>/dev/null \
|
||||
| grep -v '/opt/monitoring/grafana/' \
|
||||
| grep -v 'was tried and reverted\|was reverted\|do not re-add\|NOT recommended' \
|
||||
| grep -v 'mkdir.*grafana\|Create.*grafana' \
|
||||
@@ -71,7 +71,7 @@ else
|
||||
fi
|
||||
|
||||
# Stale CT IDs (CT 122, CT 123 as CT IDs — not IPs .122, .123)
|
||||
CT_STALE=$(grep -rn '\bCT 122\b' *.prose.md 2>/dev/null || true)
|
||||
CT_STALE=$(grep -rn '\bCT 122\b' ./*.prose.md 2>/dev/null || true)
|
||||
if [ -n "$CT_STALE" ]; then
|
||||
echo " ❌ REGRESSION: CT 122 used as CT ID — Tanko is CT 112"
|
||||
echo "$CT_STALE"
|
||||
@@ -84,6 +84,25 @@ fi
|
||||
# .122/.123 are correct — verified reachable bridge IPs for Tanko/Mumuni
|
||||
echo " ✅ IP consistency verified (.19=.122=.123 all reachable)"
|
||||
|
||||
# Report provenance — every contract report must state the absolute path it
|
||||
# executed from, so a stale-consumer report is distinguishable from a real fault
|
||||
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
|
||||
PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
|
||||
if [ -n "$PROV_FILES" ]; then
|
||||
PROV_BAD=0
|
||||
while IFS= read -r f; do
|
||||
if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then
|
||||
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
|
||||
PROV_BAD=1
|
||||
fi
|
||||
done <<< "$PROV_FILES"
|
||||
if [ "$PROV_BAD" -eq 1 ]; then
|
||||
FAILED=1
|
||||
else
|
||||
echo " ✅ Report provenance present in all report-format contracts"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 3. Cross-contract consistency ──
|
||||
echo ""
|
||||
echo "── 3. Cross-contract consistency ──"
|
||||
|
||||
Reference in New Issue
Block a user