fix(monitoring): probe-drift round 2 — gpu port, PVE liveness, report-only visibility, report provenance #70

2026-09-10T02:08:28Z

abiba-bot commented

2026-09-10 02:08:28 +00:00

Second probe-drift correction pass on monitoring contracts (backlog probe-drift-round2-20260909). no-mistakes validated (24 tests, prose-lint + shellcheck green), 5 fix rounds applied through the gate.

Four legs:

agent-health-check — 6 stale "failures" → 0 on healthy legs: abiba treated as pi-only (no Hermes wrapper expectation), koby as report-only (detect-and-report, per captain ruling), CT111 storepve node correction, .env-wrapper path; new machine-readable report_only array in --json so report-only findings are visible without counting as fleet failures.
infrastructure-monitoring — PVE API repointed to the 5 real nodes with the established alive-by-design rule; any-HTTP scoped to the auth-gated/redirect endpoints so a 401/403/500 on a bare-200 probe still alerts (regression caught and fixed in review).
gpu-monitor — probes :8080 / router /health/unified instead of bare port 80 (the recurring false DEGRADED); liveness rule scoped consistently.
Report provenance — every contract report carries its absolute execution path, lint-enforced (scoped to the Report-format paragraph), so a stale-consumer report is distinguishable from a real fault; provenance kept on the failure ALERT line under --quiet (documented quiet contract preserved).

Evidence: docs/probe-drift-round2-evidence.md with per-leg before/after.

Second probe-drift correction pass on monitoring contracts (backlog probe-drift-round2-20260909). no-mistakes validated (24 tests, prose-lint + shellcheck green), 5 fix rounds applied through the gate. Four legs: 1. **agent-health-check** — 6 stale "failures" → 0 on healthy legs: abiba treated as pi-only (no Hermes wrapper expectation), koby as report-only (detect-and-report, per captain ruling), CT111 storepve node correction, .env-wrapper path; new machine-readable `report_only` array in --json so report-only findings are visible without counting as fleet failures. 2. **infrastructure-monitoring** — PVE API repointed to the 5 real nodes with the established alive-by-design rule; any-HTTP scoped to the auth-gated/redirect endpoints so a 401/403/500 on a bare-200 probe still alerts (regression caught and fixed in review). 3. **gpu-monitor** — probes :8080 / router /health/unified instead of bare port 80 (the recurring false DEGRADED); liveness rule scoped consistently. 4. **Report provenance** — every contract report carries its absolute execution path, lint-enforced (scoped to the Report-format paragraph), so a stale-consumer report is distinguishable from a real fault; provenance kept on the failure ALERT line under --quiet (documented quiet contract preserved). Evidence: docs/probe-drift-round2-evidence.md with per-leg before/after.

abiba-bot added 7 commits 2026-09-10 02:08:29 +00:00

fix: probe-drift round 2 — stale monitoring expectations (health check, PVE API, GPU port 80, report provenance) c66d9c1e20

Second probe-drift correction pass after #65/#66/#68. All four legs were stale
consumer expectations, not live faults.

1. scripts/agent-health-check.py (v4)
   - abiba declared pi-only runtime (harness purge): Hermes-era gateway, config
     and wrapper legs are skipped instead of failing.
   - koby declared report_only (captain ruling 2026-08-17, Rule 17): every koby
     leg is detected and reported, never counted as a fleet failure or repaired.
   - koby's CT 111 mapping corrected to storepve (.6); the old amdpve mapping
     made `pct status 111` fail and read as ct-unreachable.
   - wrapper check no longer FAILs .env-based wrappers that legitimately never
     invoke infisical (koonimo).
   - keys load in main() (load_agent_keys) so the module is importable/testable.
   - every run prints absolute execution provenance (script + cwd), in the header
     and in --json.
   Before: 6 FAILURE(S). After: 0 failures, koby reported read-only.

2. infrastructure-monitoring.prose.md
   - PVE API probe repointed from CT 116 (no pveproxy, 000) to the five real
     nodes on https://<node>:8006/api2/json/version, all 401 = alive.
   - any-HTTP-response liveness rule added (401/3xx alive; 000/timeout = DOWN).
   - LiteLLM health documented as 301 -> /litellm/health/liveliness, not bare 200.

3. gpu-monitor.prose.md
   - GPU health probes on :8080 (or router /health/unified); bare port 80 on a
     GPU host is forbidden (no listener -> false DEGRADED).
   - router /health/unified 301 -> /gpu/gpu-data documented as alive.
   - port-discipline + liveness rule + direct-fallback execution step.

4. Report provenance (all contracts)
   - docs/AUTHORING-GUIDE.md documents the rule; scripts/prose-lint.sh enforces
     that any **Report format** contract states an absolute path (pwd -P).
   - provenance added to gpu-monitor, infrastructure-monitoring, proxmox-monitor.

Tests: tests/test_probe_drift.py (23 passed); prose-lint.sh clean + shellcheck
clean; CI frontmatter validation passes. Evidence with per-leg before/after and
absolute paths: docs/probe-drift-round2-evidence.md.

no-mistakes(review): Harden health-check provenance, infisical verification, report-only JSON, tests 194e256ac5

no-mistakes(review): Ignore commented infisical paths; normalize probe-model tests c59c9fb174

no-mistakes(review): Gate wrapper checks on executed infisical; scope liveness guide 1974959cc9

no-mistakes(review): Restore quiet-mode silence; keep provenance on failure alert bbdf6c1249

no-mistakes(document): Refresh stale health-check version, provenance, and lint evidence 88f27e75ed

no-mistakes(document): Scope gpu-monitor liveness rule to bare-200 probes

PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s

Details

PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s

Details

PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s

Details

PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s

Details

PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s

Details

19ed186d0a

abiba-bot merged commit b1462f3e79 into master

2026-09-10 02:17:01 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#70