The Liveness Check section now documents ALL SIX verdict shapes exactly
as emitted by proxmox-monitor.sh:
1. ✅ PBS GC: healthy (last run Nh ago, pending-bytes: N B)
2. 🔴 PBS GC: stale (last run Nh ago, pending-bytes: N B)
3. 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000)
4. 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON)
5. 🔴 PBS GC: never-run (storepve-datastore not found in GC list)
6. 🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime)
Fixed the quoted healthy example (line ~114) to include the pending-bytes
suffix the code now appends. Previously the prose only documented the
probe-failed shape, missing the PR's own headline cases (never-run).
Proof: grep -n 'never-run' proxmox-monitor.prose.md now returns two lines
(lines 106 and 108), documenting both never-run variants.
Branch: fix/pbs-gc-liveness-signal-20260919
Document the PBS GC schedule (00:00 UTC, not 20:00 UTC as PR #116 said),
what actually runs (pbs-gc.sh -> pct exec 107 -- proxmox-backup-manager),
datastore location (CT 107's /mnt/pbs-backup on /tank/pbs-backup, NOT
/media/easystore2), and the new liveness check (48h threshold, reports
age in hours, explicit healthy line).
Branch: fix/pbs-gc-liveness-signal-20260919
Second probe-drift correction pass after #65/#66/#68. All four legs were stale
consumer expectations, not live faults.
1. scripts/agent-health-check.py (v4)
- abiba declared pi-only runtime (harness purge): Hermes-era gateway, config
and wrapper legs are skipped instead of failing.
- koby declared report_only (captain ruling 2026-08-17, Rule 17): every koby
leg is detected and reported, never counted as a fleet failure or repaired.
- koby's CT 111 mapping corrected to storepve (.6); the old amdpve mapping
made `pct status 111` fail and read as ct-unreachable.
- wrapper check no longer FAILs .env-based wrappers that legitimately never
invoke infisical (koonimo).
- keys load in main() (load_agent_keys) so the module is importable/testable.
- every run prints absolute execution provenance (script + cwd), in the header
and in --json.
Before: 6 FAILURE(S). After: 0 failures, koby reported read-only.
2. infrastructure-monitoring.prose.md
- PVE API probe repointed from CT 116 (no pveproxy, 000) to the five real
nodes on https://<node>:8006/api2/json/version, all 401 = alive.
- any-HTTP-response liveness rule added (401/3xx alive; 000/timeout = DOWN).
- LiteLLM health documented as 301 -> /litellm/health/liveliness, not bare 200.
3. gpu-monitor.prose.md
- GPU health probes on :8080 (or router /health/unified); bare port 80 on a
GPU host is forbidden (no listener -> false DEGRADED).
- router /health/unified 301 -> /gpu/gpu-data documented as alive.
- port-discipline + liveness rule + direct-fallback execution step.
4. Report provenance (all contracts)
- docs/AUTHORING-GUIDE.md documents the rule; scripts/prose-lint.sh enforces
that any **Report format** contract states an absolute path (pwd -P).
- provenance added to gpu-monitor, infrastructure-monitoring, proxmox-monitor.
Tests: tests/test_probe_drift.py (23 passed); prose-lint.sh clean + shellcheck
clean; CI frontmatter validation passes. Evidence with per-leg before/after and
absolute paths: docs/probe-drift-round2-evidence.md.
Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only), not 0.0.0.0.
They must be probed from .116 via SSH. Prometheus and Grafana remain 0.0.0.0 (LAN-reachable).
Fixes false alarm from remote probe showing connection-refused (by design for localhost binds).
Revert the /grafana/ nginx sub-path I wrongly added — it broke the existing
http://192.168.68.116:3001/d/gpu-fleet URL. Restored direct 0.0.0.0:3001 LAN
access. Contract is now the source of truth: no GF_SERVER_SERVE_FROM_SUB_PATH,
no nginx /grafana/ route. gpu-fleet + 3 new dashboards all at :3001.
root
abiba-bot