fix: probe-drift round 2 — stale monitoring expectations (health check, PVE API, GPU port 80, report provenance)
Second probe-drift correction pass after #65/#66/#68. All four legs were stale
consumer expectations, not live faults.
1. scripts/agent-health-check.py (v4)
- abiba declared pi-only runtime (harness purge): Hermes-era gateway, config
and wrapper legs are skipped instead of failing.
- koby declared report_only (captain ruling 2026-08-17, Rule 17): every koby
leg is detected and reported, never counted as a fleet failure or repaired.
- koby's CT 111 mapping corrected to storepve (.6); the old amdpve mapping
made `pct status 111` fail and read as ct-unreachable.
- wrapper check no longer FAILs .env-based wrappers that legitimately never
invoke infisical (koonimo).
- keys load in main() (load_agent_keys) so the module is importable/testable.
- every run prints absolute execution provenance (script + cwd), in the header
and in --json.
Before: 6 FAILURE(S). After: 0 failures, koby reported read-only.
2. infrastructure-monitoring.prose.md
- PVE API probe repointed from CT 116 (no pveproxy, 000) to the five real
nodes on https://<node>:8006/api2/json/version, all 401 = alive.
- any-HTTP-response liveness rule added (401/3xx alive; 000/timeout = DOWN).
- LiteLLM health documented as 301 -> /litellm/health/liveliness, not bare 200.
3. gpu-monitor.prose.md
- GPU health probes on :8080 (or router /health/unified); bare port 80 on a
GPU host is forbidden (no listener -> false DEGRADED).
- router /health/unified 301 -> /gpu/gpu-data documented as alive.
- port-discipline + liveness rule + direct-fallback execution step.
4. Report provenance (all contracts)
- docs/AUTHORING-GUIDE.md documents the rule; scripts/prose-lint.sh enforces
that any **Report format** contract states an absolute path (pwd -P).
- provenance added to gpu-monitor, infrastructure-monitoring, proxmox-monitor.
Tests: tests/test_probe_drift.py (23 passed); prose-lint.sh clean + shellcheck
clean; CI frontmatter validation passes. Evidence with per-leg before/after and
absolute paths: docs/probe-drift-round2-evidence.md.
This commit is contained in:
@@ -211,6 +211,31 @@ Errors tell the operator what went wrong and what to do about it. Be specific:
|
||||
Check router /health/unified at http://192.168.68.116/health/unified instead."
|
||||
```
|
||||
|
||||
### Report provenance
|
||||
|
||||
Every report a contract produces must lead with the **absolute path the probe
|
||||
executed from** — `pwd -P`, or the running script's absolute path. A live-state
|
||||
report without provenance is unactionable: a report from a stale copy (a worktree
|
||||
clone, a retired cron entry, a diverged consumer) looks identical to a live
|
||||
fault, and the team burns rounds repairing healthy infrastructure. This is not
|
||||
optional. The 2026-09-09 probe-drift rounds cost three false `DEGRADED` reports
|
||||
because a stale consumer probed the wrong port and nothing in the report said
|
||||
where it ran.
|
||||
|
||||
Pair it with the **any-HTTP-response liveness rule**: a probe is ALIVE on ANY
|
||||
HTTP status — including `301` redirects and `401`/`403` auth challenges. A bare
|
||||
`200` is not required and must never be a pass condition for an auth-gated
|
||||
endpoint. **DOWN = connection refused (`000`) or timeout only.**
|
||||
|
||||
```markdown
|
||||
**Report format**: Begin every report with the absolute execution path
|
||||
(`pwd -P` / script path). Alive = ANY HTTP status; DOWN = `000`/timeout only.
|
||||
```
|
||||
|
||||
The lint pipeline enforces the provenance clause: any contract with a
|
||||
`**Report format**` line must state an absolute path (`pwd -P`, `absolute path`,
|
||||
or `executed from`).
|
||||
|
||||
### Comments
|
||||
|
||||
Comments in contracts explain WHY, not WHAT. The execution steps say what to
|
||||
|
||||
@@ -0,0 +1,278 @@
|
||||
# Probe-drift round 2 — per-leg before/after evidence
|
||||
|
||||
**Date:** 2026-09-10
|
||||
**Worktree (absolute execution path):** `/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts`
|
||||
**Branch:** `fm/probe-drift-round2-20260909`
|
||||
|
||||
Every command below was run from the absolute path above; output is pasted
|
||||
verbatim. This is the evidence trail for the four scoped corrections; it is not
|
||||
a contract (never `prose run` it).
|
||||
|
||||
---
|
||||
|
||||
## Leg 1 — agent-health-check (item 1)
|
||||
|
||||
**Before** — from `/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts`,
|
||||
`python3 scripts/agent-health-check.py --no-deploy` (v2, base of this branch):
|
||||
|
||||
```
|
||||
🏥 Agent Health Check v2 — 2026-09-10 01:16 UTC
|
||||
|
||||
🔑 LiteLLM Keys:
|
||||
✅ tanko: key valid → syslog-auto
|
||||
✅ abiba: key valid → syslog-auto
|
||||
✅ koby: key valid → syslog-auto
|
||||
✅ koonimo: key valid → syslog-auto
|
||||
|
||||
🎮 GPU Port Health:
|
||||
✅ gpu-rtx3090 (.8): healthy (pid=472206)
|
||||
✅ gpu-rtx5070 (.110): healthy (pid=207601)
|
||||
✅ gpu-strixhalo (.15): healthy (pid=4098872)
|
||||
|
||||
🤖 Agent Gateways:
|
||||
✅ tanko: DSH (DeepSeek Harness) — no Hermes gateway since 2026-08-27 (CT 112, SSH OK)
|
||||
⚠️ abiba: gw=no-state-file zulip=? streaming=no errors_10m=0 pid=?
|
||||
✅ koby: gw=running zulip=connected streaming=no errors_10m=0 pid=360900
|
||||
✅ koonimo: gw=running zulip=connected streaming=no errors_10m=0 pid=155125
|
||||
|
||||
🖥️ CT Liveness:
|
||||
✅ tanko (CT 112 on amdpve): running
|
||||
✅ abiba (CT 100 on minipve): running
|
||||
❌ koby (CT 111 on amdpve): PVE UNREACHABLE
|
||||
✅ koonimo (CT 113 on amdpve): running
|
||||
|
||||
📝 Config Integrity:
|
||||
⏭️ tanko: DSH — no Hermes config.yaml since 2026-08-27
|
||||
✅ abiba: config.yaml valid YAML
|
||||
✅ koby: config.yaml valid YAML
|
||||
✅ koonimo: config.yaml valid YAML
|
||||
|
||||
🔌 Wrapper/CLI Integrity:
|
||||
⏭️ tanko: DSH — no hermes CLI wrapper since 2026-08-27
|
||||
⚠️ abiba: wrapper infisical path may be wrong (infisical at /usr/bin/infisical)
|
||||
❌ abiba: hermes-real NOT FOUND (wrapper broken)
|
||||
⚠️ abiba: .env may be missing LITELLM_API_KEY entry
|
||||
⚠️ koby: wrapper infisical path may be wrong (infisical at /usr/bin/infisical)
|
||||
❌ koby: hermes-real NOT FOUND (wrapper broken)
|
||||
✅ koby: wrapper + .env key present
|
||||
⚠️ koonimo: wrapper infisical path may be wrong (infisical at /usr/bin/infisical)
|
||||
✅ koonimo: wrapper + .env key present
|
||||
|
||||
🔐 Vault Secrets:
|
||||
✅ tanko: vault TANKO_LITELLM_API_KEY=sk-...x6uw
|
||||
✅ koby: vault KOBY_LITELLM_API_KEY=sk-...jxlg
|
||||
✅ koonimo: vault KOONIMO_LITELLM_API_KEY=sk-...Y0KQ
|
||||
|
||||
❌ 6 FAILURE(S): ct-unreachable:koby:192.168.68.15 | wrapper-infisical-path:abiba | wrapper-no-hermes-real:abiba | wrapper-infisical-path:koby | wrapper-no-hermes-real:koby | wrapper-infisical-path:koonimo
|
||||
```
|
||||
|
||||
Root causes (all stale expectations; no live fault):
|
||||
|
||||
| Failure | Why it was stale |
|
||||
|---|---|
|
||||
| `ct-unreachable:koby:192.168.68.15` | CT 111 (tdunna/koby) runs on **storepve (.6)**, not amdpve (.15). |
|
||||
| `wrapper-*:abiba` | Abiba is pi-only since the harness purge. `/root/.local/bin/hermes` is a dangling symlink; no `hermes-real`, no `~/.hermes/.env`. |
|
||||
| `wrapper-*:koby` | Koby is **report-only** (captain ruling 2026-08-17, Rule 17): detect and report, never repair — its legs must not count as fleet failures. |
|
||||
| `wrapper-infisical-path:koonimo` | Koonimo's wrapper injects `KOONIMO_LITELLM_API_KEY` from `~/.hermes/.env` and never invokes infisical. The check required `/usr/bin/infisical`, which is only one valid mechanism. |
|
||||
|
||||
**After** — same absolute path, `python3 scripts/agent-health-check.py --no-deploy` (v4):
|
||||
|
||||
```
|
||||
$ pwd -P
|
||||
/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts
|
||||
$ python3 scripts/agent-health-check.py --no-deploy
|
||||
🏥 Agent Health Check v4 — 2026-09-10 01:22 UTC
|
||||
📍 executed from: script=/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts/scripts/agent-health-check.py cwd=/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts
|
||||
|
||||
🔑 LiteLLM Keys:
|
||||
✅ tanko: key valid → syslog-auto
|
||||
✅ abiba: key valid → syslog-auto
|
||||
✅ koby: key valid → syslog-auto
|
||||
✅ koonimo: key valid → syslog-auto
|
||||
|
||||
🎮 GPU Port Health:
|
||||
✅ gpu-rtx3090 (.8): healthy (pid=472206)
|
||||
✅ gpu-rtx5070 (.110): healthy (pid=207601)
|
||||
✅ gpu-strixhalo (.15): healthy (pid=4098872)
|
||||
|
||||
🤖 Agent Gateways:
|
||||
✅ tanko: DSH (DeepSeek Harness) — no Hermes gateway since 2026-08-27 (CT 112, SSH OK)
|
||||
✅ abiba: pi-only runtime — no Hermes gateway since the harness purge (CT 100, SSH OK)
|
||||
🔍 koby: REPORT-ONLY mode (diagnostic only, no repairs on .129)
|
||||
✅ koby: gateway running (pid=360900, report-only mode)
|
||||
✅ koonimo: gw=running zulip=connected streaming=no errors_10m=0 pid=155125
|
||||
|
||||
🖥️ CT Liveness:
|
||||
✅ tanko (CT 112 on amdpve): running
|
||||
✅ abiba (CT 100 on minipve): running
|
||||
✅ koby (CT 111 on storepve): running
|
||||
✅ koonimo (CT 113 on amdpve): running
|
||||
|
||||
📝 Config Integrity:
|
||||
⏭️ tanko: DSH — no Hermes config.yaml since 2026-08-27
|
||||
⏭️ abiba: pi-only runtime — no Hermes config.yaml since the harness purge
|
||||
✅ koby: config.yaml valid YAML
|
||||
✅ koonimo: config.yaml valid YAML
|
||||
|
||||
🔌 Wrapper/CLI Integrity:
|
||||
⏭️ tanko: DSH — no hermes CLI wrapper since 2026-08-27
|
||||
⏭️ abiba: pi-only runtime — no hermes CLI wrapper since the harness purge
|
||||
ℹ️ koby: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK
|
||||
❌ koby: hermes-real NOT FOUND (wrapper broken)
|
||||
🔍 report-only (koby): wrapper-no-hermes-real:koby — reported, not counted/repaired
|
||||
✅ koby: wrapper + .env key present
|
||||
✅ koonimo: wrapper infisical path OK
|
||||
✅ koonimo: wrapper + .env key present
|
||||
|
||||
🔐 Vault Secrets:
|
||||
✅ tanko: vault TANKO_LITELLM_API_KEY=sk-...x6uw
|
||||
✅ koby: vault KOBY_LITELLM_API_KEY=sk-...jxlg
|
||||
✅ koonimo: vault KOONIMO_LITELLM_API_KEY=sk-...Y0KQ
|
||||
|
||||
✅ All checks passed
|
||||
exit=0
|
||||
```
|
||||
|
||||
**Live vantage proof** (same worktree):
|
||||
|
||||
```
|
||||
$ ssh root@192.168.68.15 "pct status 111"
|
||||
Configuration file 'nodes/amdpve/lxc/111.conf' does not exist
|
||||
$ ssh root@192.168.68.6 "pct status 111; pct list | grep '^ *111'"
|
||||
status: running
|
||||
111 running tdunna
|
||||
$ ssh root@192.168.68.129 "hostname"
|
||||
tdunna
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Leg 2 — infrastructure-monitoring PVE API (item 2)
|
||||
|
||||
**Before** — the contract's probe, aimed at the monitoring host CT 116:
|
||||
|
||||
```
|
||||
$ curl -s -o /dev/null -w '%{http_code}' https://192.168.68.116:8006/api2/json
|
||||
000
|
||||
```
|
||||
|
||||
CT 116 runs no `pveproxy`, so it never answers on `:8006`. The probe target was
|
||||
wrong, which is what read as PVE-API `000`.
|
||||
|
||||
**After** — probing the five real cluster nodes (`:8006/api2/json/version`),
|
||||
alive under the any-HTTP-response rule (`401` = up, unauthenticated):
|
||||
|
||||
```
|
||||
$ for node in 192.168.68.9 192.168.68.5 192.168.68.15 192.168.68.6 192.168.68.12; do
|
||||
printf '%s:8006 -> %s\n' "$node" "$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 "https://$node:8006/api2/json/version")"
|
||||
done
|
||||
192.168.68.9:8006 -> 401
|
||||
192.168.68.5:8006 -> 401
|
||||
192.168.68.15:8006 -> 401
|
||||
192.168.68.6:8006 -> 401
|
||||
192.168.68.12:8006 -> 401
|
||||
```
|
||||
|
||||
`401` on every node = alive by design. `DOWN` is `000`/timeout only. (The
|
||||
contract's LiteLLM probe was the same class: `/litellm/health` answers `301` →
|
||||
`/litellm/health/liveliness`, so it is now specified as any-HTTP too.)
|
||||
|
||||
---
|
||||
|
||||
## Leg 3 — gpu-monitor GPU probes (item 3)
|
||||
|
||||
**Before** — the false alarm came from probing bare port 80 on GPU hosts:
|
||||
|
||||
```
|
||||
$ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.8/health
|
||||
000
|
||||
$ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.110/health
|
||||
000
|
||||
```
|
||||
|
||||
Nothing listens on GPU port 80, so the monitor reported
|
||||
`DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000` three times on 2026-09-09.
|
||||
|
||||
**After** — the real endpoints answer:
|
||||
|
||||
```
|
||||
$ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.8:8080/health
|
||||
200
|
||||
$ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.110:8080/health
|
||||
200
|
||||
$ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.15:8080/health
|
||||
200
|
||||
$ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health/unified
|
||||
301 # Location: http://192.168.68.116/gpu/gpu-data — the same payload
|
||||
$ curl -s -o /dev/null -w '%{http_code}' -L http://192.168.68.116/health/unified
|
||||
200
|
||||
```
|
||||
|
||||
`301` is healthy under the any-HTTP-response rule. The contract now requires GPU
|
||||
health on `:8080` (or router `/health/unified`) and forbids bare port 80 on a
|
||||
GPU host.
|
||||
|
||||
---
|
||||
|
||||
## Leg 4 — report provenance (item 4)
|
||||
|
||||
Every contract report must now lead with the absolute path it executed from.
|
||||
`docs/AUTHORING-GUIDE.md` documents the rule and `scripts/prose-lint.sh`
|
||||
enforces it:
|
||||
|
||||
```
|
||||
$ pwd -P
|
||||
/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts
|
||||
$ bash scripts/prose-lint.sh
|
||||
✅ Report provenance present in all report-format contracts
|
||||
...
|
||||
✅ LINT PASSED (16 warning(s))
|
||||
```
|
||||
|
||||
The health script prints `📍 executed from: script=… cwd=…` and includes
|
||||
`execution_path`/`cwd` in `--json` output.
|
||||
|
||||
---
|
||||
|
||||
## Full suite
|
||||
|
||||
```
|
||||
$ pwd -P
|
||||
/root/.treehouse/prose-contracts-9ce5f3/3/prose-contracts
|
||||
$ python3 -m pytest -q
|
||||
23 passed
|
||||
$ shellcheck scripts/prose-lint.sh
|
||||
(clean)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Follow-up findings (observed, intentionally NOT changed here)
|
||||
|
||||
These are adjacent stale expectations discovered while verifying the four
|
||||
scoped legs. Each touches a CRITICAL/HIGH-sensitivity artifact or an unrelated
|
||||
script, so it is recorded for the captain/verify mate rather than silently
|
||||
repaired.
|
||||
|
||||
1. **`infrastructure-control.prose.md` (CRITICAL) CT 111 node assignment.**
|
||||
Lines ~109 and ~615 place `tdunna` (CT 111, koby) on **amdpve**. Live
|
||||
verification on 2026-09-10 shows `pct status 111` = `running` on
|
||||
**storepve (.6)** and `Configuration file 'nodes/amdpve/lxc/111.conf' does
|
||||
not exist` on .15. `agent-health-check.py` now carries the live-verified
|
||||
`storepve` mapping (the script is not the topology source of truth); the
|
||||
CRITICAL contract itself needs an authorized correction.
|
||||
2. **Strix Halo `:8080` firewall claim is stale.** `prose-ai-review.sh`
|
||||
ground-truth rule #4 and `gpu-monitor.prose.md` say `:8080` is firewalled to
|
||||
`.116` only and `.24` cannot probe it. Live on .15:
|
||||
`-A INPUT -s 192.168.68.24/32 -p tcp --dport 8080 -j ACCEPT`, and a probe
|
||||
from .24 returns `200`. The contract keeps routing Strix via the router
|
||||
(safe), but the claim no longer matches iptables.
|
||||
3. **`contract-registry.yaml` references `agent-health-check.prose.md`**, which
|
||||
does not exist in the repo. The registry entry (with `koby_action: skip_heal`)
|
||||
is aspirational/stale.
|
||||
4. **Pre-existing script defects, untouched:** `scripts/pm2-self-heal.sh` has a
|
||||
bash syntax error at lines 19–20 (`bash -n` fails), and `shellcheck` fails on
|
||||
five untouched scripts (`netbird-add-domain.sh`, `pct-run.sh`,
|
||||
`pm2-self-heal.sh`, `prose-ai-review.sh`, `swap-gpu-dense-model.sh`).
|
||||
`scripts/prose-lint.sh` — the one shell file touched here — is now
|
||||
shellcheck-clean.
|
||||
+69
-20
@@ -31,20 +31,31 @@ agent: abiba
|
||||
└──────┘ │qwen27B│ │LiteLLM │
|
||||
└──────┘ │dashboard│
|
||||
└────────┘
|
||||
```
|
||||
|
||||
Note: JSON sidecar exporters at :8090 were never deployed on any
|
||||
GPU host. Router falls back to GPU /health direct probe. Monitor
|
||||
should use router /health/unified as source of truth for GPU status.
|
||||
Strix Halo :8080 is firewalled to .116 only — monitor on .24 cannot
|
||||
poll .15:8080 directly; must go through router on .116.
|
||||
```
|
||||
|
||||
**PORT RULE (verified 2026-09-10):** GPU per-host health lives on **:8080**
|
||||
(`http://<gpu-host>:8080/health`); Prometheus GPU exporters live on **:9400**.
|
||||
There is NO listener on bare port 80 for any GPU host — `http://192.168.68.8/health`
|
||||
and `http://192.168.68.110/health` answer `000`. Never use a bare-port-80 probe
|
||||
as a GPU liveness signal: on 2026-09-09 that produced three false
|
||||
`DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000` rounds while
|
||||
`http://192.168.68.8:8080/health` and `http://192.168.68.110:8080/health`
|
||||
answered `200`. Port 80 is valid only on the router (.116), never on a GPU host.
|
||||
|
||||
### Subsystems Polled
|
||||
|
||||
| Subsystem | Endpoint | Frequency | Metrics |
|
||||
|-----------|----------|-----------|---------|
|
||||
| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
|
||||
| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
|
||||
| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) — `301` → `/gpu/gpu-data` is **alive** |
|
||||
| GPU .8 (RTX 3090) health | `http://192.168.68.8:8080/health` | 15s | direct liveness fallback — **:8080 ONLY, never bare port 80** |
|
||||
| GPU .110 (RTX 5070) health | `http://192.168.68.110:8080/health` | 15s | direct liveness fallback — **:8080 ONLY, never bare port 80** |
|
||||
| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (`301` → `/gpu/gpu-data` = alive) |
|
||||
| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
|
||||
| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
|
||||
| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | Strix Halo status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
|
||||
@@ -61,6 +72,17 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
|
||||
|
||||
## Alert Thresholds
|
||||
|
||||
### Liveness rule (any-HTTP-response)
|
||||
|
||||
A probe is **ALIVE** if the endpoint returns **ANY** HTTP status — including
|
||||
redirects and auth challenges. A bare `200` is not required. **DOWN = connection
|
||||
refused (`000`) or timeout only.** This is the same rule zulip-health adopted for
|
||||
Tanko (loopback `:3080` + public URL). Applied here: the router's
|
||||
`/health/unified` answers `301 Moved Permanently` → `/gpu/gpu-data` (the same
|
||||
payload), so `301` is healthy and a bare-200 expectation would false-alarm.
|
||||
Statuses outside the expected set on an otherwise-alive endpoint are reported as
|
||||
a warning, never as DOWN.
|
||||
|
||||
| Metric | Warning | Critical |
|
||||
|--------|---------|----------|
|
||||
| GPU Temp | >80°C | >90°C |
|
||||
@@ -101,24 +123,44 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
|
||||
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
|
||||
|
||||
```bash
|
||||
# Provenance — run first; paste the absolute path into the report
|
||||
pwd -P
|
||||
|
||||
# GPU Monitor health
|
||||
curl http://localhost:9100/health | jq
|
||||
# Expected: 200 with {"status": "healthy", "cache_age_seconds": <n>}
|
||||
|
||||
# Router health (via nginx on port 80)
|
||||
# GPU host health — DIRECT on :8080. NEVER probe bare port 80 on a GPU host:
|
||||
# http://192.168.68.8/health has no listener and returns 000 → false DEGRADED.
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.8:8080/health
|
||||
# Expected: 200 (alive = any HTTP status; 000/timeout = DOWN)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.110:8080/health
|
||||
# Expected: 200 (alive = any HTTP status; 000/timeout = DOWN)
|
||||
|
||||
# Router unified health (source of truth; 301 → /gpu/gpu-data is HEALTHY)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health/unified
|
||||
# Expected: 301 (or 200 after following the redirect) — any HTTP status = alive
|
||||
|
||||
# Router basic health (via nginx on port 80 — router .116 only, never a GPU host)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
|
||||
# Expected: 200 (Router is up and responding)
|
||||
# Expected: 200
|
||||
|
||||
# LiteLLM health (via nginx on port 80)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
|
||||
# Expected: 200 (LiteLLM is up and responding)
|
||||
# Expected: 301 → /litellm/health/liveliness (200 after redirect) — any HTTP status = alive
|
||||
|
||||
# Dashboard
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/dashboard/
|
||||
# Expected: 200 (Dashboard is up and responding)
|
||||
# Expected: 200
|
||||
```
|
||||
|
||||
**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
|
||||
**Report format**: Begin every report with the **absolute path the probe executed
|
||||
from** (`pwd -P`, or the monitor script's absolute path) so a stale-consumer
|
||||
report is distinguishable from a real fault at read time. Summarize actual
|
||||
results from each probe. Apply the any-HTTP-response liveness rule above: only
|
||||
connection-refused (`000`) or timeout is DOWN. Flag an alert only when a probe is
|
||||
DOWN, or when an alive endpoint returns an unexpected status. Never probe a GPU
|
||||
host on bare port 80.
|
||||
|
||||
### view-dashboard
|
||||
Open `http://localhost:9100/` in browser — Live HTML dashboard
|
||||
@@ -131,9 +173,11 @@ python3 /root/scripts/gpu-monitor-server.py &
|
||||
Or via PM2: `pm2 restart gpu-monitor`
|
||||
|
||||
### check-router
|
||||
The router health is accessed through nginx on port 80 (NOT port 9000 directly).
|
||||
`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy
|
||||
The router health is accessed through nginx on port 80 on the **router**
|
||||
(.116) — NOT port 9000 directly, and NOT bare port 80 on a GPU host.
|
||||
`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy; answers `301` → `/gpu/gpu-data` (same payload) = alive
|
||||
`curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
|
||||
`curl http://192.168.68.8/health` — ❌ NEVER USE (GPU host, no port-80 listener → false `000`/DEGRADED)
|
||||
|
||||
## Configuration Files
|
||||
|
||||
@@ -145,13 +189,18 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
|
||||
|
||||
## Execution
|
||||
|
||||
1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
|
||||
2. **Poll router** (every 15s): GET .116/health via nginx:80
|
||||
3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
|
||||
4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
|
||||
5. **Poll dashboard** (every 15s): GET .116/dashboard/
|
||||
6. **Check alerts**: Compare metrics against thresholds
|
||||
7. **Compute summary**: Fleet-wide health aggregation
|
||||
8. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
|
||||
9. **Serve API**: HTTP server on port 9100
|
||||
10. **Repeat** every 15 seconds
|
||||
**Port discipline:** probe GPU hosts on `:8080` (or the router's
|
||||
`/health/unified`); probe port 80 only on the router (.116). Never bare port 80
|
||||
on a GPU host.
|
||||
|
||||
1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback). `301` → `/gpu/gpu-data` counts as alive.
|
||||
2. **Fallback direct GPU probe** (only if router /health/unified is DOWN): GET `http://192.168.68.8:8080/health` and `http://192.168.68.110:8080/health` — **:8080 only, never bare port 80**.
|
||||
3. **Poll router** (every 15s): GET .116/health via nginx:80
|
||||
4. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
|
||||
5. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
|
||||
6. **Poll dashboard** (every 15s): GET .116/dashboard/
|
||||
7. **Check alerts**: Compare metrics against thresholds
|
||||
8. **Compute summary**: Fleet-wide health aggregation
|
||||
9. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
|
||||
10. **Serve API**: HTTP server on port 9100
|
||||
11. **Repeat** every 15 seconds
|
||||
|
||||
@@ -102,11 +102,24 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
|
||||
- Stack persists across reboots (systemd for exporters, Docker restart policy)
|
||||
|
||||
## Execution
|
||||
|
||||
### Liveness rule (any-HTTP-response)
|
||||
|
||||
A probe is **ALIVE** if the endpoint returns **ANY** HTTP status — including
|
||||
`401`/`403` auth challenges and `3xx` redirects. A bare `200` is not required and
|
||||
must never be a pass condition for an auth-gated endpoint. **DOWN = connection
|
||||
refused (`000`) or timeout only.** Same rule as zulip-health (Tanko) and
|
||||
gpu-monitor. The PVE API (`:8006/api2/json`) legitimately answers `401` to an
|
||||
unauthenticated probe — that is the healthy signal, not a failure.
|
||||
|
||||
### check-health
|
||||
|
||||
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
|
||||
|
||||
```bash
|
||||
# Provenance — run first; paste the absolute path into the report
|
||||
pwd -P
|
||||
|
||||
# Zulip API health (POST ping)
|
||||
source /etc/litellm-monitor.env
|
||||
ZULIP_USER="abiba-bot@chat.sysloggh.net"
|
||||
@@ -128,11 +141,20 @@ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
|
||||
|
||||
# LiteLLM health (via nginx on port 80)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
|
||||
# Expected: 200 (LiteLLM is up and responding)
|
||||
# Expected: 301 → /litellm/health/liveliness (200 after redirect) — any HTTP status = alive
|
||||
|
||||
# PVE API (401 expected for unauthenticated probe — API is up over https)
|
||||
curl -s -o /dev/null -w '%{http_code}' https://192.168.68.116:8006/api2/json
|
||||
# Expected: 401 (unauthorized — API is up; 000 = unreachable, 500 = API down)
|
||||
# PVE API liveness — probe the REAL PVE nodes on :8006, never the monitoring
|
||||
# host CT 116. CT 116 runs no pveproxy, so probing it on :8006 returns 000 —
|
||||
# that was the stale-vantage bug this replaces (CT 116 is the monitoring host,
|
||||
# not a cluster node). Unauthenticated GET answers 401 while the API is ALIVE
|
||||
# by design. Alive = ANY HTTP status (401 is the EXPECTED healthy response);
|
||||
# DOWN = connection refused (000) or timeout only.
|
||||
for node in 192.168.68.9 192.168.68.5 192.168.68.15 192.168.68.6 192.168.68.12; do
|
||||
printf '%s:8006 -> %s\n' "$node" \
|
||||
"$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 "https://$node:8006/api2/json/version")"
|
||||
done
|
||||
# Expected: 401 on every node (acerpve .9, ocupve .5, amdpve .15, storepve .6, minipve .12)
|
||||
# A node answering 000/timeout is DOWN — flag that node. 401 is NOT a fault.
|
||||
|
||||
# Prometheus targets
|
||||
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
|
||||
@@ -147,7 +169,13 @@ curl -s http://192.168.68.116:4001/metrics | head -20
|
||||
# Expected: Prometheus-formatted metrics output
|
||||
```
|
||||
|
||||
**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
|
||||
**Report format**: Begin every report with the **absolute path the probe executed
|
||||
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
|
||||
distinguishable from a real fault at read time. Summarize actual results from
|
||||
each probe. Apply the any-HTTP-response liveness rule above: only
|
||||
connection-refused (`000`) or timeout is DOWN; empty output is a warning. A probe
|
||||
that requires a bare `200` on an auth-gated endpoint (PVE API → `401`, LiteLLM
|
||||
health → `301` redirect) is a stale expectation, not a fault.
|
||||
|
||||
|
||||
### Phase 1: GPU Exporters
|
||||
|
||||
@@ -122,7 +122,10 @@ ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1
|
||||
# Expected: 200 (pve-exporter is up and responding)
|
||||
```
|
||||
|
||||
**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
|
||||
**Report format**: Begin every report with the **absolute path the probe executed
|
||||
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
|
||||
distinguishable from a real fault at read time. Summarize actual results from
|
||||
each probe. If any probe returns non-200, flag as alert.
|
||||
|
||||
**Note**: Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only) so they must be probed from .116 via SSH. Prometheus and Grafana are bound to 0.0.0.0 so they can be probed from the LAN.
|
||||
|
||||
|
||||
+107
-39
@@ -26,6 +26,17 @@ Changelog:
|
||||
Fixed UnboundLocalError on the abiba/koonimo gateway leg (pid unbound in the
|
||||
summary f-string). Abiba's LiteLLM key now comes from /root/.pi/agent/env.sh
|
||||
(#735 agent separation; creds moved out of shared /root/.bashrc).
|
||||
v4 (2026-09-10): probe-drift round 2 (prose-contracts follow-up to #65/#66/#68).
|
||||
abiba declared pi-only runtime — Hermes-era config/wrapper/gateway checks are
|
||||
skipped (harness purge). koby declared report_only per the captain's
|
||||
2026-08-17 ruling: every koby leg is detected and reported, never counted as a
|
||||
fleet failure and never repaired. koby's PVE mapping corrected to storepve
|
||||
(CT 111 tdunna lives on .6 — the old amdpve mapping produced a false
|
||||
ct-unreachable). The wrapper infisical-path check no longer FAILs .env-based
|
||||
wrappers that legitimately never invoke infisical (koonimo/baggy). Every run
|
||||
now prints absolute execution provenance (script + cwd) in the header and in
|
||||
--json output so a stale-consumer report is distinguishable from a fault at
|
||||
read time.
|
||||
"""
|
||||
|
||||
import subprocess, json, sys, os, time
|
||||
@@ -49,10 +60,18 @@ AGENTS = {
|
||||
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
|
||||
# abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
|
||||
# local env file (key_env below), not from the shared vault or .bashrc.
|
||||
# runtime=pi: abiba has run pi-only since the harness purge. There is no
|
||||
# Hermes gateway, no ~/.hermes/config.yaml and no hermes CLI wrapper on .24
|
||||
# (the /root/.local/bin/hermes symlink is dangling), so the Hermes-era
|
||||
# config/wrapper/gateway legs are skipped rather than reported as faults.
|
||||
"abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "minipve",
|
||||
"vault_key": None,
|
||||
"vault_key": None, "runtime": "pi",
|
||||
"key_env": {"file": "/root/.pi/agent/env.sh", "var": "LITELLM_API_KEY"}},
|
||||
"koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "amdpve", "vault_key": "KOBY_LITELLM_API_KEY"},
|
||||
# koby = report-only (captain's 2026-08-17 ruling, Rule 17): detect and
|
||||
# report, NEVER repair, and never count against fleet failures. CT 111
|
||||
# (tdunna) lives on storepve (.6) — verified live 2026-09-10; the previous
|
||||
# amdpve mapping made `pct status 111` fail and read as ct-unreachable.
|
||||
"koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "storepve", "vault_key": "KOBY_LITELLM_API_KEY", "report_only": True},
|
||||
"koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root", "pve": "amdpve", "vault_key": "KOONIMO_LITELLM_API_KEY"},
|
||||
}
|
||||
|
||||
@@ -70,6 +89,22 @@ GPU_HOSTS = {
|
||||
|
||||
FAIL = []
|
||||
|
||||
|
||||
def _fail(key, agent_name=None):
|
||||
"""Record a failure, except for report-only agents.
|
||||
|
||||
Koby is report-only per the captain's 2026-08-17 ruling (Rule 17): its legs
|
||||
are detected and reported, never repaired and never counted as fleet
|
||||
failures. A red fleet alert on a known report-only leg is a false alarm. Any
|
||||
non-report-only agent (or a leg with no agent, e.g. GPU hosts) records
|
||||
normally.
|
||||
"""
|
||||
if agent_name and AGENTS.get(agent_name, {}).get("report_only"):
|
||||
print(f" 🔍 report-only ({agent_name}): {key} — reported, not counted/repaired")
|
||||
return
|
||||
FAIL.append(key)
|
||||
|
||||
|
||||
INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
|
||||
INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
|
||||
|
||||
@@ -203,17 +238,21 @@ def _read_env_export(path, var):
|
||||
return None
|
||||
|
||||
|
||||
# Inject keys for each agent:
|
||||
# - vault-backed agents (tanko/koby/koonimo): {NAME}_LITELLM_API_KEY from
|
||||
# Infisical (project 322fceab-39da-4854-a55a-568e76c0f13f, env prod).
|
||||
# - abiba (pi agent, no vault key): LITELLM_API_KEY from its local env file
|
||||
# /root/.pi/agent/env.sh (moved there from /root/.bashrc in #735).
|
||||
for agent_name in AGENTS:
|
||||
info = AGENTS[agent_name]
|
||||
key = _get_agent_key(agent_name, info.get("vault_key"))
|
||||
if not key and info.get("key_env"):
|
||||
key = _read_env_export(info["key_env"]["file"], info["key_env"]["var"])
|
||||
AGENTS[agent_name]["key"] = key
|
||||
def load_agent_keys():
|
||||
"""Populate AGENTS[*]["key"] from the vault or the agent's local env file.
|
||||
|
||||
Called from main(), not at import: keeping this out of module scope lets the
|
||||
module be imported (and unit tested) without live vault/SSH access. Vault
|
||||
format is {NAME}_LITELLM_API_KEY (project 322fceab-39da-4854-a55a-568e76c0f13f,
|
||||
env prod); abiba has no vault key and reads LITELLM_API_KEY from its local
|
||||
/root/.pi/agent/env.sh (moved there from /root/.bashrc in #735).
|
||||
"""
|
||||
for agent_name in AGENTS:
|
||||
info = AGENTS[agent_name]
|
||||
key = _get_agent_key(agent_name, info.get("vault_key"))
|
||||
if not key and info.get("key_env"):
|
||||
key = _read_env_export(info["key_env"]["file"], info["key_env"]["var"])
|
||||
AGENTS[agent_name]["key"] = key
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
@@ -225,7 +264,7 @@ def check_keys():
|
||||
key = agent.get("key")
|
||||
if not key:
|
||||
print(f" ❌ {name}: NO KEY FOUND (vault/env empty or unreachable)")
|
||||
FAIL.append(f"key:{name}:no-key")
|
||||
_fail(f"key:{name}:no-key", name)
|
||||
continue
|
||||
data = http_json(f"{LITELLM}/v1/models",
|
||||
headers={"Authorization": f"Bearer {key}"})
|
||||
@@ -234,7 +273,7 @@ def check_keys():
|
||||
print(f" ✅ {name}: key valid → {model}")
|
||||
else:
|
||||
print(f" ❌ {name}: KEY FAILURE — auth rejected or unreachable")
|
||||
FAIL.append(f"key:{name}")
|
||||
_fail(f"key:{name}", name)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
@@ -294,12 +333,17 @@ def check_agents():
|
||||
|
||||
# Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
|
||||
# Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
|
||||
if agent.get("runtime") == "dsh":
|
||||
# Non-Hermes runtimes have no gateway to probe. dsh = Tanko since
|
||||
# 2026-08-27; pi = abiba since the harness purge (.24 is pi-only).
|
||||
if agent.get("runtime") in ("dsh", "pi"):
|
||||
is_dsh = agent.get("runtime") == "dsh"
|
||||
label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
|
||||
since = "since 2026-08-27" if is_dsh else "since the harness purge"
|
||||
live = ssh(host, "true", user=user)
|
||||
print(f" {'✅' if live is not None else '❌'} {name}: DSH (DeepSeek Harness) — "
|
||||
f"no Hermes gateway since 2026-08-27 (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
|
||||
print(f" {'✅' if live is not None else '❌'} {name}: {label} — "
|
||||
f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
|
||||
if live is None:
|
||||
FAIL.append(f"unreachable:{name}")
|
||||
_fail(f"unreachable:{name}", name)
|
||||
continue
|
||||
|
||||
if not host or not user:
|
||||
@@ -323,7 +367,7 @@ def check_agents():
|
||||
# Still check gateway status for reporting purposes
|
||||
if pid == "?":
|
||||
print(f" ⚠️ {name}: GATEWAY NOT RUNNING (reported only)")
|
||||
FAIL.append(f"gateway-down:{name}")
|
||||
_fail(f"gateway-down:{name}", name)
|
||||
continue
|
||||
else:
|
||||
print(f" ✅ {name}: gateway running (pid={pid}, report-only mode)")
|
||||
@@ -386,12 +430,12 @@ def check_ct_liveness():
|
||||
status = ssh(pve_ip, f"pct status {ct} 2>/dev/null", user="root")
|
||||
if not status:
|
||||
print(f" ❌ {name} (CT {ct} on {pve_node}): PVE UNREACHABLE")
|
||||
FAIL.append(f"ct-unreachable:{name}:{pve_ip}")
|
||||
_fail(f"ct-unreachable:{name}:{pve_ip}", name)
|
||||
elif "running" in status:
|
||||
print(f" ✅ {name} (CT {ct} on {pve_node}): running")
|
||||
elif "stopped" in status:
|
||||
print(f" ❌ {name} (CT {ct} on {pve_node}): STOPPED")
|
||||
FAIL.append(f"ct-stopped:{name}")
|
||||
_fail(f"ct-stopped:{name}", name)
|
||||
else:
|
||||
print(f" ⚠️ {name} (CT {ct} on {pve_node}): {status.strip()}")
|
||||
|
||||
@@ -407,6 +451,9 @@ def check_config_integrity():
|
||||
if agent.get("runtime") == "dsh":
|
||||
print(f" ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
|
||||
continue
|
||||
if agent.get("runtime") == "pi":
|
||||
print(f" ⏭️ {name}: pi-only runtime — no Hermes config.yaml since the harness purge")
|
||||
continue
|
||||
host = agent.get("host")
|
||||
user = agent.get("user")
|
||||
if not host or not user:
|
||||
@@ -421,12 +468,12 @@ def check_config_integrity():
|
||||
user=user)
|
||||
if not yaml_ok:
|
||||
print(f" ❌ {name}: SSH UNREACHABLE (config check skipped)")
|
||||
FAIL.append(f"config-unreachable:{name}")
|
||||
_fail(f"config-unreachable:{name}", name)
|
||||
elif "OK" in yaml_ok:
|
||||
print(f" ✅ {name}: config.yaml valid YAML")
|
||||
else:
|
||||
print(f" ❌ {name}: config.yaml YAML ERROR — {yaml_ok[:120]}")
|
||||
FAIL.append(f"config-yaml-error:{name}")
|
||||
_fail(f"config-yaml-error:{name}", name)
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
@@ -440,6 +487,9 @@ def check_wrapper_integrity():
|
||||
if agent.get("runtime") == "dsh":
|
||||
print(f" ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
|
||||
continue
|
||||
if agent.get("runtime") == "pi":
|
||||
print(f" ⏭️ {name}: pi-only runtime — no hermes CLI wrapper since the harness purge")
|
||||
continue
|
||||
host = agent.get("host")
|
||||
user = agent.get("user")
|
||||
if not host or not user:
|
||||
@@ -453,24 +503,31 @@ def check_wrapper_integrity():
|
||||
wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
|
||||
if not wrapper:
|
||||
print(f" ❌ {name}: NO HERMES CLI WRAPPER FOUND")
|
||||
FAIL.append(f"wrapper-missing:{name}")
|
||||
_fail(f"wrapper-missing:{name}", name)
|
||||
continue
|
||||
else:
|
||||
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
|
||||
|
||||
# Check wrapper has correct infisical path
|
||||
infisical_path_valid = ssh(host,
|
||||
"head -20 /root/.local/bin/hermes 2>/dev/null | grep -q '/usr/bin/infisical' && echo OK || echo MISS",
|
||||
user=user)
|
||||
if infisical_path_valid == "MISS":
|
||||
# Check if infisical exists on path
|
||||
# Credential-injection mechanism. The Hermes-era wrapper injected creds
|
||||
# with `/usr/bin/infisical run`; newer wrappers source the agent key from
|
||||
# ~/.hermes/.env and never mention infisical (koonimo/baggy does exactly
|
||||
# this). A wrapper with NO infisical reference is therefore a valid
|
||||
# alternate mechanism, not a fault — only flag a wrapper that DOES call
|
||||
# infisical when the binary cannot resolve. The old check FAILed every
|
||||
# .env-based wrapper as "infisical path may be wrong": a stale
|
||||
# expectation, not a fault.
|
||||
wrapper_body = ssh(host, "cat /root/.local/bin/hermes 2>/dev/null", user=user) or ""
|
||||
if "infisical" in wrapper_body:
|
||||
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
|
||||
if not inf_actual:
|
||||
print(f" ❌ {name}: INFISICAL NOT INSTALLED (wrapper broken)")
|
||||
FAIL.append(f"wrapper-no-infisical:{name}")
|
||||
print(f" ❌ {name}: wrapper invokes infisical but the binary is MISSING")
|
||||
_fail(f"wrapper-no-infisical:{name}", name)
|
||||
elif "/usr/bin/infisical" not in wrapper_body:
|
||||
print(f" ⚠️ {name}: wrapper infisical path differs (infisical at {inf_actual}) — informational")
|
||||
else:
|
||||
print(f" ⚠️ {name}: wrapper infisical path may be wrong (infisical at {inf_actual})")
|
||||
FAIL.append(f"wrapper-infisical-path:{name}")
|
||||
print(f" ✅ {name}: wrapper infisical path OK")
|
||||
else:
|
||||
print(f" ℹ️ {name}: wrapper resolves creds without infisical (e.g. ~/.hermes/.env) — OK")
|
||||
|
||||
# Check hermes-real exists
|
||||
hermes_real = ssh(host,
|
||||
@@ -483,7 +540,7 @@ def check_wrapper_integrity():
|
||||
user=user)
|
||||
if not hermes_real or hermes_real.strip() == "MISS":
|
||||
print(f" ❌ {name}: hermes-real NOT FOUND (wrapper broken)")
|
||||
FAIL.append(f"wrapper-no-hermes-real:{name}")
|
||||
_fail(f"wrapper-no-hermes-real:{name}", name)
|
||||
else:
|
||||
print(f" ✅ {name}: hermes-real at alt path")
|
||||
|
||||
@@ -511,10 +568,10 @@ def check_vault_secrets():
|
||||
key = agent.get("key")
|
||||
if not key:
|
||||
print(f" ❌ {name}: vault secret {vault_key_name} MISSING or EMPTY")
|
||||
FAIL.append(f"vault-empty:{name}:{vault_key_name}")
|
||||
_fail(f"vault-empty:{name}:{vault_key_name}", name)
|
||||
elif not key.startswith("sk-"):
|
||||
print(f" ❌ {name}: vault secret {vault_key_name} WRONG FORMAT (starts '{key[:8]}...')")
|
||||
FAIL.append(f"vault-bad-format:{name}:{vault_key_name}")
|
||||
_fail(f"vault-bad-format:{name}:{vault_key_name}", name)
|
||||
else:
|
||||
print(f" ✅ {name}: vault {vault_key_name}=sk-...{key[-4:]}")
|
||||
|
||||
@@ -555,10 +612,20 @@ def main():
|
||||
if not quiet and "--no-deploy" not in sys.argv:
|
||||
deploy_self()
|
||||
|
||||
# Provenance: a report is only actionable if the reader can tell WHICH copy
|
||||
# of this script produced it. Emit the absolute execution path (script + cwd)
|
||||
# in both human and --json output so a stale-consumer report is
|
||||
# distinguishable from a real fault at read time.
|
||||
script_path = os.path.abspath(__file__)
|
||||
cwd = os.getcwd()
|
||||
|
||||
if not quiet:
|
||||
print(f"🏥 Agent Health Check v2 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
|
||||
print(f"🏥 Agent Health Check v4 — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
|
||||
print(f"📍 executed from: script={script_path} cwd={cwd}")
|
||||
print()
|
||||
|
||||
load_agent_keys()
|
||||
|
||||
print("🔑 LiteLLM Keys:")
|
||||
check_keys()
|
||||
print()
|
||||
@@ -595,6 +662,7 @@ def main():
|
||||
|
||||
if as_json:
|
||||
print(json.dumps({"timestamp": datetime.now().isoformat(),
|
||||
"execution_path": script_path, "cwd": cwd,
|
||||
"failures": FAIL, "healthy": len(FAIL) == 0}))
|
||||
|
||||
sys.exit(1 if FAIL else 0)
|
||||
|
||||
+21
-2
@@ -57,7 +57,7 @@ echo "── 2. Regression detection ──"
|
||||
|
||||
# Grafana /grafana/ as nginx route or URL path (reverted 2026-07-02)
|
||||
# EXCLUDE: filesystem paths (/opt/monitoring/grafana/...), directory creation, revert docs
|
||||
GRAFANA_HITS=$(grep -rn '/grafana/' *.prose.md 2>/dev/null \
|
||||
GRAFANA_HITS=$(grep -rn '/grafana/' ./*.prose.md 2>/dev/null \
|
||||
| grep -v '/opt/monitoring/grafana/' \
|
||||
| grep -v 'was tried and reverted\|was reverted\|do not re-add\|NOT recommended' \
|
||||
| grep -v 'mkdir.*grafana\|Create.*grafana' \
|
||||
@@ -71,7 +71,7 @@ else
|
||||
fi
|
||||
|
||||
# Stale CT IDs (CT 122, CT 123 as CT IDs — not IPs .122, .123)
|
||||
CT_STALE=$(grep -rn '\bCT 122\b' *.prose.md 2>/dev/null || true)
|
||||
CT_STALE=$(grep -rn '\bCT 122\b' ./*.prose.md 2>/dev/null || true)
|
||||
if [ -n "$CT_STALE" ]; then
|
||||
echo " ❌ REGRESSION: CT 122 used as CT ID — Tanko is CT 112"
|
||||
echo "$CT_STALE"
|
||||
@@ -84,6 +84,25 @@ fi
|
||||
# .122/.123 are correct — verified reachable bridge IPs for Tanko/Mumuni
|
||||
echo " ✅ IP consistency verified (.19=.122=.123 all reachable)"
|
||||
|
||||
# Report provenance — every contract report must state the absolute path it
|
||||
# executed from, so a stale-consumer report is distinguishable from a real fault
|
||||
# at read time (2026-09-09 probe-drift incident: three false DEGRADED rounds).
|
||||
PROV_FILES=$(grep -rl '\*\*Report format\*\*' ./*.prose.md 2>/dev/null || true)
|
||||
if [ -n "$PROV_FILES" ]; then
|
||||
PROV_BAD=0
|
||||
while IFS= read -r f; do
|
||||
if ! grep -qE 'absolute path|pwd -P|executed from' "$f"; then
|
||||
echo " ❌ $f: **Report format** lacks execution provenance (absolute path / pwd -P)"
|
||||
PROV_BAD=1
|
||||
fi
|
||||
done <<< "$PROV_FILES"
|
||||
if [ "$PROV_BAD" -eq 1 ]; then
|
||||
FAILED=1
|
||||
else
|
||||
echo " ✅ Report provenance present in all report-format contracts"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 3. Cross-contract consistency ──
|
||||
echo ""
|
||||
echo "── 3. Cross-contract consistency ──"
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
"""Regression tests for the 2026-09-09/10 probe-drift corrections.
|
||||
|
||||
WHY THIS FILE EXISTS: the monitoring contracts kept emitting false alarms from
|
||||
stale expectations rather than live faults.
|
||||
|
||||
* agent-health-check v3 reported 6 failures that were all stale expectations:
|
||||
abiba (pi-only since the harness purge) was tested as a Hermes host, koby
|
||||
(report-only per the captain's 2026-08-17 ruling) was counted as repairable,
|
||||
koby's CT 111 was probed on amdpve where it does not exist (it runs on
|
||||
storepve .6), and a .env-based hermes wrapper was FAILed for not mentioning
|
||||
infisical.
|
||||
* gpu-monitor emitted "DEGRADED — GPU-rtx3090 000, GPU-rtx5070 000" three
|
||||
times from probing bare port 80 on GPU hosts while :8080 answered 200.
|
||||
* infrastructure-monitoring probed CT 116 for the PVE API (no pveproxy ->
|
||||
000) instead of the five real cluster nodes, which answer 401 = alive.
|
||||
|
||||
These tests pin the corrections so the false alarms cannot return silently.
|
||||
They are static/structural over the contracts plus an inert import of the health
|
||||
script — no live network, vault, or SSH access is required.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
AHC = ROOT / "scripts" / "agent-health-check.py"
|
||||
GPU = ROOT / "gpu-monitor.prose.md"
|
||||
INFRA = ROOT / "infrastructure-monitoring.prose.md"
|
||||
PROXMOX = ROOT / "proxmox-monitor.prose.md"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def ahc():
|
||||
"""Import agent-health-check.py without live network/SSH side effects."""
|
||||
spec = importlib.util.spec_from_file_location("agent_health_check", AHC)
|
||||
assert spec and spec.loader
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
# ── agent-health-check: stale-expectation legs ───────────────────────
|
||||
|
||||
def test_import_does_not_contact_vault(ahc):
|
||||
# Keys are loaded in main() via load_agent_keys(); importing must stay inert.
|
||||
assert callable(ahc.load_agent_keys)
|
||||
assert all(agent.get("key") is None for agent in ahc.AGENTS.values())
|
||||
|
||||
|
||||
def test_abiba_is_pi_only_runtime(ahc):
|
||||
# .24 has run pi-only since the harness purge: no Hermes gateway, config, or
|
||||
# wrapper. Probing those legs produced false failures.
|
||||
assert ahc.AGENTS["abiba"]["runtime"] == "pi"
|
||||
|
||||
|
||||
def test_koby_is_report_only(ahc):
|
||||
# Captain's 2026-08-17 ruling (Rule 17): detect and report, never repair.
|
||||
assert ahc.AGENTS["koby"]["report_only"] is True
|
||||
|
||||
|
||||
def test_koby_ct111_is_on_storepve(ahc):
|
||||
# Live-verified 2026-09-10: `pct status 111` = running on storepve (.6);
|
||||
# amdpve has no lxc/111.conf, which is what false-failed before.
|
||||
assert ahc.AGENTS["koby"]["pve"] == "storepve"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("agent", ["koby", "koonimo", "tanko"])
|
||||
def test_report_only_legs_never_count_as_failures(ahc, agent):
|
||||
ahc.FAIL.clear()
|
||||
try:
|
||||
ahc._fail(f"probe:{agent}", agent)
|
||||
if ahc.AGENTS[agent].get("report_only"):
|
||||
assert ahc.FAIL == []
|
||||
else:
|
||||
assert ahc.FAIL == [f"probe:{agent}"]
|
||||
finally:
|
||||
ahc.FAIL.clear()
|
||||
|
||||
|
||||
def test_failure_recording_accepts_agentless_keys(ahc):
|
||||
ahc.FAIL.clear()
|
||||
try:
|
||||
ahc._fail("gpu-no-port:gpu-rtx3090 (.8)")
|
||||
assert ahc.FAIL == ["gpu-no-port:gpu-rtx3090 (.8)"]
|
||||
finally:
|
||||
ahc.FAIL.clear()
|
||||
|
||||
|
||||
def test_script_reports_absolute_execution_provenance(ahc):
|
||||
src = AHC.read_text()
|
||||
assert "execution_path" in src
|
||||
assert "os.getcwd()" in src
|
||||
|
||||
|
||||
def test_wrapper_check_has_no_bare_infisical_expectation(ahc):
|
||||
# A wrapper that never mentions infisical (koonimo sources ~/.hermes/.env)
|
||||
# must not be FAILed merely for lacking an infisical reference.
|
||||
assert "wrapper resolves creds without infisical" in AHC.read_text()
|
||||
|
||||
|
||||
# ── item 4: every contract report carries its execution path ──────────
|
||||
|
||||
@pytest.mark.parametrize("contract", [GPU, INFRA, PROXMOX], ids=lambda p: p.name)
|
||||
def test_report_format_requires_execution_provenance(contract):
|
||||
text = contract.read_text()
|
||||
assert "**Report format**" in text
|
||||
assert re.search(r"absolute path|pwd -P|executed from", text)
|
||||
|
||||
|
||||
# ── item 3: gpu-monitor probes the real GPU endpoints ────────────────
|
||||
|
||||
def test_gpu_monitor_probes_gpu_health_on_8080():
|
||||
text = GPU.read_text()
|
||||
assert "http://192.168.68.8:8080/health" in text
|
||||
assert "http://192.168.68.110:8080/health" in text
|
||||
|
||||
|
||||
def test_gpu_monitor_forbids_bare_port_80_on_gpu_hosts():
|
||||
text = GPU.read_text()
|
||||
assert re.search(r"never .{0,20}bare port 80", text, re.I)
|
||||
# The false-DEGRADED symptom must be documented, not just implied.
|
||||
assert "DEGRADED" in text
|
||||
|
||||
|
||||
def test_gpu_monitor_documents_router_301_as_alive():
|
||||
text = GPU.read_text()
|
||||
assert "/gpu/gpu-data" in text
|
||||
assert "301" in text
|
||||
|
||||
|
||||
# ── item 2: infrastructure-monitoring PVE API vantage ────────────────
|
||||
|
||||
def test_infra_monitoring_does_not_probe_ct116_for_pve_api():
|
||||
assert "https://192.168.68.116:8006" not in INFRA.read_text()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"node",
|
||||
["192.168.68.9", "192.168.68.5", "192.168.68.15", "192.168.68.6", "192.168.68.12"],
|
||||
)
|
||||
def test_infra_monitoring_probes_every_real_pve_node(node):
|
||||
assert node in INFRA.read_text()
|
||||
|
||||
|
||||
def test_infra_monitoring_uses_any_http_liveness_rule():
|
||||
text = INFRA.read_text()
|
||||
assert "api2/json/version" in text
|
||||
assert "ANY HTTP status" in text or "any-HTTP" in text
|
||||
Reference in New Issue
Block a user