SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

no-mistakes(review): Harden health-check provenance, infisical verification, report-only JSON, tests

Browse Source
This commit is contained in:
abiba-bot
2026-09-10 01:35:44 +00:00
parent c66d9c1e20
commit 194e256ac5
5 changed files with 367 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-monitoring.prose.md
+22 -11
View File
@@ -103,14 +103,22 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
## Execution
### Liveness rule (any-HTTP-response)
### Liveness rule (scoped)
A probe is **ALIVE** if the endpoint returns **ANY** HTTP status — including
`401`/`403` auth challenges and `3xx` redirects. A bare `200` is not required and
must never be a pass condition for an auth-gated endpoint. **DOWN = connection
refused (`000`) or timeout only.** Same rule as zulip-health (Tanko) and
gpu-monitor. The PVE API (`:8006/api2/json`) legitimately answers `401` to an
unauthenticated probe — that is the healthy signal, not a failure.
The any-HTTP-response rule applies ONLY to unauthenticated/auth-gated endpoints,
where any HTTP answer proves a listener is up: the PVE API
(`https://<node>:8006/api2/json/version`) and LiteLLM health
(`/litellm/health`, `301` → `/litellm/health/liveliness`). For those endpoints a
probe is **ALIVE** on **ANY** HTTP status — `401`/`403` auth challenges and `3xx`
redirects included — and **DOWN = connection refused (`000`) or timeout only**.
The PVE API legitimately answers `401` to an unauthenticated probe — that is the
healthy signal, not a failure. Same scoped rule as zulip-health (Tanko) and
gpu-monitor.
Probes whose success condition is specifically a bare `200` are NOT covered by
the any-HTTP rule. On those — the authenticated Zulip POST and the router
`/health` — an unexpected status (`401`/`403` from a bad or missing credential,
`5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive".
### check-health
@@ -172,10 +180,13 @@ curl -s http://192.168.68.116:4001/metrics | head -20
**Report format**: Begin every report with the **absolute path the probe executed
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
distinguishable from a real fault at read time. Summarize actual results from
each probe. Apply the any-HTTP-response liveness rule above: only
connection-refused (`000`) or timeout is DOWN; empty output is a warning. A probe
that requires a bare `200` on an auth-gated endpoint (PVE API → `401`, LiteLLM
health → `301` redirect) is a stale expectation, not a fault.
each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE
API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is
DOWN; empty output is a warning. For probes whose expected result is a bare `200`
(the authenticated Zulip POST, router `/health`), flag an alert on any unexpected
status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200`
expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect)
is a stale expectation, not a fault.
### Phase 1: GPU Exporters