From dd6e1e8b22d4da64fcf75b10fd2ef49d004f28ad Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 02:45:40 +0000 Subject: [PATCH 1/3] Add mandatory report legs to agent-health-check contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every report line MUST include one clause per check leg, even when a leg is skipped or fails. Missing leg must never look the same as healthy leg. Required legs: - LiteLLM keys: N/M (names) status - GPU ports: N/M (rtx3090, rtx5070, strixhalo) status — or SKIPPED (reason) - CTs: N/M running (names) - Vault secrets: status GPU leg is never skipped by configuration; only SSH probe failure causes degraded status. --- agent-health-check.prose.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/agent-health-check.prose.md b/agent-health-check.prose.md index cdc95cd..996c460 100644 --- a/agent-health-check.prose.md +++ b/agent-health-check.prose.md @@ -59,6 +59,16 @@ rules: any HTTP status = ALIVE; only 000/timeout/refused = probe-failed. "Agent health check: OK". If `degraded` or `critical`, report the specific failures and their severity. +**Mandatory report legs** (2026-09-17 decision, 1295.msg): Every report line +MUST include one clause per check leg, even when a leg is skipped or fails. A +missing leg must never look the same as a healthy leg. Required legs: +- `LiteLLM keys: N/M (names) status` +- `GPU ports: N/M (rtx3090, rtx5070, strixhalo) status` — or `GPU ports: SKIPPED (reason)` / `GPU ports: N/M (rtx3090 up; rtx5070 timeout; strixhalo 200)` +- `CTs: N/M running (names)` +- `Vault secrets: status` +The GPU leg is never skipped by configuration; it is only degraded when an SSH +probe fails, in which case the failure must be named per the probe rules. + ### Probe Shape (per standing rules from 1150.msg) 1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the -- 2.54.0 From 9edefe036e141f5f5eac103d356478c32cd05446 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 02:53:51 +0000 Subject: [PATCH 2/3] Fix PR #111 review findings: GPU leg failure modes + leg templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix 1: GPU leg degradation is not just SSH probe failure — it also covers gpu-no-port and gpu-ghost conditions. Rewrite to match check_gpu_ports reality. Fix 2: Add skipped and partial exemplars for all four legs (LiteLLM keys, GPU ports, CTs, Vault secrets) so the template covers the rule rather than only the happy path. Cosmetic: note that compact form (rtx5070 timeout) is acceptable in summary line when host is identifiable from context; full probe-failed: form required in detail section. --- agent-health-check.prose.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/agent-health-check.prose.md b/agent-health-check.prose.md index 996c460..d3bf236 100644 --- a/agent-health-check.prose.md +++ b/agent-health-check.prose.md @@ -61,13 +61,26 @@ failures and their severity. **Mandatory report legs** (2026-09-17 decision, 1295.msg): Every report line MUST include one clause per check leg, even when a leg is skipped or fails. A -missing leg must never look the same as a healthy leg. Required legs: -- `LiteLLM keys: N/M (names) status` -- `GPU ports: N/M (rtx3090, rtx5070, strixhalo) status` — or `GPU ports: SKIPPED (reason)` / `GPU ports: N/M (rtx3090 up; rtx5070 timeout; strixhalo 200)` -- `CTs: N/M running (names)` -- `Vault secrets: status` -The GPU leg is never skipped by configuration; it is only degraded when an SSH -probe fails, in which case the failure must be named per the probe rules. +missing leg must never look the same as a healthy leg. Required legs and their +templates in every state (healthy, skipped, partial): + +- `LiteLLM keys: 4/4 (tanko, abiba, koby, koonimo) valid` + - partial: `LiteLLM keys: 3/4 (tanko valid; koby invalid; koonimo valid; abiba probe-failed: 192.168.68.116:4000 timeout)` +- `GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy` + - skipped: `GPU ports: SKIPPED (no SSH access to GPU hosts)` + - partial: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 probe-failed: 192.168.68.110:9400 timeout; strixhalo healthy)` + - The GPU leg degrades when a probe fails OR when the port is not listening + (`gpu-no-port`) OR when the port is owned by a process other than the + service's (`gpu-ghost`) — in every case the failing host and reason must be + named. +- `CTs: 4/4 running (tanko, abiba, koby, koonimo)` + - partial: `CTs: 3/4 (tanko running; abiba running; koby probe-failed: ssh root@192.168.68.129 timeout; koonimo running)` +- `Vault secrets: 3/3 present` + - partial: `Vault secrets: 2/3 (tanko present; koby present; koonimo missing)` + +The compact form in the summary line is acceptable (e.g. `rtx5070 timeout`) as +long as the host is identifiable from context; the full `probe-failed: +` form is required when a leg reports a failure in the detail section. ### Probe Shape (per standing rules from 1150.msg) -- 2.54.0 From 0b92ab17b168a3bda3aa56060ab20fcc19469147 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 17 Sep 2026 03:05:22 +0000 Subject: [PATCH 3/3] Fix PR #111 round 2: GPU leg all 6 states + skipped templates --- agent-health-check.prose.md | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/agent-health-check.prose.md b/agent-health-check.prose.md index d3bf236..e7a438d 100644 --- a/agent-health-check.prose.md +++ b/agent-health-check.prose.md @@ -60,23 +60,31 @@ rules: any HTTP status = ALIVE; only 000/timeout/refused = probe-failed. failures and their severity. **Mandatory report legs** (2026-09-17 decision, 1295.msg): Every report line -MUST include one clause per check leg, even when a leg is skipped or fails. A -missing leg must never look the same as a healthy leg. Required legs and their -templates in every state (healthy, skipped, partial): +MUST include one clause per check leg, in every state: healthy, degraded/warn, +skipped, or failed. A missing leg must never look the same as a healthy leg. +Required legs and their templates in every state: - `LiteLLM keys: 4/4 (tanko, abiba, koby, koonimo) valid` - - partial: `LiteLLM keys: 3/4 (tanko valid; koby invalid; koonimo valid; abiba probe-failed: 192.168.68.116:4000 timeout)` + - degraded: `LiteLLM keys: 2/4 (tanko valid; koby invalid; koonimo valid; abiba probe-failed: 192.168.68.116:4000 timeout)` + - skipped: `LiteLLM keys: SKIPPED (LiteLLM router unreachable)` - `GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy` - skipped: `GPU ports: SKIPPED (no SSH access to GPU hosts)` - - partial: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 probe-failed: 192.168.68.110:9400 timeout; strixhalo healthy)` - - The GPU leg degrades when a probe fails OR when the port is not listening - (`gpu-no-port`) OR when the port is owned by a process other than the - service's (`gpu-ghost`) — in every case the failing host and reason must be - named. + - degraded/warn: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 degraded: svc=inactive, port owned by 1234; strixhalo healthy)` + - failed: `GPU ports: 2/3 (rtx3090 healthy; rtx5070 probe-failed: 192.168.68.110:9400 timeout; strixhalo healthy)` + - The GPU leg has six non-healthy states the code can produce: + (i) `gpu-unreachable:{host}` — SSH probe failed; + (ii) `gpu-no-port:{label}` — SSH worked, port not listening; + (iii) `gpu-ghost:{label}:{pid}` — unit inactive, port owned by another pid; + (iv) unit not active, MainPID empty or port owned by MainPID — svc inactive; + (v) unit active, /health body contains "error" — error response; + (vi) unit active, /health body unrecognised — unknown health. + In every case the failing host and reason must be named. - `CTs: 4/4 running (tanko, abiba, koby, koonimo)` - - partial: `CTs: 3/4 (tanko running; abiba running; koby probe-failed: ssh root@192.168.68.129 timeout; koonimo running)` + - degraded: `CTs: 3/4 (tanko running; abiba running; koby probe-failed: ssh root@192.168.68.129 timeout; koonimo running)` + - skipped: `CTs: SKIPPED (SSH access unavailable)` - `Vault secrets: 3/3 present` - - partial: `Vault secrets: 2/3 (tanko present; koby present; koonimo missing)` + - degraded: `Vault secrets: 2/3 (tanko present; koby present; koonimo missing)` + - skipped: `Vault secrets: SKIPPED (vault not configured)` The compact form in the summary line is acceptable (e.g. `rtx5070 timeout`) as long as the host is identifiable from context; the full `probe-failed: -- 2.54.0