docs(agent-health): every check leg must appear in every report - a missing line is not a pass #111

Merged

abiba-bot merged 3 commits from fix/agent-health-mandatory-report-legs-20260917 into master 2026-09-17 03:12:58 +00:00

Conversation 0 Commits 3 Files Changed 1

+31

abiba-bot commented 2026-09-17 02:49:22 +00:00

Owner

Copy Link

Copy Source

Fourth reporting wobble this week, so the rule is now written into the contract rather than corrected each time.

What happened

The agent-health-check summary line dropped its GPU clause entirely on the 2026-09-17T02:35Z run:

LiteLLM keys: 4/4 (...) all valid. CTs: 4/4 running (...). koby report-only (hermes-real not found). Vault secrets present.

Its own previous runs of the same check carried GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy. Firstmate verified the fleet independently (.8/.110/.15 all 200) and then asked whether the leg had run at all. It had — all three healthy, with their process IDs — so the check was fine and the report was wrong, which is worse: a leg that can vanish silently makes a missing check indistinguishable from a passing one.

Preceding instances of the same class, all caught by hand: GPU exporters (both) when there are three; LiteLLM keys (all 3) when there are four; 4 keys when the store reports 18.

Change (one file, +10)

agent-health-check.prose.md gains a Mandatory report legs section: every report line MUST carry one clause per leg, even when a leg is skipped or fails, with the required shapes given — LiteLLM keys: N/M (names) status, GPU ports: N/M (rtx3090, rtx5070, strixhalo) status or SKIPPED (reason) or a per-host breakdown, CTs: N/M running (names), Vault secrets: status. It also records that the GPU leg is never skipped by configuration and that a degraded probe must name the failing host per the standing probe rules.

Verification performed

• Confirmed the claim itself: the GPU leg ran (three processes alive) and the current summary includes GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy.
• Three-dot diff: exactly one file, +10/-0.
• Confirmed the responsible clone is clean afterwards — the first attempt at this change sat uncommitted on an already-merged branch and existed nowhere else; it was rebuilt on a fresh branch off master.

Reviewers: confirm the mandatory-legs wording cannot be read as optional, that the required shapes include the skipped and partial cases rather than only the healthy one, and that nothing here conflicts with the probe rules already in the file.

Fourth reporting wobble this week, so the rule is now written into the contract rather than corrected each time. ## What happened The agent-health-check summary line dropped its GPU clause entirely on the 2026-09-17T02:35Z run: ``` LiteLLM keys: 4/4 (...) all valid. CTs: 4/4 running (...). koby report-only (hermes-real not found). Vault secrets present. ``` Its own previous runs of the same check carried `GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy`. Firstmate verified the fleet independently (.8/.110/.15 all 200) and then asked whether the leg had run at all. It had — all three healthy, with their process IDs — so **the check was fine and the report was wrong**, which is worse: a leg that can vanish silently makes a missing check indistinguishable from a passing one. Preceding instances of the same class, all caught by hand: `GPU exporters (both)` when there are three; `LiteLLM keys (all 3)` when there are four; `4 keys` when the store reports 18. ## Change (one file, +10) `agent-health-check.prose.md` gains a **Mandatory report legs** section: every report line MUST carry one clause per leg, even when a leg is skipped or fails, with the required shapes given — `LiteLLM keys: N/M (names) status`, `GPU ports: N/M (rtx3090, rtx5070, strixhalo) status` or `SKIPPED (reason)` or a per-host breakdown, `CTs: N/M running (names)`, `Vault secrets: status`. It also records that the GPU leg is never skipped by configuration and that a degraded probe must name the failing host per the standing probe rules. ## Verification performed - Confirmed the claim itself: the GPU leg ran (three processes alive) and the current summary includes `GPU ports: 3/3 (rtx3090, rtx5070, strixhalo) healthy`. - Three-dot diff: exactly one file, +10/-0. - Confirmed the responsible clone is clean afterwards — the first attempt at this change sat uncommitted on an already-merged branch and existed nowhere else; it was rebuilt on a fresh branch off master. Reviewers: confirm the mandatory-legs wording cannot be read as optional, that the required shapes include the skipped and partial cases rather than only the healthy one, and that nothing here conflicts with the probe rules already in the file.

abiba-bot added 1 commit 2026-09-17 02:49:22 +00:00

Add mandatory report legs to agent-health-check contract ... dd6e1e8b22

Every report line MUST include one clause per check leg, even when a leg is
skipped or fails. Missing leg must never look the same as healthy leg.
Required legs:
- LiteLLM keys: N/M (names) status
- GPU ports: N/M (rtx3090, rtx5070, strixhalo) status — or SKIPPED (reason)
- CTs: N/M running (names)
- Vault secrets: status

GPU leg is never skipped by configuration; only SSH probe failure causes
degraded status.

abiba-bot referenced this issue from a commit 2026-09-17 02:53:56 +00:00

Fix PR #111 review findings: GPU leg failure modes + leg templates

abiba-bot added 1 commit 2026-09-17 02:53:56 +00:00

Fix PR #111 review findings: GPU leg failure modes + leg templates ... 9edefe036e

Fix 1: GPU leg degradation is not just SSH probe failure — it also covers
gpu-no-port and gpu-ghost conditions. Rewrite to match check_gpu_ports reality.

Fix 2: Add skipped and partial exemplars for all four legs (LiteLLM keys,
GPU ports, CTs, Vault secrets) so the template covers the rule rather than
only the happy path.

Cosmetic: note that compact form (rtx5070 timeout) is acceptable in summary
line when host is identifiable from context; full probe-failed: <target> <kind>
form required in detail section.

abiba-bot referenced this issue from a commit 2026-09-17 03:05:25 +00:00

Fix PR #111 round 2: GPU leg all 6 states + skipped templates

abiba-bot added 1 commit 2026-09-17 03:05:26 +00:00

Fix PR #111 round 2: GPU leg all 6 states + skipped templates 0b92ab17b1

abiba-bot merged commit a820b3f7dd into master 2026-09-17 03:12:58 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

abiba-bot jerome (Jerome Kwame Tabiri) kagentz-bot kwame (Kwame Tabiri) mumuni-bot (Mumuni 🦅 (Syslog Falcon)) tanko-bot theo

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#111