SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Details

Browse Source 
Gaps fixed:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent key lookup uses correct {NAME}_LITELLM_API_KEY format
- New CT liveness check via pct status on PVE nodes
- New config YAML integrity check via yaml.safe_load()
- New wrapper/CLI integrity check (hermes wrapper, infisical path, hermes-real)
- New vault secret non-emptiness check
- Ops escalation: failures produce ALERT lines for cron capture
This commit is contained in:
root
2026-07-26 12:06:32 +00:00
parent c380196fab
commit de9adb13cf
4 changed files with 318 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-control.prose.md
+15 -7
View File
@@ -653,24 +653,32 @@ ssh root@192.168.68.15 # Strix Halo
ssh root@192.168.68.4 # hwepve (abiba, kagentz, mumuni)
```
## Section 7: Agent Health Check (consolidated — 2026-07-05)
## Section 7: Agent Health Check v2 (2026-07-26)
Replaces 7 scattered Zulip health scripts with a single non-disruptive check
running every 10 minutes via cron (`/root/scripts/agent-health-check.py`).
Single non-disruptive check running every 10 minutes via cron
(`/root/scripts/agent-health-check.py`). v2 fixes critical gaps:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent-specific vault keys (`{NAME}_LITELLM_API_KEY`) not shared master key
- CT liveness check via `pct status` on PVE nodes
- Config YAML integrity check
- Wrapper/CLI integrity check
- Vault secret non-emptiness check
The script is **read-only** — it never restarts, kills, or modifies anything.
Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
which was kill+nohup outside systemd) are banned by policy.
### Checks Performed
| Check | Frequency | What It Detects |
|-------|-----------|-----------------|
| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models |
| LiteLLM key validation | 10 min | All 5 agent-specific keys authenticate (not shared master key) |
| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
| Gateway liveness | 10 min | Gateway process running, state file readable |
| Gateway liveness | 10 min | Gateway process running, state file readable (all 5 agents) |
| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
| Recent errors | 10 min | Error count in journald for last 10 min |
| CT liveness | 10 min | `pct status` on PVE nodes — catches stopped CTs |
| Config YAML integrity | 10 min | Python `yaml.safe_load()` — catches syntax errors |
| Wrapper/CLI integrity | 10 min | hermes wrapper exists, infisical path correct, hermes-real reachable |
| Vault secrets | 10 min | Agent-specific vault keys are non-empty and start with `sk-` |
### Disabled Scripts