fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness #30

Merged

mumuni-bot merged 5 commits from fix/agent-health-check-v2 into master 2026-07-26 12:27:21 +00:00

Conversation 0 Commits 5 Files Changed 6

+464 -95

mumuni-bot commented 2026-07-26 12:06:41 +00:00

Owner

Copy Link

Copy Source

What changed

Complete overhaul of agent-health-check.py (v2) to fix 6 monitoring gaps:

Fix 1: Koby & Koonimo now SSHa ble

Both had host: None — completely skipped by health checks. Now SSH to .129 (koby) and .114 (koonimo).

Fix 2: Agent key name format

Was looking for LITELLM_API_KEY_{NAME} but vault keys are {NAME}_LITELLM_API_KEY. Fallback never matched — all agent keys silently used the master key.

Fix 3: CT liveness check (NEW)

Checks pct status <CT> on each agent's PVE node. Catches stopped containers (Mumuni was stopped for days without detection).

Fix 4: Config YAML integrity (NEW)

Runs yaml.safe_load() on each agent's config.yaml. Catches syntax errors (Mumuni had a broken YAML on line 234).

Fix 5: Wrapper/CLI integrity (NEW)

Verifies hermes wrapper exists, infisical path is correct, hermes-real is reachable. Catches wrong binary paths (baggy's wrapper had /usr/local/bin/infisical but binary is at /usr/bin/infisical).

Fix 6: Vault secret non-emptiness (NEW)

Checks each agent-specific vault key is non-empty and starts with sk-. Baggy's BAGGY_LITELLM_API_KEY was empty for days.

Contracts updated

• litellm-self-heal.prose.md — fleet roster, v2 features
• infrastructure-control.prose.md — check table now lists all 9 checks
• hermes-agent-baseline.prose.md — script reference updated

## What changed Complete overhaul of `agent-health-check.py` (v2) to fix 6 monitoring gaps: ### Fix 1: Koby & Koonimo now SSHa ble Both had `host: None` — completely skipped by health checks. Now SSH to .129 (koby) and .114 (koonimo). ### Fix 2: Agent key name format Was looking for `LITELLM_API_KEY_{NAME}` but vault keys are `{NAME}_LITELLM_API_KEY`. Fallback never matched — all agent keys silently used the master key. ### Fix 3: CT liveness check (NEW) Checks `pct status <CT>` on each agent's PVE node. Catches stopped containers (Mumuni was stopped for days without detection). ### Fix 4: Config YAML integrity (NEW) Runs `yaml.safe_load()` on each agent's config.yaml. Catches syntax errors (Mumuni had a broken YAML on line 234). ### Fix 5: Wrapper/CLI integrity (NEW) Verifies hermes wrapper exists, infisical path is correct, hermes-real is reachable. Catches wrong binary paths (baggy's wrapper had /usr/local/bin/infisical but binary is at /usr/bin/infisical). ### Fix 6: Vault secret non-emptiness (NEW) Checks each agent-specific vault key is non-empty and starts with `sk-`. Baggy's BAGGY_LITELLM_API_KEY was empty for days. ### Contracts updated - `litellm-self-heal.prose.md` — fleet roster, v2 features - `infrastructure-control.prose.md` — check table now lists all 9 checks - `hermes-agent-baseline.prose.md` — script reference updated

mumuni-bot added 3 commits 2026-07-26 12:06:41 +00:00

fix: contract accuracy updates post fleet-wide reboot ... b17c60f997

infrastructure-control.prose.md:
- Add hwepve as 6th Proxmox node
- Fix Gitea IP: .17 (was .110)
- Fix AdGuard IP: .10 on minipve (was .102 on acerpve)
- Fix Abiba placement: hwepve (was amdpve)
- Fix Mumuni placement: hwepve (was minipve)
- Fix Authentik port: add :9000
- Add CT 118 (jdownloader), CT 119 (infisical-vault)
- Add last-verified date (2026-07-24)

infrastructure-update.prose.md:
- Add post-reboot CT sweep procedure
- Add Zulip Docker network recovery steps
- Add WireGuard tunnel verification

scripts/netbird-add-domain.sh:
- New script to register domains in Netbird proxy store.db

fix: address review findings ... c380196fab

- Remove kagentz from amdpve node table (belongs on hwepve)
- Fix kagentz pct-run table: hwepve (was amdpve)
- Fix mumuni pct-run table: hwepve (was minipve)
- Fix Zulip recovery command: wrap in single SSH call

fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness ... de9adb13cf

Gaps fixed:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent key lookup uses correct {NAME}_LITELLM_API_KEY format
- New CT liveness check via pct status on PVE nodes
- New config YAML integrity check via yaml.safe_load()
- New wrapper/CLI integrity check (hermes wrapper, infisical path, hermes-real)
- New vault secret non-emptiness check
- Ops escalation: failures produce ALERT lines for cron capture

abiba-bot added 1 commit 2026-07-26 12:22:15 +00:00

fix: verification protocol findings — CT 118 static IP, .110 llama-server restored ... 47aa92bee3

Verification protocol run 2026-07-26:
- CT 118 (jdownloader): DHCP had moved it to .131 post-reboot. Now set to static .20.
- RTX 5070 (.110) llama-server: was stopped (disabled unit). Started and verified — gemma-4-12b responding through LiteLLM.
- All 6 PVE nodes confirmed at correct IPs.
- All 18 CTs confirmed at correct IPs on correct nodes.
- All public endpoints responding (meet, chat, git, litellm, vault, auth).
- Container counts verified: docker-vm 14 ctrs, CT 116 10 ctrs, VPS 5 ctrs.

abiba-bot force-pushed fix/agent-health-check-v2 from e0f5e4f74e to 47aa92bee3 2026-07-26 12:22:15 +00:00 Compare

abiba-bot added 1 commit 2026-07-26 12:24:09 +00:00

ci: trigger recheck c869e75c61

mumuni-bot merged commit 146abf7f82 into master 2026-07-26 12:27:21 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

abiba-bot jerome (Jerome Kwame Tabiri) kagentz-bot kwame (Kwame Tabiri) mumuni-bot (Mumuni 🦅 (Syslog Falcon)) tanko-bot theo

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SyslogSolution/prose-contracts#30