Compare commits

..
Author SHA1 Message Date
root c869e75c61 ci: trigger recheck
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Passed
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Passed
test-check Manual override
2026-07-26 12:24:05 +00:00
root 47aa92bee3 fix: verification protocol findings — CT 118 static IP, .110 llama-server restored
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Verification protocol run 2026-07-26:
- CT 118 (jdownloader): DHCP had moved it to .131 post-reboot. Now set to static .20.
- RTX 5070 (.110) llama-server: was stopped (disabled unit). Started and verified — gemma-4-12b responding through LiteLLM.
- All 6 PVE nodes confirmed at correct IPs.
- All 18 CTs confirmed at correct IPs on correct nodes.
- All public endpoints responding (meet, chat, git, litellm, vault, auth).
- Container counts verified: docker-vm 14 ctrs, CT 116 10 ctrs, VPS 5 ctrs.
2026-07-26 12:20:58 +00:00
root de9adb13cf fix: agent health check v2 — CT liveness, config validation, wrapper integrity, vault emptiness
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Gaps fixed:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent key lookup uses correct {NAME}_LITELLM_API_KEY format
- New CT liveness check via pct status on PVE nodes
- New config YAML integrity check via yaml.safe_load()
- New wrapper/CLI integrity check (hermes wrapper, infisical path, hermes-real)
- New vault secret non-emptiness check
- Ops escalation: failures produce ALERT lines for cron capture
2026-07-26 12:06:32 +00:00
4 changed files with 323 additions and 71 deletions
+1 -1
View File
@@ -253,7 +253,7 @@ EnvironmentFile=/etc/environment # sources LITELLM_API_KEY
Run the consolidated health check: Run the consolidated health check:
```bash ```bash
python3 /root/scripts/agent-health-check.py python3 /root/scripts/agent-health-check.py # v2: now checks all 5 agents including Koby/Koonimo SSH, CT liveness, config YAML, wrapper integrity, vault non-emptiness
``` ```
This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes), This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
verifies gateway liveness, confirms Zulip streaming (`edit_message` present), verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
+20 -11
View File
@@ -13,10 +13,11 @@ description: >
against the live system. Policy fields are authoritative. See the against the live system. Policy fields are authoritative. See the
`verify-before-mutate` skill. `verify-before-mutate` skill.
**Last verified:** 2026-07-24corrected Gitea IP (.17 not .110), **Last verified:** 2026-07-26verification protocol run. CT 118 set to
AdGuard IP (.10 not .102), AdGuard placement (minipve not acerpve), static IP .20 (was DHCP .131). llama-server on RTX 5070 (.110) was down,
Abiba placement (hwepve not amdpve), added hwepve as 6th node, restarted and verified gemma-4-12b responding through LiteLLM. All 6 PVE
added dns.sysloggh.net route. See data/learnings.md. nodes confirmed. All CT hostnames/IPs confirmed except jdownloader (now
static). All public endpoints responding. See data/learnings.md.
--- ---
# Infrastructure Control Pattern # Infrastructure Control Pattern
@@ -653,24 +654,32 @@ ssh root@192.168.68.15 # Strix Halo
ssh root@192.168.68.4 # hwepve (abiba, kagentz, mumuni) ssh root@192.168.68.4 # hwepve (abiba, kagentz, mumuni)
``` ```
## Section 7: Agent Health Check (consolidated — 2026-07-05) ## Section 7: Agent Health Check v2 (2026-07-26)
Replaces 7 scattered Zulip health scripts with a single non-disruptive check Single non-disruptive check running every 10 minutes via cron
running every 10 minutes via cron (`/root/scripts/agent-health-check.py`). (`/root/scripts/agent-health-check.py`). v2 fixes critical gaps:
- Koby (.129) and Koonimo (.114) now have SSH hosts — no longer skipped
- Agent-specific vault keys (`{NAME}_LITELLM_API_KEY`) not shared master key
- CT liveness check via `pct status` on PVE nodes
- Config YAML integrity check
- Wrapper/CLI integrity check
- Vault secret non-emptiness check
The script is **read-only** — it never restarts, kills, or modifies anything. The script is **read-only** — it never restarts, kills, or modifies anything.
Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
which was kill+nohup outside systemd) are banned by policy.
### Checks Performed ### Checks Performed
| Check | Frequency | What It Detects | | Check | Frequency | What It Detects |
|-------|-----------|-----------------| |-------|-----------|-----------------|
| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models | | LiteLLM key validation | 10 min | All 5 agent-specific keys authenticate (not shared master key) |
| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) | | GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
| Gateway liveness | 10 min | Gateway process running, state file readable | | Gateway liveness | 10 min | Gateway process running, state file readable (all 5 agents) |
| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) | | Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
| Recent errors | 10 min | Error count in journald for last 10 min | | Recent errors | 10 min | Error count in journald for last 10 min |
| CT liveness | 10 min | `pct status` on PVE nodes — catches stopped CTs |
| Config YAML integrity | 10 min | Python `yaml.safe_load()` — catches syntax errors |
| Wrapper/CLI integrity | 10 min | hermes wrapper exists, infisical path correct, hermes-real reachable |
| Vault secrets | 10 min | Agent-specific vault keys are non-empty and start with `sk-` |
### Disabled Scripts ### Disabled Scripts
+1 -1
View File
@@ -102,7 +102,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`). - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault). - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed). - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM. - **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
## Maintains ## Maintains
+301 -58
View File
@@ -1,10 +1,10 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """
/root/scripts/agent-health-check.py — Consolidated Agent Health Verification /root/scripts/agent-health-check.py — Consolidated Agent Health Verification v2
Single non-disruptive health check replacing 7 scattered scripts. Verifies: LiteLLM keys (agent-specific), GPU port conflicts, agent Zulip streaming,
Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming, gateway liveness, gateway log health, CT liveness, config YAML integrity,
gateway liveness, and gateway log health. NEVER restarts anything. wrapper/CLI integrity, vault secret non-emptiness. NEVER restarts anything.
Usage: Usage:
python3 /root/scripts/agent-health-check.py # Full check python3 /root/scripts/agent-health-check.py # Full check
@@ -12,59 +12,40 @@ Usage:
python3 /root/scripts/agent-health-check.py --quiet # Only output on failure python3 /root/scripts/agent-health-check.py --quiet # Only output on failure
Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
Changelog:
v2 (2026-07-26): Added CT liveness, config validation, wrapper integrity,
vault secret emptiness check. Fixed Koby/Koonimo SSH hosts and agent key
name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
Fleet roster: tanko (.122), mumuni (.123), koby (.129), koonimo (.114),
abiba (.24).
""" """
import subprocess, json, sys, os, time import subprocess, json, sys, os, time
from datetime import datetime from datetime import datetime
LITELLM = "http://192.168.68.116:80" LITELLM = "http://192.168.68.116:80"
INFISICAL_PROJECT = "322fceab-39da-4854-a55a-568e76c0f13f"
INFISICAL_ENV = "prod"
def _get_agent_key(agent_name): # PVE node IPs for CT liveness checks
"""Retrieve agent key from Infisical vault.""" PVE_NODES = {
try: "hwepve": "192.168.68.4",
result = subprocess.run( "amdpve": "192.168.68.15",
["infisical", "secrets", "get", "LITELLM_API_KEY", "minipve": "192.168.68.12",
"--project=agents", "--env=production", "--plain"], "storepve": "192.168.68.6",
capture_output=True, text=True, timeout=10 "acerpve": "192.168.68.9",
) "ocupve": "192.168.68.5",
if result.returncode == 0:
return result.stdout.strip()
except Exception:
pass
# Fallback: try exporting all secrets
try:
result = subprocess.run(
["infisical", "export", "--project=agents", "--env=production",
"--format=dotenv"],
capture_output=True, text=True, timeout=10
)
if result.returncode == 0:
for line in result.stdout.splitlines():
if line.startswith(f"LITELLM_API_KEY_{agent_name.upper()}") or \
(line.startswith("LITELLM_API_KEY=") and agent_name == os.uname().nodename):
return line.split("=", 1)[1].strip().strip('"').strip("'")
except Exception:
pass
return None
# Agent keys are pulled from Infisical vault at runtime.
# The 'key' field is populated dynamically below.
AGENTS = {
"tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome"},
"mumuni": {"ct": 114, "host": "192.168.68.123", "user": "root"},
"koby": {"ct": 111, "host": None, "user": None},
"koonimo": {"ct": 113, "host": None, "user": None},
} }
# Inject keys from vault # Agent definitions: ct, host, user, pve_node, vault_key_name
for agent_name in AGENTS: AGENTS = {
key = _get_agent_key(agent_name) "tanko": {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve", "vault_key": "TANKO_LITELLM_API_KEY"},
if key: "mumuni": {"ct": 114, "host": "192.168.68.123", "user": "root", "pve": "hwepve", "vault_key": "MUMUNI_LITELLM_API_KEY"},
AGENTS[agent_name]["key"] = key "koby": {"ct": 111, "host": "192.168.68.129", "user": "root", "pve": "amdpve", "vault_key": "KOBY_LITELLM_API_KEY"},
else: "koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root", "pve": "amdpve", "vault_key": "KOONIMO_LITELLM_API_KEY"},
AGENTS[agent_name]["key"] = None "abiba": {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "hwepve", "vault_key": None}, # Pi agent, no vault key
}
GPU_HOSTS = { GPU_HOSTS = {
"gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"}, "gpu-rtx3090 (.8)": {"host": "192.168.68.8", "port": 8080, "service": "llama-server"},
@@ -74,6 +55,11 @@ GPU_HOSTS = {
FAIL = [] FAIL = []
INFISICAL_TOKEN = os.environ.get("INFISICAL_TOKEN")
INFISICAL_API_URL = os.environ.get("INFISICAL_API_URL", "https://vault.sysloggh.net")
# ── Helpers ──────────────────────────────────────────────────────────
def ssh(host, cmd, user="root"): def ssh(host, cmd, user="root"):
"""Execute a command on a remote host, return stdout or None.""" """Execute a command on a remote host, return stdout or None."""
try: try:
@@ -112,15 +98,81 @@ def http_json(url, headers=None, timeout=5):
except: except:
return None return None
def run_infisical(args, quiet=True):
"""Run infisical CLI with env-based auth, return stdout or None."""
env = os.environ.copy()
env["INFISICAL_API_URL"] = INFISICAL_API_URL
if INFISICAL_TOKEN:
env["INFISICAL_TOKEN"] = INFISICAL_TOKEN
try:
result = subprocess.run(
["/usr/bin/infisical"] + args,
capture_output=True, text=True, timeout=15, env=env
)
return result.stdout.strip() if result.returncode == 0 else None
except:
return None
# ── KEY LOOKUP FIX ───────────────────────────────────────────────────
def _get_agent_key(agent_name, vault_key_name):
"""Retrieve agent-specific key from Infisical vault.
Uses {NAME}_LITELLM_API_KEY format (e.g., TANKO_LITELLM_API_KEY,
KOONIMO_LITELLM_API_KEY) which matches actual vault key names.
"""
if not vault_key_name:
return None
# Primary: get the agent-specific key by name
key = run_infisical([
"secrets", "get", vault_key_name,
"--projectId=" + INFISICAL_PROJECT,
"--env=" + INFISICAL_ENV,
"--plain",
])
if key and key.startswith("sk-"):
return key
# Fallback: export all and search for the key name
try:
export = run_infisical([
"export",
"--projectId=" + INFISICAL_PROJECT,
"--env=" + INFISICAL_ENV,
"--format=dotenv",
])
if export:
for line in export.splitlines():
if line.startswith(vault_key_name + "="):
value = line.split("=", 1)[1].strip().strip('"').strip("'")
if value.startswith("sk-"):
return value
except:
pass
return None
# Inject keys from vault for each agent
for agent_name in AGENTS:
info = AGENTS[agent_name]
key = _get_agent_key(agent_name, info.get("vault_key"))
AGENTS[agent_name]["key"] = key
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# CHECK 1: LiteLLM Key Validation # CHECK 1: LiteLLM Key Validation (agent-specific keys)
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
def check_keys(): def check_keys():
for name, agent in AGENTS.items(): for name, agent in AGENTS.items():
key = agent.get("key")
if not key:
print(f"{name}: NO KEY FOUND (vault empty or unreachable)")
FAIL.append(f"key:{name}:no-key")
continue
data = http_json(f"{LITELLM}/v1/models", data = http_json(f"{LITELLM}/v1/models",
headers={"Authorization": f"Bearer {agent['key']}"}) headers={"Authorization": f"Bearer {key}"})
if data and data.get("data"): if data and data.get("data"):
model = data["data"][0].get("id", "?") model = data["data"][0].get("id", "?")
print(f"{name}: key valid → {model}") print(f"{name}: key valid → {model}")
@@ -130,7 +182,7 @@ def check_keys():
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# CHECK 2: GPU Port Conflict Detection # CHECK 2: GPU Port Conflict Detection (unchanged)
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
def check_gpu_ports(): def check_gpu_ports():
@@ -158,7 +210,6 @@ def check_gpu_ports():
else: else:
print(f" ⚠️ {label}: svc={svc_status}, port owned by {port_owner}") print(f" ⚠️ {label}: svc={svc_status}, port owned by {port_owner}")
else: else:
# Verify health endpoint
health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health") health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
if health and '"status":"ok"' in health: if health and '"status":"ok"' in health:
print(f"{label}: healthy (pid={port_owner})") print(f"{label}: healthy (pid={port_owner})")
@@ -171,7 +222,7 @@ def check_gpu_ports():
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# CHECK 3: Agent Gateway Liveness + Streaming # CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
def check_agents(): def check_agents():
@@ -184,8 +235,11 @@ def check_agents():
print(f"{name} (CT {ct}): cannot SSH — skip liveness check") print(f"{name} (CT {ct}): cannot SSH — skip liveness check")
continue continue
# Gateway process (exclude the infisical bash wrapper that contains the same string) # Gateway process
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user) pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
if not pid:
# Try alternate binary name
pid = ssh(host, "pgrep -f 'hermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
if not pid: if not pid:
print(f"{name}: GATEWAY NOT RUNNING") print(f"{name}: GATEWAY NOT RUNNING")
FAIL.append(f"gateway-down:{name}") FAIL.append(f"gateway-down:{name}")
@@ -203,7 +257,7 @@ def check_agents():
else: else:
gw_state, zulip = "no-state-file", "?" gw_state, zulip = "no-state-file", "?"
# Zulip streaming: does adapter have edit_message? # Zulip streaming check
adapter_paths = [ adapter_paths = [
"~/.hermes/plugins/zulip-platform/adapter.py", "~/.hermes/plugins/zulip-platform/adapter.py",
"~/.hermes/plugins/platforms/zulip/adapter.py", "~/.hermes/plugins/platforms/zulip/adapter.py",
@@ -220,13 +274,183 @@ def check_agents():
r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null " r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0", r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
user=user) user=user)
recent_errors = (recent_errors or "0").strip().split("\n")[-1] # take last line recent_errors = (recent_errors or "0").strip().split("\n")[-1]
print(f" {'' if gw_state == 'running' and zulip == 'connected' else '⚠️'} " print(f" {'' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} " f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
f"errors_10m={recent_errors.strip() or '0'} pid={pid}") f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 4: CT Liveness (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_ct_liveness():
"""Check that all agent CTs are running on their PVE nodes."""
for name, agent in AGENTS.items():
ct = agent["ct"]
pve_node = agent.get("pve")
if not pve_node:
print(f"{name} (CT {ct}): no PVE node mapped — skip")
continue
pve_ip = PVE_NODES.get(pve_node)
if not pve_ip:
print(f"{name}: unknown PVE node '{pve_node}' — skip")
continue
status = ssh(pve_ip, f"pct status {ct} 2>/dev/null", user="root")
if not status:
print(f"{name} (CT {ct} on {pve_node}): PVE UNREACHABLE")
FAIL.append(f"ct-unreachable:{name}:{pve_ip}")
elif "running" in status:
print(f"{name} (CT {ct} on {pve_node}): running")
elif "stopped" in status:
print(f"{name} (CT {ct} on {pve_node}): STOPPED")
FAIL.append(f"ct-stopped:{name}")
else:
print(f" ⚠️ {name} (CT {ct} on {pve_node}): {status.strip()}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 5: Config YAML Integrity (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_config_integrity():
"""Verify agent config.yaml parses as valid YAML."""
for name, agent in AGENTS.items():
host = agent.get("host")
user = agent.get("user")
if not host or not user:
print(f"{name}: cannot SSH — skip config check")
continue
# Check YAML parses
yaml_ok = ssh(host,
"python3 -c "
'"import yaml; yaml.safe_load(open(\'/root/.hermes/config.yaml\')); print(\'OK\')" '
"2>&1 || echo 'FAIL'",
user=user)
if not yaml_ok:
print(f"{name}: SSH UNREACHABLE (config check skipped)")
FAIL.append(f"config-unreachable:{name}")
elif "OK" in yaml_ok:
print(f"{name}: config.yaml valid YAML")
else:
print(f"{name}: config.yaml YAML ERROR — {yaml_ok[:120]}")
FAIL.append(f"config-yaml-error:{name}")
# ═══════════════════════════════════════════════════════════════════
# CHECK 6: Wrapper/CLI Integrity (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_wrapper_integrity():
"""Verify the hermes CLI wrapper exists and can reach hermes-real."""
for name, agent in AGENTS.items():
host = agent.get("host")
user = agent.get("user")
if not host or not user:
print(f"{name}: cannot SSH — skip wrapper check")
continue
# Check wrapper exists
wrapper = ssh(host, "ls -la /root/.local/bin/hermes 2>/dev/null", user=user)
if not wrapper:
# Check alternate wrapper locations
wrapper = ssh(host, "which hermes 2>/dev/null; command -v hermes 2>/dev/null", user=user)
if not wrapper:
print(f"{name}: NO HERMES CLI WRAPPER FOUND")
FAIL.append(f"wrapper-missing:{name}")
continue
else:
print(f" ⚠️ {name}: hermes at {wrapper.strip()} (not ~/.local/bin/hermes)")
# Check wrapper has correct infisical path
infisical_path_valid = ssh(host,
"head -20 /root/.local/bin/hermes 2>/dev/null | grep -q '/usr/bin/infisical' && echo OK || echo MISS",
user=user)
if infisical_path_valid == "MISS":
# Check if infisical exists on path
inf_actual = ssh(host, "command -v infisical 2>/dev/null", user=user)
if not inf_actual:
print(f"{name}: INFISICAL NOT INSTALLED (wrapper broken)")
FAIL.append(f"wrapper-no-infisical:{name}")
else:
print(f" ⚠️ {name}: wrapper infisical path may be wrong (infisical at {inf_actual})")
FAIL.append(f"wrapper-infisical-path:{name}")
# Check hermes-real exists
hermes_real = ssh(host,
"ls -la /root/.local/bin/hermes-real 2>/dev/null || echo MISS",
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
# Check venv path
hermes_real = ssh(host,
"ls -la /usr/local/lib/hermes-agent/venv/bin/hermes 2>/dev/null || echo MISS",
user=user)
if not hermes_real or hermes_real.strip() == "MISS":
print(f"{name}: hermes-real NOT FOUND (wrapper broken)")
FAIL.append(f"wrapper-no-hermes-real:{name}")
else:
print(f"{name}: hermes-real at alt path")
# Check the .env file has the key
env_has_key = ssh(host,
"grep -c 'LITELLM_API_KEY' /root/.hermes/.env 2>/dev/null || echo 0",
user=user)
if env_has_key and env_has_key.strip() not in ("", "0"):
print(f"{name}: wrapper + .env key present")
else:
print(f" ⚠️ {name}: .env may be missing LITELLM_API_KEY entry")
# ═══════════════════════════════════════════════════════════════════
# CHECK 7: Vault Secret Non-Emptiness (NEW)
# ═══════════════════════════════════════════════════════════════════
def check_vault_secrets():
"""Verify agent-specific vault secrets are non-empty and start with sk-."""
for name, agent in AGENTS.items():
vault_key_name = agent.get("vault_key")
if not vault_key_name:
continue
key = agent.get("key")
if not key:
print(f"{name}: vault secret {vault_key_name} MISSING or EMPTY")
FAIL.append(f"vault-empty:{name}:{vault_key_name}")
elif not key.startswith("sk-"):
print(f"{name}: vault secret {vault_key_name} WRONG FORMAT (starts '{key[:8]}...')")
FAIL.append(f"vault-bad-format:{name}:{vault_key_name}")
else:
print(f"{name}: vault {vault_key_name}=sk-...{key[-4:]}")
# ═══════════════════════════════════════════════════════════════════
# DEPLOY: copy updated script to /root/scripts/ on local host
# ═══════════════════════════════════════════════════════════════════
def deploy_self():
"""Copy this script to /root/scripts/agent-health-check.py if out of date."""
dest = "/root/scripts/agent-health-check.py"
try:
with open(__file__, "r") as f:
current = f.read()
if os.path.isfile(dest):
with open(dest, "r") as f:
existing = f.read()
if current == existing:
return # Already deployed
# Write new version
with open(dest, "w") as f:
f.write(current)
os.chmod(dest, 0o755)
print(f" 📦 Deployed updated script to {dest}")
except:
pass # Not fatal if deploy fails
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
# MAIN # MAIN
# ═══════════════════════════════════════════════════════════════════ # ═══════════════════════════════════════════════════════════════════
@@ -235,8 +459,12 @@ def main():
quiet = "--quiet" in sys.argv quiet = "--quiet" in sys.argv
as_json = "--json" in sys.argv as_json = "--json" in sys.argv
# Self-deploy to canonical location
if not quiet and "--no-deploy" not in sys.argv:
deploy_self()
if not quiet: if not quiet:
print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}") print(f"🏥 Agent Health Check v2 {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
print() print()
print("🔑 LiteLLM Keys:") print("🔑 LiteLLM Keys:")
@@ -249,11 +477,26 @@ def main():
print("🤖 Agent Gateways:") print("🤖 Agent Gateways:")
check_agents() check_agents()
print()
print("🖥️ CT Liveness:")
check_ct_liveness()
print()
print("📝 Config Integrity:")
check_config_integrity()
print()
print("🔌 Wrapper/CLI Integrity:")
check_wrapper_integrity()
print()
print("🔐 Vault Secrets:")
check_vault_secrets()
if FAIL: if FAIL:
print(f"\n{len(FAIL)} FAILURE(S): {' | '.join(FAIL)}") print(f"\n{len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
if quiet: if quiet:
# In quiet mode, only print failures as a single alert line
print(f"ALERT agent-health:{','.join(FAIL)}") print(f"ALERT agent-health:{','.join(FAIL)}")
elif not quiet: elif not quiet:
print("\n✅ All checks passed") print("\n✅ All checks passed")