fix: fleet config issues from 2026-07-18 relay review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s

- hermes-agent-baseline: add gpu-dense and gpu-light to models list
- hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add
  vault empty-key guard documentation in Rule 13
- litellm-api-keys: fix pgrep pattern in auditable check
- scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper

Addresses issues found during relay inbox resolution session:
1. pgrep -f 'hermes_cli.main gateway run' matches both the real python
   gateway and the infisical bash wrapper, causing false health readings
2. infisical vault stores empty key silently — no guard/monitoring
3. gpu-dense/gpu-light stable aliases missing from baseline config
This commit is contained in:
root
2026-07-18 17:39:46 +00:00
parent 31ec70ae36
commit 17a77e6b3f
4 changed files with 18 additions and 5 deletions
+2
View File
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
"models": [ "models": [
{ "id": "syslog-auto" }, { "id": "syslog-auto" },
{ "id": "strix-moe" }, { "id": "strix-moe" },
{ "id": "gpu-dense" },
{ "id": "gpu-light" },
{ "id": "qwen3.6-27B-code" }, { "id": "qwen3.6-27B-code" },
{ "id": "gemma-4-12b" } { "id": "gemma-4-12b" }
] ]
+13 -2
View File
@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
One-line agent health check (run on the agent host): One-line agent health check (run on the agent host):
```bash ```bash
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1) # Use grep -v infisical to avoid matching the bash wrapper that contains the same string
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/' cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
``` ```
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`. The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync. See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
the wrapper will inject an empty key and the gateway will silently get 401 errors
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
is present but the vault takes precedence when the secret key exists (even if empty).
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
is empty or shorter than 20 chars, log a warning and either fail with a clear error
message or fall back to the `.env` value before starting the gateway.
**Verification (all agents):** **Verification (all agents):**
```bash ```bash
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1) # Use grep -v infisical to avoid matching the bash wrapper that contains the same string
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2) K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200 curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
``` ```
+1 -1
View File
@@ -171,7 +171,7 @@ through its agent wrapper.
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense. - **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection. - **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session. - **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source. - **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
### Migration status (2026-07-17) ### Migration status (2026-07-17)
+2 -2
View File
@@ -184,8 +184,8 @@ def check_agents():
print(f"{name} (CT {ct}): cannot SSH — skip liveness check") print(f"{name} (CT {ct}): cannot SSH — skip liveness check")
continue continue
# Gateway process # Gateway process (exclude the infisical bash wrapper that contains the same string)
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user) pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
if not pid: if not pid:
print(f"{name}: GATEWAY NOT RUNNING") print(f"{name}: GATEWAY NOT RUNNING")
FAIL.append(f"gateway-down:{name}") FAIL.append(f"gateway-down:{name}")