fix: fleet config issues from 2026-07-18 relay review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- hermes-agent-baseline: add gpu-dense and gpu-light to models list - hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add vault empty-key guard documentation in Rule 13 - litellm-api-keys: fix pgrep pattern in auditable check - scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper Addresses issues found during relay inbox resolution session: 1. pgrep -f 'hermes_cli.main gateway run' matches both the real python gateway and the infisical bash wrapper, causing false health readings 2. infisical vault stores empty key silently — no guard/monitoring 3. gpu-dense/gpu-light stable aliases missing from baseline config
This commit is contained in:
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
|
||||
"models": [
|
||||
{ "id": "syslog-auto" },
|
||||
{ "id": "strix-moe" },
|
||||
{ "id": "gpu-dense" },
|
||||
{ "id": "gpu-light" },
|
||||
{ "id": "qwen3.6-27B-code" },
|
||||
{ "id": "gemma-4-12b" }
|
||||
]
|
||||
|
||||
@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
|
||||
|
||||
One-line agent health check (run on the agent host):
|
||||
```bash
|
||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
||||
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
||||
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
||||
```
|
||||
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
|
||||
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
||||
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
||||
|
||||
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
|
||||
the wrapper will inject an empty key and the gateway will silently get 401 errors
|
||||
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
|
||||
is present but the vault takes precedence when the secret key exists (even if empty).
|
||||
|
||||
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
|
||||
is empty or shorter than 20 chars, log a warning and either fail with a clear error
|
||||
message or fall back to the `.env` value before starting the gateway.
|
||||
|
||||
**Verification (all agents):**
|
||||
```bash
|
||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
||||
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
||||
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
||||
```
|
||||
|
||||
@@ -171,7 +171,7 @@ through its agent wrapper.
|
||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
||||
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
||||
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
|
||||
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
|
||||
|
||||
### Migration status (2026-07-17)
|
||||
|
||||
|
||||
@@ -184,8 +184,8 @@ def check_agents():
|
||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||
continue
|
||||
|
||||
# Gateway process
|
||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
|
||||
# Gateway process (exclude the infisical bash wrapper that contains the same string)
|
||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid:
|
||||
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
||||
FAIL.append(f"gateway-down:{name}")
|
||||
|
||||
Reference in New Issue
Block a user