fix: fleet config issues from 2026-07-18 relay review #23

Merged
jerome merged 3 commits from fix/fleet-config-issues-20260718 into master 2026-07-18 22:06:49 +00:00
4 changed files with 18 additions and 5 deletions
Showing only changes of commit 17a77e6b3f - Show all commits
+2
View File
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
"models": [
{ "id": "syslog-auto" },
{ "id": "strix-moe" },
{ "id": "gpu-dense" },
{ "id": "gpu-light" },
{ "id": "qwen3.6-27B-code" },
{ "id": "gemma-4-12b" }
]
+13 -2
View File
@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
One-line agent health check (run on the agent host):
```bash
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
```
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
the wrapper will inject an empty key and the gateway will silently get 401 errors
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
is present but the vault takes precedence when the secret key exists (even if empty).
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
is empty or shorter than 20 chars, log a warning and either fail with a clear error
message or fall back to the `.env` value before starting the gateway.
**Verification (all agents):**
```bash
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
```
+1 -1
View File
@@ -171,7 +171,7 @@ through its agent wrapper.
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
### Migration status (2026-07-17)
+2 -2
View File
@@ -184,8 +184,8 @@ def check_agents():
print(f"{name} (CT {ct}): cannot SSH — skip liveness check")
continue
# Gateway process
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
# Gateway process (exclude the infisical bash wrapper that contains the same string)
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
if not pid:
print(f"{name}: GATEWAY NOT RUNNING")
FAIL.append(f"gateway-down:{name}")