fix: fleet config issues from 2026-07-18 relay review
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
- hermes-agent-baseline: add gpu-dense and gpu-light to models list - hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add vault empty-key guard documentation in Rule 13 - litellm-api-keys: fix pgrep pattern in auditable check - scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper Addresses issues found during relay inbox resolution session: 1. pgrep -f 'hermes_cli.main gateway run' matches both the real python gateway and the infisical bash wrapper, causing false health readings 2. infisical vault stores empty key silently — no guard/monitoring 3. gpu-dense/gpu-light stable aliases missing from baseline config
This commit is contained in:
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
|
|||||||
"models": [
|
"models": [
|
||||||
{ "id": "syslog-auto" },
|
{ "id": "syslog-auto" },
|
||||||
{ "id": "strix-moe" },
|
{ "id": "strix-moe" },
|
||||||
|
{ "id": "gpu-dense" },
|
||||||
|
{ "id": "gpu-light" },
|
||||||
{ "id": "qwen3.6-27B-code" },
|
{ "id": "qwen3.6-27B-code" },
|
||||||
{ "id": "gemma-4-12b" }
|
{ "id": "gemma-4-12b" }
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
|
|||||||
|
|
||||||
One-line agent health check (run on the agent host):
|
One-line agent health check (run on the agent host):
|
||||||
```bash
|
```bash
|
||||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||||
|
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||||
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
||||||
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
||||||
```
|
```
|
||||||
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
|
|||||||
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
||||||
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
||||||
|
|
||||||
|
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
|
||||||
|
the wrapper will inject an empty key and the gateway will silently get 401 errors
|
||||||
|
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
|
||||||
|
is present but the vault takes precedence when the secret key exists (even if empty).
|
||||||
|
|
||||||
|
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
|
||||||
|
is empty or shorter than 20 chars, log a warning and either fail with a clear error
|
||||||
|
message or fall back to the `.env` value before starting the gateway.
|
||||||
|
|
||||||
**Verification (all agents):**
|
**Verification (all agents):**
|
||||||
```bash
|
```bash
|
||||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||||
|
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||||
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
||||||
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -171,7 +171,7 @@ through its agent wrapper.
|
|||||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
||||||
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
||||||
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
||||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
|
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
|
||||||
|
|
||||||
### Migration status (2026-07-17)
|
### Migration status (2026-07-17)
|
||||||
|
|
||||||
|
|||||||
@@ -184,8 +184,8 @@ def check_agents():
|
|||||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Gateway process
|
# Gateway process (exclude the infisical bash wrapper that contains the same string)
|
||||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
|
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||||
if not pid:
|
if not pid:
|
||||||
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
||||||
FAIL.append(f"gateway-down:{name}")
|
FAIL.append(f"gateway-down:{name}")
|
||||||
|
|||||||
Reference in New Issue
Block a user