diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 569755b..02b9f17 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup: "models": [ { "id": "syslog-auto" }, { "id": "strix-moe" }, + { "id": "gpu-dense" }, + { "id": "gpu-light" }, { "id": "qwen3.6-27B-code" }, { "id": "gemma-4-12b" } ] diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 693026a..a5bd9c0 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes One-line agent health check (run on the agent host): ```bash -PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1) +# Use grep -v infisical to avoid matching the bash wrapper that contains the same string +PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1) cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*//' curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models ``` @@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$_LITELLM_API_KEY"`. See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync. +**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string, +the wrapper will inject an empty key and the gateway will silently get 401 errors +on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback +is present but the vault takes precedence when the secret key exists (even if empty). + +**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY +is empty or shorter than 20 chars, log a warning and either fail with a clear error +message or fall back to the `.env` value before starting the gateway. + **Verification (all agents):** ```bash -GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1) +# Use grep -v infisical to avoid matching the bash wrapper that contains the same string +GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1) K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2) curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200 ``` diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index f8b6ef6..44624f6 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -171,7 +171,7 @@ through its agent wrapper. - **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense. - **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection. - **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session. -- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source. +- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source. ### Migration status (2026-07-17) diff --git a/scripts/agent-health-check.py b/scripts/agent-health-check.py index 97d1dce..3322b45 100755 --- a/scripts/agent-health-check.py +++ b/scripts/agent-health-check.py @@ -184,8 +184,8 @@ def check_agents(): print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check") continue - # Gateway process - pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user) + # Gateway process (exclude the infisical bash wrapper that contains the same string) + pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user) if not pid: print(f" ❌ {name}: GATEWAY NOT RUNNING") FAIL.append(f"gateway-down:{name}")