Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aebc98ead6 | ||
|
|
17a77e6b3f | ||
|
|
31ec70ae36 | ||
|
|
5eb6d3bfbd | ||
|
|
33cb88d571 |
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
|
||||
"models": [
|
||||
{ "id": "syslog-auto" },
|
||||
{ "id": "strix-moe" },
|
||||
{ "id": "gpu-dense" },
|
||||
{ "id": "gpu-light" },
|
||||
{ "id": "qwen3.6-27B-code" },
|
||||
{ "id": "gemma-4-12b" }
|
||||
]
|
||||
|
||||
@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
|
||||
|
||||
One-line agent health check (run on the agent host):
|
||||
```bash
|
||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
||||
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
||||
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
||||
```
|
||||
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
|
||||
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
||||
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
||||
|
||||
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
|
||||
the wrapper will inject an empty key and the gateway will silently get 401 errors
|
||||
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
|
||||
is present but the vault takes precedence when the secret key exists (even if empty).
|
||||
|
||||
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
|
||||
is empty or shorter than 20 chars, log a warning and either fail with a clear error
|
||||
message or fall back to the `.env` value before starting the gateway.
|
||||
|
||||
**Verification (all agents):**
|
||||
```bash
|
||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
||||
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
||||
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
||||
```
|
||||
|
||||
@@ -8,18 +8,18 @@ description: >
|
||||
Ensures agents never use the master key directly. Rotation is event-driven,
|
||||
not calendar-driven — rotate only on compromise, personnel change, or
|
||||
periodic security hygiene (quarterly/annually).
|
||||
|
||||
|
||||
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
|
||||
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
|
||||
expire/404. The .env fallback prevents agents from running without keys.
|
||||
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
||||
|
||||
|
||||
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
|
||||
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
|
||||
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
|
||||
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
|
||||
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
|
||||
|
||||
|
||||
UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
|
||||
standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
|
||||
infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
|
||||
@@ -30,7 +30,7 @@ description: >
|
||||
Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
|
||||
— hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
|
||||
(3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
|
||||
|
||||
|
||||
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
||||
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
||||
on CT 116. Last verified: 2026-07-17.
|
||||
@@ -154,7 +154,7 @@ through its agent wrapper.
|
||||
The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
|
||||
then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
|
||||
regeneration** by `hermes gateway install` — the drop-in always wins.
|
||||
|
||||
|
||||
**Why a drop-in instead of editing the unit file:** `hermes gateway install`
|
||||
(called during Hermes updates and some self-heal operations) regenerates the
|
||||
systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
|
||||
@@ -171,7 +171,7 @@ through its agent wrapper.
|
||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
||||
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
||||
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
|
||||
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
|
||||
|
||||
### Migration status (2026-07-17)
|
||||
|
||||
|
||||
@@ -184,8 +184,8 @@ def check_agents():
|
||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||
continue
|
||||
|
||||
# Gateway process
|
||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
|
||||
# Gateway process (exclude the infisical bash wrapper that contains the same string)
|
||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid:
|
||||
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
||||
FAIL.append(f"gateway-down:{name}")
|
||||
|
||||
Reference in New Issue
Block a user