Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aebc98ead6 | ||
|
|
17a77e6b3f | ||
|
|
31ec70ae36 | ||
|
|
5eb6d3bfbd | ||
|
|
33cb88d571 |
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
|
|||||||
"models": [
|
"models": [
|
||||||
{ "id": "syslog-auto" },
|
{ "id": "syslog-auto" },
|
||||||
{ "id": "strix-moe" },
|
{ "id": "strix-moe" },
|
||||||
|
{ "id": "gpu-dense" },
|
||||||
|
{ "id": "gpu-light" },
|
||||||
{ "id": "qwen3.6-27B-code" },
|
{ "id": "qwen3.6-27B-code" },
|
||||||
{ "id": "gemma-4-12b" }
|
{ "id": "gemma-4-12b" }
|
||||||
]
|
]
|
||||||
|
|||||||
@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
|
|||||||
|
|
||||||
One-line agent health check (run on the agent host):
|
One-line agent health check (run on the agent host):
|
||||||
```bash
|
```bash
|
||||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||||
|
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||||
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
||||||
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
||||||
```
|
```
|
||||||
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
|
|||||||
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
||||||
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
||||||
|
|
||||||
|
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
|
||||||
|
the wrapper will inject an empty key and the gateway will silently get 401 errors
|
||||||
|
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
|
||||||
|
is present but the vault takes precedence when the secret key exists (even if empty).
|
||||||
|
|
||||||
|
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
|
||||||
|
is empty or shorter than 20 chars, log a warning and either fail with a clear error
|
||||||
|
message or fall back to the `.env` value before starting the gateway.
|
||||||
|
|
||||||
**Verification (all agents):**
|
**Verification (all agents):**
|
||||||
```bash
|
```bash
|
||||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||||
|
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||||
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
||||||
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -8,18 +8,18 @@ description: >
|
|||||||
Ensures agents never use the master key directly. Rotation is event-driven,
|
Ensures agents never use the master key directly. Rotation is event-driven,
|
||||||
not calendar-driven — rotate only on compromise, personnel change, or
|
not calendar-driven — rotate only on compromise, personnel change, or
|
||||||
periodic security hygiene (quarterly/annually).
|
periodic security hygiene (quarterly/annually).
|
||||||
|
|
||||||
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
|
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
|
||||||
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
|
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
|
||||||
expire/404. The .env fallback prevents agents from running without keys.
|
expire/404. The .env fallback prevents agents from running without keys.
|
||||||
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
||||||
|
|
||||||
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
|
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
|
||||||
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
|
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
|
||||||
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
|
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
|
||||||
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
|
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
|
||||||
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
|
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
|
||||||
|
|
||||||
UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
|
UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
|
||||||
standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
|
standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
|
||||||
infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
|
infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
|
||||||
@@ -30,7 +30,7 @@ description: >
|
|||||||
Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
|
Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
|
||||||
— hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
|
— hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
|
||||||
(3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
|
(3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
|
||||||
|
|
||||||
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
||||||
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
||||||
on CT 116. Last verified: 2026-07-17.
|
on CT 116. Last verified: 2026-07-17.
|
||||||
@@ -154,7 +154,7 @@ through its agent wrapper.
|
|||||||
The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
|
The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
|
||||||
then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
|
then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
|
||||||
regeneration** by `hermes gateway install` — the drop-in always wins.
|
regeneration** by `hermes gateway install` — the drop-in always wins.
|
||||||
|
|
||||||
**Why a drop-in instead of editing the unit file:** `hermes gateway install`
|
**Why a drop-in instead of editing the unit file:** `hermes gateway install`
|
||||||
(called during Hermes updates and some self-heal operations) regenerates the
|
(called during Hermes updates and some self-heal operations) regenerates the
|
||||||
systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
|
systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
|
||||||
@@ -171,7 +171,7 @@ through its agent wrapper.
|
|||||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
||||||
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
||||||
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
||||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
|
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
|
||||||
|
|
||||||
### Migration status (2026-07-17)
|
### Migration status (2026-07-17)
|
||||||
|
|
||||||
|
|||||||
@@ -184,8 +184,8 @@ def check_agents():
|
|||||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Gateway process
|
# Gateway process (exclude the infisical bash wrapper that contains the same string)
|
||||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
|
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||||
if not pid:
|
if not pid:
|
||||||
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
||||||
FAIL.append(f"gateway-down:{name}")
|
FAIL.append(f"gateway-down:{name}")
|
||||||
|
|||||||
Reference in New Issue
Block a user