vault: canonical Production Vault Access Process + sync vault + migrate koby/koonimo
- Vault SYNCED: mumuni/koby/koonimo rotated keys written to Infisical (all validate 200). Vault is source of truth again (was stale since session-13 rotations). - Koby (.129) + Koonimo (.114) migrated from hardcoded systemd drop-ins to infisical-gateway.sh wrapper (live vault injection, .env fallback safety net). 4/5 agents now vault-backed (abiba/mumuni/koby/koonimo). - New § Production Vault Access Process: canonical 7-step pattern, migration status table, key rotation procedure, why-it's-non-fail rationale. - Fixed stale notes: 'vault sync PENDING' (now synced), 'Abiba key IS master key' (now a proper agent key sk-sxbphLvk1OU). - Tanko (user jerome, not systemd) documented as pending migration. - hermes-key-enforcement: cross-ref to canonical process, drop-ins deprecated.
This commit is contained in:
Abiba
@@ -14,9 +14,15 @@ description: >
|
||||
expire/404. The .env fallback prevents agents from running without keys.
|
||||
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
||||
|
||||
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
|
||||
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
|
||||
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
|
||||
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
|
||||
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
|
||||
|
||||
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
||||
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
||||
on CT 116. Last verified: 2026-07-12.
|
||||
on CT 116. Last verified: 2026-07-16.
|
||||
---
|
||||
|
||||
## Parameters
|
||||
@@ -68,6 +74,71 @@ description: >
|
||||
- Confirm key alias matches agent_name in LiteLLM key list
|
||||
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
|
||||
|
||||
## Production Vault Access Process (canonical, 2026-07-16)
|
||||
|
||||
The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
|
||||
runs as user `jerome`, not systemd root, needs user-scope adaptation).
|
||||
|
||||
### The canonical pattern
|
||||
|
||||
1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
|
||||
2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
|
||||
- Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
|
||||
- Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
|
||||
3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
|
||||
```bash
|
||||
#!/bin/bash
|
||||
export INFISICAL_API_URL="https://vault.sysloggh.net"
|
||||
TOKEN=$(cat /root/.infisical-token)
|
||||
LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
|
||||
while true; do
|
||||
infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
|
||||
--env=prod --domain=https://vault.sysloggh.net -- bash -c '
|
||||
. /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only
|
||||
export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
|
||||
exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
|
||||
' >> $LOG 2>&1
|
||||
sleep 5 # restart on exit
|
||||
done
|
||||
```
|
||||
4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
|
||||
5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
|
||||
6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
|
||||
7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
|
||||
|
||||
### Why this is non-fail
|
||||
|
||||
- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
|
||||
- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
|
||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
|
||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
|
||||
|
||||
### Migration status (2026-07-16)
|
||||
|
||||
| Agent | Host | Pattern | Vault key | Status |
|
||||
|-------|------|---------|-----------|--------|
|
||||
| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
|
||||
| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
|
||||
| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed |
|
||||
| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
|
||||
| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
|
||||
|
||||
### Tanko migration (pending)
|
||||
|
||||
Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
|
||||
`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
|
||||
Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
|
||||
with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
|
||||
enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
|
||||
|
||||
### Key rotation procedure (one vault operation with this standard)
|
||||
|
||||
1. Generate new key: `POST /key/generate` (master key, admin).
|
||||
2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
|
||||
3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
|
||||
4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
|
||||
5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
|
||||
|
||||
## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
|
||||
|
||||
**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
|
||||
@@ -95,9 +166,15 @@ Machine Identities, or an admin API call. **TODO: create `abiba-automation` mach
|
||||
and store its credentials in the vault itself (or a root-only file).**
|
||||
|
||||
**Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
|
||||
infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `MUMUNI_LITELLM_API_KEY` overrides
|
||||
the stale vault value. This is fully functional and contract-sanctioned — the vault sync above is
|
||||
only for consistency so .env and vault never drift.
|
||||
infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `<AGENT>_LITELLM_API_KEY`
|
||||
overrides a stale vault value.
|
||||
|
||||
**2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE)
|
||||
can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`,
|
||||
koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` /
|
||||
`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
|
||||
Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
|
||||
so the shared service token isn't reused across hosts — but it is no longer blocking.
|
||||
|
||||
## Key Rotation Log
|
||||
|
||||
@@ -111,5 +188,8 @@ only for consistency so .env and vault never drift.
|
||||
|
||||
- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
|
||||
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
|
||||
- **Known violation:** Abiba's LITELLM_API_KEY IS the master key (should be a dedicated `abiba` virtual key). TODO: generate `abiba` virtual key and stop using master key directly.
|
||||
- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
|
||||
It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
|
||||
The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
|
||||
see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
|
||||
- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`
|
||||
|
||||
Reference in New Issue
Block a user