SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

contract: update litellm-api-keys to v2026-07-17 — fleet standardization

Browse Source 
- Document canonical systemd drop-in pattern (ExecStart= reset + wrapper)
- Hardcode venv paths — never use variables in single-quoted bash -c
- Update migration status: all 4 agents on while-true wrapper + st.8e848433
- Infisical CLI update procedure (0.38.0 → 0.43.109 via artifacts-cli)
- Service token inventory + .env fallback inventory
- Key rotation log: fleet standardize + tanko fix-zulip entries
- Remove git merge conflict artifacts
- Add fleet-wide standardization lessons section
This commit is contained in:
root
2026-07-17 10:17:23 +00:00
parent 20cbb96e2d 24cd7f72ac
commit 65c99dab50
1 changed files with 147 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
litellm-api-keys.prose.md
+147 -105
View File
@@ -20,9 +20,20 @@ description: >
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
Abiba's key is now a proper agent key (NOT the master key — stale note removed). Abiba's key is now a proper agent key (NOT the master key — stale note removed).
UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
injection → .env fallback → exec python. Systemd drop-ins are IMMUNE to hermes gateway
install which overwrites the unit file ExecStart. Infisical CLI updated to 0.43.109 on
all agents (was 0.38.0). Service token st.8e848433 shared across fleet (st.353699cd
for tanko was deleted). .env fallback on every agent protects against token loss.
Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
— hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
(3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
on CT 116. Last verified: 2026-07-16. on CT 116. Last verified: 2026-07-17.
--- ---
## Parameters ## Parameters
@@ -74,88 +85,147 @@ description: >
- Confirm key alias matches agent_name in LiteLLM key list - Confirm key alias matches agent_name in LiteLLM key list
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run` - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
## Production Vault Access Process (canonical, 2026-07-16) ## Production Vault Access Process (canonical, 2026-07-17)
The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending — The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents
runs as user `jerome`, not systemd root, needs user-scope adaptation). (Mumuni, Tanko, Koby, Koonimo) as of 2026-07-17. Abiba (pi) uses a similar pattern
through its agent wrapper.
### The canonical pattern ### The canonical pattern
1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`). 1. **infisical CLI** installed on the host at `/usr/bin/infisical` (v0.43.109+, from
2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`). artifacts-cli.infisical.com apt repo). Update procedure:
- Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project. ```bash
- Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities). curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | sudo -E bash
3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`): sudo apt-get update && sudo apt-get install -y infisical
# Remove stale old binary if present
rm -f /usr/local/bin/infisical /bin/infisical
```
Wrappers use absolute path `/usr/bin/infisical run`. Never rely on PATH resolution.
2. **Service token** (Infisical Machine Identity, `st.…`) stored at `~/.infisical-token`
(`chmod 600`). Current: shared `st.8e848433…` (abiba, READ+WRITE on agents project).
Tanko's `st.353699cd…` (tanko-agent) was deleted — reverted to shared token.
Proper: one machine identity per agent (create in Infisical UI → Project Settings →
Machine Identities).
3. **`infisical-gateway.sh` wrapper** at `~/.hermes/infisical-gateway.sh` (`chmod 700`):
```bash ```bash
#!/bin/bash #!/bin/bash
export INFISICAL_API_URL="https://vault.sysloggh.net" export INFISICAL_API_URL="https://vault.sysloggh.net"
TOKEN=$(cat /root/.infisical-token) TOKEN=$(cat $HOME/.infisical-token)
LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs LOG=$HOME/.hermes/logs/gateway.log; mkdir -p $HOME/.hermes/logs
while true; do while true; do
infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \ echo "[$(date -Iseconds)] Starting gateway with Infisical injection..." >> $LOG
/usr/bin/infisical run --token="$TOKEN" \
--projectId=322fceab-39da-4854-a55a-568e76c0f13f \
--env=prod --domain=https://vault.sysloggh.net -- bash -c ' --env=prod --domain=https://vault.sysloggh.net -- bash -c '
. /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only . $HOME/.hermes/.env 2>/dev/null # [FALLBACK Rule 3]
export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY" export LITELLM_API_KEY="${<AGENT>_LITELLM_API_KEY}"
exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run export ZULIP_API_KEY="${<AGENT>_ZULIP_API_KEY}"
export ZULIP_SITE="https://chat.sysloggh.net"
export ZULIP_EMAIL="<agent>-bot@chat.sysloggh.net"
export SEARXNG_URL="http://192.168.68.7:8888"
# ⚠️ HARDCODE the full venv path. NEVER use $VENV inside single quotes.
exec /root/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
' >> $LOG 2>&1 ' >> $LOG 2>&1
sleep 5 # restart on exit EXIT_CODE=$?
echo "[$(date -Iseconds)] Gateway exited with code $EXIT_CODE — restarting in 5s..." >> $LOG
sleep 5
done done
``` ```
4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth. **CRITICAL: VENV PATH.** The inner `bash -c '...'` uses single quotes. Shell
5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation. variables set in the outer wrapper are NOT expanded inside single quotes.
6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot). `$VENV/bin/python` resolves to `/bin/python` (file not found). Always hardcode
7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault. the absolute path to the venv python binary.
4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` and `<AGENT>_ZULIP_API_KEY`.
Vault = source of truth for ALL platform credentials.
5. **`.env` fallback** at `~/.hermes/.env` (`chmod 600`) with agent-specific keys —
safety net for vault outage or token revocation. Must be kept in sync on rotation.
Example:
```bash
MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
```
6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
```ini
[Service]
ExecStart=
ExecStart=/root/.hermes/infisical-gateway.sh
```
The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
regeneration** by `hermes gateway install` — the drop-in always wins.
**Why a drop-in instead of editing the unit file:** `hermes gateway install`
(called during Hermes updates and some self-heal operations) regenerates the
systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
Editing the unit file directly is futile — it will be overwritten. The drop-in
approach explicitly resets ExecStart and sets the wrapper regardless of what the
main unit file says.
7. **NEVER hardcode** API keys in systemd drop-ins, config.yaml, or /etc/environment.
The wrapper injects live from vault at every start.
### Why this is non-fail ### Why this is non-fail
- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits. - **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable. - **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable or the service token is revoked.
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway. - **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source. - **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
### Migration status (2026-07-16) ### Migration status (2026-07-17)
| Agent | Host | Pattern | Vault key | Status | | Agent | Host | Pattern | Keys | Status |
|-------|------|---------|-----------|--------| |-------|------|---------|------|--------|
| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed | | abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed | | mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram | | tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed | | koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent. | koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
<<<<<<< HEAD
| tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed |
=======
| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
>>>>>>> origin/master
### Tanko migration (pending) > Tanko runs as user `jerome` — wrapper/token at `~/.hermes/infisical-gateway.sh` and
> `~/.infisical-token`. Linger enabled (`loginctl enable-linger jerome`) for boot startup.
Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in ### Tanko migration (COMPLETED 2026-07-17)
`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
### Koby migration lessons (2026-07-16) Tanko was the last agent migrated from hardcoded keys to vault wrapper.
Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
`50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
`~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
### Koby migration lessons (2026-07-16, updated 2026-07-17)
Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper. Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
**Two mistakes I made that broke the agent:** **Three mistakes made:**
1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed 1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
inherited from the pre-migration gateway env, not stored in any file. Lost on restart. inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials. 2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
Agents need ALL their platform env vars. Missing vars cause silent adapter failures. Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"`
inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path.
**How Koby actually connects (2026-07-16):** **How Koby actually connects:**
- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`). - Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config). - Telegram: token from `.env` fallback. Allowed users: 6679773481.
- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
Allowed users: 6679773481. Home channel: 6679773481.
- Both platforms now connect through the wrapper's env injection. - Both platforms now connect through the wrapper's env injection.
**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old **Golden rules for gateway restarts:**
process — captures the live env set. Especially important when migrating gateways between 1. Always `cat /proc/<pid>/environ` before killing the old process — captures the live env set.
injection mechanisms. 2. Hardcode venv python path in wrapper — never use variables inside single-quoted bash -c.
3. Use systemd drop-ins (not unit file edits) to override ExecStart — survives Hermes updates.
### Fleet-wide standardization lessons (2026-07-17)
After auditing all 4 agents, five systemic patterns caused repeated failures:
1. **Three incompatible startup patterns** coexisted (systemd drop-in, direct python, orphaned wrapper)
2. **Systemd unit files reverted** by `hermes gateway install` during updates
3. **VENV variable scoping** broke wrappers on Koby and Mumuni (single-quote bash -c)
4. **Service token expiry** — Tanko's `st.353699cd` was deleted from Infisical
5. **No ZULIP_API_KEY** in env on Tanko — wrapper bypassed by systemd direct python
All resolved by the canonical drop-in + while-true wrapper pattern documented above.
### Key rotation procedure (one vault operation with this standard) ### Key rotation procedure (one vault operation with this standard)
@@ -165,69 +235,41 @@ injection mechanisms.
4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live. 4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200. 5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300) ## Machine Identity for Vault Writes (UPDATED 2026-07-17)
**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com). **Current state:** Infisical CLI updated to v0.43.109 on all agents (from v0.38.0).
In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace The v0.38.0 bug (user-session auth fails for `secrets set`/`export`) is resolved.
key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt Service token `st.8e848433…` (abiba, READ+WRITE) can write to vault from CLI.
repo only ships 0.38.0, so `apt upgrade` does not help.
**Proper fix — Machine Identity (Infisical automation best practice):** **Proper fix — per-agent Machine Identities:**
Create a machine identity with READ+WRITE scope on the `agents` project (project_id= Create machine identities in Infisical UI → Project Settings → Machine Identities
`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely. for each agent with READ-only scope on the `agents` project. Store client_id +
Then vault writes work from any host: client_secret per agent. Then vault writes use the shared abiba identity, and
```bash reads use per-agent identities. This eliminates the single shared token risk.
# Get a machine-identity access token
TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
-H 'Content-Type: application/json' \
-d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
# Write a secret via REST API v3
curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
```
<<<<<<< HEAD
**2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE)
can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM.
4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from
hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored.
**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI. **Service Token Inventory (2026-07-17):**
Client secret needed to use universal auth for vault writes from automation. | Token ID | Name | Permissions | Used By | Status |
|----------|------|-------------|---------|--------|
| `st.8e848433…` | tanko-gateway | READ+WRITE | Mumuni, Tanko, Koby, Koonimo, Abiba | ✅ Active |
| `st.353699cd…` | tanko-agent | READ-only | — | ❌ Deleted from Infisical |
**Service Token Inventory (2026-07-16):** **Per-agent .env fallback inventory (2026-07-17):**
| Token ID | Name | Permissions | Used By | | Agent | .env Keys |
|----------|------|-------------|---------| |-------|-----------|
| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo | | Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY |
| `st.353699cd…` | tanko-agent | READ-only | Tanko | | Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
======= | Koby | (wrapper injects from vault — .env has Telegram token) |
Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings → | Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |
Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
and store its credentials in the vault itself (or a root-only file).**
**Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `<AGENT>_LITELLM_API_KEY`
overrides a stale vault value.
**2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE)
can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`,
koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` /
`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
so the shared service token isn't reused across hosts — but it is no longer blocking.
>>>>>>> origin/master
## Key Rotation Log ## Key Rotation Log
| Date | Agent | Action | Notes | | Date | Agent | Action | Notes |
|------|-------|--------|-------| |------|-------|--------|-------|
<<<<<<< HEAD | 2026-07-17 | fleet | standardize | All 4 agents standardized on systemd drop-in + while-true wrapper + infisical v0.43.109. Removed conflicting zulip-env.conf + litellm-key.conf drop-ins. Added .env fallbacks with ZULIP keys. WAL #1322. |
| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. | | 2026-07-17 | tanko | fix-zulip | Added ZULIP_API_KEY to env (was missing — systemd bypassed vault). Updated wrapper from exec to while-true. Created .env fallback. Removed hardcoded zulip-env.conf drop-in. WAL #1321. |
| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. | | 2026-07-16 | vault | cleanup | 4 stale secrets deprecated. 5 personal creds flagged. |
| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. | | 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault. Wrapper injects ZULIP_API_KEY + ZULIP_EMAIL. 3 platforms. |
======= | 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml to infisical-gateway.sh + st.353699cd. NOTE: st.353699cd later deleted — reverted to st.8e848433 on 2026-07-17. |
>>>>>>> origin/master
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. | | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. | | 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. | | 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |