SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: lessons learned from 2026-07-12 session

Browse Source 
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
root
2026-07-12 22:49:39 +00:00
parent 19b6db9891
commit 79d4a73895
14 changed files with 372 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hermes-key-enforcement.prose.md
+119 -48
View File
@@ -5,8 +5,10 @@ version: 1.0.0
description: >
Enforces standardized API key configuration across all Hermes agents. Harness/LiteLLM
providers MUST use api_key_env indirection. External providers (DeepSeek, OpenAI,
Anthropic) may use hardcoded keys. Single source of truth: /etc/environment on each
agent host. Designed to make key rotation a one-step operation.
Anthropic) may use hardcoded keys. Single source of truth: Infisical vault
(project=agents, env=production) — injected at runtime via `infisical run --` wrapper.
/etc/environment is DEPRECATED for agent keys post-migration. Designed to make key
rotation a one-step vault operation.
author: Abiba (pi agent)
---
@@ -14,7 +16,7 @@ author: Abiba (pi agent)
## Rule (One Sentence)
**Any `api_key` pointing to a Syslog-hosted LiteLLM/harness provider MUST be replaced with `api_key_env: LITELLM_API_KEY` — hardcoded harness keys are forbidden.**
**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
## Scope
@@ -26,6 +28,39 @@ Applies to all Hermes agent configs across all hosts. Covers these config sectio
- `compression.api_key` (when `provider` is `harness` or contains `litellm`)
- `fallback_providers[].api_key` (when provider is harness)
## Architecture (2026-07-10)
Syslog is migrating away from **unauthenticated direct access** to the shared inference harness.
| Path | Auth | Status |
|------|------|--------|
| `http://192.168.68.116/v1` | None (direct) | ❌ **DEPRECATED** — being phased out |
| `http://192.168.68.116/litellm/v1/responses` | Bearer `sk-*` key | ✅ **CURRENT** — authenticated LiteLLM proxy |
All harness/litellm providers MUST use the authenticated `/litellm/v1/responses` path.
Any `base_url` pointing to bare `/v1` on 192.168.68.116 is a **migration violation**.
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
When `api_mode: responses` is set, Hermes **appends `/v1/responses`** to `base_url`.
If `base_url` already includes `/litellm/v1/responses`, the result is:
```
http://192.168.68.116/litellm/v1/responses/v1/responses → 404
```
**The `base_url` must end at `/v1` — never include `/responses`:**
```yaml
# ✅ CORRECT — Hermes appends /v1/responses for api_mode: responses
base_url: http://192.168.68.116/litellm/v1
# ❌ WRONG — produces double path
base_url: http://192.168.68.116/litellm/v1/responses
```
This applies to ALL sections using the harness provider: `custom_providers`, `delegation`, `auxiliary.*`.
## Exemptions
External providers are **explicitly exempt** and may use hardcoded keys:
@@ -38,35 +73,42 @@ External providers are **explicitly exempt** and may use hardcoded keys:
## Standard Pattern
```yaml
# ✅ CORRECT — all harness/litellm providers
# ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
model:
provider: harness # or custom:litellm.sysloggh.net
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # ← indirection
provider: harness
base_url: http://192.168.68.116/litellm/v1 # ← Hermes appends /v1/responses
api_key_env: LITELLM_API_KEY
custom_providers:
- name: harness
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # ← indirection
api_mode: responses
base_url: http://192.168.68.116/litellm/v1 # ← NO /responses suffix!
api_key_env: LITELLM_API_KEY
auxiliary:
compression:
provider: harness
api_key_env: LITELLM_API_KEY # ← indirection
base_url: http://192.168.68.116/litellm/v1 # ← NO /responses suffix!
api_key_env: LITELLM_API_KEY
# ✅ ALSO CORRECT — external providers
fallback_providers:
- provider: deepseek
base_url: https://api.deepseek.com
api_key: sk-b7d9... # ← hardcoded OK (external)
api_key_env: DEEPSEEK_API_KEY # ← also OK if set in /etc/environment
api_key_env: DEEPSEEK_API_KEY # ← also OK if set in environment (vault or /etc/environment)
```
```yaml
# ❌ FORBIDDEN — hardcoded harness/litellm key
# ❌ FORBIDDEN — hardcoded key (top) OR unauthenticated path (bottom)
model:
provider: harness
api_key: sk-Flc62smlegyMEaSo1ka8JA # ← RULE VIOLATION
api_key: sk-Flc62smlegyMEaSo1ka8JA # ← RULE VIOLATION: hardcoded key
model:
provider: harness
base_url: http://192.168.68.116/v1 # ← RULE VIOLATION: unauthenticated path
api_key_env: LITELLM_API_KEY
```
## Detection Query
@@ -75,13 +117,18 @@ Run on any Hermes host to detect violations:
```bash
# 1. Check config.yaml for hardcoded harness keys
grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
grep -rn 'api_key: sk-' /root/.hermes/ \
--include='config.yaml' \
| grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
# 1b. Check for double-path bug: base_url ending with /responses
# (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
# ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /root/.config/systemd/ 2>/dev/null
# 3. Verify running process env matches dedicated key
cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
@@ -92,26 +139,32 @@ If any output from step 2 — **critical violation** (master key leaked). Fix im
## Rotation Procedure
With this standard enforced, key rotation is one step:
With this standard enforced, key rotation is one vault update:
```bash
# 1. Generate new key in LiteLLM
# 2. Update /etc/environment on agent host
ssh root@<host> "sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW_KEY/' /etc/environment"
# 3. Restart agent gateway
ssh root@<host> "pkill -f 'hermes_cli.main gateway run'; sleep 2; nohup ... &"
# 1. Generate new key in LiteLLM: POST /key/generate with agent alias
# 2. Update Infisical vault secret
infisical secrets set LITELLM_API_KEY=sk-NEW_KEY \
--project=agents --env=production
# 3. Restart agent gateway (key auto-injected via infisical run -- wrapper)
ssh root@<host> "systemctl restart hermes-gateway"
# 4. Verify
curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/litellm/v1/models
```
**Done.** No config file changes needed. The agent picks up the new key on restart.
**Done.** No config file changes needed. No /etc/environment edits needed.
The agent picks up the new key via `infisical run --` at gateway startup.
> **Post-migration note**: /etc/environment is NO LONGER the key source.
> Strip all `LITELLM_API_KEY` lines from /etc/environment (comment out with `# [INFISICAL]`)
> and let the `infisical run --` wrapper inject the key at runtime.
## Key Longevity Policy (2026-07-04)
**Keys are permanent and use bare agent name aliases.**
- **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
- **Max budget**: $100 per key (config default).
@@ -128,36 +181,54 @@ litellm_settings:
## Verified Agents (2026-07-05 update)
| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
|-------|-----|-----|---------------|-----|--------|----------------|---------------|
| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
### Systemd Service Pattern (2026-07-04 fix)
> **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
> LiteLLM key aliases use agent identity, not CT hostname.
All Hermes agents use systemd to manage their gateway. Two issues were fixed:
### Migration Status: Authenticated Path
1. **Drop-in override** — `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` (or user equivalent) had hardcoded `LITELLM_API_KEY` that bypassed `/etc/environment`.
2. **Missing EnvironmentFile** — Services did not source `/etc/environment`.
| Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
|-------|--------------------------|--------------------|--------|
| Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
| Tanko | ⚠️ No SSH access | — | Needs check |
| Koby | ⚠️ No route to host | — | Needs check |
| Koonimo | ⚠️ Connection timed out | — | Needs check |
**Correct pattern:**
### Systemd Service Pattern (2026-07-11 — vault migration)
All Hermes agents use systemd to manage their gateway. The gateway service is wrapped
with `infisical run --` to inject secrets at runtime.
**Correct pattern (post-migration):**
```ini
# In service file:
EnvironmentFile=/etc/environment
# Service file wraps gateway with Infisical:
[Service]
ExecStart=/usr/bin/infisical run --project=agents --env=production -- \
/usr/bin/hermes gateway run
# Drop-in only for overrides, NOT primary key storage.
# If a drop-in exists, it must match /etc/environment.
# /etc/environment is CLEAN — no LITELLM_API_KEY present
# (strip it and tag with # [INFISICAL] if present)
```
**Rotation procedure** (one step with this standard):
**Legacy pattern (deprecated — pre-migration only):**
```ini
# DO NOT USE post-migration:
EnvironmentFile=/etc/environment
# This pattern was replaced by infisical run -- wrapper
```
**Rotation procedure** (one vault operation with this standard):
1. Generate new key in LiteLLM: `curl /key/generate` with agent alias
2. Update `/etc/environment`: `sed -i 's/LITELLM_API_KEY=.*/LITELLM_API_KEY=sk-NEW/' /etc/environment`
3. Update drop-in (if exists): same sed on `litellm-key.conf`
4. Restart: `systemctl [--user] restart hermes-gateway`
2. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=sk-NEW --project=agents --env=production`
3. Restart: `systemctl restart hermes-gateway` (key auto-injected via wrapper)
## Violation Response
@@ -167,7 +238,7 @@ EnvironmentFile=/etc/environment
4. **Restart** — gateway must restart to pick up env var
5. **Confirm** — test key against LiteLLM: `curl -H "Authorization: Bearer $KEY" .../v1/models` → 200
6. **Update** — bump the verified table above
7. **Use safe-mutate** — if the fix requires changing `/etc/environment` or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
7. **Use safe-mutate** — if the fix requires updating vault secrets or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
## Related Contracts
@@ -206,13 +277,13 @@ task config:
```yaml
auxiliary:
vision:
api_key: sk-CggiHWlamQyShxWC3Hx6uw # ← workaround
api_key: sk-<agent-key-from-vault> # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1
model: gemma-4-12b
provider: harness
compression:
api_key: sk-CggiHWlamQyShxWC3Hx6uw # ← workaround
api_key: sk-<agent-key-from-vault> # ← workaround (same as above)
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1
model: gemma-4-12b