SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: lessons learned from 2026-07-12 session

Browse Source 
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
root
2026-07-12 22:49:39 +00:00
parent 19b6db9891
commit 79d4a73895
14 changed files with 372 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hermes-agent-baseline.prose.md
+46 -25
View File
@@ -22,30 +22,38 @@ done
## Agent Map
| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
|-------|-----|------|-----|-------------|---------------|----------|
| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
| Koby | 111 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 113 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes |
> **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo).
Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
Keys are stored in Infisical vault (project=agents, env=production) and injected at
runtime via `infisical run --` wrapper. Plaintext keys removed from this baseline.
## Key Architecture
```
Infisical vault → infisical run -- hermes gateway → LITELLM_API_KEY (runtime)
↓
Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
└── Key DB (Postgres)
└── Key DB (Postgres)
```
- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
- **Master key**: stored in Infisical vault (project=infrastructure, secret=LITELLM_MASTER_KEY) — ADMIN ONLY
- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
- **Key injection**: `infisical run --project=agents --env=production -- hermes gateway run` injects `LITELLM_API_KEY` at runtime
- **Key source**: Infisical vault → runtime env var. /etc/environment is CLEAN (stripped, tagged `# [INFISICAL]`)
- **Legacy override** (pre-migration): `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` — should be REMOVED
## Config Pattern — Mandatory Fields
### For Hermes Agents (Tanko, Mumuni, Baggy)
### For Hermes Agents (Tanko, Mumuni, Koonimo)
Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
@@ -76,7 +84,7 @@ auxiliary:
model: gemma-4-12b # or syslog-auto
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_key: <ACTUAL_KEY_FROM_/etc/environment> # ← MANDATORY workaround
api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 60
download_timeout: 30
```
@@ -91,7 +99,7 @@ auxiliary:
model: syslog-auto # or gemma-4-12b
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY
api_key: <ACTUAL_KEY_FROM_/etc/environment> # ← MANDATORY workaround
api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 120
```
@@ -110,8 +118,7 @@ LiteLLM/harness will fail with:
401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
```
**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
`api_key_env` in every auxiliary task config that uses the harness provider.
**Workaround**: Set `api_key` directly (copy the value from Infisical vault: `infisical secrets get LITELLM_API_KEY --project=agents --env=production`) alongside `api_key_env` in every auxiliary task config that uses the harness provider.
**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
`api_key` is empty:
@@ -129,7 +136,11 @@ if not cfg_api_key:
```bash
for ct in 112 114 111 113; do
echo "=== CT $ct ==="
pct-run $ct grep LITELLM_API_KEY /etc/environment
# Verify /etc/environment is CLEAN (no LITELLM_API_KEY)
pct-run $ct "grep -c LITELLM_API_KEY /etc/environment 2>/dev/null || echo '0 (clean)'"
# Verify gateway uses infisical run wrapper
pct-run $ct "ps aux | grep 'infisical run' | grep -v grep"
# Check for hardcoded harness keys
pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
echo ""
done
@@ -137,15 +148,17 @@ done
### Master Key Leak Check
```bash
# On every agent:
pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
# Must return empty
# On every agent — must return empty:
pct-run <CT> grep -rl "sk-litellm" /root/ /etc/ 2>/dev/null
# Vault is the only place the master key should exist
```
### Verify Key Works
```bash
# Retrieve key from vault and test:
KEY=$(infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
curl -s http://192.168.68.116:80/v1/models \
-H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
-H "Authorization: Bearer $KEY" | grep syslog-auto
# Must return model list
```
@@ -156,19 +169,27 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
```
### For pi Agents (Tdunna)
### For Koby (CT 111 / tdunna)
Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.
**LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.
### For pi Agents (Abiba)
Abiba (CT100) runs pi via PM2 with the Zulip extension.
Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
**models.json** — Must only list models authorized for the agent's LiteLLM key:
**models.json** — Must only list models authorized for the agent's LiteLLM key.
Key is injected via `infisical run --` wrapper at PM2 startup:
```json
{
"providers": {
"syslog-harness": {
"baseUrl": "http://192.168.68.116/v1",
"api": "openai-completions",
"apiKey": "sk-...",
"apiKey": "${LITELLM_API_KEY}",
"models": [
{ "id": "syslog-auto" },
{ "id": "ornith-1.0-35b" },
@@ -257,6 +278,6 @@ If they differ → ghost detected → kill ghost → start fresh.
| Date | Change |
|------|--------|
| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
| 2026-07-08 | Koby: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added config section. Key rotated and stored in vault. Added Failure Mode #11 to zulip-adapter-lessons. |
| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |