SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: lessons learned from 2026-07-12 session

Browse Source 
- gpu-fleet: Corrected architecture (direct GPU, no router in path).
  Updated context values (RTX 3090=256K, not 128K). Added api-key
  standardization requirement.
- gpu-self-heal: Added Lessons Learned section with 5 critical findings:
  L1: API key standardization (RTX 5070 sk-loc...5678 vs not-needed)
  L2: Fallback chain cascading failure loop detection
  L3: Verify running state, not documentation
  L4: Infisical fallback requirement (.env must have uncommented key)
  L5: Zulip event queue can silently die after ~40 reconnects
- litellm-self-heal: Updated status manual-only→deployed, cron schedule
- litellm-api-keys: Added Infisical token expiry warning + .env fallback
- hermes-config-template: Rule 3 updated with .env fallback requirement
This commit is contained in:
root
2026-07-12 22:49:39 +00:00
parent 19b6db9891
commit 79d4a73895
14 changed files with 372 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gpu-fleet.prose.md
+28 -18
View File
@@ -5,9 +5,13 @@ description: >
Manages the GPU inference fleet across all hosts. Handles model deployment,
registration, health checks, LiteLLM sync, agent key management, GPU
saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2
on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully
deprecated — nginx routes /v1 → LiteLLM directly.
UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
to llama-server on each GPU host (no router in inference path). Router
(port 9000) is running but NOT in request path. All GPUs standardized on
api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
caused cascading 401→timeout→401 fallback loops — fixed.
Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
agent: abiba
triggers:
- on model add/remove
@@ -114,10 +118,10 @@ triggers:
### sync-keys
1. List all agent keys in LiteLLM DB via `GET /key/list`
2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
3. Generate missing keys via `POST /key/generate` with unlimited budget
4. Update agent configs — `/etc/environment` LITELLM_API_KEY
5. Send Zulip DM to agents that can't be reached via SSH
4. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production`
5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
6. Verify each key with test request through full chain
7. Document keys in knowledge graph
@@ -136,19 +140,25 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
7. Check port conflicts: verify only one llama-server on :8080 per host
8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
## Agent Keys (LiteLLM DB — Current 2026-06-30)
## Agent Keys (LiteLLM DB — Current 2026-07-11)
| Agent | CT | IP | Key | Access |
|-------|-----|-----|-----|--------|
| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
| Abiba | 100 | .24 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY).
Agent gateways inject keys at runtime via `infisical run --` wrapper.
Plaintext keys removed from this contract post-vault-migration.
**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
If no SSH access, send Zulip DM via abiba-bot.
| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|-------|-----|-----|---------------|------------|--------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
| Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | no SSH |
> **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.
**Key update procedure**: Update Infisical vault → `infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production` → restart agent gateway. Agent picks up new key via `infisical run --` wrapper at startup.
If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
## Configuration Files
@@ -170,7 +180,7 @@ If no SSH access, send Zulip DM via abiba-bot.
| Component | URL | Details |
|-----------|-----|---------|
| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
| Grafana | `http://192.168.68.116:3001/` | admin / vault (`GRAFANA_ADMIN_PASSWORD`) |
| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |