SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'master' into feat/zulip-v3-resilience

Browse Source
This commit is contained in:
jerome
2026-07-16 21:20:24 +00:00
parent 622cf7b176 d6e85e68e9
commit 24cd7f72ac
12 changed files with 312 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files
abiba-zulip-restore.prose.md
+1 -1
View File
@@ -276,7 +276,7 @@ ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \
**Additional fixes applied:** **Additional fixes applied:**
- Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM) - Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM)
with `syslog-auto` + `ornith-1.0-35b` (prevents silent worker failure per with `syslog-auto` + `strix-moe` (prevents silent worker failure per
Lesson #11) Lesson #11)
- Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code) - Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code)
- PM2 saved for auto-restart on boot - PM2 saved for auto-restart on boot
delegation-prose-contract.prose.md
+4 -4
View File
@@ -72,10 +72,10 @@ reads them directly.
|--------|-------|----------|------|----------| |--------|-------|----------|------|----------|
| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | | `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | | `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | | `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | | `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
### Selection Rules ### Selection Rules
gpu-fleet.prose.md
+10 -10
View File
@@ -67,9 +67,9 @@ triggers:
│ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │ │ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │ │ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │ │ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
│ 256K ctx │ │ 131K ctx │ │ 256K ctx │ │ Watchdog │ │ 256K ctx │ │ 256K ctx │ │ 256K ctx │ │ Watchdog │
│ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │ │ qwen3.6 │ │ gemma-4-12b │ │ qwen3.6 │ │ Prometheus │
│ 27B-code │ │ :8080 │ │ :8080 │ │ exporter │ │ 27B-code │ │ :8080 │ │ -35B-udq4 │ │ exporter │
│ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │ │ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │
│ :9400 │ └─────────────┘ └───────────┘ └──────────────┘ │ :9400 │ └─────────────┘ └───────────┘ └──────────────┘
└──────────┘ └──────────┘
@@ -94,7 +94,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|-------|-----|------|------|-----|----------|----------|-------------|--------| |-------|-----|------|------|-----|----------|----------|-------------|--------|
| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | 22.2/24.6GB (90%) | **256K** 🚀 | turbo4 | 2 | default | ✅ 63 tok/s | | qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | 22.2/24.6GB (90%) | **256K** 🚀 | turbo4 | 2 | default | ✅ 63 tok/s |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy | | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 256K | q4_0 | 2 | 2048/1024 | ✅ healthy |
| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy | | qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
## Routing Configuration (LiteLLM — July 2026) ## Routing Configuration (LiteLLM — July 2026)
@@ -103,9 +103,9 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
| Model | GPU | Weight | RPM Cap | Timeout | | Model | GPU | Weight | RPM Cap | Timeout |
|-------|-----|--------|---------|---------| |-------|-----|--------|---------|---------|
| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | 90s | | qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **300s** | | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path. Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
@@ -186,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!") 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models` 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml` 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
- gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (ornith-1.0-35b does NOT exist — legacy name, do not use)
- global request_timeout: 300s, nginx proxy_read_timeout: 600s - global request_timeout: 300s, nginx proxy_read_timeout: 600s
6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power) 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
7. Check port conflicts: verify only one llama-server on :8080 per host 7. Check port conflicts: verify only one llama-server on :8080 per host
@@ -247,7 +247,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2). - **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`. - **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`. - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`.
- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`. - **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference.
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
- **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
@@ -262,7 +262,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
|-----|-------|-----------|--------------|----------|---------| |-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | 131K | | RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **256K** |
| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** | | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** |
| Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** | | Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** |
@@ -306,7 +306,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) | | `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
| `aux.web_extract.model` | `gpu-light` | Web extraction | | `aux.web_extract.model` | `gpu-light` | Web extraction |
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) | | `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
| `context.max_context_window` | 131072 (128K) | Kept at 128K to avoid compression timeout | | `context.max_context_window` | 262144 (256K) | Fixed 2026-07-16 (was 131072 — caused premature compression, WAL #1300) |
| `compression.threshold` | 0.65 | Triggers at ~85K | | `compression.threshold` | 0.65 | Triggers at ~85K |
| `compression.target_ratio` | 0.3 | Compresses to ~38K | | `compression.target_ratio` | 0.3 | Compresses to ~38K |
| `compression.protect_last_n` | 40 | Preserves last 40 messages | | `compression.protect_last_n` | 40 | Preserves last 40 messages |
gpu-self-heal.prose.md
+1 -1
View File
@@ -120,7 +120,7 @@ depends_on:
- **Detect**: Benchmark tok/s vs baseline for each GPU at current context - **Detect**: Benchmark tok/s vs baseline for each GPU at current context
- RTX 3090 (256K ctx, qwen3.6-27B-code): target 75+ tok/s — currently at baseline - RTX 3090 (256K ctx, qwen3.6-27B-code): target 75+ tok/s — currently at baseline
- RTX 5070 (131K ctx, gemma-4-12b): target 76+ tok/s — optimal for vision/web role - RTX 5070 (131K ctx, gemma-4-12b): target 76+ tok/s — optimal for vision/web role
- Strix Halo (256K ctx, ornith-1.0-35b): target 70+ tok/s — currently above baseline - Strix Halo (256K ctx, strix-moe / qwen3.6-35B-udq4): target 70+ tok/s — currently above baseline
- **Fix**: - **Fix**:
- If tok/s > baseline → context has headroom, consider increasing - If tok/s > baseline → context has headroom, consider increasing
- If tok/s < 90% baseline → reduce context by 25% and retest - If tok/s < 90% baseline → reduce context by 25% and retest
hermes-agent-baseline.prose.md
+1 -1
View File
@@ -192,7 +192,7 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
"apiKey": "${LITELLM_API_KEY}", "apiKey": "${LITELLM_API_KEY}",
"models": [ "models": [
{ "id": "syslog-auto" }, { "id": "syslog-auto" },
{ "id": "ornith-1.0-35b" }, { "id": "strix-moe" },
{ "id": "qwen3.6-27B-code" }, { "id": "qwen3.6-27B-code" },
{ "id": "gemma-4-12b" } { "id": "gemma-4-12b" }
] ]
hermes-config-template.prose.md
+95 -27
View File
@@ -5,10 +5,12 @@ description: >
Standard Hermes configuration template for Syslog Solution LLC agents. Standard Hermes configuration template for Syslog Solution LLC agents.
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices. RA-H OS MCP) while keeping agent-specific API keys and model choices.
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (ornith-1.0-35b). UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
RTX 3090 context verified at 256K (was incorrectly documented as 128K). which LiteLLM does not serve). All 3 GPUs verified 256K (RTX 5070 bumped 131K→256K on Jul 15).
RTX 5070 stays at 131K for vision/web. Infisical .env fallback required (Rule 3). Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
Parallel counts corrected (RTX 3090=1, RTX 5070=2, Strix=2). 2026-07-16 Mumuni root-cause investigation (WAL #1300).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context
verified at 256K. Infisical .env fallback required (Rule 3/13).
--- ---
## Maintains ## Maintains
@@ -31,7 +33,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Agent | Key Alias | Host | SSH | Sub-Agents | | Agent | Key Alias | Host | SSH | Sub-Agents |
|-------|-----------|------|-----|-----------| |-------|-----------|------|-----|-----------|
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — | | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
| Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ | | Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
| Abiba | `abiba-pi` | 192.168.68.24 | local | — | | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
| Koby | `koby` | CT 111 (tdunna) | Zulip | — | | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
| Koonimo | `koonimo` | CT 113 (baggy) | Zulip | — | | Koonimo | `koonimo` | CT 113 (baggy) | Zulip | — |
@@ -94,7 +96,7 @@ work immediately after restart.
```yaml ```yaml
# ─── Model Selection ─── # ─── Model Selection ───
model: model:
default: <agent_model> # e.g., ornith-1.0-35b, qwen3.6-27B-code default: <agent_model> # e.g., strix-moe, qwen3.6-27B-code, syslog-auto
provider: harness provider: harness
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper
@@ -129,10 +131,9 @@ mcp_servers:
# ─── Compression ─── # ─── Compression ───
compression: compression:
enabled: true enabled: true
model: ornith-1.0-35b # ⚠️ Must match auxiliary.compression.model model: strix-moe # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
provider: harness provider: harness
max_context_window: 262144 # For Strix Halo compression (256K ctx). max_context_window: 262144 # MUST match actual GPU capacity. All 3 GPUs are 256K (Jul 15).
# Set 131072 if using gemma directly.
threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K
target_ratio: 0.30 target_ratio: 0.30
protect_last_n: 40 protect_last_n: 40
@@ -142,37 +143,48 @@ compression:
# ─── Auxiliary Tasks (CONSISTENCY RULE) ─── # ─── Auxiliary Tasks (CONSISTENCY RULE) ───
# All auxiliary services MUST use identical model, base_url, and api_key_env: # All auxiliary services MUST use identical model, base_url, and api_key_env:
# model: gemma-4-12b # model: gpu-light # stable alias (NOT raw "gemma-4-12b")
# base_url: http://192.168.68.116/v1 # base_url: http://192.168.68.116/v1
# api_key_env: LITELLM_API_KEY # api_key_env: LITELLM_API_KEY
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU. # Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
# gemma-4-12b is a lightweight 12B model on the RTX 5070, freeing the Strix Halo # gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
# for agent reasoning. # Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
# in agent configs — use the stable aliases so model swaps don't break agents.
auxiliary: auxiliary:
vision: vision:
provider: harness provider: harness
model: gemma-4-12b model: gpu-light # stable alias for RTX 5070 (was raw gemma-4-12b)
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
timeout: 60 timeout: 60
download_timeout: 30 download_timeout: 30
web_extract: web_extract:
provider: harness provider: harness
model: gemma-4-12b model: gpu-light # stable alias for RTX 5070
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
timeout: 30 timeout: 30
compression: compression:
provider: harness provider: harness
model: ornith-1.0-35b model: strix-moe # MUST match compression.model above. Stable alias for Strix Halo.
base_url: http://192.168.68.116/v1 # Rule 5: /v1 NOT /litellm/v1
api_key_env: LITELLM_API_KEY
timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60)
# ─── Delegation / Heavy Aux (use gpu-dense = RTX 3090) ───
# delegation.model and x_search.model use gpu-dense (NOT raw qwen3.6-27B-code).
delegation:
model: gpu-dense # stable alias for RTX 3090 (was raw qwen3.6-27B-code)
provider: harness
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
timeout: 60
# ─── Custom Provider ─── # ─── Custom Provider ───
custom_providers: custom_providers:
- name: harness - name: harness
model: <agent_model> model: syslog-auto # weighted pool (default)
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
api_mode: chat_completions api_mode: chat_completions
@@ -236,27 +248,29 @@ The following MUST be identical across ALL profiles:
- Apply to BOTH main config AND all sub-agent profiles - Apply to BOTH main config AND all sub-agent profiles
- For agents needing longer outputs: raise to 8192, but never omit - For agents needing longer outputs: raise to 8192, but never omit
### Rule 7: Auxiliary Model Consistency (UPDATED July 2026) ### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized) - Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
- Compression uses `ornith-1.0-35b` (Strix Halo — 64GB, 256K ctx, compression-optimized) - Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 256K ctx, compression-optimized)
- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
(it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
- All auxiliary services MUST use identical routing: - All auxiliary services MUST use identical routing:
- `base_url: http://192.168.68.116/v1` - `base_url: http://192.168.68.116/v1` (Rule 5: `/v1`, NOT `/litellm/v1`)
- `api_key_env: LITELLM_API_KEY` - `api_key_env: LITELLM_API_KEY`
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably - **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
- **Compression moved to Strix Halo (July 2026)**: The ornith-1.0-35b model on Strix Halo - **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
(64GB UMA, 256K context, 72.4 tok/s) is the designated compression GPU. This frees the (64GB UMA, 256K context) — the designated compression GPU. This frees the
RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning. RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
- The `compression:` block's `model` MUST match `auxiliary: compression: model` - The `compression:` block's `model` MUST match `auxiliary: compression: model`
- The `compression: max_context_window: 262144` MUST match Strix Halo's actual capacity - The `compression: max_context_window: 262144` MUST match Strix Halo's actual capacity
### Rule 8: GPU Workload Distribution (July 2026) ### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
- **RTX 3090 (24GB, 256K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations - **RTX 3090 (24GB, 256K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
- **RTX 5070 (12GB, 131K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract - **RTX 5070 (12GB, 256K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (bumped 131K→256K Jul 15; IQ4_NL+MTP, 88% VRAM)
- **Strix Halo (64GB, 256K ctx, ornith-1.0-35b)**: Context compression, summarization, long docs - **Strix Halo (64GB, 256K ctx, strix-moe)**: Context compression, summarization, long docs
- Agent profiles MUST route auxiliary tasks to the correct GPU: - Agent profiles MUST route auxiliary tasks to the correct GPU:
- `auxiliary.vision.model: gemma-4-12b` (RTX 5070) - `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
- `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070) - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
- `auxiliary.compression.model: ornith-1.0-35b` (Strix Halo) - `auxiliary.compression.model: strix-moe` (Strix Halo)
- Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
- For 262K context window: `threshold: 0.65` (fires at ~170K tokens) - For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
@@ -274,7 +288,7 @@ The following MUST be identical across ALL profiles:
### Rule 10: Default Model Must Be `syslog-auto` (All Agents) ### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto` - **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json` - **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
- `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b - `syslog-auto` is the LiteLLM routing model — it load-balances between strix-moe
and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against: and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
- Model name typos that cause 403 errors and silent worker failures - Model name typos that cause 403 errors and silent worker failures
- Single GPU downtime (routing falls back automatically) - Single GPU downtime (routing falls back automatically)
@@ -293,6 +307,60 @@ The following MUST be identical across ALL profiles:
- A non-existent model ID causes 403 errors that silently break the pi RPC worker - A non-existent model ID causes 403 errors that silently break the pi RPC worker
(no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed) (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
### Rule 12: Context-Issue Diagnostic Checklist (ADDED 2026-07-16, WAL #1300)
When an agent shows "context issues" (premature compression, 401s, 504s, DeepSeek fallback),
verify ALL FOUR of these against the live config. They are the only root causes found in production:
1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
MUST be `262144` (all GPUs are 256K). A value of `131072` causes premature compression at
~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
2. **base_url uses /v1 NOT /litellm/v1?** — `custom_providers[0].base_url`, `delegation.base_url`,
and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/v1` (Rule 5). nginx `/litellm/`
has a 60s default timeout → 504 on any inference >60s; `/v1/` has 600s.
Check: `grep -n 'litellm/v1' ~/.hermes/config.yaml` (must return NOTHING)
3. **LITELLM_API_KEY valid?** — The key must be a real LiteLLM key (`sk-` + 64 hex, 67 chars).
Malformed values (e.g. `sk-_SWAl_Vu_…`, 47 chars) return 401 → DeepSeek fallback.
Verify: `curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LITELLM_API_KEY" http://192.168.68.116/v1/models` (must be 200)
4. **custom_providers aligned?** — `model: syslog-auto` (Rule 10), `api_mode: chat_completions`
(NOT `responses`). A wrong api_mode causes silent request failures.
One-line agent health check (run on the agent host):
```bash
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
```
### Rule 13: API Key Injection — Two Patterns (UPDATED 2026-07-16, WAL #1300)
Agents inject `LITELLM_API_KEY` via ONE of two mechanisms. Both are valid; the contract
requirement is that the key is a **valid LiteLLM virtual key** (HTTP 200 on /v1/models).
**Pattern A — systemd drop-in (Koby, Koonimo, and any agent without infisical wrapper):**
A systemd drop-in `/etc/systemd/system/hermes-gateway.service.d/litellm-key.conf` sets the key:
```ini
[Service]
Environment="LITELLM_API_KEY=sk-<VALID_KEY>"
```
The service unit `hermes-gateway.service` runs `python -m hermes_cli.main gateway run --replace`
directly (no infisical). Apply with `systemctl daemon-reload && systemctl restart hermes-gateway`.
- Koonimo (CT113/.114): service = `hermes-gateway.service`, drop-in has the key.
- Koby (CT111/.129): service = `hermes-gateway.service` (created 2026-07-16), ExecStart uses `--replace`
to win the lock against stray `hermes gateway restart` invocations. Key also in `/etc/environment`.
**Pattern B — infisical-gateway.sh wrapper (Mumuni):**
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
**Verification (all agents):**
```bash
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
```
- `/etc/environment` is NO LONGER the canonical key source (stale values there caused 401s).
- Do NOT leave a hardcoded stale key in `/etc/environment` — it shadows the drop-in/wrapper.
## Execution ## Execution
1. **Check current config** — Read the target agent's config.yaml 1. **Check current config** — Read the target agent's config.yaml
hermes-key-enforcement.prose.md
+5
View File
@@ -72,6 +72,11 @@ External providers are **explicitly exempt** and may use hardcoded keys:
## Standard Pattern ## Standard Pattern
> **Canonical vault process (2026-07-16):** see `litellm-api-keys` § Production Vault Access Process.
> All agents MUST use the `infisical-gateway.sh` wrapper (live vault injection). Hardcoded systemd
> drop-ins / config.yaml keys are DEPRECATED — they rot on rotation (root cause of the 2026-07-16 401 storm).
> 4/5 agents migrated; tanko (user jerome) pending.
```yaml ```yaml
# ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix) # ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
model: model:
litellm-api-keys.prose.md
+151 -3
View File
@@ -14,9 +14,15 @@ description: >
expire/404. The .env fallback prevents agents from running without keys. expire/404. The .env fallback prevents agents from running without keys.
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours. Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
on CT 116. Last verified: 2026-07-12. on CT 116. Last verified: 2026-07-16.
--- ---
## Parameters ## Parameters
@@ -51,8 +57,8 @@ description: >
- Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date) - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
- Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" } - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
- Duration is null (permanent) — inherited from litellm default_key_generate_params - Duration is null (permanent) — inherited from litellm default_key_generate_params
- Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"] - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"]
- Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU) - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
- Return the new key - Return the new key
5. **If action == "rotate"**: 5. **If action == "rotate"**:
- Generate new key with same alias (LiteLLM replaces the old key) - Generate new key with same alias (LiteLLM replaces the old key)
@@ -67,3 +73,145 @@ description: >
- Test the key against LiteLLM /v1/models - Test the key against LiteLLM /v1/models
- Confirm key alias matches agent_name in LiteLLM key list - Confirm key alias matches agent_name in LiteLLM key list
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run` - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
## Production Vault Access Process (canonical, 2026-07-16)
The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
runs as user `jerome`, not systemd root, needs user-scope adaptation).
### The canonical pattern
1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
- Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
- Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
```bash
#!/bin/bash
export INFISICAL_API_URL="https://vault.sysloggh.net"
TOKEN=$(cat /root/.infisical-token)
LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
while true; do
infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
--env=prod --domain=https://vault.sysloggh.net -- bash -c '
. /root/.hermes/.env 2>/dev/null # [FALLBACK Rule 3] safety net only
export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
' >> $LOG 2>&1
sleep 5 # restart on exit
done
```
4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
### Why this is non-fail
- **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
### Migration status (2026-07-16)
| Agent | Host | Pattern | Vault key | Status |
|-------|------|---------|-----------|--------|
| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
| tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
### Tanko migration (pending)
Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
`/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
### Koby migration lessons (2026-07-16)
Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
**Two mistakes I made that broke the agent:**
1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
**How Koby actually connects (2026-07-16):**
- Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
- Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
Allowed users: 6679773481. Home channel: 6679773481.
- Both platforms now connect through the wrapper's env injection.
**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old
process — captures the live env set. Especially important when migrating gateways between
injection mechanisms.
### Key rotation procedure (one vault operation with this standard)
1. Generate new key: `POST /key/generate` (master key, admin).
2. Update vault: `infisical secrets set <AGENT>_LITELLM_API_KEY=sk-NEW --token=$TOKEN --projectId=322fceab… --env=prod --domain=https://vault.sysloggh.net`.
3. Update `.env` fallback: `echo '<AGENT>_LITELLM_API_KEY=sk-NEW' > /root/.hermes/.env && chmod 600 /root/.hermes/.env`.
4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace
key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt
repo only ships 0.38.0, so `apt upgrade` does not help.
**Proper fix — Machine Identity (Infisical automation best practice):**
Create a machine identity with READ+WRITE scope on the `agents` project (project_id=
`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely.
Then vault writes work from any host:
```bash
# Get a machine-identity access token
TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
-H 'Content-Type: application/json' \
-d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
# Write a secret via REST API v3
curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
# OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
```
Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
and store its credentials in the vault itself (or a root-only file).**
**Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `<AGENT>_LITELLM_API_KEY`
overrides a stale vault value.
**2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE)
can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`,
koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` /
`KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
so the shared service token isn't reused across hosts — but it is no longer blocking.
## Key Rotation Log
| Date | Agent | Action | Notes |
|------|-------|--------|-------|
| 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
| 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
| 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
The master key is admin-only (/key/generate, /key/delete, /key/list). NEVER use it for inference —
see `litellm-self-heal` § "NEVER use litellm_proxy_master_key for inference".
- LiteLLM key DB: `harness-postgres` container on CT116, table `"LiteLLM_VerificationToken"` (columns: token, key_alias, key_name, created_at, expires). Query: `docker exec harness-postgres psql -U litellm -d litellm -t -c "SELECT key_alias, substr(token,1,16) FROM \"LiteLLM_VerificationToken\" ORDER BY created_at;"`
litellm-health.prose.md
+10 -10
View File
@@ -42,8 +42,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**: **What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
- Router REMOVED from request path — LiteLLM proxies directly to GPU - Router REMOVED from request path — LiteLLM proxies directly to GPU
- All GPUs at parallel 2 (was parallel 1) - All GPUs at parallel 2 (was parallel 1)
- NVIDIA context reduced 256K→128K to free VRAM - NVIDIA context reduced 256K→128K to free VRAM (SUPERSEDED 2026-07-16: all GPUs back to 256K — see litellm-self-heal)
- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s - LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal)
- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s - nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
## Parameters ## Parameters
@@ -72,18 +72,18 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
| Host | IP | Hardware | Models Served | Engine | Context | Parallel | | Host | IP | Hardware | Models Served | Engine | Context | Parallel |
|------|-----|----------|---------------|--------|---------|----------| |------|-----|----------|---------------|--------|---------|----------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 | | llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 2 |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 | | ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **256K** | 2 |
| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 | | amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 256K | 2 |
## Model Fallback Chains (LiteLLM) ## Model Fallback Chains (LiteLLM)
| Primary | Timeout | Fallback | Timeout | | Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------| |---------|---------|----------|---------|
| qwen3.6-27B-code | 90s | gemma-4-12b | 120s | | qwen3.6-27B-code | 300s | gemma-4-12b | 120s |
| gemma-4-12b | 120s | qwen3.6-27B-code | 90s | | gemma-4-12b | 120s | qwen3.6-27B-code | 300s |
| ornith-1.0-35b | 120s | qwen → gemma | — | | qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — |
| syslog-auto (balanced) | 90s | qwen → gemma | — | | syslog-auto (balanced) | 300s | qwen → gemma | — |
> Global: request_timeout=300s, nginx proxy_read_timeout=600s > Global: request_timeout=300s, nginx proxy_read_timeout=600s
@@ -129,7 +129,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
7. **Check model inference via LiteLLM** — Test each model: 7. **Check model inference via LiteLLM** — Test each model:
- POST /v1/chat/completions model=gemma-4-12b → expect 200 - POST /v1/chat/completions model=gemma-4-12b → expect 200
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
- POST /v1/chat/completions model=ornith-1.0-35b → expect 200 - POST /v1/chat/completions model=strix-moe → expect 200
- Use master key for auth - Use master key for auth
8. **Check agent keys**: 8. **Check agent keys**:
litellm-self-heal.prose.md
+27 -8
View File
@@ -6,6 +6,7 @@ note: >
DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * * DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04), Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116. now reimplemented as `litellm-health-check.sh` on CT 116.
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph. Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
GPU monitoring integrated from gpu-monitor on .24:9100. GPU monitoring integrated from gpu-monitor on .24:9100.
@@ -56,18 +57,29 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
| Host | IP | Hardware | Models Served | Engine | Context | Parallel | | Host | IP | Hardware | Models Served | Engine | Context | Parallel |
|------|-----|----------|---------------|--------|---------|----------| |------|-----|----------|---------------|--------|---------|----------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 1 | | llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 262144 --parallel 2 --ngl 99`) | **256K** | 2 |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 131K | 2 | | ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 262144 --parallel 2`, IQ4_NL + MTP draft) | **256K** | 2 |
| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 | | amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 256K | 2 |
> Verified on ground 2026-07-16 via `curl /v1/models` on each host + `llama-wrapper.sh`. The AMD host's underlying model is `qwen3.6-35B-udq4`; LiteLLM exposes it under two `model_name`s: `qwen3.6-35B-udq4` and `strix-moe` (rpm 40). The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced.
## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116)
`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`.
- `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
- `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. As of 2026-07-16 the `baggy`/`koby`/`mumuni`/`abiba-pi` keys ALSO include `qwen3.6-35B-udq4`; `abiba-pi` additionally includes `deepseek-v4-pro` (cloud fallback). `kagenz0`/`koonimo`/`pi-agents-unified` have the standard 6 only. Agents should still use the stable alias `strix-moe` (not the raw `qwen3.6-35B-udq4`) so model swaps don't break them.
- **NEVER use `litellm_proxy_master_key` (the `sk-litellm-...` master key) for inference.** It is for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated `monitor` agent key stored at `/etc/litellm-monitor.env` on CT 116 (root-only, `chmod 600`); `/key/list` is the only call that legitimately uses `$MASTER_KEY`.
## Model Fallback Chains (LiteLLM) ## Model Fallback Chains (LiteLLM)
| Primary | Timeout | Fallback | Timeout | | Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------| |---------|---------|----------|---------|
| qwen3.6-27B-code | 90s | gemma-4-12b | 120s | | qwen3.6-27B-code | 300s | gemma-4-12b | 120s |
| gemma-4-12b | 120s | qwen3.6-27B-code | 90s | | gemma-4-12b | 120s | qwen3.6-27B-code | 300s |
| ornith-1.0-35b | 120s | qwen → gemma | — | | qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — |
| syslog-auto (balanced) | 90s | qwen → gemma | — | | syslog-auto (balanced) | 300s | qwen → gemma | — |
> Global: request_timeout=300s, nginx proxy_read_timeout=600s > Global: request_timeout=300s, nginx proxy_read_timeout=600s
@@ -86,6 +98,13 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
| harness-docker-stats | python:3.12-alpine | — | container stats exporter | | harness-docker-stats | python:3.12-alpine | — | container stats exporter |
| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus | | harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
## Script Operations (synced 2026-07-16)
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 256K context steady-state is ~96% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed).
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
## Maintains ## Maintains
- litellm-admin-ui: { status: "healthy", last_check: timestamp } - litellm-admin-ui: { status: "healthy", last_check: timestamp }
@@ -128,7 +147,7 @@ Run this first on every cycle. Results feed into remediation rules below.
### 5. Check model inference via LiteLLM — test each model ### 5. Check model inference via LiteLLM — test each model
- POST /v1/chat/completions model=gemma-4-12b → expect 200 - POST /v1/chat/completions model=gemma-4-12b → expect 200
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
- POST /v1/chat/completions model=ornith-1.0-35b → expect 200 - POST /v1/chat/completions model=strix-moe → expect 200
- Use master key for auth - Use master key for auth
### 6. Check agent keys ### 6. Check agent keys
mumuni-delegation-prose-contract.prose.md
+4 -4
View File
@@ -92,10 +92,10 @@ raw data never provided.
|--------|-------|----------|------|----------| |--------|-------|----------|------|----------|
| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files |
| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks |
| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | | `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations |
| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | | `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs |
| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | | `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings |
| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | | `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting |
### Selection Rules ### Selection Rules
zulip-adapter-lessons.prose.md
+2 -2
View File
@@ -79,7 +79,7 @@ description: >
`messages_processed` stalls. `messages_processed` stalls.
- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a - **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B` model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session configured but LiteLLM only exposes `strix-moe` (alias for qwen3.6-35B-udq4) under that key. pi's session
workers emit 403 on first prompt, then never recover because the error doesn't trigger workers emit 403 on first prompt, then never recover because the error doesn't trigger
`agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue. `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows - **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
@@ -88,7 +88,7 @@ description: >
(2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process. JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model - **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should routing and fallback automatically. Direct model IDs (`strix-moe`, etc.) should
only be used when explicitly requested. Validate model IDs at agent setup time. only be used when explicitly requested. Validate model IDs at agent setup time.
- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider - **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider