Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6d975360d8 | ||
|
|
3e246835d5 |
+7
-7
@@ -125,7 +125,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
|
|||||||
|
|
||||||
| Alias | RPM Cap | Routes To | Purpose |
|
| Alias | RPM Cap | Routes To | Purpose |
|
||||||
|-------|---------|-----------|---------|
|
|-------|---------|-----------|---------|
|
||||||
| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
|
| `strix-moe` | 40 | Strix Halo | Agent reasoning, compression (~30% via syslog-auto pool) (MoE models) |
|
||||||
| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
|
| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
|
||||||
| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
|
| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
|
||||||
|
|
||||||
@@ -136,7 +136,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
|
|||||||
- syslog-auto → qwen → gemma → qwen3.6-35B-udq4
|
- syslog-auto → qwen → gemma → qwen3.6-35B-udq4
|
||||||
|
|
||||||
### Why Strix Halo RPM Is Capped
|
### Why Strix Halo RPM Is Capped
|
||||||
- Direct (strix-moe): 40 RPM (tight) — Strix Halo is shared with compression tasks
|
- Direct (strix-moe): 40 RPM (tight) — Strix Halo handles agent reasoning + compression via syslog-auto pool
|
||||||
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
|
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
|
||||||
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
|
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
|
||||||
|
|
||||||
@@ -284,7 +284,7 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.
|
|||||||
### Stable Aliases — CRITICAL
|
### Stable Aliases — CRITICAL
|
||||||
|
|
||||||
All agent configs MUST use stable role-based aliases, never model-specific names:
|
All agent configs MUST use stable role-based aliases, never model-specific names:
|
||||||
- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`)
|
- `compression.model: syslog-auto` — switched from strix-moe 2026-07-18 to distribute across weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070); relieves Strix Halo thermal pressure
|
||||||
- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
|
- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
|
||||||
- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
|
- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
|
||||||
- `auxiliary.web_extract.model: gpu-light`
|
- `auxiliary.web_extract.model: gpu-light`
|
||||||
@@ -295,7 +295,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
|
|||||||
- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
|
- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
|
||||||
- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
|
- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
|
||||||
- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
|
- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
|
||||||
- Mumuni compression model alias: `strix-moe` with 300s timeout
|
- Mumuni compression model alias: `syslog-auto` (switched from strix-moe 2026-07-18) with 300s timeout
|
||||||
|
|
||||||
### Mumuni Agent Profile
|
### Mumuni Agent Profile
|
||||||
|
|
||||||
@@ -305,8 +305,8 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
|
|||||||
|---------|-------|-------|
|
|---------|-------|-------|
|
||||||
| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
|
| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
|
||||||
| `model.provider` | `custom:litellm` | LiteLLM on CT116 |
|
| `model.provider` | `custom:litellm` | LiteLLM on CT116 |
|
||||||
| `compression.model` | `strix-moe` | Stable alias — survives model swaps |
|
| `compression.model` | `syslog-auto` | Switched from strix-moe 2026-07-18 — distributes across weighted pool |
|
||||||
| `aux.compression.model` | `strix-moe` | Compression auxiliary model |
|
| `aux.compression.model` | `syslog-auto` | Compression auxiliary — switched from strix-moe 2026-07-18 |
|
||||||
| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
|
| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
|
||||||
| `aux.web_extract.model` | `gpu-light` | Web extraction |
|
| `aux.web_extract.model` | `gpu-light` | Web extraction |
|
||||||
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
|
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
|
||||||
@@ -318,7 +318,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
|
|||||||
| `personalities` | `creative` | Creative assistant personality |
|
| `personalities` | `creative` | Creative assistant personality |
|
||||||
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
||||||
| Main model timeout | 300s | LiteLLM global timeout |
|
| Main model timeout | 300s | LiteLLM global timeout |
|
||||||
| Compression model timeout | 300s | strix-moe timeout increased from 120s |
|
| Compression model timeout | 300s | syslog-auto timeout (switched from strix-moe 2026-07-18) |
|
||||||
|
|
||||||
### Agent Update Status (2026-07-15)
|
### Agent Update Status (2026-07-15)
|
||||||
|
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ depends_on:
|
|||||||
|-------|-----|------|-------|------|-----|-------|------|
|
|-------|-----|------|-------|------|-----|-------|------|
|
||||||
| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
|
| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
|
||||||
| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
|
| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
|
||||||
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
|
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Agent reasoning, compression (~30% via syslog-auto pool), summarization, long docs |
|
||||||
|
|
||||||
Key notes:
|
Key notes:
|
||||||
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
|
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
|
||||||
@@ -157,7 +157,7 @@ Key notes:
|
|||||||
- **Target distribution**:
|
- **Target distribution**:
|
||||||
- RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
|
- RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
|
||||||
- RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
|
- RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
|
||||||
- Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
|
- Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Agent reasoning, compression (~30% via syslog-auto pool), summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
|
||||||
- **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
|
- **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
|
||||||
- **Fix**:
|
- **Fix**:
|
||||||
- Alert if any GPU is handling workload outside its designated role
|
- Alert if any GPU is handling workload outside its designated role
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ version: 1.0.0
|
|||||||
description: >
|
description: >
|
||||||
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
||||||
configuration state, keys, workarounds, and audit procedure. When an agent's
|
configuration state, keys, workarounds, and audit procedure. When an agent's
|
||||||
configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
|
configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 128K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (compression via syslog-auto pool — switched from strix-moe 2026-07-18).
|
||||||
author: Abiba (pi agent)
|
author: Abiba (pi agent)
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -5,11 +5,14 @@ description: >
|
|||||||
Standard Hermes configuration template for Syslog Solution LLC agents.
|
Standard Hermes configuration template for Syslog Solution LLC agents.
|
||||||
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
|
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
|
||||||
RA-H OS MCP) while keeping agent-specific API keys and model choices.
|
RA-H OS MCP) while keeping agent-specific API keys and model choices.
|
||||||
UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
|
UPDATED 2026-07-18: Compression model switched to `syslog-auto` (was `strix-moe`)
|
||||||
|
to relieve Strix Halo pressure. syslog-auto distributes compression across the
|
||||||
|
weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
|
||||||
|
UPDATED 2026-07-16: Compression model was the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
|
||||||
which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
|
which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
|
||||||
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
|
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
|
||||||
2026-07-16 Mumuni root-cause investigation (WAL #1300).
|
2026-07-16 Mumuni root-cause investigation (WAL #1300).
|
||||||
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
|
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (later switched to syslog-auto 2026-07-18). RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
|
||||||
---
|
---
|
||||||
|
|
||||||
## Maintains
|
## Maintains
|
||||||
@@ -130,7 +133,9 @@ mcp_servers:
|
|||||||
# ─── Compression ───
|
# ─── Compression ───
|
||||||
compression:
|
compression:
|
||||||
enabled: true
|
enabled: true
|
||||||
model: strix-moe # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
|
model: syslog-auto # ⚠️ Switched from strix-moe 2026-07-18 to relieve Strix Halo.
|
||||||
|
# syslog-auto distributes across weighted pool (55% RTX 3090,
|
||||||
|
# 30% Strix Halo, 15% RTX 5070). All GPUs at 128K.
|
||||||
provider: harness
|
provider: harness
|
||||||
max_context_window: 131072 # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
|
max_context_window: 131072 # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
|
||||||
threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K
|
threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K
|
||||||
@@ -145,8 +150,9 @@ compression:
|
|||||||
# model: gpu-light # stable alias (NOT raw "gemma-4-12b")
|
# model: gpu-light # stable alias (NOT raw "gemma-4-12b")
|
||||||
# base_url: http://192.168.68.116/v1
|
# base_url: http://192.168.68.116/v1
|
||||||
# api_key_env: LITELLM_API_KEY
|
# api_key_env: LITELLM_API_KEY
|
||||||
# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
|
# Compression uses syslog-auto (switched from strix-moe 2026-07-18) to distribute
|
||||||
# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
|
# load across the weighted pool and relieve Strix Halo pressure.
|
||||||
|
# Vision and web_extract use gpu-light = RTX 5070 (12B).
|
||||||
# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
|
# Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
|
||||||
# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
|
# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
|
||||||
# in agent configs — use the stable aliases so model swaps don't break agents.
|
# in agent configs — use the stable aliases so model swaps don't break agents.
|
||||||
@@ -166,7 +172,7 @@ auxiliary:
|
|||||||
timeout: 30
|
timeout: 30
|
||||||
compression:
|
compression:
|
||||||
provider: harness
|
provider: harness
|
||||||
model: strix-moe # MUST match compression.model above. Stable alias for Strix Halo.
|
model: syslog-auto # Switched from strix-moe 2026-07-18. Relieves Strix Halo pressure.
|
||||||
base_url: http://192.168.68.116/v1 # Rule 5: /v1 NOT /litellm/v1
|
base_url: http://192.168.68.116/v1 # Rule 5: /v1 NOT /litellm/v1
|
||||||
api_key_env: LITELLM_API_KEY
|
api_key_env: LITELLM_API_KEY
|
||||||
timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60)
|
timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60)
|
||||||
@@ -247,29 +253,30 @@ The following MUST be identical across ALL profiles:
|
|||||||
- Apply to BOTH main config AND all sub-agent profiles
|
- Apply to BOTH main config AND all sub-agent profiles
|
||||||
- For agents needing longer outputs: raise to 8192, but never omit
|
- For agents needing longer outputs: raise to 8192, but never omit
|
||||||
|
|
||||||
### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
|
### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-18)
|
||||||
- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
|
- Vision and web_extract use `gpu-light` (stable alias, RTX 5070 — 12GB, vision-optimized)
|
||||||
- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized)
|
- Compression now uses `syslog-auto` (switched from `strix-moe` 2026-07-18) to distribute
|
||||||
- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
|
compression load across the weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
|
||||||
(it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
|
This relieves Strix Halo pressure while keeping compression functional on all GPUs.
|
||||||
|
- **`syslog-auto` is the valid compression model** — LiteLLM serves it as the weighted pool.
|
||||||
|
Old configs with `strix-moe` for compression should be updated to `syslog-auto`.
|
||||||
- All auxiliary services MUST use identical routing:
|
- All auxiliary services MUST use identical routing:
|
||||||
- `base_url: http://192.168.68.116/v1` (Rule 5: `/v1`, NOT `/litellm/v1`)
|
- `base_url: http://192.168.68.116/v1` (Rule 5: `/v1`, NOT `/litellm/v1`)
|
||||||
- `api_key_env: LITELLM_API_KEY`
|
- `api_key_env: LITELLM_API_KEY`
|
||||||
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
|
- **Compression via syslog-auto**: Routes through the weighted pool. Strix Halo still handles
|
||||||
- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
|
~30% of compression calls (at 60 RPM via pool vs 40 RPM direct), but the bulk (55%)
|
||||||
(64GB UMA, 128K context) — the designated compression GPU. This frees the
|
goes to RTX 3090 which has ample spare capacity.
|
||||||
RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
|
|
||||||
- The `compression:` block's `model` MUST match `auxiliary: compression: model`
|
- The `compression:` block's `model` MUST match `auxiliary: compression: model`
|
||||||
- The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
|
- The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
|
||||||
|
|
||||||
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
|
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
|
||||||
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
|
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
|
||||||
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
|
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
|
||||||
- **Strix Halo (64GB, 128K ctx, strix-moe)**: Context compression, summarization, long docs
|
- **Strix Halo (64GB, 128K ctx, Genesis Hermes V3 APEX)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
|
||||||
- Agent profiles MUST route auxiliary tasks to the correct GPU:
|
- Agent profiles MUST route auxiliary tasks to the correct GPU:
|
||||||
- `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
|
- `auxiliary.vision.model: gpu-light` (RTX 5070)
|
||||||
- `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
|
- `auxiliary.web_extract.model: gpu-light` (RTX 5070)
|
||||||
- `auxiliary.compression.model: strix-moe` (Strix Halo)
|
- `auxiliary.compression.model: syslog-auto` (distributed pool, switched from strix-moe 2026-07-18)
|
||||||
- Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
|
- Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
|
||||||
- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
|
- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
|
||||||
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
|
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ duration.
|
|||||||
prefill time at 532 tok/s. Fix context first, routing second.
|
prefill time at 532 tok/s. Fix context first, routing second.
|
||||||
|
|
||||||
- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
|
- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
|
||||||
queries; gemma for compression/auxiliary. Never send simple completion to a
|
queries; gemma for vision/web extraction; syslog-auto for compression (via weighted pool). Never send simple completion to a
|
||||||
35B MoE.
|
35B MoE.
|
||||||
- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
|
- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
|
||||||
compact at 102K, not 166K. Target 15% tail (not 30%).
|
compact at 102K, not 166K. Target 15% tail (not 30%).
|
||||||
|
|||||||
Reference in New Issue
Block a user