contract updates: ornith decommissioned, 256K→128K context, Mumuni Discord disabled
Change 1: Strix Halo — ornith decommissioned - gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout - inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4 - gpu-monitor: ornith status → Strix Halo status - infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4 - infrastructure-update: ornith→strix-moe via router - proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe) Change 2: GPU context 256K→128K fleet-wide - hermes-agent-baseline: frontmatter description updated - litellm-health: GPU Fleet Topology table 256K→128K - litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert - inference-optimization: compress threshold 256K→128K compact at 85K - gpu-fleet: instability note updated Change 3: Mumuni Discord platform disabled - gpu-fleet: Mumuni platforms: removed discord
This commit is contained in:
@@ -22,14 +22,14 @@ management, and prompt caching — without sacrificing agent capability.
|
||||
- `agent-configs`: current config.yaml from each active Hermes agent (Mumuni
|
||||
.123, any others on .129/.122) including compression, model, context_window,
|
||||
prompt_caching, memory settings
|
||||
- `gpu-health`: health check response from all 3 GPU backends (ornith .15:8080,
|
||||
- `gpu-health`: health check response from all 3 GPU backends (strix-moe .15:8080,
|
||||
qwen .8:8080, gemma .110:8080)
|
||||
|
||||
### Maintains
|
||||
|
||||
The optimized inference stack configuration — every change is applied and
|
||||
verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of
|
||||
non-ornith traffic; ≤ 30000 for ornith-bound agentic calls.
|
||||
inference calls.
|
||||
|
||||
#### liteLLM-routing
|
||||
The syslog-auto routing weights, model-specific timeouts, RPM limits, and
|
||||
@@ -53,18 +53,17 @@ duration.
|
||||
|
||||
### Strategies
|
||||
|
||||
**Context is the root cause.** Every ~46K prompt token costs ~87s of ornith
|
||||
**Context is the root cause.** Every ~46K prompt token costs ~87s of
|
||||
prefill time at 532 tok/s. Fix context first, routing second.
|
||||
|
||||
- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
|
||||
queries; gemma for compression/auxiliary. Never send simple completion to a
|
||||
35B MoE.
|
||||
- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
|
||||
compact at 102K, not 166K. Target 15% tail (not 30%).
|
||||
- **Route by task**: qwen for code/standard queries; gemma for
|
||||
compression/auxiliary; strix-moe for compression tasks.
|
||||
- **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should
|
||||
compact at 85K, not 166K. Target 15% tail (not 30%).
|
||||
- **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these
|
||||
never change between turns. Single-digit cache hit rate is unacceptable.
|
||||
- **Lower context ceiling**: 128K window is the stable ceiling for agent conversations.
|
||||
GPUs reduced from 256K to 128K (2026-07-17). For larger contexts, route to external providers.
|
||||
GPUs reduced from 256K to 128K (2026-07-17). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers.
|
||||
|
||||
### Shape
|
||||
|
||||
@@ -100,5 +99,5 @@ call enable-prompt-caching
|
||||
|
||||
call verify-latency
|
||||
host: 192.168.68.116
|
||||
models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b, ornith-1.0-35b]
|
||||
models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b]
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user