root
b9149bce47
GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)
3.4 KiB
name, kind, id
| name | kind | id |
|---|---|---|
| inference-optimization | responsibility | 067NC6KP02RG60S50M40E30928 |
Goal
Syslog inference response times reduced to sub-15s average by optimizing the full stack: LiteLLM routing weights, GPU model assignments, Hermes agent context management, and prompt caching — without sacrificing agent capability.
Requires
inference-metrics: current SpendLogs from CT116 LiteLLM Postgres — avg request_duration_ms, prompt_tokens, completion_tokens, model_group breakdown, cache_hit rate over the last 3 hoursagent-configs: current config.yaml from each active Hermes agent (Mumuni .123, any others on .129/.122) including compression, model, context_window, prompt_caching, memory settingsgpu-health: health check response from all 3 GPU backends (ornith .15:8080, qwen .8:8080, gemma .110:8080)
Maintains
The optimized inference stack configuration — every change is applied and verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of non-ornith traffic; ≤ 30000 for ornith-bound agentic calls.
liteLLM-routing
The syslog-auto routing weights, model-specific timeouts, RPM limits, and
model_list entries on CT116 /opt/inference-harness/litellm_config.yaml.
agent-compression
Each Hermes agent's ~/.hermes/config.yaml compression, context_window,
prompt_caching, and model sections.
prompt-caching
LiteLLM cache configuration and llama.cpp --cache-prompt flag on GPU hosts.
verification
End-to-end latency measurements after changes applied — at least 3 test inference calls per model path measuring ttft (time-to-first-token) and total duration.
Continuity
- input-driven
Strategies
Context is the root cause. Every ~46K prompt token costs ~87s of ornith prefill time at 532 tok/s. Fix context first, routing second.
- Route by task: ornith for multi-step reasoning only; qwen for code/standard queries; gemma for compression/auxiliary. Never send simple completion to a 35B MoE.
- Compress aggressively: threshold at 40% (not 65%) — a 256K window should compact at 102K, not 166K. Target 15% tail (not 30%).
- Cache everything repeated: system prompts, skill docs, AGENTS.md — these never change between turns. Single-digit cache hit rate is unacceptable.
- Lower context ceiling: 128K window is the stable ceiling for agent conversations. GPUs reduced from 256K to 128K (2026-07-17). For larger contexts, route to external providers.
Shape
self: analyze metrics, compute optimal configs, apply changes, verifydelegates:apply-liteLLM: update litellm_config.yaml and reloadapply-agent-config: update hermes config.yaml per agentverify-latency: run test inference calls and measure response
Execution
-- Phase 1: Analyze current state (already complete)
-- Phase 2: Apply LiteLLM routing optimization
call apply-liteLLM-routing
config_path: /opt/inference-harness/litellm_config.yaml
host: 192.168.68.116
-- Phase 3: Apply agent context compression optimization
call apply-agent-compression
agent: mumuni
host: 192.168.68.123
config_path: /root/.hermes/config.yaml
-- Phase 4: Enable llama.cpp prompt caching on GPU hosts
call enable-prompt-caching
hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110]
-- Phase 5: Verify end-to-end latency
call verify-latency
host: 192.168.68.116
models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b, ornith-1.0-35b]