diff --git a/gpu-self-heal.prose.md b/gpu-self-heal.prose.md index 62cf205..911ed08 100644 --- a/gpu-self-heal.prose.md +++ b/gpu-self-heal.prose.md @@ -46,7 +46,7 @@ depends_on: |-------|-----|------|-------|------|-----|-------|------| | `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen | | `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks | -| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs | +| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs | Key notes: - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path. @@ -143,7 +143,7 @@ Key notes: - **Detect**: Benchmark tok/s vs baseline for each GPU at current context (all 128K) - RTX 3090 (128K ctx, ThinkingCap): baseline 74.8 tok/s — currently at 74.9 (100%) - RTX 5070 (128K ctx, HauhauCS QAT): baseline 165.2 tok/s — currently at 169.6 (103%) - - Strix Halo (128K ctx, Genesis Hermes V3): baseline 70.5 tok/s — currently at 62.9 (89%) + - Strix Halo (128K ctx, qwen3.6-35B-udq4): baseline 70.5 tok/s — currently at 62.9 (89%) - **Fix**: - If tok/s > baseline → context has headroom, consider increasing - If tok/s < 90% baseline → reduce context by 25% and retest diff --git a/inference-optimization.prose.md b/inference-optimization.prose.md index 646828f..318a4a9 100644 --- a/inference-optimization.prose.md +++ b/inference-optimization.prose.md @@ -59,7 +59,7 @@ prefill time at 532 tok/s. Fix context first, routing second. - **Route by task**: qwen for code/standard queries; gemma for compression/auxiliary; strix-moe for compression tasks. - **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should - compact at 85K, not 166K. Target 15% tail (not 30%). + compact at 51K, not 85K. Target 15% tail (not 30%). - **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these never change between turns. Single-digit cache hit rate is unacceptable. - **Lower context ceiling**: 128K window is the stable ceiling for agent conversations.