Merge pull request 'fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)' (#34 ) from fix/gpu-dense-q3-correction into master

fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)
2026-07-27 20:26:13 +00:00 · 2026-07-27 20:25:51 +00:00
1 changed files with 8 additions and 7 deletions
@@ -15,8 +15,9 @@ description: >
  For larger context needs → fall back to external providers (deepseek).
  VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
  UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled
-  (UD-Q4_K_XL, ~17.9GB). ~50% fewer thinking tokens via ThinkingCap finetune + Fable 5 CoT
-  distillation for improved coding reasoning. Same 27B base, cleaner fits on RTX 3090.
+  (UD-Q3_K_XL, ~14.7GB — Q4 was too large for 24GB VRAM with 128K KV cache). ~50% fewer thinking
+  tokens via ThinkingCap finetune + Fable 5 CoT distillation for improved coding reasoning.
+  VRAM ~22.4/24.6GB (91%).
 agent: abiba
 triggers:
  - on model add/remove
@@ -99,7 +100,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap

 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| SmartCode-Fable-5-27B-UD-Q4_K_XL | RTX 3090 | .8 (llm-gpu) | ~17.9/24.6GB (73%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ testing |
+| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 | .8 (llm-gpu) | ~22.4/24.6GB (91%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ healthy |
 | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
 | qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |

@@ -109,7 +110,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap

 | Model | GPU | Weight | RPM Cap | Timeout |
 |-------|-----|--------|---------|---------|
-| SmartCode-Fable-5-27B-UD-Q4_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
+| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
 | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
 | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |

@@ -120,7 +121,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
 | strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
-| SmartCode-Fable-5-27B-UD-Q4_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
+| SmartCode-Fable-5-27B-UD-Q3_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
 | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |

 ### Stable Aliases (for agent configs — never change)
@@ -250,7 +251,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
 - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
- **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q4_K_XL. Same 27B base, distilled for ~50% fewer thinking tokens + Fable 5 CoT coding reasoning. Runs via direct llama-server (no MTP for this model). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. VRAM: ~73%. Service: direct `llama-server` call (no wrapper script).
+- **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q3_K_XL (14.7GB). Q4 was too large for 24GB VRAM with 128K context + KV cache overhead. Q3 fits at ~22.4GB (91%). Uses standard llama.cpp build b9190 (turboquant b9150 incompatible with qwen3_5 arch). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. Service: `/home/llmuser/llama-fable-wrapper.sh`.
 - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
 - **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
@@ -267,7 +268,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.

 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | **TBD** | — | — | **128K** |
+| RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q3_K_XL | **TBD** | — | — | **128K** |
 | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
 | Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |
Author	SHA1	Message	Date
mumuni-bot	835647e241	Merge pull request 'fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)' (#34 ) from fix/gpu-dense-q3-correction into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-07-27 20:26:13 +00:00
root	22b015e182	fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM) PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped Details	2026-07-27 20:25:51 +00:00