fix: swap gpu-dense to SmartCode-Fable-5 (ThinkingCap + Fable 5 CoT)

gpu-dense on RTX 3090 swapped from qwen3.6-27B-code to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled (UD-Q4_K_XL, ~17.9GB).\n\nImprovements:\n- ~50% fewer thinking tokens via ThinkingCap finetune\n- Fable 5 CoT distillation for improved coding reasoning\n- Same 27B base, fits RTX 3090 at ~73% VRAM\n- Recommended samplers: temp 0.9, top-p 0.95, top-k 60\n- Context: 128K (fleet standard)

gpu-fleet.prose.md

@@ -14,6 +14,9 @@ description: >
   Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling.
   For larger context needs → fall back to external providers (deepseek).
   VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
+  UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled
+  (UD-Q4_K_XL, ~17.9GB). ~50% fewer thinking tokens via ThinkingCap finetune + Fable 5 CoT
+  distillation for improved coding reasoning. Same 27B base, cleaner fits on RTX 3090.
 agent: abiba
 triggers:
   - on model add/remove
@@ -86,7 +89,7 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
 | Alias | GPU | Current Model | Will Route To |
 |-------|-----|---------------|---------------|
 | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
-| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
+| `gpu-dense` | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | Whatever runs on RTX 3090 |
 | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
 
 **Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
@@ -96,7 +99,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
 
 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s |
+| SmartCode-Fable-5-27B-UD-Q4_K_XL | RTX 3090 | .8 (llm-gpu) | ~17.9/24.6GB (73%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ testing |
 | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
 | qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
 
@@ -106,7 +109,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
 
 | Model | GPU | Weight | RPM Cap | Timeout |
 |-------|-----|--------|---------|---------|
-| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
+| SmartCode-Fable-5-27B-UD-Q4_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
 | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
 | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
 
@@ -117,7 +120,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
 | strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
-| qwen3.6-27B-code | 500 | High cap — primary workhorse |
+| SmartCode-Fable-5-27B-UD-Q4_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
 | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
 
 ### Stable Aliases (for agent configs — never change)
@@ -247,10 +250,9 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
   Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
 - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
-- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
+- **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q4_K_XL. Same 27B base, distilled for ~50% fewer thinking tokens + Fable 5 CoT coding reasoning. Runs via direct llama-server (no MTP for this model). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. VRAM: ~73%. Service: direct `llama-server` call (no wrapper script).
-- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
 - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
-- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
+- **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
@@ -265,7 +267,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 
 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** |
+| RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | **TBD** | — | — | **128K** |
 | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
 | Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |
 

infrastructure-control.prose.md

@@ -13,11 +13,10 @@ description: >
   against the live system. Policy fields are authoritative. See the
   `verify-before-mutate` skill.
 
-  **Last verified:** 2026-07-26 — verification protocol run. CT 118 set to
+  **Last verified:** 2026-07-27 — gpu-dense swapped to SmartCode-Fable-5
-  static IP .20 (was DHCP .131). llama-server on RTX 5070 (.110) was down,
+  (Qwen3.6-27B distilled, ~50% fewer thinking tokens, improved coding reasoning).
-  restarted and verified gemma-4-12b responding through LiteLLM. All 6 PVE
+  Model pricing reduced 100x across all models ($0.15/$0.60 per 1M tokens).
-  nodes confirmed. All CT hostnames/IPs confirmed except jdownloader (now
+  All LiteLLM models set to 128K max_model_tokens. See data/learnings.md.
-  static). All public endpoints responding. See data/learnings.md.
 ---
 
 # Infrastructure Control Pattern
@@ -113,9 +112,9 @@ description: >
 | hwepve | .4 | ? | ? | abiba, (mumuni CT 114 stopped) | Agents (new node) |
 
 > **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
-> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve, not amdpve. Mumuni
+> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve and now runs Mumuni
-> (CT 114) is on hwepve (currently stopped), not minipve. Mumuni also has a
+> Zulip gateway internally. CT 114 (old mumuni container) destroyed 2026-07-26.
-> second instance on minipve at .123 — distinguish by CT ID, not hostname.
+> Mumuni also has a second instance on minipve at .123 — distinguish by CT ID.
 
 ### Checks (every 5 min)
 
@@ -605,7 +604,6 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
 | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
-| 114 | mumuni | **hwepve** | .123 | Hermes agent (stopped) | ✅ |
 | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
 | 117 | zulip | storepve | .19 | Chat | ❌ |
@@ -638,7 +636,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
 | 115 | scottdenya | amdpve | `pct-run 115` |
 | 104 | authentik | minipve | `pct-run 104` |
 | 110 | gitea | minipve | `pct-run 110` |
-| 114 | mumuni | **hwepve** | `pct-run 114` |
+
 | 116 | syslog-api | minipve | `pct-run 116` |
 | 106 | ra-h-os | storepve | `pct-run 106` |
 | 107 | proxmox-backup | storepve | `pct-run 107` |
@@ -651,7 +649,7 @@ GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use S
 ssh root@192.168.68.8   # RTX 3090
 ssh root@192.168.68.110 # RTX 5070
 ssh root@192.168.68.15  # Strix Halo
-ssh root@192.168.68.4   # hwepve (abiba, kagentz, mumuni)
+ssh root@192.168.68.4   # hwepve (abiba, kagentz) — Mumuni runs inside CT 100
 ```
 
 ## Section 7: Agent Health Check v2 (2026-07-26)

scripts/agent-health-check.py

@@ -41,10 +41,9 @@ PVE_NODES = {
 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
     "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve",  "vault_key": "TANKO_LITELLM_API_KEY"},
-    "mumuni":  {"ct": 114, "host": "192.168.68.123", "user": "root",   "pve": "hwepve",  "vault_key": "MUMUNI_LITELLM_API_KEY"},
+    "abiba":   {"ct": 100, "host": "192.168.68.24",  "user": "root",   "pve": "hwepve",  "vault_key": None},  # Pi agent + Mumuni Zulip, no vault key
     "koby":    {"ct": 111, "host": "192.168.68.129", "user": "root",   "pve": "amdpve",  "vault_key": "KOBY_LITELLM_API_KEY"},
     "koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root",   "pve": "amdpve",  "vault_key": "KOONIMO_LITELLM_API_KEY"},
-    "abiba":   {"ct": 100, "host": "192.168.68.24",  "user": "root",   "pve": "hwepve",  "vault_key": None},  # Pi agent, no vault key
 }
 
 GPU_HOSTS = {