diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 360ba94..eea7074 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -14,10 +14,9 @@ description: > Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling. For larger context needs → fall back to external providers (deepseek). VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%. - UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled - (UD-Q3_K_XL, ~14.7GB — Q4 was too large for 24GB VRAM with 128K KV cache). ~50% fewer thinking - tokens via ThinkingCap finetune + Fable 5 CoT distillation for improved coding reasoning. - VRAM ~22.4/24.6GB (91%). + UPDATED 2026-08-15: gpu-dense swapped to Qwen3.8-27B-Uncensored-Q4_K_M + (~16.8GB, 128K ctx, --spec-type draft-mtp (v2)). Served on .8:8080 under the legacy + alias qwen3.6-27B-code for LiteLLM routing continuity. Replaces SmartCode-Fable-5-27B. agent: abiba triggers: - on model add/remove @@ -90,7 +89,7 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen | Alias | GPU | Current Model | Will Route To | |-------|-----|---------------|---------------| | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo | -| `gpu-dense` | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | Whatever runs on RTX 3090 | +| `gpu-dense` | RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | Whatever runs on RTX 3090 | | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 | **Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work @@ -100,7 +99,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | |-------|-----|------|------|-----|----------|----------|-------------|--------| -| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 | .8 (llm-gpu) | ~22.4/24.6GB (91%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ healthy | +| Qwen3.8-27B-Uncensored-Q4_K_M | RTX 3090 | .8 (llm-gpu) | ~16.8/24.6GB | **128K** | turbo4 | 1 | 2048/1024 | ✅ healthy | | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy | | qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s | @@ -110,7 +109,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap | Model | GPU | Weight | RPM Cap | Timeout | |-------|-----|--------|---------|---------| -| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | +| Qwen3.8-27B-Uncensored-Q4_K_M | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** | @@ -121,7 +120,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. | Model | RPM Cap | Notes | |-------|---------|-------| | strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload | -| SmartCode-Fable-5-27B-UD-Q3_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) | +| Qwen3.8-27B-Uncensored-Q4_K_M | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) | | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s | ### Stable Aliases (for agent configs — never change) @@ -251,9 +250,13 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster(). - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround. - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB. -- **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q3_K_XL (14.7GB). Q4 was too large for 24GB VRAM with 128K context + KV cache overhead. Q3 fits at ~22.4GB (91%). Uses standard llama.cpp build b9190 (turboquant b9150 incompatible with qwen3_5 arch). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. Service: `/home/llmuser/llama-fable-wrapper.sh`. +- **RTX 3090 (2026-08-15)**: Swapped to Qwen3.8-27B-Uncensored-Q4_K_M (~16.8GB, replaced + SmartCode-Fable-5-27B-UD-Q3_K_XL). Service: `/home/llmuser/llama-fable-wrapper.sh`. + Served under alias `qwen3.6-27B-code` for LiteLLM routing continuity. - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`. -- **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. +- **LiteLLM timeout tuning (verified 2026-08-16)**: Qwen3.8-27B (alias qwen3.6-27B-code) + 300s, gemma-4-12b 120s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all + 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. @@ -268,7 +271,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | |-----|-------|-----------|--------------|----------|---------| -| RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q3_K_XL | **TBD** | — | — | **128K** | +| RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | **TBD** | — | — | **128K** | | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** | | Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** | diff --git a/gpu-self-heal.prose.md b/gpu-self-heal.prose.md index 403657a..6afe6d5 100644 --- a/gpu-self-heal.prose.md +++ b/gpu-self-heal.prose.md @@ -44,7 +44,7 @@ depends_on: | Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role | |-------|-----|------|-------|------|-----|-------|------| -| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen | +| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | Qwen3.8-27B-Uncensored-Q4_K_M (alias qwen3.6-27B-code) | ~16.8/24.6GB | 128K | — | Heavy reasoning, code gen | | `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks | | `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs | diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index c9b27b6..c020116 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -41,10 +41,15 @@ description: > - **Verify**: Re-check status after 5 seconds - **Escalate**: If still failed after 2 retries, send Zulip DM to owner -### Rule 2: Process Restarting Too Often -- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter) +### Rule 2: Process Restarting Too Often (crash-loop guard) +- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter) **or** a + process reporting a restart count > 1000 while showing "online" (a crash-loop mask) - **Note**: PM2 counter never decrements; only full delete+re-add resets it - **Fix**: `pm2 delete && pm2 start --only ` +- **Script guard (2026-08-16)**: `scripts/pm2-self-heal.sh` now restarts `abiba-telegram` + when `TEL_RESTARTS > 1000` even if the process reports "online" — catches a quiet + crash-loop that never toggles status to "stopped"/"errored" (e.g. the 10k-restarts + spoton incident). Alerts include the restart count. - **Escalate**: Only when restarts > 30 — alerts to Zulip DM - **Historical fix**: Previous cycles were caused by abiba-zulip extension's stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2). diff --git a/scripts/pm2-self-heal.sh b/scripts/pm2-self-heal.sh index 45d454d..381a818 100755 --- a/scripts/pm2-self-heal.sh +++ b/scripts/pm2-self-heal.sh @@ -33,7 +33,7 @@ TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram") TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs) TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs) -if [ "$TEL_STATUS" != "online" ]; then +if [ "$TEL_STATUS" != "online" ] || [ "$TEL_RESTARTS" -gt 1000 ]; then pm2 restart abiba-telegram > /dev/null 2>&1 sleep 3 TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")