From aebc98ead6a3972f0eb52d859c7853d62f8a2dcc Mon Sep 17 00:00:00 2001 From: root Date: Sat, 18 Jul 2026 17:48:52 +0000 Subject: [PATCH] fix: remove trailing whitespace from litellm-api-keys.prose.md --- gpu-fleet.prose.md | 40 ++++++++++++++------------------------- litellm-api-keys.prose.md | 10 +++++----- 2 files changed, 19 insertions(+), 31 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 7a6dc86..7e938d9 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -9,9 +9,9 @@ description: > gpu-dense, gpu-light. These never change — only the underlying model does. Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB). RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster). - UPDATED 2026-07-17: Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored, + UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability. + Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored, Hermes agent fine-tune, tensor repair, multimodal with mmproj). - RTX 5070 swapped to HauhauCS Gemma4-12B QAT Uncensored Balanced (Q4_K_M, 87 tok/s, 0/465 refusals). Instability observed near 100K at 256K. 128K is the stable ceiling. For larger context needs → fall back to external providers (deepseek). VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%. @@ -87,32 +87,20 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen | Alias | GPU | Current Model | Will Route To | |-------|-----|---------------|---------------| | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo | -| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code (ThinkingCap) | Whatever runs on RTX 3090 | +| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 | | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 | **Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work but are deprecated for agent configs. Only the stable aliases survive model swaps. -## Current Model Assignments (2026-07-17) +## Current Model Assignments (2026-07-15) | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | |-------|-----|------|------|-----|----------|----------|-------------|--------| -| qwen3.6-27B-code (ThinkingCap) | RTX 3090 | .8 (llm-gpu) | ~20.9/24.6GB (85%) | **128K** | turbo4 | 1 | default | ✅ 68 tok/s | -| gemma-4-12b (HauhauCS QAT) | RTX 5070 | .110 (ocu-llm) | ~10.0/12.2GB (82%) | 128K | q4_0 | 1 | 2048/1024 | ✅ 87 tok/s | +| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s | +| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy | | Genesis Hermes V3 APEX | Strix Halo Vulkan | .15 (amdpve) | ~10GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s | -> **RTX 5070 model swap (2026-07-17)**: Switched from `gemma-4-12b-it-IQ4_NL` (Unsloth, 191 tok/s) -> to `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` (Q4_K_M QAT, 87 tok/s). -> Trade: 54% slower generation for QAT quality, 0/465 refusals, and agent-optimized tuning. -> MTP draft also swapped: Q8_0 (444MB) → tuned draft (242MB), saving 200MB VRAM. -> Role unchanged: gpu-light (vision, web extract, light auxiliary tasks). - -> **RTX 3090 model swap (2026-07-17)**: Switched from `Qwopus3.6-27B-v2-MTP-Q4_K_M` (63 tok/s) -> to `bottlecapai/ThinkingCap-Qwen3.6-27B` (Q4_K_M QAT, 68 tok/s). -> RL-finetuned: 50% fewer thinking tokens, MMLU-Pro 0.85 vs 0.83 base. -> Self-spec MTP REQUIRED (crashes without it on turboquant build). -> Added vision via mmproj (0.9GB). VRAM 85%. - ## Routing Configuration (LiteLLM — July 2026) ### syslog-auto Weighted Pool (Direct GPU — bypasses router) @@ -130,16 +118,16 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. | Model | RPM Cap | Notes | |-------|---------|-------| | strix-moe (Hermes V3) | 40 | Tight cap — prevents Strix overload | -| qwen3.6-27B-code | 500 | ThinkingCap Q4_K_M + MTP self-spec + vision, 68 tok/s | -| gemma-4-12b | 500 | HauhauCS QAT Uncensored Balanced + MTP, 87 tok/s | +| qwen3.6-27B-code | 500 | High cap — primary workhorse | +| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s | ### Stable Aliases (for agent configs — never change) | Alias | RPM Cap | Routes To | Purpose | |-------|---------|-----------|---------| | `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) | -| `gpu-dense` | 500 | RTX 3090 (ThinkingCap) | Heavy reasoning, code gen, delegation | -| `gpu-light` | 500 | RTX 5070 (HauhauCS QAT) | Vision, web extract, light tasks | +| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning | +| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks | ### Fallback Chains - gemma → qwen @@ -261,8 +249,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround. - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB. - **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2). -- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching --spec-type draft-mtp --spec-draft-n-max 4`. ThinkingCap Qwen3.6-27B Q4_K_M (15.7GB) + mmproj (0.9GB) + MTP self-spec. VRAM: ~85%. Service: `/home/llmuser/llama-wrapper.sh`. ⚠️ MTP REQUIRED for stability on this turboquant build — model segfaults without `--spec-type draft-mtp`. Outputs reasoning_content (hidden from agent, improves answer quality). -- **RTX 5070 config (2026-07-17)**: HauhauCS Gemma4-12B QAT Uncensored Balanced (Q4_K_M) + tuned MTP draft (242MB) at 128K context, single slot. Gen speed: 87 tok/s (vs 191 IQ4_NL). VRAM: ~10.0/12.2GB (~82%). Service: `/home/llmuser/llama-wrapper.sh`. Model: `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`, MTP: `mtp-gemma-4-12B-it.gguf`, mmproj: `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`. Recommended sampling: temp 0.6, top_k 64, top_p 0.9, min_p 0.05, repeat_penalty 1.1. +- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`. +- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`. - **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V3-GGUF` (APEX quant), alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). Hermes agent fine-tune, tensor repair (SSM layers fixed via SVD), uncensored (0/465 refusals). - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). @@ -278,8 +266,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | |-----|-------|-----------|--------------|----------|---------| -| RTX 3090 (.8) | ThinkingCap-Qwen3.6-27B Q4_K_M | **68** | — | — | **128K** | -| RTX 5070 (.110) | HauhauCS QAT Uncensored Balanced | **87** | — | — | **128K** | +| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** | +| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** | | Strix Halo (.15) | Genesis Hermes V3 APEX | **65** | 140 | — | **128K** | Benchmarks from 2026-07-17. Strix Halo swapped to Genesis Hermes V3 APEX (LuffyTheFox). RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index 44624f6..c492206 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -8,18 +8,18 @@ description: > Ensures agents never use the master key directly. Rotation is event-driven, not calendar-driven — rotate only on compromise, personnel change, or periodic security hygiene (quarterly/annually). - + UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production) BUT each agent host MUST keep a local .env fallback. Infisical service tokens can expire/404. The .env fallback prevents agents from running without keys. Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours. - + UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed. Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending. Abiba's key is now a proper agent key (NOT the master key — stale note removed). - + UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo) standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) → infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key @@ -30,7 +30,7 @@ description: > Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently. (3) Capture /proc//environ before gateway restarts to preserve running env set. - + Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml on CT 116. Last verified: 2026-07-17. @@ -154,7 +154,7 @@ through its agent wrapper. The `ExecStart=` (empty reset) clears any ExecStart from the main unit file, then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file regeneration** by `hermes gateway install` — the drop-in always wins. - + **Why a drop-in instead of editing the unit file:** `hermes gateway install` (called during Hermes updates and some self-heal operations) regenerates the systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.