From 1c44bf12590a4653d39c99e48a0d428abcc978f9 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 23 Jul 2026 08:59:35 +0000 Subject: [PATCH 1/3] =?UTF-8?q?contract=20updates:=20ornith=20decommission?= =?UTF-8?q?ed,=20256K=E2=86=92128K=20context,=20Mumuni=20Discord=20disable?= =?UTF-8?q?d?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change 1: Strix Halo — ornith decommissioned - gpu-fleet: Genesis Hermes V3 APEX → qwen3.6-35B-udq4 throughout - inference-optimization: ornith→strix-moe/qwen3.6-35B-udq4 - gpu-monitor: ornith status → Strix Halo status - infrastructure-control: Strix Halo — ornith → qwen3.6-35B-udq4 - infrastructure-update: ornith→strix-moe via router - proxmox-monitor: Strix Halo LLM (ornith) → (qwen3.6-35B-udq4, strix-moe) Change 2: GPU context 256K→128K fleet-wide - hermes-agent-baseline: frontmatter description updated - litellm-health: GPU Fleet Topology table 256K→128K - litellm-self-heal: GPU Fleet Topology, engine flags, VRAM alert - inference-optimization: compress threshold 256K→128K compact at 85K - gpu-fleet: instability note updated Change 3: Mumuni Discord platform disabled - gpu-fleet: Mumuni platforms: removed discord --- gpu-fleet.prose.md | 17 ++++++++--------- gpu-monitor.prose.md | 2 +- hermes-agent-baseline.prose.md | 2 +- inference-optimization.prose.md | 19 +++++++++---------- infrastructure-control.prose.md | 2 +- infrastructure-update.prose.md | 4 ++-- litellm-health.prose.md | 8 ++++---- litellm-self-heal.prose.md | 8 ++++---- proxmox-monitor.prose.md | 2 +- 9 files changed, 31 insertions(+), 33 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 7e938d9..05c00ff 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -10,9 +10,8 @@ description: > Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB). RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster). UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability. - Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored, - Hermes agent fine-tune, tensor repair, multimodal with mmproj). - Instability observed near 100K at 256K. 128K is the stable ceiling. + Strix Halo model swapped to qwen3.6-35B-udq4 (22GB, strix-moe alias). + Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling. For larger context needs → fall back to external providers (deepseek). VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%. agent: abiba @@ -99,7 +98,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap |-------|-----|------|------|-----|----------|----------|-------------|--------| | qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s | | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy | -| Genesis Hermes V3 APEX | Strix Halo Vulkan | .15 (amdpve) | ~10GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s | +| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s | ## Routing Configuration (LiteLLM — July 2026) @@ -108,7 +107,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap | Model | GPU | Weight | RPM Cap | Timeout | |-------|-----|--------|---------|---------| | qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | -| Genesis Hermes V3 APEX | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | +| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** | Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path. @@ -252,7 +251,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`. - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`. - **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. -- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V3-GGUF` (APEX quant), alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). Hermes agent fine-tune, tensor repair (SSM layers fixed via SVD), uncensored (0/465 refusals). +- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116. @@ -268,9 +267,9 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. |-----|-------|-----------|--------------|----------|---------| | RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** | | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** | -| Strix Halo (.15) | Genesis Hermes V3 APEX | **65** | 140 | — | **128K** | +| Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** | -Benchmarks from 2026-07-17. Strix Halo swapped to Genesis Hermes V3 APEX (LuffyTheFox). RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. +Benchmarks from 2026-07-17. Strix Halo model: qwen3.6-35B-udq4. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. All 3 GPUs now at 128K context (2026-07-17, reduced from 256K for stability). Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes. @@ -316,7 +315,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i | `compression.protect_last_n` | 40 | Preserves last 40 messages | | `memory.memory_char_limit` | 800 | Brief memory entries | | `personalities` | `creative` | Creative assistant personality | -| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms | +| Platforms | cli, homeassistant, signal, telegram, zulip | All Hermes platforms | | Main model timeout | 300s | LiteLLM global timeout | | Compression model timeout | 300s | strix-moe timeout increased from 120s | diff --git a/gpu-monitor.prose.md b/gpu-monitor.prose.md index a368c32..ee98873 100644 --- a/gpu-monitor.prose.md +++ b/gpu-monitor.prose.md @@ -47,7 +47,7 @@ poll .15:8080 directly; must go through router on .116. | Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status | | Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness | | LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count | -| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) | +| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | Strix Halo status via router — cannot poll .15:8080 directly (firewalled to .116 only) | | Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness | ### Alert Delivery diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 02b9f17..6b70252 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -5,7 +5,7 @@ version: 1.0.0 description: > Canonical known-good baseline for all Syslog Hermes agents. Captures the exact configuration state, keys, workarounds, and audit procedure. When an agent's - configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo). + configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 128K context (reduced from 256K for stability Jul 2026) (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo). author: Abiba (pi agent) --- diff --git a/inference-optimization.prose.md b/inference-optimization.prose.md index 91a10e6..646828f 100644 --- a/inference-optimization.prose.md +++ b/inference-optimization.prose.md @@ -22,14 +22,14 @@ management, and prompt caching — without sacrificing agent capability. - `agent-configs`: current config.yaml from each active Hermes agent (Mumuni .123, any others on .129/.122) including compression, model, context_window, prompt_caching, memory settings -- `gpu-health`: health check response from all 3 GPU backends (ornith .15:8080, +- `gpu-health`: health check response from all 3 GPU backends (strix-moe .15:8080, qwen .8:8080, gemma .110:8080) ### Maintains The optimized inference stack configuration — every change is applied and verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of -non-ornith traffic; ≤ 30000 for ornith-bound agentic calls. +inference calls. #### liteLLM-routing The syslog-auto routing weights, model-specific timeouts, RPM limits, and @@ -53,18 +53,17 @@ duration. ### Strategies -**Context is the root cause.** Every ~46K prompt token costs ~87s of ornith +**Context is the root cause.** Every ~46K prompt token costs ~87s of prefill time at 532 tok/s. Fix context first, routing second. -- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard - queries; gemma for compression/auxiliary. Never send simple completion to a - 35B MoE. -- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should - compact at 102K, not 166K. Target 15% tail (not 30%). +- **Route by task**: qwen for code/standard queries; gemma for + compression/auxiliary; strix-moe for compression tasks. +- **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should + compact at 85K, not 166K. Target 15% tail (not 30%). - **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these never change between turns. Single-digit cache hit rate is unacceptable. - **Lower context ceiling**: 128K window is the stable ceiling for agent conversations. - GPUs reduced from 256K to 128K (2026-07-17). For larger contexts, route to external providers. + GPUs reduced from 256K to 128K (2026-07-17). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers. ### Shape @@ -100,5 +99,5 @@ call enable-prompt-caching call verify-latency host: 192.168.68.116 - models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b, ornith-1.0-35b] + models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b] ``` diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index a29412c..8471017 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -199,7 +199,7 @@ description: > **Prometheus targets**: - 192.168.68.8:9400 (RTX 3090 — qwen) - 192.168.68.110:9400 (RTX 5070 — gemma) -- 192.168.68.15:9400 (Strix Halo — ornith) +- 192.168.68.15:9400 (Strix Halo — qwen3.6-35B-udq4) - 192.168.68.24:9401 (Router metrics exporter) - harness-litellm:4000 (LiteLLM health) diff --git a/infrastructure-update.prose.md b/infrastructure-update.prose.md index 301e48c..014044b 100644 --- a/infrastructure-update.prose.md +++ b/infrastructure-update.prose.md @@ -67,7 +67,7 @@ Before ANY update wave: - All VMs/CTs running: check via Proxmox API - LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116) - LiteLLM MCP tools: `curl localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` → 90 tools -- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only) +- GPU servers responding: check :8080 on VM 101, VM 103; check strix-moe via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only) - Zulip agents connected: check Mumuni/Tanko gateway state - Abiba PM2 processes online: `pm2 status` @@ -124,7 +124,7 @@ Before Wave 1, snapshot these files: /opt/home_stack/docker-compose.yml (VM 109 .7) /opt/audiobookshelf/docker-compose.yml (VM 109 .7) /root/.pi/agent/extensions/config.yaml (CT 100 .24) -/etc/systemd/system/ornith-server.service (amdpve .15) +/etc/systemd/system/ornith-server.service (amdpve .15 — strix-moe) /etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110) # Hermes agent configs (key enforcement — 2026-07-10) /root/.hermes/config.yaml (Mumuni CT 114, Tanko CT 112, etc.) diff --git a/litellm-health.prose.md b/litellm-health.prose.md index 66cfc36..615ab3c 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -42,7 +42,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) **What changed (v3.2.0 → v4.0.0 — 2026-07-08)**: - Router REMOVED from request path — LiteLLM proxies directly to GPU - All GPUs at parallel 2 (was parallel 1) -- NVIDIA context reduced 256K→128K to free VRAM (SUPERSEDED 2026-07-16: all GPUs back to 256K — see litellm-self-heal) +- NVIDIA context reduced 256K→128K to free VRAM — now the stable ceiling across all GPUs (2026-07-17) - LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal) - nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s @@ -72,9 +72,9 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) | Host | IP | Hardware | Models Served | Engine | Context | Parallel | |------|-----|----------|---------------|--------|---------|----------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 2 | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **256K** | 2 | -| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 256K | 2 | +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **128K** | 2 | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **128K** | 2 | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 128K | 2 | ## Model Fallback Chains (LiteLLM) diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 71ade65..86ffdd5 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -57,9 +57,9 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) | Host | IP | Hardware | Models Served | Engine | Context | Parallel | |------|-----|----------|---------------|--------|---------|----------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 262144 --parallel 2 --ngl 99`) | **256K** | 2 | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 262144 --parallel 2`, IQ4_NL + MTP draft) | **256K** | 2 | -| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 256K | 2 | +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 131072 --parallel 2 --ngl 99`) | **128K** | 2 | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 131072 --parallel 2`, IQ4_NL + MTP draft) | **128K** | 2 | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 128K | 2 | > Verified on ground 2026-07-16 via `curl /v1/models` on each host + `llama-wrapper.sh`. The AMD host's underlying model is `qwen3.6-35B-udq4`; LiteLLM exposes it under two `model_name`s: `qwen3.6-35B-udq4` and `strix-moe` (rpm 40). The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced. @@ -101,7 +101,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) ## Script Operations (synced 2026-07-16) - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`). -- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 256K context steady-state is ~96% on RTX 3090, not a fault). +- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault). - **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed). - **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM. diff --git a/proxmox-monitor.prose.md b/proxmox-monitor.prose.md index 33a0151..90fadaa 100644 --- a/proxmox-monitor.prose.md +++ b/proxmox-monitor.prose.md @@ -87,7 +87,7 @@ agent: abiba | storepve | 192.168.68.6 | PVE | | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) | | minipve | 192.168.68.12 | PVE | -| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) | +| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) | ## Operations -- 2.54.0 From e6c52bf0716af97937bab23721ad54aa6caa9a62 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 23 Jul 2026 09:04:51 +0000 Subject: [PATCH 2/3] no-mistakes(review): Fix F1 model rename and F2 128K compaction math in contracts --- gpu-self-heal.prose.md | 4 ++-- inference-optimization.prose.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/gpu-self-heal.prose.md b/gpu-self-heal.prose.md index 62cf205..911ed08 100644 --- a/gpu-self-heal.prose.md +++ b/gpu-self-heal.prose.md @@ -46,7 +46,7 @@ depends_on: |-------|-----|------|-------|------|-----|-------|------| | `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen | | `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks | -| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs | +| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs | Key notes: - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path. @@ -143,7 +143,7 @@ Key notes: - **Detect**: Benchmark tok/s vs baseline for each GPU at current context (all 128K) - RTX 3090 (128K ctx, ThinkingCap): baseline 74.8 tok/s — currently at 74.9 (100%) - RTX 5070 (128K ctx, HauhauCS QAT): baseline 165.2 tok/s — currently at 169.6 (103%) - - Strix Halo (128K ctx, Genesis Hermes V3): baseline 70.5 tok/s — currently at 62.9 (89%) + - Strix Halo (128K ctx, qwen3.6-35B-udq4): baseline 70.5 tok/s — currently at 62.9 (89%) - **Fix**: - If tok/s > baseline → context has headroom, consider increasing - If tok/s < 90% baseline → reduce context by 25% and retest diff --git a/inference-optimization.prose.md b/inference-optimization.prose.md index 646828f..318a4a9 100644 --- a/inference-optimization.prose.md +++ b/inference-optimization.prose.md @@ -59,7 +59,7 @@ prefill time at 532 tok/s. Fix context first, routing second. - **Route by task**: qwen for code/standard queries; gemma for compression/auxiliary; strix-moe for compression tasks. - **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should - compact at 85K, not 166K. Target 15% tail (not 30%). + compact at 51K, not 85K. Target 15% tail (not 30%). - **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these never change between turns. Single-digit cache hit rate is unacceptable. - **Lower context ceiling**: 128K window is the stable ceiling for agent conversations. -- 2.54.0 From 2b9b545ca92e7c1331709815bce455d36bbb08a7 Mon Sep 17 00:00:00 2001 From: root Date: Thu, 23 Jul 2026 09:10:43 +0000 Subject: [PATCH 3/3] no-mistakes(document): Fixed 2 stale model-name references in doc files --- gpu-fleet.prose.md | 2 +- hermes-config-template.prose.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 05c00ff..fcb7023 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -116,7 +116,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. | Model | RPM Cap | Notes | |-------|---------|-------| -| strix-moe (Hermes V3) | 40 | Tight cap — prevents Strix overload | +| strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload | | qwen3.6-27B-code | 500 | High cap — primary workhorse | | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s | diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 36171ce..b3d0dda 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -272,7 +272,7 @@ The following MUST be identical across ALL profiles: ### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18) - **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool - **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool -- **Strix Halo (64GB, 128K ctx, Geneis Hermes V3 APEX)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs +- **Strix Halo (64GB, 128K ctx, qwen3.6-35B-udq4)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs - Agent profiles MUST route auxiliary tasks to the correct GPU: - `auxiliary.vision.model: gpu-light` (RTX 5070) - `auxiliary.web_extract.model: gpu-light` (RTX 5070) -- 2.54.0