SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gpu-fleet: Strix Halo Vulkan rebuild + LiteLLM alignment (2026-07-01)

Browse Source 
- ornith now on Vulkan build 4fc4ec5, 256K ctx, ~70 tok/s
- Fixed GPU_MOE_URL bug in docker-compose (.110→.15)
- Added router sidecar fallback for health-checks
- Updated roster context 4096→262144 + correct model_path
- Documented port 8080 firewall (restricted to .116)
This commit is contained in:
root
2026-07-02 00:01:56 +00:00
parent 38d2f7d956
commit 49e8d968f5
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gpu-fleet.prose.md
+5 -2
View File
@@ -77,7 +77,7 @@ triggers:
|-------|-----|------|------|--------| |-------|-----|------|------|--------|
| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy | | qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy | | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
| ornith-1.0-35b | Strix Halo CPU | .15 (amdpve) | 28.2/64GB | ✅ healthy | | ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |
## Operations ## Operations
@@ -170,7 +170,10 @@ If no SSH access, send Zulip DM via abiba-bot.
- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy. - **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput. - **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`. - **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
- **Strix Halo GPU**: ROCm 6.3.1 installed, HIP compiler works via clang++ + device libs. **Blocked**: HSA runtime driver needs `rocm-llvm` → `libstdc++-11-dev` not on Debian 13. Fixes: Docker with ROCm, or reinstall to Ubuntu 24.04. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. Localhost curls hang (SYN dropped). Always test from .116.
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled. - **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds. - **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down. - **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.