diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index c0e2770..29b172c 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -77,7 +77,7 @@ triggers: |-------|-----|------|------|--------| | qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy | | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy | -| ornith-1.0-35b | Strix Halo CPU | .15 (amdpve) | 28.2/64GB | ✅ healthy | +| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy | ## Operations @@ -170,7 +170,10 @@ If no SSH access, send Zulip DM via abiba-bot. - **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy. - **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput. - **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`. -- **Strix Halo GPU**: ROCm 6.3.1 installed, HIP compiler works via clang++ + device libs. **Blocked**: HSA runtime driver needs `rocm-llvm` → `libstdc++-11-dev` not on Debian 13. Fixes: Docker with ROCm, or reinstall to Ubuntu 24.04. +- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV. +- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. Localhost curls hang (SYN dropped). Always test from .116. +- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback. +- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected. - **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled. - **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds. - **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.