diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 37925bf..390c36d 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -84,6 +84,35 @@ triggers: | gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy | | ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy | +## Routing Configuration (LiteLLM — July 2026) + +### syslog-auto Weighted Pool + +| Model | GPU | Weight | RPM Cap | Purpose | +|-------|-----|--------|---------|---------| +| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context | +| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling | +| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision | + +### Direct Model Endpoints + +| Model | RPM Cap | Notes | +|-------|---------|-------| +| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload | +| qwen3.6-27B-code | 500 | High cap — primary workhorse | +| gemma-4-12b | 500 | High cap — fast 12B | + +### Fallback Chains +- gemma → qwen +- qwen → gemma +- ornith → qwen → gemma +- syslog-auto → qwen → gemma → ornith + +### Why ornith RPM Is Capped +- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks +- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously +- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C + ## Operations ### add-model