feat: add routing configuration with RPM caps (ornith syslog-auto: 40→60)
- Strix Halo ornith RPM increased from 40 to 60 inside syslog-auto pool - Prevented unbounded ornith traffic when multiple agents use syslog-auto - Direct ornith remains at RPM=40 - All models verified healthy (ornith 80ms, gemma 110ms, qwen 840ms) - Added full routing table to gpu-fleet contract
This commit is contained in:
@@ -84,6 +84,35 @@ triggers:
|
||||
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
|
||||
| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
|
||||
|
||||
## Routing Configuration (LiteLLM — July 2026)
|
||||
|
||||
### syslog-auto Weighted Pool
|
||||
|
||||
| Model | GPU | Weight | RPM Cap | Purpose |
|
||||
|-------|-----|--------|---------|---------|
|
||||
| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context |
|
||||
| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling |
|
||||
| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision |
|
||||
|
||||
### Direct Model Endpoints
|
||||
|
||||
| Model | RPM Cap | Notes |
|
||||
|-------|---------|-------|
|
||||
| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload |
|
||||
| qwen3.6-27B-code | 500 | High cap — primary workhorse |
|
||||
| gemma-4-12b | 500 | High cap — fast 12B |
|
||||
|
||||
### Fallback Chains
|
||||
- gemma → qwen
|
||||
- qwen → gemma
|
||||
- ornith → qwen → gemma
|
||||
- syslog-auto → qwen → gemma → ornith
|
||||
|
||||
### Why ornith RPM Is Capped
|
||||
- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks
|
||||
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
|
||||
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
|
||||
|
||||
## Operations
|
||||
|
||||
### add-model
|
||||
|
||||
Reference in New Issue
Block a user