VLM migration: qwen3.5-9b-vlm → gemma-4-12b across entire harness
- router/router.py: model ID, context window (131K→262K), VRAM comment - litellm_config.yaml: model name updated - gpu-router.conf, gpu-router-docker.conf: pool names, comments - dashboard/dashboard.py: MODELS array refactor for single-point config - README.md: architecture diagram, model table
This commit is contained in:
@@ -8,7 +8,7 @@ CT 116 Docker stack for routing local GPU models through a unified OpenAI-compat
|
||||
nginx :80 → router :9000 → GPU backends
|
||||
├─ qwen3.6-35B-A3B (MoE) @ 192.168.68.15:8080 [2 slots]
|
||||
├─ qwen3.6-27B-code (Dense) @ 192.168.68.8:8080 [2 slots]
|
||||
└─ qwen3.5-9b-vlm (VLM) @ 192.168.68.110:8080 [2 slots]
|
||||
└─ gemma-4-12b (VLM) @ 192.168.68.110:8080 [2 slots]
|
||||
Total: 6 concurrent slots
|
||||
|
||||
LiteLLM :8081 (fallback) | Dashboard :3000 | Redis :6379 (local)
|
||||
@@ -63,7 +63,7 @@ When all GPUs are saturated, requests enter a polling queue (500ms intervals) in
|
||||
|-----|-------|------|-------|
|
||||
| Strix Halo | qwen3.6-35B-A3B (MoE) | 65GB | 2 |
|
||||
| RTX 3090 | qwen3.6-27B-code (Dense) | 24GB | 2 |
|
||||
| RTX 5070 | qwen3.5-9b-vlm (VLM) | 12GB | 2 |
|
||||
| New Backend | gemma-4-12b (VLM) | 7.1GB | 2 |
|
||||
|
||||
## Maintenance
|
||||
|
||||
|
||||
Reference in New Issue
Block a user