abiba-bot/inference-harness
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

chore(ct116): capture production state (LiteLLM 1.99.1, nginx /ui //docs, router decommission, trove agent)

Browse Source
This commit is contained in:
agent-zero
2026-09-11 19:34:30 +00:00
parent 7cad063e27
commit 93e7605aa2
21 changed files with 196 additions and 1929 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -2
View File
@@ -8,7 +8,7 @@ CT 116 Docker stack for routing local GPU models through a unified OpenAI-compat
nginx :80 → router :9000 → GPU backends
├─ qwen3.6-35B-A3B (MoE) @ 192.168.68.15:8080 [2 slots, 262K ctx]
├─ qwen3.6-27B-code (Dense) @ 192.168.68.8:8080 [2 slots, 262K ctx]
└─ gemma-4-12b (VLM) @ 192.168.68.110:8080 [2 slots, 262K ctx]
└─ gpu-vision (VLM) @ 192.168.68.110:8080 [2 slots, 262K ctx]
Total: 6 concurrent slots
LiteLLM :8081 (fallback) | Dashboard :3000 | Redis :6379 (local)
@@ -63,7 +63,7 @@ When all GPUs are saturated, requests enter a polling queue (500ms intervals) in
|-----|-------|------|-------|
| Strix Halo | qwen3.6-35B-A3B (MoE) | 65GB | 2 | 262K | General quality |
| RTX 3090 | qwen3.6-27B-code (Dense) | 24GB | 2 | 262K | Code, reasoning |
| RTX 5070 | gemma-4-12b (VLM) | 12GB | 2 | 262K | Speed, vision |
| RTX 5070 | gpu-vision (VLM) | 12GB | 2 | 262K | Speed, vision |
## Maintenance