Abiba (pi) bf90e57c5f Load-aware routing: tracks active GPU requests in Redis, distributes overflow when MoE saturated. 6 concurrent requests now spread across all 3 GPUs instead of queuing on one.

2026-05-16 20:23:32 +00:00

..

Dockerfile

Initial commit: CT 116 inference harness — nginx, LiteLLM, router, dashboard, Redis

2026-05-16 18:51:50 +00:00

requirements.txt

Initial commit: CT 116 inference harness — nginx, LiteLLM, router, dashboard, Redis

2026-05-16 18:51:50 +00:00

router.py

Load-aware routing: tracks active GPU requests in Redis, distributes overflow when MoE saturated. 6 concurrent requests now spread across all 3 GPUs instead of queuing on one.

2026-05-16 20:23:32 +00:00