Abiba f2f8e8c921 feat: add request queuing to router (replaces hard 503 on saturation) ...

When all GPUs are saturated, requests now enter a queue loop (poll every 500ms)
instead of immediately returning 503. Configurable via QUEUE_TIMEOUT env var
(default 30s) or X-Queue-Timeout header per-request.

This prevents agent failures from cluster saturation — agents wait for a slot
instead of crashing on fallback.

2026-05-19 15:55:05 +00:00

..

Dockerfile

Initial commit: CT 116 inference harness — nginx, LiteLLM, router, dashboard, Redis

2026-05-16 18:51:50 +00:00

requirements.txt

Initial commit: CT 116 inference harness — nginx, LiteLLM, router, dashboard, Redis

2026-05-16 18:51:50 +00:00

router.py

feat: add request queuing to router (replaces hard 503 on saturation)

2026-05-19 15:55:05 +00:00