- Added extra_hosts for auth.sysloggh.net to LiteLLM container
- Fixed DOCS_URL=/docs (was /litellm/docs - path mismatch)
- Added Authentik self-signed cert to CA bundle
- Added nginx auth proxy for token/userinfo endpoints (SSL verify off)
- Changed OIDC token/userinfo endpoints to use nginx internal proxy
- Admin UI serving correctly on :4001/ui/ and /litellm/ui/
- Swagger API docs working at /docs and /litellm/docs
- ReDoc API docs working at /redoc and /litellm/redoc
- OIDC login flow verified working end-to-end
The nginx catch-all location / used proxy_pass $litellm_backend_url/ which
stripped the URI path, causing /openapi.json to return the Swagger UI HTML
page instead of the actual OpenAPI spec JSON. This broke the Swagger UI
rendering with "Unable to render this definition" error.
Fix: Add a dedicated location /openapi.json block before the catch-all /
that preserves the full path, so LiteLLM returns its valid openapi: 3.1.0
spec JSON at the public endpoint.
- Added LiteLLM fallback chains for explicit GPU models
- Changed allowed_fails: 0 -> 100 (router returns 503 on saturated)
- Documented strict passthrough router change (already deployed)
- Rewrote Appendix A as actual Model Identity Gap Analysis
- Added risk mitigation for fallback chain masking real failures
- Updated success metrics to reflect accurate per-model tracking
- Reviewed and approved by Mumuni and Kagenz0
AMD sidecar returns null for temp_c/power_w fields.
gpu_health_score now uses fallback for all numeric fields.
check_gpu_health also defaults temp_c and gpu_util_pct to 0.
Fixes 500 error on /metrics/gpu-health
Mumuni pushed parallel Phase 1-2 commits to syslog/main. Our branch
has the deployed-and-verified versions with gpu-health endpoint and
dashboard v2. Using ours strategy to accept our working tree.
- Added /metrics/gpu-health endpoint with live health scores (VRAM 40%, temp 30%, load 30%)
- Added /metrics/latency endpoint for dashboard KPIs
- Added GPU_LABELS for human-readable model names
- Dashboard v2: rewired to real data endpoints
- KPI cards: GPUs online, circuit trips, avg latency, req/min, active requests
- Health scores from actual gpu_health_score() function
- Rolling 60-sample history chart (real data, no simulation)
- Status: green/yellow/red based on tripped circuits
- No CDN dependency (pure CSS)
- Auto-refresh every 15s
- nginx: /dashboard/ serves static files with cache headers
- docker-compose: dashboard volume mount
Co-authored-by: Abiba <abiba@sysloggh.com>
- Added is_circuit_tripped(), trip_circuit(), half_open_probe() functions
- Filters out models with tripped circuits in route() function
- Trips circuit on 502/504 errors and timeouts in chat() function
- Prevents hung GPU cascades (Node #480 scenario)
Approved by Abiba via relay #635
Signed-off-by: Mumuni <mumuni@sysloggh.com>
- Added Nginx location block for circuit breaker metrics endpoint
- Enables visibility into which GPUs degrade most often
Signed-off-by: Mumuni <mumuni@sysloggh.com>
Restores the Phase 0 security features that were lost during the merge of
SyslogSolution/syslog-harness into abiba-bot/inference-harness (commit 815ed799).
Changes:
- Added ADMIN_KEY env var + _admin_auth() for endpoint protection
- Added GET /admin/keys — list all keys with deprecation status
- Added GET /admin/keys/deprecation-summary — deprecated key usage from Redis
- Added POST /admin/keys/generate — generate new keys (memory-only)
- Added POST /admin/keys/revoke — revoke deprecated keys (memory-only)
- Added dual-key deprecated key logging to chat() endpoint
- Preserves: Redis lazy reconnect, MoE spillover, 5-tier routing, vision guard
- Added qwen3.5-9b-vlm to MODELS array as (retired) with gray color
- Added modelLabel() / modelColor() safe lookup helpers
- Replaced all raw mlab[] and mcol[] lookups with safe fallbacks
- Unknown model IDs now show raw ID instead of 'undefined'
- Prevents stale Redis perf data from breaking renderPerf/renderScatter
- Removed all hardcoded API_KEYS from source (env var now REQUIRED)
- Added dual-key transition support (old keys deprecated, logged)
- Model sync: qwen3.5-9b-vlm → gemma-4-12b everywhere
- Dense context: 131K → 262K (all 3 models at 262K)
- Added /admin/keys endpoints for key lifecycle management
- Added ADMIN_KEY env var for admin auth
- Dashboard already synced to gemma-4-12b labels
Deployment: set API_KEYS from phase0-dual-keys.json in docker-compose
Transition: 7 days dual-key, then POST /admin/keys/revoke
- Added API_KEYS env var to router service in docker-compose.yml
- Replaced hardcoded agent keys in router.py fallback with dev-only placeholder
- Production now loads keys from environment, not source code
- Resolves the final remaining item from CT116 security deep-dive
Co-authored-by: Abiba via Kwame
Cross-agent GPU awareness ensures Tanko+Mumuni never
simultaneously hit MoE. Second agent always overflows
to Dense/VLM. MoE can safely use its extra VRAM with
2 slots since distinct agents never pile on.
OLD: checked only if CURRENT agent was on a GPU
Tanko→MoE, Mumuni also→MoE (didnt see Tanko)
NEW: checks if ANY agent is on a GPU (cross-agent awareness)
Pass 1: prefer GPUs with 0 agents
Pass 2: prefer GPU this agent is not already on
Pass 3: any non-busy GPU
Prevents Tanko+Mumuni piling onto same GPU simultaneously
even when both slots are free. Combined with MoE=1 slot,
guarantees overflow goes to idle Dense.
MoE at 95C with p50=13s latency — thermal throttling causing
death spiral. Both slots stuck processing for 113s p95.
Dense idle at 38C with 2 free slots. Reducing MoE to 1 slot
forces heavy overflow to Dense, giving MoE thermal headroom.
Heavy tier: MoE → Dense → VLM still valid — first heavy goes
to MoE, second overflows to Dense.
Mumuni 23K-token responses split the final SSE timings chunk
across HTTP frames. The old per-chunk check missed timings when
split. Now accumulates lines in a buffer before parsing.
Also fixed: store_perf_record accidentally dropped in prior edit.
select_best_gpu() now spreads different agents across GPUs:
- If agent already has a request on a GPU, prefer other GPUs first
- Tracked via Redis agent_gpu:{agent}:{model} with 120s TTL
- Same agent can still use multiple slots on same GPU if needed
- Falls back to normal priority when only one option available
Prevents Tanko+Mumuni from piling onto MoE simultaneously
while Dense sits idle. Each agent naturally spreads across
available GPUs.
Heavy tier keeps MoE primary (workhorse for >25K tok).
Default tier routes Dense → VLM → MoE to prevent MoE overload.
MoE had 5 timeouts in 15 min when Default pushed overflow to it.
Abiba
jerome