AMD sidecar returns null for temp_c/power_w fields.
gpu_health_score now uses fallback for all numeric fields.
check_gpu_health also defaults temp_c and gpu_util_pct to 0.
Fixes 500 error on /metrics/gpu-health
Mumuni pushed parallel Phase 1-2 commits to syslog/main. Our branch
has the deployed-and-verified versions with gpu-health endpoint and
dashboard v2. Using ours strategy to accept our working tree.
- Added /metrics/gpu-health endpoint with live health scores (VRAM 40%, temp 30%, load 30%)
- Added /metrics/latency endpoint for dashboard KPIs
- Added GPU_LABELS for human-readable model names
- Dashboard v2: rewired to real data endpoints
- KPI cards: GPUs online, circuit trips, avg latency, req/min, active requests
- Health scores from actual gpu_health_score() function
- Rolling 60-sample history chart (real data, no simulation)
- Status: green/yellow/red based on tripped circuits
- No CDN dependency (pure CSS)
- Auto-refresh every 15s
- nginx: /dashboard/ serves static files with cache headers
- docker-compose: dashboard volume mount
Co-authored-by: Abiba <abiba@sysloggh.com>
- Added is_circuit_tripped(), trip_circuit(), half_open_probe() functions
- Filters out models with tripped circuits in route() function
- Trips circuit on 502/504 errors and timeouts in chat() function
- Prevents hung GPU cascades (Node #480 scenario)
Approved by Abiba via relay #635
Signed-off-by: Mumuni <mumuni@sysloggh.com>
- Added Nginx location block for circuit breaker metrics endpoint
- Enables visibility into which GPUs degrade most often
Signed-off-by: Mumuni <mumuni@sysloggh.com>
Restores the Phase 0 security features that were lost during the merge of
SyslogSolution/syslog-harness into abiba-bot/inference-harness (commit 815ed799).
Changes:
- Added ADMIN_KEY env var + _admin_auth() for endpoint protection
- Added GET /admin/keys — list all keys with deprecation status
- Added GET /admin/keys/deprecation-summary — deprecated key usage from Redis
- Added POST /admin/keys/generate — generate new keys (memory-only)
- Added POST /admin/keys/revoke — revoke deprecated keys (memory-only)
- Added dual-key deprecated key logging to chat() endpoint
- Preserves: Redis lazy reconnect, MoE spillover, 5-tier routing, vision guard
- Added qwen3.5-9b-vlm to MODELS array as (retired) with gray color
- Added modelLabel() / modelColor() safe lookup helpers
- Replaced all raw mlab[] and mcol[] lookups with safe fallbacks
- Unknown model IDs now show raw ID instead of 'undefined'
- Prevents stale Redis perf data from breaking renderPerf/renderScatter
- Removed all hardcoded API_KEYS from source (env var now REQUIRED)
- Added dual-key transition support (old keys deprecated, logged)
- Model sync: qwen3.5-9b-vlm → gemma-4-12b everywhere
- Dense context: 131K → 262K (all 3 models at 262K)
- Added /admin/keys endpoints for key lifecycle management
- Added ADMIN_KEY env var for admin auth
- Dashboard already synced to gemma-4-12b labels
Deployment: set API_KEYS from phase0-dual-keys.json in docker-compose
Transition: 7 days dual-key, then POST /admin/keys/revoke
- Added API_KEYS env var to router service in docker-compose.yml
- Replaced hardcoded agent keys in router.py fallback with dev-only placeholder
- Production now loads keys from environment, not source code
- Resolves the final remaining item from CT116 security deep-dive
Co-authored-by: Abiba via Kwame
Cross-agent GPU awareness ensures Tanko+Mumuni never
simultaneously hit MoE. Second agent always overflows
to Dense/VLM. MoE can safely use its extra VRAM with
2 slots since distinct agents never pile on.
OLD: checked only if CURRENT agent was on a GPU
Tanko→MoE, Mumuni also→MoE (didnt see Tanko)
NEW: checks if ANY agent is on a GPU (cross-agent awareness)
Pass 1: prefer GPUs with 0 agents
Pass 2: prefer GPU this agent is not already on
Pass 3: any non-busy GPU
Prevents Tanko+Mumuni piling onto same GPU simultaneously
even when both slots are free. Combined with MoE=1 slot,
guarantees overflow goes to idle Dense.
MoE at 95C with p50=13s latency — thermal throttling causing
death spiral. Both slots stuck processing for 113s p95.
Dense idle at 38C with 2 free slots. Reducing MoE to 1 slot
forces heavy overflow to Dense, giving MoE thermal headroom.
Heavy tier: MoE → Dense → VLM still valid — first heavy goes
to MoE, second overflows to Dense.
Mumuni 23K-token responses split the final SSE timings chunk
across HTTP frames. The old per-chunk check missed timings when
split. Now accumulates lines in a buffer before parsing.
Also fixed: store_perf_record accidentally dropped in prior edit.
select_best_gpu() now spreads different agents across GPUs:
- If agent already has a request on a GPU, prefer other GPUs first
- Tracked via Redis agent_gpu:{agent}:{model} with 120s TTL
- Same agent can still use multiple slots on same GPU if needed
- Falls back to normal priority when only one option available
Prevents Tanko+Mumuni from piling onto MoE simultaneously
while Dense sits idle. Each agent naturally spreads across
available GPUs.
Heavy tier keeps MoE primary (workhorse for >25K tok).
Default tier routes Dense → VLM → MoE to prevent MoE overload.
MoE had 5 timeouts in 15 min when Default pushed overflow to it.
More conversations now route to VLM as primary. 9B VLM has 262K
context window and 88 tok/s average — well suited for moderate
conversations. Dense absorbs overflow and heavy reasoning.
Strix Halo running qwen3.6-35B-A3B was hitting 94°C with 2 concurrent
slots, causing 300s request timeouts. Mumuni + Koby accumulated 15
timeouts in the last hour. Reduced to 1 slot for thermal headroom.
Medium and Default tiers already route VLM before MoE as fallback,
minimizing overflow traffic to the hot GPU.
Router: new /metrics/scatter endpoint returns individual data points
(prompt_tokens, inference_ms, model, agent, reason, stream)
for scatter visualization.
Dashboard: new panel showing latency vs prompt size by model.
- Log-scale X axis (prompt tokens) with model color coding
- Dropdown to filter by individual model or view all
- Hover tooltips with details per point
- Auto-refresh every 30s
Enables direct observation of context-length vs latency
relationship — validates routing tier decisions.
Router now buffers streaming response chunks to extract timings
(prompt_n, predicted_n, predicted_per_second) from the final
SSE data frame before yielding to the client. Streaming requests
get real throughput data instead of 0 tok/s.
Uses llama.cpp timings field in the last content chunk:
- completion_tokens = predicted_n
- tokens_per_sec = predicted_per_second
- inference_ms = predicted_ms (generation only)
Client sees identical stream, no perceptible delay.
- Dashboard: when a model has zero non-streaming records, shows
"streaming only" instead of misleading 0 tok/s
- Dashboard: minimum bar width enforced (6% avg, 4% p50) so
low-tps models are always visible
- Router: removed inflated streaming tps estimate (prompt tokens
skewed results for long conversations)
Fixes Dense model appearing to "register nothing" when Mumuni
sends mostly streaming requests.
Abiba
jerome