SyslogSolution/syslog-harness
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: SyslogSolution/syslog-harness:867a6189dff4de55d2f881720225f1c2ec1c67a6
Branches Tags
SyslogSolution/syslog-harness:main SyslogSolution/syslog-harness:master
...
compare: SyslogSolution/syslog-harness:815ed7991fb16184260d474c9db644d2cd33fb64
Branches Tags
SyslogSolution/syslog-harness:main SyslogSolution/syslog-harness:master
43 Commits
867a6189df ... 815ed7991f
Author SHA1 Message Date
Abiba 815ed7991f Merge SyslogSolution/syslog-harness: accept current state (Phase 0 + Redis lazy reconnect + dashboard fix) 2026-06-07 23:14:28 +00:00
Abiba 633afc5e29 Router: lazy Redis reconnect (survives Redis restarts/reboots) 2026-06-07 23:01:20 +00:00
Abiba 85608d7c60 Dashboard + LiteLLM config updates from maintenance 2026-06-07 22:49:50 +00:00
Abiba 24f0928ea1 Phase 0: model migration (qwen3.5-9b-vlm → gemma-4-12b), context alignment (all 262K), routing tiers + MoE spillover, fix dashboard window=1h parsing 2026-06-07 22:49:46 +00:00
AbibaandAbiba via Kwame 0cb4597b0e security: move API keys to env var, strip from source code fallback 
- Added API_KEYS env var to router service in docker-compose.yml
- Replaced hardcoded agent keys in router.py fallback with dev-only placeholder
- Production now loads keys from environment, not source code
- Resolves the final remaining item from CT116 security deep-dive

Co-authored-by: Abiba via Kwame
 2026-06-03 12:40:04 +00:00
Abiba 9a633583ab fix: Security hardening from CT116 deep-dive review 
- API keys moved from hardcoded dict to env var (API_KEYS JSON) with fallback
- Rate limiting added: token bucket per API key (Redis-backed), 429 responses with Retry-After
- Rate limit tiers: enterprise 120/min, professional 60/min, starter 20/min
- X-RateLimit-* headers on all responses
- Dashboard polling reduced from 3s to 5s backend, 10s JS fallback
- SSE detection disables redundant polling when stream is connected
- Deleted ts_patch.py (dead one-shot migration, already applied)
- Added ssl/README.md documenting upstream SSL termination

Ref: Relay #444 (Mumuni CT116 harness deep-dive)
Reviewed-by: Abiba <abiba@sysloggh.com>
 2026-06-02 10:37:10 +00:00
Abiba 060a47fce9 revert: MoE back to 2 slots (cross-agent spread now prevents hotspot) 
Cross-agent GPU awareness ensures Tanko+Mumuni never
simultaneously hit MoE. Second agent always overflows
to Dense/VLM. MoE can safely use its extra VRAM with
2 slots since distinct agents never pile on.
 2026-05-30 13:15:19 +00:00
Abiba 34fb7516e1 fix: cross-agent GPU spreading prevents hotspot hammering 
OLD: checked only if CURRENT agent was on a GPU
  Tanko→MoE, Mumuni also→MoE (didnt see Tanko)

NEW: checks if ANY agent is on a GPU (cross-agent awareness)
  Pass 1: prefer GPUs with 0 agents
  Pass 2: prefer GPU this agent is not already on
  Pass 3: any non-busy GPU

Prevents Tanko+Mumuni piling onto same GPU simultaneously
even when both slots are free. Combined with MoE=1 slot,
guarantees overflow goes to idle Dense.
 2026-05-30 12:55:29 +00:00
Abiba acbcb20837 fix: MoE concurrency 2→1 (95C thermal emergency) 
MoE at 95C with p50=13s latency — thermal throttling causing
death spiral. Both slots stuck processing for 113s p95.
Dense idle at 38C with 2 free slots. Reducing MoE to 1 slot
forces heavy overflow to Dense, giving MoE thermal headroom.

Heavy tier: MoE → Dense → VLM still valid — first heavy goes
to MoE, second overflows to Dense.
 2026-05-30 12:52:23 +00:00
Abiba a3bca93d9b fix: buffer SSE chunks for large streaming responses 
Mumuni 23K-token responses split the final SSE timings chunk
across HTTP frames. The old per-chunk check missed timings when
split. Now accumulates lines in a buffer before parsing.

Also fixed: store_perf_record accidentally dropped in prior edit.
 2026-05-29 09:45:41 +00:00
Abiba d53685d874 feat: agent-aware GPU load balancing 
select_best_gpu() now spreads different agents across GPUs:
- If agent already has a request on a GPU, prefer other GPUs first
- Tracked via Redis agent_gpu:{agent}:{model} with 120s TTL
- Same agent can still use multiple slots on same GPU if needed
- Falls back to normal priority when only one option available

Prevents Tanko+Mumuni from piling onto MoE simultaneously
while Dense sits idle. Each agent naturally spreads across
available GPUs.
 2026-05-28 21:45:23 +00:00
Abiba 54a4f26db7 fix: Default tier back to Dense-first (MoE overheating at 91°C) 
Heavy tier keeps MoE primary (workhorse for >25K tok).
Default tier routes Dense → VLM → MoE to prevent MoE overload.
MoE had 5 timeouts in 15 min when Default pushed overflow to it.
 2026-05-28 21:40:18 +00:00
Abiba fb1d51b93b restructure: routing prioritized by reasoning requirements 
Tier 1 (Lightweight): VLM → Dense → MoE     ≤500 tok, 1 turn
Tier 2 (Simple):      VLM → Dense → MoE     ≤15K tok, ≤12 turns (was 10K/10)
Tier 3 (Medium):      Dense → VLM → MoE     ≤25K tok
Tier 4 (Heavy):       MoE → Dense → VLM     >25K tok (MoE PRIMARY workhorse)
Tier 5 (Default):     MoE → Dense → VLM     MoE primary fallback

Target: MoE ~50% (heavy primary), VLM ~25% (raised simple + fallback),
        Dense ~25% (medium primary + heavy fallback)

Removed turn limit from Medium tier — Simple tier handles conversational
requests up to 12 turns now.
 2026-05-27 07:22:30 +00:00
Abiba 9a0d69ce8d feat: Dense 128K context + 2 slots, VLM second in Heavy tier 
- Dense GPU_CONTEXT: 192K→128K (131072) to free VRAM
- Dense max_concurrent: 1→2 (VRAM now sufficient)
- Heavy tier: Dense → VLM → MoE (VLM handles 262K context)
- Total slots: 6 (2 Dense + 2 MoE + 2 VLM)

Distribution target: Dense 50%, VLM 30%, MoE 20%

NOTE: Requires llama.cpp restart on 192.168.68.8 with --ctx-size 131072
 2026-05-27 07:15:58 +00:00
Abiba 621a897bec tune: raise Tier 2 threshold 4K→10K tok, 6→10 turns for VLM 
More conversations now route to VLM as primary. 9B VLM has 262K
context window and 88 tok/s average — well suited for moderate
conversations. Dense absorbs overflow and heavy reasoning.
 2026-05-27 00:29:25 +00:00
Abiba 93d0d3cc4b revert: MoE concurrency back to 2 (Dense-first routing handles thermal) 2026-05-27 00:04:42 +00:00
Abiba c4ea5e3a98 fix: flip Tier 4 (Heavy) to Dense-first for thermal safety 
Dense → MoE → VLM instead of MoE → Dense → VLM.
Combined with MoE at 1 concurrent slot, Dense absorbs all
primary traffic. MoE only activates when Dense saturated.
Prevents Strix Halo from hitting 94C thermal limit.
 2026-05-27 00:01:33 +00:00
Abiba ebe8f9ced4 fix: reduce MoE concurrency 2→1 to prevent thermal timeout (94°C) 
Strix Halo running qwen3.6-35B-A3B was hitting 94°C with 2 concurrent
slots, causing 300s request timeouts. Mumuni + Koby accumulated 15
timeouts in the last hour. Reduced to 1 slot for thermal headroom.

Medium and Default tiers already route VLM before MoE as fallback,
minimizing overflow traffic to the hot GPU.
 2026-05-26 23:47:08 +00:00
Abiba b3db0841ef feat: redesigned routing tiers for even GPU distribution + speed priority 
OLD: Dense was last choice in every tier, got 4% of auto-routed traffic
NEW: 5-tier routing with speed-first prioritization

Tier 1 (Lightweight): VLM → Dense → MoE    (≤500 tok, ≤100 words)
Tier 2 (Simple):      VLM → Dense → MoE    (≤4000 tok, ≤6 turns)
Tier 3 (Medium):      DENSE → MoE → VLM    (≤25000 tok, ≤15 turns)
Tier 4 (Heavy):       MoE → Dense → VLM    (>25000 tok or >15 turns)
Tier 5 (Default):     DENSE → MoE → VLM    (balanced fallback)

Also: quality hint now routes to MoE (better reasoning)
Bugfix: Tier 1 now checks token count to prevent giant single-word
inputs from being routed as lightweight
 2026-05-26 22:00:20 +00:00
Abiba 80362fa528 fix: default performance window to 24h so all models appear immediately 2026-05-26 12:37:52 +00:00
Abiba 7ef9e58f61 fix: restore /api/performance route in dashboard (was overwritten to /api/timeseries) 2026-05-26 12:31:53 +00:00
Abiba f47c3f3304 feat: latency vs prompt size scatter plot on dashboard 
Router: new /metrics/scatter endpoint returns individual data points
(prompt_tokens, inference_ms, model, agent, reason, stream)
for scatter visualization.

Dashboard: new panel showing latency vs prompt size by model.
- Log-scale X axis (prompt tokens) with model color coding
- Dropdown to filter by individual model or view all
- Hover tooltips with details per point
- Auto-refresh every 30s

Enables direct observation of context-length vs latency
relationship — validates routing tier decisions.
 2026-05-26 12:18:31 +00:00
Abiba cfb05fa501 feat: capture streaming token counts from SSE final chunk 
Router now buffers streaming response chunks to extract timings
(prompt_n, predicted_n, predicted_per_second) from the final
SSE data frame before yielding to the client. Streaming requests
get real throughput data instead of 0 tok/s.

Uses llama.cpp timings field in the last content chunk:
- completion_tokens = predicted_n
- tokens_per_sec = predicted_per_second
- inference_ms = predicted_ms (generation only)

Client sees identical stream, no perceptible delay.
 2026-05-25 19:58:51 +00:00
Abiba b2ec4b0572 fix: throughput panel handles streaming-only models gracefully 
- Dashboard: when a model has zero non-streaming records, shows
  "streaming only" instead of misleading 0 tok/s
- Dashboard: minimum bar width enforced (6% avg, 4% p50) so
  low-tps models are always visible
- Router: removed inflated streaming tps estimate (prompt tokens
  skewed results for long conversations)

Fixes Dense model appearing to "register nothing" when Mumuni
sends mostly streaming requests.
 2026-05-25 19:45:21 +00:00
Abiba 8c5c922a4e fix: handle single data point in performance percentiles 2026-05-25 17:00:40 +00:00
Abiba f42747d721 feat: performance analytics panel on dashboard 
dashboard/dashboard.py (+61 lines):
- New /api/performance endpoint proxying to router metrics/performance
- Performance Analytics row with 4 panels:
  - Latency distribution (p50/p95/p99 per model) with stacked bars
  - Throughput comparison (avg + p50 tokens/sec per model)
  - Routing effectiveness table by reason
  - Agent performance bars with latency
- 1h/24h window toggle, auto-refresh every 15s
- Color-coded per model (purple=MoE, amber=Dense, green=VLM)
 2026-05-25 16:58:15 +00:00
Abiba b849cd3395 feat: per-request performance tracking + /metrics/performance endpoint 
router/router.py (+158 lines):
- store_perf_record(): captures queue_ms, inference_ms, prompt_tokens,
  completion_tokens, tokens_per_sec per request in Redis
- Per-model, per-reason, per-agent rolling windows (last 200-500)
- /metrics/performance?window=N endpoint with percentiles (p50/p95/p99)
  for latency, throughput, and queue time per model/reason/agent
- Queue time now surfaced in routing metadata and routes:recent
- Streaming requests tracked with estimated prompt tokens

nginx/nginx.conf:
- Added /metrics/ proxy pass to router_api

Enables model performance comparison and routing tier validation.
 2026-05-25 16:50:45 +00:00
Abiba b7882b2434 fix: reduce 27B Dense context to 192K to free VRAM 
RTX 3090 was at 94.9% VRAM at 262K context. Reduced to 192K (196608),
freeing ~2.4GB. VRAM now at 85% with room for active inference.
 2026-05-25 00:31:40 +00:00
Abiba ddde6646de fix: decouple VRAM usage from saturation status 
VRAM percentage no longer marks GPU as saturated.
Saturation is about slot availability (handled by is_gpu_busy()),
not memory usage. Added vram_warning boolean flag (≥95% threshold)
for informational monitoring without affecting routing decisions.

27B Dense now correctly shows healthy at 91% VRAM.
 2026-05-23 06:00:37 +00:00
Abiba 41939104c7 fix: non-blocking GPU health checks + 256K turboquant context upgrade 
router/router.py:
- check_gpu_health() now accepts configurable timeouts (sidecar_timeout, gpu_timeout)
- /health and /v1/models endpoints use fast 1.5s/1s timeouts (non-blocking)
- /v1/models now calls check_gpu_health once per model instead of twice
- GPU_CONTEXT updated to 262144 across all models (turboquant upgrade)
- 27B max_concurrent reduced 2→1 (24GB VRAM saturated at 256K context)

docker-compose.yml:
- Router healthcheck timeout 5s→15s, interval 15s→30s
- Nginx healthcheck timeout 5s→15s, interval 15s→30s

Fixes dashboard hang when any GPU is unreachable.
 2026-05-23 05:57:13 +00:00
Abiba 0983337fdb fix: heavy tier Dense→MoE→VLM 2026-05-19 21:24:36 +00:00
Abiba 28d62e27ba feat: context-aware routing + compaction signals 2026-05-19 21:13:57 +00:00
Abiba 714ebb003e fix: heavy threshold → 50000 tokens, 25 turns 2026-05-19 21:08:18 +00:00
Abiba e90bf0216d fix: raise heavy threshold — 4000→12000 tokens, 8→15 turns 2026-05-19 20:10:07 +00:00
Abiba 5971ceee4e security: reject requests without valid API key (401) 2026-05-19 19:15:13 +00:00
Abiba 5f05f46c7c fix: heavy tier — Dense first for reasoning, MoE workhorse, VLM overflow 2026-05-19 18:27:24 +00:00
Abiba 911fdc9f3f fix: routing priority — MoE first, VLM second, Dense last 2026-05-19 17:38:29 +00:00
Abiba d9d2c213f6 fix: routing — remove turn limit from default tier, no gaps 2026-05-19 17:24:41 +00:00
Abiba 6625892908 feat: redesigned routing tiers — VLM handles more traffic 2026-05-19 17:01:58 +00:00
Abiba fcb99a26c8 revert: remove Ollama endpoints 2026-05-19 16:57:05 +00:00
Abiba 2234d03079 fix: add /v1/props and /v1/models/<id> endpoints 2026-05-19 16:08:58 +00:00
Abiba 5b99b16712 feat: add request queuing to router (replaces hard 503) 2026-05-19 15:55:13 +00:00
Abiba 28fc57c5c7 May 19, 2026: Full harness update 
- Model migration: gemma-4-E4B → qwen3.5-9b-vlm
- Dashboard reorder: Usage Over Time + GPU Metrics to top
- Router counter leak fix (gpu_decr in except handler)
- VLM slot upgrade 1→2
- Automated maintenance cron job
- LiteLLM config update
 2026-05-19 15:03:47 +00:00
12 changed files with 1810 additions and 218 deletions
Expand all files Collapse all files
backups/20260602_103344/dashboard.py.bak
+356
View File
@@ -0,0 +1,356 @@
"""SyslogAI Harness Dashboard — Modern Design."""
import os, json, time, queue, threading
import requests
from flask import Flask, request, render_template_string, Response, stream_with_context
ROUTER_METRICS = os.environ.get("ROUTER_METRICS_URL", "http://router:9000/metrics")
app = Flask(__name__)
sse_subscribers = []; sse_lock = threading.Lock()
def fetch_state():
try:
r = requests.get(ROUTER_METRICS, timeout=5)
if r.status_code == 200: return r.json()
except Exception: pass
return {"gpus":[],"route_counts":{},"agent_counts":{},"recent":[],"timestamp":time.time()}
def broadcast_loop():
while True:
time.sleep(3)
data = fetch_state(); payload = json.dumps(data)
with sse_lock:
dead = [q for q in sse_subscribers if not q.put(payload)]
for q in dead: sse_subscribers.remove(q)
threading.Thread(target=broadcast_loop, daemon=True).start()
DASHBOARD_HTML = r"""<!DOCTYPE html>
<html lang="en" data-bs-theme="dark">
<head>
<meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>SyslogAI Harness</title>
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" rel="stylesheet">
<style>
body { background: #0b0f17; color: #bcc3cd; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', system-ui, sans-serif; padding: 20px 24px; }
.card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; height: 100%; }
.stat-card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; padding: 18px 20px; text-align: center; }
.stat-value { font-size: 28px; font-weight: 700; line-height: 1.1; }
.stat-label { font-size: 11px; text-transform: uppercase; letter-spacing: 0.6px; color: #64748b; margin-top: 4px; }
.gpu-card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; padding: 16px 18px; height: 100%; }
.gpu-card .title { font-size: 13px; font-weight: 600; color: #e2e8f0; margin-bottom: 12px; display: flex; align-items: center; gap: 8px; }
.gpu-card .status-dot { width: 8px; height: 8px; border-radius: 50%; flex-shrink: 0; }
.gpu-card .row-metric { display: flex; justify-content: space-between; font-size: 12px; padding: 2px 0; }
.gpu-card .row-metric .lbl { color: #64748b; }
.gpu-card .row-metric .val { color: #e2e8f0; font-variant-numeric: tabular-nums; }
.gpu-card .slot-bar { display: flex; gap: 3px; margin-top: 8px; }
.gpu-card .slot-bar .s { flex: 1; height: 5px; border-radius: 2px; background: #1e293b; }
.gpu-card .slot-bar .s.active { background: #38bdf8; }
.chart-card { background: #111827; border: 1px solid #1e293b; border-radius: 10px; padding: 16px 18px; height: 100%; display: flex; flex-direction: column; }
.chart-card .title { font-size: 13px; font-weight: 600; color: #e2e8f0; margin-bottom: 12px; }
.bar-row { margin-bottom: 8px; }
.bar-label { display: flex; justify-content: space-between; font-size: 11px; margin-bottom: 3px; color: #64748b; }
.bar-label .name { color: #cbd5e1; }
.bar-track { height: 5px; background: #1e293b; border-radius: 3px; overflow: hidden; }
.bar-fill { height: 100%; border-radius: 3px; transition: width 0.6s ease; }
.table-custom { font-size: 11px; margin: 0; }
.table-custom th { color: #64748b; font-weight: 500; font-size: 10px; text-transform: uppercase; border-color: #1e293b; padding: 8px 10px; }
.table-custom td { color: #94a3b8; border-color: rgba(30,41,59,0.5); padding: 6px 10px; }
.agent-badge { font-size: 10px; padding: 2px 7px; border-radius: 8px; font-weight: 600; }
.btn-sm-period { font-size: 10px; padding: 3px 10px; border-radius: 6px; border: 1px solid #1e293b; color: #64748b; background: transparent; cursor: pointer; }
.btn-sm-period.active { background: #1d4ed8; color: #fff; border-color: #1d4ed8; }
.ring-label { font-size: 22px; font-weight: 700; }
.ring-sublabel { font-size: 10px; color: #64748b; }
</style>
</head>
<body>
<!-- HEADER -->
<div class="d-flex justify-content-between align-items-center mb-4">
<div>
<h5 class="mb-0 text-white fw-bold">&#x26A1; SyslogAI Harness</h5>
<div class="small text-secondary" id="live-indicator">
<span class="status-dot" id="live-dot" style="width:6px;height:6px;border-radius:50%;display:inline-block;background:#22c55e;animation:pulse 2s infinite"></span>
<span id="connection-status">live</span> &middot; <span id="update-time"></span>
</div>
</div>
<div class="d-flex gap-2">
<div class="stat-card" style="min-width:100px"><div class="stat-value text-info" id="kpi-total">0</div><div class="stat-label">Requests</div></div>
<div class="stat-card" style="min-width:100px"><div class="stat-value text-warning" id="kpi-active">0</div><div class="stat-label">Active</div></div>
<div class="stat-card" style="min-width:100px"><div class="stat-value" style="color:#a78bfa" id="kpi-agents">0</div><div class="stat-label">Agents</div></div>
</div>
</div>
<div class="row g-3 align-items-stretch">
<!-- ROW 1: Usage Chart (8) + GPU Metrics (4) -->
<div class="col-md-8"><div class="chart-card"><div class="title d-flex justify-content-between align-items-center">
<span>Usage Over Time</span>
<div class="d-flex gap-1">
<button class="btn-sm-period active" onclick="switchPeriod('day')">24h</button>
<button class="btn-sm-period" onclick="switchPeriod('week')">7d</button>
<button class="btn-sm-period" onclick="switchPeriod('month')">30d</button>
</div>
</div><div id="timeseries-chart" style="height:150px"></div><div id="timeseries-legend" class="d-flex justify-content-center gap-3 mt-2 flex-wrap small"></div></div></div>
<div class="col-md-4"><div class="chart-card"><div class="title">GPU Metrics</div><div id="gpu-metrics-card"></div></div></div>
<!-- ROW 2: 3 GPU Cards -->
<div class="col-md-4"><div class="gpu-card" id="gpu-moe"><div class="text-secondary small">Loading...</div></div></div>
<div class="col-md-4"><div class="gpu-card" id="gpu-dense"><div class="text-secondary small">Loading...</div></div></div>
<div class="col-md-4"><div class="gpu-card" id="gpu-light"><div class="text-secondary small">Loading...</div></div></div>
<!-- ROW 3: Queue + Model + Agent -->
<div class="col-md-4"><div class="chart-card"><div class="title">Queue Status</div><div class="text-center" id="queue-viz"></div></div></div>
<div class="col-md-4"><div class="chart-card"><div class="title">Model Distribution</div><div id="route-bars"></div></div></div>
<div class="col-md-4"><div class="chart-card"><div class="title">Agent Activity</div><div id="agent-bars"></div></div></div>
<!-- ROW 4: Performance Analytics -->
<div class="col-12 mb-2"><div class="d-flex align-items-center gap-2"><span class="fw-bold text-white" style="font-size:14px">&#x1F4CA; Performance Analytics</span>
<div class="d-flex gap-1 ms-auto">
<button class="btn-sm-period active" onclick="switchPerfWindow('1')">1h</button>
<button class="btn-sm-period" onclick="switchPerfWindow('24')">24h</button>
</div>
</div></div>
<div class="col-md-6"><div class="chart-card"><div class="title">Latency — P50 / P95 / P99 (ms)</div><div id="perf-latency"></div></div></div>
<div class="col-md-6"><div class="chart-card"><div class="title">Throughput — Tokens / sec</div><div id="perf-throughput"></div></div></div>
<div class="col-md-6"><div class="chart-card"><div class="title">Routing Effectiveness — by Reason</div><div id="perf-reasons"></div></div></div>
<div class="col-md-6"><div class="chart-card"><div class="title">Agent Performance</div><div id="perf-agents"></div></div></div>
<!-- ROW 5: Latency vs Context Scatter -->
<div class="col-12"><div class="chart-card"><div class="title d-flex justify-content-between align-items-center">
<span>Latency vs Prompt Size — by Model</span>
<div class="d-flex gap-2">
<select id="scatter-model" onchange="loadScatter()" style="font-size:10px;background:#1e293b;color:#94a3b8;border:1px solid #334155;border-radius:4px;padding:2px 6px">
<option value="all">All Models</option>
<option value="qwen3.5-9b-vlm">9B VLM</option>
<option value="qwen3.6-27B-code">27B Dense</option>
<option value="qwen3.6-35B-A3B">35B MoE</option>
</select>
</div>
</div><div id="scatter-plot" style="height:200px;position:relative"></div><div id="scatter-legend" class="d-flex justify-content-center gap-3 mt-2 flex-wrap small"></div></div></div>
<!-- ROW 6: Live Stream -->
<div class="col-12"><div class="chart-card"><div class="title">Live Stream</div>
<div class="table-responsive"><table class="table table-custom mb-0">
<thead><tr><th>Time</th><th>Agent</th><th>Model</th><th>Reason</th><th>Tier</th></tr></thead>
<tbody id="route-tbody"></tbody>
</table></div>
</div></div>
</div>
<script>
var MC={'qwen3.5-9b-vlm':'#22c55e','qwen3.6-27B-code':'#f59e0b','qwen3.6-35B-A3B':'#a78bfa'};
var ML={'qwen3.5-9b-vlm':'Qwen3.5 9B VLM','qwen3.6-27B-code':'Qwen Code','qwen3.6-35B-A3B':'Qwen MoE'};
var GL={'qwen3.6-35B-A3B':'MoE - Strix Halo','qwen3.6-27B-code':'Dense - RTX 3090','qwen3.5-9b-vlm':'VLM - RTX 5070'};
function $(id){return document.getElementById(id);}
function render(data){
if(!data||!data.gpus)return;
var t=Object.values(data.route_counts||{}).reduce((a,b)=>a+b,0);
var ta=0,tm=0;data.gpus.forEach(function(g){ta+=(g.active_requests||0);tm+=(g.max_concurrent||1)});
$('kpi-total').textContent=t;$('kpi-active').textContent=ta+'/'+tm;$('kpi-agents').textContent=Object.keys(data.agent_counts||{}).length;
$('update-time').textContent=new Date().toLocaleTimeString();
var ids={'qwen3.6-35B-A3B':'gpu-moe','qwen3.6-27B-code':'gpu-dense','qwen3.5-9b-vlm':'gpu-light'};
data.gpus.forEach(function(g){
var el=$(ids[g.id]);if(!el)return;
var a=g.active_requests||0,mx=g.max_concurrent||1;
var sc=g.status==='healthy'?'#22c55e':g.status==='saturated'?'#f59e0b':'#ef4444';
var ss=g.status==='healthy'?'Online':g.status==='saturated'?'Busy':'Offline';
var slots='';for(var i=0;i<mx;i++)slots+='<span class=\"s'+(i<a?' active':'')+'\"></span>';
var h='<div class=\"title\"><span class=\"status-dot\" style=\"background:'+sc+'\"></span>'+GL[g.id]+'<span class=\"ms-auto small\" style=\"color:'+sc+'\">'+ss+'</span></div>';
h+='<div class=\"row-metric\"><span class=\"lbl\">VRAM</span><span class=\"val\">'+g.vram_used_mb+' / '+g.vram_total_mb+' MB</span></div>';
h+='<div class=\"row-metric\"><span class=\"lbl\">Utilization</span><span class=\"val\">'+g.gpu_util_pct+'%</span></div>';
h+='<div class=\"row-metric\"><span class=\"lbl\">Temperature</span><span class=\"val\" style=\"color:'+(g.temp_c>85?'#ef4444':g.temp_c>70?'#f59e0b':'#22c55e')+'\">'+g.temp_c+'C</span></div>';
if(g.power_w)h+='<div class=\"row-metric\"><span class=\"lbl\">Power</span><span class=\"val\">'+g.power_w+'W'+(g.power_limit_w?'/'+g.power_limit_w+'W':'')+'</span></div>';
h+='<div class=\"row-metric\"><span class=\"lbl\">Slots</span><span class=\"val\" style=\"color:'+(a>=mx?'#ef4444':'#e2e8f0')+'\">'+a+' / '+mx+'</span></div>';
h+='<div class=\"slot-bar\">'+slots+'</div>';el.innerHTML=h;
});
renderQueue(data);renderGPUMetrics(data);
var rc=data.route_counts||{},mr=Math.max(1,...Object.values(rc));
$('route-bars').innerHTML=Object.entries(rc).length?Object.entries(rc).sort((a,b)=>b[1]-a[1]).map(function(e){var m=e[0],c=e[1];return'<div class=\"bar-row\"><div class=\"bar-label\"><span class=\"name\">'+(ML[m]||m)+'</span><span>'+c+' ('+(t?Math.round(c/t*100):0)+'%)</span></div><div class=\"bar-track\"><div class=\"bar-fill\" style=\"width:'+(c/mr*100)+'%;background:'+(MC[m]||'#38bdf8')+'\"></div></div></div>';}).join(''):'<div class=\"text-secondary small\">-</div>';
var ac=data.agent_counts||{},ma=Math.max(1,...Object.values(ac));
$('agent-bars').innerHTML=Object.entries(ac).length?Object.entries(ac).sort((a,b)=>b[1]-a[1]).map(function(e){return'<div class=\"bar-row\"><div class=\"bar-label\"><span class=\"name\">'+e[0]+'</span><span>'+e[1]+'</span></div><div class=\"bar-track\"><div class=\"bar-fill\" style=\"width:'+(e[1]/ma*100)+'%;background:#38bdf8\"></div></div></div>';}).join(''):'<div class=\"text-secondary small\">-</div>';
var recent=data.recent||[];
$('route-tbody').innerHTML=recent.length?recent.slice(0,20).map(function(r){var d=new Date(r.ts*1000),ag=r.agent||'?';return'<tr><td class=\"text-secondary\">'+d.toLocaleTimeString()+'</td><td><span class=\"agent-badge\" style=\"background:rgba(56,189,248,0.12);color:#38bdf8\">'+ag+'</span></td><td>'+(ML[r.model]||r.model)+'</td><td class=\"text-secondary\">'+(r.reason||'')+'</td><td class=\"text-uppercase\" style=\"font-size:10px;color:'+(r.tier==='enterprise'?'#a78bfa':'#64748b')+'\">'+(r.tier||'')+'</td></tr>';}).join(''):'<tr><td colspan=\"5\" class=\"text-secondary\">Waiting...</td></tr>';
}
function renderQueue(data){
var el=$('queue-viz');if(!el)return;
var ta=0,tm=0;data.gpus.forEach(function(g){ta+=(g.active_requests||0);tm+=(g.max_concurrent||1)});
var pct=tm>0?Math.round(ta/tm*100):0,st=pct>=100?'SATURATED':pct>=50?'BUSY':'IDLE';
var sc=pct>=100?'#ef4444':pct>=50?'#f59e0b':'#22c55e';
var circ=188.5,dash=(pct/100)*circ;
var h='<div class=\"d-inline-block position-relative mb-2\"><svg width=\"72\" height=\"72\"><circle cx=\"36\" cy=\"36\" r=\"30\" fill=\"none\" stroke=\"#1e293b\" stroke-width=\"6\"/><circle cx=\"36\" cy=\"36\" r=\"30\" fill=\"none\" stroke=\"'+sc+'\" stroke-width=\"6\" stroke-dasharray=\"'+dash+' '+(circ-dash)+'\" stroke-linecap=\"round\" transform=\"rotate(-90 36 36)\"/></svg><div style=\"position:absolute;top:50%;left:50%;transform:translate(-50%,-50%);text-align:center\"><div class=\"ring-label\" style=\"color:'+sc+'\">'+ta+'</div><div class=\"ring-sublabel\">/ '+tm+' slots</div></div></div>';
h+='<div class=\"fw-bold mb-2 small\" style=\"color:'+sc+'\">'+st+'</div>';
var lb={'qwen3.6-35B-A3B':'MoE','qwen3.6-27B-code':'Dense','qwen3.5-9b-vlm':'VLM'};
data.gpus.forEach(function(g){var a=g.active_requests||0,mx=g.max_concurrent||1,gp=mx>0?Math.round(a/mx*100):0;h+='<div class=\"d-flex align-items-center gap-2 mb-1 justify-content-center\"><span class=\"small\" style=\"min-width:32px;text-align:right;font-size:10px\">'+(lb[g.id]||g.id)+'</span><div style=\"flex:1;max-width:70px;height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+gp+'%;background:'+sc+';border-radius:2px\"></div></div><span class=\"small\" style=\"min-width:22px;font-size:10px\">'+a+'/'+mx+'</span></div>'});
el.innerHTML=h;
}
function renderGPUMetrics(data){
var el=$('gpu-metrics-card');if(!el)return;
var lb={'qwen3.6-35B-A3B':'MoE','qwen3.6-27B-code':'Dense','qwen3.5-9b-vlm':'VLM'};
var h='';data.gpus.forEach(function(g){
var nm=lb[g.id]||g.id,tp=g.temp_c||0,ut=g.gpu_util_pct||0,pw=g.power_w||0,pl=g.power_limit_w||0;
var tc=tp>85?'#ef4444':tp>70?'#f59e0b':'#22c55e',uc=ut>90?'#ef4444':ut>70?'#f59e0b':'#22c55e';
h+='<div class=\"mb-3\"><div class=\"fw-bold small text-white-50 mb-1\">'+nm+'</div>';
h+='<div class=\"d-flex align-items-center gap-2 mb-1\"><span class=\"small text-secondary\" style=\"min-width:30px\">T</span><div class=\"flex-grow-1\" style=\"height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+Math.min(tp,100)+'%;background:'+tc+';border-radius:2px\"></div></div><span class=\"small\" style=\"color:'+tc+';min-width:30px;text-align:right\">'+tp+'C</span></div>';
h+='<div class=\"d-flex align-items-center gap-2 mb-1\"><span class=\"small text-secondary\" style=\"min-width:30px\">U</span><div class=\"flex-grow-1\" style=\"height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+ut+'%;background:'+uc+';border-radius:2px\"></div></div><span class=\"small\" style=\"color:'+uc+';min-width:30px;text-align:right\">'+ut+'%</span></div>';
if(pw>0){var pp=pl>0?Math.round(pw/pl*100):0,pc=pp>90?'#ef4444':pp>70?'#f59e0b':'#22c55e';h+='<div class=\"d-flex align-items-center gap-2\"><span class=\"small text-secondary\" style=\"min-width:30px\">P</span><div class=\"flex-grow-1\" style=\"height:3px;background:#1e293b;border-radius:2px;overflow:hidden\"><div style=\"height:100%;width:'+pp+'%;background:'+pc+';border-radius:2px\"></div></div><span class=\"small\" style=\"color:'+pc+';min-width:30px;text-align:right\">'+pw+'W</span></div>';}
h+='</div>';});
el.innerHTML=h;
}
var cp='day';
function switchPeriod(p){cp=p;document.querySelectorAll('.btn-sm-period').forEach(function(b){b.classList.remove('active')});event.target.classList.add('active');loadTS();}
function loadTS(){fetch('/api/timeseries?period='+cp).then(function(r){return r.json()}).then(renderTS).catch(function(){})}
function renderTS(d){
var models=d.models||{},labels=d.labels||[];
if(!labels.length)return;
var cn=$('timeseries-chart'),lg=$('timeseries-legend'),mn=Object.keys(models);
if(!mn.length){cn.innerHTML='<div class=\"text-secondary small text-center py-4\">-</div>';return;}
var mv=1;for(var m in models)for(var i=0;i<models[m].length;i++)if(models[m][i]>mv)mv=models[m][i];mv=Math.ceil(mv*1.15)||1;
var W=labels.length>1?100/(labels.length-1):100,H=130;
var paths='';for(var mi=0;mi<mn.length;mi++){var m=mn[mi],vals=models[m]||[],d='';for(var i=0;i<vals.length;i++){var x=i*W,y=H-(vals[i]/mv)*H;d+=(i===0?'M':'L')+x.toFixed(1)+','+y.toFixed(1)+' ';}paths+='<path d=\"'+d+'\" fill=\"none\" stroke=\"'+(MC[m]||'#38bdf8')+'\" stroke-width=\"2\" stroke-linecap=\"round\" opacity=\"0.8\"/>';}
var grid='';for(var g=0;g<=4;g++){var y=(g/4)*H;grid+='<line x1=\"0\" y1=\"'+y.toFixed(1)+'\" x2=\"100\" y2=\"'+y.toFixed(1)+'\" stroke=\"#1e293b\" stroke-width=\"1\"/>';}
cn.innerHTML='<svg viewBox=\"0 0 100 '+(H+16)+'\" style=\"width:100%;height:'+(H+20)+'px;display:block\" preserveAspectRatio=\"none\">'+grid+paths+'</svg>';
lg.innerHTML=mn.map(function(m){return'<span class=\"d-flex align-items-center gap-1\"><svg width=\"14\" height=\"8\"><line x1=\"0\" y1=\"4\" x2=\"14\" y2=\"4\" stroke=\"'+(MC[m]||'#38bdf8')+'\" stroke-width=\"2\"/></svg>'+(ML[m]||m)+'</span>';}).join('');
}
var perfWindow='24';
function switchPerfWindow(w){perfWindow=w;document.querySelectorAll('.btn-sm-period').forEach(function(b,i){if(i>=4)b.classList.toggle('active',b.textContent.trim().replace('h','')===w)});loadPerf();}
function loadPerf(){fetch('/api/performance?window='+perfWindow).then(function(r){return r.json()}).then(renderPerf).catch(function(){})}
function renderPerf(d){
var models=d.models||[],reasons=d.reasons||[],agents=d.agents||[],sum=d.summary||{};
// Latency bars: p50/p95/p99 per model
var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','qwen3.5-9b-vlm':'9B VLM'};
var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','qwen3.5-9b-vlm':'#22c55e'};
if(!models.length){$('perf-latency').innerHTML='<div class="text-secondary small text-center py-4">Accumulating data...</div>';return;}
var maxLat=Math.max(...models.map(function(m){return m.latency.p99||0}),1);
var latHTML=models.map(function(m){
var l=m.latency||{},p50=l.p50||0,p95=l.p95||0,p99=l.p99||0,c=mcol[m.model]||'#38bdf8';
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span class="text-secondary">'+m.count+' reqs</span></div>'+
'<div class="d-flex align-items-center gap-2 mb-1"><span class="text-secondary" style="min-width:28px">p50</span><div class="flex-grow-1" style="height:14px;background:#1e293b;border-radius:4px;overflow:hidden;position:relative"><div style="position:absolute;left:0;top:0;height:100%;width:'+(p50/maxLat*100)+'%;background:'+c+';opacity:0.3;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p95/maxLat*100)+'%;background:'+c+';opacity:0.5;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p99/maxLat*100)+'%;background:'+c+';border-radius:4px"></div></div><span style="color:'+c+';min-width:48px;text-align:right;font-variant-numeric:tabular-nums">'+p99+'ms</span></div>'+
'<div class="d-flex gap-3" style="font-size:10px;color:#64748b;padding-left:32px"><span>p50: '+p50+'ms</span><span>p95: '+p95+'ms</span><span>p99: '+p99+'ms</span></div></div>';
}).join('');
$('perf-latency').innerHTML=latHTML;
// Throughput comparison
var maxTps=Math.max(...models.map(function(m){return m.throughput.avg_tokens_per_sec||0}),1);
var tpsHTML=models.map(function(m){
var t=m.throughput||{},avg=t.avg_tokens_per_sec||0,p50=t.p50||0,c=mcol[m.model]||'#38bdf8';
var isAllStreaming = avg===0 && p50===0;
if(isAllStreaming){
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:#64748b;font-style:italic">streaming only</span></div><div class="text-secondary" style="font-size:10px">t/s available for non-streaming requests only</div></div>';
}
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:'+c+'" class="fw-bold">'+avg+' tok/s</span></div>'+
'<div class="d-flex align-items-center gap-2"><span class="text-secondary" style="min-width:28px">avg</span><div class="flex-grow-1" style="height:6px;background:#1e293b;border-radius:3px;overflow:hidden"><div style="height:100%;width:'+(Math.max(avg/maxTps*100,6))+'%;background:'+c+';border-radius:3px"></div></div><span class="small" style="color:'+c+';min-width:54px;text-align:right">'+avg+' tok/s</span></div>'+
'<div class="d-flex align-items-center gap-2 mt-1"><span class="text-secondary" style="min-width:28px;font-size:10px">p50</span><div class="flex-grow-1" style="height:4px;background:#1e293b;border-radius:2px;overflow:hidden"><div style="height:100%;width:'+(Math.max(p50/maxTps*100,4))+'%;background:'+c+';opacity:0.5;border-radius:2px"></div></div><span style="font-size:10px;color:#64748b">'+p50+' tok/s</span></div></div>';
}).join('');
$('perf-throughput').innerHTML=tpsHTML;
// Routing reasons table
if(reasons.length){
var rHTML='<table class="table table-custom mb-0"><thead><tr><th>Reason</th><th>Count</th><th>Avg Lat</th><th>P95 Lat</th></tr></thead><tbody>';
reasons.forEach(function(r){rHTML+='<tr><td>'+r.reason+'</td><td>'+r.count+'</td><td>'+r.avg_total_ms+'ms</td><td>'+r.p95_total_ms+'ms</td></tr>';});
rHTML+='</tbody></table>';$('perf-reasons').innerHTML=rHTML;
}else{$('perf-reasons').innerHTML='<div class="text-secondary small text-center py-3">-</div>';}
// Agent performance
if(agents.length){
var maxAc=Math.max(...agents.map(function(a){return a.count||0}),1);
var aHTML=agents.map(function(a){return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+a.agent+'</span><span class="text-secondary">'+a.count+' reqs</span></div><div class="d-flex align-items-center gap-2"><div class="flex-grow-1" style="height:4px;background:#1e293b;border-radius:2px;overflow:hidden"><div style="height:100%;width:'+(a.count/maxAc*100)+'%;background:#38bdf8;border-radius:2px"></div></div><span class="small" style="color:#38bdf8;min-width:60px;text-align:right">'+a.avg_total_ms+'ms avg</span></div></div>';}).join('');
$('perf-agents').innerHTML=aHTML;
}else{$('perf-agents').innerHTML='<div class="text-secondary small text-center py-3">-</div>';}
}
function poll(){fetch('/api/state').then(function(r){return r.json()}).then(function(data){render(data);$('connection-status').textContent='live';}).catch(function(){$('connection-status').textContent='reconnecting';});}
function loadScatter(){
var m=$('scatter-model').value;
fetch('/api/scatter?window=24&model='+m).then(function(r){return r.json()}).then(renderScatter).catch(function(){});
}
function renderScatter(d){
var pts=d.points||[],el=$('scatter-plot'),lg=$('scatter-legend');
if(!pts.length){el.innerHTML='<div class="text-secondary small text-center py-5">No data yet</div>';return;}
var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','qwen3.5-9b-vlm':'#22c55e','unknown':'#38bdf8'};
var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','qwen3.5-9b-vlm':'9B VLM'};
var maxX=Math.max.apply(null,pts.map(function(p){return p.prompt_tokens||0}))||1000;
var maxY=Math.max.apply(null,pts.map(function(p){return p.inference_ms||0}))||5000;
// Log scale for X axis (prompt tokens vary widely)
var toX=function(t){return Math.log10(Math.max(t,1))/Math.log10(Math.max(maxX,10))*100;};
var toY=function(t){return (t/maxY)*100;};
var dots='';
pts.forEach(function(p){
var x=toX(p.prompt_tokens),y=toY(p.inference_ms),c=mcol[p.model]||'#38bdf8';
var r=p.stream?1.5:2.5,o=p.stream?0.4:0.8;
dots+='<circle cx="'+x+'" cy="'+(100-y)+'" r="'+r+'" fill="'+c+'" opacity="'+o+'"><title>'+mlab[p.model]+' | '+p.prompt_tokens+' tok | '+p.inference_ms+'ms | '+p.agent+'</title></circle>';
});
// Grid lines
var grid='';
for(var i=1;i<=4;i++){grid+='<line x1="0" y1="'+(i*20)+'" x2="100" y2="'+(i*20)+'" stroke="#1e293b" stroke-width="0.5"/>';}
for(var i=1;i<=4;i++){grid+='<line x1="'+(i*20)+'" y1="0" x2="'+(i*20)+'" y2="100" stroke="#1e293b" stroke-width="0.5"/>';}
// Axis labels
var xTicks='';
var xVals=[10,100,1000,10000,100000];
xVals.forEach(function(v){if(v<=maxX)xTicks+='<text x="'+toX(v)+'" y="103" text-anchor="middle" font-size="8" fill="#64748b">'+(v>=1000?(v/1000)+'k':v)+'</text>';});
var yTicks='';
var yVals=[500,1000,5000,10000,50000,100000];
yVals.forEach(function(v){if(v<=maxY)yTicks+='<text x="-2" y="'+(97-toY(v))+'" text-anchor="end" font-size="8" fill="#64748b">'+(v>=1000?(v/1000)+'s':v+'ms')+'</text>';});
el.innerHTML='<svg viewBox="-35 0 140 115" style="width:100%;height:200px">'+grid+dots+xTicks+yTicks+'<text x="50" y="112" text-anchor="middle" font-size="9" fill="#475569">Prompt Tokens (log scale)</text><text x="-38" y="50" text-anchor="middle" font-size="9" fill="#475569" transform="rotate(-90,-38,50)">Inference Time</text></svg>';
// Legend
var models=[];pts.forEach(function(p){if(models.indexOf(p.model)===-1)models.push(p.model);});
lg.innerHTML=models.map(function(m){return'<span class="d-flex align-items-center gap-1 small"><svg width="10" height="10"><circle cx="5" cy="5" r="3.5" fill="'+(mcol[m]||'#38bdf8')+'"/></svg>'+mlab[m]+'</span>';}).join('');
}
poll();setInterval(poll,3000);loadTS();loadPerf();setInterval(loadPerf,15000);loadScatter();setInterval(loadScatter,30000);
</script>
</body>
</html>"""
@app.route("/")
def dashboard(): return render_template_string(DASHBOARD_HTML)
@app.route("/api/state")
def api_state(): return fetch_state()
@app.route("/api/scatter")
def api_scatter():
window = request.args.get("window", "24")
model = request.args.get("model", "all")
try:
r = requests.get(f"http://router:9000/metrics/scatter?window={window}&model={model}", timeout=10)
if r.status_code == 200: return r.json()
except Exception: pass
return {"points": [], "count": 0}
@app.route("/api/performance")
def api_performance():
window = request.args.get("window", "24")
model = request.args.get("model", "all")
try:
r = requests.get(f"http://router:9000/metrics/performance?window={window}&model={model}", timeout=10)
if r.status_code == 200: return r.json()
except Exception: pass
return {"models": [], "reasons": [], "agents": [], "summary": {"total_requests": 0}}
@app.route("/api/timeseries")
def api_timeseries():
period = request.args.get("period", "day")
try:
r = requests.get("http://router:9000/metrics/timeseries?period=" + period, timeout=5)
if r.status_code == 200: return r.json()
except Exception: pass
return {"models": {}, "labels": []}
@app.route("/api/stream")
def api_stream():
def ev():
q = queue.Queue()
with sse_lock: sse_subscribers.append(q)
try:
yield "data: "+json.dumps(fetch_state())+"\n\n"
while True:
try: msg = q.get(timeout=3); yield "data: "+msg+"\n\n"
except queue.Empty: yield "data: "+json.dumps(fetch_state())+"\n\n"
except GeneratorExit: pass
finally:
with sse_lock:
if q in sse_subscribers: sse_subscribers.remove(q)
return Response(stream_with_context(ev()), mimetype="text/event-stream", headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
@app.route("/health")
def health(): return {"status":"healthy","service":"harness-dashboard"}
if __name__ == "__main__":
app.run(host="0.0.0.0", port=3000, debug=False)
backups/20260602_103344/router.py.bak
+659
View File
@@ -0,0 +1,659 @@
import os, json, time, logging, traceback, threading, queue, statistics, math
import requests, redis
from flask import Flask, request, jsonify, Response, stream_with_context
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
GPU_SIDECARS = {
"qwen3.6-35B-A3B": "http://192.168.68.15:8090",
"qwen3.6-27B-code": "http://192.168.68.8:8090",
"qwen3.5-9b-vlm": "http://192.168.68.110:8090",
}
GPU_URLS = {
"qwen3.6-35B-A3B": GPU_MOE_URL,
"qwen3.6-27B-code": GPU_DENSE_URL,
"qwen3.5-9b-vlm": GPU_LIGHT_URL,
}
# Max concurrent requests per GPU (based on llama.cpp --parallel)
GPU_MAX_CONCURRENT = {
"qwen3.6-35B-A3B": 2, # 2 slots (cross-agent spread prevents overheating)
"qwen3.6-27B-code": 2, # 2 slots (128K context frees VRAM)
"qwen3.5-9b-vlm": 2, # 2 slots (12GB VRAM, 4GB headroom)
}
# Context window sizes (tokens) — used for compaction signals
GPU_CONTEXT = {
"qwen3.6-35B-A3B": 262144,
"qwen3.6-27B-code": 131072,
"qwen3.5-9b-vlm": 262144,
}
TIER_MODELS = {
"starter": ["qwen3.5-9b-vlm"],
"professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
"enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
}
API_KEYS = {
"sk-syslog-local-master-key": {"tier": "enterprise", "agent": "admin"},
"sk-syslog-abiba": {"tier": "enterprise", "agent": "Abiba"},
"sk-syslog-mumuni": {"tier": "enterprise", "agent": "Mumuni"},
"sk-syslog-tanko": {"tier": "enterprise", "agent": "Tanko"},
"sk-syslog-koby": {"tier": "enterprise", "agent": "Koby"},
"sk-syslog-kagenz0": {"tier": "enterprise", "agent": "Kagenz0"},
"sk-syslog-koonimo": {"tier": "enterprise", "agent": "Koonimo"},
"sk-starter-abc123": {"tier": "starter", "agent": "test-starter"},
"sk-professional-xyz789": {"tier": "professional", "agent": "test-pro"},
}
logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
log = logging.getLogger("router")
try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
except Exception: r = None
def counter_audit_loop():
"""Every 30s, check GPU slots and reset counters if all slots idle."""
while True:
time.sleep(30)
if not r: continue
for model, url in GPU_URLS.items():
try:
resp = requests.get(url.replace("/v1","") + "/slots",
headers={"Authorization": "Bearer not-needed"}, timeout=5)
if resp.status_code == 200:
slots = resp.json()
all_idle = all(not s.get("is_processing", False) for s in slots)
if all_idle:
current = int(r.get("active:" + model) or 0)
if current > 0:
r.set("active:" + model, 0)
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
except Exception:
pass
threading.Thread(target=counter_audit_loop, daemon=True).start()
app = Flask(__name__)
sse_subscribers = []; sse_lock = threading.Lock()
def gpu_active_count(model):
"""Get number of in-flight requests for a GPU."""
if r:
return int(r.get("active:" + model) or 0)
return 0
def gpu_incr(model):
if r: r.incr("active:" + model)
def gpu_decr(model):
if r:
v = r.decr("active:" + model)
if v and int(v) < 0:
r.set("active:" + model, 0) # never go negative
def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
url = GPU_SIDECARS.get(model)
if not url: return {"status": "unknown"}
try:
resp = requests.get(url, timeout=sidecar_timeout)
if resp.status_code == 200:
d = resp.json()
pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
status = "healthy" # VRAM usage != saturation; busy slots handled by is_gpu_busy()
vram_warning = pct >= 95
# Also check if llama.cpp endpoint is actually responding
gpu_url = GPU_URLS.get(model, "")
try:
hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=gpu_timeout)
if hr.status_code != 200:
status = "down"
except Exception:
status = "down"
return {"status": status, "vram_warning": vram_warning, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
except Exception: pass
return {"status": "down"}
def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
def estimate_tokens(msgs):
"""Estimate token count from messages. Uses JSON length / 3.5 (closer to real tokenizer ratios for dense text)."""
return len(json.dumps(msgs, default=str)) // 3.5
def store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, stream):
"""Store detailed performance record in Redis for analytics."""
if not r: return
try:
total_ms = queue_ms + inference_ms
tps = completion_tokens / (inference_ms / 1000) if inference_ms > 0 and completion_tokens > 0 else 0
rec = json.dumps({
"ts": time.time(),
"model": model, "agent": agent, "tier": tier, "reason": reason,
"queue_ms": round(queue_ms, 1),
"inference_ms": round(inference_ms, 1),
"total_ms": round(total_ms, 1),
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"tokens_per_sec": round(tps, 1),
"stream": stream
})
# Global recent list (last 500)
r.lpush("perf:recent", rec)
r.ltrim("perf:recent", 0, 499)
# Per-model list (last 200)
r.lpush("perf:model:" + model, rec)
r.ltrim("perf:model:" + model, 0, 199)
# Per-reason list (last 200)
r.lpush("perf:reason:" + reason, rec)
r.ltrim("perf:reason:" + reason, 0, 199)
# Per-agent list (last 200)
r.lpush("perf:agent:" + agent, rec)
r.ltrim("perf:agent:" + agent, 0, 199)
except Exception:
pass
def is_gpu_busy(model):
"""Check if GPU is at or near max concurrent capacity."""
active = gpu_active_count(model)
max_c = GPU_MAX_CONCURRENT.get(model, 1)
return active >= max_c
def select_best_gpu(candidates, reason, agent=""):
"""Pick best GPU, spreading agents across GPUs to prevent hotspots."""
# Count how many distinct agents are on each GPU
gpu_agent_counts = {}
if r:
for m in GPU_URLS:
count = 0
for ak in API_KEYS.values():
if r.get("agent_gpu:" + ak["agent"] + ":" + m):
count += 1
gpu_agent_counts[m] = count
# First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
for m in candidates:
if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
return {"model": m, "reason": reason}
# Second pass: prefer GPU this agent is NOT already on (skip own GPU)
if agent:
for m in candidates:
if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
return {"model": m, "reason": reason}
# Third pass: any non-busy GPU
for m in candidates:
if not is_gpu_busy(m):
return {"model": m, "reason": reason}
# All busy — pick least loaded
best = None
best_load = 999
for m in candidates:
load = gpu_active_count(m)
if load < best_load:
best_load = load
best = m
if best:
return {"model": best, "reason": "load_balanced_" + reason}
return None
def route(rd, tier, agent=""):
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
sys = any(m.get("role")=="system" for m in msgs)
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
hints = rd.get("routing_hints",{})
allowed = TIER_MODELS.get(tier, ["qwen3.5-9b-vlm"])
avail = [m for m in available_models() if m in allowed]
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
# Check if all available GPUs are at max capacity
if all(is_gpu_busy(m) for m in avail):
return {"model": avail[0], "reason": "all_saturated", "saturated": True}
req = rd.get("model","auto")
if req != "auto":
target = req if req in avail else avail[0]
# If explicit model is busy, check if another can take it
if is_gpu_busy(target) and req in allowed:
alts = [m for m in avail if m != target and m in allowed]
if alts:
alt = select_best_gpu(alts, "explicit", agent)
if alt: return alt
return {"model": target, "reason": "explicit"}
if hints:
if hints.get("priority")=="speed" and "qwen3.5-9b-vlm" in avail:
return select_best_gpu(["qwen3.5-9b-vlm"], "hint_speed", agent) or {"model":"qwen3.5-9b-vlm","reason":"hint_speed"}
if hints.get("priority")=="quality" and "qwen3.6-35B-A3B" in avail:
return select_best_gpu(["qwen3.6-35B-A3B"], "hint_quality", agent) or {"model":"qwen3.6-35B-A3B","reason":"hint_quality"}
first_msg = msgs[0].get("content","") if msgs else ""
words = len(first_msg.split()) if isinstance(first_msg, str) else 99
# TIER 1: Lightweight — single-turn short queries → VLM (fastest)
if not sys and turns <= 1 and t <= 500 and words <= 100 and "qwen3.5-9b-vlm" in avail:
if not is_gpu_busy("qwen3.5-9b-vlm"):
return {"model":"qwen3.5-9b-vlm","reason":"lightweight"}
# VLM busy — Dense is faster for short queries than MoE
fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(fallback, "lightweight_fallback", agent)
if result: return result
# TIER 2: Simple conversations — VLM primary (up to 15K tok), fastest for moderate chat
if t <= 15000 and turns <= 12 and "qwen3.5-9b-vlm" in avail:
if not is_gpu_busy("qwen3.5-9b-vlm"):
return {"model":"qwen3.5-9b-vlm","reason":"simple_conv"}
# VLM busy — fall back to Dense, then MoE
fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(fallback, "simple_conv_fallback", agent)
if result: return result
# TIER 3: Medium complexity — Dense primary, VLM fallback (quality + speed balance)
if t <= 25000:
candidates = [m for m in ["qwen3.6-27B-code","qwen3.5-9b-vlm","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(candidates, "medium", agent)
if result: return result
# TIER 4: Heavy reasoning — MoE primary (workhorse), Dense fallback
if t > 25000:
candidates = [m for m in ["qwen3.6-35B-A3B","qwen3.6-27B-code","qwen3.5-9b-vlm"] if m in avail]
result = select_best_gpu(candidates, "heavy_reasoning", agent)
if result: return result
# TIER 5: Default — Dense primary, MoE fallback
candidates = [m for m in ["qwen3.6-27B-code","qwen3.5-9b-vlm","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(candidates, "default", agent)
if result: return result
return {"model":avail[0],"reason":"last_resort"}
def clean_unicode(text):
if not isinstance(text, str): return text
text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
return text.encode("ascii", "ignore").decode("ascii")
def clean_response(d):
if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
if isinstance(d, list): return [clean_response(v) for v in d]
if isinstance(d, str): return clean_unicode(d)
return d
def get_metrics():
d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
for m in GPU_URLS:
h = check_gpu_health(m)
d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
d["active_requests"][m] = gpu_active_count(m)
if r:
try:
for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
for k,v in API_KEYS.items():
c = int(r.get("routes:agent:"+v["agent"]) or 0)
if c>0: d["agent_counts"][v["agent"]] = c
for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
raw = r.lrange("routes:recent",0,49)
d["recent"] = [json.loads(x) for x in raw] if raw else []
except Exception: pass
return d
def bcast():
data = get_metrics(); payload = json.dumps(data)
with sse_lock:
dead = []
for q in sse_subscribers:
try: q.put(payload)
except Exception: dead.append(q)
for q in dead: sse_subscribers.remove(q)
QUEUE_TIMEOUT = int(os.environ.get("QUEUE_TIMEOUT", "30")) # max seconds to queue before 503
@app.route("/v1/chat/completions", methods=["POST"])
def chat():
try:
rd = request.get_json(force=True)
ak = request.headers.get("Authorization","").replace("Bearer ","")
if not ak or ak not in API_KEYS:
log.warning("AUTH_REJECTED: no/invalid API key from %s", request.remote_addr)
return jsonify({"error": "Unauthorized — valid API key required"}), 401
ki = API_KEYS[ak]
tier, agent = ki["tier"], ki["agent"]
# Allow agent to override queue timeout via header
q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
# Cross-turn context tracking: accumulate tokens per session
session_id = request.headers.get("X-Session-Id", "")
session_tokens = 0
if session_id and r:
try:
prev = int(r.get("session:" + session_id) or 0)
current = estimate_tokens(rd.get("messages",[]))
session_tokens = max(prev, current) # context only grows
r.set("session:" + session_id, session_tokens, ex=86400) # TTL 24h
except Exception: pass
d = route(rd, tier, agent)
queue_start = time.time()
# Queue loop: wait for a GPU slot instead of immediate 503
while d.get("saturated"):
elapsed = time.time() - queue_start
if elapsed > q_timeout:
resp = jsonify({"error": "All GPUs saturated", "queued_s": round(elapsed,1), "retry_after_s": 5})
resp.headers["Retry-After"] = "5"
log.warning("QUEUE_TIMEOUT: %s waited %.1fs, all GPUs saturated", agent, elapsed)
return resp, 503
time.sleep(0.5) # poll every 500ms
d = route(rd, tier, agent)
queue_ms = (time.time() - queue_start) * 1000
if queue_ms > 500:
log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
is_stream = rd.get("stream", False)
gpu_incr(model)
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
# Track which GPU this agent is using (TTL 120s covers typical request)
if r and agent:
try: r.setex("agent_gpu:" + agent + ":" + model, 120, "1")
except: pass
if r:
try:
r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent,"queue_ms": round(queue_ms,1)}))
r.ltrim("routes:recent",0,999)
except Exception: pass
start = time.time()
resp = requests.post(url+"/chat/completions", json=rd,
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
lat = int((time.time()-start)*1000)
gpu_decr(model)
if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
if is_stream:
# Buffer SSE chunks, handle split lines for large responses
chunks = []
stream_timings = {}
buf = "" # accumulate partial lines
for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
if raw:
cleaned = clean_unicode(raw)
chunks.append(cleaned)
buf += cleaned
# Process complete lines from buffer
while "\n" in buf:
line, buf = buf.split("\n", 1)
line = line.strip()
if line.startswith("data: ") and not stream_timings:
js = line[6:].strip()
if js.startswith("{") and "timings" in js and "predicted_n" in js:
try:
tj = json.loads(js).get("timings", {})
if tj:
stream_timings = tj
except: pass
# Store perf record with real token counts from stream
if stream_timings:
pt = stream_timings.get("prompt_n", 0)
ct = stream_timings.get("predicted_n", 0)
tps = stream_timings.get("predicted_per_second", 0)
gen_ms = stream_timings.get("predicted_ms", lat)
store_perf_record(model, agent, tier, reason, queue_ms, gen_ms, pt, ct, True)
else:
store_perf_record(model, agent, tier, reason, queue_ms, lat, estimate_tokens(rd.get("messages",[])), 0, True)
# Yield all chunks to client
def gen():
for c in chunks: yield c
bcast()
ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
sse_resp = Response(stream_with_context(gen()), mimetype="text/event-stream")
sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
sse_resp.headers["X-Context-Warning"] = ctx_warning
sse_resp.headers["X-Context-Model"] = model
return sse_resp
data = clean_response(resp.json())
for c in data.get("choices",[]):
msg = c.get("message",{})
if not msg.get("content") and msg.get("reasoning_content"):
msg["content"] = msg["reasoning_content"]
# Extract performance data from llama.cpp response
usage = data.get("usage", {})
timings = data.get("timings", {})
prompt_tokens = usage.get("prompt_tokens", 0)
completion_tokens = usage.get("completion_tokens", 0)
inference_ms = lat # total GPU round-trip
store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, False)
ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"queue_ms": round(queue_ms,1),"active_gpu":gpu_active_count(model),"context_remaining": max(0, ctx_remaining),"context_pct": round(ctx_pct,1),"context_warning": ctx_warning}
resp = jsonify(data)
resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
resp.headers["X-Context-Warning"] = ctx_warning
resp.headers["X-Context-Model"] = model
bcast()
return resp
except requests.Timeout:
gpu_decr(model)
log.error("TIMEOUT: %s -> %s", agent, model)
return jsonify({"error":"timeout"}), 504
except Exception as e:
gpu_decr(model)
log.error("Error: %s\n%s", e, traceback.format_exc())
return jsonify({"error":str(e)}), 500
@app.route("/metrics/performance")
def performance():
"""Per-request performance analytics with percentiles per model/reason/agent."""
if not r: return jsonify({"error": "Redis unavailable"}), 503
try:
window_hours = int(request.args.get("window", "24"))
model_filter = request.args.get("model", "all")
# Load recent records
cutoff = time.time() - (window_hours * 3600)
raw = r.lrange("perf:recent", 0, -1)
records = []
for x in raw:
try:
rec = json.loads(x)
if rec["ts"] >= cutoff:
records.append(rec)
except: pass
# Filter by model if specified
if model_filter != "all":
records = [r for r in records if r["model"] == model_filter]
if not records:
return jsonify({"models": [], "reasons": [], "agents": [], "summary": {"total_requests": 0}})
def pct(values, p):
if len(values) < 2: return round(values[0], 1) if values else 0
return round(statistics.quantiles(sorted(values), n=100, method='inclusive')[min(p-1, 98)], 1)
# Per-model stats
model_groups = {}
for rec in records:
m = rec["model"]
if m not in model_groups: model_groups[m] = []
model_groups[m].append(rec)
models = []
for m, recs in sorted(model_groups.items()):
latencies = [r["total_ms"] for r in recs]
tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
non_stream = [r for r in recs if not r["stream"]]
queue_times = [r["queue_ms"] for r in non_stream]
models.append({
"model": m,
"count": len(recs),
"stream_pct": round(len([r for r in recs if r["stream"]]) / len(recs) * 100, 1),
"latency": {
"avg": round(statistics.mean(latencies), 1),
"p50": pct(latencies, 50),
"p95": pct(latencies, 95),
"p99": pct(latencies, 99)
},
"throughput": {
"avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0,
"p50": pct(tps_vals, 50) if tps_vals else 0,
"p95": pct(tps_vals, 95) if tps_vals else 0,
},
"queue": {
"avg_ms": round(statistics.mean(queue_times), 1) if queue_times else 0,
"p95_ms": pct(queue_times, 95) if queue_times else 0,
} if queue_times else None
})
# Per-reason stats
reason_groups = {}
for rec in records:
rsn = rec["reason"]
if rsn not in reason_groups: reason_groups[rsn] = []
reason_groups[rsn].append(rec)
reasons = []
for rsn, recs in sorted(reason_groups.items(), key=lambda x: -len(x[1])):
latencies = [r["total_ms"] for r in recs]
reasons.append({
"reason": rsn,
"count": len(recs),
"avg_total_ms": round(statistics.mean(latencies), 1),
"p95_total_ms": pct(latencies, 95)
})
# Per-agent stats
agent_groups = {}
for rec in records:
ag = rec["agent"]
if ag not in agent_groups: agent_groups[ag] = []
agent_groups[ag].append(rec)
agents = []
for ag, recs in sorted(agent_groups.items(), key=lambda x: -len(x[1])):
latencies = [r["total_ms"] for r in recs]
tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
agents.append({
"agent": ag,
"count": len(recs),
"avg_total_ms": round(statistics.mean(latencies), 1),
"avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0
})
all_lat = [r["total_ms"] for r in records]
all_tps = [r["tokens_per_sec"] for r in records if r["tokens_per_sec"] > 0]
summary = {
"total_requests": len(records),
"window_hours": window_hours,
"latency": {
"avg_ms": round(statistics.mean(all_lat), 1),
"p50_ms": pct(all_lat, 50),
"p95_ms": pct(all_lat, 95),
"p99_ms": pct(all_lat, 99)
},
"throughput_avg_tps": round(statistics.mean(all_tps), 1) if all_tps else 0
}
return jsonify({"models": models, "reasons": reasons, "agents": agents, "summary": summary})
except Exception as e:
return jsonify({"error": str(e)}), 500
@app.route("/metrics/scatter")
def scatter():
"""Return individual data points for scatter plots (prompt_tokens vs latency)."""
if not r: return jsonify({"error": "Redis unavailable"}), 503
try:
window_hours = int(request.args.get("window", "24"))
model_filter = request.args.get("model", "all")
cutoff = time.time() - (window_hours * 3600)
raw = r.lrange("perf:recent", 0, -1)
points = []
for x in raw:
try:
rec = json.loads(x)
if rec["ts"] >= cutoff:
if model_filter == "all" or rec["model"] == model_filter:
points.append({
"model": rec["model"],
"agent": rec["agent"],
"reason": rec["reason"],
"prompt_tokens": int(rec.get("prompt_tokens", 0)),
"completion_tokens": rec.get("completion_tokens", 0),
"inference_ms": round(rec["inference_ms"], 1),
"tokens_per_sec": rec.get("tokens_per_sec", 0),
"stream": rec.get("stream", False)
})
except: pass
return jsonify({"points": points, "count": len(points)})
except Exception as e:
return jsonify({"error": str(e)}), 500
@app.route("/v1/models")
def models():
def _h(m): return check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":_h(m).get("status"),"gpu":_h(m).get("gpu_name")} for m in GPU_URLS]})
@app.route("/health")
def health():
gpus = {}
for m in GPU_URLS:
h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
h["active_requests"] = gpu_active_count(m)
h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
gpus[m] = h
return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
@app.route("/metrics")
def metrics(): return jsonify(get_metrics())
@app.route("/metrics/timeseries")
def metrics_timeseries():
period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
data = {"models": {}, "labels": []}
if period == "day":
buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
elif period == "week":
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
else:
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
if r:
for model in models_list:
counts = []
for bucket in buckets:
total = 0
if period in ("week","month"):
for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
else: total = int(r.get("ts:"+model+":"+bucket) or 0)
counts.append(total)
data["models"][model] = counts
return jsonify(data)
@app.route("/stream")
def stream():
def ev():
q = queue.Queue()
with sse_lock: sse_subscribers.append(q)
try:
yield "data: "+json.dumps(get_metrics())+"\n\n"
while True:
try: yield "data: "+q.get(timeout=3)+"\n\n"
except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
except GeneratorExit: pass
finally:
with sse_lock:
if q in sse_subscribers: sse_subscribers.remove(q)
return Response(stream_with_context(ev()), mimetype="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
if __name__ == "__main__":
log.info("Router on :9000 (load-aware)")
app.run(host="0.0.0.0", port=9000, debug=False)
backups/20260602_103344/ts_patch.py.bak
+80
View File
@@ -0,0 +1,80 @@
# Add time-series tracking and endpoint to router
with open('/opt/inference-harness/router/router.py') as f:
code = f.read()
# Add time-series tracking in the chat handler (after Redis incr)
old_track = '''r.incr('routes:'+model); r.incr('routes:tier:'+tier); r.incr('routes:agent:'+agent)
r.lpush('routes:recent', json.dumps'''
new_track = '''r.incr('routes:'+model); r.incr('routes:tier:'+tier); r.incr('routes:agent:'+agent)
# Time-series: hourly bucket
hour_key = 'ts:'+model+':'+time.strftime('%Y%m%d%H')
r.incr(hour_key)
r.expire(hour_key, 86400*31) # keep 31 days
r.lpush('routes:recent', json.dumps'''
code = code.replace(old_track, new_track)
# Add /metrics/timeseries endpoint before if __name__
ts_endpoint = '''
@app.route('/metrics/timeseries')
def metrics_timeseries():
period = request.args.get('period', 'day')
models = list(GPU_URLS.keys())
data = {'models': {}, 'labels': []}
if period == 'day':
# Last 24 hours, hourly buckets
buckets = []
for h in range(23, -1, -1):
t = time.time() - h * 3600
buckets.append(time.strftime('%Y%m%d%H', time.gmtime(t)))
data['labels'] = [time.strftime('%H:00', time.gmtime(time.time() - h*3600)) for h in range(23, -1, -1)]
elif period == 'week':
# Last 7 days, daily buckets
buckets = []
for d in range(6, -1, -1):
t = time.time() - d * 86400
buckets.append(time.strftime('%Y%m%d', time.gmtime(t)))
data['labels'] = [time.strftime('%a', time.gmtime(time.time() - d*86400)) for d in range(6, -1, -1)]
else:
# Month — last 30 days, 3-day buckets
buckets = []
for d in range(29, -1, -3):
t = time.time() - d * 86400
buckets.append(time.strftime('%Y%m%d', time.gmtime(t)))
data['labels'] = [time.strftime('%m/%d', time.gmtime(time.time() - d*86400)) for d in range(29, -1, -3)]
if r:
for model in models:
counts = []
for bucket in buckets:
if period == 'month':
# Sum 3 consecutive days per bucket
total = 0
base = time.strptime(bucket, '%Y%m%d')
for offset in range(3):
d = time.strftime('%Y%m%d', time.gmtime(time.mktime(base) + offset*86400))
total += int(r.get('ts:'+model+':'+d) or 0)
# Also check hourly keys for today
for hh in range(24):
total += int(r.get('ts:'+model+':'+d+'{:02d}'.format(hh)) or 0)
counts.append(total)
else:
key = 'ts:'+model+':'+bucket
if period == 'week':
# Sum all hours in the day
total = sum(int(r.get(key+'{:02d}'.format(h)) or 0) for h in range(24))
else:
total = int(r.get(key) or 0)
counts.append(total)
data['models'][model] = counts
return jsonify(data)
'''
# Insert before if __name__
code = code.replace(if __name__ == __main__:, ts_endpoint + nif __name__ == __main__:)
with open('/opt/inference-harness/router/router.py', 'w') as f:
f.write(code)
print('Time-series tracking and endpoint added')
dashboard/dashboard.py
+16 -24
View File
@@ -136,19 +136,9 @@ body { background: #0b0f17; color: #bcc3cd; font-family: -apple-system, BlinkMac
</div>
<script>
// ═══ MODEL CONFIG — Update ONLY this array for model migrations ═══
var MODELS=[
{id:'qwen3.6-35B-A3B',label:'Qwen MoE',gpu:'MoE - Strix Halo',color:'#a78bfa',short:'MoE',domId:'gpu-moe',perf:'35B MoE'},
{id:'qwen3.6-27B-code',label:'Qwen Code',gpu:'Dense - RTX 3090',color:'#f59e0b',short:'Dense',domId:'gpu-dense',perf:'27B Dense'},
{id:'gemma-4-12b',label:'Gemma 4 12B',gpu:'VLM - RTX 5070',color:'#22c55e',short:'VLM',domId:'gpu-light',perf:'12B VLM'},
{id:'qwen3.5-9b-vlm',label:'Qwen VLM (retired)',gpu:'RTX 5070 (legacy)',color:'#64748b',short:'OLD',domId:'gpu-light',perf:'9B VLM'}
];
// Auto-derived lookups — DO NOT EDIT below
var MC={},ML={},GL={},ids={},lb={},mlab={},mcol={};
MODELS.forEach(function(m){MC[m.id]=m.color;ML[m.id]=m.label;GL[m.id]=m.gpu;ids[m.id]=m.domId;lb[m.id]=m.short;mlab[m.id]=m.perf;mcol[m.id]=m.color;});
// Safe lookup helpers — fall back to raw model ID if not in MODELS
function modelLabel(id){return mlab[id]||id||'Unknown';}
function modelColor(id){return mcol[id]||'#64748b';}
var MC={'gemma-4-12b':'#22c55e','qwen3.6-27B-code':'#f59e0b','qwen3.6-35B-A3B':'#a78bfa'};
var ML={'gemma-4-12b':'Gemma 4 12B','qwen3.6-27B-code':'Qwen Code','qwen3.6-35B-A3B':'Qwen MoE'};
var GL={'qwen3.6-35B-A3B':'MoE - Strix Halo','qwen3.6-27B-code':'Dense - RTX 3090','gemma-4-12b':'VLM - RTX 5070'};
function $(id){return document.getElementById(id);}
function render(data){
@@ -157,6 +147,7 @@ var t=Object.values(data.route_counts||{}).reduce((a,b)=>a+b,0);
var ta=0,tm=0;data.gpus.forEach(function(g){ta+=(g.active_requests||0);tm+=(g.max_concurrent||1)});
$('kpi-total').textContent=t;$('kpi-active').textContent=ta+'/'+tm;$('kpi-agents').textContent=Object.keys(data.agent_counts||{}).length;
$('update-time').textContent=new Date().toLocaleTimeString();
var ids={'qwen3.6-35B-A3B':'gpu-moe','qwen3.6-27B-code':'gpu-dense','gemma-4-12b':'gpu-light'};
data.gpus.forEach(function(g){
var el=$(ids[g.id]);if(!el)return;
var a=g.active_requests||0,mx=g.max_concurrent||1;
@@ -228,12 +219,13 @@ function loadPerf(){fetch('/api/performance?window='+perfWindow).then(function(r
function renderPerf(d){
var models=d.models||[],reasons=d.reasons||[],agents=d.agents||[],sum=d.summary||{};
// Latency bars: p50/p95/p99 per model
// mlab/mcol auto-derived from MODELS above
var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','gemma-4-12b':'12B VLM'};
var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','gemma-4-12b':'#22c55e'};
if(!models.length){$('perf-latency').innerHTML='<div class="text-secondary small text-center py-4">Accumulating data...</div>';return;}
var maxLat=Math.max(...models.map(function(m){return m.latency.p99||0}),1);
var latHTML=models.map(function(m){
var l=m.latency||{},p50=l.p50||0,p95=l.p95||0,p99=l.p99||0,c=modelColor(m.model);
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+modelLabel(m.model)+'</span><span class="text-secondary">'+m.count+' reqs</span></div>'+
var l=m.latency||{},p50=l.p50||0,p95=l.p95||0,p99=l.p99||0,c=mcol[m.model]||'#38bdf8';
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span class="text-secondary">'+m.count+' reqs</span></div>'+
'<div class="d-flex align-items-center gap-2 mb-1"><span class="text-secondary" style="min-width:28px">p50</span><div class="flex-grow-1" style="height:14px;background:#1e293b;border-radius:4px;overflow:hidden;position:relative"><div style="position:absolute;left:0;top:0;height:100%;width:'+(p50/maxLat*100)+'%;background:'+c+';opacity:0.3;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p95/maxLat*100)+'%;background:'+c+';opacity:0.5;border-radius:4px"></div><div style="position:absolute;left:0;top:0;height:100%;width:'+(p99/maxLat*100)+'%;background:'+c+';border-radius:4px"></div></div><span style="color:'+c+';min-width:48px;text-align:right;font-variant-numeric:tabular-nums">'+p99+'ms</span></div>'+
'<div class="d-flex gap-3" style="font-size:10px;color:#64748b;padding-left:32px"><span>p50: '+p50+'ms</span><span>p95: '+p95+'ms</span><span>p99: '+p99+'ms</span></div></div>';
}).join('');
@@ -241,12 +233,12 @@ $('perf-latency').innerHTML=latHTML;
// Throughput comparison
var maxTps=Math.max(...models.map(function(m){return m.throughput.avg_tokens_per_sec||0}),1);
var tpsHTML=models.map(function(m){
var t=m.throughput||{},avg=t.avg_tokens_per_sec||0,p50=t.p50||0,c=modelColor(m.model);
var t=m.throughput||{},avg=t.avg_tokens_per_sec||0,p50=t.p50||0,c=mcol[m.model]||'#38bdf8';
var isAllStreaming = avg===0 && p50===0;
if(isAllStreaming){
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+modelLabel(m.model)+'</span><span style="color:#64748b;font-style:italic">streaming only</span></div><div class="text-secondary" style="font-size:10px">t/s available for non-streaming requests only</div></div>';
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:#64748b;font-style:italic">streaming only</span></div><div class="text-secondary" style="font-size:10px">t/s available for non-streaming requests only</div></div>';
}
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+modelLabel(m.model)+'</span><span style="color:'+c+'" class="fw-bold">'+avg+' tok/s</span></div>'+
return'<div class="mb-2" style="font-size:11px"><div class="d-flex justify-content-between mb-1"><span style="color:#e2e8f0">'+mlab[m.model]+'</span><span style="color:'+c+'" class="fw-bold">'+avg+' tok/s</span></div>'+
'<div class="d-flex align-items-center gap-2"><span class="text-secondary" style="min-width:28px">avg</span><div class="flex-grow-1" style="height:6px;background:#1e293b;border-radius:3px;overflow:hidden"><div style="height:100%;width:'+(Math.max(avg/maxTps*100,6))+'%;background:'+c+';border-radius:3px"></div></div><span class="small" style="color:'+c+';min-width:54px;text-align:right">'+avg+' tok/s</span></div>'+
'<div class="d-flex align-items-center gap-2 mt-1"><span class="text-secondary" style="min-width:28px;font-size:10px">p50</span><div class="flex-grow-1" style="height:4px;background:#1e293b;border-radius:2px;overflow:hidden"><div style="height:100%;width:'+(Math.max(p50/maxTps*100,4))+'%;background:'+c+';opacity:0.5;border-radius:2px"></div></div><span style="font-size:10px;color:#64748b">'+p50+' tok/s</span></div></div>';
}).join('');
@@ -278,8 +270,8 @@ fetch('/api/scatter?window=24&model='+m).then(function(r){return r.json()}).then
function renderScatter(d){
var pts=d.points||[],el=$('scatter-plot'),lg=$('scatter-legend');
if(!pts.length){el.innerHTML='<div class="text-secondary small text-center py-5">No data yet</div>';return;}
var mcol=Object.assign({unknown:'#38bdf8'},mcol);
// mlab auto-derived from MODELS above
var mcol={'qwen3.6-35B-A3B':'#a78bfa','qwen3.6-27B-code':'#f59e0b','gemma-4-12b':'#22c55e','unknown':'#38bdf8'};
var mlab={'qwen3.6-35B-A3B':'35B MoE','qwen3.6-27B-code':'27B Dense','gemma-4-12b':'12B VLM'};
var maxX=Math.max.apply(null,pts.map(function(p){return p.prompt_tokens||0}))||1000;
var maxY=Math.max.apply(null,pts.map(function(p){return p.inference_ms||0}))||5000;
// Log scale for X axis (prompt tokens vary widely)
@@ -287,9 +279,9 @@ var toX=function(t){return Math.log10(Math.max(t,1))/Math.log10(Math.max(maxX,10
var toY=function(t){return (t/maxY)*100;};
var dots='';
pts.forEach(function(p){
var x=toX(p.prompt_tokens),y=toY(p.inference_ms),c=modelColor(p.model);
var x=toX(p.prompt_tokens),y=toY(p.inference_ms),c=mcol[p.model]||'#38bdf8';
var r=p.stream?1.5:2.5,o=p.stream?0.4:0.8;
dots+='<circle cx="'+x+'" cy="'+(100-y)+'" r="'+r+'" fill="'+c+'" opacity="'+o+'"><title>'+modelLabel(p.model)+' | '+p.prompt_tokens+' tok | '+p.inference_ms+'ms | '+p.agent+'</title></circle>';
dots+='<circle cx="'+x+'" cy="'+(100-y)+'" r="'+r+'" fill="'+c+'" opacity="'+o+'"><title>'+mlab[p.model]+' | '+p.prompt_tokens+' tok | '+p.inference_ms+'ms | '+p.agent+'</title></circle>';
});
// Grid lines
var grid='';
@@ -305,7 +297,7 @@ yVals.forEach(function(v){if(v<=maxY)yTicks+='<text x="-2" y="'+(97-toY(v))+'" t
el.innerHTML='<svg viewBox="-35 0 140 115" style="width:100%;height:200px">'+grid+dots+xTicks+yTicks+'<text x="50" y="112" text-anchor="middle" font-size="9" fill="#475569">Prompt Tokens (log scale)</text><text x="-38" y="50" text-anchor="middle" font-size="9" fill="#475569" transform="rotate(-90,-38,50)">Inference Time</text></svg>';
// Legend
var models=[];pts.forEach(function(p){if(models.indexOf(p.model)===-1)models.push(p.model);});
lg.innerHTML=models.map(function(m){return'<span class="d-flex align-items-center gap-1 small"><svg width="10" height="10"><circle cx="5" cy="5" r="3.5" fill="'+modelColor(m)+'"/></svg>'+modelLabel(m)+'</span>';}).join('');
lg.innerHTML=models.map(function(m){return'<span class="d-flex align-items-center gap-1 small"><svg width="10" height="10"><circle cx="5" cy="5" r="3.5" fill="'+(mcol[m]||'#38bdf8')+'"/></svg>'+mlab[m]+'</span>';}).join('');
}
poll();setInterval(poll,10000);loadTS();loadPerf();setInterval(loadPerf,15000);loadScatter();setInterval(loadScatter,30000);
</script>
docker-compose.yml
+12 -9
View File
@@ -21,16 +21,17 @@ services:
container_name: harness-router
restart: unless-stopped
ports:
- "9000:9000"
- "127.0.0.1:9000:9000"
environment:
- REDIS_URL=redis://redis:6379
- GPU_MOE_URL=http://192.168.68.15:8080/v1
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro"}}
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
interval: 15s
timeout: 5s
interval: 30s
timeout: 15s
retries: 3
depends_on:
redis:
@@ -42,13 +43,15 @@ services:
container_name: harness-litellm
restart: unless-stopped
ports:
- "8081:4000"
- "127.0.0.1:8081:4000"
volumes:
- ./litellm_config.yaml:/app/config.yaml
environment:
- LITELLM_MASTER_KEY=sk-syslog-local-master-key
- LITELLM_MASTER_KEY=sk-sys...-key
extra_hosts:
- "host.docker.internal:host-gateway"
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
interval: 15s
timeout: 5s
retries: 3
@@ -66,8 +69,8 @@ services:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
healthcheck:
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
interval: 15s
timeout: 5s
interval: 30s
timeout: 15s
retries: 3
depends_on:
- litellm
@@ -78,7 +81,7 @@ services:
container_name: harness-dashboard
restart: unless-stopped
ports:
- "3000:3000"
- "127.0.0.1:3000:3000"
environment:
- REDIS_URL=redis://redis:6379
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
docker-compose.yml.bak
+97
View File
@@ -0,0 +1,97 @@
version: '3.8'
services:
redis:
image: redis:7-alpine
container_name: harness-redis
restart: unless-stopped
ports:
- "127.0.0.1:6379:6379"
volumes:
- redis-data:/data
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 3s
retries: 5
router:
build: ./router
container_name: harness-router
restart: unless-stopped
ports:
- "9000:9000"
environment:
- REDIS_URL=redis://redis:6379
- GPU_MOE_URL=http://192.168.68.15:8080/v1
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
interval: 15s
timeout: 5s
retries: 3
depends_on:
redis:
condition: service_healthy
litellm:
image: ghcr.io/berriai/litellm:main-stable
command: ["--config", "/app/config.yaml", "--port", "4000"]
container_name: harness-litellm
restart: unless-stopped
ports:
- "8081:4000"
volumes:
- ./litellm_config.yaml:/app/config.yaml
environment:
- LITELLM_MASTER_KEY=sk-syslog-local-master-key
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
interval: 15s
timeout: 5s
retries: 3
depends_on:
redis:
condition: service_healthy
nginx:
image: nginx:alpine
container_name: harness-nginx
restart: unless-stopped
ports:
- "80:80"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
healthcheck:
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
interval: 15s
timeout: 5s
retries: 3
depends_on:
- litellm
- dashboard
dashboard:
build: ./dashboard
container_name: harness-dashboard
restart: unless-stopped
ports:
- "3000:3000"
environment:
- REDIS_URL=redis://redis:6379
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
interval: 15s
timeout: 5s
retries: 3
depends_on:
- redis
volumes:
redis-data:
# LiteLLM command override to load config
# (appended to fix config loading issue)
maintenance.sh
Executable
+37
View File
@@ -0,0 +1,37 @@
#!/bin/bash
# SyslogAI Harness — Automated Maintenance
# Runs daily via cron
LOG="/var/log/harness-maintenance.log"
echo "=== $(date) ===" >> "$LOG"
# 1. Clean Redis timeseries keys older than 60 days
CUTOFF=$(date -d "60 days ago" +%Y%m%d%H)
echo "Redis: removing ts:* keys older than $CUTOFF" >> "$LOG"
DELETED=0
for key in $(docker exec harness-redis redis-cli KEYS "ts:*" 2>/dev/null); do
TS=$(echo "$key" | grep -oP '\d{10}$')
if [ -n "$TS" ] && [ "$TS" -lt "$CUTOFF" ] 2>/dev/null; then
docker exec harness-redis redis-cli DEL "$key" > /dev/null 2>&1
DELETED=$((DELETED + 1))
fi
done
echo "Redis: deleted $DELETED stale timeseries keys" >> "$LOG"
# 2. Log stale model keys (leftover from migrations)
STALE=$(docker exec harness-redis redis-cli KEYS "*gemma*" 2>/dev/null)
if [ -n "$STALE" ]; then
echo "WARNING: stale gemma keys found: $STALE" >> "$LOG"
fi
# 3. Prune Docker build cache (older than 7 days)
echo "Docker: pruning build cache" >> "$LOG"
docker builder prune -f --filter until=168h >> "$LOG" 2>&1
# 4. Log container health status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.RunningFor}}" >> "$LOG" 2>&1
# 5. Log Redis memory
docker exec harness-redis redis-cli INFO memory | grep used_memory_human >> "$LOG" 2>&1
echo "" >> "$LOG"
nginx/nginx.conf
+19 -3
View File
@@ -8,7 +8,9 @@ http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main launching rt=;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" rt=$request_time';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
sendfile on;
@@ -21,6 +23,11 @@ http {
server {
listen 80;
# Security headers
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options SAMEORIGIN always;
add_header X-XSS-Protection "1; mode=block" always;
# Disable buffering for SSE streams
proxy_buffering off;
@@ -71,9 +78,18 @@ http {
proxy_buffering off;
}
# Performance analytics
location /metrics/ {
proxy_pass http://router_api;
proxy_http_version 1.1;
proxy_set_header Host $host;
}
location /health {
return 200 "{\"status\":\"healthy\"}";
add_header Content-Type application/json;
proxy_pass http://router_api/health;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
}
router/http_patch.py
+90
View File
@@ -0,0 +1,90 @@
# Insert streaming support before the gpu_resp call
import re
with open('/opt/inference-harness/router/router.py') as f:
code = f.read()
# Find the gpu_resp block and replace with streaming-aware version
old = ''' start = time.time()
gpu_resp = requests.post(
gpu_url + "/chat/completions",
json=req_data,
headers={"Content-Type": "application/json", "Authorization": "Bearer not-needed"},
timeout=120,
)
latency_ms = int((time.time() - start) * 1000)
if gpu_resp.status_code != 200:
log.error("GPU error: %s %s", gpu_resp.status_code, gpu_resp.text[:200])
return jsonify({"error": "GPU backend returned " + str(gpu_resp.status_code)}), 502
response_data = gpu_resp.json()
response_data = fix_reasoning_content(response_data)
response_data["routing"] = {
"model": model, "reason": reason, "gpu": gpu_url,
"tier": tier, "agent": agent, "latency_ms": latency_ms,
}
return jsonify(response_data)'''
new = ''' start = time.time()
is_stream = req_data.get("stream", False)
gpu_resp = requests.post(
gpu_url + "/chat/completions",
json=req_data,
headers={"Content-Type": "application/json", "Authorization": "Bearer not-needed"},
timeout=120,
stream=is_stream,
)
latency_ms = int((time.time() - start) * 1000)
if gpu_resp.status_code != 200:
log.error("GPU error: %s %s", gpu_resp.status_code, gpu_resp.text[:200])
return jsonify({"error": "GPU backend returned " + str(gpu_resp.status_code)}), 502
if is_stream:
# Stream response back to client
def generate():
first = True
for line in gpu_resp.iter_lines(decode_unicode=True):
if line:
if first and line.startswith("data: "):
# Inject routing into first chunk
try:
chunk = json.loads(line[6:])
chunk["routing"] = {
"model": model, "reason": reason, "gpu": gpu_url,
"tier": tier, "agent": agent, "latency_ms": latency_ms,
}
yield "data: " + json.dumps(chunk) + "\n\n"
first = False
continue
except Exception:
pass
yield line + "\n"
yield "data: [DONE]\n\n"
return Response(stream_with_context(generate()), mimetype="text/event-stream")
response_data = gpu_resp.json()
response_data = fix_reasoning_content(response_data)
response_data["routing"] = {
"model": model, "reason": reason, "gpu": gpu_url,
"tier": tier, "agent": agent, "latency_ms": latency_ms,
}
return jsonify(response_data)'''
code = code.replace(old, new)
# Add missing import
if 'from flask import Flask, request, jsonify' in code:
code = code.replace(
'from flask import Flask, request, jsonify',
'from flask import Flask, request, jsonify, Response, stream_with_context'
)
with open('/opt/inference-harness/router/router.py', 'w') as f:
f.write(code)
print('Streaming support added')
router/router.py
+88 -174
View File
@@ -20,8 +20,8 @@ GPU_URLS = {
# Max concurrent requests per GPU (based on llama.cpp --parallel)
GPU_MAX_CONCURRENT = {
"qwen3.6-35B-A3B": 2, # 2 slots (cross-agent spread prevents overheating)
"qwen3.6-27B-code": 2, # 2 slots
"gemma-4-12b": 2, # 2 slots (12GB VRAM, 4GB headroom)
"qwen3.6-27B-code": 2, # 2 slots (128K context frees VRAM)
"gemma-4-12b": 2, # 2 slots (7.1GB VRAM)
}
# Context window sizes (tokens) — used for compaction signals
@@ -36,18 +36,11 @@ TIER_MODELS = {
"professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
"enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
}
# ── PHASE 0: Dual-Key API Key System ──
# API_KEYS env var is REQUIRED (JSON string). No hardcoded fallback.
# Format: {"sk-xxx": {"tier": "enterprise", "agent": "Name"}}
# Deprecated keys get {"deprecated": true} — still accepted, logged with warning.
_raw_keys = os.environ.get("API_KEYS")
if not _raw_keys:
raise RuntimeError("FATAL: API_KEYS environment variable is required. "
"Set it in docker-compose.yml or .env file. "
"No hardcoded keys fallback — this is a security feature.")
API_KEYS = json.loads(_raw_keys)
log.info("Loaded %d API keys from env var (%d deprecated)",
len(API_KEYS), sum(1 for v in API_KEYS.values() if v.get("deprecated")))
# API keys loaded from API_KEYS env var (set in docker-compose.yml)
# Fallback is dev-only — production MUST set API_KEYS env var
API_KEYS = json.loads(os.environ.get("API_KEYS", json.dumps({
"sk-dev-local-only": {"tier": "enterprise", "agent": "dev"},
})))
# Rate limits: requests per minute per API key tier
RATE_LIMIT_RPM = {
"enterprise": 120,
@@ -57,7 +50,7 @@ RATE_LIMIT_RPM = {
def check_rate_limit(api_key, tier):
"""Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remaining, reset_seconds)."""
if not r:
if not get_redis():
return True, 999, 60
limit = RATE_LIMIT_RPM.get(tier, 30)
key = f"ratelimit:{api_key}"
@@ -77,7 +70,25 @@ def check_rate_limit(api_key, tier):
logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
log = logging.getLogger("router")
try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
# Redis connection — initialized lazily, retries on first use
def get_redis():
global r
if r is not None:
try:
r.ping()
return r
except Exception:
r = None
try:
r = redis.from_url(REDIS_URL, decode_responses=True)
r.ping()
return r
except Exception:
return None
r = None
try: get_redis()
except Exception: pass
except Exception: r = None
@@ -85,7 +96,7 @@ def counter_audit_loop():
"""Every 30s, check GPU slots and reset counters if all slots idle."""
while True:
time.sleep(30)
if not r: continue
if not get_redis(): continue
for model, url in GPU_URLS.items():
try:
resp = requests.get(url.replace("/v1","") + "/slots",
@@ -96,7 +107,7 @@ def counter_audit_loop():
if all_idle:
current = int(r.get("active:" + model) or 0)
if current > 0:
r.set("active:" + model, 0)
rd.set("active:" + model, 0)
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
except Exception:
pass
@@ -113,14 +124,14 @@ def gpu_active_count(model):
return 0
def gpu_incr(model):
if r: r.incr("active:" + model)
if get_redis(): get_redis().incr("active:" + model)
def gpu_decr(model):
if r:
v = r.decr("active:" + model)
rd = get_redis()
if rd:
v = rd.decr("active:" + model)
if v and int(v) < 0:
r.set("active:" + model, 0) # never go negative
rd.set("active:" + model, 0) # never go negative
def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
url = GPU_SIDECARS.get(model)
if not url: return {"status": "unknown"}
@@ -151,7 +162,7 @@ def estimate_tokens(msgs):
def store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, stream):
"""Store detailed performance record in Redis for analytics."""
if not r: return
if not get_redis(): return
try:
total_ms = queue_ms + inference_ms
tps = completion_tokens / (inference_ms / 1000) if inference_ms > 0 and completion_tokens > 0 else 0
@@ -223,6 +234,8 @@ def select_best_gpu(candidates, reason, agent=""):
return {"model": best, "reason": "load_balanced_" + reason}
return None
def route(rd, tier, agent=""):
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
sys = any(m.get("role")=="system" for m in msgs)
@@ -231,14 +244,26 @@ def route(rd, tier, agent=""):
allowed = TIER_MODELS.get(tier, ["gemma-4-12b"])
avail = [m for m in available_models() if m in allowed]
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
# Check if all available GPUs are at max capacity
if all(is_gpu_busy(m) for m in avail):
return {"model": avail[0], "reason": "all_saturated", "saturated": True}
# GUARD: multimodal -> VLM only (sole vision model)
has_image = any(
isinstance(m.get("content"), list) and
any(p.get("type") == "image_url" for p in m["content"] if isinstance(p, dict))
for m in msgs
)
if has_image:
if "gemma-4-12b" in avail and not is_gpu_busy("gemma-4-12b"):
return {"model": "gemma-4-12b", "reason": "vision"}
elif "gemma-4-12b" in avail:
return {"model": "gemma-4-12b", "reason": "vision_saturated", "saturated": True}
else:
return {"model": allowed[0], "reason": "vision_unavailable"}
req = rd.get("model","auto")
if req != "auto":
target = req if req in avail else avail[0]
# If explicit model is busy, check if another can take it
if is_gpu_busy(target) and req in allowed:
alts = [m for m in avail if m != target and m in allowed]
if alts:
@@ -251,46 +276,58 @@ def route(rd, tier, agent=""):
return select_best_gpu(["gemma-4-12b"], "hint_speed", agent) or {"model":"gemma-4-12b","reason":"hint_speed"}
if hints.get("priority")=="quality" and "qwen3.6-35B-A3B" in avail:
return select_best_gpu(["qwen3.6-35B-A3B"], "hint_quality", agent) or {"model":"qwen3.6-35B-A3B","reason":"hint_quality"}
if hints.get("priority")=="code" and "qwen3.6-27B-code" in avail:
return select_best_gpu(["qwen3.6-27B-code"], "hint_code", agent) or {"model":"qwen3.6-27B-code","reason":"hint_code"}
first_msg = msgs[0].get("content","") if msgs else ""
words = len(first_msg.split()) if isinstance(first_msg, str) else 99
# TIER 1: Lightweight — single-turn short queries → VLM (fastest)
if not sys and turns <= 1 and t <= 500 and words <= 100 and "gemma-4-12b" in avail:
# TIER 1: Tiny - single-turn micro queries -> VLM (fastest)
if not sys and turns <= 1 and t <= 300 and words <= 100 and "gemma-4-12b" in avail:
if not is_gpu_busy("gemma-4-12b"):
return {"model":"gemma-4-12b","reason":"lightweight"}
# VLM busy — Dense is faster for short queries than MoE
return {"model":"gemma-4-12b","reason":"tiny"}
fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(fallback, "lightweight_fallback", agent)
result = select_best_gpu(fallback, "tiny_fallback", agent)
if result: return result
# TIER 2: Simple conversations — VLM primary (up to 15K tok), fastest for moderate chat
if t <= 15000 and turns <= 12 and "gemma-4-12b" in avail:
if not is_gpu_busy("gemma-4-12b"):
return {"model":"gemma-4-12b","reason":"simple_conv"}
# VLM busy — fall back to Dense, then MoE
fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(fallback, "simple_conv_fallback", agent)
# TIER 2: Light - moderate chat -> VLM first (fastest), Dense fallback
if t <= 5000 and turns <= 4:
candidates = [m for m in ["gemma-4-12b","qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
result = select_best_gpu(candidates, "light", agent)
if result: return result
# TIER 3: Medium complexity — Dense primary, VLM fallback (quality + speed balance)
if t <= 25000:
candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
# TIER 3: Medium - quality matters -> MoE primary (60%), Dense spillover (40%)
if t <= 30000:
candidates = moe_spillover(avail, ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"])
result = select_best_gpu(candidates, "medium", agent)
if result: return result
# TIER 4: Heavy reasoning — MoE primary (workhorse), Dense fallback
if t > 25000:
candidates = [m for m in ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"] if m in avail]
result = select_best_gpu(candidates, "heavy_reasoning", agent)
# TIER 4: Heavy - quality first -> Dense primary, MoE fallback
if t > 30000:
candidates = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B","gemma-4-12b"] if m in avail]
result = select_best_gpu(candidates, "heavy", agent)
if result: return result
# TIER 5: Default — Dense primary, MoE fallback
candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
# TIER 5: Default - MoE primary (60%), Dense spillover (40%)
candidates = moe_spillover(avail, ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"])
result = select_best_gpu(candidates, "default", agent)
if result: return result
return {"model":avail[0],"reason":"last_resort"}
def moe_spillover(avail, default_order):
"""Spill 40% of MoE-first traffic to Dense to prevent Strix Halo overheating.
Only applies when MoE is first candidate, available, and not busy."""
import random
if (default_order[0] == "qwen3.6-35B-A3B"
and "qwen3.6-35B-A3B" in avail
and not is_gpu_busy("qwen3.6-35B-A3B")
and "qwen3.6-27B-code" in avail
and not is_gpu_busy("qwen3.6-27B-code")
and random.random() < 0.4):
# Swap: Dense first, MoE second
return ["qwen3.6-27B-code","qwen3.6-35B-A3B"] + [m for m in default_order[2:] if m in avail and m not in ("qwen3.6-27B-code","qwen3.6-35B-A3B")]
return [m for m in default_order if m in avail]
def clean_unicode(text):
if not isinstance(text, str): return text
text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
@@ -344,16 +381,6 @@ def chat():
return jsonify({"error": "Unauthorized — valid API key required"}), 401
ki = API_KEYS[ak]
tier, agent = ki["tier"], ki["agent"]
# Phase 0: dual-key transition — log deprecated key usage
if ki.get("deprecated"):
new_key = next((k for k, v in API_KEYS.items()
if v.get("agent") == agent and not v.get("deprecated")), None)
log.warning("DEPRECATED_KEY: agent=%s using old key %s...%s — switch to %s...%s",
agent, ak[:12], ak[-8:],
new_key[:12] if new_key else "N/A",
new_key[-8:] if new_key else "N/A")
if r:
r.incr("deprecated_usage:" + agent)
# Rate limit check
allowed, rl_val, reset_sec = check_rate_limit(ak, tier)
@@ -509,9 +536,9 @@ def chat():
@app.route("/metrics/performance")
def performance():
"""Per-request performance analytics with percentiles per model/reason/agent."""
if not r: return jsonify({"error": "Redis unavailable"}), 503
if not get_redis(): return jsonify({"error": "Redis unavailable"}), 503
try:
window_hours = int(request.args.get("window", "24"))
window_hours = int(request.args.get("window", "24").replace("h",""))
model_filter = request.args.get("model", "all")
# Load recent records
@@ -626,9 +653,9 @@ def performance():
@app.route("/metrics/scatter")
def scatter():
"""Return individual data points for scatter plots (prompt_tokens vs latency)."""
if not r: return jsonify({"error": "Redis unavailable"}), 503
if not get_redis(): return jsonify({"error": "Redis unavailable"}), 503
try:
window_hours = int(request.args.get("window", "24"))
window_hours = int(request.args.get("window", "24").replace("h",""))
model_filter = request.args.get("model", "all")
cutoff = time.time() - (window_hours * 3600)
raw = r.lrange("perf:recent", 0, -1)
@@ -713,119 +740,6 @@ def stream():
return Response(stream_with_context(ev()), mimetype="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
# ── Phase 0: Admin Key Management ──
ADMIN_KEY = os.environ.get("ADMIN_KEY", "")
def _admin_auth():
"""Require admin key for management endpoints."""
if not ADMIN_KEY:
return False, "ADMIN_KEY not configured on server"
ak = request.headers.get("Authorization","").replace("Bearer ","")
if ak != ADMIN_KEY:
return False, "Admin key required"
return True, None
@app.route("/admin/keys")
def admin_keys():
"""List all API keys (masked) with agent, tier, and deprecation status."""
ok, err = _admin_auth()
if not ok: return jsonify({"error": err}), 401
keys = []
for key, info in API_KEYS.items():
masked = key[:8] + "..." + key[-8:]
keys.append({
"masked": masked,
"prefix": key[:8],
"agent": info["agent"],
"tier": info["tier"],
"deprecated": info.get("deprecated", False),
"length": len(key)
})
return jsonify({
"total": len(keys),
"active": sum(1 for k in keys if not k["deprecated"]),
"deprecated": sum(1 for k in keys if k["deprecated"]),
"keys": sorted(keys, key=lambda k: (k["deprecated"], k["agent"]))
})
@app.route("/admin/keys/deprecation-summary")
def admin_deprecation_summary():
"""Summary of deprecated key usage (from Redis logs, if available)."""
ok, err = _admin_auth()
if not ok: return jsonify({"error": err}), 401
deprecated_agents = []
for key, info in API_KEYS.items():
if info.get("deprecated"):
# Check Redis for usage count
count = 0
if r:
count = int(r.get("deprecated_usage:" + info["agent"]) or 0)
deprecated_agents.append({
"agent": info["agent"],
"deprecated_uses": count,
"needs_migration": count > 0
})
return jsonify({
"deprecated_agents": sorted(deprecated_agents, key=lambda d: -d["deprecated_uses"]),
"recommendation": "Run POST /admin/keys/revoke to remove keys with 0 usage"
})
@app.route("/admin/keys/generate", methods=["POST"])
def admin_generate_key():
"""Generate a new API key for an agent. Body: {"agent": "Name", "tier": "enterprise"}"""
ok, err = _admin_auth()
if not ok: return jsonify({"error": err}), 401
body = request.get_json(force=True)
agent = body.get("agent", "").strip()
tier = body.get("tier", "enterprise")
if not agent:
return jsonify({"error": "agent field required"}), 400
if tier not in ("starter", "professional", "enterprise"):
return jsonify({"error": "tier must be starter/professional/enterprise"}), 400
# Generate secure key
import secrets, hashlib
prefix = hashlib.sha256(secrets.token_bytes(12)).hexdigest()[:8]
suffix = secrets.token_hex(20)
new_key = f"sk-{prefix}-{suffix}"
# Update in-memory dict (note: not persisted across restarts without env var update)
API_KEYS[new_key] = {"tier": tier, "agent": agent}
log.info("KEY_GENERATED: agent=%s tier=%s key=%s...%s", agent, tier, new_key[:8], new_key[-8:])
return jsonify({
"agent": agent,
"tier": tier,
"key": new_key,
"masked": new_key[:8] + "..." + new_key[-8:],
"warning": "This key exists in memory only. Update API_KEYS env var and redeploy to persist."
}), 201
@app.route("/admin/keys/revoke", methods=["POST"])
def admin_revoke_key():
"""Revoke a deprecated key. Body: {"agent": "Name"} or {"key_prefix": "sk-xxxx"}"""
ok, err = _admin_auth()
if not ok: return jsonify({"error": err}), 401
body = request.get_json(force=True)
agent = body.get("agent", "")
key_prefix = body.get("key_prefix", "")
revoked = []
keys_to_remove = []
for key, info in API_KEYS.items():
if not info.get("deprecated"):
continue
if agent and info["agent"] == agent:
keys_to_remove.append(key)
elif key_prefix and key.startswith(key_prefix):
keys_to_remove.append(key)
for key in keys_to_remove:
info = API_KEYS.pop(key)
revoked.append({"agent": info["agent"], "masked": key[:8] + "..." + key[-8:]})
log.warning("KEY_REVOKED: agent=%s key=%s...%s", info["agent"], key[:8], key[-8:])
return jsonify({
"revoked": len(revoked),
"keys": revoked,
"remaining_total": len(API_KEYS),
"warning": "Memory-only revoke. Update API_KEYS env var and redeploy to persist."
})
if __name__ == "__main__":
log.info("Router on :9000 (load-aware)")
app.run(host="0.0.0.0", port=9000, debug=False)
router/router.py.bak.20260518074236
+342
View File
@@ -0,0 +1,342 @@
import os, json, time, logging, traceback, threading, queue
import requests, redis
from flask import Flask, request, jsonify, Response, stream_with_context
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
GPU_SIDECARS = {
"qwen3.6-35B-A3B": "http://192.168.68.15:8090",
"qwen3.6-27B-code": "http://192.168.68.8:8090",
"qwen3.5-9b-vlm": "http://192.168.68.110:8090",
}
GPU_URLS = {
"qwen3.6-35B-A3B": GPU_MOE_URL,
"qwen3.6-27B-code": GPU_DENSE_URL,
"qwen3.5-9b-vlm": GPU_LIGHT_URL,
}
# Max concurrent requests per GPU (based on llama.cpp --parallel)
GPU_MAX_CONCURRENT = {
"qwen3.6-35B-A3B": 2, # 2 slots
"qwen3.6-27B-code": 2, # 2 slots
"qwen3.5-9b-vlm": 1, # 1 slot
}
TIER_MODELS = {
"starter": ["qwen3.5-9b-vlm"],
"professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
"enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
}
API_KEYS = {
"sk-syslog-local-master-key": {"tier": "enterprise", "agent": "admin"},
"sk-syslog-abiba": {"tier": "enterprise", "agent": "Abiba"},
"sk-syslog-mumuni": {"tier": "enterprise", "agent": "Mumuni"},
"sk-syslog-tanko": {"tier": "enterprise", "agent": "Tanko"},
"sk-syslog-koby": {"tier": "enterprise", "agent": "Koby"},
"sk-syslog-kagenz0": {"tier": "enterprise", "agent": "Kagenz0"},
"sk-syslog-koonimo": {"tier": "enterprise", "agent": "Koonimo"},
"sk-starter-abc123": {"tier": "starter", "agent": "test-starter"},
"sk-professional-xyz789": {"tier": "professional", "agent": "test-pro"},
}
logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
log = logging.getLogger("router")
try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
except Exception: r = None
def counter_audit_loop():
"""Every 30s, check GPU slots and reset counters if all slots idle."""
while True:
time.sleep(30)
if not r: continue
for model, url in GPU_URLS.items():
try:
resp = requests.get(url.replace("/v1","") + "/slots",
headers={"Authorization": "Bearer not-needed"}, timeout=5)
if resp.status_code == 200:
slots = resp.json()
all_idle = all(not s.get("is_processing", False) for s in slots)
if all_idle:
current = int(r.get("active:" + model) or 0)
if current > 0:
r.set("active:" + model, 0)
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
except Exception:
pass
threading.Thread(target=counter_audit_loop, daemon=True).start()
app = Flask(__name__)
sse_subscribers = []; sse_lock = threading.Lock()
def gpu_active_count(model):
"""Get number of in-flight requests for a GPU."""
if r:
return int(r.get("active:" + model) or 0)
return 0
def gpu_incr(model):
if r: r.incr("active:" + model)
def gpu_decr(model):
if r:
v = r.decr("active:" + model)
if v and int(v) < 0:
r.set("active:" + model, 0) # never go negative
def check_gpu_health(model):
url = GPU_SIDECARS.get(model)
if not url: return {"status": "unknown"}
try:
resp = requests.get(url, timeout=5)
if resp.status_code == 200:
d = resp.json()
pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
status = "healthy" if pct < 90 else "saturated"
# Also check if llama.cpp endpoint is actually responding
gpu_url = GPU_URLS.get(model, "")
try:
hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=3)
if hr.status_code != 200:
status = "down"
except Exception:
status = "down"
return {"status": status, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
except Exception: pass
return {"status": "down"}
def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
def estimate_tokens(msgs): return sum(len(str(m.get("content",""))) for m in msgs) // 4
def is_gpu_busy(model):
"""Check if GPU is at or near max concurrent capacity."""
active = gpu_active_count(model)
max_c = GPU_MAX_CONCURRENT.get(model, 1)
return active >= max_c
def select_best_gpu(candidates, reason):
"""Pick the best GPU from candidates, preferring least-loaded."""
best = None
best_load = 999
for m in candidates:
load = gpu_active_count(m)
if load < best_load:
best_load = load
best = m
if best:
actual_reason = reason
if is_gpu_busy(best):
actual_reason = "load_balanced_" + reason
return {"model": best, "reason": actual_reason}
return None
def route(rd, tier):
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
sys = any(m.get("role")=="system" for m in msgs)
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
hints = rd.get("routing_hints",{})
allowed = TIER_MODELS.get(tier, ["qwen3.5-9b-vlm"])
avail = [m for m in available_models() if m in allowed]
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
req = rd.get("model","auto")
if req != "auto":
target = req if req in avail else avail[0]
# If explicit model is busy, check if another can take it
if is_gpu_busy(target) and req in allowed:
alts = [m for m in avail if m != target and m in allowed]
if alts:
alt = select_best_gpu(alts, "explicit")
if alt: return alt
return {"model": target, "reason": "explicit"}
if hints:
if hints.get("priority")=="speed" and "qwen3.5-9b-vlm" in avail:
return select_best_gpu(["qwen3.5-9b-vlm"], "hint_speed") or {"model":"qwen3.5-9b-vlm","reason":"hint_speed"}
if hints.get("priority")=="quality" and "qwen3.6-27B-code" in avail:
return select_best_gpu(["qwen3.6-27B-code"], "hint_quality") or {"model":"qwen3.6-27B-code","reason":"hint_quality"}
# Heavy -> dense (but fall back to MoE if dense is busy)
if t > 4000 or sys or turns > 6:
candidates = ["qwen3.6-27B-code","qwen3.6-35B-A3B","qwen3.5-9b-vlm"]
candidates = [m for m in candidates if m in avail]
result = select_best_gpu(candidates, "heavy_reasoning")
if result: return result
# Ultra-light -> VLM
first_msg = msgs[0].get("content","") if msgs else ""
words = len(first_msg.split()) if isinstance(first_msg, str) else 99
if words <= 3 and turns <= 1 and not sys and "qwen3.5-9b-vlm" in avail:
if not is_gpu_busy("qwen3.5-9b-vlm"):
return {"model":"qwen3.5-9b-vlm","reason":"ultra_light"}
# Default: MoE, fall back to dense if MoE is busy
if "qwen3.6-35B-A3B" in avail:
if is_gpu_busy("qwen3.6-35B-A3B") and "qwen3.6-27B-code" in avail:
return {"model": "qwen3.6-27B-code", "reason": "load_balanced_default"}
return {"model":"qwen3.6-35B-A3B","reason":"default_moe"}
return {"model":avail[0],"reason":"fallback"}
def clean_unicode(text):
if not isinstance(text, str): return text
text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
return text.encode("ascii", "ignore").decode("ascii")
def clean_response(d):
if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
if isinstance(d, list): return [clean_response(v) for v in d]
if isinstance(d, str): return clean_unicode(d)
return d
def get_metrics():
d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
for m in GPU_URLS:
h = check_gpu_health(m)
d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
d["active_requests"][m] = gpu_active_count(m)
if r:
try:
for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
for k,v in API_KEYS.items():
c = int(r.get("routes:agent:"+v["agent"]) or 0)
if c>0: d["agent_counts"][v["agent"]] = c
for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
raw = r.lrange("routes:recent",0,49)
d["recent"] = [json.loads(x) for x in raw] if raw else []
except Exception: pass
return d
def bcast():
data = get_metrics(); payload = json.dumps(data)
with sse_lock:
dead = []
for q in sse_subscribers:
try: q.put(payload)
except Exception: dead.append(q)
for q in dead: sse_subscribers.remove(q)
@app.route("/v1/chat/completions", methods=["POST"])
def chat():
try:
rd = request.get_json(force=True)
ak = request.headers.get("Authorization","").replace("Bearer ","")
ki = API_KEYS.get(ak, {"tier":"starter","agent":"unknown"})
tier, agent = ki["tier"], ki["agent"]
d = route(rd, tier)
if d.get("saturated"):
resp = jsonify({"error": "All GPUs saturated", "retry_after_s": 5})
resp.headers["Retry-After"] = "5"
return resp, 503
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
is_stream = rd.get("stream", False)
gpu_incr(model)
decremented = False
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
if r:
try:
r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent}))
r.ltrim("routes:recent",0,999)
except Exception: pass
start = time.time()
resp = requests.post(url+"/chat/completions", json=rd,
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
lat = int((time.time()-start)*1000)
gpu_decr(model)
decremented = True # Release slot
if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
if is_stream:
def gen():
for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
if raw: yield clean_unicode(raw)
bcast()
return Response(stream_with_context(gen()), mimetype="text/event-stream")
data = clean_response(resp.json())
for c in data.get("choices",[]):
msg = c.get("message",{})
if not msg.get("content") and msg.get("reasoning_content"):
msg["content"] = msg["reasoning_content"]
data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"active_gpu":gpu_active_count(model)}
bcast()
return jsonify(data)
if not decremented:
try: gpu_decr(model)
except: pass
except requests.Timeout:
return jsonify({"error":"timeout"}), 504
log.error("Error: %s\n%s", e, traceback.format_exc())
return jsonify({"error":str(e)}), 500
@app.route("/v1/models")
def models(): return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":check_gpu_health(m).get("status"),"gpu":check_gpu_health(m).get("gpu_name")} for m in GPU_URLS]})
@app.route("/health")
def health():
gpus = {}
for m in GPU_URLS:
h = check_gpu_health(m)
h["active_requests"] = gpu_active_count(m)
h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
gpus[m] = h
return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
@app.route("/metrics")
def metrics(): return jsonify(get_metrics())
@app.route("/metrics/timeseries")
def metrics_timeseries():
period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
data = {"models": {}, "labels": []}
if period == "day":
buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
elif period == "week":
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
else:
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
if r:
for model in models_list:
counts = []
for bucket in buckets:
total = 0
if period in ("week","month"):
for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
else: total = int(r.get("ts:"+model+":"+bucket) or 0)
counts.append(total)
data["models"][model] = counts
return jsonify(data)
@app.route("/stream")
def stream():
def ev():
q = queue.Queue()
with sse_lock: sse_subscribers.append(q)
try:
yield "data: "+json.dumps(get_metrics())+"\n\n"
while True:
try: yield "data: "+q.get(timeout=3)+"\n\n"
except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
except GeneratorExit: pass
finally:
with sse_lock:
if q in sse_subscribers: sse_subscribers.remove(q)
return Response(stream_with_context(ev()), mimetype="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
if __name__ == "__main__":
log.info("Router on :9000 (load-aware)")
app.run(host="0.0.0.0", port=9000, debug=False)
ssl/README.md
+6
View File
@@ -0,0 +1,6 @@
# SSL Directory
SSL termination is handled upstream by NetBird/Authentik.
This directory is intentionally empty — no certs stored here.
For local dev SSL, use the docker-compose.override.yml pattern.