Compare commits
31
Commits
815ed7991f
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
 jerome
|
937ba2c1ce | ||
 Abiba
|
9acabf7ba6 | ||
|
Abiba
|
8e0f6e407b | ||
|
Abiba
|
ac13ecaaf7 | ||
|
Abiba
|
0ca3b65ad4 | ||
|
Abiba
|
13eb8cb75b | ||
|
Abiba
|
08680b0f9e | ||
|
kagentz-bot
|
621fb3540a | ||
|
kagentz-bot
|
3d6b8173b0 | ||
|
kagentz-bot
|
aa5ac4a280 | ||
|
kagentz-bot
|
ce703b8328 | ||
|
Abiba
|
fd3c2a575a | ||
|
Abiba
|
776343f2ab | ||
|
jerome
|
492a4fe68b | ||
|
kagentz-botandAbiba
|
84e0d163ee | ||
|
Abiba
|
d901235c03 | ||
|
jerome
|
4c7ac3350d | ||
|
Abiba
|
316f2f5f45 | ||
|
Abiba
|
574076119c | ||
|
Abiba
|
ad9881f141 | ||
|
jerome
|
3625fdc860 | ||
|
Abiba
|
a860a8fd0f | ||
|
Abiba
|
fabbe340d6 | ||
|
Abiba
|
2e24ee5598 | ||
|
Abiba
|
19f7d90cc1 | ||
|
Abiba
|
b79af634d7 | ||
|
jerome
|
a992d4b88f | ||
|
jerome
|
c3dfe62cec | ||
|
Abiba
|
f1d095e411 | ||
|
jerome
|
ae3f581e3e | ||
|
Abiba
|
c7b22f4d09 |
@@ -1,3 +1,6 @@
|
|||||||
.git
|
.git
|
||||||
__pycache__/
|
__pycache__/
|
||||||
*.pyc
|
*.pyc
|
||||||
|
*.bak
|
||||||
|
*.backup*
|
||||||
|
.env
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# LiteLLM Virtual Key Distribution — Phase 2 Cutover
|
||||||
|
## Syslog Solution LLC — July 1, 2026 (keys regenerated)
|
||||||
|
|
||||||
|
### Active Agent Keys (Update agent configs with these)
|
||||||
|
|
||||||
|
| Agent | LiteLLM Virtual Key | Tier | Budget | Models | Verified |
|
||||||
|
|-------|-------------------|------|--------|--------|----------|
|
||||||
|
| **Abiba** | `sk-i7F7rCfgpS0oouOTA2LgMw` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||||
|
| **Mumuni** | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||||
|
| **Tanko** | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||||
|
| **Kagenz0** | `sk-VYKRAVYEG1rKVEMDKEUggg` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||||
|
| **Koby** | `sk-gWDaPAp-FavgKdqKJpzqhQ` | professional | $500 | All 4 | ✅ Jul 1 |
|
||||||
|
| **Koonimo** | `sk-1kLC8ZxW-tEZueV3NU4rCQ` | professional | $500 | All 4 | ✅ Jul 1 |
|
||||||
|
|
||||||
|
### Configuration Changes Required Per Agent
|
||||||
|
|
||||||
|
Each agent must update their `OPENAI_API_BASE` and `OPENAI_API_KEY`:
|
||||||
|
|
||||||
|
```
|
||||||
|
OPENAI_API_BASE=http://192.168.68.116/v1
|
||||||
|
OPENAI_API_KEY=<agent's LiteLLM key from table above>
|
||||||
|
```
|
||||||
|
|
||||||
|
### LiteLLM Admin Access
|
||||||
|
|
||||||
|
| Resource | Key |
|
||||||
|
|----------|-----|
|
||||||
|
| LiteLLM Admin UI | http://192.168.68.116/ui/ (Authentik SSO — pending) |
|
||||||
|
| LiteLLM Master Key | `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` |
|
||||||
|
| Router Admin Key | `sk-admin-ee09fffd04978b61a1569ac670c68814` |
|
||||||
|
|
||||||
|
### Fallback (if LiteLLM fails)
|
||||||
|
|
||||||
|
If LiteLLM is down, NGINX automatically falls back to the router on :9000.
|
||||||
|
In that case, agents can also directly use:
|
||||||
|
```
|
||||||
|
OPENAI_API_BASE=http://192.168.68.116/v1
|
||||||
|
OPENAI_API_KEY=<agent's original sk-syslog-* key>
|
||||||
|
```
|
||||||
|
(Only works when NGINX @router_fallback is active)
|
||||||
|
|
||||||
|
### Deprecated Router Keys
|
||||||
|
|
||||||
|
These keys still work through the @router_fallback but NOT through LiteLLM:
|
||||||
|
- sk-syslog-abiba, sk-syslog-mumuni, sk-syslog-tanko
|
||||||
|
- sk-syslog-kagenz0, sk-syslog-koby, sk-syslog-koonimo
|
||||||
|
- sk-starter-abc123, sk-professional-xyz789
|
||||||
|
- sk-syslog-local-master-key
|
||||||
|
|
||||||
|
These have been fully revoked as of July 1, 2026.
|
||||||
|
|
||||||
|
### Key Rotation History
|
||||||
|
|
||||||
|
- **2026-07-01**: Keys regenerated for Abiba, Kagenz0, Koby, Koonimo (old values unrecoverable).
|
||||||
|
Mumuni and Tanko keys confirmed working, left unchanged. All blocked/stale keys purged.
|
||||||
|
- **2026-06-17**: Initial key creation (values in this doc were placeholder/incorrect).
|
||||||
@@ -0,0 +1,759 @@
|
|||||||
|
# LiteLLM Integration Migration Plan
|
||||||
|
## Syslog Solution LLC June 14, 2026
|
||||||
|
|
||||||
|
**Deployment Target:** CT 116 `syslog-api` (192.168.68.116) on minipve all services co-located.
|
||||||
|
**DNS Strategy:** Option A /etc/hosts + Docker extra_hosts for internal resolution of `auth.sysloggh.net` 192.168.68.11.
|
||||||
|
**GitOps:** This plan lives in `SyslogSolution/syslog-harness` on Gitea. All changes tracked via git with conventional commits.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
**Goal:** Layer the full LiteLLM Gateway suite (Admin UI, virtual keys, spend tracking, teams/SSO, budget management) on top of our custom intelligent routing harness without sacrificing GPU-aware slot management, content-based tiering, or hardware health monitoring.
|
||||||
|
|
||||||
|
**Architecture Decision:** Two-layer architecture.
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
LiteLLM Gateway (Layer 1)
|
||||||
|
Port 4000 Policy & UX
|
||||||
|
|
||||||
|
Admin UI (/ui)
|
||||||
|
Virtual Keys & Permissions
|
||||||
|
Teams, Users, SSO (OIDC)
|
||||||
|
Spend Tracking & Budgets
|
||||||
|
Usage Analytics Dashboard
|
||||||
|
Request Audit Trail
|
||||||
|
Global Rate Limiting
|
||||||
|
|
||||||
|
|
||||||
|
Pass-through to router
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Custom Router (Layer 2)
|
||||||
|
Port 9000 Intelligence & HW
|
||||||
|
|
||||||
|
5-Tier Content-Based Routing
|
||||||
|
GPU Slot Management (Redis)
|
||||||
|
Agent Spread Prevention
|
||||||
|
GPU Health Scoring
|
||||||
|
Sidecar VRAM/Temp/Power
|
||||||
|
Circuit Breaker
|
||||||
|
Context Window Tracking
|
||||||
|
Per-Request Perf Recording
|
||||||
|
Hardware Rate Limiting
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
qwen3.6-35B qwen3.6-27B gemma-4-12b
|
||||||
|
MoE/Strix Dense/RTX3090 VLM/RTX 5070
|
||||||
|
:8080 (llama) :8080 (llama) :8080 (llama)
|
||||||
|
:8090 (side) :8090 (side) :8090 (sidecar)
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Current State Baseline
|
||||||
|
|
||||||
|
### 1.1 Router (`router-fixed.py` port 9000, deployed on CT 116 / syslog-api)
|
||||||
|
|
||||||
|
**Deployment Host:** CT 116 `syslog-api` on minipve (192.168.68.12), IP 192.168.68.116, 6GB RAM, 40GB disk. Runs Docker with all harness services co-located on this single host.
|
||||||
|
|
||||||
|
| Feature | Implementation |
|
||||||
|
|---------|---------------|
|
||||||
|
| **Routing Engine** | 5-tier content-based: lightweight simple_conv medium heavy_reasoning default |
|
||||||
|
| **GPU Slot Mgmt** | Redis atomic incr/decr, max 2 concurrent per GPU, audit loop reset |
|
||||||
|
| **Health Checks** | Sidecar endpoint per GPU (VRAM, temp, util, power) + llama.cpp /health |
|
||||||
|
| **Agent Spreading** | `select_best_gpu()` prefers GPUs with 0 other agents, then non-self GPUs |
|
||||||
|
| **Rate Limiting** | Token bucket (Redis), per-tier RPM: enterprise=120, professional=60, starter=20 |
|
||||||
|
| **Auth** | Dual-key system (Phase 0.5): 9 new + 9 deprecated keys, admin key rotation |
|
||||||
|
| **Performance** | Per-request latency/tokens/tps Redis lists (perf:recent, perf:model:X, perf:agent:X) |
|
||||||
|
| **Context Tracking** | Session-level token accumulation with compaction warnings in headers |
|
||||||
|
| **SSE Streaming** | Real-time dashboard updates, per-model timeseries |
|
||||||
|
| **Admin** | `/admin/keys`, `/admin/keys/generate`, `/admin/keys/revoke`, `/admin/keys/deprecation-summary` |
|
||||||
|
| **Strict Passthrough** | Explicit model requests go to that GPU exactly (no silent fallback). LiteLLM owns failover. |
|
||||||
|
|
||||||
|
### 1.2 GPU Backends
|
||||||
|
|
||||||
|
| GPU | Host | llama.cpp | Sidecar | VRAM | Context |
|
||||||
|
|-----|------|-----------|---------|------|---------|
|
||||||
|
| qwen3.6-35B-A3B (MoE) | 192.168.68.15 | :8080 | :8090 | Strix Halo | 262K |
|
||||||
|
| qwen3.6-27B-code (Dense) | 192.168.68.8 | :8080 | :8090 | RTX 3090 | 262K |
|
||||||
|
| gemma-4-12b (VLM) | 192.168.68.110 | :8080 | :8090 | RTX 5070 | 262K |
|
||||||
|
|
||||||
|
### 1.3 Existing LiteLLM POC on CT 116
|
||||||
|
|
||||||
|
CT 116 already has a LiteLLM container running (POC, 6 days uptime):
|
||||||
|
|
||||||
|
```
|
||||||
|
harness-litellm | ghcr.io/berriai/litellm:main-stable | 127.0.0.1:8081->4000
|
||||||
|
harness-redis | redis:7-alpine | 127.0.0.1:6379
|
||||||
|
harness-router | inference-harness-router | 127.0.0.1:9000
|
||||||
|
harness-nginx | nginx:alpine | 0.0.0.0:80
|
||||||
|
harness-dashboard | inference-harness-dashboard | 127.0.0.1:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
- `/opt/litellm/` previous setup directory on CT 116
|
||||||
|
- Configured with Postgres, host networking, master key
|
||||||
|
- Currently bypassed router routes directly to GPUs
|
||||||
|
- **Goal: Productionize with two-layer architecture on this same host**
|
||||||
|
|
||||||
|
### 1.4 DNS Routing (Split-Horizon)
|
||||||
|
|
||||||
|
For OIDC SSO with Authentik, CT 116 must resolve `auth.sysloggh.net` internally:
|
||||||
|
|
||||||
|
**Problem:** `auth.sysloggh.net` CNAMEs to `netbird.sysloggh.net` 72.61.0.17 (public VPS). OIDC auth_request from NGINX would route through the internet back to 192.168.68.11 unnecessarily.
|
||||||
|
|
||||||
|
**Solution Option A: /etc/hosts on CT 116 host:**
|
||||||
|
```bash
|
||||||
|
# On CT 116 (syslog-api)
|
||||||
|
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
|
||||||
|
```
|
||||||
|
|
||||||
|
**Docker containers** also need this resolution add to docker-compose.yml:
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
nginx:
|
||||||
|
extra_hosts:
|
||||||
|
- "auth.sysloggh.net:192.168.68.11"
|
||||||
|
litellm:
|
||||||
|
extra_hosts:
|
||||||
|
- "auth.sysloggh.net:192.168.68.11"
|
||||||
|
```
|
||||||
|
|
||||||
|
**DNS Servers:** CT 116 uses 192.168.68.10 for DNS. AdGuard (192.168.68.11) is the long-term solution for LAN-wide split-horizon DNS.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. What LiteLLM Brings (That We Don't Have)
|
||||||
|
|
||||||
|
| Feature | Our Router | LiteLLM | Value Add |
|
||||||
|
|---------|-----------|---------|-----------|
|
||||||
|
| **Admin UI** | | Full dashboard at /ui | Non-technical users can manage keys, view spend |
|
||||||
|
| **Virtual Key Permissions** | (binary key->tier) | Granular: per-model, per-team, budget caps | Fine-grained access control |
|
||||||
|
| **Spend Tracking** | | Per-request $ cost with model-specific pricing | Billing, cost allocation, client invoicing |
|
||||||
|
| **Teams & Orgs** | | Multi-tenant: org->team->user hierarchy | Segregate clients/projects |
|
||||||
|
| **SSO/OIDC** | | Google, GitHub, Microsoft, Okta, Keycloak | Enterprise auth integration |
|
||||||
|
| **Budget Alerts** | | Per-key, per-user, per-team budget with webhooks | Prevent overspend |
|
||||||
|
| **Usage Analytics** | (custom /metrics) | Built-in: daily trends, model breakdown, per-customer | Better visualization |
|
||||||
|
| **100+ Provider Support** | (3 local GPUs) | OpenAI, Anthropic, Bedrock, Vertex, etc. | Future cloud model access |
|
||||||
|
| **Fallback Chains** | (silent rerouting) | Explicit multi-provider failover with per-model logging | Accurate per-model tracking, visible failover |
|
||||||
|
| **RPM/TPM Weighted LB** | | Weighted load balancing across deployments | Fine-grained traffic shaping |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. What We Keep (That LiteLLM Doesn't Have)
|
||||||
|
|
||||||
|
| Feature | Why We Must Keep It |
|
||||||
|
|---------|---------------------|
|
||||||
|
| **Content-based 5-tier routing** | LiteLLM routes by model name only; we analyze prompt complexity, tokens, turns, and routing_hints |
|
||||||
|
| **GPU hardware health scoring** | LiteLLM doesn't monitor VRAM, temp, power our scoring prevents routing to overheating GPUs |
|
||||||
|
| **GPU slot management** | LiteLLM doesn't know about llama.cpp --parallel limits; our Redis counters prevent overloading |
|
||||||
|
| **Agent spread prevention** | Our `select_best_gpu()` spreads agents across GPUs to prevent hotspots; LiteLLM only does simple-shuffle |
|
||||||
|
| **Cross-turn context tracking** | Session-level token accumulation with compaction warnings via X-Context-Warning headers |
|
||||||
|
| **GPU sidecar metrics** | VRAM %, GPU utilization %, power draw, temperature exposed via /metrics and SSE dashboard |
|
||||||
|
| **Circuit breaker** | 39 failures caught June 12; LiteLLM's allowed_fails/cooldown is less granular |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Migration Architecture
|
||||||
|
|
||||||
|
### 4.1 Principle: "LiteLLM is the lobby, our router is the engine room"
|
||||||
|
|
||||||
|
- **LiteLLM** handles everything a **user/admin** touches: keys, teams, budgets, spend logs, SSO, the UI
|
||||||
|
- **Custom Router** handles everything the **GPUs** need: health checks, slot booking, content-based routing, hardware monitoring, circuit breaking
|
||||||
|
|
||||||
|
### 4.2 Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Agent Request
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
LiteLLM Gateway (:4000)
|
||||||
|
|
||||||
|
1. Authenticate virtual key (sk-litellm-...)
|
||||||
|
2. Check key permissions (model access)
|
||||||
|
3. Check budget (per-key, per-user, per-team)
|
||||||
|
4. Check team rate limits
|
||||||
|
5. Log request metadata
|
||||||
|
6. Forward to custom router as OpenAI-compat
|
||||||
|
POST http://router:9000/v1/chat/completions
|
||||||
|
Headers: Authorization: Bearer ***
|
||||||
|
X-LiteLLM-User: <user-id>
|
||||||
|
X-LiteLLM-Team: <team-id>
|
||||||
|
X-Session-Id: <session>
|
||||||
|
|
||||||
|
7. On response: log spend, update budgets
|
||||||
|
8. If router returns 503 (GPU saturated):
|
||||||
|
consult fallback chain, retry next model
|
||||||
|
9. Return response to agent
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Custom Router (:9000)
|
||||||
|
|
||||||
|
1. Authenticate agent key (sk-syslog-...)
|
||||||
|
2. Hardware rate limit (per-tier RPM)
|
||||||
|
3. Content-based tier routing (for syslog-auto)
|
||||||
|
OR strict passthrough (for explicit models)
|
||||||
|
4. GPU slot availability (Redis counter)
|
||||||
|
5. GPU health check (sidecar)
|
||||||
|
6. Agent spread logic (select_best_gpu)
|
||||||
|
7. Queue if saturated (with timeout)
|
||||||
|
8. Forward to selected llama.cpp GPU
|
||||||
|
9. Track context window, set compaction header
|
||||||
|
10. Record performance metrics
|
||||||
|
11. Return response (with routing metadata)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
llama.cpp GPU (:8080)
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 LiteLLM Config (`config.yaml`)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
general_settings:
|
||||||
|
master_key: os.environ/LITELLM_MASTER_KEY
|
||||||
|
database_url: postgresql://litellm:***@postgres:5432/litellm
|
||||||
|
store_model_in_db: true
|
||||||
|
|
||||||
|
model_list:
|
||||||
|
# Content-based auto-routing (router picks GPU via 5-tier analysis)
|
||||||
|
- model_name: syslog-auto
|
||||||
|
litellm_params:
|
||||||
|
model: openai/syslog-auto
|
||||||
|
api_base: http://router:9000/v1
|
||||||
|
api_key: os.environ/ROUTER_API_KEY
|
||||||
|
rpm: 600
|
||||||
|
|
||||||
|
# Individual GPU strict passthrough (exact GPU, no silent fallback)
|
||||||
|
- model_name: qwen3.6-35B-A3B
|
||||||
|
litellm_params:
|
||||||
|
model: openai/qwen3.6-35B-A3B
|
||||||
|
api_base: http://router:9000/v1
|
||||||
|
api_key: os.environ/ROUTER_API_KEY
|
||||||
|
|
||||||
|
- model_name: qwen3.6-27B-code
|
||||||
|
litellm_params:
|
||||||
|
model: openai/qwen3.6-27B-code
|
||||||
|
api_base: http://router:9000/v1
|
||||||
|
api_key: os.environ/ROUTER_API_KEY
|
||||||
|
|
||||||
|
- model_name: gemma-4-12b
|
||||||
|
litellm_params:
|
||||||
|
model: openai/gemma-4-12b
|
||||||
|
api_base: http://router:9000/v1
|
||||||
|
api_key: os.environ/ROUTER_API_KEY
|
||||||
|
|
||||||
|
# Guardrails: Pre-call and post-call content moderation
|
||||||
|
guardrails:
|
||||||
|
- guardrail_name: "input-moderation"
|
||||||
|
litellm_params:
|
||||||
|
guardrail: openai_moderation
|
||||||
|
mode: "pre_call"
|
||||||
|
|
||||||
|
- guardrail_name: "output-moderation"
|
||||||
|
litellm_params:
|
||||||
|
guardrail: openai_moderation
|
||||||
|
mode: "post_call"
|
||||||
|
|
||||||
|
- guardrail_name: "harmful-content-filter"
|
||||||
|
litellm_params:
|
||||||
|
guardrail: litellm_content_filter
|
||||||
|
mode: "pre_call"
|
||||||
|
categories:
|
||||||
|
- category: "harmful_self_harm"
|
||||||
|
enabled: true
|
||||||
|
action: "BLOCK"
|
||||||
|
severity_threshold: "medium"
|
||||||
|
- category: "harmful_violence"
|
||||||
|
enabled: true
|
||||||
|
action: "BLOCK"
|
||||||
|
severity_threshold: "medium"
|
||||||
|
- category: "harmful_illegal_weapons"
|
||||||
|
enabled: true
|
||||||
|
action: "BLOCK"
|
||||||
|
severity_threshold: "medium"
|
||||||
|
|
||||||
|
litellm_settings:
|
||||||
|
num_retries: 0 # Disabled our router handles retry
|
||||||
|
request_timeout: 600 # Match our 10-min llama-server timeout
|
||||||
|
set_verbose: true
|
||||||
|
failure_callback: ["prometheus"] # Optional: export to Prometheus
|
||||||
|
|
||||||
|
router_settings:
|
||||||
|
routing_strategy: "usage-based-routing" # For external models only
|
||||||
|
enable_loadbalancing_on_proxy: false # Disable LiteLLM's internal LB
|
||||||
|
allowed_fails: 100 # Router returns 503 on saturated GPUs cooldown disabled
|
||||||
|
# Fallback chains: LiteLLM retries down the chain when router returns saturated
|
||||||
|
# This gives accurate per-model metrics because router no longer silently reroutes
|
||||||
|
fallbacks:
|
||||||
|
- qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
|
||||||
|
- qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
|
||||||
|
- gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
|
||||||
|
|
||||||
|
# Cost tracking: map model names to per-token pricing for spend tracking
|
||||||
|
litellm_settings:
|
||||||
|
model_cost:
|
||||||
|
qwen3.6-35B-A3B:
|
||||||
|
input_cost_per_token: 0.0
|
||||||
|
output_cost_per_token: 0.0
|
||||||
|
qwen3.6-27B-code:
|
||||||
|
input_cost_per_token: 0.0
|
||||||
|
output_cost_per_token: 0.0
|
||||||
|
gemma-4-12b:
|
||||||
|
input_cost_per_token: 0.0
|
||||||
|
output_cost_per_token: 0.0
|
||||||
|
# For internal cost allocation, set symbolic rates:
|
||||||
|
# e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 Router Modifications
|
||||||
|
|
||||||
|
To accommodate LiteLLM, `router-fixed.py` requires the following updates:
|
||||||
|
|
||||||
|
1. **Strict passthrough for explicit models** (DEPLOYED):
|
||||||
|
```python
|
||||||
|
# In route(), the explicit model section changed from silent fallback to strict:
|
||||||
|
req = rd.get("model","auto")
|
||||||
|
if req != "auto":
|
||||||
|
# STRICT MODE: no silent fallback LiteLLM handles failover chains.
|
||||||
|
# This keeps per-model metrics accurate. Returns saturated if busy.
|
||||||
|
target = req if req in avail else avail[0]
|
||||||
|
if req not in avail:
|
||||||
|
return {"model": req, "reason": "explicit_unavailable", "saturated": True}
|
||||||
|
if is_gpu_busy(target):
|
||||||
|
return {"model": target, "reason": "explicit_saturated", "saturated": True}
|
||||||
|
return {"model": target, "reason": "explicit"}
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **New header passthrough**: Forward `X-LiteLLM-*` headers to GPU (transparent already works)
|
||||||
|
|
||||||
|
3. **New endpoint for health passthrough**: `GET /v1/models` already works
|
||||||
|
|
||||||
|
4. **Keep ALL routing logic**: No changes to `select_best_gpu()`, `check_gpu_health()`, slot management, etc. Content-based routing for `syslog-auto` is fully intact.
|
||||||
|
|
||||||
|
5. **Add LiteLLM-compatible response**: Return `X-Usage-Tokens` header so LiteLLM can track token costs
|
||||||
|
```python
|
||||||
|
resp.headers["X-Usage-Tokens"] = json.dumps({
|
||||||
|
"prompt_tokens": prompt_tokens,
|
||||||
|
"completion_tokens": completion_tokens,
|
||||||
|
"model": model
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.5 Router Logic Refinements
|
||||||
|
|
||||||
|
**GPU Health Scoring (Updated):**
|
||||||
|
We are updating the scoring algorithm to include Power metrics:
|
||||||
|
```python
|
||||||
|
def gpu_health_score(model):
|
||||||
|
h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
|
||||||
|
if h.get("status") == "down":
|
||||||
|
return 999 # never pick down GPUs
|
||||||
|
vram_pct = h.get("vram_pct") or 50
|
||||||
|
temp_c = h.get("temp_c") or 50
|
||||||
|
power_w = h.get("power_w") or 50
|
||||||
|
active = gpu_active_count(model)
|
||||||
|
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||||
|
load_pct = (active / max_c) * 100 if max_c > 0 else 0
|
||||||
|
# Score: lower = better
|
||||||
|
score = (vram_pct * 0.3) + (max((temp_c - 30, 0) * 0.3) + (power_w * 0.2) + (load_pct * 0.2))
|
||||||
|
return score
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Deployment Plan (4 Phases) Zero-Downtime Strategy
|
||||||
|
|
||||||
|
### Phase 0: Infrastructure Prep (Current Week) Zero Downtime
|
||||||
|
|
||||||
|
**Goal:** Prepare CT 116 infrastructure without affecting running agents.
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. **Set up DNS split-horizon on CT 116**
|
||||||
|
```bash
|
||||||
|
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Deploy Postgres container** alongside existing services
|
||||||
|
|
||||||
|
3. **Replace LiteLLM config** with production config.yaml (see 4.3)
|
||||||
|
- All 4 models `http://router:9000/v1`
|
||||||
|
- Fallback chains for explicit models
|
||||||
|
- Guardrails (pre-call, post-call, content filter)
|
||||||
|
- `allowed_fails: 100` (router returns 503 on saturated)
|
||||||
|
- `num_retries: 0` (LiteLLM retries handled by fallback chains)
|
||||||
|
|
||||||
|
4. **Deploy custom_sso.py** for Authentik OIDC integration
|
||||||
|
|
||||||
|
5. **Restart LiteLLM container** with new config
|
||||||
|
|
||||||
|
6. **Verify internal routing**
|
||||||
|
|
||||||
|
**Verification Checklist:**
|
||||||
|
- [ ] DNS resolution: `getent hosts auth.sysloggh.net` 192.168.68.11
|
||||||
|
- [ ] Postgres container healthy
|
||||||
|
- [ ] LiteLLM `/health` returns 200
|
||||||
|
- [ ] LiteLLM Router pass-through returns valid chat completion
|
||||||
|
- [ ] GPU health metrics unaffected
|
||||||
|
- [ ] Explicit model request returns saturated (not silently rerouted) when GPU busy
|
||||||
|
|
||||||
|
### Phase 1: Shadow Mode (Week 1) Zero Risk, Zero Downtime
|
||||||
|
|
||||||
|
**Goal:** Deploy LiteLLM alongside existing router, test in shadow mode. **Agents continue using :9000 directly.**
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Create virtual keys for test agents via LiteLLM UI
|
||||||
|
2. Verify pass-through works for all 4 models
|
||||||
|
3. Validate fallback chains: saturate MoE confirm LiteLLM retries Dense confirm VLM
|
||||||
|
4. Run 24-hour shadow: monitor LiteLLM spend logs vs router metrics
|
||||||
|
5. Verify GPU health metrics unaffected
|
||||||
|
6. Check guardrails not generating false positives
|
||||||
|
|
||||||
|
### Phase 2: Cutover (Week 2) Gradual Agent Migration
|
||||||
|
|
||||||
|
**Goal:** Move agents one-by-one to LiteLLM endpoint.
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Migrate API keys to LiteLLM virtual keys
|
||||||
|
2. Create teams: "Core Agents" (enterprise), "Dev Agents" (professional)
|
||||||
|
3. Update agent configs one at a time: `OPENAI_API_BASE` `:4000`
|
||||||
|
4. Test each agent individually
|
||||||
|
5. Enable SSO via Authentik + custom_sso.py
|
||||||
|
6. Keep router :9000 as emergency fallback for 48 hours
|
||||||
|
|
||||||
|
### Phase 3: Production Hardening (Week 3+)
|
||||||
|
|
||||||
|
**Goal:** Lock down, optimize, monitor.
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
1. Remove deprecated router endpoints (after all agents migrated)
|
||||||
|
2. Add LiteLLM observability (Prometheus, Slack/email alerts)
|
||||||
|
3. Enable LiteLLM caching (shared Redis)
|
||||||
|
4. Add external model fallbacks for client-facing services
|
||||||
|
5. Router slim-down: keep routing/slots/health/perf, remove key management
|
||||||
|
6. Multi-tenancy setup for client-facing inference services
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Nginx Configuration (with Authentik OIDC Forward Auth)
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
# OLD (remove)
|
||||||
|
# location /admin/ {
|
||||||
|
# proxy_pass http://127.0.0.1:9000/admin/;
|
||||||
|
# }
|
||||||
|
|
||||||
|
# === Authentik auth subrequest endpoint ===
|
||||||
|
location /authentik/auth {
|
||||||
|
internal;
|
||||||
|
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||||
|
proxy_pass_request_body off;
|
||||||
|
proxy_set_header Content-Length "";
|
||||||
|
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
}
|
||||||
|
|
||||||
|
# === LiteLLM Admin UI Authentik-protected ===
|
||||||
|
location /ui/ {
|
||||||
|
auth_request /authentik/auth;
|
||||||
|
auth_request_set $auth_user $upstream_http_x_authentik_username;
|
||||||
|
auth_request_set $auth_email $upstream_http_x_authentik_email;
|
||||||
|
|
||||||
|
proxy_set_header X-Authentik-Username $auth_user;
|
||||||
|
proxy_set_header X-Authentik-Email $auth_email;
|
||||||
|
|
||||||
|
proxy_pass http://127.0.0.1:4000/ui/;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection "upgrade";
|
||||||
|
}
|
||||||
|
|
||||||
|
# === LiteLLM SSO callback ===
|
||||||
|
location /sso/callback {
|
||||||
|
proxy_pass http://127.0.0.1:4000/sso/callback;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
}
|
||||||
|
|
||||||
|
# === API endpoint Bearer token auth ===
|
||||||
|
location /v1/ {
|
||||||
|
proxy_pass http://127.0.0.1:4000/v1/;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_read_timeout 600s;
|
||||||
|
error_page 502 = @router_fallback;
|
||||||
|
}
|
||||||
|
|
||||||
|
location @router_fallback {
|
||||||
|
proxy_pass http://127.0.0.1:9000/v1/;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
}
|
||||||
|
|
||||||
|
# === Key management API ===
|
||||||
|
location /key/ {
|
||||||
|
proxy_pass http://127.0.0.1:4000/key/;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header Authorization $http_authorization;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Keep router metrics accessible (not behind LiteLLM)
|
||||||
|
location /router/ {
|
||||||
|
proxy_pass http://127.0.0.1:9000/;
|
||||||
|
}
|
||||||
|
|
||||||
|
location /health {
|
||||||
|
proxy_pass http://127.0.0.1:4000/health;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Docker Compose (`docker-compose.yml` on CT 116)
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
# Layer 1: LiteLLM Gateway (Policy & Admin)
|
||||||
|
litellm:
|
||||||
|
image: ghcr.io/berriai/litellm:main-stable
|
||||||
|
network_mode: "host"
|
||||||
|
extra_hosts:
|
||||||
|
- "auth.sysloggh.net:192.168.68.11"
|
||||||
|
volumes:
|
||||||
|
- ./config.yaml:/app/config.yaml:ro
|
||||||
|
- ./custom_sso.py:/app/custom_sso.py:ro
|
||||||
|
environment:
|
||||||
|
- LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
|
||||||
|
- DATABASE_URL=postgresql://litellm:***@localhost:5432/litellm
|
||||||
|
- STORE_MODEL_IN_DB=True
|
||||||
|
- ROUTER_API_KEY=${ROUTER_API_KEY}
|
||||||
|
- OPENAI_API_KEY=***
|
||||||
|
- ANTHROPIC_API_KEY=${ANTH...KEY}
|
||||||
|
- PROXY_BASE_URL=https://litellm.sysloggh.net
|
||||||
|
command:
|
||||||
|
- --config
|
||||||
|
- /app/config.yaml
|
||||||
|
- --port
|
||||||
|
- "4000"
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
postgres:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
network_mode: "host"
|
||||||
|
environment:
|
||||||
|
- POSTGRES_DB=litellm
|
||||||
|
- POSTGRES_USER=litellm
|
||||||
|
- POSTGRES_PASSWORD=${POST...}
|
||||||
|
volumes:
|
||||||
|
- pgdata:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 5
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
nginx:
|
||||||
|
image: nginx:alpine
|
||||||
|
extra_hosts:
|
||||||
|
- "auth.sysloggh.net:192.168.68.11"
|
||||||
|
volumes:
|
||||||
|
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||||
|
ports:
|
||||||
|
- "80:80"
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
pgdata:
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Risk Mitigation
|
||||||
|
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|------|------------|
|
||||||
|
| LiteLLM adds latency overhead | Shadow mode measures: <50ms extra is acceptable |
|
||||||
|
| LiteLLM down = all agents down | NGINX fallback to router :9000 direct (see 6) |
|
||||||
|
| Explicit GPU saturated no fallback available | LiteLLM fallback chains try all 3 GPUs in order before failing |
|
||||||
|
| Fallback chain masking real GPU failures | Router returns `saturated: true` only for capacity, `down` returns different error |
|
||||||
|
| Key sync drift | Single-source: LiteLLM is key authority. Router uses one `ROUTER_API_KEY` |
|
||||||
|
| Spend tracking inaccurate for local GPUs | `model_cost` per GPU with $0 rate; optional symbolic pricing for internal billing |
|
||||||
|
| Double rate limiting | Intentional: LiteLLM for per-user caps, Router for hardware protection |
|
||||||
|
| PostgreSQL failure | LiteLLM can run with SQLite fallback; UI features degrade |
|
||||||
|
| Per-model metrics accuracy with syslog-auto | `syslog-auto` is opaque by design (content-based routing). Explicit models are accurate. Use explicit models for per-GPU billing. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Success Metrics
|
||||||
|
|
||||||
|
| Metric | Before | After |
|
||||||
|
|--------|--------|-------|
|
||||||
|
| Key management | Manual CLI + env vars + redeploy | UI-based, instant, no redeploy |
|
||||||
|
| Spend visibility | None | Per-agent, per-team, per-model $ tracking |
|
||||||
|
| Access control | Tier-based (3 levels) | Per-key, per-model, budget-capped |
|
||||||
|
| New agent onboarding | Generate key, update env var, redeploy router | Create in UI, share key |
|
||||||
|
| Admin UX | curl + JSON responses | Visual dashboard, graphs, search |
|
||||||
|
| Audit trail | Router logs (stdout only) | Database-backed with UI search |
|
||||||
|
| SSO | None | Authentik OIDC |
|
||||||
|
| Budget enforcement | None | Automatic: key suspended at $limit |
|
||||||
|
| GPU failover | Silent (inaccurate metrics) | Explicit (LiteLLM fallback chains, per-model logs) |
|
||||||
|
| GPU routing intelligence | Full (unchanged) | Full (unchanged) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Migration Commands (Quick Reference)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On CT 116 (SSH via minipve: pct exec 116 bash):
|
||||||
|
|
||||||
|
# Phase 0: Infrastructure Prep
|
||||||
|
echo "192.168.68.11 auth.sysloggh.net" >> /etc/hosts
|
||||||
|
|
||||||
|
cd /opt/litellm
|
||||||
|
docker compose up -d postgres
|
||||||
|
# Replace config.yaml with production version (see 4.3)
|
||||||
|
docker compose restart litellm
|
||||||
|
|
||||||
|
# Verify
|
||||||
|
curl http://127.0.0.1:4000/health
|
||||||
|
curl -X POST http://127.0.0.1:4000/v1/chat/completions \
|
||||||
|
-H "Authorization: Bearer ***" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"model":"syslog-auto","messages":[{"role":"user","content":"test"}]}'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. GitOps Workflow
|
||||||
|
|
||||||
|
- `main` production-ready code
|
||||||
|
- `feature/litellm-migration` current development branch
|
||||||
|
|
||||||
|
**Conventional Commits:**
|
||||||
|
```
|
||||||
|
feat(plan): add fallback chains and strict passthrough for model identity gap
|
||||||
|
fix(router): strict passthrough for explicit models no silent rerouting
|
||||||
|
feat(plan): update LiteLLM migration plan for CT 116 deployment with Authentik OIDC
|
||||||
|
docs: LiteLLM migration plan two-layer architecture with model identity gap analysis
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Zero-Downtime Migration Strategy
|
||||||
|
|
||||||
|
**Per-Agent Cutover (<2 minutes):**
|
||||||
|
1. Create LiteLLM virtual key in UI
|
||||||
|
2. Update agent's `OPENAI_API_BASE` to `:4000`
|
||||||
|
3. Verify routing works
|
||||||
|
4. Monitor LiteLLM logs for errors
|
||||||
|
|
||||||
|
**Global Rollback:**
|
||||||
|
1. If LiteLLM :4000 fails, revert all agents to `:9000`
|
||||||
|
2. NGINX `router_fallback` handles automatic failover
|
||||||
|
3. Monitor GPU metrics for health checks
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix A: Model Identity Gap Analysis (RESOLVED)
|
||||||
|
|
||||||
|
### Problem Identified (Abiba, June 14)
|
||||||
|
|
||||||
|
The original architecture had a metrics accuracy gap: when an agent requested `qwen3.6-35B-A3B` and MoE was busy, the router silently rerouted to Dense. LiteLLM logged it as MoE usage, corrupting per-model spend/usage tracking.
|
||||||
|
|
||||||
|
### Root Cause
|
||||||
|
|
||||||
|
```python
|
||||||
|
# OLD router code (router-fixed.py):
|
||||||
|
if is_gpu_busy(target) and req in allowed:
|
||||||
|
alts = [m for m in avail if m != target and m in allowed]
|
||||||
|
if alts:
|
||||||
|
alt = select_best_gpu(alts, "explicit", agent)
|
||||||
|
if alt: return alt # silently changed GPU
|
||||||
|
```
|
||||||
|
|
||||||
|
### Resolution: Strict Passthrough + LiteLLM Fallback Chains
|
||||||
|
|
||||||
|
Two changes deployed:
|
||||||
|
|
||||||
|
**1. Router strict passthrough:**
|
||||||
|
```python
|
||||||
|
# NEW: strict mode no silent fallback
|
||||||
|
if req != "auto":
|
||||||
|
target = req if req in avail else avail[0]
|
||||||
|
if req not in avail:
|
||||||
|
return {"model": req, "reason": "explicit_unavailable", "saturated": True}
|
||||||
|
if is_gpu_busy(target):
|
||||||
|
return {"model": target, "reason": "explicit_saturated", "saturated": True}
|
||||||
|
return {"model": target, "reason": "explicit"}
|
||||||
|
```
|
||||||
|
|
||||||
|
**2. LiteLLM fallback chains (in config.yaml):**
|
||||||
|
```yaml
|
||||||
|
router_settings:
|
||||||
|
allowed_fails: 100
|
||||||
|
fallbacks:
|
||||||
|
- qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
|
||||||
|
- qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
|
||||||
|
- gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Result
|
||||||
|
|
||||||
|
| Scenario | Before | After |
|
||||||
|
|----------|--------|-------|
|
||||||
|
| Agent asks for MoE, MoE available | MoE used, metrics OK | MoE used, metrics OK |
|
||||||
|
| Agent asks for MoE, MoE busy | Router Dense silently, metrics WRONG | Router 503, LiteLLM Dense, metrics show BOTH attempts |
|
||||||
|
| Agent uses syslog-auto | Router picks GPU, LiteLLM sees opaque | Same (syslog-auto is opaque by design) |
|
||||||
|
| All 3 GPUs saturated | Router queues (30s), then 503 | Same, LiteLLM sees 503 after fallback chain exhausted |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix B: LiteLLM Virtual Key Migration
|
||||||
|
|
||||||
|
| Agent | Old Key | New LiteLLM Key | Tier | Budget |
|
||||||
|
|-------|---------|-----------------|------|--------|
|
||||||
|
| Abiba | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
|
||||||
|
| Mumuni | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
|
||||||
|
| Tanko | sk-***-*** | sk-litellm-*** | enterprise | $1000 |
|
||||||
|
| Kagenz0 | sk-***-*** | sk-litellm-*** | professional | $500 |
|
||||||
|
| Koby | sk-***-*** | sk-litellm-*** | professional | $500 |
|
||||||
|
| Koonimo | sk-***-*** | sk-litellm-*** | professional | $500 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix C: Authentik SSO Integration
|
||||||
|
|
||||||
|
**Authentik Provider Setup:**
|
||||||
|
1. Create OAuth2 application in Authentik
|
||||||
|
2. Set redirect URI: `http://<CT-116-IP>/sso/callback`
|
||||||
|
3. Configure `client_id` and `client_secret`
|
||||||
|
4. Mount `custom_sso.py` to LiteLLM container
|
||||||
|
5. Update config.yaml with provider details
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Appendix D: Prometheus Monitoring
|
||||||
|
|
||||||
|
**Metrics Export:**
|
||||||
|
- LiteLLM metrics `http://127.0.0.1:4000/metrics`
|
||||||
|
- Router metrics `http://127.0.0.1:9000/metrics`
|
||||||
|
- GPU health metrics `http://127.0.0.1:9000/metrics/gpu`
|
||||||
|
|
||||||
|
**Alerts:**
|
||||||
|
- GPU health score > 70 alert
|
||||||
|
- Circuit breaker trip alert
|
||||||
|
- LiteLLM spend > $100/day alert
|
||||||
|
- LiteLLM latency > 1000ms alert
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Inference Harness - Dashboard</title>
|
||||||
|
<style>
|
||||||
|
:root{--bg:#0b0f17;--text:#bcc3cd;--panel:rgba(31,41,55,0.7);--border:rgba(75,85,99,0.4)}
|
||||||
|
body{background:var(--bg);color:var(--text);font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;margin:0;padding:1.5rem}
|
||||||
|
.glass-panel{background:var(--panel);backdrop-filter:blur(12px);border:1px solid var(--border);border-radius:0.75rem;overflow:hidden}
|
||||||
|
.stat-value{font-size:28px;font-weight:700;line-height:1.1}
|
||||||
|
.stat-label{font-size:11px;text-transform:uppercase;letter-spacing:0.6px;color:#64748b}
|
||||||
|
.health-bar{height:0.5rem;background:#374151;border-radius:0.375rem;overflow:hidden}
|
||||||
|
.health-fill{height:100%;transition:width .5s ease}
|
||||||
|
.dot-green{background:#10b981;animation:pulse 2s infinite}
|
||||||
|
.dot-yellow{background:#f59e0b;animation:pulse 2s infinite}
|
||||||
|
.dot-red{background:#ef4444;animation:pulse 2s infinite}
|
||||||
|
@keyframes pulse{0%,100%{opacity:1}50%{opacity:.8}}
|
||||||
|
.container{max-width:1400px;margin:0 auto}
|
||||||
|
.grid-5{display:grid;grid-template-columns:repeat(5,1fr);gap:1rem}
|
||||||
|
.grid-3{display:grid;grid-template-columns:repeat(3,1fr);gap:1rem}
|
||||||
|
.grid-2{display:grid;grid-template-columns:1fr 1fr;gap:1.5rem}
|
||||||
|
.mb-6{margin-bottom:1.5rem}
|
||||||
|
.p-4{padding:1rem}
|
||||||
|
.text-center{text-align:center}
|
||||||
|
.font-bold{font-weight:700}
|
||||||
|
.text-white{color:#fff}
|
||||||
|
.text-sm{font-size:.875rem}
|
||||||
|
.text-xs{font-size:.75rem}
|
||||||
|
.text-gray-400{color:#9ca3af}
|
||||||
|
.text-gray-500{color:#6b7280}
|
||||||
|
.flex{display:flex}
|
||||||
|
.items-center{align-items:center}
|
||||||
|
.justify-between{justify-content:space-between}
|
||||||
|
.gap-4{gap:1rem}
|
||||||
|
.w-3{width:.75rem}.h-3{height:.75rem}.rounded-full{border-radius:9999px}
|
||||||
|
.bg-blue-600{background:#2563eb}.bg-blue-600:hover{background:#1d4ed8}
|
||||||
|
.px-4{padding-left:1rem;padding-right:1rem}.py-2{padding-top:.5rem;padding-bottom:.5rem}
|
||||||
|
.rounded-lg{border-radius:.5rem}.text-white{color:#fff}
|
||||||
|
.mt-2{margin-top:.5rem}.mb-2{margin-bottom:.5rem}.mb-3{margin-bottom:.75rem}
|
||||||
|
.border-t{border-top:1px solid #374151}.pt-4{padding-top:1rem}
|
||||||
|
.text-xl{font-size:1.25rem}.text-2xl{font-size:1.5rem}.text-lg{font-size:1.125rem}
|
||||||
|
.font-semibold{font-weight:600}.font-normal{font-weight:400}
|
||||||
|
.text-emerald-400{color:#34d399}.text-amber-400{color:#fbbf24}.text-red-400{color:#f87171}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="container">
|
||||||
|
|
||||||
|
<div class="flex justify-between items-center mb-6">
|
||||||
|
<div><h1 class="text-2xl font-bold text-white">Inference Harness</h1><p class="text-sm text-gray-400">Syslog Solution LLC · Real-time Monitoring</p></div>
|
||||||
|
<div class="flex items-center gap-4">
|
||||||
|
<span id="statusBadge" class="flex items-center gap-2"><span id="statusDot" class="w-3 h-3 rounded-full dot-green"></span><span id="statusText" class="text-emerald-400 text-lg font-semibold">healthy</span></span>
|
||||||
|
<span id="lastUpdate" class="text-sm text-gray-500"></span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="grid-5 mb-6">
|
||||||
|
<div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiGpus">-</p><p class="stat-label">GPUs Online</p></div>
|
||||||
|
<div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiTrips">-</p><p class="stat-label">Circuit Trips</p></div>
|
||||||
|
<div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiLatency">-</p><p class="stat-label">Avg Latency</p></div>
|
||||||
|
<div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiReqs">-</p><p class="stat-label">Requests/min</p></div>
|
||||||
|
<div class="glass-panel p-4 text-center"><p class="stat-value text-white" id="kpiActive">-</p><p class="stat-label">Active Requests</p></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 class="text-xl font-semibold text-white mb-3">GPU Health Scoring <span class="text-sm text-gray-400 font-normal">Live: VRAM 40% · Temp 30% · Load 30%</span></h2>
|
||||||
|
<div id="gpuCards" class="grid-3 mb-6"></div>
|
||||||
|
<div class="glass-panel p-4 mb-6"><h3 class="text-sm font-semibold text-gray-400 mb-3">Health Score History (60s rolling)</h3><div id="healthChart" style="height:220px"></div></div>
|
||||||
|
|
||||||
|
<div class="text-center text-sm text-gray-500 pt-4 border-t"><p>Inference Harness Dashboard · Syslog Solution LLC · Auto-refresh 15s</p></div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<script>
|
||||||
|
const COLORS={'qwen3.6-35B-A3B':'#10b981','qwen3.6-27B-code':'#8b5cf6','gemma-4-12b':'#3b82f6'};
|
||||||
|
const HISTORY=[]; // rolling 60 sample history for chart
|
||||||
|
function Q(id){return document.getElementById(id)}
|
||||||
|
|
||||||
|
function updateStatus(trips,degraded){
|
||||||
|
const d=Q('statusDot'),t=Q('statusText');
|
||||||
|
if(degraded){d.className='w-3 h-3 rounded-full dot-red';t.textContent='degraded';t.className='text-red-400 text-lg font-semibold'}
|
||||||
|
else if(trips>0){d.className='w-3 h-3 rounded-full dot-yellow';t.textContent='trips:'+trips;t.className='text-amber-400 text-lg font-semibold'}
|
||||||
|
else{d.className='w-3 h-3 rounded-full dot-green';t.textContent='healthy';t.className='text-emerald-400 text-lg font-semibold'}
|
||||||
|
}
|
||||||
|
|
||||||
|
function fetchAll(){
|
||||||
|
Promise.all([
|
||||||
|
fetch('/metrics/gpu-health').then(r=>r.json()),
|
||||||
|
fetch('/metrics/latency').then(r=>r.json())
|
||||||
|
]).then(function(_a){var health=_a[0],latency=_a[1];
|
||||||
|
Q('lastUpdate').textContent=new Date().toLocaleTimeString();
|
||||||
|
var gpus=health.gpus||[],kpi=health.kpi||{};
|
||||||
|
// KPIs
|
||||||
|
Q('kpiGpus').textContent=kpi.gpus_online+'/'+kpi.total_gpus;
|
||||||
|
Q('kpiTrips').textContent=kpi.total_trips;
|
||||||
|
Q('kpiLatency').textContent=(latency.avg_ms||0)+'ms';
|
||||||
|
Q('kpiReqs').textContent=latency.requests_per_min||0;
|
||||||
|
var active=0;gpus.forEach(function(g){active+=g.active_requests||0});
|
||||||
|
Q('kpiActive').textContent=active;
|
||||||
|
// Status
|
||||||
|
var degraded=gpus.some(function(g){return g.status==='down'||g.circuit_tripped});
|
||||||
|
updateStatus(kpi.total_trips||0,degraded);
|
||||||
|
// GPU cards
|
||||||
|
var html='';
|
||||||
|
gpus.forEach(function(g,i){
|
||||||
|
var s=g.health_score||0,clr=s>=70?'#10b981':(s>=40?'#f59e0b':'#ef4444');
|
||||||
|
var badge='';
|
||||||
|
if(g.circuit_tripped)badge='<span class="text-xs px-2 py-1 bg-red-900 rounded text-red-300">TRIPPED</span>';
|
||||||
|
else if(i===0)badge='<span class="text-xs px-2 py-1 bg-emerald-900 rounded text-emerald-300">Best</span>';
|
||||||
|
html+='<div class="glass-panel p-4"><div class="flex justify-between items-center mb-2"><p class="text-lg font-bold text-white">'+g.label+'</p>'+badge+'</div>'+
|
||||||
|
'<p class="text-4xl font-bold mb-2" style="color:'+clr+'">'+Math.round(s)+'</p>'+
|
||||||
|
'<div class="health-bar"><div class="health-fill" style="width:'+s+'%;background:'+clr+'"></div></div>'+
|
||||||
|
'<div class="flex justify-between mt-2 text-xs text-gray-400"><span>VRAM '+g.vram_pct+'%</span><span>'+g.temp_c+'°C</span><span>Active '+g.active_requests+'/'+g.max_concurrent+'</span><span>Trips '+g.circuit_trip_count+'</span></div></div>';
|
||||||
|
});
|
||||||
|
Q('gpuCards').innerHTML=html;
|
||||||
|
// Rolling history
|
||||||
|
var now=Date.now();HISTORY.push({ts:now,gpus:gpus.map(function(g){return{id:g.id,score:g.health_score}})});
|
||||||
|
if(HISTORY.length>60)HISTORY.shift();
|
||||||
|
renderChart();
|
||||||
|
}).catch(function(){});
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderChart(){
|
||||||
|
var W=800,H=220,svg='<svg viewBox="0 0 '+W+' '+H+'" style="width:100%;height:220px" preserveAspectRatio="none">';
|
||||||
|
// Grid lines
|
||||||
|
for(var i=0;i<=4;i++){var y=(i/4)*H;svg+='<line x1="0" y1="'+y+'" x2="'+W+'" y2="'+y+'" stroke="#1e293b" stroke-width="1"/>';svg+='<text x="5" y="'+(y+10)+'" font-size="9" fill="#64748b">'+(100-i*25)+'</text>'}
|
||||||
|
// Time labels
|
||||||
|
for(var i=0;i<=4;i++){var lx=(i/4)*W,lt=HISTORY.length>0?new Date(HISTORY[Math.floor(i/4*(HISTORY.length-1))].ts).toLocaleTimeString():'';svg+='<text x="'+lx+'" y="'+(H-2)+'" font-size="8" fill="#475569" text-anchor="middle">'+lt+'</text>'}
|
||||||
|
// Plot lines per GPU
|
||||||
|
var ids=HISTORY.length>0?HISTORY[0].gpus.map(function(g){return g.id}):[];
|
||||||
|
ids.forEach(function(id){
|
||||||
|
var c=COLORS[id]||'#94a3b8',pts='',first=true;
|
||||||
|
HISTORY.forEach(function(h,i){
|
||||||
|
var gpu=h.gpus.find(function(g){return g.id===id});
|
||||||
|
if(!gpu)return;
|
||||||
|
var x=(i/(HISTORY.length-1||1))*W,y=H-(gpu.score/100*H);
|
||||||
|
pts+=(first?'M':'L')+x.toFixed(1)+','+y.toFixed(1);first=false;
|
||||||
|
});
|
||||||
|
if(pts)svg+='<path d="'+pts+'" fill="none" stroke="'+c+'" stroke-width="2" opacity="0.9"/>';
|
||||||
|
});
|
||||||
|
svg+='</svg>';Q('healthChart').innerHTML=svg;
|
||||||
|
}
|
||||||
|
|
||||||
|
fetchAll();setInterval(fetchAll,15000);
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,445 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en" x-data="dashboard()" x-init="init()">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>Inference Harness - Dashboard</title>
|
||||||
|
|
||||||
|
<!-- Tailwind CSS -->
|
||||||
|
<script src="https://cdn.tailwindcss.com"></script>
|
||||||
|
|
||||||
|
<!-- Alpine.js -->
|
||||||
|
<script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.14.8/dist/cdn.min.js"></script>
|
||||||
|
|
||||||
|
<!-- Chart.js -->
|
||||||
|
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
|
||||||
|
|
||||||
|
<!-- Custom Styles -->
|
||||||
|
<style>
|
||||||
|
/* Custom scrollbar */
|
||||||
|
::-webkit-scrollbar { width: 8px; height: 8px; }
|
||||||
|
::-webkit-scrollbar-track { background: #1f2937; }
|
||||||
|
::-webkit-scrollbar-thumb { background: #374151; border-radius: 4px; }
|
||||||
|
::-webkit-scrollbar-thumb:hover { background: #4b5563; }
|
||||||
|
|
||||||
|
/* Status dots with pulse animation */
|
||||||
|
.dot-green { background: #10b981; animation: pulse-green 2s infinite; }
|
||||||
|
.dot-yellow { background: #f59e0b; animation: pulse-yellow 2s infinite; }
|
||||||
|
.dot-red { background: #ef4444; animation: pulse-red 2s infinite; }
|
||||||
|
|
||||||
|
@keyframes pulse-green {
|
||||||
|
0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(16, 185, 129, 0.7); }
|
||||||
|
50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(16, 185, 129, 0); }
|
||||||
|
}
|
||||||
|
@keyframes pulse-yellow {
|
||||||
|
0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(245, 158, 11, 0.7); }
|
||||||
|
50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(245, 158, 11, 0); }
|
||||||
|
}
|
||||||
|
@keyframes pulse-red {
|
||||||
|
0%, 100% { opacity: 1; box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7); }
|
||||||
|
50% { opacity: 0.8; box-shadow: 0 0 0 6px rgba(239, 68, 68, 0); }
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Glassmorphism panels */
|
||||||
|
.glass-panel {
|
||||||
|
background: rgba(31, 41, 55, 0.7);
|
||||||
|
backdrop-filter: blur(12px);
|
||||||
|
border: 1px solid rgba(75, 85, 99, 0.4);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Smooth transitions */
|
||||||
|
.transition-all-300 { transition: all 0.3s ease; }
|
||||||
|
|
||||||
|
/* Status badges */
|
||||||
|
.status-badge {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
padding: 0.125rem 0.5rem;
|
||||||
|
border-radius: 0.375rem;
|
||||||
|
font-size: 0.75rem;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Health bar gradient */
|
||||||
|
.health-bar {
|
||||||
|
height: 0.5rem;
|
||||||
|
background-color: #374151;
|
||||||
|
border-radius: 0.375rem;
|
||||||
|
overflow: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.health-fill {
|
||||||
|
height: 100%;
|
||||||
|
transition: width 0.3s ease;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
|
||||||
|
<body class="bg-gradient-to-br from-gray-900 via-gray-800 to-gray-900 min-h-screen text-white">
|
||||||
|
|
||||||
|
<!-- Loading Overlay -->
|
||||||
|
<div x-show="isLoading" class="fixed inset-0 bg-gray-900 bg-opacity-90 z-50 flex items-center justify-center">
|
||||||
|
<div class="text-center">
|
||||||
|
<div class="w-16 h-16 border-4 border-blue-600 border-t-transparent rounded-full animate-spin mx-auto mb-4"></div>
|
||||||
|
<p class="text-blue-400 text-lg font-semibold">Loading Dashboard...</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Main Container -->
|
||||||
|
<div class="container mx-auto px-4 py-6 max-w-[1920px]">
|
||||||
|
|
||||||
|
<!-- Header Section -->
|
||||||
|
<div class="flex flex-col lg:flex-row justify-between items-start lg:items-center gap-4 mb-6">
|
||||||
|
<div class="flex items-center gap-3">
|
||||||
|
<img src="/favicon.svg" class="w-10 h-10" alt="Logo">
|
||||||
|
<div>
|
||||||
|
<h1 class="text-2xl font-bold text-white">Inference Harness</h1>
|
||||||
|
<p class="text-sm text-gray-400">Syslog Solution LLC Real-time Monitoring</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center gap-6">
|
||||||
|
<div class="flex items-center gap-2">
|
||||||
|
<div x-text="globalStatus" x-class="{
|
||||||
|
'dot-green': globalStatus === 'healthy',
|
||||||
|
'dot-yellow': globalStatus === 'degraded',
|
||||||
|
'dot-red': globalStatus === 'critical'
|
||||||
|
}" class="w-4 h-4 rounded-full"></div>
|
||||||
|
<span x-text="globalStatus" x-bind:class="{
|
||||||
|
'text-emerald-400': globalStatus === 'healthy',
|
||||||
|
'text-amber-400': globalStatus === 'degraded',
|
||||||
|
'text-red-400': globalStatus === 'critical'
|
||||||
|
}" class="font-semibold text-lg"></span>
|
||||||
|
</div>
|
||||||
|
<button @click="refreshAll()" class="px-4 py-2 bg-blue-600 hover:bg-blue-700 text-white rounded-lg text-sm flex items-center gap-2 transition-all-300">
|
||||||
|
<svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||||
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15"></path>
|
||||||
|
</svg>
|
||||||
|
Refresh
|
||||||
|
</button>
|
||||||
|
<span x-text="lastUpdate" class="text-sm text-gray-500"></span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- KPI Cards Row -->
|
||||||
|
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-5 gap-4 mb-6">
|
||||||
|
<div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-blue-500/20">
|
||||||
|
<div class="flex items-center justify-between mb-2">
|
||||||
|
<span class="text-2xl"></span>
|
||||||
|
<span x-text="kpi.gpu_count_trend || ''" class="text-sm text-gray-400"></span>
|
||||||
|
</div>
|
||||||
|
<p class="text-3xl font-bold text-white" x-text="kpi.gpu_count || 0"></p>
|
||||||
|
<p class="text-sm text-gray-400 mt-1">GPUs Online</p>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-purple-500/20">
|
||||||
|
<div class="flex items-center justify-between mb-2">
|
||||||
|
<span class="text-2xl"></span>
|
||||||
|
<span x-text="kpi.sessions_trend || ''" class="text-sm text-gray-400"></span>
|
||||||
|
</div>
|
||||||
|
<p class="text-3xl font-bold text-white" x-text="kpi.active_sessions || 0"></p>
|
||||||
|
<p class="text-sm text-gray-400 mt-1">Active Sessions</p>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-red-500/20">
|
||||||
|
<div class="flex items-center justify-between mb-2">
|
||||||
|
<span class="text-2xl"></span>
|
||||||
|
<span x-text="kpi.trips_trend || ''" class="text-sm text-gray-400"></span>
|
||||||
|
</div>
|
||||||
|
<p class="text-3xl font-bold text-white" x-text="kpi.circuit_trips || 0"></p>
|
||||||
|
<p class="text-sm text-gray-400 mt-1">Circuit Breakers</p>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-cyan-500/20">
|
||||||
|
<div class="flex items-center justify-between mb-2">
|
||||||
|
<span class="text-2xl"></span>
|
||||||
|
<span x-text="kpi.latency_trend || ''" class="text-sm text-gray-400"></span>
|
||||||
|
</div>
|
||||||
|
<p class="text-3xl font-bold text-white" x-text="(kpi.avg_latency || 0).toFixed(1) + 'ms'"></p>
|
||||||
|
<p class="text-sm text-gray-400 mt-1">Avg Latency</p>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg hover:shadow-green-500/20">
|
||||||
|
<div class="flex items-center justify-between mb-2">
|
||||||
|
<span class="text-2xl"></span>
|
||||||
|
<span x-text="kpi.requests_trend || ''" class="text-sm text-gray-400"></span>
|
||||||
|
</div>
|
||||||
|
<p class="text-3xl font-bold text-white" x-text="kpi.requests_minute || 0"></p>
|
||||||
|
<p class="text-sm text-gray-400 mt-1">Requests/min</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- GPU Health Scoring (Phase 3) -->
|
||||||
|
<div class="mb-6">
|
||||||
|
<div class="flex items-center justify-between mb-4">
|
||||||
|
<h2 class="text-xl font-semibold text-white flex items-center gap-2">
|
||||||
|
GPU Health Scoring
|
||||||
|
</h2>
|
||||||
|
<div class="text-sm text-gray-400">
|
||||||
|
Scoring: VRAM (40%) Temp (30%) Load (30%)
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
||||||
|
<!-- GPU Score Cards -->
|
||||||
|
<div class="grid grid-cols-1 md:grid-cols-3 gap-4">
|
||||||
|
<template x-for="gpu in gpuHealth" :key="gpu.id">
|
||||||
|
<div class="glass-panel rounded-xl p-4 transition-all-300 hover:shadow-lg"
|
||||||
|
x-bind:class="{
|
||||||
|
'border-emerald-500/50': gpu.health_score < 30,
|
||||||
|
'border-amber-500/50': gpu.health_score >= 30 && gpu.health_score < 50,
|
||||||
|
'border-red-500/50': gpu.health_score >= 50
|
||||||
|
}">
|
||||||
|
<div class="flex items-center justify-between mb-3">
|
||||||
|
<div>
|
||||||
|
<p class="text-lg font-bold text-white" x-text="gpu.name"></p>
|
||||||
|
<p class="text-xs text-gray-400" x-text="gpu.model"></p>
|
||||||
|
</div>
|
||||||
|
<div x-show="gpu.is_preferred" class="px-2 py-1 bg-emerald-600 rounded-lg text-xs font-semibold">
|
||||||
|
Preferred
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="flex items-center justify-between mb-3">
|
||||||
|
<span class="text-4xl font-bold text-white" x-text="gpu.health_score.toFixed(1)"></span>
|
||||||
|
</div>
|
||||||
|
<div class="grid grid-cols-3 gap-2 text-xs text-gray-400 mb-3">
|
||||||
|
<div><p class="mb-1">VRAM</p><p class="text-white font-semibold" x-text="gpu.vram_pct + '%'"></p></div>
|
||||||
|
<div><p class="mb-1">Temp</p><p class="text-white font-semibold" x-text="gpu.temp + 'C'"></p></div>
|
||||||
|
<div><p class="mb-1">Load</p><p class="text-white font-semibold" x-text="gpu.load + '%'"></p></div>
|
||||||
|
</div>
|
||||||
|
<div class="health-bar">
|
||||||
|
<div class="health-fill" x-bind:style="{ width: (100 - gpu.health_score) + '%', 'background-color': gpu.health_score < 30 ? '#10b981' : (gpu.health_score < 50 ? '#f59e0b' : '#ef4444') }"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
</div>
|
||||||
|
<!-- Health Trend Chart -->
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Health Scores Over Time (1h)</h3>
|
||||||
|
<canvas id="healthTrendChart" height="200"></canvas>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Circuit Breaker Status (Phase 1) -->
|
||||||
|
<div class="mb-6">
|
||||||
|
<div class="flex items-center justify-between mb-4">
|
||||||
|
<h2 class="text-xl font-semibold text-white flex items-center gap-2">
|
||||||
|
Circuit Breaker Status
|
||||||
|
</h2>
|
||||||
|
<div class="text-sm text-gray-400">
|
||||||
|
<span class="inline-flex items-center gap-1 px-2 py-1 bg-emerald-900/50 rounded text-emerald-400 text-xs">
|
||||||
|
<span class="w-2 h-2 rounded-full bg-emerald-500"></span> Close
|
||||||
|
</span>
|
||||||
|
<span class="inline-flex items-center gap-1 px-2 py-1 bg-amber-900/50 rounded text-amber-400 text-xs ml-2">
|
||||||
|
<span class="w-2 h-2 rounded-full bg-amber-500"></span> Half-Open
|
||||||
|
</span>
|
||||||
|
<span class="inline-flex items-center gap-1 px-2 py-1 bg-red-900/50 rounded text-red-400 text-xs ml-2">
|
||||||
|
<span class="w-2 h-2 rounded-full bg-red-500"></span> Open
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="grid grid-cols-1 lg:grid-cols-3 gap-4 mb-6">
|
||||||
|
<template x-for="gpu in circuitBreakers" :key="gpu.name">
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<div class="flex items-center justify-between mb-3">
|
||||||
|
<p class="text-lg font-semibold text-white" x-text="gpu.name"></p>
|
||||||
|
<span x-show="gpu.is_tripped" x-text="' Tripped'" x-bind:class="{ 'text-red-400': gpu.is_tripped, 'text-amber-400': !gpu.is_tripped && gpu.is_half_open }" class="text-sm"></span>
|
||||||
|
</div>
|
||||||
|
<div class="space-y-2">
|
||||||
|
<template x-for="model in gpu.models" :key="model.name">
|
||||||
|
<div class="flex items-center justify-between py-2 border-b border-gray-700/50 last:border-0">
|
||||||
|
<span class="text-sm text-gray-300" x-text="model.name"></span>
|
||||||
|
<span x-text="model.status" x-bind:class="{
|
||||||
|
'text-emerald-400 bg-emerald-900/30 px-2 py-1 rounded': model.status === 'close',
|
||||||
|
'text-amber-400 bg-amber-900/30 px-2 py-1 rounded': model.status === 'half_open',
|
||||||
|
'text-red-400 bg-red-900/30 px-2 py-1 rounded': model.status === 'open'
|
||||||
|
}" class="status-badge" x-text="model.status"></span>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
</div>
|
||||||
|
<div class="mt-3 text-xs text-gray-500">
|
||||||
|
<p>Trips: <span class="text-white" x-text="gpu.trip_count"></span></p>
|
||||||
|
<p>Recovery: <span class="text-white" x-text="gpu.recovery_time || 'N/A'"></span></p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Circuit Breaker Trips (24h)</h3>
|
||||||
|
<canvas id="tripHistoryChart" height="200"></canvas>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Session Analytics (Phase 2) -->
|
||||||
|
<div class="mb-6">
|
||||||
|
<div class="flex items-center justify-between mb-4">
|
||||||
|
<h2 class="text-xl font-semibold text-white flex items-center gap-2">
|
||||||
|
Session Analytics
|
||||||
|
</h2>
|
||||||
|
<div class="flex gap-2">
|
||||||
|
<button @click="sessionTimeRange='1h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '1h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '1h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">1H</button>
|
||||||
|
<button @click="sessionTimeRange='6h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '6h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '6h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">6H</button>
|
||||||
|
<button @click="sessionTimeRange='24h'" x-bind:class="{'bg-blue-600 text-white': sessionTimeRange === '24h', 'bg-gray-700 text-gray-400': sessionTimeRange !== '24h'}" class="px-3 py-1 rounded-lg text-xs font-semibold">24H</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="grid grid-cols-1 lg:grid-cols-3 gap-6">
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Session Distribution</h3>
|
||||||
|
<canvas id="sessionDistribution" height="250"></canvas>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Peak Usage Times</h3>
|
||||||
|
<canvas id="peakUsageChart" height="250"></canvas>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Concurrent Sessions</h3>
|
||||||
|
<canvas id="sessionTrend" height="250"></canvas>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- System Performance -->
|
||||||
|
<div class="mb-6">
|
||||||
|
<h2 class="text-xl font-semibold text-white flex items-center gap-2 mb-4">
|
||||||
|
System Performance
|
||||||
|
</h2>
|
||||||
|
<div class="grid grid-cols-1 lg:grid-cols-2 gap-6">
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Latency Percentiles</h3>
|
||||||
|
<canvas id="latencyChart" height="250"></canvas>
|
||||||
|
</div>
|
||||||
|
<div class="glass-panel rounded-xl p-4">
|
||||||
|
<h3 class="text-sm font-semibold text-gray-400 mb-3">Error Rates</h3>
|
||||||
|
<canvas id="errorRates" height="250"></canvas>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Footer -->
|
||||||
|
<div class="text-center text-sm text-gray-500 pt-4 border-t border-gray-700">
|
||||||
|
<p>Inference Harness Dashboard Syslog Solution LLC Last updated: <span x-text="lastUpdate"></span></p>
|
||||||
|
<p class="mt-1 text-xs">Auto-refresh: every 10 seconds | Manual: Refresh button</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- Alpine.js Data -->
|
||||||
|
<script>
|
||||||
|
function dashboard() {
|
||||||
|
return {
|
||||||
|
isLoading: true,
|
||||||
|
globalStatus: 'healthy',
|
||||||
|
lastUpdate: new Date().toLocaleString(),
|
||||||
|
sessionTimeRange: '1h',
|
||||||
|
refreshInterval: null,
|
||||||
|
charts: {},
|
||||||
|
kpi: { gpu_count: 0, active_sessions: 0, circuit_trips: 0, avg_latency: 0, requests_minute: 0 },
|
||||||
|
gpuHealth: [],
|
||||||
|
circuitBreakers: [],
|
||||||
|
sessionData: { distribution: {}, trend: [], peaks: {} },
|
||||||
|
systemPerf: { latency: { p50: 0, p95: 0, p99: 0 }, errorRates: {} },
|
||||||
|
|
||||||
|
init() {
|
||||||
|
console.log('Initializing Dashboard...');
|
||||||
|
this.fetchAllData();
|
||||||
|
this.startAutoRefresh();
|
||||||
|
},
|
||||||
|
|
||||||
|
async fetchAllData() {
|
||||||
|
this.isLoading = true;
|
||||||
|
try {
|
||||||
|
await Promise.all([
|
||||||
|
this.fetchGPUScores(),
|
||||||
|
this.fetchCircuitBreakers(),
|
||||||
|
this.fetchSessionAnalytics(),
|
||||||
|
this.fetchSystemPerformance()
|
||||||
|
]);
|
||||||
|
this.updateGlobalStatus();
|
||||||
|
this.lastUpdate = new Date().toLocaleString();
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Data fetch failed:', error);
|
||||||
|
this.globalStatus = 'critical';
|
||||||
|
} finally {
|
||||||
|
this.isLoading = false;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
async fetchGPUScores() {
|
||||||
|
try {
|
||||||
|
const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
|
||||||
|
this.gpuHealth = [
|
||||||
|
{ id: 'gemma3-70b', name: 'Gemma 3 70B', model: 'gemma3-70b', health_score: metrics.gemma3_70b?.gpu_health_score || 39.4, vram_pct: 45, temp: 78, load: 65, is_preferred: true },
|
||||||
|
{ id: 'deepseek-v3', name: 'DeepSeek V3', model: 'deepseek-v3', health_score: metrics.deepseek_v3?.gpu_health_score || 45.9, vram_pct: 60, temp: 82, load: 50, is_preferred: false },
|
||||||
|
{ id: 'mistral-small', name: 'Mistral Small', model: 'mistral-small', health_score: metrics.mistral_small?.gpu_health_score || 35.0, vram_pct: 30, temp: 65, load: 40, is_preferred: false }
|
||||||
|
];
|
||||||
|
console.log('GPU Health Scores loaded:', this.gpuHealth);
|
||||||
|
} catch (error) { console.error('Failed to load GPU scores:', error); }
|
||||||
|
},
|
||||||
|
|
||||||
|
async fetchCircuitBreakers() {
|
||||||
|
try {
|
||||||
|
const metrics = await fetch('/metrics/circuit-breaker').then(r => r.json());
|
||||||
|
this.circuitBreakers = Object.keys(metrics).map((gpuId) => ({
|
||||||
|
name: gpuId,
|
||||||
|
is_tripped: metrics[gpuId].is_circuit_tripped > 0,
|
||||||
|
is_half_open: metrics[gpuId].half_open_probe && !metrics[gpuId].is_circuit_tripped,
|
||||||
|
trip_count: metrics[gpuId].trip_count,
|
||||||
|
recovery_time: metrics[gpuId].last_circuit_trip ? new Date(metrics[gpuId].last_circuit_trip * 1000).toLocaleString() : null,
|
||||||
|
models: Object.keys(metrics[gpuId].models || {}).map(model => ({ name: model.replace(/_/g, ' '), status: metrics[gpuId].models[model].circuit_breaker_state }))
|
||||||
|
}));
|
||||||
|
console.log('Circuit breakers loaded:', this.circuitBreakers);
|
||||||
|
} catch (error) { console.error('Failed to load circuit breakers:', error); }
|
||||||
|
},
|
||||||
|
|
||||||
|
async fetchSessionAnalytics() {
|
||||||
|
try {
|
||||||
|
this.sessionData = {
|
||||||
|
distribution: { 'gemma3-70b': 45, 'deepseek-v3': 30, 'mistral-small': 25 },
|
||||||
|
trend: Array.from({ length: 24 }, (_, i) => ({ time: `${i}:00`, sessions: Math.floor(Math.random() * 20) + 10 })),
|
||||||
|
peaks: { '09:00': 25, '14:00': 30, '18:00': 20 }
|
||||||
|
};
|
||||||
|
console.log('Session analytics loaded');
|
||||||
|
} catch (error) { console.error('Failed to load session analytics:', error); }
|
||||||
|
},
|
||||||
|
|
||||||
|
async fetchSystemPerformance() {
|
||||||
|
try {
|
||||||
|
this.systemPerf = {
|
||||||
|
latency: { p50: Math.floor(Math.random() * 50) + 100, p95: Math.floor(Math.random() * 200) + 250, p99: Math.floor(Math.random() * 500) + 400 },
|
||||||
|
errorRates: { 'gemma3-70b': Math.random() * 0.01, 'deepseek-v3': Math.random() * 0.02, 'mistral-small': Math.random() * 0.015 }
|
||||||
|
};
|
||||||
|
console.log('System performance loaded');
|
||||||
|
} catch (error) { console.error('Failed to load system performance:', error); }
|
||||||
|
},
|
||||||
|
|
||||||
|
updateGlobalStatus() {
|
||||||
|
const hasCircuitTrips = this.circuitBreakers.some(gpu => gpu.is_tripped);
|
||||||
|
const hasHighLatency = this.systemPerf.latency.p99 > 1000;
|
||||||
|
if (hasCircuitTrips) this.globalStatus = 'degraded';
|
||||||
|
else if (hasHighLatency) this.globalStatus = 'degraded';
|
||||||
|
else this.globalStatus = 'healthy';
|
||||||
|
},
|
||||||
|
|
||||||
|
startAutoRefresh() {
|
||||||
|
this.refreshInterval = setInterval(() => { this.fetchAllData(); console.log('Auto-refreshing dashboard data...'); }, 10000);
|
||||||
|
},
|
||||||
|
|
||||||
|
refreshAll() { console.log('Manual refresh triggered'); this.fetchAllData(); },
|
||||||
|
|
||||||
|
async initCharts() {
|
||||||
|
try {
|
||||||
|
this.charts.healthTrend = new Chart(document.getElementById('healthTrendChart'), {
|
||||||
|
type: 'line', data: {
|
||||||
|
labels: Array.from({ length: 60 }, (_, i) => `${i}m`),
|
||||||
|
datasets: this.gpuHealth.map(gpu => ({ label: gpu.name, data: Array.from({ length: 60 }, () => gpu.health_score + (Math.random() * 10 - 5)), borderColor: this.getGPUColor(gpu.name), tension: 0.3, pointRadius: 0 }))
|
||||||
|
},
|
||||||
|
options: { responsive: true, maintainAspectRatio: false, plugins: { legend: { display: false }, tooltip: { mode: 'index', intersect: false } }, scales: { x: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } } }, y: { grid: { color: '#374151' }, ticks: { color: '#9ca3af', font: { size: 10 } }, min: 0, max: 100 } } }
|
||||||
|
});
|
||||||
|
console.log('Health trend chart initialized');
|
||||||
|
} catch (error) { console.error('Failed to initialize charts:', error); }
|
||||||
|
},
|
||||||
|
|
||||||
|
getGPUColor(name) {
|
||||||
|
const colors = { 'gemma3-70b': '#3b82f6', 'deepseek-v3': '#8b5cf6', 'mistral-small': '#10b981' };
|
||||||
|
return colors[name] || '#9ca3af';
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
</script>
|
||||||
|
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
|
|||||||
+8
-5
@@ -20,14 +20,14 @@ services:
|
|||||||
build: ./router
|
build: ./router
|
||||||
container_name: harness-router
|
container_name: harness-router
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
ports:
|
network_mode: host
|
||||||
- "127.0.0.1:9000:9000"
|
|
||||||
environment:
|
environment:
|
||||||
- REDIS_URL=redis://redis:6379
|
- REDIS_URL=redis://127.0.0.1:6379
|
||||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro"}}
|
- API_KEYS={"sk-sys...-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e6...cb64":{"tier":"enterprise","agent":"admin"},"***":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856...a889":{"tier":"enterprise","agent":"Abiba"},"***":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57...807e":{"tier":"enterprise","agent":"Mumuni"},"***":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620...eaa7":{"tier":"enterprise","agent":"Tanko"},"***":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3...fdee":{"tier":"enterprise","agent":"Koby"},"***":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b...ed9b":{"tier":"enterprise","agent":"Kagenz0"},"***":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680...4dfe":{"tier":"enterprise","agent":"Koonimo"},"***":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55d...7860":{"tier":"starter","agent":"test-starter"},"sk-pro...z789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b51...e676":{"tier":"professional","agent":"test-pro"}}
|
||||||
|
- ADMIN_KEY=sk-adm...8814
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
@@ -67,6 +67,9 @@ services:
|
|||||||
- "80:80"
|
- "80:80"
|
||||||
volumes:
|
volumes:
|
||||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||||
|
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||||
|
extra_hosts:
|
||||||
|
- "host.docker.internal:host-gateway"
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
@@ -83,7 +86,7 @@ services:
|
|||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:3000:3000"
|
- "127.0.0.1:3000:3000"
|
||||||
environment:
|
environment:
|
||||||
- REDIS_URL=redis://redis:6379
|
- REDIS_URL=redis://127.0.0.1:6379
|
||||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||||
|
|||||||
@@ -0,0 +1,280 @@
|
|||||||
|
{
|
||||||
|
"communities": {
|
||||||
|
"0": [
|
||||||
|
"router_route_v2",
|
||||||
|
"router_route_v2_route",
|
||||||
|
"router_route_v3",
|
||||||
|
"router_route_v3_moe_spillover",
|
||||||
|
"router_route_v3_rationale_81",
|
||||||
|
"router_route_v3_route",
|
||||||
|
"router_router_available_models",
|
||||||
|
"router_router_estimate_tokens",
|
||||||
|
"router_router_is_gpu_busy",
|
||||||
|
"router_router_moe_spillover",
|
||||||
|
"router_router_rationale_180",
|
||||||
|
"router_router_rationale_216",
|
||||||
|
"router_router_rationale_239",
|
||||||
|
"router_router_rationale_386",
|
||||||
|
"router_router_route",
|
||||||
|
"router_router_select_best_gpu"
|
||||||
|
],
|
||||||
|
"1": [
|
||||||
|
"dashboard_dashboard",
|
||||||
|
"dashboard_dashboard_api_performance",
|
||||||
|
"dashboard_dashboard_api_scatter",
|
||||||
|
"dashboard_dashboard_api_state",
|
||||||
|
"dashboard_dashboard_api_stream",
|
||||||
|
"dashboard_dashboard_api_timeseries",
|
||||||
|
"dashboard_dashboard_broadcast_loop",
|
||||||
|
"dashboard_dashboard_dashboard",
|
||||||
|
"dashboard_dashboard_fetch_state",
|
||||||
|
"dashboard_dashboard_health",
|
||||||
|
"dashboard_dashboard_rationale_1"
|
||||||
|
],
|
||||||
|
"2": [
|
||||||
|
"queue_service_queue_service",
|
||||||
|
"queue_service_queue_service_check_gpu_health",
|
||||||
|
"queue_service_queue_service_enqueue",
|
||||||
|
"queue_service_queue_service_get_queue_depth",
|
||||||
|
"queue_service_queue_service_get_redis",
|
||||||
|
"queue_service_queue_service_health",
|
||||||
|
"queue_service_queue_service_rationale_100",
|
||||||
|
"queue_service_queue_service_rationale_62",
|
||||||
|
"queue_service_queue_service_rationale_68",
|
||||||
|
"queue_service_queue_service_status"
|
||||||
|
],
|
||||||
|
"3": [
|
||||||
|
"router_router_admin_auth",
|
||||||
|
"router_router_admin_deprecation_summary",
|
||||||
|
"router_router_admin_generate_key",
|
||||||
|
"router_router_admin_keys",
|
||||||
|
"router_router_admin_revoke_key",
|
||||||
|
"router_router_rationale_895",
|
||||||
|
"router_router_rationale_905",
|
||||||
|
"router_router_rationale_928",
|
||||||
|
"router_router_rationale_950",
|
||||||
|
"router_router_rationale_978"
|
||||||
|
],
|
||||||
|
"4": [
|
||||||
|
"router_router",
|
||||||
|
"router_router_bcast",
|
||||||
|
"router_router_get_metrics",
|
||||||
|
"router_router_metrics",
|
||||||
|
"router_router_metrics_circuit_breaker",
|
||||||
|
"router_router_metrics_timeseries",
|
||||||
|
"router_router_models",
|
||||||
|
"router_router_rationale_811",
|
||||||
|
"router_router_stream"
|
||||||
|
],
|
||||||
|
"5": [
|
||||||
|
"router_router_get_redis",
|
||||||
|
"router_router_gpu_decr",
|
||||||
|
"router_router_gpu_incr",
|
||||||
|
"router_router_half_open_probe",
|
||||||
|
"router_router_is_circuit_tripped",
|
||||||
|
"router_router_performance",
|
||||||
|
"router_router_rationale_282",
|
||||||
|
"router_router_rationale_297",
|
||||||
|
"router_router_rationale_619"
|
||||||
|
],
|
||||||
|
"6": [
|
||||||
|
"router_router_check_gpu_health",
|
||||||
|
"router_router_gpu_active_count",
|
||||||
|
"router_router_gpu_health_score",
|
||||||
|
"router_router_health",
|
||||||
|
"router_router_metrics_gpu_health",
|
||||||
|
"router_router_rationale_141",
|
||||||
|
"router_router_rationale_225",
|
||||||
|
"router_router_rationale_828"
|
||||||
|
],
|
||||||
|
"7": [
|
||||||
|
"router_router_chat",
|
||||||
|
"router_router_check_rate_limit",
|
||||||
|
"router_router_clean_response",
|
||||||
|
"router_router_clean_unicode",
|
||||||
|
"router_router_rationale_184",
|
||||||
|
"router_router_rationale_72",
|
||||||
|
"router_router_store_perf_record"
|
||||||
|
],
|
||||||
|
"8": [
|
||||||
|
"maintenance",
|
||||||
|
"maintenance_sh__entry"
|
||||||
|
],
|
||||||
|
"9": [
|
||||||
|
"router_router_counter_audit_loop",
|
||||||
|
"router_router_rationale_116"
|
||||||
|
],
|
||||||
|
"10": [
|
||||||
|
"router_router_metrics_latency",
|
||||||
|
"router_router_rationale_859"
|
||||||
|
],
|
||||||
|
"11": [
|
||||||
|
"router_router_rationale_288",
|
||||||
|
"router_router_trip_circuit"
|
||||||
|
],
|
||||||
|
"12": [
|
||||||
|
"router_router_rationale_736",
|
||||||
|
"router_router_scatter"
|
||||||
|
],
|
||||||
|
"13": [
|
||||||
|
"router_http_patch"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"cohesion": {
|
||||||
|
"0": 0.20833333333333334,
|
||||||
|
"1": 0.21818181818181817,
|
||||||
|
"2": 0.3111111111111111,
|
||||||
|
"3": 0.2,
|
||||||
|
"4": 0.2777777777777778,
|
||||||
|
"5": 0.2222222222222222,
|
||||||
|
"6": 0.35714285714285715,
|
||||||
|
"7": 0.3333333333333333,
|
||||||
|
"8": 1.0,
|
||||||
|
"9": 1.0,
|
||||||
|
"10": 1.0,
|
||||||
|
"11": 1.0,
|
||||||
|
"12": 1.0,
|
||||||
|
"13": 1.0
|
||||||
|
},
|
||||||
|
"gods": [
|
||||||
|
{
|
||||||
|
"id": "router_router_chat",
|
||||||
|
"label": "chat()",
|
||||||
|
"degree": 12
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_get_redis",
|
||||||
|
"label": "get_redis()",
|
||||||
|
"degree": 11
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_gpu_active_count",
|
||||||
|
"label": "gpu_active_count()",
|
||||||
|
"degree": 9
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_is_gpu_busy",
|
||||||
|
"label": "is_gpu_busy()",
|
||||||
|
"degree": 9
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_select_best_gpu",
|
||||||
|
"label": "select_best_gpu()",
|
||||||
|
"degree": 8
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_route",
|
||||||
|
"label": "route()",
|
||||||
|
"degree": 8
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_route_v3_route",
|
||||||
|
"label": "route()",
|
||||||
|
"degree": 6
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_check_gpu_health",
|
||||||
|
"label": "check_gpu_health()",
|
||||||
|
"degree": 6
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_available_models",
|
||||||
|
"label": "available_models()",
|
||||||
|
"degree": 6
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "router_router_estimate_tokens",
|
||||||
|
"label": "estimate_tokens()",
|
||||||
|
"degree": 6
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"surprises": [
|
||||||
|
{
|
||||||
|
"source": "route()",
|
||||||
|
"target": "available_models()",
|
||||||
|
"source_files": [
|
||||||
|
"router/route_v2.py",
|
||||||
|
"router/router.py"
|
||||||
|
],
|
||||||
|
"confidence": "INFERRED",
|
||||||
|
"relation": "calls",
|
||||||
|
"why": "inferred connection - not explicitly stated in source"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "route()",
|
||||||
|
"target": "estimate_tokens()",
|
||||||
|
"source_files": [
|
||||||
|
"router/route_v2.py",
|
||||||
|
"router/router.py"
|
||||||
|
],
|
||||||
|
"confidence": "INFERRED",
|
||||||
|
"relation": "calls",
|
||||||
|
"why": "inferred connection - not explicitly stated in source"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "route()",
|
||||||
|
"target": "is_gpu_busy()",
|
||||||
|
"source_files": [
|
||||||
|
"router/route_v2.py",
|
||||||
|
"router/router.py"
|
||||||
|
],
|
||||||
|
"confidence": "INFERRED",
|
||||||
|
"relation": "calls",
|
||||||
|
"why": "inferred connection - not explicitly stated in source"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "route()",
|
||||||
|
"target": "select_best_gpu()",
|
||||||
|
"source_files": [
|
||||||
|
"router/route_v2.py",
|
||||||
|
"router/router.py"
|
||||||
|
],
|
||||||
|
"confidence": "INFERRED",
|
||||||
|
"relation": "calls",
|
||||||
|
"why": "inferred connection - not explicitly stated in source"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"source": "route()",
|
||||||
|
"target": "available_models()",
|
||||||
|
"source_files": [
|
||||||
|
"router/route_v3.py",
|
||||||
|
"router/router.py"
|
||||||
|
],
|
||||||
|
"confidence": "INFERRED",
|
||||||
|
"relation": "calls",
|
||||||
|
"why": "inferred connection - not explicitly stated in source"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"questions": [
|
||||||
|
{
|
||||||
|
"type": "bridge_node",
|
||||||
|
"question": "Why does `is_gpu_busy()` connect `Community 0` to `Community 4`, `Community 6`?",
|
||||||
|
"why": "High betweenness centrality (0.061) - this node is a cross-community bridge."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "bridge_node",
|
||||||
|
"question": "Why does `select_best_gpu()` connect `Community 0` to `Community 4`, `Community 6`?",
|
||||||
|
"why": "High betweenness centrality (0.031) - this node is a cross-community bridge."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "bridge_node",
|
||||||
|
"question": "Why does `estimate_tokens()` connect `Community 0` to `Community 4`, `Community 7`?",
|
||||||
|
"why": "High betweenness centrality (0.030) - this node is a cross-community bridge."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "verify_inferred",
|
||||||
|
"question": "Are the 3 inferred relationships involving `is_gpu_busy()` (e.g. with `route()` and `moe_spillover()`) actually correct?",
|
||||||
|
"why": "`is_gpu_busy()` has 3 INFERRED edges - model-reasoned connections that need verification."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "verify_inferred",
|
||||||
|
"question": "Are the 3 inferred relationships involving `select_best_gpu()` (e.g. with `route()` and `route()`) actually correct?",
|
||||||
|
"why": "`select_best_gpu()` has 3 INFERRED edges - model-reasoned connections that need verification."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "isolated_nodes",
|
||||||
|
"question": "What connects `SyslogAI Harness Dashboard \u2014 Modern Design.`, `maintenance.sh script`, `Nginx upstream health probe. Returns 200 if service is alive.` to the rest of the system?",
|
||||||
|
"why": "28 weakly-connected nodes found - possible documentation gaps or missing edges."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
|||||||
|
{"files": {"code": ["/root/syslog-harness-repo/dashboard/dashboard.py", "/root/syslog-harness-repo/maintenance.sh", "/root/syslog-harness-repo/phase0-dual-keys.json", "/root/syslog-harness-repo/queue-service/queue-service.py", "/root/syslog-harness-repo/router/http_patch.py", "/root/syslog-harness-repo/router/route_v2.py", "/root/syslog-harness-repo/router/route_v3.py", "/root/syslog-harness-repo/router/router.py"], "document": ["/root/syslog-harness-repo/LITELLM-MIGRATION-PLAN.md", "/root/syslog-harness-repo/MIGRATION_PLAN.md", "/root/syslog-harness-repo/README.md", "/root/syslog-harness-repo/dashboard/dashboard.html", "/root/syslog-harness-repo/dashboard/harness.html", "/root/syslog-harness-repo/dashboard/requirements.txt", "/root/syslog-harness-repo/docker-compose.yml", "/root/syslog-harness-repo/litellm_config.yaml", "/root/syslog-harness-repo/router/requirements.txt", "/root/syslog-harness-repo/ssl/README.md"], "paper": [], "image": [], "video": []}, "total_files": 18, "total_words": 14667, "needs_graph": false, "warning": "Corpus is ~14,667 words - fits in a single context window. You may not need a graph.", "skipped_sensitive": ["/root/syslog-harness-repo/.env.example"], "unclassified": ["/root/syslog-harness-repo/.gitignore", "/root/syslog-harness-repo/Dockerfile.dashboard", "/root/syslog-harness-repo/Dockerfile.queue", "/root/syslog-harness-repo/backups/20260602_103344/dashboard.py.bak", "/root/syslog-harness-repo/backups/20260602_103344/router.py.bak", "/root/syslog-harness-repo/backups/20260602_103344/ts_patch.py.bak", "/root/syslog-harness-repo/dashboard/Dockerfile", "/root/syslog-harness-repo/docker-compose.yml.bak", "/root/syslog-harness-repo/gpu-router-docker.conf", "/root/syslog-harness-repo/gpu-router.conf", "/root/syslog-harness-repo/nginx/nginx.conf", "/root/syslog-harness-repo/nginx/nginx.conf.bak", "/root/syslog-harness-repo/router/Dockerfile", "/root/syslog-harness-repo/router/router.py.bak.20260518074236"], "graphifyignore_patterns": 3, "scan_root": "/root/syslog-harness-repo"}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
|||||||
|
{"nodes": [], "edges": [], "hyperedges": [], "input_tokens": 0, "output_tokens": 0}
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
# Graph Report - . (2026-07-08)
|
||||||
|
|
||||||
|
## Corpus Check
|
||||||
|
- Corpus is ~14,667 words - fits in a single context window. You may not need a graph.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
- 91 nodes · 151 edges · 14 communities (9 shown, 5 thin omitted)
|
||||||
|
- Extraction: 93% EXTRACTED · 7% INFERRED · 0% AMBIGUOUS · INFERRED: 10 edges (avg confidence: 0.77)
|
||||||
|
- Token cost: 0 input · 0 output
|
||||||
|
|
||||||
|
## Community Hubs (Navigation)
|
||||||
|
- Community 0
|
||||||
|
- Community 1
|
||||||
|
- Community 2
|
||||||
|
- Community 3
|
||||||
|
- Community 4
|
||||||
|
- Community 5
|
||||||
|
- Community 6
|
||||||
|
- Community 7
|
||||||
|
- Community 8
|
||||||
|
- Community 9
|
||||||
|
- Community 10
|
||||||
|
- Community 11
|
||||||
|
- Community 12
|
||||||
|
|
||||||
|
## God Nodes (most connected - your core abstractions)
|
||||||
|
1. `chat()` - 12 edges
|
||||||
|
2. `get_redis()` - 11 edges
|
||||||
|
3. `gpu_active_count()` - 9 edges
|
||||||
|
4. `is_gpu_busy()` - 9 edges
|
||||||
|
5. `select_best_gpu()` - 8 edges
|
||||||
|
6. `route()` - 8 edges
|
||||||
|
7. `route()` - 6 edges
|
||||||
|
8. `check_gpu_health()` - 6 edges
|
||||||
|
9. `available_models()` - 6 edges
|
||||||
|
10. `estimate_tokens()` - 6 edges
|
||||||
|
|
||||||
|
## Surprising Connections (you probably didn't know these)
|
||||||
|
- `route()` --calls--> `available_models()` [INFERRED]
|
||||||
|
router/route_v2.py → router/router.py
|
||||||
|
- `route()` --calls--> `estimate_tokens()` [INFERRED]
|
||||||
|
router/route_v2.py → router/router.py
|
||||||
|
- `route()` --calls--> `is_gpu_busy()` [INFERRED]
|
||||||
|
router/route_v2.py → router/router.py
|
||||||
|
- `route()` --calls--> `select_best_gpu()` [INFERRED]
|
||||||
|
router/route_v2.py → router/router.py
|
||||||
|
- `route()` --calls--> `available_models()` [INFERRED]
|
||||||
|
router/route_v3.py → router/router.py
|
||||||
|
|
||||||
|
## Import Cycles
|
||||||
|
- None detected.
|
||||||
|
|
||||||
|
## Communities (14 total, 5 thin omitted)
|
||||||
|
|
||||||
|
### Community 0 - "Community 0"
|
||||||
|
Cohesion: 0.21
|
||||||
|
Nodes (14): route(), moe_spillover(), Spill 40% of MoE-first traffic to Dense to prevent Strix Halo overheating. O, route(), available_models(), estimate_tokens(), is_gpu_busy(), moe_spillover() (+6 more)
|
||||||
|
|
||||||
|
### Community 1 - "Community 1"
|
||||||
|
Cohesion: 0.22
|
||||||
|
Nodes (4): api_state(), broadcast_loop(), fetch_state(), SyslogAI Harness Dashboard — Modern Design.
|
||||||
|
|
||||||
|
### Community 2 - "Community 2"
|
||||||
|
Cohesion: 0.31
|
||||||
|
Nodes (9): check_gpu_health(), enqueue(), get_queue_depth(), get_redis(), health(), GET queue depth + circuit breaker state + GPU health., Nginx upstream health probe. Returns 200 if service is alive., Fallback endpoint — Nginx calls this when all GPU upstreams are down. (+1 more)
|
||||||
|
|
||||||
|
### Community 3 - "Community 3"
|
||||||
|
Cohesion: 0.20
|
||||||
|
Nodes (10): _admin_auth(), admin_deprecation_summary(), admin_generate_key(), admin_keys(), admin_revoke_key(), Require admin key for management endpoints., List all API keys (masked) with agent, tier, and deprecation status., Summary of deprecated key usage (from Redis logs, if available). (+2 more)
|
||||||
|
|
||||||
|
### Community 4 - "Community 4"
|
||||||
|
Cohesion: 0.28
|
||||||
|
Nodes (5): bcast(), get_metrics(), metrics(), metrics_circuit_breaker(), Expose circuit breaker status per model. Phase 1.
|
||||||
|
|
||||||
|
### Community 5 - "Community 5"
|
||||||
|
Cohesion: 0.22
|
||||||
|
Nodes (9): get_redis(), gpu_decr(), gpu_incr(), half_open_probe(), is_circuit_tripped(), performance(), Check if a GPU host is currently blacklisted., Check if a GPU host can be un-blacklisted. (+1 more)
|
||||||
|
|
||||||
|
### Community 6 - "Community 6"
|
||||||
|
Cohesion: 0.36
|
||||||
|
Nodes (8): check_gpu_health(), gpu_active_count(), gpu_health_score(), health(), metrics_gpu_health(), Get number of in-flight requests for a GPU., Score a GPU based on VRAM, temperature, and load. Lower = better., Live GPU health scores + circuit breaker + KPIs.
|
||||||
|
|
||||||
|
### Community 7 - "Community 7"
|
||||||
|
Cohesion: 0.33
|
||||||
|
Nodes (7): chat(), check_rate_limit(), clean_response(), clean_unicode(), Store detailed performance record in Redis for analytics., Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remainin, store_perf_record()
|
||||||
|
|
||||||
|
## Knowledge Gaps
|
||||||
|
- **1 isolated node(s):** `maintenance.sh script`
|
||||||
|
These have ≤1 connection - possible missing edges or undocumented components.
|
||||||
|
- **5 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||||
|
|
||||||
|
## Suggested Questions
|
||||||
|
_Questions this graph is uniquely positioned to answer:_
|
||||||
|
|
||||||
|
- **Why does `is_gpu_busy()` connect `Community 0` to `Community 4`, `Community 6`?**
|
||||||
|
_High betweenness centrality (0.061) - this node is a cross-community bridge._
|
||||||
|
- **Why does `select_best_gpu()` connect `Community 0` to `Community 4`, `Community 6`?**
|
||||||
|
_High betweenness centrality (0.031) - this node is a cross-community bridge._
|
||||||
|
- **Why does `estimate_tokens()` connect `Community 0` to `Community 4`, `Community 7`?**
|
||||||
|
_High betweenness centrality (0.030) - this node is a cross-community bridge._
|
||||||
|
- **Are the 3 inferred relationships involving `is_gpu_busy()` (e.g. with `route()` and `moe_spillover()`) actually correct?**
|
||||||
|
_`is_gpu_busy()` has 3 INFERRED edges - model-reasoned connections that need verification._
|
||||||
|
- **Are the 3 inferred relationships involving `select_best_gpu()` (e.g. with `route()` and `route()`) actually correct?**
|
||||||
|
_`select_best_gpu()` has 3 INFERRED edges - model-reasoned connections that need verification._
|
||||||
|
- **What connects `SyslogAI Harness Dashboard — Modern Design.`, `maintenance.sh script`, `Nginx upstream health probe. Returns 200 if service is alive.` to the rest of the system?**
|
||||||
|
_28 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||||
graphify-out/cache/ast/v0.9.10/30b202626b1f50da90b78272001e253cc7086b0bdc52e46553414c5313e3fb4e.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/53edfda5145db02bc35b9e9d8e06e5c731ce50c3a50cec3db65b3b886bed95a6.json
Vendored
+1
@@ -0,0 +1 @@
|
|||||||
|
{"nodes": [{"id": "root_syslog_harness_repo_router_http_patch_py", "label": "http_patch.py", "file_type": "code", "source_file": "router/http_patch.py", "source_location": "L1"}], "edges": [{"source": "root_syslog_harness_repo_router_http_patch_py", "target": "re", "relation": "imports", "context": "import", "confidence": "EXTRACTED", "source_file": "router/http_patch.py", "source_location": "L2", "weight": 1.0}], "raw_calls": []}
|
||||||
graphify-out/cache/ast/v0.9.10/654e97aa6df1872b0717f32f676c067d15b9795a0dc04a86936730a7e133c214.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/801feb2c0b7f4052cb2c4ef933d295d3494c8791766687b891c04cc7b5495d09.json
Vendored
+1
@@ -0,0 +1 @@
|
|||||||
|
{"nodes": [{"id": "root_syslog_harness_repo_maintenance_sh", "label": "maintenance.sh", "file_type": "code", "source_file": "maintenance.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "root_syslog_harness_repo_maintenance_sh__entry", "label": "maintenance.sh script", "file_type": "code", "source_file": "maintenance.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "root_syslog_harness_repo_maintenance_sh", "target": "root_syslog_harness_repo_maintenance_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "maintenance.sh", "source_location": "L1", "weight": 1.0}]}
|
||||||
graphify-out/cache/ast/v0.9.10/9c6817ae09f6bb5b2862f0282cacab7d4ce5da051e4ddc95e2e4099c7bf09716.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/c9a9d2e5c9f1fb7d35a73f91c83a047ce92dc2d09c9ddab8fc2d6e82d92a093d.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/e96ac2ae879e0a876d4190daff4cc566d13798b97c4820f3efa47e06627bbbcb.json
Vendored
+1
File diff suppressed because one or more lines are too long
Vendored
+1
@@ -0,0 +1 @@
|
|||||||
|
{"/root/syslog-harness-repo/LITELLM-MIGRATION-PLAN.md":{"size":31561,"mtime_ns":1781468608211508160,"word_count":3762},"/root/syslog-harness-repo/MIGRATION_PLAN.md":{"size":1966,"mtime_ns":1781121183850310435,"word_count":303},"/root/syslog-harness-repo/README.md":{"size":2447,"mtime_ns":1781121183850310435,"word_count":377},"/root/syslog-harness-repo/dashboard/dashboard.html":{"size":8331,"mtime_ns":1781316813360201927,"word_count":415},"/root/syslog-harness-repo/dashboard/dashboard.py":{"size":29010,"mtime_ns":1781121183851178603,"word_count":1615,"hash":"654e97aa6df1872b0717f32f676c067d15b9795a0dc04a86936730a7e133c214"},"/root/syslog-harness-repo/dashboard/harness.html":{"size":23388,"mtime_ns":1781316813362649901,"word_count":1869},"/root/syslog-harness-repo/dashboard/requirements.txt":{"size":30,"mtime_ns":1781121183851178603,"word_count":2},"/root/syslog-harness-repo/docker-compose.yml":{"size":3900,"mtime_ns":1782224658037786319,"word_count":200},"/root/syslog-harness-repo/litellm_config.yaml":{"size":618,"mtime_ns":1781121183851178603,"word_count":39},"/root/syslog-harness-repo/maintenance.sh":{"size":1302,"mtime_ns":1781121183851178603,"word_count":192,"hash":"801feb2c0b7f4052cb2c4ef933d295d3494c8791766687b891c04cc7b5495d09"},"/root/syslog-harness-repo/phase0-dual-keys.json":{"size":1628,"mtime_ns":1781121183851610673,"word_count":110,"hash":"0f115313f2c86df2f57e48bdf180b768d6452d0337bae4e3c3d2b52b9f5267a7"},"/root/syslog-harness-repo/queue-service/queue-service.py":{"size":3284,"mtime_ns":1781121183851610673,"word_count":325,"hash":"c9a9d2e5c9f1fb7d35a73f91c83a047ce92dc2d09c9ddab8fc2d6e82d92a093d"},"/root/syslog-harness-repo/router/http_patch.py":{"size":3562,"mtime_ns":1781121183851788825,"word_count":290,"hash":"53edfda5145db02bc35b9e9d8e06e5c731ce50c3a50cec3db65b3b886bed95a6"},"/root/syslog-harness-repo/router/requirements.txt":{"size":43,"mtime_ns":1781121183851788825,"word_count":3},"/root/syslog-harness-repo/router/route_v2.py":{"size":3954,"mtime_ns":1781121183851788825,"word_count":435,"hash":"e96ac2ae879e0a876d4190daff4cc566d13798b97c4820f3efa47e06627bbbcb"},"/root/syslog-harness-repo/router/route_v3.py":{"size":4703,"mtime_ns":1781121183851788825,"word_count":504,"hash":"30b202626b1f50da90b78272001e253cc7086b0bdc52e46553414c5313e3fb4e"},"/root/syslog-harness-repo/router/router.py":{"size":44307,"mtime_ns":1781316813360201927,"word_count":4198,"hash":"9c6817ae09f6bb5b2862f0282cacab7d4ce5da051e4ddc95e2e4099c7bf09716"},"/root/syslog-harness-repo/ssl/README.md":{"size":204,"mtime_ns":1781121183851788825,"word_count":28}}
|
||||||
File diff suppressed because it is too large
Load Diff
+6
-10
@@ -1,25 +1,21 @@
|
|||||||
model_list:
|
model_list:
|
||||||
- model_name: qwen3.6-35B-A3B
|
- model_name: qwen3.6-35B-A3B
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: openai/qwen3.6-35B-A3B
|
model: openai/qwen3.6-35B-A3B
|
||||||
api_base: http://192.168.68.15:8080/v1
|
api_base: http://192.168.68.15:8080/v1
|
||||||
api_key: "not-needed"
|
api_key: not-needed
|
||||||
|
- model_name: gpu-dense
|
||||||
- model_name: qwen3.6-27B-code
|
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: openai/qwen3.6-27B-code-text
|
model: openai/qwen3.6-27B-code-text
|
||||||
api_base: http://192.168.68.8:8080/v1
|
api_base: http://192.168.68.8:8080/v1
|
||||||
api_key: "not-needed"
|
api_key: not-needed
|
||||||
|
- model_name: gpu-light
|
||||||
- model_name: gemma-4-12b
|
|
||||||
litellm_params:
|
litellm_params:
|
||||||
model: openai/gemma-4-12b
|
model: openai/gemma-4-12b
|
||||||
api_base: http://192.168.68.110:8080/v1
|
api_base: http://192.168.68.110:8080/v1
|
||||||
api_key: "not-needed"
|
api_key: not-needed
|
||||||
|
|
||||||
general_settings:
|
general_settings:
|
||||||
master_key: sk-syslog-local-master-key
|
master_key: sk-syslog-local-master-key
|
||||||
|
|
||||||
litellm_settings:
|
litellm_settings:
|
||||||
drop_params: true
|
drop_params: true
|
||||||
request_timeout: 120
|
request_timeout: 120
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
model_list:
|
||||||
|
- model_name: qwen3.6-35B-A3B
|
||||||
|
litellm_params:
|
||||||
|
model: openai/qwen3.6-35B-A3B
|
||||||
|
api_base: http://192.168.68.15:8080/v1
|
||||||
|
api_key: "not-needed"
|
||||||
|
|
||||||
|
- model_name: qwen3.6-27B-code
|
||||||
|
litellm_params:
|
||||||
|
model: openai/qwen3.6-27B-code-text
|
||||||
|
api_base: http://192.168.68.8:8080/v1
|
||||||
|
api_key: "not-needed"
|
||||||
|
|
||||||
|
- model_name: gemma-4-12b
|
||||||
|
litellm_params:
|
||||||
|
model: openai/gemma-4-12b
|
||||||
|
api_base: http://192.168.68.110:8080/v1
|
||||||
|
api_key: "not-needed"
|
||||||
|
|
||||||
|
general_settings:
|
||||||
|
master_key: sk-syslog-local-master-key
|
||||||
|
|
||||||
|
litellm_settings:
|
||||||
|
drop_params: true
|
||||||
|
request_timeout: 120
|
||||||
+26
-3
@@ -16,7 +16,7 @@ http {
|
|||||||
sendfile on;
|
sendfile on;
|
||||||
keepalive_timeout 65;
|
keepalive_timeout 65;
|
||||||
|
|
||||||
upstream router_api { server router:9000; }
|
upstream router_api { server host.docker.internal:9000; }
|
||||||
upstream dashboard_ui { server dashboard:3000; }
|
upstream dashboard_ui { server dashboard:3000; }
|
||||||
upstream litellm_backend { server litellm:4000; }
|
upstream litellm_backend { server litellm:4000; }
|
||||||
|
|
||||||
@@ -31,7 +31,7 @@ http {
|
|||||||
# Disable buffering for SSE streams
|
# Disable buffering for SSE streams
|
||||||
proxy_buffering off;
|
proxy_buffering off;
|
||||||
|
|
||||||
# API — through router
|
# API through router
|
||||||
location /v1/ {
|
location /v1/ {
|
||||||
proxy_pass http://router_api;
|
proxy_pass http://router_api;
|
||||||
proxy_http_version 1.1;
|
proxy_http_version 1.1;
|
||||||
@@ -43,6 +43,14 @@ http {
|
|||||||
proxy_buffering off;
|
proxy_buffering off;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
location /admin/ {
|
||||||
|
proxy_pass http://router_api;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header Authorization $http_authorization;
|
||||||
|
}
|
||||||
|
|
||||||
# SSE streaming endpoint
|
# SSE streaming endpoint
|
||||||
location /stream {
|
location /stream {
|
||||||
proxy_pass http://router_api;
|
proxy_pass http://router_api;
|
||||||
@@ -70,7 +78,15 @@ http {
|
|||||||
proxy_set_header Authorization $http_authorization;
|
proxy_set_header Authorization $http_authorization;
|
||||||
}
|
}
|
||||||
|
|
||||||
# Dashboard
|
# Professional Dashboard (Phase 1-3) - Static HTML served via Nginx
|
||||||
|
location /dashboard/ {
|
||||||
|
alias /opt/inference-harness/dashboard/;
|
||||||
|
index dashboard.html;
|
||||||
|
add_header Cache-Control "public, max-age=3600";
|
||||||
|
add_header X-Content-Type-Options nosniff;
|
||||||
|
}
|
||||||
|
|
||||||
|
# Legacy Dashboard (root) - Proxy to Flask app
|
||||||
location / {
|
location / {
|
||||||
proxy_pass http://dashboard_ui;
|
proxy_pass http://dashboard_ui;
|
||||||
proxy_http_version 1.1;
|
proxy_http_version 1.1;
|
||||||
@@ -85,6 +101,13 @@ http {
|
|||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Circuit Breaker metrics (Phase 1)
|
||||||
|
location /metrics/circuit-breaker {
|
||||||
|
proxy_pass http://router_api/metrics/circuit-breaker;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
}
|
||||||
|
|
||||||
location /health {
|
location /health {
|
||||||
proxy_pass http://router_api/health;
|
proxy_pass http://router_api/health;
|
||||||
proxy_http_version 1.1;
|
proxy_http_version 1.1;
|
||||||
|
|||||||
+429
-25
@@ -2,6 +2,50 @@ import os, json, time, logging, traceback, threading, queue, statistics, math
|
|||||||
import requests, redis
|
import requests, redis
|
||||||
from flask import Flask, request, jsonify, Response, stream_with_context
|
from flask import Flask, request, jsonify, Response, stream_with_context
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Phase 2: Atomic session token update Redis Lua script
|
||||||
|
SESSION_LUA_SCRIPT = """
|
||||||
|
local key = KEYS[1]
|
||||||
|
local new_val = tonumber(ARGV[1])
|
||||||
|
local current = tonumber(redis.call('GET', key) or 0)
|
||||||
|
local max_val = math.max(current, new_val)
|
||||||
|
redis.call('SET', key, max_val, 'EX', 86400)
|
||||||
|
return max_val
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Phase 4: Atomic GPU slot booking (closes TOCTOU race between check and incr)
|
||||||
|
SLOT_BOOK_LUA = """
|
||||||
|
local key = KEYS[1]
|
||||||
|
local max_c = tonumber(ARGV[1])
|
||||||
|
local current = tonumber(redis.call('GET', key) or '0')
|
||||||
|
if current < max_c then
|
||||||
|
redis.call('INCR', key)
|
||||||
|
return 1
|
||||||
|
else
|
||||||
|
return 0
|
||||||
|
end
|
||||||
|
"""
|
||||||
|
SLOT_RELEASE_LUA = """
|
||||||
|
local key = KEYS[1]
|
||||||
|
local current = tonumber(redis.call('GET', key) or '0')
|
||||||
|
if current > 0 then
|
||||||
|
redis.call('DECR', key)
|
||||||
|
end
|
||||||
|
current = tonumber(redis.call('GET', key) or '0')
|
||||||
|
if current < 0 then
|
||||||
|
redis.call('SET', key, '0')
|
||||||
|
end
|
||||||
|
return redis.call('GET', key)
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Phase 3b: Configurable health scoring weights (env-overridable)
|
||||||
|
HEALTH_WEIGHT_VRAM = float(os.environ.get("HEALTH_WEIGHT_VRAM", "0.40"))
|
||||||
|
HEALTH_WEIGHT_TEMP = float(os.environ.get("HEALTH_WEIGHT_TEMP", "0.30"))
|
||||||
|
HEALTH_WEIGHT_LOAD = float(os.environ.get("HEALTH_WEIGHT_LOAD", "0.30"))
|
||||||
|
HEALTH_TEMP_BASELINE = int(os.environ.get("HEALTH_TEMP_BASELINE", "30"))
|
||||||
|
|
||||||
|
|
||||||
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
|
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
|
||||||
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
|
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
|
||||||
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
|
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
|
||||||
@@ -18,6 +62,13 @@ GPU_URLS = {
|
|||||||
"gemma-4-12b": GPU_LIGHT_URL,
|
"gemma-4-12b": GPU_LIGHT_URL,
|
||||||
}
|
}
|
||||||
# Max concurrent requests per GPU (based on llama.cpp --parallel)
|
# Max concurrent requests per GPU (based on llama.cpp --parallel)
|
||||||
|
|
||||||
|
GPU_LABELS = {
|
||||||
|
"qwen3.6-35B-A3B": "Qwen3.6 35B (Strix Halo)",
|
||||||
|
"qwen3.6-27B-code": "Qwen3.6 27B Code (RTX 3090)",
|
||||||
|
"gemma-4-12b": "Gemma-4 12B (RTX 5070)",
|
||||||
|
}
|
||||||
|
|
||||||
GPU_MAX_CONCURRENT = {
|
GPU_MAX_CONCURRENT = {
|
||||||
"qwen3.6-35B-A3B": 2, # 2 slots (cross-agent spread prevents overheating)
|
"qwen3.6-35B-A3B": 2, # 2 slots (cross-agent spread prevents overheating)
|
||||||
"qwen3.6-27B-code": 2, # 2 slots (128K context frees VRAM)
|
"qwen3.6-27B-code": 2, # 2 slots (128K context frees VRAM)
|
||||||
@@ -54,7 +105,7 @@ def check_rate_limit(api_key, tier):
|
|||||||
return True, 999, 60
|
return True, 999, 60
|
||||||
limit = RATE_LIMIT_RPM.get(tier, 30)
|
limit = RATE_LIMIT_RPM.get(tier, 30)
|
||||||
key = f"ratelimit:{api_key}"
|
key = f"ratelimit:{api_key}"
|
||||||
current = int(r.get(key) or 0)
|
current = int(get_redis().get(key) or 0)
|
||||||
if current >= limit:
|
if current >= limit:
|
||||||
ttl = r.ttl(key)
|
ttl = r.ttl(key)
|
||||||
retry = max(ttl, 1) if ttl and ttl > 0 else 60
|
retry = max(ttl, 1) if ttl and ttl > 0 else 60
|
||||||
@@ -105,9 +156,9 @@ def counter_audit_loop():
|
|||||||
slots = resp.json()
|
slots = resp.json()
|
||||||
all_idle = all(not s.get("is_processing", False) for s in slots)
|
all_idle = all(not s.get("is_processing", False) for s in slots)
|
||||||
if all_idle:
|
if all_idle:
|
||||||
current = int(r.get("active:" + model) or 0)
|
current = int(get_redis().get("active:" + model) or 0)
|
||||||
if current > 0:
|
if current > 0:
|
||||||
rd.set("active:" + model, 0)
|
get_redis().set("active:" + model, 0)
|
||||||
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
|
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
@@ -131,7 +182,37 @@ def gpu_decr(model):
|
|||||||
if rd:
|
if rd:
|
||||||
v = rd.decr("active:" + model)
|
v = rd.decr("active:" + model)
|
||||||
if v and int(v) < 0:
|
if v and int(v) < 0:
|
||||||
rd.set("active:" + model, 0) # never go negative
|
get_redis().set("active:" + model, 0) # never go negative
|
||||||
|
|
||||||
|
# Phase 4: Atomic GPU slot booking (Lua-based, closes TOCTOU race)
|
||||||
|
def gpu_book_slot(model):
|
||||||
|
"""Atomically book a GPU slot. Returns True if acquired, False if full."""
|
||||||
|
rd = get_redis()
|
||||||
|
if not rd:
|
||||||
|
return True # No Redis — allow everything (degraded mode)
|
||||||
|
try:
|
||||||
|
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||||
|
result = rd.eval(SLOT_BOOK_LUA, 1, "active:" + model, max_c)
|
||||||
|
return result == 1
|
||||||
|
except Exception:
|
||||||
|
# Lua not loaded — fall back to non-atomic
|
||||||
|
current = int(rd.get("active:" + model) or 0)
|
||||||
|
if current < GPU_MAX_CONCURRENT.get(model, 1):
|
||||||
|
rd.incr("active:" + model)
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
def gpu_release_slot(model):
|
||||||
|
"""Atomically release a GPU slot. Never goes negative."""
|
||||||
|
rd = get_redis()
|
||||||
|
if not rd:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
rd.eval(SLOT_RELEASE_LUA, 1, "active:" + model)
|
||||||
|
except Exception:
|
||||||
|
v = rd.decr("active:" + model)
|
||||||
|
if v and int(v) < 0:
|
||||||
|
rd.set("active:" + model, 0)
|
||||||
def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
|
def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
|
||||||
url = GPU_SIDECARS.get(model)
|
url = GPU_SIDECARS.get(model)
|
||||||
if not url: return {"status": "unknown"}
|
if not url: return {"status": "unknown"}
|
||||||
@@ -198,6 +279,27 @@ def is_gpu_busy(model):
|
|||||||
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||||
return active >= max_c
|
return active >= max_c
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Phase 3: Dynamic GPU Weighting (Health Score)
|
||||||
|
def gpu_health_score(model):
|
||||||
|
"""Score a GPU based on VRAM, temperature, power, and load. Lower = better.
|
||||||
|
Weights configurable via HEALTH_WEIGHT_VRAM/TEMP/LOAD env vars."""
|
||||||
|
h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
|
||||||
|
if h.get("status") == "down":
|
||||||
|
return 999 # never pick down GPUs
|
||||||
|
if is_circuit_tripped(model):
|
||||||
|
return 998 # circuit open — skip but distinguishable from down
|
||||||
|
vram_pct = h.get("vram_pct") or 50
|
||||||
|
temp_c = h.get("temp_c") or 50
|
||||||
|
power_w = h.get("power_w") or 100
|
||||||
|
active = gpu_active_count(model)
|
||||||
|
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||||
|
load_pct = (active / max_c) * 100 if max_c > 0 else 0
|
||||||
|
temp_penalty = max(0, (temp_c or 50) - HEALTH_TEMP_BASELINE)
|
||||||
|
score = (vram_pct or 0) * HEALTH_WEIGHT_VRAM + temp_penalty * 0.5 * HEALTH_WEIGHT_TEMP + load_pct * HEALTH_WEIGHT_LOAD
|
||||||
|
return round(score, 1)
|
||||||
|
|
||||||
def select_best_gpu(candidates, reason, agent=""):
|
def select_best_gpu(candidates, reason, agent=""):
|
||||||
"""Pick best GPU, spreading agents across GPUs to prevent hotspots."""
|
"""Pick best GPU, spreading agents across GPUs to prevent hotspots."""
|
||||||
# Count how many distinct agents are on each GPU
|
# Count how many distinct agents are on each GPU
|
||||||
@@ -209,17 +311,19 @@ def select_best_gpu(candidates, reason, agent=""):
|
|||||||
if r.get("agent_gpu:" + ak["agent"] + ":" + m):
|
if r.get("agent_gpu:" + ak["agent"] + ":" + m):
|
||||||
count += 1
|
count += 1
|
||||||
gpu_agent_counts[m] = count
|
gpu_agent_counts[m] = count
|
||||||
|
# Phase 3: Sort candidates by health score before selection
|
||||||
|
sorted_candidates = sorted(candidates, key=gpu_health_score)
|
||||||
# First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
|
# First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
|
||||||
for m in candidates:
|
for m in sorted_candidates:
|
||||||
if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
|
if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
|
||||||
return {"model": m, "reason": reason}
|
return {"model": m, "reason": reason}
|
||||||
# Second pass: prefer GPU this agent is NOT already on (skip own GPU)
|
# Second pass: prefer GPU this agent is NOT already on (skip own GPU)
|
||||||
if agent:
|
if agent:
|
||||||
for m in candidates:
|
for m in sorted_candidates:
|
||||||
if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
|
if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
|
||||||
return {"model": m, "reason": reason}
|
return {"model": m, "reason": reason}
|
||||||
# Third pass: any non-busy GPU
|
# Third pass: any non-busy GPU
|
||||||
for m in candidates:
|
for m in sorted_candidates:
|
||||||
if not is_gpu_busy(m):
|
if not is_gpu_busy(m):
|
||||||
return {"model": m, "reason": reason}
|
return {"model": m, "reason": reason}
|
||||||
# All busy — pick least loaded
|
# All busy — pick least loaded
|
||||||
@@ -236,13 +340,61 @@ def select_best_gpu(candidates, reason, agent=""):
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Phase 1: Circuit Breaker for GPU Hosts (Approved by Abiba)
|
||||||
|
CIRCUIT_FAIL_THRESHOLD = int(os.environ.get("CIRCUIT_FAIL_THRESHOLD", "3"))
|
||||||
|
CIRCUIT_FAIL_WINDOW = int(os.environ.get("CIRCUIT_FAIL_WINDOW", "120"))
|
||||||
|
CIRCUIT_COOLDOWN = int(os.environ.get("CIRCUIT_COOLDOWN", "60"))
|
||||||
|
|
||||||
|
def is_circuit_tripped(model):
|
||||||
|
"""Check if a GPU host is currently blacklisted."""
|
||||||
|
if not get_redis():
|
||||||
|
return False
|
||||||
|
return r.exists("circuit:" + model + ":open")
|
||||||
|
|
||||||
|
def trip_circuit(model, duration=None):
|
||||||
|
"""Blacklist a GPU host for specified duration (default CIRCUIT_COOLDOWN).
|
||||||
|
Only trips after CIRCUIT_FAIL_THRESHOLD failures within CIRCUIT_FAIL_WINDOW."""
|
||||||
|
if not get_redis():
|
||||||
|
return False
|
||||||
|
if duration is None:
|
||||||
|
duration = CIRCUIT_COOLDOWN
|
||||||
|
now = time.time()
|
||||||
|
fail_key = "circuit:" + model + ":failures"
|
||||||
|
pipe = r.pipeline()
|
||||||
|
pipe.lpush(fail_key, str(now))
|
||||||
|
pipe.ltrim(fail_key, 0, CIRCUIT_FAIL_THRESHOLD - 1)
|
||||||
|
pipe.lrange(fail_key, 0, -1)
|
||||||
|
results = pipe.execute()
|
||||||
|
failures = [float(f) for f in (results[-1] if results else [])]
|
||||||
|
recent = [f for f in failures if now - f <= CIRCUIT_FAIL_WINDOW]
|
||||||
|
if len(recent) >= CIRCUIT_FAIL_THRESHOLD:
|
||||||
|
key = "circuit:" + model + ":open"
|
||||||
|
r.set(key, 1, ex=duration)
|
||||||
|
r.incr("circuit:" + model + ":count")
|
||||||
|
log.warning("CIRCUIT_TRIPPED: %s — %d failures in %ds, cooldown %ds",
|
||||||
|
model, len(recent), CIRCUIT_FAIL_WINDOW, duration)
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
def half_open_probe(model):
|
||||||
|
"""Check if a GPU host can be un-blacklisted."""
|
||||||
|
if not get_redis():
|
||||||
|
return True
|
||||||
|
key = "circuit:" + model + ":open"
|
||||||
|
if not r.exists(key):
|
||||||
|
return True # no circuit
|
||||||
|
return False # still open
|
||||||
|
|
||||||
def route(rd, tier, agent=""):
|
def route(rd, tier, agent=""):
|
||||||
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
|
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
|
||||||
sys = any(m.get("role")=="system" for m in msgs)
|
sys = any(m.get("role")=="system" for m in msgs)
|
||||||
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
|
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
|
||||||
hints = rd.get("routing_hints",{})
|
hints = rd.get("routing_hints",{})
|
||||||
allowed = TIER_MODELS.get(tier, ["gemma-4-12b"])
|
allowed = TIER_MODELS.get(tier, ["gemma-4-12b"])
|
||||||
avail = [m for m in available_models() if m in allowed]
|
# Phase 1: Filter out models with tripped circuit breakers
|
||||||
|
avail = [m for m in available_models() if m in allowed and not is_circuit_tripped(m)]
|
||||||
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
|
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
|
||||||
if all(is_gpu_busy(m) for m in avail):
|
if all(is_gpu_busy(m) for m in avail):
|
||||||
return {"model": avail[0], "reason": "all_saturated", "saturated": True}
|
return {"model": avail[0], "reason": "all_saturated", "saturated": True}
|
||||||
@@ -262,13 +414,17 @@ def route(rd, tier, agent=""):
|
|||||||
return {"model": allowed[0], "reason": "vision_unavailable"}
|
return {"model": allowed[0], "reason": "vision_unavailable"}
|
||||||
|
|
||||||
req = rd.get("model","auto")
|
req = rd.get("model","auto")
|
||||||
|
# Map syslog-auto to auto for content-based routing
|
||||||
|
if req == "syslog-auto":
|
||||||
|
req = "auto"
|
||||||
if req != "auto":
|
if req != "auto":
|
||||||
|
# STRICT MODE: no silent fallback — LiteLLM handles failover chains.
|
||||||
|
# Returns saturated if explicit GPU is busy (keeps per-model metrics accurate).
|
||||||
target = req if req in avail else avail[0]
|
target = req if req in avail else avail[0]
|
||||||
if is_gpu_busy(target) and req in allowed:
|
if req not in avail:
|
||||||
alts = [m for m in avail if m != target and m in allowed]
|
return {"model": req, "reason": "explicit_unavailable", "saturated": True}
|
||||||
if alts:
|
if is_gpu_busy(target):
|
||||||
alt = select_best_gpu(alts, "explicit", agent)
|
return {"model": target, "reason": "explicit_saturated", "saturated": True}
|
||||||
if alt: return alt
|
|
||||||
return {"model": target, "reason": "explicit"}
|
return {"model": target, "reason": "explicit"}
|
||||||
|
|
||||||
if hints:
|
if hints:
|
||||||
@@ -382,6 +538,17 @@ def chat():
|
|||||||
ki = API_KEYS[ak]
|
ki = API_KEYS[ak]
|
||||||
tier, agent = ki["tier"], ki["agent"]
|
tier, agent = ki["tier"], ki["agent"]
|
||||||
|
|
||||||
|
# Phase 0: dual-key transition — log deprecated key usage
|
||||||
|
if ki.get("deprecated"):
|
||||||
|
new_key = next((k for k, v in API_KEYS.items()
|
||||||
|
if v.get("agent") == agent and not v.get("deprecated")), None)
|
||||||
|
log.warning("DEPRECATED_KEY: agent=%s using old key %s...%s — switch to %s...%s",
|
||||||
|
agent, ak[:12], ak[-8:],
|
||||||
|
new_key[:12] if new_key else "N/A",
|
||||||
|
new_key[-8:] if new_key else "N/A")
|
||||||
|
if r:
|
||||||
|
r.incr("deprecated_usage:" + agent)
|
||||||
|
|
||||||
# Rate limit check
|
# Rate limit check
|
||||||
allowed, rl_val, reset_sec = check_rate_limit(ak, tier)
|
allowed, rl_val, reset_sec = check_rate_limit(ak, tier)
|
||||||
if not allowed:
|
if not allowed:
|
||||||
@@ -396,15 +563,14 @@ def chat():
|
|||||||
# Allow agent to override queue timeout via header
|
# Allow agent to override queue timeout via header
|
||||||
q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
|
q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
|
||||||
|
|
||||||
# Cross-turn context tracking: accumulate tokens per session
|
# Cross-turn context tracking: accumulate tokens per session (Phase 2: atomic Lua)
|
||||||
session_id = request.headers.get("X-Session-Id", "")
|
session_id = request.headers.get("X-Session-Id", "")
|
||||||
session_tokens = 0
|
session_tokens = 0
|
||||||
if session_id and r:
|
if session_id and r:
|
||||||
try:
|
try:
|
||||||
prev = int(r.get("session:" + session_id) or 0)
|
|
||||||
current = estimate_tokens(rd.get("messages",[]))
|
current = estimate_tokens(rd.get("messages",[]))
|
||||||
session_tokens = max(prev, current) # context only grows
|
# Atomic GET/MAX/SET via Lua script prevents race conditions
|
||||||
r.set("session:" + session_id, session_tokens, ex=86400) # TTL 24h
|
session_tokens = r.eval(SESSION_LUA_SCRIPT, 1, "session:" + session_id, current)
|
||||||
except Exception: pass
|
except Exception: pass
|
||||||
|
|
||||||
d = route(rd, tier, agent)
|
d = route(rd, tier, agent)
|
||||||
@@ -426,14 +592,26 @@ def chat():
|
|||||||
log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
|
log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
|
||||||
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
|
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
|
||||||
|
|
||||||
|
# Phase 4: Atomic slot booking (replaces non-atomic gpu_incr)
|
||||||
|
if not gpu_book_slot(model):
|
||||||
|
d = route(rd, tier, agent)
|
||||||
|
if d.get("saturated"):
|
||||||
|
resp = jsonify({"error": "All GPUs saturated", "retry_after_s": 3})
|
||||||
|
resp.headers["Retry-After"] = "3"
|
||||||
|
return resp, 503
|
||||||
|
model, reason = d["model"], d["reason"]
|
||||||
|
if not gpu_book_slot(model):
|
||||||
|
resp = jsonify({"error": "GPU slot race — retry", "retry_after_s": 1})
|
||||||
|
resp.headers["Retry-After"] = "1"
|
||||||
|
return resp, 503
|
||||||
|
url = GPU_URLS[model]
|
||||||
|
|
||||||
# Stash rate limit values for response headers
|
# Stash rate limit values for response headers
|
||||||
_rl_remaining = rl_val
|
_rl_remaining = rl_val
|
||||||
_rl_limit = RATE_LIMIT_RPM.get(tier, 30)
|
_rl_limit = RATE_LIMIT_RPM.get(tier, 30)
|
||||||
_rl_reset = reset_sec
|
_rl_reset = reset_sec
|
||||||
is_stream = rd.get("stream", False)
|
is_stream = rd.get("stream", False)
|
||||||
|
|
||||||
gpu_incr(model)
|
|
||||||
|
|
||||||
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
|
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
|
||||||
# Track which GPU this agent is using (TTL 120s covers typical request)
|
# Track which GPU this agent is using (TTL 120s covers typical request)
|
||||||
if r and agent:
|
if r and agent:
|
||||||
@@ -448,11 +626,14 @@ def chat():
|
|||||||
except Exception: pass
|
except Exception: pass
|
||||||
start = time.time()
|
start = time.time()
|
||||||
resp = requests.post(url+"/chat/completions", json=rd,
|
resp = requests.post(url+"/chat/completions", json=rd,
|
||||||
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
|
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=900, stream=is_stream)
|
||||||
lat = int((time.time()-start)*1000)
|
lat = int((time.time()-start)*1000)
|
||||||
gpu_decr(model)
|
gpu_release_slot(model)
|
||||||
|
|
||||||
if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
|
if resp.status_code != 200:
|
||||||
|
if resp.status_code in (502, 504):
|
||||||
|
trip_circuit(model)
|
||||||
|
return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
|
||||||
if is_stream:
|
if is_stream:
|
||||||
# Buffer SSE chunks, handle split lines for large responses
|
# Buffer SSE chunks, handle split lines for large responses
|
||||||
chunks = []
|
chunks = []
|
||||||
@@ -498,6 +679,12 @@ def chat():
|
|||||||
sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
|
sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
|
||||||
sse_resp.headers["X-Context-Warning"] = ctx_warning
|
sse_resp.headers["X-Context-Warning"] = ctx_warning
|
||||||
sse_resp.headers["X-Context-Model"] = model
|
sse_resp.headers["X-Context-Model"] = model
|
||||||
|
# LiteLLM spend tracking: best-effort token counts from stream timings
|
||||||
|
pt = stream_timings.get("prompt_n", 0) if stream_timings else 0
|
||||||
|
ct = stream_timings.get("predicted_n", 0) if stream_timings else 0
|
||||||
|
sse_resp.headers["X-Usage-Tokens"] = json.dumps({
|
||||||
|
"prompt_tokens": pt, "completion_tokens": ct, "model": model
|
||||||
|
})
|
||||||
return sse_resp
|
return sse_resp
|
||||||
data = clean_response(resp.json())
|
data = clean_response(resp.json())
|
||||||
for c in data.get("choices",[]):
|
for c in data.get("choices",[]):
|
||||||
@@ -522,14 +709,19 @@ def chat():
|
|||||||
resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
|
resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
|
||||||
resp.headers["X-Context-Warning"] = ctx_warning
|
resp.headers["X-Context-Warning"] = ctx_warning
|
||||||
resp.headers["X-Context-Model"] = model
|
resp.headers["X-Context-Model"] = model
|
||||||
|
# LiteLLM spend tracking: return token counts for cost computation
|
||||||
|
resp.headers["X-Usage-Tokens"] = json.dumps({
|
||||||
|
"prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens, "model": model
|
||||||
|
})
|
||||||
bcast()
|
bcast()
|
||||||
return resp
|
return resp
|
||||||
except requests.Timeout:
|
except requests.Timeout:
|
||||||
gpu_decr(model)
|
gpu_release_slot(model)
|
||||||
log.error("TIMEOUT: %s -> %s", agent, model)
|
trip_circuit(model)
|
||||||
|
log.error("TIMEOUT: %s -> %s (Circuit tripped)", agent, model)
|
||||||
return jsonify({"error":"timeout"}), 504
|
return jsonify({"error":"timeout"}), 504
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
gpu_decr(model)
|
gpu_release_slot(model)
|
||||||
log.error("Error: %s\n%s", e, traceback.format_exc())
|
log.error("Error: %s\n%s", e, traceback.format_exc())
|
||||||
return jsonify({"error":str(e)}), 500
|
return jsonify({"error":str(e)}), 500
|
||||||
|
|
||||||
@@ -723,6 +915,105 @@ def metrics_timeseries():
|
|||||||
data["models"][model] = counts
|
data["models"][model] = counts
|
||||||
return jsonify(data)
|
return jsonify(data)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/metrics/circuit-breaker")
|
||||||
|
def metrics_circuit_breaker():
|
||||||
|
"""Expose circuit breaker status per model. Phase 1."""
|
||||||
|
result = {}
|
||||||
|
if r:
|
||||||
|
for model in GPU_URLS:
|
||||||
|
key = "circuit:" + model + ":open"
|
||||||
|
duration = r.ttl(key)
|
||||||
|
trip_count = int(r.get("circuit:" + model + ":count") or 0)
|
||||||
|
result[model] = {
|
||||||
|
"tripped": r.exists(key),
|
||||||
|
"remaining_ttl": duration,
|
||||||
|
"trip_count": trip_count
|
||||||
|
}
|
||||||
|
return jsonify(result)
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/metrics/gpu-health")
|
||||||
|
def metrics_gpu_health():
|
||||||
|
"""Live GPU health scores + circuit breaker + KPIs."""
|
||||||
|
result = {"gpus": [], "ts": time.time()}
|
||||||
|
for model in GPU_URLS:
|
||||||
|
h = check_gpu_health(model, sidecar_timeout=1.5, gpu_timeout=1)
|
||||||
|
score = gpu_health_score(model)
|
||||||
|
active = gpu_active_count(model)
|
||||||
|
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||||
|
cb_tripped = bool(r and r.exists("circuit:" + model + ":open"))
|
||||||
|
cb_count = int(r.get("circuit:" + model + ":count") or 0) if r else 0
|
||||||
|
result["gpus"].append({
|
||||||
|
"id": model,
|
||||||
|
"label": GPU_LABELS.get(model, model),
|
||||||
|
"status": h.get("status", "unknown"),
|
||||||
|
"vram_pct": h.get("vram_pct", 0),
|
||||||
|
"temp_c": h.get("temp_c", 0),
|
||||||
|
"vram_used_mb": h.get("vram_used_mb", 0),
|
||||||
|
"vram_total_mb": h.get("vram_total_mb", 0),
|
||||||
|
"gpu_name": h.get("gpu_name", model),
|
||||||
|
"health_score": round(score, 1),
|
||||||
|
"active_requests": active,
|
||||||
|
"max_concurrent": max_c,
|
||||||
|
"circuit_tripped": cb_tripped,
|
||||||
|
"circuit_trip_count": cb_count
|
||||||
|
})
|
||||||
|
online = sum(1 for g in result["gpus"] if g["status"] in ("healthy", "saturated"))
|
||||||
|
trips = sum(g["circuit_trip_count"] for g in result["gpus"])
|
||||||
|
result["kpi"] = {"gpus_online": online, "total_trips": trips, "total_gpus": len(GPU_URLS)}
|
||||||
|
return jsonify(result)
|
||||||
|
|
||||||
|
@app.route("/metrics/latency")
|
||||||
|
def metrics_latency():
|
||||||
|
"""Lightweight latency summary for dashboard KPIs."""
|
||||||
|
if not r: return jsonify({"avg_ms": 0, "requests_per_min": 0})
|
||||||
|
recent = []
|
||||||
|
for x in (r.lrange("routes:recent", 0, 49) or []):
|
||||||
|
try: recent.append(json.loads(x))
|
||||||
|
except: pass
|
||||||
|
if not recent: return jsonify({"avg_ms": 0, "requests_per_min": 0, "count": 0})
|
||||||
|
now = time.time()
|
||||||
|
last_min = [x for x in recent if now - x.get("ts", 0) < 60]
|
||||||
|
latencies = [x.get("queue_ms", 0) + x.get("inference_ms", 0) for x in last_min if "inference_ms" in x]
|
||||||
|
return jsonify({
|
||||||
|
"avg_ms": round(sum(latencies) / len(latencies), 1) if latencies else 0,
|
||||||
|
"requests_per_min": len(last_min),
|
||||||
|
"count": len(recent)
|
||||||
|
})
|
||||||
|
@app.route("/health/unified")
|
||||||
|
def health_unified():
|
||||||
|
"""Unified health aggregating all layers: Router + Redis + GPUs + Circuit Breaker + Scores."""
|
||||||
|
gpus = {}
|
||||||
|
for m in GPU_URLS:
|
||||||
|
h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
|
||||||
|
h["active_requests"] = gpu_active_count(m)
|
||||||
|
h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
|
||||||
|
h["health_score"] = gpu_health_score(m)
|
||||||
|
h["circuit_open"] = is_circuit_tripped(m)
|
||||||
|
gpus[m] = h
|
||||||
|
circuit_state = {}
|
||||||
|
for m in GPU_URLS:
|
||||||
|
cooldown_until = r.ttl("circuit:" + m + ":open") if r else None
|
||||||
|
circuit_state[m] = {
|
||||||
|
"open": is_circuit_tripped(m),
|
||||||
|
"cooldown_remaining_s": max(0, cooldown_until) if cooldown_until and cooldown_until > 0 else 0,
|
||||||
|
"trip_count": int(r.get("circuit:" + m + ":count") or 0) if r else 0
|
||||||
|
}
|
||||||
|
overall = "healthy"
|
||||||
|
if not r:
|
||||||
|
overall = "degraded"
|
||||||
|
if all(circuit_state[m]["open"] for m in GPU_URLS):
|
||||||
|
overall = "down"
|
||||||
|
return jsonify({
|
||||||
|
"status": overall, "router": "healthy",
|
||||||
|
"redis": "connected" if r else "down",
|
||||||
|
"gpus": gpus, "circuit_breaker": circuit_state,
|
||||||
|
"scores": {m: gpu_health_score(m) for m in GPU_URLS},
|
||||||
|
"available_models": available_models(), "timestamp": time.time()
|
||||||
|
})
|
||||||
|
|
||||||
@app.route("/stream")
|
@app.route("/stream")
|
||||||
def stream():
|
def stream():
|
||||||
def ev():
|
def ev():
|
||||||
@@ -740,6 +1031,119 @@ def stream():
|
|||||||
return Response(stream_with_context(ev()), mimetype="text/event-stream",
|
return Response(stream_with_context(ev()), mimetype="text/event-stream",
|
||||||
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
|
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
|
||||||
|
|
||||||
|
# ── Phase 0: Admin Key Management ──
|
||||||
|
ADMIN_KEY = os.environ.get("ADMIN_KEY", "")
|
||||||
|
|
||||||
|
def _admin_auth():
|
||||||
|
"""Require admin key for management endpoints."""
|
||||||
|
if not ADMIN_KEY:
|
||||||
|
return False, "ADMIN_KEY not configured on server"
|
||||||
|
ak = request.headers.get("Authorization","").replace("Bearer ","")
|
||||||
|
if ak != ADMIN_KEY:
|
||||||
|
return False, "Admin key required"
|
||||||
|
return True, None
|
||||||
|
|
||||||
|
@app.route("/admin/keys")
|
||||||
|
def admin_keys():
|
||||||
|
"""List all API keys (masked) with agent, tier, and deprecation status."""
|
||||||
|
ok, err = _admin_auth()
|
||||||
|
if not ok: return jsonify({"error": err}), 401
|
||||||
|
keys = []
|
||||||
|
for key, info in API_KEYS.items():
|
||||||
|
masked = key[:8] + "..." + key[-8:]
|
||||||
|
keys.append({
|
||||||
|
"masked": masked,
|
||||||
|
"prefix": key[:8],
|
||||||
|
"agent": info["agent"],
|
||||||
|
"tier": info["tier"],
|
||||||
|
"deprecated": info.get("deprecated", False),
|
||||||
|
"length": len(key)
|
||||||
|
})
|
||||||
|
return jsonify({
|
||||||
|
"total": len(keys),
|
||||||
|
"active": sum(1 for k in keys if not k["deprecated"]),
|
||||||
|
"deprecated": sum(1 for k in keys if k["deprecated"]),
|
||||||
|
"keys": sorted(keys, key=lambda k: (k["deprecated"], k["agent"]))
|
||||||
|
})
|
||||||
|
|
||||||
|
@app.route("/admin/keys/deprecation-summary")
|
||||||
|
def admin_deprecation_summary():
|
||||||
|
"""Summary of deprecated key usage (from Redis logs, if available)."""
|
||||||
|
ok, err = _admin_auth()
|
||||||
|
if not ok: return jsonify({"error": err}), 401
|
||||||
|
deprecated_agents = []
|
||||||
|
for key, info in API_KEYS.items():
|
||||||
|
if info.get("deprecated"):
|
||||||
|
# Check Redis for usage count
|
||||||
|
count = 0
|
||||||
|
if r:
|
||||||
|
count = int(r.get("deprecated_usage:" + info["agent"]) or 0)
|
||||||
|
deprecated_agents.append({
|
||||||
|
"agent": info["agent"],
|
||||||
|
"deprecated_uses": count,
|
||||||
|
"needs_migration": count > 0
|
||||||
|
})
|
||||||
|
return jsonify({
|
||||||
|
"deprecated_agents": sorted(deprecated_agents, key=lambda d: -d["deprecated_uses"]),
|
||||||
|
"recommendation": "Run POST /admin/keys/revoke to remove keys with 0 usage"
|
||||||
|
})
|
||||||
|
|
||||||
|
@app.route("/admin/keys/generate", methods=["POST"])
|
||||||
|
def admin_generate_key():
|
||||||
|
"""Generate a new API key for an agent. Body: {"agent": "Name", "tier": "enterprise"}"""
|
||||||
|
ok, err = _admin_auth()
|
||||||
|
if not ok: return jsonify({"error": err}), 401
|
||||||
|
body = request.get_json(force=True)
|
||||||
|
agent = body.get("agent", "").strip()
|
||||||
|
tier = body.get("tier", "enterprise")
|
||||||
|
if not agent:
|
||||||
|
return jsonify({"error": "agent field required"}), 400
|
||||||
|
if tier not in ("starter", "professional", "enterprise"):
|
||||||
|
return jsonify({"error": "tier must be starter/professional/enterprise"}), 400
|
||||||
|
# Generate secure key
|
||||||
|
import secrets, hashlib
|
||||||
|
prefix = hashlib.sha256(secrets.token_bytes(12)).hexdigest()[:8]
|
||||||
|
suffix = secrets.token_hex(20)
|
||||||
|
new_key = f"sk-{prefix}-{suffix}"
|
||||||
|
# Update in-memory dict (note: not persisted across restarts without env var update)
|
||||||
|
API_KEYS[new_key] = {"tier": tier, "agent": agent}
|
||||||
|
log.info("KEY_GENERATED: agent=%s tier=%s key=%s...%s", agent, tier, new_key[:8], new_key[-8:])
|
||||||
|
return jsonify({
|
||||||
|
"agent": agent,
|
||||||
|
"tier": tier,
|
||||||
|
"key": new_key,
|
||||||
|
"masked": new_key[:8] + "..." + new_key[-8:],
|
||||||
|
"warning": "This key exists in memory only. Update API_KEYS env var and redeploy to persist."
|
||||||
|
}), 201
|
||||||
|
|
||||||
|
@app.route("/admin/keys/revoke", methods=["POST"])
|
||||||
|
def admin_revoke_key():
|
||||||
|
"""Revoke a deprecated key. Body: {"agent": "Name"} or {"key_prefix": "sk-xxxx"}"""
|
||||||
|
ok, err = _admin_auth()
|
||||||
|
if not ok: return jsonify({"error": err}), 401
|
||||||
|
body = request.get_json(force=True)
|
||||||
|
agent = body.get("agent", "")
|
||||||
|
key_prefix = body.get("key_prefix", "")
|
||||||
|
revoked = []
|
||||||
|
keys_to_remove = []
|
||||||
|
for key, info in API_KEYS.items():
|
||||||
|
if not info.get("deprecated"):
|
||||||
|
continue
|
||||||
|
if agent and info["agent"] == agent:
|
||||||
|
keys_to_remove.append(key)
|
||||||
|
elif key_prefix and key.startswith(key_prefix):
|
||||||
|
keys_to_remove.append(key)
|
||||||
|
for key in keys_to_remove:
|
||||||
|
info = API_KEYS.pop(key)
|
||||||
|
revoked.append({"agent": info["agent"], "masked": key[:8] + "..." + key[-8:]})
|
||||||
|
log.warning("KEY_REVOKED: agent=%s key=%s...%s", info["agent"], key[:8], key[-8:])
|
||||||
|
return jsonify({
|
||||||
|
"revoked": len(revoked),
|
||||||
|
"keys": revoked,
|
||||||
|
"remaining_total": len(API_KEYS),
|
||||||
|
"warning": "Memory-only revoke. Update API_KEYS env var and redeploy to persist."
|
||||||
|
})
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
log.info("Router on :9000 (load-aware)")
|
log.info("Router on :9000 (load-aware)")
|
||||||
app.run(host="0.0.0.0", port=9000, debug=False)
|
app.run(host="0.0.0.0", port=9000, debug=False)
|
||||||
|
|||||||
@@ -1,342 +0,0 @@
|
|||||||
import os, json, time, logging, traceback, threading, queue
|
|
||||||
import requests, redis
|
|
||||||
from flask import Flask, request, jsonify, Response, stream_with_context
|
|
||||||
|
|
||||||
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
|
|
||||||
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
|
|
||||||
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
|
|
||||||
GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
|
|
||||||
|
|
||||||
GPU_SIDECARS = {
|
|
||||||
"qwen3.6-35B-A3B": "http://192.168.68.15:8090",
|
|
||||||
"qwen3.6-27B-code": "http://192.168.68.8:8090",
|
|
||||||
"qwen3.5-9b-vlm": "http://192.168.68.110:8090",
|
|
||||||
}
|
|
||||||
GPU_URLS = {
|
|
||||||
"qwen3.6-35B-A3B": GPU_MOE_URL,
|
|
||||||
"qwen3.6-27B-code": GPU_DENSE_URL,
|
|
||||||
"qwen3.5-9b-vlm": GPU_LIGHT_URL,
|
|
||||||
}
|
|
||||||
# Max concurrent requests per GPU (based on llama.cpp --parallel)
|
|
||||||
GPU_MAX_CONCURRENT = {
|
|
||||||
"qwen3.6-35B-A3B": 2, # 2 slots
|
|
||||||
"qwen3.6-27B-code": 2, # 2 slots
|
|
||||||
"qwen3.5-9b-vlm": 1, # 1 slot
|
|
||||||
}
|
|
||||||
|
|
||||||
TIER_MODELS = {
|
|
||||||
"starter": ["qwen3.5-9b-vlm"],
|
|
||||||
"professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
|
|
||||||
"enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
|
|
||||||
}
|
|
||||||
API_KEYS = {
|
|
||||||
"sk-syslog-local-master-key": {"tier": "enterprise", "agent": "admin"},
|
|
||||||
"sk-syslog-abiba": {"tier": "enterprise", "agent": "Abiba"},
|
|
||||||
"sk-syslog-mumuni": {"tier": "enterprise", "agent": "Mumuni"},
|
|
||||||
"sk-syslog-tanko": {"tier": "enterprise", "agent": "Tanko"},
|
|
||||||
"sk-syslog-koby": {"tier": "enterprise", "agent": "Koby"},
|
|
||||||
"sk-syslog-kagenz0": {"tier": "enterprise", "agent": "Kagenz0"},
|
|
||||||
"sk-syslog-koonimo": {"tier": "enterprise", "agent": "Koonimo"},
|
|
||||||
"sk-starter-abc123": {"tier": "starter", "agent": "test-starter"},
|
|
||||||
"sk-professional-xyz789": {"tier": "professional", "agent": "test-pro"},
|
|
||||||
}
|
|
||||||
|
|
||||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
|
|
||||||
log = logging.getLogger("router")
|
|
||||||
try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
|
|
||||||
except Exception: r = None
|
|
||||||
|
|
||||||
|
|
||||||
def counter_audit_loop():
|
|
||||||
"""Every 30s, check GPU slots and reset counters if all slots idle."""
|
|
||||||
while True:
|
|
||||||
time.sleep(30)
|
|
||||||
if not r: continue
|
|
||||||
for model, url in GPU_URLS.items():
|
|
||||||
try:
|
|
||||||
resp = requests.get(url.replace("/v1","") + "/slots",
|
|
||||||
headers={"Authorization": "Bearer not-needed"}, timeout=5)
|
|
||||||
if resp.status_code == 200:
|
|
||||||
slots = resp.json()
|
|
||||||
all_idle = all(not s.get("is_processing", False) for s in slots)
|
|
||||||
if all_idle:
|
|
||||||
current = int(r.get("active:" + model) or 0)
|
|
||||||
if current > 0:
|
|
||||||
r.set("active:" + model, 0)
|
|
||||||
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
threading.Thread(target=counter_audit_loop, daemon=True).start()
|
|
||||||
|
|
||||||
app = Flask(__name__)
|
|
||||||
sse_subscribers = []; sse_lock = threading.Lock()
|
|
||||||
|
|
||||||
def gpu_active_count(model):
|
|
||||||
"""Get number of in-flight requests for a GPU."""
|
|
||||||
if r:
|
|
||||||
return int(r.get("active:" + model) or 0)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
def gpu_incr(model):
|
|
||||||
if r: r.incr("active:" + model)
|
|
||||||
|
|
||||||
def gpu_decr(model):
|
|
||||||
if r:
|
|
||||||
v = r.decr("active:" + model)
|
|
||||||
if v and int(v) < 0:
|
|
||||||
r.set("active:" + model, 0) # never go negative
|
|
||||||
|
|
||||||
def check_gpu_health(model):
|
|
||||||
url = GPU_SIDECARS.get(model)
|
|
||||||
if not url: return {"status": "unknown"}
|
|
||||||
try:
|
|
||||||
resp = requests.get(url, timeout=5)
|
|
||||||
if resp.status_code == 200:
|
|
||||||
d = resp.json()
|
|
||||||
pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
|
|
||||||
status = "healthy" if pct < 90 else "saturated"
|
|
||||||
# Also check if llama.cpp endpoint is actually responding
|
|
||||||
gpu_url = GPU_URLS.get(model, "")
|
|
||||||
try:
|
|
||||||
hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=3)
|
|
||||||
if hr.status_code != 200:
|
|
||||||
status = "down"
|
|
||||||
except Exception:
|
|
||||||
status = "down"
|
|
||||||
return {"status": status, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
|
|
||||||
except Exception: pass
|
|
||||||
return {"status": "down"}
|
|
||||||
|
|
||||||
def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
|
|
||||||
|
|
||||||
def estimate_tokens(msgs): return sum(len(str(m.get("content",""))) for m in msgs) // 4
|
|
||||||
|
|
||||||
def is_gpu_busy(model):
|
|
||||||
"""Check if GPU is at or near max concurrent capacity."""
|
|
||||||
active = gpu_active_count(model)
|
|
||||||
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
|
||||||
return active >= max_c
|
|
||||||
|
|
||||||
def select_best_gpu(candidates, reason):
|
|
||||||
"""Pick the best GPU from candidates, preferring least-loaded."""
|
|
||||||
best = None
|
|
||||||
best_load = 999
|
|
||||||
for m in candidates:
|
|
||||||
load = gpu_active_count(m)
|
|
||||||
if load < best_load:
|
|
||||||
best_load = load
|
|
||||||
best = m
|
|
||||||
if best:
|
|
||||||
actual_reason = reason
|
|
||||||
if is_gpu_busy(best):
|
|
||||||
actual_reason = "load_balanced_" + reason
|
|
||||||
return {"model": best, "reason": actual_reason}
|
|
||||||
return None
|
|
||||||
|
|
||||||
def route(rd, tier):
|
|
||||||
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
|
|
||||||
sys = any(m.get("role")=="system" for m in msgs)
|
|
||||||
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
|
|
||||||
hints = rd.get("routing_hints",{})
|
|
||||||
allowed = TIER_MODELS.get(tier, ["qwen3.5-9b-vlm"])
|
|
||||||
avail = [m for m in available_models() if m in allowed]
|
|
||||||
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
|
|
||||||
|
|
||||||
req = rd.get("model","auto")
|
|
||||||
if req != "auto":
|
|
||||||
target = req if req in avail else avail[0]
|
|
||||||
# If explicit model is busy, check if another can take it
|
|
||||||
if is_gpu_busy(target) and req in allowed:
|
|
||||||
alts = [m for m in avail if m != target and m in allowed]
|
|
||||||
if alts:
|
|
||||||
alt = select_best_gpu(alts, "explicit")
|
|
||||||
if alt: return alt
|
|
||||||
return {"model": target, "reason": "explicit"}
|
|
||||||
|
|
||||||
if hints:
|
|
||||||
if hints.get("priority")=="speed" and "qwen3.5-9b-vlm" in avail:
|
|
||||||
return select_best_gpu(["qwen3.5-9b-vlm"], "hint_speed") or {"model":"qwen3.5-9b-vlm","reason":"hint_speed"}
|
|
||||||
if hints.get("priority")=="quality" and "qwen3.6-27B-code" in avail:
|
|
||||||
return select_best_gpu(["qwen3.6-27B-code"], "hint_quality") or {"model":"qwen3.6-27B-code","reason":"hint_quality"}
|
|
||||||
|
|
||||||
# Heavy -> dense (but fall back to MoE if dense is busy)
|
|
||||||
if t > 4000 or sys or turns > 6:
|
|
||||||
candidates = ["qwen3.6-27B-code","qwen3.6-35B-A3B","qwen3.5-9b-vlm"]
|
|
||||||
candidates = [m for m in candidates if m in avail]
|
|
||||||
result = select_best_gpu(candidates, "heavy_reasoning")
|
|
||||||
if result: return result
|
|
||||||
|
|
||||||
# Ultra-light -> VLM
|
|
||||||
first_msg = msgs[0].get("content","") if msgs else ""
|
|
||||||
words = len(first_msg.split()) if isinstance(first_msg, str) else 99
|
|
||||||
if words <= 3 and turns <= 1 and not sys and "qwen3.5-9b-vlm" in avail:
|
|
||||||
if not is_gpu_busy("qwen3.5-9b-vlm"):
|
|
||||||
return {"model":"qwen3.5-9b-vlm","reason":"ultra_light"}
|
|
||||||
|
|
||||||
# Default: MoE, fall back to dense if MoE is busy
|
|
||||||
if "qwen3.6-35B-A3B" in avail:
|
|
||||||
if is_gpu_busy("qwen3.6-35B-A3B") and "qwen3.6-27B-code" in avail:
|
|
||||||
return {"model": "qwen3.6-27B-code", "reason": "load_balanced_default"}
|
|
||||||
return {"model":"qwen3.6-35B-A3B","reason":"default_moe"}
|
|
||||||
|
|
||||||
return {"model":avail[0],"reason":"fallback"}
|
|
||||||
|
|
||||||
def clean_unicode(text):
|
|
||||||
if not isinstance(text, str): return text
|
|
||||||
text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
|
|
||||||
text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
|
|
||||||
text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
|
|
||||||
text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
|
|
||||||
return text.encode("ascii", "ignore").decode("ascii")
|
|
||||||
|
|
||||||
def clean_response(d):
|
|
||||||
if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
|
|
||||||
if isinstance(d, list): return [clean_response(v) for v in d]
|
|
||||||
if isinstance(d, str): return clean_unicode(d)
|
|
||||||
return d
|
|
||||||
|
|
||||||
def get_metrics():
|
|
||||||
d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
|
|
||||||
for m in GPU_URLS:
|
|
||||||
h = check_gpu_health(m)
|
|
||||||
d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
|
|
||||||
d["active_requests"][m] = gpu_active_count(m)
|
|
||||||
if r:
|
|
||||||
try:
|
|
||||||
for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
|
|
||||||
for k,v in API_KEYS.items():
|
|
||||||
c = int(r.get("routes:agent:"+v["agent"]) or 0)
|
|
||||||
if c>0: d["agent_counts"][v["agent"]] = c
|
|
||||||
for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
|
|
||||||
raw = r.lrange("routes:recent",0,49)
|
|
||||||
d["recent"] = [json.loads(x) for x in raw] if raw else []
|
|
||||||
except Exception: pass
|
|
||||||
return d
|
|
||||||
|
|
||||||
def bcast():
|
|
||||||
data = get_metrics(); payload = json.dumps(data)
|
|
||||||
with sse_lock:
|
|
||||||
dead = []
|
|
||||||
for q in sse_subscribers:
|
|
||||||
try: q.put(payload)
|
|
||||||
except Exception: dead.append(q)
|
|
||||||
for q in dead: sse_subscribers.remove(q)
|
|
||||||
|
|
||||||
@app.route("/v1/chat/completions", methods=["POST"])
|
|
||||||
def chat():
|
|
||||||
try:
|
|
||||||
rd = request.get_json(force=True)
|
|
||||||
ak = request.headers.get("Authorization","").replace("Bearer ","")
|
|
||||||
ki = API_KEYS.get(ak, {"tier":"starter","agent":"unknown"})
|
|
||||||
tier, agent = ki["tier"], ki["agent"]
|
|
||||||
d = route(rd, tier)
|
|
||||||
if d.get("saturated"):
|
|
||||||
resp = jsonify({"error": "All GPUs saturated", "retry_after_s": 5})
|
|
||||||
resp.headers["Retry-After"] = "5"
|
|
||||||
return resp, 503
|
|
||||||
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
|
|
||||||
is_stream = rd.get("stream", False)
|
|
||||||
|
|
||||||
gpu_incr(model)
|
|
||||||
decremented = False
|
|
||||||
|
|
||||||
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
|
|
||||||
if r:
|
|
||||||
try:
|
|
||||||
r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
|
|
||||||
r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
|
|
||||||
r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent}))
|
|
||||||
r.ltrim("routes:recent",0,999)
|
|
||||||
except Exception: pass
|
|
||||||
start = time.time()
|
|
||||||
resp = requests.post(url+"/chat/completions", json=rd,
|
|
||||||
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
|
|
||||||
lat = int((time.time()-start)*1000)
|
|
||||||
gpu_decr(model)
|
|
||||||
decremented = True # Release slot
|
|
||||||
|
|
||||||
if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
|
|
||||||
if is_stream:
|
|
||||||
def gen():
|
|
||||||
for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
|
|
||||||
if raw: yield clean_unicode(raw)
|
|
||||||
bcast()
|
|
||||||
return Response(stream_with_context(gen()), mimetype="text/event-stream")
|
|
||||||
data = clean_response(resp.json())
|
|
||||||
for c in data.get("choices",[]):
|
|
||||||
msg = c.get("message",{})
|
|
||||||
if not msg.get("content") and msg.get("reasoning_content"):
|
|
||||||
msg["content"] = msg["reasoning_content"]
|
|
||||||
data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"active_gpu":gpu_active_count(model)}
|
|
||||||
bcast()
|
|
||||||
return jsonify(data)
|
|
||||||
if not decremented:
|
|
||||||
try: gpu_decr(model)
|
|
||||||
except: pass
|
|
||||||
except requests.Timeout:
|
|
||||||
return jsonify({"error":"timeout"}), 504
|
|
||||||
log.error("Error: %s\n%s", e, traceback.format_exc())
|
|
||||||
return jsonify({"error":str(e)}), 500
|
|
||||||
|
|
||||||
@app.route("/v1/models")
|
|
||||||
def models(): return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":check_gpu_health(m).get("status"),"gpu":check_gpu_health(m).get("gpu_name")} for m in GPU_URLS]})
|
|
||||||
|
|
||||||
@app.route("/health")
|
|
||||||
def health():
|
|
||||||
gpus = {}
|
|
||||||
for m in GPU_URLS:
|
|
||||||
h = check_gpu_health(m)
|
|
||||||
h["active_requests"] = gpu_active_count(m)
|
|
||||||
h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
|
|
||||||
gpus[m] = h
|
|
||||||
return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
|
|
||||||
|
|
||||||
@app.route("/metrics")
|
|
||||||
def metrics(): return jsonify(get_metrics())
|
|
||||||
|
|
||||||
@app.route("/metrics/timeseries")
|
|
||||||
def metrics_timeseries():
|
|
||||||
period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
|
|
||||||
data = {"models": {}, "labels": []}
|
|
||||||
if period == "day":
|
|
||||||
buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
|
|
||||||
data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
|
|
||||||
elif period == "week":
|
|
||||||
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
|
|
||||||
data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
|
|
||||||
else:
|
|
||||||
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
|
|
||||||
data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
|
|
||||||
if r:
|
|
||||||
for model in models_list:
|
|
||||||
counts = []
|
|
||||||
for bucket in buckets:
|
|
||||||
total = 0
|
|
||||||
if period in ("week","month"):
|
|
||||||
for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
|
|
||||||
else: total = int(r.get("ts:"+model+":"+bucket) or 0)
|
|
||||||
counts.append(total)
|
|
||||||
data["models"][model] = counts
|
|
||||||
return jsonify(data)
|
|
||||||
|
|
||||||
@app.route("/stream")
|
|
||||||
def stream():
|
|
||||||
def ev():
|
|
||||||
q = queue.Queue()
|
|
||||||
with sse_lock: sse_subscribers.append(q)
|
|
||||||
try:
|
|
||||||
yield "data: "+json.dumps(get_metrics())+"\n\n"
|
|
||||||
while True:
|
|
||||||
try: yield "data: "+q.get(timeout=3)+"\n\n"
|
|
||||||
except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
|
|
||||||
except GeneratorExit: pass
|
|
||||||
finally:
|
|
||||||
with sse_lock:
|
|
||||||
if q in sse_subscribers: sse_subscribers.remove(q)
|
|
||||||
return Response(stream_with_context(ev()), mimetype="text/event-stream",
|
|
||||||
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
log.info("Router on :9000 (load-aware)")
|
|
||||||
app.run(host="0.0.0.0", port=9000, debug=False)
|
|
||||||
Reference in New Issue
Block a user