Phase 0.5: Re-add ADMIN_KEY, /admin/keys endpoints, dual-key logging #1
@@ -0,0 +1,56 @@
|
||||
# LiteLLM Virtual Key Distribution — Phase 2 Cutover
|
||||
## Syslog Solution LLC — July 1, 2026 (keys regenerated)
|
||||
|
||||
### Active Agent Keys (Update agent configs with these)
|
||||
|
||||
| Agent | LiteLLM Virtual Key | Tier | Budget | Models | Verified |
|
||||
|-------|-------------------|------|--------|--------|----------|
|
||||
| **Abiba** | `sk-i7F7rCfgpS0oouOTA2LgMw` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||
| **Mumuni** | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||
| **Tanko** | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||
| **Kagenz0** | `sk-VYKRAVYEG1rKVEMDKEUggg` | enterprise | $1,000 | All 4 | ✅ Jul 1 |
|
||||
| **Koby** | `sk-gWDaPAp-FavgKdqKJpzqhQ` | professional | $500 | All 4 | ✅ Jul 1 |
|
||||
| **Koonimo** | `sk-1kLC8ZxW-tEZueV3NU4rCQ` | professional | $500 | All 4 | ✅ Jul 1 |
|
||||
|
||||
### Configuration Changes Required Per Agent
|
||||
|
||||
Each agent must update their `OPENAI_API_BASE` and `OPENAI_API_KEY`:
|
||||
|
||||
```
|
||||
OPENAI_API_BASE=http://192.168.68.116/v1
|
||||
OPENAI_API_KEY=<agent's LiteLLM key from table above>
|
||||
```
|
||||
|
||||
### LiteLLM Admin Access
|
||||
|
||||
| Resource | Key |
|
||||
|----------|-----|
|
||||
| LiteLLM Admin UI | http://192.168.68.116/ui/ (Authentik SSO — pending) |
|
||||
| LiteLLM Master Key | `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` |
|
||||
| Router Admin Key | `sk-admin-ee09fffd04978b61a1569ac670c68814` |
|
||||
|
||||
### Fallback (if LiteLLM fails)
|
||||
|
||||
If LiteLLM is down, NGINX automatically falls back to the router on :9000.
|
||||
In that case, agents can also directly use:
|
||||
```
|
||||
OPENAI_API_BASE=http://192.168.68.116/v1
|
||||
OPENAI_API_KEY=<agent's original sk-syslog-* key>
|
||||
```
|
||||
(Only works when NGINX @router_fallback is active)
|
||||
|
||||
### Deprecated Router Keys
|
||||
|
||||
These keys still work through the @router_fallback but NOT through LiteLLM:
|
||||
- sk-syslog-abiba, sk-syslog-mumuni, sk-syslog-tanko
|
||||
- sk-syslog-kagenz0, sk-syslog-koby, sk-syslog-koonimo
|
||||
- sk-starter-abc123, sk-professional-xyz789
|
||||
- sk-syslog-local-master-key
|
||||
|
||||
These have been fully revoked as of July 1, 2026.
|
||||
|
||||
### Key Rotation History
|
||||
|
||||
- **2026-07-01**: Keys regenerated for Abiba, Kagenz0, Koby, Koonimo (old values unrecoverable).
|
||||
Mumuni and Tanko keys confirmed working, left unchanged. All blocked/stale keys purged.
|
||||
- **2026-06-17**: Initial key creation (values in this doc were placeholder/incorrect).
|
||||
@@ -1,135 +0,0 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>Syslog GPU Monitor</title>
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #0d1117; color: #c9d1d9; padding: 20px; }
|
||||
h1 { font-size: 22px; margin-bottom: 8px; color: #58a6ff; }
|
||||
.subtitle { color: #8b949e; font-size: 13px; margin-bottom: 24px; }
|
||||
.grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 16px; margin-bottom: 24px; }
|
||||
.card { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 20px; }
|
||||
.card h2 { font-size: 16px; margin-bottom: 12px; color: #f0f6fc; }
|
||||
.card.down { border-color: #da3633; }
|
||||
.card.warn { border-color: #d29922; }
|
||||
.metric { display: flex; justify-content: space-between; padding: 4px 0; font-size: 14px; }
|
||||
.metric .label { color: #8b949e; }
|
||||
.metric .value { font-weight: 600; font-variant-numeric: tabular-nums; }
|
||||
.value.good { color: #3fb950; }
|
||||
.value.warn { color: #d29922; }
|
||||
.value.bad { color: #da3633; }
|
||||
.bar-bg { background: #21262d; border-radius: 4px; height: 8px; margin: 4px 0 8px; overflow: hidden; }
|
||||
.bar { height: 100%; border-radius: 4px; transition: width 0.5s; }
|
||||
.bar.good { background: #3fb950; }
|
||||
.bar.warn { background: #d29922; }
|
||||
.bar.bad { background: #da3633; }
|
||||
.summary { display: grid; grid-template-columns: repeat(4, 1fr); gap: 12px; margin-bottom: 24px; }
|
||||
.stat { background: #161b22; border: 1px solid #30363d; border-radius: 8px; padding: 14px; text-align: center; }
|
||||
.stat .num { font-size: 28px; font-weight: 700; }
|
||||
.stat .lbl { font-size: 11px; color: #8b949e; margin-top: 4px; text-transform: uppercase; letter-spacing: 0.5px; }
|
||||
.status-dot { display: inline-block; width: 10px; height: 10px; border-radius: 50%; margin-right: 6px; }
|
||||
.status-dot.healthy { background: #3fb950; }
|
||||
.status-dot.warning { background: #d29922; }
|
||||
.status-dot.down { background: #da3633; }
|
||||
.circuit { font-size: 12px; padding: 2px 8px; border-radius: 4px; display: inline-block; }
|
||||
.circuit.open { background: #da363322; color: #da3633; border: 1px solid #da363344; }
|
||||
.circuit.closed { background: #3fb95022; color: #3fb950; border: 1px solid #3fb95044; }
|
||||
footer { text-align: center; color: #484f58; font-size: 11px; margin-top: 20px; }
|
||||
.refresh { animation: pulse 0.3s; }
|
||||
@keyframes pulse { 0%{opacity:0.4} 100%{opacity:1} }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>⚡ Syslog GPU Monitor</h1>
|
||||
<p class="subtitle">Real-time GPU health — <span id="updated">loading...</span></p>
|
||||
|
||||
<div class="summary">
|
||||
<div class="stat"><div class="num good" id="gpus-online">-</div><div class="lbl">GPUs Online</div></div>
|
||||
<div class="stat"><div class="num" id="total-requests">-</div><div class="lbl">Active Requests</div></div>
|
||||
<div class="stat"><div class="num warn" id="trip-count">-</div><div class="lbl">Circuit Trips</div></div>
|
||||
<div class="stat"><div class="num" id="slots-used">-</div><div class="lbl">Slots Used</div></div>
|
||||
</div>
|
||||
|
||||
<div class="grid" id="gpu-grid"></div>
|
||||
|
||||
<footer>Syslog Solution LLC — Auto-refreshes every 5s · <span id="last-refresh">—</span></footer>
|
||||
|
||||
<script>
|
||||
const API = '';
|
||||
|
||||
async function fetchJSON(url) {
|
||||
const r = await fetch(url);
|
||||
return r.json();
|
||||
}
|
||||
|
||||
function barClass(pct) {
|
||||
if (pct > 90) return 'bad';
|
||||
if (pct > 75) return 'warn';
|
||||
return 'good';
|
||||
}
|
||||
|
||||
function tempClass(temp) {
|
||||
if (temp > 80) return 'bad';
|
||||
if (temp > 65) return 'warn';
|
||||
return 'good';
|
||||
}
|
||||
|
||||
function scoreClass(score) {
|
||||
if (score > 60) return 'bad';
|
||||
if (score > 35) return 'warn';
|
||||
return 'good';
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
try {
|
||||
const [health, perf] = await Promise.all([
|
||||
fetchJSON(API + '/metrics/gpu-health'),
|
||||
fetchJSON(API + '/metrics/performance?window=1')
|
||||
]);
|
||||
|
||||
// Summary stats
|
||||
document.getElementById('gpus-online').textContent = health.kpi.gpus_online + '/' + health.kpi.total_gpus;
|
||||
document.getElementById('trip-count').textContent = health.kpi.total_trips;
|
||||
document.getElementById('total-requests').textContent = perf.summary?.total_requests || 0;
|
||||
const used = health.gpus.reduce((s,g) => s + g.active_requests, 0);
|
||||
const total = health.gpus.reduce((s,g) => s + g.max_concurrent, 0);
|
||||
document.getElementById('slots-used').textContent = used + '/' + total;
|
||||
|
||||
// GPU cards
|
||||
const grid = document.getElementById('gpu-grid');
|
||||
grid.innerHTML = health.gpus.map(g => {
|
||||
const statusClass = g.status === 'healthy' ? 'healthy' : g.status === 'down' ? 'down' : 'warning';
|
||||
const circuitLabel = g.circuit_tripped ? 'OPEN' : 'closed';
|
||||
const circuitClass = g.circuit_tripped ? 'open' : 'closed';
|
||||
const cardClass = g.circuit_tripped ? 'warn' : g.status === 'down' ? 'down' : '';
|
||||
const activeLabel = `${g.active_requests}/${g.max_concurrent}`;
|
||||
const name = g.label || g.id;
|
||||
|
||||
return `
|
||||
<div class="card ${cardClass}">
|
||||
<h2><span class="status-dot ${statusClass}"></span>${name}</h2>
|
||||
<div class="metric"><span class="label">Health Score</span><span class="value ${scoreClass(g.health_score)}">${g.health_score}</span></div>
|
||||
<div class="metric"><span class="label">VRAM</span><span class="value ${barClass(g.vram_pct)}">${g.vram_pct}%</span></div>
|
||||
<div class="bar-bg"><div class="bar ${barClass(g.vram_pct)}" style="width:${g.vram_pct}%"></div></div>
|
||||
<div class="metric"><span class="label">Temperature</span><span class="value ${tempClass(g.temp_c)}">${g.temp_c}°C</span></div>
|
||||
<div class="bar-bg"><div class="bar ${tempClass(g.temp_c)}" style="width:${Math.min(g.temp_c,100)}%"></div></div>
|
||||
<div class="metric"><span class="label">Active</span><span class="value">${activeLabel}</span></div>
|
||||
<div class="metric"><span class="label">Circuit</span><span class="circuit ${circuitClass}">${circuitLabel}</span></div>
|
||||
<div class="metric"><span class="label">Trips</span><span class="value">${g.circuit_trip_count}</span></div>
|
||||
</div>`;
|
||||
}).join('');
|
||||
|
||||
document.getElementById('updated').textContent = new Date().toLocaleTimeString();
|
||||
document.getElementById('last-refresh').textContent = 'Last refresh: ' + new Date().toLocaleTimeString();
|
||||
} catch(e) {
|
||||
console.error(e);
|
||||
}
|
||||
}
|
||||
|
||||
refresh();
|
||||
setInterval(refresh, 5000);
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
+13
-53
@@ -1,4 +1,4 @@
|
||||
version: "3.8"
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
redis:
|
||||
@@ -16,37 +16,18 @@ services:
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: harness-postgres
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
- POSTGRES_DB=litellm
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
network_mode: host
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- REDIS_URL=redis://127.0.0.1:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
- API_KEYS={"sk-sys...-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e6...cb64":{"tier":"enterprise","agent":"admin"},"***":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856...a889":{"tier":"enterprise","agent":"Abiba"},"***":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57...807e":{"tier":"enterprise","agent":"Mumuni"},"***":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620...eaa7":{"tier":"enterprise","agent":"Tanko"},"***":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3...fdee":{"tier":"enterprise","agent":"Koby"},"***":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b...ed9b":{"tier":"enterprise","agent":"Kagenz0"},"***":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680...4dfe":{"tier":"enterprise","agent":"Koonimo"},"***":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55d...7860":{"tier":"starter","agent":"test-starter"},"sk-pro...z789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b51...e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-adm...8814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
@@ -57,47 +38,24 @@ services:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
|
||||
image: ghcr.io/berriai/litellm:main-stable
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4001:4000"
|
||||
- "127.0.0.1:8081:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
- /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
|
||||
- ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
|
||||
- DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_UI_USERNAME=admin
|
||||
- LITELLM_UI_PASSWORD=syslog-admin-2026
|
||||
- UI_USERNAME=admin
|
||||
- UI_PASSWORD=syslog-admin-2026
|
||||
- OPENAI_API_KEY=not-used
|
||||
- PROXY_BASE_URL=https://litellm.sysloggh.net
|
||||
- DOCS_URL=/docs
|
||||
- ANTHROPIC_API_KEY=not-used
|
||||
- GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
|
||||
- GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
|
||||
- GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
|
||||
- GENERIC_TOKEN_ENDPOINT=http://harness-nginx/application/o/token/
|
||||
- GENERIC_USERINFO_ENDPOINT=http://harness-nginx/application/o/userinfo/
|
||||
- GENERIC_SCOPE=openid email profile
|
||||
- GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
|
||||
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
|
||||
- LITELLM_MASTER_KEY=sk-sys...-key
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
@@ -111,7 +69,7 @@ services:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
extra_hosts:
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
- "host.docker.internal:host-gateway"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
@@ -128,7 +86,7 @@ services:
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- REDIS_URL=redis://127.0.0.1:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
@@ -140,4 +98,6 @@ services:
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
pgdata:
|
||||
|
||||
# LiteLLM command override to load config
|
||||
# (appended to fix config loading issue)
|
||||
|
||||
@@ -1,102 +0,0 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: harness-redis
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: ghcr.io/berriai/litellm:main-stable
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:8081:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-sys...-key
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: harness-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- litellm
|
||||
- dashboard
|
||||
|
||||
dashboard:
|
||||
build: ./dashboard
|
||||
container_name: harness-dashboard
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
|
||||
# LiteLLM command override to load config
|
||||
# (appended to fix config loading issue)
|
||||
@@ -1,131 +0,0 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: harness-redis
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: harness-postgres
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
- POSTGRES_DB=litellm
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: ghcr.io/berriai/litellm:main-stable
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4000:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
|
||||
- ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
|
||||
- DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_UI_USERNAME=admin
|
||||
- LITELLM_UI_PASSWORD=syslog-admin-2026
|
||||
- OPENAI_API_KEY=not-used
|
||||
- PROXY_BASE_URL=http://192.168.68.116/litellm
|
||||
- ANTHROPIC_API_KEY=not-used
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: harness-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
extra_hosts:
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- litellm
|
||||
- dashboard
|
||||
|
||||
dashboard:
|
||||
build: ./dashboard
|
||||
container_name: harness-dashboard
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
pgdata:
|
||||
@@ -1,142 +0,0 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: harness-redis
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: harness-postgres
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
- POSTGRES_DB=litellm
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4001:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
- /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
|
||||
- ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
|
||||
- DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_UI_USERNAME=admin
|
||||
- LITELLM_UI_PASSWORD=syslog-admin-2026
|
||||
- UI_USERNAME=admin
|
||||
- UI_PASSWORD=syslog-admin-2026
|
||||
- OPENAI_API_KEY=not-used
|
||||
- PROXY_BASE_URL=https://litellm.sysloggh.net
|
||||
- DOCS_URL=/litellm/docs
|
||||
- ANTHROPIC_API_KEY=not-used
|
||||
- GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
|
||||
- GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
|
||||
- GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
|
||||
- GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
|
||||
- GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
|
||||
- GENERIC_SCOPE=openid email profile
|
||||
- GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
|
||||
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: harness-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
extra_hosts:
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- litellm
|
||||
- dashboard
|
||||
|
||||
dashboard:
|
||||
build: ./dashboard
|
||||
container_name: harness-dashboard
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
pgdata:
|
||||
@@ -1,143 +0,0 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: harness-redis
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: harness-postgres
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
- POSTGRES_DB=litellm
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4001:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
- /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
|
||||
- ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
|
||||
- DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_UI_USERNAME=admin
|
||||
- LITELLM_UI_PASSWORD=syslog-admin-2026
|
||||
- UI_USERNAME=admin
|
||||
- UI_PASSWORD=syslog-admin-2026
|
||||
- OPENAI_API_KEY=not-used
|
||||
- PROXY_BASE_URL=https://litellm.sysloggh.net
|
||||
- DOCS_URL=/litellm/docs
|
||||
- ANTHROPIC_API_KEY=not-used
|
||||
- GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
|
||||
- GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
|
||||
- GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
|
||||
- GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
|
||||
- GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
|
||||
- GENERIC_SCOPE=openid email profile
|
||||
- GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
|
||||
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: harness-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
extra_hosts:
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- litellm
|
||||
- dashboard
|
||||
|
||||
dashboard:
|
||||
build: ./dashboard
|
||||
container_name: harness-dashboard
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
pgdata:
|
||||
@@ -1,142 +0,0 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: harness-redis
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: harness-postgres
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
- POSTGRES_DB=litellm
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4001:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
- /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
|
||||
- ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
|
||||
- DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_UI_USERNAME=admin
|
||||
- LITELLM_UI_PASSWORD=syslog-admin-2026
|
||||
- UI_USERNAME=admin
|
||||
- UI_PASSWORD=syslog-admin-2026
|
||||
- OPENAI_API_KEY=not-used
|
||||
- PROXY_BASE_URL=https://litellm.sysloggh.net
|
||||
- ANTHROPIC_API_KEY=not-used
|
||||
- GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
|
||||
- GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
|
||||
- GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
|
||||
- GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
|
||||
- GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
|
||||
- GENERIC_SCOPE=openid email profile
|
||||
- GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
|
||||
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: harness-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
extra_hosts:
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- litellm
|
||||
- dashboard
|
||||
|
||||
dashboard:
|
||||
build: ./dashboard
|
||||
container_name: harness-dashboard
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
pgdata:
|
||||
@@ -1,143 +0,0 @@
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: harness-redis
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis-data:/data
|
||||
command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: harness-postgres
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:5432:5432"
|
||||
environment:
|
||||
- POSTGRES_DB=litellm
|
||||
- POSTGRES_USER=litellm
|
||||
- POSTGRES_PASSWORD=d9fc143e3dc1a7a8e672c359fea95c5e
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U litellm"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
router:
|
||||
build: ./router
|
||||
container_name: harness-router
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:9000:9000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_MOE_URL=http://192.168.68.15:8080/v1
|
||||
- GPU_DENSE_URL=http://192.168.68.8:8080/v1
|
||||
- GPU_LIGHT_URL=http://192.168.68.110:8080/v1
|
||||
- API_KEYS={"sk-syslog-local-master-key":{"tier":"enterprise","agent":"admin","deprecated":true},"sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64":{"tier":"enterprise","agent":"admin"},"sk-syslog-abiba":{"tier":"enterprise","agent":"Abiba","deprecated":true},"sk-856ffb0bbb-e5aaf78b10054eca608f8fbcbd73a889":{"tier":"enterprise","agent":"Abiba"},"sk-syslog-mumuni":{"tier":"enterprise","agent":"Mumuni","deprecated":true},"sk-b57e6e042e-47573660114f3138c852c47f62da807e":{"tier":"enterprise","agent":"Mumuni"},"sk-syslog-tanko":{"tier":"enterprise","agent":"Tanko","deprecated":true},"sk-620a05e95a-e93d875476b650a4d1137249ead8eaa7":{"tier":"enterprise","agent":"Tanko"},"sk-syslog-koby":{"tier":"enterprise","agent":"Koby","deprecated":true},"sk-eb3e6fc1c0-de1bf2edf35a53cb3749a2400483fdee":{"tier":"enterprise","agent":"Koby"},"sk-syslog-kagenz0":{"tier":"enterprise","agent":"Kagenz0","deprecated":true},"sk-12b66b3392-b548aed9138aeb6f698e8e521650ed9b":{"tier":"enterprise","agent":"Kagenz0"},"sk-syslog-koonimo":{"tier":"enterprise","agent":"Koonimo","deprecated":true},"sk-680d06686c-00ee8bf9dc3c93b276af122d49a14dfe":{"tier":"enterprise","agent":"Koonimo"},"sk-starter-abc123":{"tier":"starter","agent":"test-starter","deprecated":true},"sk-55da55907a-1bd7ff344e26feda50e9ac2219697860":{"tier":"starter","agent":"test-starter"},"sk-professional-xyz789":{"tier":"professional","agent":"test-pro","deprecated":true},"sk-b5159863e6-8df3ae52fb958cfe76cc2888c8c8e676":{"tier":"professional","agent":"test-pro"}}
|
||||
- ADMIN_KEY=sk-admin-ee09fffd04978b61a1569ac670c68814
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/health')"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
litellm:
|
||||
image: docker.litellm.ai/berriai/litellm:1.90.0-rc.1
|
||||
command: ["--config", "/app/config.yaml", "--port", "4000"]
|
||||
container_name: harness-litellm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "4001:4000"
|
||||
volumes:
|
||||
- ./litellm_config.yaml:/app/config.yaml
|
||||
- /opt/combined-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro
|
||||
environment:
|
||||
- LITELLM_MASTER_KEY=sk-litellm-7f96080dd99b15c36bd4b333b58a6796
|
||||
- ROUTER_API_KEY=sk-9e65b69a67-e54af421c1b09fb8bd4f75dacb38cb64
|
||||
- DATABASE_URL=postgresql://litellm:d9fc143e3dc1a7a8e672c359fea95c5e@postgres:5432/litellm
|
||||
- STORE_MODEL_IN_DB=True
|
||||
- LITELLM_UI_USERNAME=admin
|
||||
- LITELLM_UI_PASSWORD=syslog-admin-2026
|
||||
- UI_USERNAME=admin
|
||||
- UI_PASSWORD=syslog-admin-2026
|
||||
- OPENAI_API_KEY=not-used
|
||||
- PROXY_BASE_URL=https://litellm.sysloggh.net
|
||||
- DOCS_URL=/docs
|
||||
- ANTHROPIC_API_KEY=not-used
|
||||
- GENERIC_CLIENT_ID=FHd7bs9dP5gHad2Ki23iUL5kQvFa0GRaj3nlLnNU
|
||||
- GENERIC_CLIENT_SECRET=aDkXQx82duqpxc98xxqp0quzzUf1mawsnOTqj7sx1acaS7rWSt02N5ksBCi92n8ZilRavigoYME6fLakP20Ixc9H2pxnSZFiOqQLb7BPi8UtsvfxmzXklD0HJIdbKFxe
|
||||
- GENERIC_AUTHORIZATION_ENDPOINT=https://auth.sysloggh.net/application/o/authorize/
|
||||
- GENERIC_TOKEN_ENDPOINT=https://auth.sysloggh.net/application/o/token/
|
||||
- GENERIC_USERINFO_ENDPOINT=https://auth.sysloggh.net/application/o/userinfo/
|
||||
- GENERIC_SCOPE=openid email profile
|
||||
- GENERIC_ROLE_MAPPINGS_DEFAULT_ROLE=proxy_admin
|
||||
- SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: harness-nginx
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "80:80"
|
||||
volumes:
|
||||
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
- ./dashboard:/opt/inference-harness/dashboard:ro
|
||||
extra_hosts:
|
||||
- "auth.sysloggh.net:192.168.68.11"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://127.0.0.1/health"]
|
||||
interval: 30s
|
||||
timeout: 15s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- litellm
|
||||
- dashboard
|
||||
|
||||
dashboard:
|
||||
build: ./dashboard
|
||||
container_name: harness-dashboard
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- GPU_SIDECARS=192.168.68.15:8090,192.168.68.8:8090,192.168.68.110:8090
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3000/health')"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
volumes:
|
||||
redis-data:
|
||||
pgdata:
|
||||
@@ -0,0 +1,280 @@
|
||||
{
|
||||
"communities": {
|
||||
"0": [
|
||||
"router_route_v2",
|
||||
"router_route_v2_route",
|
||||
"router_route_v3",
|
||||
"router_route_v3_moe_spillover",
|
||||
"router_route_v3_rationale_81",
|
||||
"router_route_v3_route",
|
||||
"router_router_available_models",
|
||||
"router_router_estimate_tokens",
|
||||
"router_router_is_gpu_busy",
|
||||
"router_router_moe_spillover",
|
||||
"router_router_rationale_180",
|
||||
"router_router_rationale_216",
|
||||
"router_router_rationale_239",
|
||||
"router_router_rationale_386",
|
||||
"router_router_route",
|
||||
"router_router_select_best_gpu"
|
||||
],
|
||||
"1": [
|
||||
"dashboard_dashboard",
|
||||
"dashboard_dashboard_api_performance",
|
||||
"dashboard_dashboard_api_scatter",
|
||||
"dashboard_dashboard_api_state",
|
||||
"dashboard_dashboard_api_stream",
|
||||
"dashboard_dashboard_api_timeseries",
|
||||
"dashboard_dashboard_broadcast_loop",
|
||||
"dashboard_dashboard_dashboard",
|
||||
"dashboard_dashboard_fetch_state",
|
||||
"dashboard_dashboard_health",
|
||||
"dashboard_dashboard_rationale_1"
|
||||
],
|
||||
"2": [
|
||||
"queue_service_queue_service",
|
||||
"queue_service_queue_service_check_gpu_health",
|
||||
"queue_service_queue_service_enqueue",
|
||||
"queue_service_queue_service_get_queue_depth",
|
||||
"queue_service_queue_service_get_redis",
|
||||
"queue_service_queue_service_health",
|
||||
"queue_service_queue_service_rationale_100",
|
||||
"queue_service_queue_service_rationale_62",
|
||||
"queue_service_queue_service_rationale_68",
|
||||
"queue_service_queue_service_status"
|
||||
],
|
||||
"3": [
|
||||
"router_router_admin_auth",
|
||||
"router_router_admin_deprecation_summary",
|
||||
"router_router_admin_generate_key",
|
||||
"router_router_admin_keys",
|
||||
"router_router_admin_revoke_key",
|
||||
"router_router_rationale_895",
|
||||
"router_router_rationale_905",
|
||||
"router_router_rationale_928",
|
||||
"router_router_rationale_950",
|
||||
"router_router_rationale_978"
|
||||
],
|
||||
"4": [
|
||||
"router_router",
|
||||
"router_router_bcast",
|
||||
"router_router_get_metrics",
|
||||
"router_router_metrics",
|
||||
"router_router_metrics_circuit_breaker",
|
||||
"router_router_metrics_timeseries",
|
||||
"router_router_models",
|
||||
"router_router_rationale_811",
|
||||
"router_router_stream"
|
||||
],
|
||||
"5": [
|
||||
"router_router_get_redis",
|
||||
"router_router_gpu_decr",
|
||||
"router_router_gpu_incr",
|
||||
"router_router_half_open_probe",
|
||||
"router_router_is_circuit_tripped",
|
||||
"router_router_performance",
|
||||
"router_router_rationale_282",
|
||||
"router_router_rationale_297",
|
||||
"router_router_rationale_619"
|
||||
],
|
||||
"6": [
|
||||
"router_router_check_gpu_health",
|
||||
"router_router_gpu_active_count",
|
||||
"router_router_gpu_health_score",
|
||||
"router_router_health",
|
||||
"router_router_metrics_gpu_health",
|
||||
"router_router_rationale_141",
|
||||
"router_router_rationale_225",
|
||||
"router_router_rationale_828"
|
||||
],
|
||||
"7": [
|
||||
"router_router_chat",
|
||||
"router_router_check_rate_limit",
|
||||
"router_router_clean_response",
|
||||
"router_router_clean_unicode",
|
||||
"router_router_rationale_184",
|
||||
"router_router_rationale_72",
|
||||
"router_router_store_perf_record"
|
||||
],
|
||||
"8": [
|
||||
"maintenance",
|
||||
"maintenance_sh__entry"
|
||||
],
|
||||
"9": [
|
||||
"router_router_counter_audit_loop",
|
||||
"router_router_rationale_116"
|
||||
],
|
||||
"10": [
|
||||
"router_router_metrics_latency",
|
||||
"router_router_rationale_859"
|
||||
],
|
||||
"11": [
|
||||
"router_router_rationale_288",
|
||||
"router_router_trip_circuit"
|
||||
],
|
||||
"12": [
|
||||
"router_router_rationale_736",
|
||||
"router_router_scatter"
|
||||
],
|
||||
"13": [
|
||||
"router_http_patch"
|
||||
]
|
||||
},
|
||||
"cohesion": {
|
||||
"0": 0.20833333333333334,
|
||||
"1": 0.21818181818181817,
|
||||
"2": 0.3111111111111111,
|
||||
"3": 0.2,
|
||||
"4": 0.2777777777777778,
|
||||
"5": 0.2222222222222222,
|
||||
"6": 0.35714285714285715,
|
||||
"7": 0.3333333333333333,
|
||||
"8": 1.0,
|
||||
"9": 1.0,
|
||||
"10": 1.0,
|
||||
"11": 1.0,
|
||||
"12": 1.0,
|
||||
"13": 1.0
|
||||
},
|
||||
"gods": [
|
||||
{
|
||||
"id": "router_router_chat",
|
||||
"label": "chat()",
|
||||
"degree": 12
|
||||
},
|
||||
{
|
||||
"id": "router_router_get_redis",
|
||||
"label": "get_redis()",
|
||||
"degree": 11
|
||||
},
|
||||
{
|
||||
"id": "router_router_gpu_active_count",
|
||||
"label": "gpu_active_count()",
|
||||
"degree": 9
|
||||
},
|
||||
{
|
||||
"id": "router_router_is_gpu_busy",
|
||||
"label": "is_gpu_busy()",
|
||||
"degree": 9
|
||||
},
|
||||
{
|
||||
"id": "router_router_select_best_gpu",
|
||||
"label": "select_best_gpu()",
|
||||
"degree": 8
|
||||
},
|
||||
{
|
||||
"id": "router_router_route",
|
||||
"label": "route()",
|
||||
"degree": 8
|
||||
},
|
||||
{
|
||||
"id": "router_route_v3_route",
|
||||
"label": "route()",
|
||||
"degree": 6
|
||||
},
|
||||
{
|
||||
"id": "router_router_check_gpu_health",
|
||||
"label": "check_gpu_health()",
|
||||
"degree": 6
|
||||
},
|
||||
{
|
||||
"id": "router_router_available_models",
|
||||
"label": "available_models()",
|
||||
"degree": 6
|
||||
},
|
||||
{
|
||||
"id": "router_router_estimate_tokens",
|
||||
"label": "estimate_tokens()",
|
||||
"degree": 6
|
||||
}
|
||||
],
|
||||
"surprises": [
|
||||
{
|
||||
"source": "route()",
|
||||
"target": "available_models()",
|
||||
"source_files": [
|
||||
"router/route_v2.py",
|
||||
"router/router.py"
|
||||
],
|
||||
"confidence": "INFERRED",
|
||||
"relation": "calls",
|
||||
"why": "inferred connection - not explicitly stated in source"
|
||||
},
|
||||
{
|
||||
"source": "route()",
|
||||
"target": "estimate_tokens()",
|
||||
"source_files": [
|
||||
"router/route_v2.py",
|
||||
"router/router.py"
|
||||
],
|
||||
"confidence": "INFERRED",
|
||||
"relation": "calls",
|
||||
"why": "inferred connection - not explicitly stated in source"
|
||||
},
|
||||
{
|
||||
"source": "route()",
|
||||
"target": "is_gpu_busy()",
|
||||
"source_files": [
|
||||
"router/route_v2.py",
|
||||
"router/router.py"
|
||||
],
|
||||
"confidence": "INFERRED",
|
||||
"relation": "calls",
|
||||
"why": "inferred connection - not explicitly stated in source"
|
||||
},
|
||||
{
|
||||
"source": "route()",
|
||||
"target": "select_best_gpu()",
|
||||
"source_files": [
|
||||
"router/route_v2.py",
|
||||
"router/router.py"
|
||||
],
|
||||
"confidence": "INFERRED",
|
||||
"relation": "calls",
|
||||
"why": "inferred connection - not explicitly stated in source"
|
||||
},
|
||||
{
|
||||
"source": "route()",
|
||||
"target": "available_models()",
|
||||
"source_files": [
|
||||
"router/route_v3.py",
|
||||
"router/router.py"
|
||||
],
|
||||
"confidence": "INFERRED",
|
||||
"relation": "calls",
|
||||
"why": "inferred connection - not explicitly stated in source"
|
||||
}
|
||||
],
|
||||
"questions": [
|
||||
{
|
||||
"type": "bridge_node",
|
||||
"question": "Why does `is_gpu_busy()` connect `Community 0` to `Community 4`, `Community 6`?",
|
||||
"why": "High betweenness centrality (0.061) - this node is a cross-community bridge."
|
||||
},
|
||||
{
|
||||
"type": "bridge_node",
|
||||
"question": "Why does `select_best_gpu()` connect `Community 0` to `Community 4`, `Community 6`?",
|
||||
"why": "High betweenness centrality (0.031) - this node is a cross-community bridge."
|
||||
},
|
||||
{
|
||||
"type": "bridge_node",
|
||||
"question": "Why does `estimate_tokens()` connect `Community 0` to `Community 4`, `Community 7`?",
|
||||
"why": "High betweenness centrality (0.030) - this node is a cross-community bridge."
|
||||
},
|
||||
{
|
||||
"type": "verify_inferred",
|
||||
"question": "Are the 3 inferred relationships involving `is_gpu_busy()` (e.g. with `route()` and `moe_spillover()`) actually correct?",
|
||||
"why": "`is_gpu_busy()` has 3 INFERRED edges - model-reasoned connections that need verification."
|
||||
},
|
||||
{
|
||||
"type": "verify_inferred",
|
||||
"question": "Are the 3 inferred relationships involving `select_best_gpu()` (e.g. with `route()` and `route()`) actually correct?",
|
||||
"why": "`select_best_gpu()` has 3 INFERRED edges - model-reasoned connections that need verification."
|
||||
},
|
||||
{
|
||||
"type": "isolated_nodes",
|
||||
"question": "What connects `SyslogAI Harness Dashboard \u2014 Modern Design.`, `maintenance.sh script`, `Nginx upstream health probe. Returns 200 if service is alive.` to the rest of the system?",
|
||||
"why": "28 weakly-connected nodes found - possible documentation gaps or missing edges."
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
||||
{"files": {"code": ["/root/syslog-harness-repo/dashboard/dashboard.py", "/root/syslog-harness-repo/maintenance.sh", "/root/syslog-harness-repo/phase0-dual-keys.json", "/root/syslog-harness-repo/queue-service/queue-service.py", "/root/syslog-harness-repo/router/http_patch.py", "/root/syslog-harness-repo/router/route_v2.py", "/root/syslog-harness-repo/router/route_v3.py", "/root/syslog-harness-repo/router/router.py"], "document": ["/root/syslog-harness-repo/LITELLM-MIGRATION-PLAN.md", "/root/syslog-harness-repo/MIGRATION_PLAN.md", "/root/syslog-harness-repo/README.md", "/root/syslog-harness-repo/dashboard/dashboard.html", "/root/syslog-harness-repo/dashboard/harness.html", "/root/syslog-harness-repo/dashboard/requirements.txt", "/root/syslog-harness-repo/docker-compose.yml", "/root/syslog-harness-repo/litellm_config.yaml", "/root/syslog-harness-repo/router/requirements.txt", "/root/syslog-harness-repo/ssl/README.md"], "paper": [], "image": [], "video": []}, "total_files": 18, "total_words": 14667, "needs_graph": false, "warning": "Corpus is ~14,667 words - fits in a single context window. You may not need a graph.", "skipped_sensitive": ["/root/syslog-harness-repo/.env.example"], "unclassified": ["/root/syslog-harness-repo/.gitignore", "/root/syslog-harness-repo/Dockerfile.dashboard", "/root/syslog-harness-repo/Dockerfile.queue", "/root/syslog-harness-repo/backups/20260602_103344/dashboard.py.bak", "/root/syslog-harness-repo/backups/20260602_103344/router.py.bak", "/root/syslog-harness-repo/backups/20260602_103344/ts_patch.py.bak", "/root/syslog-harness-repo/dashboard/Dockerfile", "/root/syslog-harness-repo/docker-compose.yml.bak", "/root/syslog-harness-repo/gpu-router-docker.conf", "/root/syslog-harness-repo/gpu-router.conf", "/root/syslog-harness-repo/nginx/nginx.conf", "/root/syslog-harness-repo/nginx/nginx.conf.bak", "/root/syslog-harness-repo/router/Dockerfile", "/root/syslog-harness-repo/router/router.py.bak.20260518074236"], "graphifyignore_patterns": 3, "scan_root": "/root/syslog-harness-repo"}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
||||
{"nodes": [], "edges": [], "hyperedges": [], "input_tokens": 0, "output_tokens": 0}
|
||||
@@ -0,0 +1,106 @@
|
||||
# Graph Report - . (2026-07-08)
|
||||
|
||||
## Corpus Check
|
||||
- Corpus is ~14,667 words - fits in a single context window. You may not need a graph.
|
||||
|
||||
## Summary
|
||||
- 91 nodes · 151 edges · 14 communities (9 shown, 5 thin omitted)
|
||||
- Extraction: 93% EXTRACTED · 7% INFERRED · 0% AMBIGUOUS · INFERRED: 10 edges (avg confidence: 0.77)
|
||||
- Token cost: 0 input · 0 output
|
||||
|
||||
## Community Hubs (Navigation)
|
||||
- Community 0
|
||||
- Community 1
|
||||
- Community 2
|
||||
- Community 3
|
||||
- Community 4
|
||||
- Community 5
|
||||
- Community 6
|
||||
- Community 7
|
||||
- Community 8
|
||||
- Community 9
|
||||
- Community 10
|
||||
- Community 11
|
||||
- Community 12
|
||||
|
||||
## God Nodes (most connected - your core abstractions)
|
||||
1. `chat()` - 12 edges
|
||||
2. `get_redis()` - 11 edges
|
||||
3. `gpu_active_count()` - 9 edges
|
||||
4. `is_gpu_busy()` - 9 edges
|
||||
5. `select_best_gpu()` - 8 edges
|
||||
6. `route()` - 8 edges
|
||||
7. `route()` - 6 edges
|
||||
8. `check_gpu_health()` - 6 edges
|
||||
9. `available_models()` - 6 edges
|
||||
10. `estimate_tokens()` - 6 edges
|
||||
|
||||
## Surprising Connections (you probably didn't know these)
|
||||
- `route()` --calls--> `available_models()` [INFERRED]
|
||||
router/route_v2.py → router/router.py
|
||||
- `route()` --calls--> `estimate_tokens()` [INFERRED]
|
||||
router/route_v2.py → router/router.py
|
||||
- `route()` --calls--> `is_gpu_busy()` [INFERRED]
|
||||
router/route_v2.py → router/router.py
|
||||
- `route()` --calls--> `select_best_gpu()` [INFERRED]
|
||||
router/route_v2.py → router/router.py
|
||||
- `route()` --calls--> `available_models()` [INFERRED]
|
||||
router/route_v3.py → router/router.py
|
||||
|
||||
## Import Cycles
|
||||
- None detected.
|
||||
|
||||
## Communities (14 total, 5 thin omitted)
|
||||
|
||||
### Community 0 - "Community 0"
|
||||
Cohesion: 0.21
|
||||
Nodes (14): route(), moe_spillover(), Spill 40% of MoE-first traffic to Dense to prevent Strix Halo overheating. O, route(), available_models(), estimate_tokens(), is_gpu_busy(), moe_spillover() (+6 more)
|
||||
|
||||
### Community 1 - "Community 1"
|
||||
Cohesion: 0.22
|
||||
Nodes (4): api_state(), broadcast_loop(), fetch_state(), SyslogAI Harness Dashboard — Modern Design.
|
||||
|
||||
### Community 2 - "Community 2"
|
||||
Cohesion: 0.31
|
||||
Nodes (9): check_gpu_health(), enqueue(), get_queue_depth(), get_redis(), health(), GET queue depth + circuit breaker state + GPU health., Nginx upstream health probe. Returns 200 if service is alive., Fallback endpoint — Nginx calls this when all GPU upstreams are down. (+1 more)
|
||||
|
||||
### Community 3 - "Community 3"
|
||||
Cohesion: 0.20
|
||||
Nodes (10): _admin_auth(), admin_deprecation_summary(), admin_generate_key(), admin_keys(), admin_revoke_key(), Require admin key for management endpoints., List all API keys (masked) with agent, tier, and deprecation status., Summary of deprecated key usage (from Redis logs, if available). (+2 more)
|
||||
|
||||
### Community 4 - "Community 4"
|
||||
Cohesion: 0.28
|
||||
Nodes (5): bcast(), get_metrics(), metrics(), metrics_circuit_breaker(), Expose circuit breaker status per model. Phase 1.
|
||||
|
||||
### Community 5 - "Community 5"
|
||||
Cohesion: 0.22
|
||||
Nodes (9): get_redis(), gpu_decr(), gpu_incr(), half_open_probe(), is_circuit_tripped(), performance(), Check if a GPU host is currently blacklisted., Check if a GPU host can be un-blacklisted. (+1 more)
|
||||
|
||||
### Community 6 - "Community 6"
|
||||
Cohesion: 0.36
|
||||
Nodes (8): check_gpu_health(), gpu_active_count(), gpu_health_score(), health(), metrics_gpu_health(), Get number of in-flight requests for a GPU., Score a GPU based on VRAM, temperature, and load. Lower = better., Live GPU health scores + circuit breaker + KPIs.
|
||||
|
||||
### Community 7 - "Community 7"
|
||||
Cohesion: 0.33
|
||||
Nodes (7): chat(), check_rate_limit(), clean_response(), clean_unicode(), Store detailed performance record in Redis for analytics., Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remainin, store_perf_record()
|
||||
|
||||
## Knowledge Gaps
|
||||
- **1 isolated node(s):** `maintenance.sh script`
|
||||
These have ≤1 connection - possible missing edges or undocumented components.
|
||||
- **5 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
|
||||
|
||||
## Suggested Questions
|
||||
_Questions this graph is uniquely positioned to answer:_
|
||||
|
||||
- **Why does `is_gpu_busy()` connect `Community 0` to `Community 4`, `Community 6`?**
|
||||
_High betweenness centrality (0.061) - this node is a cross-community bridge._
|
||||
- **Why does `select_best_gpu()` connect `Community 0` to `Community 4`, `Community 6`?**
|
||||
_High betweenness centrality (0.031) - this node is a cross-community bridge._
|
||||
- **Why does `estimate_tokens()` connect `Community 0` to `Community 4`, `Community 7`?**
|
||||
_High betweenness centrality (0.030) - this node is a cross-community bridge._
|
||||
- **Are the 3 inferred relationships involving `is_gpu_busy()` (e.g. with `route()` and `moe_spillover()`) actually correct?**
|
||||
_`is_gpu_busy()` has 3 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **Are the 3 inferred relationships involving `select_best_gpu()` (e.g. with `route()` and `route()`) actually correct?**
|
||||
_`select_best_gpu()` has 3 INFERRED edges - model-reasoned connections that need verification._
|
||||
- **What connects `SyslogAI Harness Dashboard — Modern Design.`, `maintenance.sh script`, `Nginx upstream health probe. Returns 200 if service is alive.` to the rest of the system?**
|
||||
_28 weakly-connected nodes found - possible documentation gaps or missing edges._
|
||||
graphify-out/cache/ast/v0.9.10/30b202626b1f50da90b78272001e253cc7086b0bdc52e46553414c5313e3fb4e.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/53edfda5145db02bc35b9e9d8e06e5c731ce50c3a50cec3db65b3b886bed95a6.json
Vendored
+1
@@ -0,0 +1 @@
|
||||
{"nodes": [{"id": "root_syslog_harness_repo_router_http_patch_py", "label": "http_patch.py", "file_type": "code", "source_file": "router/http_patch.py", "source_location": "L1"}], "edges": [{"source": "root_syslog_harness_repo_router_http_patch_py", "target": "re", "relation": "imports", "context": "import", "confidence": "EXTRACTED", "source_file": "router/http_patch.py", "source_location": "L2", "weight": 1.0}], "raw_calls": []}
|
||||
graphify-out/cache/ast/v0.9.10/654e97aa6df1872b0717f32f676c067d15b9795a0dc04a86936730a7e133c214.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/801feb2c0b7f4052cb2c4ef933d295d3494c8791766687b891c04cc7b5495d09.json
Vendored
+1
@@ -0,0 +1 @@
|
||||
{"nodes": [{"id": "root_syslog_harness_repo_maintenance_sh", "label": "maintenance.sh", "file_type": "code", "source_file": "maintenance.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "file"}}, {"id": "root_syslog_harness_repo_maintenance_sh__entry", "label": "maintenance.sh script", "file_type": "code", "source_file": "maintenance.sh", "source_location": "L1", "metadata": {"language": "bash", "kind": "bash_entrypoint"}}], "edges": [{"source": "root_syslog_harness_repo_maintenance_sh", "target": "root_syslog_harness_repo_maintenance_sh__entry", "relation": "contains", "confidence": "EXTRACTED", "source_file": "maintenance.sh", "source_location": "L1", "weight": 1.0}]}
|
||||
graphify-out/cache/ast/v0.9.10/9c6817ae09f6bb5b2862f0282cacab7d4ce5da051e4ddc95e2e4099c7bf09716.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/c9a9d2e5c9f1fb7d35a73f91c83a047ce92dc2d09c9ddab8fc2d6e82d92a093d.json
Vendored
+1
File diff suppressed because one or more lines are too long
graphify-out/cache/ast/v0.9.10/e96ac2ae879e0a876d4190daff4cc566d13798b97c4820f3efa47e06627bbbcb.json
Vendored
+1
File diff suppressed because one or more lines are too long
Vendored
+1
@@ -0,0 +1 @@
|
||||
{"/root/syslog-harness-repo/LITELLM-MIGRATION-PLAN.md":{"size":31561,"mtime_ns":1781468608211508160,"word_count":3762},"/root/syslog-harness-repo/MIGRATION_PLAN.md":{"size":1966,"mtime_ns":1781121183850310435,"word_count":303},"/root/syslog-harness-repo/README.md":{"size":2447,"mtime_ns":1781121183850310435,"word_count":377},"/root/syslog-harness-repo/dashboard/dashboard.html":{"size":8331,"mtime_ns":1781316813360201927,"word_count":415},"/root/syslog-harness-repo/dashboard/dashboard.py":{"size":29010,"mtime_ns":1781121183851178603,"word_count":1615,"hash":"654e97aa6df1872b0717f32f676c067d15b9795a0dc04a86936730a7e133c214"},"/root/syslog-harness-repo/dashboard/harness.html":{"size":23388,"mtime_ns":1781316813362649901,"word_count":1869},"/root/syslog-harness-repo/dashboard/requirements.txt":{"size":30,"mtime_ns":1781121183851178603,"word_count":2},"/root/syslog-harness-repo/docker-compose.yml":{"size":3900,"mtime_ns":1782224658037786319,"word_count":200},"/root/syslog-harness-repo/litellm_config.yaml":{"size":618,"mtime_ns":1781121183851178603,"word_count":39},"/root/syslog-harness-repo/maintenance.sh":{"size":1302,"mtime_ns":1781121183851178603,"word_count":192,"hash":"801feb2c0b7f4052cb2c4ef933d295d3494c8791766687b891c04cc7b5495d09"},"/root/syslog-harness-repo/phase0-dual-keys.json":{"size":1628,"mtime_ns":1781121183851610673,"word_count":110,"hash":"0f115313f2c86df2f57e48bdf180b768d6452d0337bae4e3c3d2b52b9f5267a7"},"/root/syslog-harness-repo/queue-service/queue-service.py":{"size":3284,"mtime_ns":1781121183851610673,"word_count":325,"hash":"c9a9d2e5c9f1fb7d35a73f91c83a047ce92dc2d09c9ddab8fc2d6e82d92a093d"},"/root/syslog-harness-repo/router/http_patch.py":{"size":3562,"mtime_ns":1781121183851788825,"word_count":290,"hash":"53edfda5145db02bc35b9e9d8e06e5c731ce50c3a50cec3db65b3b886bed95a6"},"/root/syslog-harness-repo/router/requirements.txt":{"size":43,"mtime_ns":1781121183851788825,"word_count":3},"/root/syslog-harness-repo/router/route_v2.py":{"size":3954,"mtime_ns":1781121183851788825,"word_count":435,"hash":"e96ac2ae879e0a876d4190daff4cc566d13798b97c4820f3efa47e06627bbbcb"},"/root/syslog-harness-repo/router/route_v3.py":{"size":4703,"mtime_ns":1781121183851788825,"word_count":504,"hash":"30b202626b1f50da90b78272001e253cc7086b0bdc52e46553414c5313e3fb4e"},"/root/syslog-harness-repo/router/router.py":{"size":44307,"mtime_ns":1781316813360201927,"word_count":4198,"hash":"9c6817ae09f6bb5b2862f0282cacab7d4ce5da051e4ddc95e2e4099c7bf09716"},"/root/syslog-harness-repo/ssl/README.md":{"size":204,"mtime_ns":1781121183851788825,"word_count":28}}
|
||||
File diff suppressed because it is too large
Load Diff
+18
-86
@@ -1,89 +1,21 @@
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
store_model_in_db: true
|
||||
guardrails:
|
||||
- guardrail_name: input-moderation
|
||||
litellm_params:
|
||||
guardrail: openai_moderation
|
||||
mode: pre_call
|
||||
- guardrail_name: output-moderation
|
||||
litellm_params:
|
||||
guardrail: openai_moderation
|
||||
mode: post_call
|
||||
- guardrail_name: harmful-content-filter
|
||||
litellm_params:
|
||||
categories:
|
||||
- action: BLOCK
|
||||
category: harmful_self_harm
|
||||
enabled: true
|
||||
severity_threshold: medium
|
||||
- action: BLOCK
|
||||
category: harmful_violence
|
||||
enabled: true
|
||||
severity_threshold: medium
|
||||
- action: BLOCK
|
||||
category: harmful_illegal_weapons
|
||||
enabled: true
|
||||
severity_threshold: medium
|
||||
guardrail: litellm_content_filter
|
||||
mode: pre_call
|
||||
litellm_settings:
|
||||
failure_callback:
|
||||
- prometheus
|
||||
model_cost:
|
||||
gemma-4-12b:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
qwen3.6-27B-code:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
qwen3.6-35B-A3B:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
syslog-auto:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
num_retries: 0
|
||||
request_timeout: 600
|
||||
set_verbose: true
|
||||
sso_callback: /sso/callback
|
||||
model_list:
|
||||
- litellm_params:
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
model: openai/syslog-auto
|
||||
rpm: 600
|
||||
model_name: syslog-auto
|
||||
- litellm_params:
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
- model_name: qwen3.6-35B-A3B
|
||||
litellm_params:
|
||||
model: openai/qwen3.6-35B-A3B
|
||||
model_name: qwen3.6-35B-A3B
|
||||
- litellm_params:
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
model: openai/qwen3.6-27B-code
|
||||
model_name: qwen3.6-27B-code
|
||||
- litellm_params:
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
api_base: http://192.168.68.15:8080/v1
|
||||
api_key: not-needed
|
||||
- model_name: gpu-dense
|
||||
litellm_params:
|
||||
model: openai/qwen3.6-27B-code-text
|
||||
api_base: http://192.168.68.8:8080/v1
|
||||
api_key: not-needed
|
||||
- model_name: gpu-light
|
||||
litellm_params:
|
||||
model: openai/gemma-4-12b
|
||||
model_name: gemma-4-12b
|
||||
router_settings:
|
||||
allowed_fails: 100
|
||||
enable_loadbalancing_on_proxy: false
|
||||
fallbacks:
|
||||
- syslog-auto:
|
||||
- qwen3.6-35B-A3B
|
||||
- qwen3.6-27B-code
|
||||
- gemma-4-12b
|
||||
- qwen3.6-35B-A3B:
|
||||
- qwen3.6-27B-code
|
||||
- gemma-4-12b
|
||||
- qwen3.6-27B-code:
|
||||
- qwen3.6-35B-A3B
|
||||
- gemma-4-12b
|
||||
- gemma-4-12b:
|
||||
- qwen3.6-27B-code
|
||||
- qwen3.6-35B-A3B
|
||||
routing_strategy: usage-based-routing
|
||||
api_base: http://192.168.68.110:8080/v1
|
||||
api_key: not-needed
|
||||
general_settings:
|
||||
master_key: sk-syslog-local-master-key
|
||||
litellm_settings:
|
||||
drop_params: true
|
||||
request_timeout: 120
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
# LiteLLM Gateway Configuration — Layer 1 of 2-Layer Architecture
|
||||
# Deployed on CT 116 (192.168.68.116) alongside custom router on :9000
|
||||
# Last updated: 2026-06-16
|
||||
|
||||
general_settings:
|
||||
master_key: os.environ/LITELLM_MASTER_KEY
|
||||
# database_url: using DATABASE_URL env var instead
|
||||
store_model_in_db: true
|
||||
|
||||
model_list:
|
||||
# Content-based auto-routing (router picks GPU via 5-tier analysis)
|
||||
- model_name: syslog-auto
|
||||
litellm_params:
|
||||
model: openai/syslog-auto
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
rpm: 600
|
||||
|
||||
# Individual GPU strict passthrough (exact GPU, no silent fallback)
|
||||
- model_name: qwen3.6-35B-A3B
|
||||
litellm_params:
|
||||
model: openai/qwen3.6-35B-A3B
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
|
||||
- model_name: qwen3.6-27B-code
|
||||
litellm_params:
|
||||
model: openai/qwen3.6-27B-code
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
|
||||
- model_name: gemma-4-12b
|
||||
litellm_params:
|
||||
model: openai/gemma-4-12b
|
||||
api_base: http://router:9000/v1
|
||||
api_key: os.environ/ROUTER_API_KEY
|
||||
|
||||
# Guardrails: Pre-call and post-call content moderation
|
||||
guardrails:
|
||||
- guardrail_name: "input-moderation"
|
||||
litellm_params:
|
||||
guardrail: openai_moderation
|
||||
mode: "pre_call"
|
||||
|
||||
- guardrail_name: "output-moderation"
|
||||
litellm_params:
|
||||
guardrail: openai_moderation
|
||||
mode: "post_call"
|
||||
|
||||
- guardrail_name: "harmful-content-filter"
|
||||
litellm_params:
|
||||
guardrail: litellm_content_filter
|
||||
mode: "pre_call"
|
||||
categories:
|
||||
- category: "harmful_self_harm"
|
||||
enabled: true
|
||||
action: "BLOCK"
|
||||
severity_threshold: "medium"
|
||||
- category: "harmful_violence"
|
||||
enabled: true
|
||||
action: "BLOCK"
|
||||
severity_threshold: "medium"
|
||||
- category: "harmful_illegal_weapons"
|
||||
enabled: true
|
||||
action: "BLOCK"
|
||||
severity_threshold: "medium"
|
||||
|
||||
litellm_settings:
|
||||
num_retries: 0 # Disabled — our router handles retry logic
|
||||
request_timeout: 600 # Match 10-min llama-server timeout
|
||||
set_verbose: true
|
||||
failure_callback: ["prometheus"] # Export metrics to Prometheus
|
||||
|
||||
router_settings:
|
||||
routing_strategy: "usage-based-routing" # For external models only
|
||||
enable_loadbalancing_on_proxy: false # Disable LiteLLM internal LB
|
||||
allowed_fails: 100 # Router returns 503 on saturated GPUs
|
||||
# Fallback chains: LiteLLM retries down the chain when router returns saturated.
|
||||
# This gives accurate per-model metrics because router no longer silently reroutes.
|
||||
# The router's circuit breaker prevents cascading failures to dead GPUs.
|
||||
fallbacks:
|
||||
- syslog-auto: ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"]
|
||||
- qwen3.6-35B-A3B: ["qwen3.6-27B-code", "gemma-4-12b"]
|
||||
- qwen3.6-27B-code: ["qwen3.6-35B-A3B", "gemma-4-12b"]
|
||||
- gemma-4-12b: ["qwen3.6-27B-code", "qwen3.6-35B-A3B"]
|
||||
|
||||
# Cost tracking for spend analytics (local GPUs at $0, symbolic rates optional)
|
||||
litellm_settings:
|
||||
model_cost:
|
||||
syslog-auto:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
qwen3.6-35B-A3B:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
qwen3.6-27B-code:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
gemma-4-12b:
|
||||
input_cost_per_token: 0.0
|
||||
output_cost_per_token: 0.0
|
||||
# For internal cost allocation, set symbolic rates:
|
||||
# e.g., MoE = $2/M tokens, Dense = $1/M tokens, VLM = $0.50/M tokens
|
||||
|
||||
# SSO/OIDC Configuration
|
||||
litellm_settings:
|
||||
sso_callback: "/sso/callback"
|
||||
+56
-112
@@ -1,51 +1,39 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
worker_processes auto;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
pid /var/run/nginx.pid;
|
||||
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
||||
'$status $body_bytes_sent "$http_referer" '
|
||||
'"$http_user_agent" rt=$request_time';
|
||||
access_log /var/log/nginx/access.log main;
|
||||
error_log /var/log/nginx/error.log;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
# Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
|
||||
# Without this, nginx resolves upstream hostnames at config load time,
|
||||
# which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
|
||||
resolver 127.0.0.11 valid=30s;
|
||||
upstream router_api { server host.docker.internal:9000; }
|
||||
upstream dashboard_ui { server dashboard:3000; }
|
||||
upstream litellm_backend { server litellm:4000; }
|
||||
|
||||
# Dynamic upstream resolution via nginx variables.
|
||||
# Using $var in proxy_pass forces request-time resolution through the resolver.
|
||||
# Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
|
||||
map $host $router_api_url {
|
||||
default http://harness-router:9000;
|
||||
}
|
||||
map $host $dashboard_ui_url {
|
||||
default http://harness-dashboard:3000;
|
||||
}
|
||||
map $host $litellm_backend_url {
|
||||
default http://harness-litellm:4000;
|
||||
}
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :80 — harness entrypoint
|
||||
# dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
|
||||
# router fallback, health
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Authentik OIDC subrequest
|
||||
location /authentik/auth {
|
||||
internal;
|
||||
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
# Security headers
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
add_header X-Frame-Options SAMEORIGIN always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
|
||||
# 2-Layer: /v1/ → router (existing keys work unchanged)
|
||||
# Disable buffering for SSE streams
|
||||
proxy_buffering off;
|
||||
|
||||
# API through router
|
||||
location /v1/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
@@ -53,122 +41,78 @@ http {
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
error_page 502 503 = @router_fallback;
|
||||
}
|
||||
|
||||
location @router_fallback {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# SSE streaming endpoint
|
||||
location /stream {
|
||||
proxy_pass $router_api_url/stream;
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Connection "";
|
||||
proxy_buffering off;
|
||||
chunked_transfer_encoding off;
|
||||
}
|
||||
|
||||
# Dashboard API proxy for SSE
|
||||
location /api/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_pass http://dashboard_ui;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
|
||||
|
||||
location /dashboard/ {
|
||||
proxy_pass $dashboard_ui_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM redirect target /litellm (no trailing slash) -> add slash back
|
||||
location = /litellm {
|
||||
return 301 /litellm/;
|
||||
}
|
||||
|
||||
# LiteLLM static assets (Next.js chunks, CSS, fonts)
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 60s;
|
||||
}
|
||||
|
||||
# LiteLLM admin UI and API proxy — strip /litellm prefix so /litellm/ui/ → /ui/
|
||||
location /litellm/ {
|
||||
rewrite ^/litellm(/.*)$ $1 break;
|
||||
proxy_pass $litellm_backend_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Auth proxy to Authentik — accepts HTTP from LiteLLM, proxies HTTPS to .11 with SSL verify off
|
||||
location /application/o/ {
|
||||
proxy_pass https://192.168.68.11;
|
||||
proxy_ssl_verify off;
|
||||
proxy_set_header Host auth.sysloggh.net;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 30s;
|
||||
# LiteLLM debug
|
||||
location /litellm/ {
|
||||
rewrite ^/litellm/(.*) /$1 break;
|
||||
proxy_pass http://litellm_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# All other requests → 404
|
||||
# Professional Dashboard (Phase 1-3) - Static HTML served via Nginx
|
||||
location /dashboard/ {
|
||||
alias /opt/inference-harness/dashboard/;
|
||||
index dashboard.html;
|
||||
add_header Cache-Control "public, max-age=3600";
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
}
|
||||
|
||||
# Legacy Dashboard (root) - Proxy to Flask app
|
||||
location / {
|
||||
return 404;
|
||||
proxy_pass http://dashboard_ui;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Performance analytics
|
||||
location /metrics/ {
|
||||
proxy_pass $router_api_url/metrics/;
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Circuit Breaker metrics (Phase 1)
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass $router_api_url/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /router/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass $router_api_url/health/unified;
|
||||
proxy_pass http://router_api/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass $router_api_url/health;
|
||||
proxy_pass http://router_api/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
worker_processes auto;
|
||||
error_log /var/log/nginx/error.log warn;
|
||||
pid /var/run/nginx.pid;
|
||||
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
||||
'$status $body_bytes_sent "$http_referer" '
|
||||
'"$http_user_agent" rt=$request_time';
|
||||
access_log /var/log/nginx/access.log main;
|
||||
error_log /var/log/nginx/error.log;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
upstream router_api { server router:9000; }
|
||||
upstream dashboard_ui { server dashboard:3000; }
|
||||
upstream litellm_backend { server litellm:4000; }
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Security headers
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
add_header X-Frame-Options SAMEORIGIN always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
|
||||
# Disable buffering for SSE streams
|
||||
proxy_buffering off;
|
||||
|
||||
# API through router
|
||||
location /v1/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# SSE streaming endpoint
|
||||
location /stream {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Connection "";
|
||||
proxy_buffering off;
|
||||
chunked_transfer_encoding off;
|
||||
}
|
||||
|
||||
# Dashboard API proxy for SSE
|
||||
location /api/ {
|
||||
proxy_pass http://dashboard_ui;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# LiteLLM debug
|
||||
location /litellm/ {
|
||||
rewrite ^/litellm/(.*) /$1 break;
|
||||
proxy_pass http://litellm_backend;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# Professional Dashboard (Phase 1-3) - Static HTML served via Nginx
|
||||
location /dashboard/ {
|
||||
alias /opt/inference-harness/dashboard/;
|
||||
index dashboard.html;
|
||||
add_header Cache-Control "public, max-age=3600";
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
}
|
||||
|
||||
# Legacy Dashboard (root) - Proxy to Flask app
|
||||
location / {
|
||||
proxy_pass http://dashboard_ui;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Performance analytics
|
||||
location /metrics/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Circuit Breaker metrics (Phase 1)
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass http://router_api/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass http://router_api/health/unified;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass http://router_api/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
upstream router_api { server router:9000; }
|
||||
upstream dashboard_ui { server dashboard:3000; }
|
||||
upstream litellm_backend { server litellm:4000; }
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Authentik OIDC subrequest
|
||||
location /authentik/auth {
|
||||
internal;
|
||||
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# 2-Layer: LiteLLM (Layer 1) -> Router (Layer 2) -> GPUs
|
||||
# /v1/ routes to router (all existing keys work)
|
||||
# Agents using new LiteLLM keys: change OPENAI_API_BASE to /litellm/v1
|
||||
location /v1/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
error_page 502 503 = @router_fallback;
|
||||
}
|
||||
|
||||
location @router_fallback {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /stream {
|
||||
proxy_pass http://router_api/stream;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass http://router_api/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM gateway access (for agents with new virtual keys)
|
||||
location /litellm/v1/ {
|
||||
proxy_pass http://litellm_backend/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass http://litellm_backend/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI (WebSocket support for live updates)
|
||||
location /litellm/ {
|
||||
proxy_pass http://litellm_backend/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
# Rewrite all redirects to include /litellm/ prefix
|
||||
proxy_redirect http://172.18.0.7:4000/ /litellm/;
|
||||
proxy_redirect http://127.0.0.1:4000/ /litellm/;
|
||||
}
|
||||
|
||||
location /dashboard/ {
|
||||
proxy_pass http://dashboard_ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location / {
|
||||
root /opt/inference-harness/dashboard;
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
location /metrics/ {
|
||||
proxy_pass http://router_api/metrics/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass http://router_api/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /router/ {
|
||||
proxy_pass http://router_api/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /ui/ {
|
||||
return 302 http://192.168.68.116:4000/ui/;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass http://router_api/health/unified;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /ui/ {
|
||||
return 302 http://192.168.68.116:4000/ui/;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass http://router_api/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,351 +0,0 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
# Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
|
||||
# Without this, nginx resolves upstream hostnames at config load time,
|
||||
# which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
|
||||
resolver 127.0.0.11 valid=30s;
|
||||
|
||||
# Dynamic upstream resolution via nginx variables.
|
||||
# Using $var in proxy_pass forces request-time resolution through the resolver.
|
||||
# Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
|
||||
map $host $router_api_url {
|
||||
default http://harness-router:9000;
|
||||
}
|
||||
map $host $dashboard_ui_url {
|
||||
default http://harness-dashboard:3000;
|
||||
}
|
||||
map $host $litellm_backend_url {
|
||||
default http://harness-litellm:4000;
|
||||
}
|
||||
|
||||
# Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
|
||||
# Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
|
||||
# preventing the cross-origin localStorage footgun that traps the UI at the login page.
|
||||
map $http_cf_connecting_ip $is_cloudflared {
|
||||
default 1; # any non-empty value = request came through Cloudflare
|
||||
"" 0; # empty = direct access
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :80 — existing harness entrypoint
|
||||
# dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
|
||||
# router fallback, LiteLLM UI via /litellm/ prefix, health
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Authentik OIDC subrequest
|
||||
location /authentik/auth {
|
||||
internal;
|
||||
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# 2-Layer: /v1/ → router (existing keys work unchanged)
|
||||
location /v1/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
error_page 502 503 = @router_fallback;
|
||||
}
|
||||
|
||||
location @router_fallback {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /stream {
|
||||
proxy_pass $router_api_url/stream;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM gateway access (agents with new virtual keys)
|
||||
location /litellm/v1/ {
|
||||
proxy_pass $litellm_backend_url/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI at /ui/ (public access via Traefik → port 80)
|
||||
# LiteLLM Admin UI at /ui/ (must be before catch-all /)
|
||||
location /ui/ {
|
||||
proxy_pass http://litellm_backend/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
error_page 301 302 = @ui_redirect;
|
||||
}
|
||||
|
||||
location @ui_redirect {
|
||||
proxy_pass http://litellm_backend/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass $litellm_backend_url/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets on port 80
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
|
||||
location /litellm/ {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://$host/ /litellm/;
|
||||
proxy_redirect https://$host/ /litellm/;
|
||||
}
|
||||
|
||||
location /dashboard/ {
|
||||
proxy_pass $dashboard_ui_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Dedicated /openapi.json block — must come before catch-all /
|
||||
# LiteLLM serves valid openapi: 3.1.0 JSON at this path internally,
|
||||
# but the catch-all location / strips the URI path. This block
|
||||
# preserves the full path so the spec JSON is returned instead of
|
||||
# the Swagger UI SPA HTML.
|
||||
location /openapi.json {
|
||||
proxy_pass $litellm_backend_url/openapi.json;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /metrics/ {
|
||||
proxy_pass $router_api_url/metrics/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass $router_api_url/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /router/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass $router_api_url/health/unified;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass $router_api_url/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :4000 — external LiteLLM entrypoint (cloudflared target)
|
||||
# Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
|
||||
# builds external URLs instead of leaking 192.168.68.116:4000.
|
||||
# LiteLLM itself is now bound to 127.0.0.1 only (see compose).
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 4000;
|
||||
|
||||
# Canonical external identity — overrides whatever Host cloudflared sends
|
||||
proxy_set_header Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
|
||||
# /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
|
||||
# so a relative return would be absolutized to http://192.168.68.116:4000/).
|
||||
location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
|
||||
|
||||
# LiteLLM Admin UI + WebSocket
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect $litellm_backend_url/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
|
||||
# UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass $litellm_backend_url/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# API
|
||||
location /v1/ {
|
||||
proxy_pass $litellm_backend_url/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Key management + admin API
|
||||
location /key/ {
|
||||
proxy_pass $litellm_backend_url/key/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /user/ {
|
||||
proxy_pass $litellm_backend_url/user/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /model/ {
|
||||
proxy_pass $litellm_backend_url/model/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /team/ {
|
||||
proxy_pass $litellm_backend_url/team/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# Health
|
||||
location /health {
|
||||
proxy_pass $litellm_backend_url/health;
|
||||
proxy_http_version 1.1;
|
||||
}
|
||||
|
||||
# Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
|
||||
location / {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,262 +0,0 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
upstream router_api { server router:9000; }
|
||||
upstream dashboard_ui { server dashboard:3000; }
|
||||
upstream litellm_backend { server litellm:4000; }
|
||||
|
||||
# Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
|
||||
# Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
|
||||
# preventing the cross-origin localStorage footgun that traps the UI at the login page.
|
||||
map $http_cf_connecting_ip $is_cloudflared {
|
||||
default 1; # any non-empty value = request came through Cloudflare
|
||||
"" 0; # empty = direct access
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :80 — existing harness entrypoint
|
||||
# dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
|
||||
# router fallback, LiteLLM UI via /litellm/ prefix, health
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Authentik OIDC subrequest
|
||||
location /authentik/auth {
|
||||
internal;
|
||||
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# 2-Layer: /v1/ → router (existing keys work unchanged)
|
||||
location /v1/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
error_page 502 503 = @router_fallback;
|
||||
}
|
||||
|
||||
location @router_fallback {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass http://router_api;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /stream {
|
||||
proxy_pass http://router_api/stream;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass http://router_api/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM gateway access (agents with new virtual keys)
|
||||
location /litellm/v1/ {
|
||||
proxy_pass http://litellm_backend/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass http://litellm_backend/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
|
||||
location /litellm/ {
|
||||
proxy_pass http://litellm_backend/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://$host/ /litellm/;
|
||||
proxy_redirect https://$host/ /litellm/;
|
||||
}
|
||||
|
||||
location /dashboard/ {
|
||||
proxy_pass http://dashboard_ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location / {
|
||||
root /opt/inference-harness/dashboard;
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
location /metrics/ {
|
||||
proxy_pass http://router_api/metrics/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass http://router_api/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /router/ {
|
||||
proxy_pass http://router_api/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass http://router_api/health/unified;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass http://router_api/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :4000 — external LiteLLM entrypoint (cloudflared target)
|
||||
# Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
|
||||
# builds external URLs instead of leaking 192.168.68.116:4000.
|
||||
# LiteLLM itself is now bound to 127.0.0.1 only (see compose).
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 4000;
|
||||
|
||||
# Canonical external identity — overrides whatever Host cloudflared sends
|
||||
proxy_set_header Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
|
||||
# /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
|
||||
# so a relative return would be absolutized to http://192.168.68.116:4000/).
|
||||
location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
|
||||
|
||||
# LiteLLM Admin UI + WebSocket
|
||||
location /ui/ {
|
||||
proxy_pass http://litellm_backend/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://litellm_backend/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
|
||||
# UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass http://litellm_backend/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass http://litellm_backend/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# API
|
||||
location /v1/ {
|
||||
proxy_pass http://litellm_backend/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Key management + admin API
|
||||
location /key/ {
|
||||
proxy_pass http://litellm_backend/key/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /user/ {
|
||||
proxy_pass http://litellm_backend/user/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /model/ {
|
||||
proxy_pass http://litellm_backend/model/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /team/ {
|
||||
proxy_pass http://litellm_backend/team/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# Health
|
||||
location /health {
|
||||
proxy_pass http://litellm_backend/health;
|
||||
proxy_http_version 1.1;
|
||||
}
|
||||
|
||||
# Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
|
||||
location / {
|
||||
proxy_pass http://litellm_backend/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,351 +0,0 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
# Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
|
||||
# Without this, nginx resolves upstream hostnames at config load time,
|
||||
# which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
|
||||
resolver 127.0.0.11 valid=30s;
|
||||
|
||||
# Dynamic upstream resolution via nginx variables.
|
||||
# Using $var in proxy_pass forces request-time resolution through the resolver.
|
||||
# Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
|
||||
map $host $router_api_url {
|
||||
default http://harness-router:9000;
|
||||
}
|
||||
map $host $dashboard_ui_url {
|
||||
default http://harness-dashboard:3000;
|
||||
}
|
||||
map $host $litellm_backend_url {
|
||||
default http://harness-litellm:4000;
|
||||
}
|
||||
|
||||
# Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
|
||||
# Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
|
||||
# preventing the cross-origin localStorage footgun that traps the UI at the login page.
|
||||
map $http_cf_connecting_ip $is_cloudflared {
|
||||
default 1; # any non-empty value = request came through Cloudflare
|
||||
"" 0; # empty = direct access
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :80 — existing harness entrypoint
|
||||
# dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
|
||||
# router fallback, LiteLLM UI via /litellm/ prefix, health
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Authentik OIDC subrequest
|
||||
location /authentik/auth {
|
||||
internal;
|
||||
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# 2-Layer: /v1/ → router (existing keys work unchanged)
|
||||
location /v1/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
error_page 502 503 = @router_fallback;
|
||||
}
|
||||
|
||||
location @router_fallback {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /stream {
|
||||
proxy_pass $router_api_url/stream;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM gateway access (agents with new virtual keys)
|
||||
location /litellm/v1/ {
|
||||
proxy_pass $litellm_backend_url/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI at /ui/ (public access via Traefik → port 80)
|
||||
# LiteLLM Admin UI at /ui/ (must be before catch-all /)
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
error_page 301 302 = @ui_redirect;
|
||||
}
|
||||
|
||||
location @ui_redirect {
|
||||
proxy_pass $litellm_backend_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass $litellm_backend_url/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets on port 80
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
|
||||
location /litellm/ {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://$host/ /litellm/;
|
||||
proxy_redirect https://$host/ /litellm/;
|
||||
}
|
||||
|
||||
location /dashboard/ {
|
||||
proxy_pass $dashboard_ui_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Dedicated /openapi.json block — must come before catch-all /
|
||||
# LiteLLM serves valid openapi: 3.1.0 JSON at this path internally,
|
||||
# but the catch-all location / strips the URI path. This block
|
||||
# preserves the full path so the spec JSON is returned instead of
|
||||
# the Swagger UI SPA HTML.
|
||||
location /openapi.json {
|
||||
proxy_pass $litellm_backend_url/openapi.json;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /metrics/ {
|
||||
proxy_pass $router_api_url/metrics/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass $router_api_url/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /router/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass $router_api_url/health/unified;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass $router_api_url/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :4000 — external LiteLLM entrypoint (cloudflared target)
|
||||
# Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
|
||||
# builds external URLs instead of leaking 192.168.68.116:4000.
|
||||
# LiteLLM itself is now bound to 127.0.0.1 only (see compose).
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 4000;
|
||||
|
||||
# Canonical external identity — overrides whatever Host cloudflared sends
|
||||
proxy_set_header Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
|
||||
# /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
|
||||
# so a relative return would be absolutized to http://192.168.68.116:4000/).
|
||||
location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
|
||||
|
||||
# LiteLLM Admin UI + WebSocket
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect $litellm_backend_url/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
|
||||
# UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass $litellm_backend_url/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# API
|
||||
location /v1/ {
|
||||
proxy_pass $litellm_backend_url/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Key management + admin API
|
||||
location /key/ {
|
||||
proxy_pass $litellm_backend_url/key/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /user/ {
|
||||
proxy_pass $litellm_backend_url/user/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /model/ {
|
||||
proxy_pass $litellm_backend_url/model/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /team/ {
|
||||
proxy_pass $litellm_backend_url/team/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# Health
|
||||
location /health {
|
||||
proxy_pass $litellm_backend_url/health;
|
||||
proxy_http_version 1.1;
|
||||
}
|
||||
|
||||
# Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
|
||||
location / {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,343 +0,0 @@
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
sendfile on;
|
||||
keepalive_timeout 65;
|
||||
|
||||
# Docker DNS resolver — forces request-time resolution for variable-based proxy_pass.
|
||||
# Without this, nginx resolves upstream hostnames at config load time,
|
||||
# which fails when Docker DNS (127.0.0.11) isn't ready yet on container start.
|
||||
resolver 127.0.0.11 valid=30s;
|
||||
|
||||
# Dynamic upstream resolution via nginx variables.
|
||||
# Using $var in proxy_pass forces request-time resolution through the resolver.
|
||||
# Without this, 'host not found in upstream' crashes nginx when Docker DNS is slow.
|
||||
map $host $router_api_url {
|
||||
default http://harness-router:9000;
|
||||
}
|
||||
map $host $dashboard_ui_url {
|
||||
default http://harness-dashboard:3000;
|
||||
}
|
||||
map $host $litellm_backend_url {
|
||||
default http://harness-litellm:4000;
|
||||
}
|
||||
|
||||
# Detect Cloudflare Tunnel requests (cloudflared always sets CF-Connecting-IP).
|
||||
# Direct LAN browser access to :4000 has no such header -> redirect to canonical https,
|
||||
# preventing the cross-origin localStorage footgun that traps the UI at the login page.
|
||||
map $http_cf_connecting_ip $is_cloudflared {
|
||||
default 1; # any non-empty value = request came through Cloudflare
|
||||
"" 0; # empty = direct access
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :80 — existing harness entrypoint
|
||||
# dashboard (/), router API (/v1/, /admin/, /stream, /api/, /metrics),
|
||||
# router fallback, LiteLLM UI via /litellm/ prefix, health
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
# Authentik OIDC subrequest
|
||||
location /authentik/auth {
|
||||
internal;
|
||||
proxy_pass https://auth.sysloggh.net/outpost.goauthentik.io/auth/nginx;
|
||||
proxy_pass_request_body off;
|
||||
proxy_set_header Content-Length "";
|
||||
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
}
|
||||
|
||||
# 2-Layer: /v1/ → router (existing keys work unchanged)
|
||||
location /v1/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
error_page 502 503 = @router_fallback;
|
||||
}
|
||||
|
||||
location @router_fallback {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /admin/ {
|
||||
proxy_pass $router_api_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
location /stream {
|
||||
proxy_pass $router_api_url/stream;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /api/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM gateway access (agents with new virtual keys)
|
||||
location /litellm/v1/ {
|
||||
proxy_pass $litellm_backend_url/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# LiteLLM UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# LiteLLM Admin UI at /ui/ (public access via Traefik → port 80)
|
||||
# LiteLLM Admin UI at /ui/ (must be before catch-all /)
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
error_page 301 302 = @ui_redirect;
|
||||
}
|
||||
|
||||
location @ui_redirect {
|
||||
proxy_pass $litellm_backend_url;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass $litellm_backend_url/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
|
||||
# LiteLLM Admin UI via /litellm/ prefix (LAN/internal access on :80)
|
||||
location /litellm/ {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://$host/ /litellm/;
|
||||
proxy_redirect https://$host/ /litellm/;
|
||||
}
|
||||
|
||||
location /dashboard/ {
|
||||
proxy_pass $dashboard_ui_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
# Dedicated /openapi.json block — must come before catch-all /
|
||||
# LiteLLM serves valid openapi: 3.1.0 JSON at this path internally,
|
||||
# but the catch-all location / strips the URI path. This block
|
||||
# preserves the full path so the spec JSON is returned instead of
|
||||
# the Swagger UI SPA HTML.
|
||||
location /openapi.json {
|
||||
proxy_pass $litellm_backend_url/openapi.json;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_connect_timeout 10s;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
location /metrics/ {
|
||||
proxy_pass $router_api_url/metrics/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /metrics/circuit-breaker {
|
||||
proxy_pass $router_api_url/metrics/circuit-breaker;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /router/ {
|
||||
proxy_pass $router_api_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health/unified {
|
||||
proxy_pass $router_api_url/health/unified;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location /health {
|
||||
proxy_pass $router_api_url/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
}
|
||||
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
# Server :4000 — external LiteLLM entrypoint (cloudflared target)
|
||||
# Fixes the /ui redirect: forces correct Host + scheme so LiteLLM
|
||||
# builds external URLs instead of leaking 192.168.68.116:4000.
|
||||
# LiteLLM itself is now bound to 127.0.0.1 only (see compose).
|
||||
# ════════════════════════════════════════════════════════════════
|
||||
server {
|
||||
listen 4000;
|
||||
|
||||
# Canonical external identity — overrides whatever Host cloudflared sends
|
||||
proxy_set_header Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Host litellm.sysloggh.net;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_set_header X-Forwarded-Port 443;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
|
||||
# /ui → /ui/ — absolute redirect (cloudflared rewrites Host to origin IP,
|
||||
# so a relative return would be absolutized to http://192.168.68.116:4000/).
|
||||
location = /ui { return 301 https://litellm.sysloggh.net/ui/; }
|
||||
|
||||
# LiteLLM Admin UI + WebSocket
|
||||
location /ui/ {
|
||||
proxy_pass $litellm_backend_url/ui/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect $litellm_backend_url/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect http://litellm.sysloggh.net:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
|
||||
# UI static assets
|
||||
location /litellm-asset-prefix/ {
|
||||
proxy_pass $litellm_backend_url/litellm-asset-prefix/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
|
||||
# SSO callback
|
||||
location /sso/ {
|
||||
proxy_pass $litellm_backend_url/sso/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 600s;
|
||||
}
|
||||
|
||||
# API
|
||||
location /v1/ {
|
||||
proxy_pass $litellm_backend_url/v1/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
proxy_read_timeout 600s;
|
||||
proxy_buffering off;
|
||||
}
|
||||
|
||||
# Key management + admin API
|
||||
location /key/ {
|
||||
proxy_pass $litellm_backend_url/key/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /user/ {
|
||||
proxy_pass $litellm_backend_url/user/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /model/ {
|
||||
proxy_pass $litellm_backend_url/model/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
location /team/ {
|
||||
proxy_pass $litellm_backend_url/team/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Authorization $http_authorization;
|
||||
}
|
||||
|
||||
# Health
|
||||
location /health {
|
||||
proxy_pass $litellm_backend_url/health;
|
||||
proxy_http_version 1.1;
|
||||
}
|
||||
|
||||
# Root + everything else → LiteLLM (UI index, /configs, /spend, etc.)
|
||||
location / {
|
||||
proxy_pass $litellm_backend_url/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_buffering off;
|
||||
proxy_redirect http://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
proxy_redirect https://192.168.68.116:4000/ https://litellm.sysloggh.net/;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,831 +0,0 @@
|
||||
import os, json, time, logging, traceback, threading, queue, statistics, math
|
||||
import requests, redis
|
||||
from flask import Flask, request, jsonify, Response, stream_with_context
|
||||
|
||||
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
|
||||
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
|
||||
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
|
||||
GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
|
||||
|
||||
GPU_SIDECARS = {
|
||||
"qwen3.6-35B-A3B": "http://192.168.68.15:8090",
|
||||
"qwen3.6-27B-code": "http://192.168.68.8:8090",
|
||||
"gemma-4-12b": "http://192.168.68.110:8090",
|
||||
}
|
||||
GPU_URLS = {
|
||||
"qwen3.6-35B-A3B": GPU_MOE_URL,
|
||||
"qwen3.6-27B-code": GPU_DENSE_URL,
|
||||
"gemma-4-12b": GPU_LIGHT_URL,
|
||||
}
|
||||
# Max concurrent requests per GPU (based on llama.cpp --parallel)
|
||||
GPU_MAX_CONCURRENT = {
|
||||
"qwen3.6-35B-A3B": 2, # 2 slots (cross-agent spread prevents overheating)
|
||||
"qwen3.6-27B-code": 2, # 2 slots
|
||||
"gemma-4-12b": 2, # 2 slots (12GB VRAM, 4GB headroom)
|
||||
}
|
||||
|
||||
# Context window sizes (tokens) — used for compaction signals
|
||||
GPU_CONTEXT = {
|
||||
"qwen3.6-35B-A3B": 262144,
|
||||
"qwen3.6-27B-code": 262144,
|
||||
"gemma-4-12b": 262144,
|
||||
}
|
||||
|
||||
TIER_MODELS = {
|
||||
"starter": ["gemma-4-12b"],
|
||||
"professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
|
||||
"enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"],
|
||||
}
|
||||
# ── PHASE 0: Dual-Key API Key System ──
|
||||
# API_KEYS env var is REQUIRED (JSON string). No hardcoded fallback.
|
||||
# Format: {"sk-xxx": {"tier": "enterprise", "agent": "Name"}}
|
||||
# Deprecated keys get {"deprecated": true} — still accepted, logged with warning.
|
||||
_raw_keys = os.environ.get("API_KEYS")
|
||||
if not _raw_keys:
|
||||
raise RuntimeError("FATAL: API_KEYS environment variable is required. "
|
||||
"Set it in docker-compose.yml or .env file. "
|
||||
"No hardcoded keys fallback — this is a security feature.")
|
||||
API_KEYS = json.loads(_raw_keys)
|
||||
log.info("Loaded %d API keys from env var (%d deprecated)",
|
||||
len(API_KEYS), sum(1 for v in API_KEYS.values() if v.get("deprecated")))
|
||||
# Rate limits: requests per minute per API key tier
|
||||
RATE_LIMIT_RPM = {
|
||||
"enterprise": 120,
|
||||
"professional": 60,
|
||||
"starter": 20,
|
||||
}
|
||||
|
||||
def check_rate_limit(api_key, tier):
|
||||
"""Token bucket rate limiter using Redis. Returns (allowed, retry_after_or_remaining, reset_seconds)."""
|
||||
if not r:
|
||||
return True, 999, 60
|
||||
limit = RATE_LIMIT_RPM.get(tier, 30)
|
||||
key = f"ratelimit:{api_key}"
|
||||
current = int(r.get(key) or 0)
|
||||
if current >= limit:
|
||||
ttl = r.ttl(key)
|
||||
retry = max(ttl, 1) if ttl and ttl > 0 else 60
|
||||
return False, retry, 0
|
||||
pipe = r.pipeline()
|
||||
pipe.incr(key)
|
||||
pipe.expire(key, 60) # 1-minute sliding window
|
||||
pipe.execute()
|
||||
remaining = limit - (current + 1)
|
||||
reset_seconds = r.ttl(key) or 60
|
||||
return True, remaining, reset_seconds
|
||||
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
|
||||
log = logging.getLogger("router")
|
||||
try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
|
||||
except Exception: r = None
|
||||
|
||||
|
||||
def counter_audit_loop():
|
||||
"""Every 30s, check GPU slots and reset counters if all slots idle."""
|
||||
while True:
|
||||
time.sleep(30)
|
||||
if not r: continue
|
||||
for model, url in GPU_URLS.items():
|
||||
try:
|
||||
resp = requests.get(url.replace("/v1","") + "/slots",
|
||||
headers={"Authorization": "Bearer not-needed"}, timeout=5)
|
||||
if resp.status_code == 200:
|
||||
slots = resp.json()
|
||||
all_idle = all(not s.get("is_processing", False) for s in slots)
|
||||
if all_idle:
|
||||
current = int(r.get("active:" + model) or 0)
|
||||
if current > 0:
|
||||
r.set("active:" + model, 0)
|
||||
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
threading.Thread(target=counter_audit_loop, daemon=True).start()
|
||||
|
||||
app = Flask(__name__)
|
||||
sse_subscribers = []; sse_lock = threading.Lock()
|
||||
|
||||
def gpu_active_count(model):
|
||||
"""Get number of in-flight requests for a GPU."""
|
||||
if r:
|
||||
return int(r.get("active:" + model) or 0)
|
||||
return 0
|
||||
|
||||
def gpu_incr(model):
|
||||
if r: r.incr("active:" + model)
|
||||
|
||||
def gpu_decr(model):
|
||||
if r:
|
||||
v = r.decr("active:" + model)
|
||||
if v and int(v) < 0:
|
||||
r.set("active:" + model, 0) # never go negative
|
||||
|
||||
def check_gpu_health(model, sidecar_timeout=5, gpu_timeout=3):
|
||||
url = GPU_SIDECARS.get(model)
|
||||
if not url: return {"status": "unknown"}
|
||||
try:
|
||||
resp = requests.get(url, timeout=sidecar_timeout)
|
||||
if resp.status_code == 200:
|
||||
d = resp.json()
|
||||
pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
|
||||
status = "healthy" # VRAM usage != saturation; busy slots handled by is_gpu_busy()
|
||||
vram_warning = pct >= 95
|
||||
# Also check if llama.cpp endpoint is actually responding
|
||||
gpu_url = GPU_URLS.get(model, "")
|
||||
try:
|
||||
hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=gpu_timeout)
|
||||
if hr.status_code != 200:
|
||||
status = "down"
|
||||
except Exception:
|
||||
status = "down"
|
||||
return {"status": status, "vram_warning": vram_warning, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
|
||||
except Exception: pass
|
||||
return {"status": "down"}
|
||||
|
||||
def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
|
||||
|
||||
def estimate_tokens(msgs):
|
||||
"""Estimate token count from messages. Uses JSON length / 3.5 (closer to real tokenizer ratios for dense text)."""
|
||||
return len(json.dumps(msgs, default=str)) // 3.5
|
||||
|
||||
def store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, stream):
|
||||
"""Store detailed performance record in Redis for analytics."""
|
||||
if not r: return
|
||||
try:
|
||||
total_ms = queue_ms + inference_ms
|
||||
tps = completion_tokens / (inference_ms / 1000) if inference_ms > 0 and completion_tokens > 0 else 0
|
||||
rec = json.dumps({
|
||||
"ts": time.time(),
|
||||
"model": model, "agent": agent, "tier": tier, "reason": reason,
|
||||
"queue_ms": round(queue_ms, 1),
|
||||
"inference_ms": round(inference_ms, 1),
|
||||
"total_ms": round(total_ms, 1),
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"completion_tokens": completion_tokens,
|
||||
"tokens_per_sec": round(tps, 1),
|
||||
"stream": stream
|
||||
})
|
||||
# Global recent list (last 500)
|
||||
r.lpush("perf:recent", rec)
|
||||
r.ltrim("perf:recent", 0, 499)
|
||||
# Per-model list (last 200)
|
||||
r.lpush("perf:model:" + model, rec)
|
||||
r.ltrim("perf:model:" + model, 0, 199)
|
||||
# Per-reason list (last 200)
|
||||
r.lpush("perf:reason:" + reason, rec)
|
||||
r.ltrim("perf:reason:" + reason, 0, 199)
|
||||
# Per-agent list (last 200)
|
||||
r.lpush("perf:agent:" + agent, rec)
|
||||
r.ltrim("perf:agent:" + agent, 0, 199)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def is_gpu_busy(model):
|
||||
"""Check if GPU is at or near max concurrent capacity."""
|
||||
active = gpu_active_count(model)
|
||||
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||
return active >= max_c
|
||||
|
||||
def select_best_gpu(candidates, reason, agent=""):
|
||||
"""Pick best GPU, spreading agents across GPUs to prevent hotspots."""
|
||||
# Count how many distinct agents are on each GPU
|
||||
gpu_agent_counts = {}
|
||||
if r:
|
||||
for m in GPU_URLS:
|
||||
count = 0
|
||||
for ak in API_KEYS.values():
|
||||
if r.get("agent_gpu:" + ak["agent"] + ":" + m):
|
||||
count += 1
|
||||
gpu_agent_counts[m] = count
|
||||
# First pass: prefer GPUs with 0 other agents (fresh GPU for this agent)
|
||||
for m in candidates:
|
||||
if not is_gpu_busy(m) and gpu_agent_counts.get(m, 0) == 0:
|
||||
return {"model": m, "reason": reason}
|
||||
# Second pass: prefer GPU this agent is NOT already on (skip own GPU)
|
||||
if agent:
|
||||
for m in candidates:
|
||||
if not is_gpu_busy(m) and not r.get("agent_gpu:" + agent + ":" + m):
|
||||
return {"model": m, "reason": reason}
|
||||
# Third pass: any non-busy GPU
|
||||
for m in candidates:
|
||||
if not is_gpu_busy(m):
|
||||
return {"model": m, "reason": reason}
|
||||
# All busy — pick least loaded
|
||||
best = None
|
||||
best_load = 999
|
||||
for m in candidates:
|
||||
load = gpu_active_count(m)
|
||||
if load < best_load:
|
||||
best_load = load
|
||||
best = m
|
||||
if best:
|
||||
return {"model": best, "reason": "load_balanced_" + reason}
|
||||
return None
|
||||
|
||||
def route(rd, tier, agent=""):
|
||||
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
|
||||
sys = any(m.get("role")=="system" for m in msgs)
|
||||
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
|
||||
hints = rd.get("routing_hints",{})
|
||||
allowed = TIER_MODELS.get(tier, ["gemma-4-12b"])
|
||||
avail = [m for m in available_models() if m in allowed]
|
||||
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
|
||||
# Check if all available GPUs are at max capacity
|
||||
if all(is_gpu_busy(m) for m in avail):
|
||||
return {"model": avail[0], "reason": "all_saturated", "saturated": True}
|
||||
|
||||
req = rd.get("model","auto")
|
||||
if req != "auto":
|
||||
# STRICT MODE: no silent fallback — LiteLLM handles failover chains.
|
||||
# This keeps per-model metrics accurate. Returns saturated if busy.
|
||||
target = req if req in avail else avail[0]
|
||||
if req not in avail:
|
||||
return {"model": req, "reason": "explicit_unavailable", "saturated": True}
|
||||
if is_gpu_busy(target):
|
||||
return {"model": target, "reason": "explicit_saturated", "saturated": True}
|
||||
return {"model": target, "reason": "explicit"}
|
||||
|
||||
if hints:
|
||||
if hints.get("priority")=="speed" and "gemma-4-12b" in avail:
|
||||
return select_best_gpu(["gemma-4-12b"], "hint_speed", agent) or {"model":"gemma-4-12b","reason":"hint_speed"}
|
||||
if hints.get("priority")=="quality" and "qwen3.6-35B-A3B" in avail:
|
||||
return select_best_gpu(["qwen3.6-35B-A3B"], "hint_quality", agent) or {"model":"qwen3.6-35B-A3B","reason":"hint_quality"}
|
||||
|
||||
first_msg = msgs[0].get("content","") if msgs else ""
|
||||
words = len(first_msg.split()) if isinstance(first_msg, str) else 99
|
||||
|
||||
# TIER 1: Lightweight — single-turn short queries → VLM (fastest)
|
||||
if not sys and turns <= 1 and t <= 500 and words <= 100 and "gemma-4-12b" in avail:
|
||||
if not is_gpu_busy("gemma-4-12b"):
|
||||
return {"model":"gemma-4-12b","reason":"lightweight"}
|
||||
# VLM busy — Dense is faster for short queries than MoE
|
||||
fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
|
||||
result = select_best_gpu(fallback, "lightweight_fallback", agent)
|
||||
if result: return result
|
||||
|
||||
# TIER 2: Simple conversations — VLM primary (up to 15K tok), fastest for moderate chat
|
||||
if t <= 15000 and turns <= 12 and "gemma-4-12b" in avail:
|
||||
if not is_gpu_busy("gemma-4-12b"):
|
||||
return {"model":"gemma-4-12b","reason":"simple_conv"}
|
||||
# VLM busy — fall back to Dense, then MoE
|
||||
fallback = [m for m in ["qwen3.6-27B-code","qwen3.6-35B-A3B"] if m in avail]
|
||||
result = select_best_gpu(fallback, "simple_conv_fallback", agent)
|
||||
if result: return result
|
||||
|
||||
# TIER 3: Medium complexity — Dense primary, VLM fallback (quality + speed balance)
|
||||
if t <= 25000:
|
||||
candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
|
||||
result = select_best_gpu(candidates, "medium", agent)
|
||||
if result: return result
|
||||
|
||||
# TIER 4: Heavy reasoning — MoE primary (workhorse), Dense fallback
|
||||
if t > 25000:
|
||||
candidates = [m for m in ["qwen3.6-35B-A3B","qwen3.6-27B-code","gemma-4-12b"] if m in avail]
|
||||
result = select_best_gpu(candidates, "heavy_reasoning", agent)
|
||||
if result: return result
|
||||
|
||||
# TIER 5: Default — Dense primary, MoE fallback
|
||||
candidates = [m for m in ["qwen3.6-27B-code","gemma-4-12b","qwen3.6-35B-A3B"] if m in avail]
|
||||
result = select_best_gpu(candidates, "default", agent)
|
||||
if result: return result
|
||||
return {"model":avail[0],"reason":"last_resort"}
|
||||
|
||||
def clean_unicode(text):
|
||||
if not isinstance(text, str): return text
|
||||
text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
|
||||
text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
|
||||
text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
|
||||
text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
|
||||
return text.encode("ascii", "ignore").decode("ascii")
|
||||
|
||||
def clean_response(d):
|
||||
if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
|
||||
if isinstance(d, list): return [clean_response(v) for v in d]
|
||||
if isinstance(d, str): return clean_unicode(d)
|
||||
return d
|
||||
|
||||
def get_metrics():
|
||||
d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
|
||||
for m in GPU_URLS:
|
||||
h = check_gpu_health(m)
|
||||
d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
|
||||
d["active_requests"][m] = gpu_active_count(m)
|
||||
if r:
|
||||
try:
|
||||
for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
|
||||
for k,v in API_KEYS.items():
|
||||
c = int(r.get("routes:agent:"+v["agent"]) or 0)
|
||||
if c>0: d["agent_counts"][v["agent"]] = c
|
||||
for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
|
||||
raw = r.lrange("routes:recent",0,49)
|
||||
d["recent"] = [json.loads(x) for x in raw] if raw else []
|
||||
except Exception: pass
|
||||
return d
|
||||
|
||||
def bcast():
|
||||
data = get_metrics(); payload = json.dumps(data)
|
||||
with sse_lock:
|
||||
dead = []
|
||||
for q in sse_subscribers:
|
||||
try: q.put(payload)
|
||||
except Exception: dead.append(q)
|
||||
for q in dead: sse_subscribers.remove(q)
|
||||
|
||||
QUEUE_TIMEOUT = int(os.environ.get("QUEUE_TIMEOUT", "30")) # max seconds to queue before 503
|
||||
|
||||
@app.route("/v1/chat/completions", methods=["POST"])
|
||||
def chat():
|
||||
try:
|
||||
rd = request.get_json(force=True)
|
||||
ak = request.headers.get("Authorization","").replace("Bearer ","")
|
||||
if not ak or ak not in API_KEYS:
|
||||
log.warning("AUTH_REJECTED: no/invalid API key from %s", request.remote_addr)
|
||||
return jsonify({"error": "Unauthorized — valid API key required"}), 401
|
||||
ki = API_KEYS[ak]
|
||||
tier, agent = ki["tier"], ki["agent"]
|
||||
# Phase 0: dual-key transition — log deprecated key usage
|
||||
if ki.get("deprecated"):
|
||||
new_key = next((k for k, v in API_KEYS.items()
|
||||
if v.get("agent") == agent and not v.get("deprecated")), None)
|
||||
log.warning("DEPRECATED_KEY: agent=%s using old key %s...%s — switch to %s...%s",
|
||||
agent, ak[:12], ak[-8:],
|
||||
new_key[:12] if new_key else "N/A",
|
||||
new_key[-8:] if new_key else "N/A")
|
||||
if r:
|
||||
r.incr("deprecated_usage:" + agent)
|
||||
|
||||
# Rate limit check
|
||||
allowed, rl_val, reset_sec = check_rate_limit(ak, tier)
|
||||
if not allowed:
|
||||
resp = jsonify({"error": "Rate limit exceeded", "retry_after_s": rl_val})
|
||||
resp.headers["Retry-After"] = str(rl_val)
|
||||
resp.headers["X-RateLimit-Limit"] = str(RATE_LIMIT_RPM.get(tier, 30))
|
||||
resp.headers["X-RateLimit-Remaining"] = "0"
|
||||
resp.headers["X-RateLimit-Reset"] = str(int(time.time() + rl_val))
|
||||
log.warning("RATE_LIMIT: %s (%s) exceeded limit", agent, ak[-8:])
|
||||
return resp, 429
|
||||
|
||||
# Allow agent to override queue timeout via header
|
||||
q_timeout = int(request.headers.get("X-Queue-Timeout", str(QUEUE_TIMEOUT)))
|
||||
|
||||
# Cross-turn context tracking: accumulate tokens per session
|
||||
session_id = request.headers.get("X-Session-Id", "")
|
||||
session_tokens = 0
|
||||
if session_id and r:
|
||||
try:
|
||||
prev = int(r.get("session:" + session_id) or 0)
|
||||
current = estimate_tokens(rd.get("messages",[]))
|
||||
session_tokens = max(prev, current) # context only grows
|
||||
r.set("session:" + session_id, session_tokens, ex=86400) # TTL 24h
|
||||
except Exception: pass
|
||||
|
||||
d = route(rd, tier, agent)
|
||||
queue_start = time.time()
|
||||
|
||||
# Queue loop: wait for a GPU slot instead of immediate 503
|
||||
while d.get("saturated"):
|
||||
elapsed = time.time() - queue_start
|
||||
if elapsed > q_timeout:
|
||||
resp = jsonify({"error": "All GPUs saturated", "queued_s": round(elapsed,1), "retry_after_s": 5})
|
||||
resp.headers["Retry-After"] = "5"
|
||||
log.warning("QUEUE_TIMEOUT: %s waited %.1fs, all GPUs saturated", agent, elapsed)
|
||||
return resp, 503
|
||||
time.sleep(0.5) # poll every 500ms
|
||||
d = route(rd, tier, agent)
|
||||
|
||||
queue_ms = (time.time() - queue_start) * 1000
|
||||
if queue_ms > 500:
|
||||
log.info("QUEUED: %s waited %.0fms before slot opened", agent, queue_ms)
|
||||
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
|
||||
|
||||
# Stash rate limit values for response headers
|
||||
_rl_remaining = rl_val
|
||||
_rl_limit = RATE_LIMIT_RPM.get(tier, 30)
|
||||
_rl_reset = reset_sec
|
||||
is_stream = rd.get("stream", False)
|
||||
|
||||
gpu_incr(model)
|
||||
|
||||
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
|
||||
# Track which GPU this agent is using (TTL 120s covers typical request)
|
||||
if r and agent:
|
||||
try: r.setex("agent_gpu:" + agent + ":" + model, 120, "1")
|
||||
except: pass
|
||||
if r:
|
||||
try:
|
||||
r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
|
||||
r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
|
||||
r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent,"queue_ms": round(queue_ms,1)}))
|
||||
r.ltrim("routes:recent",0,999)
|
||||
except Exception: pass
|
||||
start = time.time()
|
||||
resp = requests.post(url+"/chat/completions", json=rd,
|
||||
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
|
||||
lat = int((time.time()-start)*1000)
|
||||
gpu_decr(model)
|
||||
|
||||
if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
|
||||
if is_stream:
|
||||
# Buffer SSE chunks, handle split lines for large responses
|
||||
chunks = []
|
||||
stream_timings = {}
|
||||
buf = "" # accumulate partial lines
|
||||
for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
|
||||
if raw:
|
||||
cleaned = clean_unicode(raw)
|
||||
chunks.append(cleaned)
|
||||
buf += cleaned
|
||||
# Process complete lines from buffer
|
||||
while "\n" in buf:
|
||||
line, buf = buf.split("\n", 1)
|
||||
line = line.strip()
|
||||
if line.startswith("data: ") and not stream_timings:
|
||||
js = line[6:].strip()
|
||||
if js.startswith("{") and "timings" in js and "predicted_n" in js:
|
||||
try:
|
||||
tj = json.loads(js).get("timings", {})
|
||||
if tj:
|
||||
stream_timings = tj
|
||||
except: pass
|
||||
# Store perf record with real token counts from stream
|
||||
if stream_timings:
|
||||
pt = stream_timings.get("prompt_n", 0)
|
||||
ct = stream_timings.get("predicted_n", 0)
|
||||
tps = stream_timings.get("predicted_per_second", 0)
|
||||
gen_ms = stream_timings.get("predicted_ms", lat)
|
||||
store_perf_record(model, agent, tier, reason, queue_ms, gen_ms, pt, ct, True)
|
||||
else:
|
||||
store_perf_record(model, agent, tier, reason, queue_ms, lat, estimate_tokens(rd.get("messages",[])), 0, True)
|
||||
# Yield all chunks to client
|
||||
def gen():
|
||||
for c in chunks: yield c
|
||||
bcast()
|
||||
ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
|
||||
ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
|
||||
ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
|
||||
sse_resp = Response(stream_with_context(gen()), mimetype="text/event-stream")
|
||||
sse_resp.headers["X-RateLimit-Limit"] = str(_rl_limit)
|
||||
sse_resp.headers["X-RateLimit-Remaining"] = str(max(0, _rl_remaining))
|
||||
sse_resp.headers["X-RateLimit-Reset"] = str(int(time.time() + _rl_reset))
|
||||
sse_resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
|
||||
sse_resp.headers["X-Context-Warning"] = ctx_warning
|
||||
sse_resp.headers["X-Context-Model"] = model
|
||||
return sse_resp
|
||||
data = clean_response(resp.json())
|
||||
for c in data.get("choices",[]):
|
||||
msg = c.get("message",{})
|
||||
if not msg.get("content") and msg.get("reasoning_content"):
|
||||
msg["content"] = msg["reasoning_content"]
|
||||
# Extract performance data from llama.cpp response
|
||||
usage = data.get("usage", {})
|
||||
timings = data.get("timings", {})
|
||||
prompt_tokens = usage.get("prompt_tokens", 0)
|
||||
completion_tokens = usage.get("completion_tokens", 0)
|
||||
inference_ms = lat # total GPU round-trip
|
||||
store_perf_record(model, agent, tier, reason, queue_ms, inference_ms, prompt_tokens, completion_tokens, False)
|
||||
ctx_remaining = GPU_CONTEXT.get(model, 65536) - max(session_tokens, estimate_tokens(rd.get("messages",[])))
|
||||
ctx_pct = ctx_remaining / GPU_CONTEXT.get(model, 65536) * 100
|
||||
ctx_warning = "compact_urgent" if ctx_pct < 5 else ("compact_recommended" if ctx_pct < 15 else ("compact_soon" if ctx_pct < 30 else "ok"))
|
||||
data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"queue_ms": round(queue_ms,1),"active_gpu":gpu_active_count(model),"context_remaining": max(0, ctx_remaining),"context_pct": round(ctx_pct,1),"context_warning": ctx_warning}
|
||||
resp = jsonify(data)
|
||||
resp.headers["X-RateLimit-Limit"] = str(_rl_limit)
|
||||
resp.headers["X-RateLimit-Remaining"] = str(max(0, _rl_remaining))
|
||||
resp.headers["X-RateLimit-Reset"] = str(int(time.time() + _rl_reset))
|
||||
resp.headers["X-Context-Remaining"] = str(max(0, ctx_remaining))
|
||||
resp.headers["X-Context-Warning"] = ctx_warning
|
||||
resp.headers["X-Context-Model"] = model
|
||||
bcast()
|
||||
return resp
|
||||
except requests.Timeout:
|
||||
gpu_decr(model)
|
||||
log.error("TIMEOUT: %s -> %s", agent, model)
|
||||
return jsonify({"error":"timeout"}), 504
|
||||
except Exception as e:
|
||||
gpu_decr(model)
|
||||
log.error("Error: %s\n%s", e, traceback.format_exc())
|
||||
return jsonify({"error":str(e)}), 500
|
||||
|
||||
@app.route("/metrics/performance")
|
||||
def performance():
|
||||
"""Per-request performance analytics with percentiles per model/reason/agent."""
|
||||
if not r: return jsonify({"error": "Redis unavailable"}), 503
|
||||
try:
|
||||
window_hours = int(request.args.get("window", "24"))
|
||||
model_filter = request.args.get("model", "all")
|
||||
|
||||
# Load recent records
|
||||
cutoff = time.time() - (window_hours * 3600)
|
||||
raw = r.lrange("perf:recent", 0, -1)
|
||||
records = []
|
||||
for x in raw:
|
||||
try:
|
||||
rec = json.loads(x)
|
||||
if rec["ts"] >= cutoff:
|
||||
records.append(rec)
|
||||
except: pass
|
||||
|
||||
# Filter by model if specified
|
||||
if model_filter != "all":
|
||||
records = [r for r in records if r["model"] == model_filter]
|
||||
|
||||
if not records:
|
||||
return jsonify({"models": [], "reasons": [], "agents": [], "summary": {"total_requests": 0}})
|
||||
|
||||
def pct(values, p):
|
||||
if len(values) < 2: return round(values[0], 1) if values else 0
|
||||
return round(statistics.quantiles(sorted(values), n=100, method='inclusive')[min(p-1, 98)], 1)
|
||||
|
||||
# Per-model stats
|
||||
model_groups = {}
|
||||
for rec in records:
|
||||
m = rec["model"]
|
||||
if m not in model_groups: model_groups[m] = []
|
||||
model_groups[m].append(rec)
|
||||
|
||||
models = []
|
||||
for m, recs in sorted(model_groups.items()):
|
||||
latencies = [r["total_ms"] for r in recs]
|
||||
tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
|
||||
non_stream = [r for r in recs if not r["stream"]]
|
||||
queue_times = [r["queue_ms"] for r in non_stream]
|
||||
models.append({
|
||||
"model": m,
|
||||
"count": len(recs),
|
||||
"stream_pct": round(len([r for r in recs if r["stream"]]) / len(recs) * 100, 1),
|
||||
"latency": {
|
||||
"avg": round(statistics.mean(latencies), 1),
|
||||
"p50": pct(latencies, 50),
|
||||
"p95": pct(latencies, 95),
|
||||
"p99": pct(latencies, 99)
|
||||
},
|
||||
"throughput": {
|
||||
"avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0,
|
||||
"p50": pct(tps_vals, 50) if tps_vals else 0,
|
||||
"p95": pct(tps_vals, 95) if tps_vals else 0,
|
||||
},
|
||||
"queue": {
|
||||
"avg_ms": round(statistics.mean(queue_times), 1) if queue_times else 0,
|
||||
"p95_ms": pct(queue_times, 95) if queue_times else 0,
|
||||
} if queue_times else None
|
||||
})
|
||||
|
||||
# Per-reason stats
|
||||
reason_groups = {}
|
||||
for rec in records:
|
||||
rsn = rec["reason"]
|
||||
if rsn not in reason_groups: reason_groups[rsn] = []
|
||||
reason_groups[rsn].append(rec)
|
||||
|
||||
reasons = []
|
||||
for rsn, recs in sorted(reason_groups.items(), key=lambda x: -len(x[1])):
|
||||
latencies = [r["total_ms"] for r in recs]
|
||||
reasons.append({
|
||||
"reason": rsn,
|
||||
"count": len(recs),
|
||||
"avg_total_ms": round(statistics.mean(latencies), 1),
|
||||
"p95_total_ms": pct(latencies, 95)
|
||||
})
|
||||
|
||||
# Per-agent stats
|
||||
agent_groups = {}
|
||||
for rec in records:
|
||||
ag = rec["agent"]
|
||||
if ag not in agent_groups: agent_groups[ag] = []
|
||||
agent_groups[ag].append(rec)
|
||||
|
||||
agents = []
|
||||
for ag, recs in sorted(agent_groups.items(), key=lambda x: -len(x[1])):
|
||||
latencies = [r["total_ms"] for r in recs]
|
||||
tps_vals = [r["tokens_per_sec"] for r in recs if r["tokens_per_sec"] > 0]
|
||||
agents.append({
|
||||
"agent": ag,
|
||||
"count": len(recs),
|
||||
"avg_total_ms": round(statistics.mean(latencies), 1),
|
||||
"avg_tokens_per_sec": round(statistics.mean(tps_vals), 1) if tps_vals else 0
|
||||
})
|
||||
|
||||
all_lat = [r["total_ms"] for r in records]
|
||||
all_tps = [r["tokens_per_sec"] for r in records if r["tokens_per_sec"] > 0]
|
||||
summary = {
|
||||
"total_requests": len(records),
|
||||
"window_hours": window_hours,
|
||||
"latency": {
|
||||
"avg_ms": round(statistics.mean(all_lat), 1),
|
||||
"p50_ms": pct(all_lat, 50),
|
||||
"p95_ms": pct(all_lat, 95),
|
||||
"p99_ms": pct(all_lat, 99)
|
||||
},
|
||||
"throughput_avg_tps": round(statistics.mean(all_tps), 1) if all_tps else 0
|
||||
}
|
||||
|
||||
return jsonify({"models": models, "reasons": reasons, "agents": agents, "summary": summary})
|
||||
except Exception as e:
|
||||
return jsonify({"error": str(e)}), 500
|
||||
|
||||
@app.route("/metrics/scatter")
|
||||
def scatter():
|
||||
"""Return individual data points for scatter plots (prompt_tokens vs latency)."""
|
||||
if not r: return jsonify({"error": "Redis unavailable"}), 503
|
||||
try:
|
||||
window_hours = int(request.args.get("window", "24"))
|
||||
model_filter = request.args.get("model", "all")
|
||||
cutoff = time.time() - (window_hours * 3600)
|
||||
raw = r.lrange("perf:recent", 0, -1)
|
||||
points = []
|
||||
for x in raw:
|
||||
try:
|
||||
rec = json.loads(x)
|
||||
if rec["ts"] >= cutoff:
|
||||
if model_filter == "all" or rec["model"] == model_filter:
|
||||
points.append({
|
||||
"model": rec["model"],
|
||||
"agent": rec["agent"],
|
||||
"reason": rec["reason"],
|
||||
"prompt_tokens": int(rec.get("prompt_tokens", 0)),
|
||||
"completion_tokens": rec.get("completion_tokens", 0),
|
||||
"inference_ms": round(rec["inference_ms"], 1),
|
||||
"tokens_per_sec": rec.get("tokens_per_sec", 0),
|
||||
"stream": rec.get("stream", False)
|
||||
})
|
||||
except: pass
|
||||
return jsonify({"points": points, "count": len(points)})
|
||||
except Exception as e:
|
||||
return jsonify({"error": str(e)}), 500
|
||||
|
||||
@app.route("/v1/models")
|
||||
def models():
|
||||
def _h(m): return check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
|
||||
return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":_h(m).get("status"),"gpu":_h(m).get("gpu_name")} for m in GPU_URLS]})
|
||||
|
||||
@app.route("/health")
|
||||
def health():
|
||||
gpus = {}
|
||||
for m in GPU_URLS:
|
||||
h = check_gpu_health(m, sidecar_timeout=1.5, gpu_timeout=1)
|
||||
h["active_requests"] = gpu_active_count(m)
|
||||
h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
|
||||
gpus[m] = h
|
||||
return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
|
||||
|
||||
@app.route("/metrics")
|
||||
def metrics(): return jsonify(get_metrics())
|
||||
|
||||
@app.route("/metrics/timeseries")
|
||||
def metrics_timeseries():
|
||||
period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
|
||||
data = {"models": {}, "labels": []}
|
||||
if period == "day":
|
||||
buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
|
||||
data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
|
||||
elif period == "week":
|
||||
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
|
||||
data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
|
||||
else:
|
||||
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
|
||||
data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
|
||||
if r:
|
||||
for model in models_list:
|
||||
counts = []
|
||||
for bucket in buckets:
|
||||
total = 0
|
||||
if period in ("week","month"):
|
||||
for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
|
||||
else: total = int(r.get("ts:"+model+":"+bucket) or 0)
|
||||
counts.append(total)
|
||||
data["models"][model] = counts
|
||||
return jsonify(data)
|
||||
|
||||
@app.route("/stream")
|
||||
def stream():
|
||||
def ev():
|
||||
q = queue.Queue()
|
||||
with sse_lock: sse_subscribers.append(q)
|
||||
try:
|
||||
yield "data: "+json.dumps(get_metrics())+"\n\n"
|
||||
while True:
|
||||
try: yield "data: "+q.get(timeout=3)+"\n\n"
|
||||
except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
|
||||
except GeneratorExit: pass
|
||||
finally:
|
||||
with sse_lock:
|
||||
if q in sse_subscribers: sse_subscribers.remove(q)
|
||||
return Response(stream_with_context(ev()), mimetype="text/event-stream",
|
||||
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
|
||||
|
||||
# ── Phase 0: Admin Key Management ──
|
||||
ADMIN_KEY = os.environ.get("ADMIN_KEY", "")
|
||||
|
||||
def _admin_auth():
|
||||
"""Require admin key for management endpoints."""
|
||||
if not ADMIN_KEY:
|
||||
return False, "ADMIN_KEY not configured on server"
|
||||
ak = request.headers.get("Authorization","").replace("Bearer ","")
|
||||
if ak != ADMIN_KEY:
|
||||
return False, "Admin key required"
|
||||
return True, None
|
||||
|
||||
@app.route("/admin/keys")
|
||||
def admin_keys():
|
||||
"""List all API keys (masked) with agent, tier, and deprecation status."""
|
||||
ok, err = _admin_auth()
|
||||
if not ok: return jsonify({"error": err}), 401
|
||||
keys = []
|
||||
for key, info in API_KEYS.items():
|
||||
masked = key[:8] + "..." + key[-8:]
|
||||
keys.append({
|
||||
"masked": masked,
|
||||
"prefix": key[:8],
|
||||
"agent": info["agent"],
|
||||
"tier": info["tier"],
|
||||
"deprecated": info.get("deprecated", False),
|
||||
"length": len(key)
|
||||
})
|
||||
return jsonify({
|
||||
"total": len(keys),
|
||||
"active": sum(1 for k in keys if not k["deprecated"]),
|
||||
"deprecated": sum(1 for k in keys if k["deprecated"]),
|
||||
"keys": sorted(keys, key=lambda k: (k["deprecated"], k["agent"]))
|
||||
})
|
||||
|
||||
@app.route("/admin/keys/deprecation-summary")
|
||||
def admin_deprecation_summary():
|
||||
"""Summary of deprecated key usage (from Redis logs, if available)."""
|
||||
ok, err = _admin_auth()
|
||||
if not ok: return jsonify({"error": err}), 401
|
||||
deprecated_agents = []
|
||||
for key, info in API_KEYS.items():
|
||||
if info.get("deprecated"):
|
||||
# Check Redis for usage count
|
||||
count = 0
|
||||
if r:
|
||||
count = int(r.get("deprecated_usage:" + info["agent"]) or 0)
|
||||
deprecated_agents.append({
|
||||
"agent": info["agent"],
|
||||
"deprecated_uses": count,
|
||||
"needs_migration": count > 0
|
||||
})
|
||||
return jsonify({
|
||||
"deprecated_agents": sorted(deprecated_agents, key=lambda d: -d["deprecated_uses"]),
|
||||
"recommendation": "Run POST /admin/keys/revoke to remove keys with 0 usage"
|
||||
})
|
||||
|
||||
@app.route("/admin/keys/generate", methods=["POST"])
|
||||
def admin_generate_key():
|
||||
"""Generate a new API key for an agent. Body: {"agent": "Name", "tier": "enterprise"}"""
|
||||
ok, err = _admin_auth()
|
||||
if not ok: return jsonify({"error": err}), 401
|
||||
body = request.get_json(force=True)
|
||||
agent = body.get("agent", "").strip()
|
||||
tier = body.get("tier", "enterprise")
|
||||
if not agent:
|
||||
return jsonify({"error": "agent field required"}), 400
|
||||
if tier not in ("starter", "professional", "enterprise"):
|
||||
return jsonify({"error": "tier must be starter/professional/enterprise"}), 400
|
||||
# Generate secure key
|
||||
import secrets, hashlib
|
||||
prefix = hashlib.sha256(secrets.token_bytes(12)).hexdigest()[:8]
|
||||
suffix = secrets.token_hex(20)
|
||||
new_key = f"sk-{prefix}-{suffix}"
|
||||
# Update in-memory dict (note: not persisted across restarts without env var update)
|
||||
API_KEYS[new_key] = {"tier": tier, "agent": agent}
|
||||
log.info("KEY_GENERATED: agent=%s tier=%s key=%s...%s", agent, tier, new_key[:8], new_key[-8:])
|
||||
return jsonify({
|
||||
"agent": agent,
|
||||
"tier": tier,
|
||||
"key": new_key,
|
||||
"masked": new_key[:8] + "..." + new_key[-8:],
|
||||
"warning": "This key exists in memory only. Update API_KEYS env var and redeploy to persist."
|
||||
}), 201
|
||||
|
||||
@app.route("/admin/keys/revoke", methods=["POST"])
|
||||
def admin_revoke_key():
|
||||
"""Revoke a deprecated key. Body: {"agent": "Name"} or {"key_prefix": "sk-xxxx"}"""
|
||||
ok, err = _admin_auth()
|
||||
if not ok: return jsonify({"error": err}), 401
|
||||
body = request.get_json(force=True)
|
||||
agent = body.get("agent", "")
|
||||
key_prefix = body.get("key_prefix", "")
|
||||
revoked = []
|
||||
keys_to_remove = []
|
||||
for key, info in API_KEYS.items():
|
||||
if not info.get("deprecated"):
|
||||
continue
|
||||
if agent and info["agent"] == agent:
|
||||
keys_to_remove.append(key)
|
||||
elif key_prefix and key.startswith(key_prefix):
|
||||
keys_to_remove.append(key)
|
||||
for key in keys_to_remove:
|
||||
info = API_KEYS.pop(key)
|
||||
revoked.append({"agent": info["agent"], "masked": key[:8] + "..." + key[-8:]})
|
||||
log.warning("KEY_REVOKED: agent=%s key=%s...%s", info["agent"], key[:8], key[-8:])
|
||||
return jsonify({
|
||||
"revoked": len(revoked),
|
||||
"keys": revoked,
|
||||
"remaining_total": len(API_KEYS),
|
||||
"warning": "Memory-only revoke. Update API_KEYS env var and redeploy to persist."
|
||||
})
|
||||
|
||||
if __name__ == "__main__":
|
||||
log.info("Router on :9000 (load-aware)")
|
||||
app.run(host="0.0.0.0", port=9000, debug=False)
|
||||
@@ -1,342 +0,0 @@
|
||||
import os, json, time, logging, traceback, threading, queue
|
||||
import requests, redis
|
||||
from flask import Flask, request, jsonify, Response, stream_with_context
|
||||
|
||||
REDIS_URL = os.environ.get("REDIS_URL", "redis://redis:6379")
|
||||
GPU_MOE_URL = os.environ.get("GPU_MOE_URL", "http://192.168.68.15:8080/v1")
|
||||
GPU_DENSE_URL = os.environ.get("GPU_DENSE_URL", "http://192.168.68.8:8080/v1")
|
||||
GPU_LIGHT_URL = os.environ.get("GPU_LIGHT_URL", "http://192.168.68.110:8080/v1")
|
||||
|
||||
GPU_SIDECARS = {
|
||||
"qwen3.6-35B-A3B": "http://192.168.68.15:8090",
|
||||
"qwen3.6-27B-code": "http://192.168.68.8:8090",
|
||||
"qwen3.5-9b-vlm": "http://192.168.68.110:8090",
|
||||
}
|
||||
GPU_URLS = {
|
||||
"qwen3.6-35B-A3B": GPU_MOE_URL,
|
||||
"qwen3.6-27B-code": GPU_DENSE_URL,
|
||||
"qwen3.5-9b-vlm": GPU_LIGHT_URL,
|
||||
}
|
||||
# Max concurrent requests per GPU (based on llama.cpp --parallel)
|
||||
GPU_MAX_CONCURRENT = {
|
||||
"qwen3.6-35B-A3B": 2, # 2 slots
|
||||
"qwen3.6-27B-code": 2, # 2 slots
|
||||
"qwen3.5-9b-vlm": 1, # 1 slot
|
||||
}
|
||||
|
||||
TIER_MODELS = {
|
||||
"starter": ["qwen3.5-9b-vlm"],
|
||||
"professional": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
|
||||
"enterprise": ["qwen3.6-35B-A3B", "qwen3.6-27B-code", "qwen3.5-9b-vlm"],
|
||||
}
|
||||
API_KEYS = {
|
||||
"sk-syslog-local-master-key": {"tier": "enterprise", "agent": "admin"},
|
||||
"sk-syslog-abiba": {"tier": "enterprise", "agent": "Abiba"},
|
||||
"sk-syslog-mumuni": {"tier": "enterprise", "agent": "Mumuni"},
|
||||
"sk-syslog-tanko": {"tier": "enterprise", "agent": "Tanko"},
|
||||
"sk-syslog-koby": {"tier": "enterprise", "agent": "Koby"},
|
||||
"sk-syslog-kagenz0": {"tier": "enterprise", "agent": "Kagenz0"},
|
||||
"sk-syslog-koonimo": {"tier": "enterprise", "agent": "Koonimo"},
|
||||
"sk-starter-abc123": {"tier": "starter", "agent": "test-starter"},
|
||||
"sk-professional-xyz789": {"tier": "professional", "agent": "test-pro"},
|
||||
}
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s [ROUTER] %(levelname)s %(message)s")
|
||||
log = logging.getLogger("router")
|
||||
try: r = redis.from_url(REDIS_URL, decode_responses=True); r.ping()
|
||||
except Exception: r = None
|
||||
|
||||
|
||||
def counter_audit_loop():
|
||||
"""Every 30s, check GPU slots and reset counters if all slots idle."""
|
||||
while True:
|
||||
time.sleep(30)
|
||||
if not r: continue
|
||||
for model, url in GPU_URLS.items():
|
||||
try:
|
||||
resp = requests.get(url.replace("/v1","") + "/slots",
|
||||
headers={"Authorization": "Bearer not-needed"}, timeout=5)
|
||||
if resp.status_code == 200:
|
||||
slots = resp.json()
|
||||
all_idle = all(not s.get("is_processing", False) for s in slots)
|
||||
if all_idle:
|
||||
current = int(r.get("active:" + model) or 0)
|
||||
if current > 0:
|
||||
r.set("active:" + model, 0)
|
||||
log.info("AUDIT: Reset stuck counter for %s (was %d)", model, current)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
threading.Thread(target=counter_audit_loop, daemon=True).start()
|
||||
|
||||
app = Flask(__name__)
|
||||
sse_subscribers = []; sse_lock = threading.Lock()
|
||||
|
||||
def gpu_active_count(model):
|
||||
"""Get number of in-flight requests for a GPU."""
|
||||
if r:
|
||||
return int(r.get("active:" + model) or 0)
|
||||
return 0
|
||||
|
||||
def gpu_incr(model):
|
||||
if r: r.incr("active:" + model)
|
||||
|
||||
def gpu_decr(model):
|
||||
if r:
|
||||
v = r.decr("active:" + model)
|
||||
if v and int(v) < 0:
|
||||
r.set("active:" + model, 0) # never go negative
|
||||
|
||||
def check_gpu_health(model):
|
||||
url = GPU_SIDECARS.get(model)
|
||||
if not url: return {"status": "unknown"}
|
||||
try:
|
||||
resp = requests.get(url, timeout=5)
|
||||
if resp.status_code == 200:
|
||||
d = resp.json()
|
||||
pct = (d.get("vram_used_mb",0) / max(d.get("vram_total_mb",1), 1)) * 100
|
||||
status = "healthy" if pct < 90 else "saturated"
|
||||
# Also check if llama.cpp endpoint is actually responding
|
||||
gpu_url = GPU_URLS.get(model, "")
|
||||
try:
|
||||
hr = requests.get(gpu_url.replace("/v1","") + "/health", headers={"Authorization": "Bearer not-needed"}, timeout=3)
|
||||
if hr.status_code != 200:
|
||||
status = "down"
|
||||
except Exception:
|
||||
status = "down"
|
||||
return {"status": status, "vram_used_mb": d.get("vram_used_mb"), "vram_total_mb": d.get("vram_total_mb"), "vram_pct": round(pct,1), "temp_c": d.get("temp_c"), "gpu_util_pct": d.get("gpu_util_pct"), "gpu_name": d.get("gpu_name"), "power_w": d.get("power_w"), "power_limit_w": d.get("power_limit_w")}
|
||||
except Exception: pass
|
||||
return {"status": "down"}
|
||||
|
||||
def available_models(): return [m for m in GPU_URLS if check_gpu_health(m)["status"] in ("healthy","saturated")]
|
||||
|
||||
def estimate_tokens(msgs): return sum(len(str(m.get("content",""))) for m in msgs) // 4
|
||||
|
||||
def is_gpu_busy(model):
|
||||
"""Check if GPU is at or near max concurrent capacity."""
|
||||
active = gpu_active_count(model)
|
||||
max_c = GPU_MAX_CONCURRENT.get(model, 1)
|
||||
return active >= max_c
|
||||
|
||||
def select_best_gpu(candidates, reason):
|
||||
"""Pick the best GPU from candidates, preferring least-loaded."""
|
||||
best = None
|
||||
best_load = 999
|
||||
for m in candidates:
|
||||
load = gpu_active_count(m)
|
||||
if load < best_load:
|
||||
best_load = load
|
||||
best = m
|
||||
if best:
|
||||
actual_reason = reason
|
||||
if is_gpu_busy(best):
|
||||
actual_reason = "load_balanced_" + reason
|
||||
return {"model": best, "reason": actual_reason}
|
||||
return None
|
||||
|
||||
def route(rd, tier):
|
||||
msgs = rd.get("messages",[]); t = estimate_tokens(msgs)
|
||||
sys = any(m.get("role")=="system" for m in msgs)
|
||||
turns = len([m for m in msgs if m.get("role") in ("user","assistant")])
|
||||
hints = rd.get("routing_hints",{})
|
||||
allowed = TIER_MODELS.get(tier, ["qwen3.5-9b-vlm"])
|
||||
avail = [m for m in available_models() if m in allowed]
|
||||
if not avail: return {"model": allowed[0], "reason": "all_saturated", "saturated": True}
|
||||
|
||||
req = rd.get("model","auto")
|
||||
if req != "auto":
|
||||
target = req if req in avail else avail[0]
|
||||
# If explicit model is busy, check if another can take it
|
||||
if is_gpu_busy(target) and req in allowed:
|
||||
alts = [m for m in avail if m != target and m in allowed]
|
||||
if alts:
|
||||
alt = select_best_gpu(alts, "explicit")
|
||||
if alt: return alt
|
||||
return {"model": target, "reason": "explicit"}
|
||||
|
||||
if hints:
|
||||
if hints.get("priority")=="speed" and "qwen3.5-9b-vlm" in avail:
|
||||
return select_best_gpu(["qwen3.5-9b-vlm"], "hint_speed") or {"model":"qwen3.5-9b-vlm","reason":"hint_speed"}
|
||||
if hints.get("priority")=="quality" and "qwen3.6-27B-code" in avail:
|
||||
return select_best_gpu(["qwen3.6-27B-code"], "hint_quality") or {"model":"qwen3.6-27B-code","reason":"hint_quality"}
|
||||
|
||||
# Heavy -> dense (but fall back to MoE if dense is busy)
|
||||
if t > 4000 or sys or turns > 6:
|
||||
candidates = ["qwen3.6-27B-code","qwen3.6-35B-A3B","qwen3.5-9b-vlm"]
|
||||
candidates = [m for m in candidates if m in avail]
|
||||
result = select_best_gpu(candidates, "heavy_reasoning")
|
||||
if result: return result
|
||||
|
||||
# Ultra-light -> VLM
|
||||
first_msg = msgs[0].get("content","") if msgs else ""
|
||||
words = len(first_msg.split()) if isinstance(first_msg, str) else 99
|
||||
if words <= 3 and turns <= 1 and not sys and "qwen3.5-9b-vlm" in avail:
|
||||
if not is_gpu_busy("qwen3.5-9b-vlm"):
|
||||
return {"model":"qwen3.5-9b-vlm","reason":"ultra_light"}
|
||||
|
||||
# Default: MoE, fall back to dense if MoE is busy
|
||||
if "qwen3.6-35B-A3B" in avail:
|
||||
if is_gpu_busy("qwen3.6-35B-A3B") and "qwen3.6-27B-code" in avail:
|
||||
return {"model": "qwen3.6-27B-code", "reason": "load_balanced_default"}
|
||||
return {"model":"qwen3.6-35B-A3B","reason":"default_moe"}
|
||||
|
||||
return {"model":avail[0],"reason":"fallback"}
|
||||
|
||||
def clean_unicode(text):
|
||||
if not isinstance(text, str): return text
|
||||
text = text.replace(chr(0x2014), "-"); text = text.replace(chr(0x2013), "-")
|
||||
text = text.replace(chr(0x2018), "'"); text = text.replace(chr(0x2019), "'")
|
||||
text = text.replace(chr(0x201C), '"'); text = text.replace(chr(0x201D), '"')
|
||||
text = text.replace(chr(0x2026), "..."); text = text.replace(chr(0x00A0), " ")
|
||||
return text.encode("ascii", "ignore").decode("ascii")
|
||||
|
||||
def clean_response(d):
|
||||
if isinstance(d, dict): return {k: clean_response(v) for k,v in d.items()}
|
||||
if isinstance(d, list): return [clean_response(v) for v in d]
|
||||
if isinstance(d, str): return clean_unicode(d)
|
||||
return d
|
||||
|
||||
def get_metrics():
|
||||
d = {"gpus":[],"route_counts":{},"agent_counts":{},"tier_counts":{},"recent":[],"timestamp":time.time(),"active_requests":{}}
|
||||
for m in GPU_URLS:
|
||||
h = check_gpu_health(m)
|
||||
d["gpus"].append({"id":m,"gpu_name":h.get("gpu_name",m),"status":h.get("status"),"vram_used_mb":h.get("vram_used_mb"),"vram_total_mb":h.get("vram_total_mb"),"vram_pct":h.get("vram_pct"),"temp_c":h.get("temp_c"),"gpu_util_pct":h.get("gpu_util_pct"),"power_w":h.get("power_w"),"power_limit_w":h.get("power_limit_w"),"active_requests":gpu_active_count(m), "max_concurrent": GPU_MAX_CONCURRENT.get(m, 1)})
|
||||
d["active_requests"][m] = gpu_active_count(m)
|
||||
if r:
|
||||
try:
|
||||
for m in GPU_URLS: d["route_counts"][m] = int(r.get("routes:"+m) or 0)
|
||||
for k,v in API_KEYS.items():
|
||||
c = int(r.get("routes:agent:"+v["agent"]) or 0)
|
||||
if c>0: d["agent_counts"][v["agent"]] = c
|
||||
for t in TIER_MODELS: d["tier_counts"][t] = int(r.get("routes:tier:"+t) or 0)
|
||||
raw = r.lrange("routes:recent",0,49)
|
||||
d["recent"] = [json.loads(x) for x in raw] if raw else []
|
||||
except Exception: pass
|
||||
return d
|
||||
|
||||
def bcast():
|
||||
data = get_metrics(); payload = json.dumps(data)
|
||||
with sse_lock:
|
||||
dead = []
|
||||
for q in sse_subscribers:
|
||||
try: q.put(payload)
|
||||
except Exception: dead.append(q)
|
||||
for q in dead: sse_subscribers.remove(q)
|
||||
|
||||
@app.route("/v1/chat/completions", methods=["POST"])
|
||||
def chat():
|
||||
try:
|
||||
rd = request.get_json(force=True)
|
||||
ak = request.headers.get("Authorization","").replace("Bearer ","")
|
||||
ki = API_KEYS.get(ak, {"tier":"starter","agent":"unknown"})
|
||||
tier, agent = ki["tier"], ki["agent"]
|
||||
d = route(rd, tier)
|
||||
if d.get("saturated"):
|
||||
resp = jsonify({"error": "All GPUs saturated", "retry_after_s": 5})
|
||||
resp.headers["Retry-After"] = "5"
|
||||
return resp, 503
|
||||
model, reason, url = d["model"], d["reason"], GPU_URLS[d["model"]]
|
||||
is_stream = rd.get("stream", False)
|
||||
|
||||
gpu_incr(model)
|
||||
decremented = False
|
||||
|
||||
log.info("ROUTE: %s -> %s (%s) stream=%s active=%d/%d", agent, model, reason, is_stream, gpu_active_count(model), GPU_MAX_CONCURRENT.get(model,1))
|
||||
if r:
|
||||
try:
|
||||
r.incr("routes:"+model); r.incr("routes:tier:"+tier); r.incr("routes:agent:"+agent)
|
||||
r.incr("ts:"+model+":"+time.strftime("%Y%m%d%H"))
|
||||
r.lpush("routes:recent", json.dumps({"ts":time.time(),"model":model,"reason":reason,"tier":tier,"agent":agent}))
|
||||
r.ltrim("routes:recent",0,999)
|
||||
except Exception: pass
|
||||
start = time.time()
|
||||
resp = requests.post(url+"/chat/completions", json=rd,
|
||||
headers={"Content-Type":"application/json","Authorization":"Bearer not-needed"}, timeout=300, stream=is_stream)
|
||||
lat = int((time.time()-start)*1000)
|
||||
gpu_decr(model)
|
||||
decremented = True # Release slot
|
||||
|
||||
if resp.status_code != 200: return jsonify({"error":"GPU error "+str(resp.status_code)}), 502
|
||||
if is_stream:
|
||||
def gen():
|
||||
for raw in resp.iter_content(chunk_size=None, decode_unicode=True):
|
||||
if raw: yield clean_unicode(raw)
|
||||
bcast()
|
||||
return Response(stream_with_context(gen()), mimetype="text/event-stream")
|
||||
data = clean_response(resp.json())
|
||||
for c in data.get("choices",[]):
|
||||
msg = c.get("message",{})
|
||||
if not msg.get("content") and msg.get("reasoning_content"):
|
||||
msg["content"] = msg["reasoning_content"]
|
||||
data["routing"] = {"model":model,"reason":reason,"gpu":url,"tier":tier,"agent":agent,"latency_ms":lat,"active_gpu":gpu_active_count(model)}
|
||||
bcast()
|
||||
return jsonify(data)
|
||||
if not decremented:
|
||||
try: gpu_decr(model)
|
||||
except: pass
|
||||
except requests.Timeout:
|
||||
return jsonify({"error":"timeout"}), 504
|
||||
log.error("Error: %s\n%s", e, traceback.format_exc())
|
||||
return jsonify({"error":str(e)}), 500
|
||||
|
||||
@app.route("/v1/models")
|
||||
def models(): return jsonify({"object":"list","data":[{"id":m,"object":"model","owned_by":"syslog","status":check_gpu_health(m).get("status"),"gpu":check_gpu_health(m).get("gpu_name")} for m in GPU_URLS]})
|
||||
|
||||
@app.route("/health")
|
||||
def health():
|
||||
gpus = {}
|
||||
for m in GPU_URLS:
|
||||
h = check_gpu_health(m)
|
||||
h["active_requests"] = gpu_active_count(m)
|
||||
h["max_concurrent"] = GPU_MAX_CONCURRENT.get(m, 1)
|
||||
gpus[m] = h
|
||||
return jsonify({"status":"healthy","redis":"connected" if r else "down","gpus":gpus,"available_models":available_models()})
|
||||
|
||||
@app.route("/metrics")
|
||||
def metrics(): return jsonify(get_metrics())
|
||||
|
||||
@app.route("/metrics/timeseries")
|
||||
def metrics_timeseries():
|
||||
period = request.args.get("period", "day"); models_list = list(GPU_URLS.keys())
|
||||
data = {"models": {}, "labels": []}
|
||||
if period == "day":
|
||||
buckets = [time.strftime("%Y%m%d%H", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
|
||||
data["labels"] = [time.strftime("%H:00", time.gmtime(time.time()-h*3600)) for h in range(23,-1,-1)]
|
||||
elif period == "week":
|
||||
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
|
||||
data["labels"] = [time.strftime("%a", time.gmtime(time.time()-d*86400)) for d in range(6,-1,-1)]
|
||||
else:
|
||||
buckets = [time.strftime("%Y%m%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
|
||||
data["labels"] = [time.strftime("%m/%d", time.gmtime(time.time()-d*86400)) for d in range(29,-1,-1)]
|
||||
if r:
|
||||
for model in models_list:
|
||||
counts = []
|
||||
for bucket in buckets:
|
||||
total = 0
|
||||
if period in ("week","month"):
|
||||
for hh in range(24): total += int(r.get("ts:"+model+":"+bucket+"{:02d}".format(hh)) or 0)
|
||||
else: total = int(r.get("ts:"+model+":"+bucket) or 0)
|
||||
counts.append(total)
|
||||
data["models"][model] = counts
|
||||
return jsonify(data)
|
||||
|
||||
@app.route("/stream")
|
||||
def stream():
|
||||
def ev():
|
||||
q = queue.Queue()
|
||||
with sse_lock: sse_subscribers.append(q)
|
||||
try:
|
||||
yield "data: "+json.dumps(get_metrics())+"\n\n"
|
||||
while True:
|
||||
try: yield "data: "+q.get(timeout=3)+"\n\n"
|
||||
except queue.Empty: yield "data: "+json.dumps(get_metrics())+"\n\n"
|
||||
except GeneratorExit: pass
|
||||
finally:
|
||||
with sse_lock:
|
||||
if q in sse_subscribers: sse_subscribers.remove(q)
|
||||
return Response(stream_with_context(ev()), mimetype="text/event-stream",
|
||||
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no","Access-Control-Allow-Origin":"*"})
|
||||
|
||||
if __name__ == "__main__":
|
||||
log.info("Router on :9000 (load-aware)")
|
||||
app.run(host="0.0.0.0", port=9000, debug=False)
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user