root
49e8d968f5
- ornith now on Vulkan build 4fc4ec5, 256K ctx, ~70 tok/s - Fixed GPU_MOE_URL bug in docker-compose (.110→.15) - Added router sidecar fallback for health-checks - Updated roster context 4096→262144 + correct model_path - Documented port 8080 firewall (restricted to .116)
192 lines
12 KiB
Markdown
192 lines
12 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: gpu-fleet
|
|
description: >
|
|
Manages the GPU inference fleet across all hosts. Handles model deployment,
|
|
registration, health checks, LiteLLM sync, agent key management, GPU
|
|
saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
|
|
Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
|
|
router runs behind LiteLLM on :9000 (internal-only).
|
|
agent: abiba
|
|
triggers:
|
|
- on model add/remove
|
|
- on GPU health degradation
|
|
- on agent key rotation
|
|
- on router restart (roster must be loaded)
|
|
---
|
|
|
|
## Maintains
|
|
|
|
- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
|
|
- router: { status: "healthy", roster_loaded: bool, models: array }
|
|
- litellm: { status: "healthy", keys: array, models: array }
|
|
- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
|
|
- health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
|
|
- monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
|
|
- watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
|
|
- benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
|
|
- grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
|
|
- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
|
|
|
|
## Fleet Topology (Current — June 2026)
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────────────┐
|
|
│ CT 116 (192.168.68.116) — Inference Harness Host │
|
|
│ │
|
|
│ nginx:80 (entrypoint) │
|
|
│ ├─ /v1/* → harness-litellm:4000 (API requests) │
|
|
│ ├─ /admin/* → harness-litellm:4000 (admin endpoints) │
|
|
│ ├─ /dashboard/ → harness-dashboard:3000 (harness UI) │
|
|
│ ├─ /litellm/* → harness-litellm:4000 (LiteLLM UI + API) │
|
|
│ ├─ /grafana/* → harness-grafana:3000 (dashboards) │
|
|
│ ├─ /health/* → harness-litellm:4000 (health probes) │
|
|
│ └─ /gpu/* → 192.168.68.24:9100 (fleet monitor) │
|
|
│ │
|
|
│ Containers: │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │ LiteLLM │ │ Router │ │Dashboard │ │ Grafana │ │
|
|
│ │ :4000 │─▶│ :9000 │ │ :3000 │ │ :3000 │ │
|
|
│ │ keys+sync│ │internal │ │ harness │ │ Prometheus│ │
|
|
│ │ fallback │ │only! │ │ UI │ │ data src │ │
|
|
│ └──────────┘ └───┬──────┘ └──────────┘ └──────────┘ │
|
|
│ │ │
|
|
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │PostgreSQL│ │ Redis │ │Prometheus│ │
|
|
│ │ :5432 │ │ :6379 │ │ :9090 │ │
|
|
│ └──────────┘ └──────────┘ └──────────┘ │
|
|
└────────────────────┼─────────────────────────────────────────────┘
|
|
│
|
|
┌───────────────┼───────────────┬──────────────────┐
|
|
│ │ │ │
|
|
┌────▼─────┐ ┌──────▼──────┐ ┌────▼──────┐ ┌───────▼──────┐
|
|
│ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │
|
|
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │
|
|
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
|
|
│ │ │ │ │ llama-srv │ │ Watchdog │
|
|
│ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │
|
|
│ 27B-code │ │ :8090 │ │ :8080 │ │ exporter │
|
|
│ :8090 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │
|
|
│ :9400 │ └─────────────┘ └───────────┘ └──────────────┘
|
|
└──────────┘
|
|
```
|
|
|
|
## Current Model Assignments
|
|
|
|
| Model | GPU | Host | VRAM | Status |
|
|
|-------|-----|------|------|--------|
|
|
| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
|
|
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
|
|
| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |
|
|
|
|
## Operations
|
|
|
|
### add-model
|
|
1. Download model file from Hugging Face or source
|
|
2. Check disk space + GPU VRAM compatibility
|
|
3. Start llama-server via systemd service on GPU host
|
|
4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
|
|
5. Hot-reload or restart router
|
|
6. Add model to LiteLLM config (`model_list` + `fallbacks`)
|
|
7. Generate agent keys for new model access via `/key/generate`
|
|
8. Add Prometheus scrape target for the new GPU exporter
|
|
9. Verify end-to-end: LiteLLM → Router → Model
|
|
|
|
### remove-model
|
|
1. Drain active requests (wait for active=0)
|
|
2. Remove from LiteLLM config
|
|
3. Remove from router config
|
|
4. Stop llama-server (systemd)
|
|
5. Remove Prometheus scrape target
|
|
6. Cleanup model files (optional)
|
|
|
|
### heal
|
|
1. Check all GPUs via router internal `:9000/health/unified`
|
|
2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
|
|
3. Reset stuck circuit breakers if idle (Redis)
|
|
4. Restart dead llama-server instances via SSH
|
|
5. Flush Redis active counters if stale
|
|
6. Verify GPU monitor server is running on pi (:9100)
|
|
7. Verify watchdog is running on pi
|
|
8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
|
|
9. Reload roster via `POST :9000/admin/roster/reload` if available
|
|
|
|
### sync-keys
|
|
1. List all agent keys in LiteLLM DB via `GET /key/list`
|
|
2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
|
|
3. Generate missing keys via `POST /key/generate` with unlimited budget
|
|
4. Update agent configs — `/etc/environment` LITELLM_API_KEY
|
|
5. Send Zulip DM to agents that can't be reached via SSH
|
|
6. Verify each key with test request through full chain
|
|
7. Document keys in knowledge graph
|
|
|
|
### list
|
|
Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys
|
|
|
|
## Agent Keys (LiteLLM DB — Current 2026-06-30)
|
|
|
|
| Agent | Key | Host | Access |
|
|
|-------|-----|------|--------|
|
|
| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
|
|
| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
|
|
| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
|
|
| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip DM |
|
|
| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
|
|
| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
|
|
|
|
**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
|
|
If no SSH access, send Zulip DM via abiba-bot.
|
|
|
|
## Configuration Files
|
|
|
|
| File | Host | Purpose |
|
|
|------|------|---------|
|
|
| `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
|
|
| `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
|
|
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
|
|
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /grafana/, /health |
|
|
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
|
|
| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
|
|
| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
|
|
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
|
|
| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
|
|
| `/etc/systemd/system/llama-server.service` | .8, .110, .15 | llama-server daemons |
|
|
|
|
## Prometheus & Grafana
|
|
|
|
| Component | URL | Details |
|
|
|-----------|-----|---------|
|
|
| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
|
|
| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
|
|
| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
|
|
| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
|
|
| Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |
|
|
|
|
## Known Issues & Watch Points
|
|
|
|
- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
|
|
Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
|
|
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
|
|
- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
|
|
- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
|
|
- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
|
|
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
|
|
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. Localhost curls hang (SYN dropped). Always test from .116.
|
|
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
|
|
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
|
|
- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
|
|
- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
|
|
- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.
|
|
|
|
## GPU Inference Benchmarks (Current)
|
|
|
|
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
|
|
|-----|-------|-----------|--------------|----------|---------|
|
|
| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
|
|
| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
|
|
| Strix Halo (.15) | ornith-1.0-35b | 70 | 209 | 70 | 6 |
|
|
|
|
Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
|
|
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
|
|
History stored at `/root/data/toks-history.json` with 7-day rolling window.
|