root
45641e7ae7
- Polls GPU sidecars every 15s for temp, VRAM, power, utilization - Creates responsive HTML dashboard at /root/dashboard/gpu-fleet.html - Served via nginx at /gpu/ on CT 116 - PM2-managed gpu-monitor server on port 9100 - Tracks Strix Halo llama-server via SSH - Exposes JSON API at /gpu-data for integrations
3.0 KiB
3.0 KiB
kind, name, description
| kind | name | description |
|---|---|---|
| responsibility | gpu-monitor | Monitors GPU fleet health, utilization, and model routing in real-time. Polls sidecar metrics every 30s, logs to knowledge graph, generates a live HTML dashboard, and alerts on anomalies. |
Maintains
- dense-gpu: { host: "192.168.68.8", name: "RTX 3090", vram: "24GB", models: ["gemma-4-12b"] }
- moe-gpu: { host: "192.168.68.110", name: "RTX 5070", vram: "12GB", models: ["qwen3.6-27B-code"] }
- strix-halo: { host: "192.168.68.15", name: "AMD Strix Halo", models: ["ornith-1.0-35b"], engine: "llama-server CPU" }
- router-health: { model_availability: array, circuit_breakers: array }
Dashboard (live HTML)
Generated at /root/dashboard/gpu-fleet.html — responsive, mobile-friendly, auto-refreshes every 30s.
Layout
┌─────────────────────────────────────────────────────┐
│ 🖥️ GPU Fleet Dashboard [live] │
├──────────────────┬──────────────────┬────────────────┤
│ Dense (RTX 3090)│ MoE (RTX 5070) │ Strix Halo │
│ ─────────────── │ ─────────────── │ ───────────── │
│ Temp: 41°C │ Temp: 39°C │ CPU: 16t │
│ VRAM: 23.4/24.6 │ VRAM: 11.2/12.2│ Context: 262K │
│ Power: 12W │ Power: 5W │ Model: 35B │
│ Model: gemma │ Model: qwen27B │ │
├──────────────────┴──────────────────┴────────────────┤
│ Router Health: ✅ All models available │
│ Circuit Breakers: gemma=0 qwen=0 ornith=0 │
│ Last updated: 2026-06-29 22:30 UTC │
└─────────────────────────────────────────────────────┘
Alert Thresholds
| Metric | Warning | Critical |
|---|---|---|
| GPU Temp | >80°C | >90°C |
| VRAM Usage | >90% | >95% |
| GPU Util | >95% for 5min | >95% for 15min |
| Model Down | — | No response to test prompt |
| Circuit Breaker Open | 1 model | >1 model |
Execution
- Poll sidecars (every 30s):
- GET http://192.168.68.8:8090/health → RTX 3090 metrics
- GET http://192.168.68.110:8090/health → RTX 5070 metrics
- SSH 192.168.68.15 → llama-server process stats
- Poll router: GET /health/unified for model health + circuit breakers
- Render dashboard: Generate HTML at /root/dashboard/gpu-fleet.html
- Check thresholds: Alert via Zulip DM if any metric out of bounds
- Log to KG: Create [LEARN] node hourly with aggregated metrics
- Repeat every 30 seconds