feat: gpu-monitor — real-time GPU fleet dashboard with live metrics

- Polls GPU sidecars every 15s for temp, VRAM, power, utilization - Creates responsive HTML dashboard at /root/dashboard/gpu-fleet.html - Served via nginx at /gpu/ on CT 116 - PM2-managed gpu-monitor server on port 9100 - Tracks Strix Halo llama-server via SSH - Exposes JSON API at /gpu-data for integrations
2026-06-29 22:35:30 +00:00
parent 4ee441f622
commit 45641e7ae7
1 changed files with 59 additions and 0 deletions
@@ -0,0 +1,59 @@
+---
+kind: responsibility
+name: gpu-monitor
+description: >
+  Monitors GPU fleet health, utilization, and model routing in
+  real-time. Polls sidecar metrics every 30s, logs to knowledge graph,
+  generates a live HTML dashboard, and alerts on anomalies.
+---
+
+## Maintains
+
+- dense-gpu: { host: "192.168.68.8", name: "RTX 3090", vram: "24GB", models: ["gemma-4-12b"] }
+- moe-gpu: { host: "192.168.68.110", name: "RTX 5070", vram: "12GB", models: ["qwen3.6-27B-code"] }
+- strix-halo: { host: "192.168.68.15", name: "AMD Strix Halo", models: ["ornith-1.0-35b"], engine: "llama-server CPU" }
+- router-health: { model_availability: array, circuit_breakers: array }
+
+## Dashboard (live HTML)
+
+Generated at `/root/dashboard/gpu-fleet.html` — responsive, mobile-friendly, auto-refreshes every 30s.
+
+### Layout
+```
+┌─────────────────────────────────────────────────────┐
+│ 🖥️  GPU Fleet Dashboard                    [live]   │
+├──────────────────┬──────────────────┬────────────────┤
+│  Dense (RTX 3090)│  MoE (RTX 5070)  │  Strix Halo    │
+│  ─────────────── │  ─────────────── │  ───────────── │
+│  Temp: 41°C      │  Temp: 39°C      │  CPU: 16t      │
+│  VRAM: 23.4/24.6 │  VRAM: 11.2/12.2│  Context: 262K │
+│  Power: 12W      │  Power: 5W       │  Model: 35B    │
+│  Model: gemma    │  Model: qwen27B  │                │
+├──────────────────┴──────────────────┴────────────────┤
+│  Router Health: ✅ All models available              │
+│  Circuit Breakers: gemma=0 qwen=0 ornith=0         │
+│  Last updated: 2026-06-29 22:30 UTC                 │
+└─────────────────────────────────────────────────────┘
+```
+
+## Alert Thresholds
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| GPU Temp | >80°C | >90°C |
+| VRAM Usage | >90% | >95% |
+| GPU Util | >95% for 5min | >95% for 15min |
+| Model Down | — | No response to test prompt |
+| Circuit Breaker Open | 1 model | >1 model |
+
+## Execution
+
+1. **Poll sidecars** (every 30s):
+   - GET http://192.168.68.8:8090/health → RTX 3090 metrics
+   - GET http://192.168.68.110:8090/health → RTX 5070 metrics
+   - SSH 192.168.68.15 → llama-server process stats
+2. **Poll router**: GET /health/unified for model health + circuit breakers
+3. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
+4. **Check thresholds**: Alert via Zulip DM if any metric out of bounds
+5. **Log to KG**: Create [LEARN] node hourly with aggregated metrics
+6. **Repeat** every 30 seconds