From 45641e7ae778f3ec35fddbae5f76c4abb5e433a3 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 29 Jun 2026 22:35:30 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20gpu-monitor=20=E2=80=94=20real-time=20G?= =?UTF-8?q?PU=20fleet=20dashboard=20with=20live=20metrics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Polls GPU sidecars every 15s for temp, VRAM, power, utilization - Creates responsive HTML dashboard at /root/dashboard/gpu-fleet.html - Served via nginx at /gpu/ on CT 116 - PM2-managed gpu-monitor server on port 9100 - Tracks Strix Halo llama-server via SSH - Exposes JSON API at /gpu-data for integrations --- gpu-monitor.prose.md | 59 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 gpu-monitor.prose.md diff --git a/gpu-monitor.prose.md b/gpu-monitor.prose.md new file mode 100644 index 0000000..88f13f7 --- /dev/null +++ b/gpu-monitor.prose.md @@ -0,0 +1,59 @@ +--- +kind: responsibility +name: gpu-monitor +description: > + Monitors GPU fleet health, utilization, and model routing in + real-time. Polls sidecar metrics every 30s, logs to knowledge graph, + generates a live HTML dashboard, and alerts on anomalies. +--- + +## Maintains + +- dense-gpu: { host: "192.168.68.8", name: "RTX 3090", vram: "24GB", models: ["gemma-4-12b"] } +- moe-gpu: { host: "192.168.68.110", name: "RTX 5070", vram: "12GB", models: ["qwen3.6-27B-code"] } +- strix-halo: { host: "192.168.68.15", name: "AMD Strix Halo", models: ["ornith-1.0-35b"], engine: "llama-server CPU" } +- router-health: { model_availability: array, circuit_breakers: array } + +## Dashboard (live HTML) + +Generated at `/root/dashboard/gpu-fleet.html` — responsive, mobile-friendly, auto-refreshes every 30s. + +### Layout +``` +┌─────────────────────────────────────────────────────┐ +│ 🖥️ GPU Fleet Dashboard [live] │ +├──────────────────┬──────────────────┬────────────────┤ +│ Dense (RTX 3090)│ MoE (RTX 5070) │ Strix Halo │ +│ ─────────────── │ ─────────────── │ ───────────── │ +│ Temp: 41°C │ Temp: 39°C │ CPU: 16t │ +│ VRAM: 23.4/24.6 │ VRAM: 11.2/12.2│ Context: 262K │ +│ Power: 12W │ Power: 5W │ Model: 35B │ +│ Model: gemma │ Model: qwen27B │ │ +├──────────────────┴──────────────────┴────────────────┤ +│ Router Health: ✅ All models available │ +│ Circuit Breakers: gemma=0 qwen=0 ornith=0 │ +│ Last updated: 2026-06-29 22:30 UTC │ +└─────────────────────────────────────────────────────┘ +``` + +## Alert Thresholds + +| Metric | Warning | Critical | +|--------|---------|----------| +| GPU Temp | >80°C | >90°C | +| VRAM Usage | >90% | >95% | +| GPU Util | >95% for 5min | >95% for 15min | +| Model Down | — | No response to test prompt | +| Circuit Breaker Open | 1 model | >1 model | + +## Execution + +1. **Poll sidecars** (every 30s): + - GET http://192.168.68.8:8090/health → RTX 3090 metrics + - GET http://192.168.68.110:8090/health → RTX 5070 metrics + - SSH 192.168.68.15 → llama-server process stats +2. **Poll router**: GET /health/unified for model health + circuit breakers +3. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html +4. **Check thresholds**: Alert via Zulip DM if any metric out of bounds +5. **Log to KG**: Create [LEARN] node hourly with aggregated metrics +6. **Repeat** every 30 seconds