root
45641e7ae7
- Polls GPU sidecars every 15s for temp, VRAM, power, utilization - Creates responsive HTML dashboard at /root/dashboard/gpu-fleet.html - Served via nginx at /gpu/ on CT 116 - PM2-managed gpu-monitor server on port 9100 - Tracks Strix Halo llama-server via SSH - Exposes JSON API at /gpu-data for integrations
60 lines
3.0 KiB
Markdown
60 lines
3.0 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: gpu-monitor
|
|
description: >
|
|
Monitors GPU fleet health, utilization, and model routing in
|
|
real-time. Polls sidecar metrics every 30s, logs to knowledge graph,
|
|
generates a live HTML dashboard, and alerts on anomalies.
|
|
---
|
|
|
|
## Maintains
|
|
|
|
- dense-gpu: { host: "192.168.68.8", name: "RTX 3090", vram: "24GB", models: ["gemma-4-12b"] }
|
|
- moe-gpu: { host: "192.168.68.110", name: "RTX 5070", vram: "12GB", models: ["qwen3.6-27B-code"] }
|
|
- strix-halo: { host: "192.168.68.15", name: "AMD Strix Halo", models: ["ornith-1.0-35b"], engine: "llama-server CPU" }
|
|
- router-health: { model_availability: array, circuit_breakers: array }
|
|
|
|
## Dashboard (live HTML)
|
|
|
|
Generated at `/root/dashboard/gpu-fleet.html` — responsive, mobile-friendly, auto-refreshes every 30s.
|
|
|
|
### Layout
|
|
```
|
|
┌─────────────────────────────────────────────────────┐
|
|
│ 🖥️ GPU Fleet Dashboard [live] │
|
|
├──────────────────┬──────────────────┬────────────────┤
|
|
│ Dense (RTX 3090)│ MoE (RTX 5070) │ Strix Halo │
|
|
│ ─────────────── │ ─────────────── │ ───────────── │
|
|
│ Temp: 41°C │ Temp: 39°C │ CPU: 16t │
|
|
│ VRAM: 23.4/24.6 │ VRAM: 11.2/12.2│ Context: 262K │
|
|
│ Power: 12W │ Power: 5W │ Model: 35B │
|
|
│ Model: gemma │ Model: qwen27B │ │
|
|
├──────────────────┴──────────────────┴────────────────┤
|
|
│ Router Health: ✅ All models available │
|
|
│ Circuit Breakers: gemma=0 qwen=0 ornith=0 │
|
|
│ Last updated: 2026-06-29 22:30 UTC │
|
|
└─────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Alert Thresholds
|
|
|
|
| Metric | Warning | Critical |
|
|
|--------|---------|----------|
|
|
| GPU Temp | >80°C | >90°C |
|
|
| VRAM Usage | >90% | >95% |
|
|
| GPU Util | >95% for 5min | >95% for 15min |
|
|
| Model Down | — | No response to test prompt |
|
|
| Circuit Breaker Open | 1 model | >1 model |
|
|
|
|
## Execution
|
|
|
|
1. **Poll sidecars** (every 30s):
|
|
- GET http://192.168.68.8:8090/health → RTX 3090 metrics
|
|
- GET http://192.168.68.110:8090/health → RTX 5070 metrics
|
|
- SSH 192.168.68.15 → llama-server process stats
|
|
2. **Poll router**: GET /health/unified for model health + circuit breakers
|
|
3. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
|
|
4. **Check thresholds**: Alert via Zulip DM if any metric out of bounds
|
|
5. **Log to KG**: Create [LEARN] node hourly with aggregated metrics
|
|
6. **Repeat** every 30 seconds
|