---
kind: responsibility
name: gpu-monitor
description: >
  Monitors GPU fleet health, utilization, and model routing in
  real-time. Polls sidecar metrics every 30s, logs to knowledge graph,
  generates a live HTML dashboard, and alerts on anomalies.
---

## Maintains

- dense-gpu: { host: "192.168.68.8", name: "RTX 3090", vram: "24GB", models: ["gemma-4-12b"] }
- moe-gpu: { host: "192.168.68.110", name: "RTX 5070", vram: "12GB", models: ["qwen3.6-27B-code"] }
- strix-halo: { host: "192.168.68.15", name: "AMD Strix Halo", models: ["ornith-1.0-35b"], engine: "llama-server CPU" }
- router-health: { model_availability: array, circuit_breakers: array }

## Dashboard (live HTML)

Generated at `/root/dashboard/gpu-fleet.html` — responsive, mobile-friendly, auto-refreshes every 30s.

### Layout
```
┌─────────────────────────────────────────────────────┐
│ 🖥️  GPU Fleet Dashboard                    [live]   │
├──────────────────┬──────────────────┬────────────────┤
│  Dense (RTX 3090)│  MoE (RTX 5070)  │  Strix Halo    │
│  ─────────────── │  ─────────────── │  ───────────── │
│  Temp: 41°C      │  Temp: 39°C      │  CPU: 16t      │
│  VRAM: 23.4/24.6 │  VRAM: 11.2/12.2│  Context: 262K │
│  Power: 12W      │  Power: 5W       │  Model: 35B    │
│  Model: gemma    │  Model: qwen27B  │                │
├──────────────────┴──────────────────┴────────────────┤
│  Router Health: ✅ All models available              │
│  Circuit Breakers: gemma=0 qwen=0 ornith=0         │
│  Last updated: 2026-06-29 22:30 UTC                 │
└─────────────────────────────────────────────────────┘
```

## Alert Thresholds

| Metric | Warning | Critical |
|--------|---------|----------|
| GPU Temp | >80°C | >90°C |
| VRAM Usage | >90% | >95% |
| GPU Util | >95% for 5min | >95% for 15min |
| Model Down | — | No response to test prompt |
| Circuit Breaker Open | 1 model | >1 model |

## Execution

1. **Poll sidecars** (every 30s):
   - GET http://192.168.68.8:8090/health → RTX 3090 metrics
   - GET http://192.168.68.110:8090/health → RTX 5070 metrics
   - SSH 192.168.68.15 → llama-server process stats
2. **Poll router**: GET /health/unified for model health + circuit breakers
3. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
4. **Check thresholds**: Alert via Zulip DM if any metric out of bounds
5. **Log to KG**: Create [LEARN] node hourly with aggregated metrics
6. **Repeat** every 30 seconds