root
38d2f7d956
- gpu-fleet: documented parallel=2/batch=4096 optimizations for RTX 3090/5070 - gpu-fleet: added Strix Halo ROCm status (installed, blocked by HSA runtime on Debian 13) - gpu-fleet: documented alert migration from DM → #agent-hub stream topics - gpu-monitor: added Alert Delivery section with stream topic conventions - All GPU alerts now go to #agent-hub for cross-agent visibility
132 lines
5.5 KiB
Markdown
132 lines
5.5 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: gpu-monitor
|
|
description: >
|
|
Comprehensive GPU fleet monitor — polls every subsystem (sidecars, router,
|
|
LiteLLM, Strix Halo, dashboard) every 15s, renders a live HTML dashboard,
|
|
checks alert thresholds, and exposes a JSON API for downstream consumers.
|
|
agent: abiba
|
|
---
|
|
|
|
## Maintains
|
|
|
|
- gpu-fleet-data: { gpus: array, strix: object, router: object, litellm: object, summary: object, alerts: array } — Full fleet snapshot refreshed every 15s
|
|
- dashboard: live HTML at `/root/dashboard/gpu-fleet.html`, served on port 9100
|
|
- gpu-data-api: JSON at `GET /gpu-data` on port 9100
|
|
- health-endpoint: `GET /health` on port 9100 → 200 if cache populated, 503 if warming up
|
|
|
|
## GPU Fleet Topology
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ GPU Monitor Server (:9100) — 192.168.68.65 │
|
|
│ Polls every 15s, serves dashboard + JSON API │
|
|
└──┬──────────┬──────────┬──────────┬─────────────────────┘
|
|
│ │ │ │
|
|
▼ ▼ ▼ ▼
|
|
┌──────┐ ┌──────┐ ┌────────┐ ┌──────────┐
|
|
│.8:8090│ │.110 │ │.116:80 │ │.15 (SSH) │
|
|
│RTX3090│ │:8090 │ │nginx │ │Strix Halo│
|
|
│gemma │ │RTX5070│ │router │ │ornith35B │
|
|
└──────┘ │qwen27B│ │LiteLLM │ └──────────┘
|
|
└──────┘ │dashboard│
|
|
└────────┘
|
|
```
|
|
|
|
### Subsystems Polled
|
|
|
|
| Subsystem | Endpoint | Frequency | Metrics |
|
|
|-----------|----------|-----------|---------|
|
|
| GPU Sidecar (RTX 3090) | `http://192.168.68.8:8090/health` | 15s | temp, VRAM, util, power, fan |
|
|
| GPU Sidecar (RTX 5070) | `http://192.168.68.110:8090/health` | 15s | temp, VRAM, util, power, fan |
|
|
| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
|
|
| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
|
|
| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
|
|
| Strix Halo | SSH `192.168.68.15` + `:8080/health` | 15s | process status, CPU load, llama health |
|
|
| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
|
|
|
|
### Alert Delivery
|
|
|
|
All alerts are sent to `#agent-hub` stream topics:
|
|
- `alerts-gpu` — GPU fleet alerts (temp, VRAM, utilization)
|
|
- `alerts-pm2` — PM2 process health (restarts, crashes)
|
|
- `alerts-infra` — Infrastructure health (LiteLLM, Zulip, containers)
|
|
|
|
This replaces the previous DM-only delivery. All agents on the mesh can see and respond to alerts.
|
|
|
|
## Alert Thresholds
|
|
|
|
| Metric | Warning | Critical |
|
|
|--------|---------|----------|
|
|
| GPU Temp | >80°C | >90°C |
|
|
| VRAM Usage | >90% | >95% |
|
|
| GPU Util | >95% | >98% |
|
|
| Sidecar Unreachable | — | critical (connectivity) |
|
|
| Model Down | — | critical (circuit breaker open) |
|
|
|
|
### JSON API Response Schema (/gpu-data)
|
|
|
|
```json
|
|
{
|
|
"gpus": [{ "name", "gpu_name", "temp_c", "vram_used_mb", "vram_total_mb",
|
|
"gpu_util_pct", "power_w", "power_limit_w", "fan_pct",
|
|
"models", "hostname", "active_requests", "health_score",
|
|
"circuit_open" }],
|
|
"strix": { "status", "cpu_load", "llama_health" },
|
|
"router": { "available_models", "circuit_breaker", "gpus", "scores",
|
|
"status", "redis", "_basic" },
|
|
"litellm": { ... },
|
|
"dashboard": { "reachable": bool },
|
|
"summary": { "fleet_status", "models_available", "models_total",
|
|
"circuit_breakers_open", "gpu_count", "gpu_errors",
|
|
"strix_running", "router_reachable", "litellm_reachable" },
|
|
"alerts": [{ "gpu", "metric", "level", "value", "threshold" }],
|
|
"updated": "ISO8601",
|
|
"monitor_version": "2.0.0"
|
|
}
|
|
```
|
|
|
|
## Operations
|
|
|
|
### list
|
|
`curl http://localhost:9100/gpu-data | jq` — Full fleet status
|
|
|
|
### check-health
|
|
`curl http://localhost:9100/health` — Monitor self-check
|
|
|
|
### view-dashboard
|
|
Open `http://localhost:9100/` in browser — Live HTML dashboard
|
|
|
|
### restart
|
|
```bash
|
|
pkill -f gpu-monitor-server.py
|
|
python3 /root/scripts/gpu-monitor-server.py &
|
|
```
|
|
Or via PM2: `pm2 restart gpu-monitor`
|
|
|
|
### check-router
|
|
The router health is accessed through nginx on port 80 (NOT port 9000 directly).
|
|
`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy
|
|
`curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
|
|
|
|
## Configuration Files
|
|
|
|
| File | Host | Purpose |
|
|
|------|------|---------|
|
|
| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
|
|
| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
|
|
| `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |
|
|
|
|
## Execution
|
|
|
|
1. **Poll sidecars** (every 15s): GET .8:8090/health, .110:8090/health
|
|
2. **Poll router** (every 15s): GET .116/health/unified + /health via nginx:80
|
|
3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
|
|
4. **Poll Strix** (every 15s): SSH .15 for process check + GET .15:8080/health
|
|
5. **Poll dashboard** (every 15s): GET .116/dashboard/
|
|
6. **Check alerts**: Compare metrics against thresholds
|
|
7. **Compute summary**: Fleet-wide health aggregation
|
|
8. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
|
|
9. **Serve API**: HTTP server on port 9100
|
|
10. **Repeat** every 15 seconds
|