feat: update contracts with GPU optimizations, Strix Halo/ROCm status, alert migration

- gpu-fleet: documented parallel=2/batch=4096 optimizations for RTX 3090/5070 - gpu-fleet: added Strix Halo ROCm status (installed, blocked by HSA runtime on Debian 13) - gpu-fleet: documented alert migration from DM → #agent-hub stream topics - gpu-monitor: added Alert Delivery section with stream topic conventions - All GPU alerts now go to #agent-hub for cross-agent visibility
2026-07-01 22:25:26 +00:00
parent 0c976ddbf7
commit 38d2f7d956
2 changed files with 253 additions and 104 deletions
@@ -2,58 +2,130 @@
 kind: responsibility
 name: gpu-monitor
 description: >
-  Monitors GPU fleet health, utilization, and model routing in
-  real-time. Polls sidecar metrics every 30s, logs to knowledge graph,
-  generates a live HTML dashboard, and alerts on anomalies.
+  Comprehensive GPU fleet monitor — polls every subsystem (sidecars, router,
+  LiteLLM, Strix Halo, dashboard) every 15s, renders a live HTML dashboard,
+  checks alert thresholds, and exposes a JSON API for downstream consumers.
+agent: abiba
 ---

 ## Maintains

- dense-gpu: { host: "192.168.68.8", name: "RTX 3090", vram: "24GB", models: ["gemma-4-12b"] }
- moe-gpu: { host: "192.168.68.110", name: "RTX 5070", vram: "12GB", models: ["qwen3.6-27B-code"] }
- strix-halo: { host: "192.168.68.15", name: "AMD Strix Halo", models: ["ornith-1.0-35b"], engine: "llama-server CPU" }
- router-health: { model_availability: array, circuit_breakers: array }
+- gpu-fleet-data: { gpus: array, strix: object, router: object, litellm: object, summary: object, alerts: array } — Full fleet snapshot refreshed every 15s
+- dashboard: live HTML at `/root/dashboard/gpu-fleet.html`, served on port 9100
+- gpu-data-api: JSON at `GET /gpu-data` on port 9100
+- health-endpoint: `GET /health` on port 9100 → 200 if cache populated, 503 if warming up

-## Dashboard (live HTML)
+## GPU Fleet Topology

-Generated at `/root/dashboard/gpu-fleet.html` — responsive, mobile-friendly, auto-refreshes every 30s.
-
-### Layout
 ```
-┌─────────────────────────────────────────────────────┐
-│ 🖥️  GPU Fleet Dashboard                    [live]   │
-├──────────────────┬──────────────────┬────────────────┤
-│  Dense (RTX 3090)│  MoE (RTX 5070)  │  Strix Halo    │
-│  ─────────────── │  ─────────────── │  ───────────── │
-│  Temp: 41°C      │  Temp: 39°C      │  CPU: 16t      │
-│  VRAM: 23.4/24.6 │  VRAM: 11.2/12.2│  Context: 262K │
-│  Power: 12W      │  Power: 5W       │  Model: 35B    │
-│  Model: gemma    │  Model: qwen27B  │                │
-├──────────────────┴──────────────────┴────────────────┤
-│  Router Health: ✅ All models available              │
-│  Circuit Breakers: gemma=0 qwen=0 ornith=0         │
-│  Last updated: 2026-06-29 22:30 UTC                 │
-└─────────────────────────────────────────────────────┘
+┌─────────────────────────────────────────────────────────┐
+│  GPU Monitor Server (:9100) — 192.168.68.65              │
+│  Polls every 15s, serves dashboard + JSON API            │
+└──┬──────────┬──────────┬──────────┬─────────────────────┘
+   │          │          │          │
+   ▼          ▼          ▼          ▼
+┌──────┐ ┌──────┐ ┌────────┐ ┌──────────┐
+│.8:8090│ │.110  │ │.116:80 │ │.15 (SSH) │
+│RTX3090│ │:8090 │ │nginx   │ │Strix Halo│
+│gemma  │ │RTX5070│ │router  │ │ornith35B │
+└──────┘ │qwen27B│ │LiteLLM │ └──────────┘
+         └──────┘ │dashboard│
+                  └────────┘
 ```

+### Subsystems Polled
+
+| Subsystem | Endpoint | Frequency | Metrics |
+|-----------|----------|-----------|---------|
+| GPU Sidecar (RTX 3090) | `http://192.168.68.8:8090/health` | 15s | temp, VRAM, util, power, fan |
+| GPU Sidecar (RTX 5070) | `http://192.168.68.110:8090/health` | 15s | temp, VRAM, util, power, fan |
+| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
+| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
+| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
+| Strix Halo | SSH `192.168.68.15` + `:8080/health` | 15s | process status, CPU load, llama health |
+| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
+
+### Alert Delivery
+
+All alerts are sent to `#agent-hub` stream topics:
+- `alerts-gpu` — GPU fleet alerts (temp, VRAM, utilization)
+- `alerts-pm2` — PM2 process health (restarts, crashes)
+- `alerts-infra` — Infrastructure health (LiteLLM, Zulip, containers)
+
+This replaces the previous DM-only delivery. All agents on the mesh can see and respond to alerts.
+
 ## Alert Thresholds

 | Metric | Warning | Critical |
 |--------|---------|----------|
 | GPU Temp | >80°C | >90°C |
 | VRAM Usage | >90% | >95% |
-| GPU Util | >95% for 5min | >95% for 15min |
-| Model Down | — | No response to test prompt |
-| Circuit Breaker Open | 1 model | >1 model |
+| GPU Util | >95% | >98% |
+| Sidecar Unreachable | — | critical (connectivity) |
+| Model Down | — | critical (circuit breaker open) |
+
+### JSON API Response Schema (/gpu-data)
+
+```json
+{
+  "gpus": [{ "name", "gpu_name", "temp_c", "vram_used_mb", "vram_total_mb",
+             "gpu_util_pct", "power_w", "power_limit_w", "fan_pct",
+             "models", "hostname", "active_requests", "health_score",
+             "circuit_open" }],
+  "strix": { "status", "cpu_load", "llama_health" },
+  "router": { "available_models", "circuit_breaker", "gpus", "scores",
+              "status", "redis", "_basic" },
+  "litellm": { ... },
+  "dashboard": { "reachable": bool },
+  "summary": { "fleet_status", "models_available", "models_total",
+               "circuit_breakers_open", "gpu_count", "gpu_errors",
+               "strix_running", "router_reachable", "litellm_reachable" },
+  "alerts": [{ "gpu", "metric", "level", "value", "threshold" }],
+  "updated": "ISO8601",
+  "monitor_version": "2.0.0"
+}
+```
+
+## Operations
+
+### list
+`curl http://localhost:9100/gpu-data | jq` — Full fleet status
+
+### check-health
+`curl http://localhost:9100/health` — Monitor self-check
+
+### view-dashboard
+Open `http://localhost:9100/` in browser — Live HTML dashboard
+
+### restart
+```bash
+pkill -f gpu-monitor-server.py
+python3 /root/scripts/gpu-monitor-server.py &
+```
+Or via PM2: `pm2 restart gpu-monitor`
+
+### check-router
+The router health is accessed through nginx on port 80 (NOT port 9000 directly).
+`curl http://192.168.68.116/health/unified` — Router unified health via nginx proxy
+`curl http://192.168.68.116:9000/health/unified` — ❌ WILL FAIL (port bound to 127.0.0.1 only)
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `/root/scripts/gpu-monitor-server.py` | pi (.65) | Monitor server v2.0.0 |
+| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/etc/nginx/nginx.conf` | CT 116 | Routes /health/* → router |

 ## Execution

-1. **Poll sidecars** (every 30s):
-   - GET http://192.168.68.8:8090/health → RTX 3090 metrics
-   - GET http://192.168.68.110:8090/health → RTX 5070 metrics
-   - SSH 192.168.68.15 → llama-server process stats
-2. **Poll router**: GET /health/unified for model health + circuit breakers
-3. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
-4. **Check thresholds**: Alert via Zulip DM if any metric out of bounds
-5. **Log to KG**: Create [LEARN] node hourly with aggregated metrics
-6. **Repeat** every 30 seconds
+1. **Poll sidecars** (every 15s): GET .8:8090/health, .110:8090/health
+2. **Poll router** (every 15s): GET .116/health/unified + /health via nginx:80
+3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
+4. **Poll Strix** (every 15s): SSH .15 for process check + GET .15:8080/health
+5. **Poll dashboard** (every 15s): GET .116/dashboard/
+6. **Check alerts**: Compare metrics against thresholds
+7. **Compute summary**: Fleet-wide health aggregation
+8. **Render dashboard**: Generate HTML at /root/dashboard/gpu-fleet.html
+9. **Serve API**: HTTP server on port 9100
+10. **Repeat** every 15 seconds