fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring)

- litellm-health: 'check-litellm-health' -> 'litellm-health' - infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring' Fixes 'prose run <filename>' mismatch. No external references broken.
2026-07-02 20:35:44 +00:00
parent b15771bfd9
commit bb20637b32
2 changed files with 240 additions and 49 deletions
@@ -0,0 +1,150 @@
+---
+kind: function
+name: infrastructure-monitoring
+description: >
+  Deploys Prometheus + GPU exporters + Grafana to monitor the entire
+  inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
+  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
+  via existing /metrics Prometheus endpoint. Replaces the deprecated
+  harness-dashboard with a production-grade monitoring stack.
+version: 1.0.0
+---
+
+## Architecture
+
+```
+GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
+  nvidia-exporter       nvidia-exporter       amdgpu-exporter
+  :9400                 :9400                 :9400
+       │                     │                     │
+       └─────────────────────┼─────────────────────┘
+                             ▼
+              ┌──────────────────────────┐
+              │       Prometheus         │
+              │       CT 116 :9090       │
+              │                          │
+              │  Scrape targets:         │
+              │  • 192.168.68.8:9400     │
+              │  • 192.168.68.110:9400   │
+              │  • 192.168.68.15:9400    │
+              │  • litellm:4000/metrics  │
+              └──────────┬───────────────┘
+                         │
+              ┌──────────▼───────────────┐
+              │        Grafana           │
+              │       CT 116 :3001       │
+              │                          │
+              │  Preloaded dashboards:   │
+              │  • GPU Fleet Overview    │
+              │  • LiteLLM Proxy Stats   │
+              └──────────────────────────┘
+```
+
+## Components
+
+### 1. NVIDIA GPU Exporter (hosts: .8, .110)
+- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
+
+### 2. AMD GPU Exporter (host: .15)
+- Custom exporter: Python script wrapping `amdgpu_top --json`
+- Listens on `:9400`, exposes `/metrics` in Prometheus format
+- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
+- Runs as systemd service for persistence
+
+### 3. Prometheus (CT 116)
+- Container: `prom/prometheus:latest`
+- Port: `9090` (internal Docker network)
+- Scrape interval: 15s
+- Config: `/opt/monitoring/prometheus.yml`
+- Storage: Docker volume `prometheus-data`
+
+### 4. Grafana (CT 116)
+- Container: `grafana/grafana:latest`
+- Port: `3001` (mapped to host)
+- Data source: Prometheus at `http://prometheus:9090`
+- Provisioned dashboards for GPU fleet + LiteLLM
+- Accessible at `http://192.168.68.116:3001`
+
+## Parameters
+
+- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
+- gpu_amd_hosts: ["192.168.68.15"]
+- monitoring_host: "192.168.68.116"
+- prometheus_port: 9090
+- grafana_port: 3001
+- gpu_exporter_port: 9400
+
+## Requires
+
+- SSH access to all GPU hosts for exporter deployment
+- Docker on CT 116 for Prometheus + Grafana containers
+- Python 3 on AMD host for custom exporter
+- nvidia-smi on NVIDIA hosts
+
+## Maintains
+
+- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
+- Prometheus scrapes all targets every 15s
+- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
+- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
+- Stack persists across reboots (systemd for exporters, Docker restart policy)
+
+## Execution
+
+### Phase 1: GPU Exporters
+
+**NVIDIA (.8 and .110)**:
+1. Download `nvidia_gpu_exporter` binary
+2. Create systemd service `nvidia-gpu-exporter.service`
+3. Start and enable
+
+**AMD (.15)**:
+1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
+2. Parses `amdgpu_top --json -d 1000` output
+3. Exposes key metrics at `:9400/metrics` via Python http.server
+4. Create systemd service
+5. Start and enable
+
+### Phase 2: Prometheus
+
+1. Create `/opt/monitoring/` directory on CT 116
+2. Write `prometheus.yml` with scrape configs for all targets
+3. Add to docker-compose (or separate compose file)
+4. Start container
+
+### Phase 3: Grafana
+
+1. Create `/opt/monitoring/grafana/` directories
+2. Provision Prometheus datasource
+3. Provision GPU fleet dashboard JSON
+4. Provision LiteLLM dashboard JSON
+5. Add to docker-compose
+6. Start container
+
+### Phase 4: Verification
+
+1. Verify all 3 GPU exporters return 200 at :9400/metrics
+2. Verify Prometheus targets all UP at :9090/targets
+3. Verify Grafana accessible at :3001 with dashboards
+4. Verify LiteLLM metrics flowing to Prometheus
+5. Update nginx to proxy `/monitoring/` → Grafana (optional)
+
+## Verification Commands
+
+```bash
+# GPU exporters
+curl -s http://192.168.68.8:9400/metrics | grep nvidia
+curl -s http://192.168.68.110:9400/metrics | grep nvidia
+curl -s http://192.168.68.15:9400/metrics | grep amdgpu
+
+# Prometheus
+curl -s http://192.168.68.116:9090/api/v1/targets
+
+# Grafana
+curl -s http://192.168.68.116:3001/api/health
+
+# LiteLLM metrics (already live)
+curl -s http://192.168.68.116:4001/metrics | head -20
+```