---
kind: function
name: infrastructure-monitoring
description: >
  Deploys Prometheus + GPU exporters + Grafana to monitor the entire
  inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
  from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
  via existing /metrics Prometheus endpoint. Replaces the deprecated
  harness-dashboard with a production-grade monitoring stack.
version: 1.0.0
---

## Architecture

```
GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
  nvidia-exporter       nvidia-exporter       amdgpu-exporter
  :9400                 :9400                 :9400
       │                     │                     │
       └─────────────────────┼─────────────────────┘
                             ▼
              ┌──────────────────────────┐
              │       Prometheus         │
              │       CT 116 :9090       │
              │                          │
              │  Scrape targets:         │
              │  • 192.168.68.8:9400     │
              │  • 192.168.68.110:9400   │
              │  • 192.168.68.15:9400    │
              │  • litellm:4000/metrics  │
              └──────────┬───────────────┘
                         │
              ┌──────────▼───────────────┐
              │        Grafana           │
              │       CT 116 :3001       │
              │                          │
              │  Preloaded dashboards:   │
              │  • GPU Fleet Overview    │
              │  • LiteLLM Proxy Stats   │
              └──────────────────────────┘
```

## Components

### 1. NVIDIA GPU Exporter (hosts: .8, .110)
- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
- Listens on `:9400`, exposes `/metrics` in Prometheus format
- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed

### 2. AMD GPU Exporter (host: .15)
- Custom exporter: Python script wrapping `amdgpu_top --json`
- Listens on `:9400`, exposes `/metrics` in Prometheus format
- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
- Runs as systemd service for persistence

### 3. Prometheus (CT 116)
- Container: `prom/prometheus:latest`
- Port: `9090` (internal Docker network)
- Scrape interval: 15s
- Config: `/opt/monitoring/prometheus.yml`
- Storage: Docker volume `prometheus-data`

### 4. Grafana (CT 116)
- Container: `grafana/grafana:latest`
- Port: `3001` (mapped to host)
- Data source: Prometheus at `http://prometheus:9090`
- Provisioned dashboards for GPU fleet + LiteLLM
- Accessible at `http://192.168.68.116:3001`

## Parameters

- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
- gpu_amd_hosts: ["192.168.68.15"]
- monitoring_host: "192.168.68.116"
- prometheus_port: 9090
- grafana_port: 3001
- gpu_exporter_port: 9400

## Requires

- SSH access to all GPU hosts for exporter deployment
- Docker on CT 116 for Prometheus + Grafana containers
- Python 3 on AMD host for custom exporter
- nvidia-smi on NVIDIA hosts

## Maintains

- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
- Prometheus scrapes all targets every 15s
- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
- Stack persists across reboots (systemd for exporters, Docker restart policy)

## Execution

### Phase 1: GPU Exporters

**NVIDIA (.8 and .110)**:
1. Download `nvidia_gpu_exporter` binary
2. Create systemd service `nvidia-gpu-exporter.service`
3. Start and enable

**AMD (.15)**:
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
2. Parses `amdgpu_top --json -d 1000` output
3. Exposes key metrics at `:9400/metrics` via Python http.server
4. Create systemd service
5. Start and enable

### Phase 2: Prometheus

1. Create `/opt/monitoring/` directory on CT 116
2. Write `prometheus.yml` with scrape configs for all targets
3. Add to docker-compose (or separate compose file)
4. Start container

### Phase 3: Grafana

1. Create `/opt/monitoring/grafana/` directories
2. Provision Prometheus datasource
3. Provision GPU fleet dashboard JSON
4. Provision LiteLLM dashboard JSON
5. Add to docker-compose
6. Start container

### Phase 4: Verification

1. Verify all 3 GPU exporters return 200 at :9400/metrics
2. Verify Prometheus targets all UP at :9090/targets
3. Verify Grafana accessible at :3001 with dashboards
4. Verify LiteLLM metrics flowing to Prometheus
5. Update nginx to proxy `/monitoring/` → Grafana (optional)

## Verification Commands

```bash
# GPU exporters
curl -s http://192.168.68.8:9400/metrics | grep nvidia
curl -s http://192.168.68.110:9400/metrics | grep nvidia
curl -s http://192.168.68.15:9400/metrics | grep amdgpu

# Prometheus
curl -s http://192.168.68.116:9090/api/v1/targets

# Grafana
curl -s http://192.168.68.116:3001/api/health

# LiteLLM metrics (already live)
curl -s http://192.168.68.116:4001/metrics | head -20
```