prose-contracts/infrastructure-monitoring.prose.md

kind, name, description, version

kind	name	description	version
function	infrastructure-monitoring	Deploys Prometheus + GPU exporters + Grafana to monitor the entire inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics via existing /metrics Prometheus endpoint. DEPLOYMENT STATUS (2026-07-09): ✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters (via proxmox-monitor contract). Grafana at :3001, 5 scrape targets active. ❌ GPU exporters NOT deployed: gpu-exporter crash-loops on .15, NVIDIA sidecar exporters (.8/.110:9400) never installed. Router falls back to direct GPU /health probes. ⚠️ This contract is target-state aspirational — not as-built. As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).	1.0.0

Architecture

GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
  nvidia-exporter       nvidia-exporter       amdgpu-exporter
  :9400                 :9400                 :9400
       │                     │                     │
       └─────────────────────┼─────────────────────┘
                             ▼
              ┌──────────────────────────┐
              │       Prometheus         │
              │       CT 116 :9090       │
              │                          │
              │  Scrape targets:         │
              │  • 192.168.68.8:9400     │
              │  • 192.168.68.110:9400   │
              │  • 192.168.68.15:9400    │
              │  • litellm:4000/metrics  │
              └──────────┬───────────────┘
                         │
              ┌──────────▼───────────────┐
              │        Grafana           │
              │       CT 116 :3001       │
              │                          │
              │  Preloaded dashboards:   │
              │  • GPU Fleet Overview    │
              │  • LiteLLM Proxy Stats   │
              └──────────────────────────┘

Components

1. NVIDIA GPU Exporter (hosts: .8, .110)

• Tool: utkuozdemir/nvidia_gpu_exporter (Go binary, single static binary)
• Listens on :9400, exposes /metrics in Prometheus format
• Metrics: utilization, temp, VRAM, power, clock speeds, fan speed

2. AMD GPU Exporter (host: .15)

• Custom exporter: Python script wrapping amdgpu_top --json
• Listens on :9400, exposes /metrics in Prometheus format
• Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
• Runs as systemd service for persistence

3. Prometheus (CT 116)

• Container: prom/prometheus:latest
• Port: 9090 (internal Docker network)
• Scrape interval: 15s
• Config: /opt/monitoring/prometheus.yml
• Storage: Docker volume prometheus-data

4. Grafana (CT 116)

• Container: grafana/grafana:latest
• Port: 3001 (mapped to host)
• Data source: Prometheus at http://prometheus:9090
• Provisioned dashboards for GPU fleet + LiteLLM
• Accessible at http://192.168.68.116:3001

Parameters

• gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
• gpu_amd_hosts: ["192.168.68.15"]
• monitoring_host: "192.168.68.116"
• prometheus_port: 9090
• grafana_port: 3001
• gpu_exporter_port: 9400

Requires

• SSH access to all GPU hosts for exporter deployment
• Docker on CT 116 for Prometheus + Grafana containers
• Python 3 on AMD host for custom exporter
• nvidia-smi on NVIDIA hosts

Maintains

• All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
• Prometheus scrapes all targets every 15s
• Grafana dashboards show real-time GPU utilization, temp, VRAM, power
• LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
• Stack persists across reboots (systemd for exporters, Docker restart policy)

Execution

Phase 1: GPU Exporters

NVIDIA (.8 and .110):

1. Download nvidia_gpu_exporter binary
2. Create systemd service nvidia-gpu-exporter.service
3. Start and enable

AMD (.15):

1. Create Python exporter script at /opt/amdgpu-exporter/exporter.py
2. Parses amdgpu_top --json -d 1000 output
3. Exposes key metrics at :9400/metrics via Python http.server
4. Create systemd service
5. Start and enable

Phase 2: Prometheus

1. Create /opt/monitoring/ directory on CT 116
2. Write prometheus.yml with scrape configs for all targets
3. Add to docker-compose (or separate compose file)
4. Start container

Phase 3: Grafana

1. Create /opt/monitoring/grafana/ directories
2. Provision Prometheus datasource
3. Provision GPU fleet dashboard JSON
4. Provision LiteLLM dashboard JSON
5. Add to docker-compose
6. Start container

Phase 4: Verification

1. Verify all 3 GPU exporters return 200 at :9400/metrics
2. Verify Prometheus targets all UP at :9090/targets
3. Verify Grafana accessible at :3001 with dashboards
4. Verify LiteLLM metrics flowing to Prometheus
5. Update nginx to proxy /monitoring/ → Grafana (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)

Verification Commands

# GPU exporters
curl -s http://192.168.68.8:9400/metrics | grep nvidia
curl -s http://192.168.68.110:9400/metrics | grep nvidia
curl -s http://192.168.68.15:9400/metrics | grep amdgpu

# Prometheus
curl -s http://192.168.68.116:9090/api/v1/targets

# Grafana
curl -s http://192.168.68.116:3001/api/health

# LiteLLM metrics (already live)
curl -s http://192.168.68.116:4001/metrics | head -20