root
0d0215e9dd
- Core stack (Prometheus + Grafana + pve/node/docker exporters) already deployed by proxmox-monitor contract (Jul 2) - GPU exporters (.8/.110/.15) NOT yet live — gpu-exporter crash-loops, sidecars never deployed - Strike nginx /monitoring/ proxy suggestion (sub-path pattern already tried and reverted for /grafana/ per proxmox-monitor) - Add cross-reference: this contract defines target state, proxmox-monitor is the as-built
155 lines
5.5 KiB
Markdown
155 lines
5.5 KiB
Markdown
---
|
|
kind: function
|
|
name: infrastructure-monitoring
|
|
description: >
|
|
Deploys Prometheus + GPU exporters + Grafana to monitor the entire
|
|
inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
|
|
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
|
|
via existing /metrics Prometheus endpoint.
|
|
|
|
STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
|
|
already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
|
|
NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
|
|
This contract defines the target state; proxmox-monitor is the as-built.
|
|
version: 1.0.0
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
|
|
nvidia-exporter nvidia-exporter amdgpu-exporter
|
|
:9400 :9400 :9400
|
|
│ │ │
|
|
└─────────────────────┼─────────────────────┘
|
|
▼
|
|
┌──────────────────────────┐
|
|
│ Prometheus │
|
|
│ CT 116 :9090 │
|
|
│ │
|
|
│ Scrape targets: │
|
|
│ • 192.168.68.8:9400 │
|
|
│ • 192.168.68.110:9400 │
|
|
│ • 192.168.68.15:9400 │
|
|
│ • litellm:4000/metrics │
|
|
└──────────┬───────────────┘
|
|
│
|
|
┌──────────▼───────────────┐
|
|
│ Grafana │
|
|
│ CT 116 :3001 │
|
|
│ │
|
|
│ Preloaded dashboards: │
|
|
│ • GPU Fleet Overview │
|
|
│ • LiteLLM Proxy Stats │
|
|
└──────────────────────────┘
|
|
```
|
|
|
|
## Components
|
|
|
|
### 1. NVIDIA GPU Exporter (hosts: .8, .110)
|
|
- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
|
|
- Listens on `:9400`, exposes `/metrics` in Prometheus format
|
|
- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
|
|
|
|
### 2. AMD GPU Exporter (host: .15)
|
|
- Custom exporter: Python script wrapping `amdgpu_top --json`
|
|
- Listens on `:9400`, exposes `/metrics` in Prometheus format
|
|
- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
|
|
- Runs as systemd service for persistence
|
|
|
|
### 3. Prometheus (CT 116)
|
|
- Container: `prom/prometheus:latest`
|
|
- Port: `9090` (internal Docker network)
|
|
- Scrape interval: 15s
|
|
- Config: `/opt/monitoring/prometheus.yml`
|
|
- Storage: Docker volume `prometheus-data`
|
|
|
|
### 4. Grafana (CT 116)
|
|
- Container: `grafana/grafana:latest`
|
|
- Port: `3001` (mapped to host)
|
|
- Data source: Prometheus at `http://prometheus:9090`
|
|
- Provisioned dashboards for GPU fleet + LiteLLM
|
|
- Accessible at `http://192.168.68.116:3001`
|
|
|
|
## Parameters
|
|
|
|
- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
|
|
- gpu_amd_hosts: ["192.168.68.15"]
|
|
- monitoring_host: "192.168.68.116"
|
|
- prometheus_port: 9090
|
|
- grafana_port: 3001
|
|
- gpu_exporter_port: 9400
|
|
|
|
## Requires
|
|
|
|
- SSH access to all GPU hosts for exporter deployment
|
|
- Docker on CT 116 for Prometheus + Grafana containers
|
|
- Python 3 on AMD host for custom exporter
|
|
- nvidia-smi on NVIDIA hosts
|
|
|
|
## Maintains
|
|
|
|
- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
|
|
- Prometheus scrapes all targets every 15s
|
|
- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
|
|
- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
|
|
- Stack persists across reboots (systemd for exporters, Docker restart policy)
|
|
|
|
## Execution
|
|
|
|
### Phase 1: GPU Exporters
|
|
|
|
**NVIDIA (.8 and .110)**:
|
|
1. Download `nvidia_gpu_exporter` binary
|
|
2. Create systemd service `nvidia-gpu-exporter.service`
|
|
3. Start and enable
|
|
|
|
**AMD (.15)**:
|
|
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
|
|
2. Parses `amdgpu_top --json -d 1000` output
|
|
3. Exposes key metrics at `:9400/metrics` via Python http.server
|
|
4. Create systemd service
|
|
5. Start and enable
|
|
|
|
### Phase 2: Prometheus
|
|
|
|
1. Create `/opt/monitoring/` directory on CT 116
|
|
2. Write `prometheus.yml` with scrape configs for all targets
|
|
3. Add to docker-compose (or separate compose file)
|
|
4. Start container
|
|
|
|
### Phase 3: Grafana
|
|
|
|
1. Create `/opt/monitoring/grafana/` directories
|
|
2. Provision Prometheus datasource
|
|
3. Provision GPU fleet dashboard JSON
|
|
4. Provision LiteLLM dashboard JSON
|
|
5. Add to docker-compose
|
|
6. Start container
|
|
|
|
### Phase 4: Verification
|
|
|
|
1. Verify all 3 GPU exporters return 200 at :9400/metrics
|
|
2. Verify Prometheus targets all UP at :9090/targets
|
|
3. Verify Grafana accessible at :3001 with dashboards
|
|
4. Verify LiteLLM metrics flowing to Prometheus
|
|
5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
|
|
|
|
## Verification Commands
|
|
|
|
```bash
|
|
# GPU exporters
|
|
curl -s http://192.168.68.8:9400/metrics | grep nvidia
|
|
curl -s http://192.168.68.110:9400/metrics | grep nvidia
|
|
curl -s http://192.168.68.15:9400/metrics | grep amdgpu
|
|
|
|
# Prometheus
|
|
curl -s http://192.168.68.116:9090/api/v1/targets
|
|
|
|
# Grafana
|
|
curl -s http://192.168.68.116:3001/api/health
|
|
|
|
# LiteLLM metrics (already live)
|
|
curl -s http://192.168.68.116:4001/metrics | head -20
|
|
```
|