root
2dfc3e1530
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
267 lines
9.8 KiB
Markdown
267 lines
9.8 KiB
Markdown
---
|
|
kind: function
|
|
name: infrastructure-monitoring
|
|
description: >
|
|
Deploys Prometheus + GPU exporters + Grafana to monitor the entire
|
|
inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
|
|
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
|
|
via existing /metrics Prometheus endpoint.
|
|
|
|
DEPLOYMENT STATUS (2026-08-09):
|
|
✅ Core stack deployed: Prometheus + Grafana + pve/node/docker exporters
|
|
(via proxmox-monitor contract). Grafana at :3001, all scrape targets active.
|
|
✅ GPU exporters DEPLOYED: all 3 GPU hosts (.8/.110/.15) run exporters on
|
|
:9400 (nvidia_gpu_exporter / amdgpu exporter) — verified 200 on 2026-08-09.
|
|
✅ LiteLLM /metrics scraping live (success_callback: prometheus; auth via
|
|
master key) + Alertmanager + Zulip bridge (alerts-infra) added 2026-08-09.
|
|
⚠️ This contract is target-state aspirational — but GPU export + alerting
|
|
are now as-built (verified 2026-08-09).
|
|
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
|
|
version: 1.0.0
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
|
|
nvidia-exporter nvidia-exporter amdgpu-exporter
|
|
:9400 :9400 :9400
|
|
│ │ │
|
|
└─────────────────────┼─────────────────────┘
|
|
▼
|
|
┌──────────────────────────┐
|
|
│ Prometheus │
|
|
│ CT 116 :9090 │
|
|
│ │
|
|
│ Scrape targets: │
|
|
│ • 192.168.68.8:9400 │
|
|
│ • 192.168.68.110:9400 │
|
|
│ • 192.168.68.15:9400 │
|
|
│ • litellm:4000/metrics │
|
|
└──────────┬───────────────┘
|
|
│
|
|
┌──────────▼───────────────┐
|
|
│ Grafana │
|
|
│ CT 116 :3001 │
|
|
│ │
|
|
│ Preloaded dashboards: │
|
|
│ • GPU Fleet Overview │
|
|
│ • LiteLLM Proxy Stats │
|
|
└──────────────────────────┘
|
|
```
|
|
|
|
## Components
|
|
|
|
### 1. NVIDIA GPU Exporter (hosts: .8, .110)
|
|
- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
|
|
- Listens on `:9400`, exposes `/metrics` in Prometheus format
|
|
- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
|
|
|
|
### 2. AMD GPU Exporter (host: .15)
|
|
- Custom exporter: Python script wrapping `amdgpu_top --json`
|
|
- Listens on `:9400`, exposes `/metrics` in Prometheus format
|
|
- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
|
|
- Runs as systemd service for persistence
|
|
|
|
### 3. Prometheus (CT 116)
|
|
- Container: `prom/prometheus:latest`
|
|
- Port: `9090` (internal Docker network)
|
|
- Scrape interval: 15s
|
|
- Config: `/opt/monitoring/prometheus.yml`
|
|
- Storage: Docker volume `prometheus-data`
|
|
|
|
### 4. Grafana (CT 116)
|
|
- Container: `grafana/grafana:latest`
|
|
- Port: `3001` (mapped to host)
|
|
- Data source: Prometheus at `http://prometheus:9090`
|
|
- Provisioned dashboards for GPU fleet + LiteLLM
|
|
- Accessible at `http://192.168.68.116:3001`
|
|
|
|
## Parameters
|
|
|
|
- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
|
|
- gpu_amd_hosts: ["192.168.68.15"]
|
|
- monitoring_host: "192.168.68.116"
|
|
- prometheus_port: 9090
|
|
- grafana_port: 3001
|
|
- gpu_exporter_port: 9400
|
|
|
|
## Requires
|
|
|
|
- SSH access to all GPU hosts for exporter deployment
|
|
- Docker on CT 116 for Prometheus + Grafana containers
|
|
- Python 3 on AMD host for custom exporter
|
|
- nvidia-smi on NVIDIA hosts
|
|
|
|
## Maintains
|
|
|
|
- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
|
|
- Prometheus scrapes all targets every 15s
|
|
- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
|
|
- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
|
|
- Stack persists across reboots (systemd for exporters, Docker restart policy)
|
|
|
|
## Execution
|
|
### check-health
|
|
|
|
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
|
|
|
|
```bash
|
|
# Zulip API health (POST ping)
|
|
curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u 'abiba-bot@chat.sysloggh.net:KEY'
|
|
# Expected: 200 (HTTP 000 = unreachable/cache)
|
|
|
|
# PM2 process health
|
|
pm2 jlist
|
|
# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
|
|
|
|
# GPU exporters (may be down per DEPLOYMENT STATUS)
|
|
curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
|
|
curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
|
|
curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
|
|
|
|
# Prometheus targets
|
|
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
|
|
# Expected: All targets UP (may show some down if exporters not deployed)
|
|
|
|
# Grafana health
|
|
curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
|
|
# Expected: {"status":"ok","version":"..."}
|
|
|
|
# LiteLLM metrics
|
|
curl -s http://192.168.68.116:4001/metrics | head -20
|
|
# Expected: Prometheus-formatted metrics output
|
|
```
|
|
|
|
**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
|
|
|
|
|
|
### Phase 1: GPU Exporters
|
|
|
|
**NVIDIA (.8 and .110)**:
|
|
1. Download `nvidia_gpu_exporter` binary
|
|
2. Create systemd service `nvidia-gpu-exporter.service`
|
|
3. Start and enable
|
|
|
|
**AMD (.15)**:
|
|
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
|
|
2. Parses `amdgpu_top --json -d 1000` output
|
|
3. Exposes key metrics at `:9400/metrics` via Python http.server
|
|
4. Create systemd service
|
|
5. Start and enable
|
|
|
|
### Phase 2: Prometheus
|
|
|
|
1. Create `/opt/monitoring/` directory on CT 116
|
|
2. Write `prometheus.yml` with scrape configs for all targets
|
|
3. Add to docker-compose (or separate compose file)
|
|
4. Start container
|
|
|
|
### Phase 3: Grafana
|
|
|
|
1. Create `/opt/monitoring/grafana/` directories
|
|
2. Provision Prometheus datasource
|
|
3. Provision GPU fleet dashboard JSON
|
|
4. Provision LiteLLM dashboard JSON
|
|
5. Add to docker-compose
|
|
6. Start container
|
|
|
|
### Phase 4: Verification
|
|
|
|
1. Verify all 3 GPU exporters return 200 at :9400/metrics
|
|
2. Verify Prometheus targets all UP at :9090/targets
|
|
3. Verify Grafana accessible at :3001 with dashboards
|
|
4. Verify LiteLLM metrics flowing to Prometheus
|
|
5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
|
|
|
|
## Execution
|
|
|
|
### check-health
|
|
|
|
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
|
|
|
|
```bash
|
|
# Zulip API health (POST ping)
|
|
curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u 'abiba-bot@chat.sysloggh.net:KEY'
|
|
# Expected: 200 (HTTP 000 = unreachable/cache)
|
|
|
|
# PM2 process health
|
|
pm2 jlist
|
|
# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
|
|
|
|
# GPU exporters (may be down per DEPLOYMENT STATUS)
|
|
curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
|
|
curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
|
|
curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
|
|
|
|
# Prometheus targets
|
|
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
|
|
# Expected: All targets UP (may show some down if exporters not deployed)
|
|
|
|
# Grafana health
|
|
curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
|
|
# Expected: {"status":"ok","version":"..."}
|
|
|
|
# LiteLLM metrics
|
|
curl -s http://192.168.68.116:4001/metrics | head -20
|
|
# Expected: Prometheus-formatted metrics output
|
|
```
|
|
|
|
**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
|
|
|
|
### Phase 1: GPU Exporters
|
|
|
|
**NVIDIA (.8 and .110)**:
|
|
1. Download `nvidia_gpu_exporter` binary
|
|
2. Create systemd service `nvidia-gpu-exporter.service`
|
|
3. Start and enable
|
|
|
|
**AMD (.15)**:
|
|
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
|
|
2. Parses `amdgpu_top --json -d 1000` output
|
|
3. Exposes key metrics at `:9400/metrics` via Python http.server
|
|
4. Create systemd service
|
|
5. Start and enable
|
|
|
|
### Phase 2: Prometheus
|
|
|
|
1. Create `/opt/monitoring/` directory on CT 116
|
|
2. Write `prometheus.yml` with scrape configs for all targets
|
|
3. Add to docker-compose (or separate compose file)
|
|
4. Start container
|
|
|
|
### Phase 3: Grafana
|
|
|
|
1. Create `/opt/monitoring/grafana/` directories
|
|
2. Provision Prometheus datasource
|
|
3. Provision GPU fleet dashboard JSON
|
|
4. Provision LiteLLM dashboard JSON
|
|
5. Add to docker-compose
|
|
6. Start container
|
|
|
|
### Phase 4: Verification
|
|
|
|
1. Verify all 3 GPU exporters return 200 at :9400/metrics
|
|
2. Verify Prometheus targets all UP at :9090/targets
|
|
3. Verify Grafana accessible at :3001 with dashboards
|
|
4. Verify LiteLLM metrics flowing to Prometheus
|
|
5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
|
|
|
|
## Verification Commands
|
|
|
|
```bash
|
|
# GPU exporters
|
|
curl -s http://192.168.68.8:9400/metrics | grep nvidia
|
|
curl -s http://192.168.68.110:9400/metrics | grep nvidia
|
|
curl -s http://192.168.68.15:9400/metrics | grep amdgpu
|
|
|
|
# Prometheus
|
|
curl -s http://192.168.68.116:9090/api/v1/targets
|
|
|
|
# Grafana
|
|
curl -s http://192.168.68.116:3001/api/health
|
|
|
|
# LiteLLM metrics (already live)
|
|
curl -s http://192.168.68.116:4001/metrics | head -20
|
|
```
|