root
4df2fa8329
- New proxmox-monitor.prose.md: pve-exporter + node_exporter(5) + docker-stats-exporter - Replaces Pulse with file-provisioned Grafana dashboards (cluster/node/docker) - Documents cAdvisor/Docker-29 incompatibility + custom exporter workaround - PVE API token, access URL, config file map, operations
117 lines
6.9 KiB
Markdown
117 lines
6.9 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: proxmox-monitor
|
|
description: >
|
|
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
|
|
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
|
|
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
|
|
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
|
|
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
|
|
the layerdb). Dashboards exposed at http://192.168.68.116/grafana/.
|
|
agent: abiba
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ CT 116 (syslog-api) — monitoring compose /opt/monitoring/ │
|
|
│ Prometheus :9090 → Grafana :3001 (nginx /grafana/) │
|
|
└───────┬──────────────┬──────────────┬───────────────────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
pve-exporter docker-stats (scrapes 5x node_exporter)
|
|
:9221 :9324
|
|
│ │
|
|
▼ ▼
|
|
PVE API Docker API
|
|
(amdpve .15, (unix socket,
|
|
cluster-wide) 10 containers)
|
|
```
|
|
|
|
## Exporters
|
|
|
|
| Exporter | Host:Port | Scope | Notes |
|
|
|----------|-----------|-------|-------|
|
|
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
|
|
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
|
|
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
|
|
|
|
## PVE API Token
|
|
|
|
- User: `monitoring@pve` (cluster-replicated)
|
|
- Role: `PVEAuditor` on `/` (read-only, whole cluster)
|
|
- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
|
|
- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
|
|
|
|
## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
|
|
|
|
| UID | Title | Panels | Source |
|
|
|-----|-------|--------|--------|
|
|
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
|
|
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
|
|
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
|
|
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
|
|
|
|
- Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/`
|
|
- Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml`
|
|
- Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`)
|
|
- Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks
|
|
|
|
## Access
|
|
|
|
- **URL**: `http://192.168.68.116/grafana/` (LAN, via nginx sub-path)
|
|
- **Credentials**: admin / syslog-grafana-2026
|
|
- nginx route: `/opt/inference-harness/nginx/nginx.conf` `location /grafana/` → `proxy_pass http://harness-grafana:3000;` (no trailing slash — preserves prefix for serve_from_sub_path)
|
|
- harness-nginx connected to `monitoring_default` network (`docker network connect monitoring_default harness-nginx`)
|
|
- grafana env: `GF_SERVER_ROOT_URL=http://192.168.68.116/grafana/`, `GF_SERVER_SERVE_FROM_SUB_PATH=true`
|
|
|
|
## Configuration Files
|
|
|
|
| File | Host | Purpose |
|
|
|------|------|---------|
|
|
| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
|
|
| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
|
|
| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
|
|
| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
|
|
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
|
|
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
|
|
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
|
|
| `/opt/inference-harness/nginx/nginx.conf` | .116 | `/grafana/` reverse proxy route |
|
|
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
|
|
|
|
## Cluster "Tabiri" — 5 Nodes
|
|
|
|
| Node | IP | Role |
|
|
|------|----|----|
|
|
| ocupve | 192.168.68.5 | PVE |
|
|
| storepve | 192.168.68.6 | PVE |
|
|
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
|
|
| minipve | 192.168.68.12 | PVE |
|
|
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
|
|
|
|
## Operations
|
|
|
|
### view-dashboards
|
|
Open `http://192.168.68.116/grafana/` → "Syslog Fleet" folder
|
|
|
|
### add-dashboard
|
|
Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
|
|
|
|
### check-targets
|
|
`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
|
|
|
|
### restart-exporter
|
|
`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
|
|
|
|
### rotate-pve-token
|
|
`pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter`
|
|
|
|
## Known Issues & Notes
|
|
|
|
- **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter.
|
|
- **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health.
|
|
- **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
|
|
- **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
|
|
- **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.
|