From 4df2fa8329852954a65f57bc490dab1d38608bef Mon Sep 17 00:00:00 2001 From: root Date: Thu, 2 Jul 2026 02:50:59 +0000 Subject: [PATCH] prose: add proxmox-monitor contract (Grafana Pulse replacement) - New proxmox-monitor.prose.md: pve-exporter + node_exporter(5) + docker-stats-exporter - Replaces Pulse with file-provisioned Grafana dashboards (cluster/node/docker) - Documents cAdvisor/Docker-29 incompatibility + custom exporter workaround - PVE API token, access URL, config file map, operations --- proxmox-monitor.prose.md | 116 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 proxmox-monitor.prose.md diff --git a/proxmox-monitor.prose.md b/proxmox-monitor.prose.md new file mode 100644 index 0000000..742e2af --- /dev/null +++ b/proxmox-monitor.prose.md @@ -0,0 +1,116 @@ +--- +kind: responsibility +name: proxmox-monitor +description: > + Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack + on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three + exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single + instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter + (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve + the layerdb). Dashboards exposed at http://192.168.68.116/grafana/. +agent: abiba +--- + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ CT 116 (syslog-api) — monitoring compose /opt/monitoring/ │ +│ Prometheus :9090 → Grafana :3001 (nginx /grafana/) │ +└───────┬──────────────┬──────────────┬───────────────────────┘ + │ │ │ + ▼ ▼ ▼ + pve-exporter docker-stats (scrapes 5x node_exporter) + :9221 :9324 + │ │ + ▼ ▼ + PVE API Docker API + (amdpve .15, (unix socket, + cluster-wide) 10 containers) +``` + +## Exporters + +| Exporter | Host:Port | Scope | Notes | +|----------|-----------|-------|-------| +| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). | +| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. | +| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. | + +## PVE API Token + +- User: `monitoring@pve` (cluster-replicated) +- Role: `PVEAuditor` on `/` (read-only, whole cluster) +- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c` +- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`) + +## Grafana Dashboards (file-provisioned, folder "Syslog Fleet") + +| UID | Title | Panels | Source | +|-----|-------|--------|--------| +| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries | +| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) | +| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) | +| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) | + +- Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/` +- Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml` +- Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`) +- Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks + +## Access + +- **URL**: `http://192.168.68.116/grafana/` (LAN, via nginx sub-path) +- **Credentials**: admin / syslog-grafana-2026 +- nginx route: `/opt/inference-harness/nginx/nginx.conf` `location /grafana/` → `proxy_pass http://harness-grafana:3000;` (no trailing slash — preserves prefix for serve_from_sub_path) +- harness-nginx connected to `monitoring_default` network (`docker network connect monitoring_default harness-nginx`) +- grafana env: `GF_SERVER_ROOT_URL=http://192.168.68.116/grafana/`, `GF_SERVER_SERVE_FROM_SUB_PATH=true` + +## Configuration Files + +| File | Host | Purpose | +|------|------|---------| +| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) | +| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) | +| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) | +| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter | +| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator | +| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions | +| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning | +| `/opt/inference-harness/nginx/nginx.conf` | .116 | `/grafana/` reverse proxy route | +| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config | + +## Cluster "Tabiri" — 5 Nodes + +| Node | IP | Role | +|------|----|----| +| ocupve | 192.168.68.5 | PVE | +| storepve | 192.168.68.6 | PVE | +| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) | +| minipve | 192.168.68.12 | PVE | +| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) | + +## Operations + +### view-dashboards +Open `http://192.168.68.116/grafana/` → "Syslog Fleet" folder + +### add-dashboard +Edit `build-dashboards.py`, run it, `docker restart harness-grafana` + +### check-targets +`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'` + +### restart-exporter +`cd /opt/monitoring && docker compose restart pve-exporter docker-stats` + +### rotate-pve-token +`pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter` + +## Known Issues & Notes + +- **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter. +- **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health. +- **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage). +- **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13). +- **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.