prose-contracts/proxmox-monitor.prose.md

kind, name, description, agent

kind	name	description	agent
responsibility	proxmox-monitor	Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).	abiba

Architecture

┌─────────────────────────────────────────────────────────────┐
│  CT 116 (syslog-api) — monitoring compose /opt/monitoring/   │
│  Prometheus :9090  →  Grafana :3001 (direct LAN, 0.0.0.0)    │
└───────┬──────────────┬──────────────┬───────────────────────┘
        │              │              │
        ▼              ▼              ▼
   pve-exporter    docker-stats    (scrapes 5x node_exporter)
   :9221           :9324
        │              │
        ▼              ▼
   PVE API         Docker API
   (amdpve .15,    (unix socket,
    cluster-wide)  10 containers)

Exporters

Exporter	Host:Port	Scope	Notes
prometheus-pve-exporter	.116:9221 (container)	All 5 nodes + 14 guests + 36 storage pools	Single instance, cluster-aware via amdpve API. Config /opt/monitoring/pve.yml (token monitoring@pve!prometheus, PVEAuditor role). Metric schema is label-based (id=node/amdpve, id=lxc/100).
node_exporter	.5/.6/.9/.12/.15:9100 (systemd)	Per-node CPU/mem/disk/net/temp	Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool.
docker-stats-exporter	.116:9324 (container)	10 Docker containers on .116	Custom (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script /opt/monitoring/docker-stats-exporter.py.

PVE API Token

• User: monitoring@pve (cluster-replicated)
• Role: PVEAuditor on / (read-only, whole cluster)
• Token: monitoring@pve!prometheus — stored in Infisical vault (PROXMOX_MONITOR_TOKEN)
• verify_ssl: false (proxmoxer uses verify_ssl, NOT verify_tls)

Grafana Dashboards (file-provisioned, folder "Syslog Fleet")

UID	Title	Panels	Source
proxmox-cluster	Proxmox Cluster Overview	16	cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries
proxmox-node	Proxmox Node Detail	13	per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node)
docker-containers	Docker Containers	10	per-container CPU/mem/network, restarts, memory limit ratio (variable: $container)
gpu-fleet	GPU Fleet	7	(existing, preserved in DB, not provisioned)

• Dashboards built by /opt/monitoring/grafana/dashboards/build-dashboards.py → JSON in .../dashboards/json/
• Provider config: /opt/monitoring/grafana/dashboards/dashboards.yml
• Datasource: Prometheus uid afqpgfay4g9hce (provisioned, /opt/monitoring/grafana/datasources/prometheus.yml)
• Edit dashboards in build-dashboards.py + re-run; allowUiUpdates: true for ad-hoc UI tweaks

Access

• URL: http://192.168.68.116:3001/ (LAN, direct — Grafana bound to 0.0.0.0:3001)
• Dashboards: http://192.168.68.116:3001/d/gpu-fleet, .../d/proxmox-cluster, .../d/proxmox-node, .../d/docker-containers
• Credentials: admin / password stored in Infisical vault (GRAFANA_ADMIN_PASSWORD)
• Grafana is NOT behind nginx — access port 3001 directly. The harness-nginx /grafana/ sub-path route was tried and reverted (broke the existing :3001 URL and gpu-fleet path). Do not re-add GF_SERVER_SERVE_FROM_SUB_PATH or an nginx /grafana/ route.
• grafana compose port mapping: "3001:3000" (0.0.0.0, not 127.0.0.1)

Configuration Files

File	Host	Purpose
/opt/monitoring/docker-compose.yml	.116	monitoring stack (prometheus, grafana, pve-exporter, docker-stats)
/opt/monitoring/prometheus.yml	.116	6 scrape jobs (3 GPU, pve, node x5, docker-stats)
/opt/monitoring/pve.yml	.116	PVE API credentials (chmod 644, contains token)
/opt/monitoring/docker-stats-exporter.py	.116	custom Docker metrics exporter
/opt/monitoring/grafana/dashboards/build-dashboards.py	.116	dashboard JSON generator
/opt/monitoring/grafana/dashboards/json/*.json	.116	provisioned dashboard definitions
/opt/monitoring/grafana/datasources/prometheus.yml	.116	datasource provisioning
/etc/default/prometheus-node-exporter	.5/.6/.9/.12/.15	node_exporter collector config

Cluster "Tabiri" — 5 Nodes

Node	IP	Role
ocupve	192.168.68.5	PVE
storepve	192.168.68.6	PVE
acerpve	192.168.68.9	PVE (hosts llm-gpu qemu/101)
minipve	192.168.68.12	PVE
amdpve	192.168.68.15	PVE + Strix Halo LLM (strix-moe)

PBS GC (Proxmox Backup Server)

Schedule

Cron 0 20 * * * on the storepve HOST (192.168.68.6) = 20:00 America/New_York local = 00:00 UTC.

NOTE: The closed PR #116 said "20:00 UTC" — this is WRONG by four hours. Do not copy it.

What Actually Runs

Host script /usr/local/bin/pbs-gc.sh runs pct exec 107 -- proxmox-backup-manager garbage-collection start storepve-datastore.

IMPORTANT: The tool proxmox-backup-manager exists only inside CT 107 (where the PBS server runs). The storepve host has only proxmox-backup-client. This is why the job had never worked before 2026-09-19 00:00 UTC.

Datastore Location

• Datastore: CT 107's /mnt/pbs-backup on the storepve ZFS dataset /tank/pbs-backup (pool tank, ~11T free)
• NOT /media/easystore2 (media library, 3.7T, 96% used — separate volume)

Liveness Check

The new proxmox-monitor.sh leg checks storepve-datastore GC health:

• Reads GC state from CT 107: pct exec 107 -- proxmox-backup-manager garbage-collection list --output-format json
• FAILS if last-run-endtime is older than 48 hours
• Reports age in hours and pending-bytes

All six verdict shapes (exactly as emitted by the script):

1. Healthy (fresh GC, 0 B pending): ✅ PBS GC: healthy (last run 1h ago, pending-bytes: 0 B)

2. Stale (GC ran >48h ago): 🔴 PBS GC: stale (last run 49h ago, pending-bytes: 1048576 B)

3. Probe-failed: empty read (000/timeout/unreadable): 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000)

4. Probe-failed: unparseable (non-empty but invalid JSON — the "command not found" case): 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON)

5. Never-run: datastore absent (valid JSON but storepve-datastore not in list): 🔴 PBS GC: never-run (storepve-datastore not found in GC list)

6. Never-run: no endtime (valid JSON with datastore present but last-run-endtime is null/0): 🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime)

Operations

view-dashboards

Open http://192.168.68.116:3001/ → "Syslog Fleet" folder

add-dashboard

Edit build-dashboards.py, run it, docker restart harness-grafana

check-targets

curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'

check-health

RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.

# Prometheus health (bound to 0.0.0.0:9090 on .116)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9090/-/healthy
# Expected: 200 (Prometheus is up and healthy)

# Grafana health (bound to 0.0.0.0:3001 on .116)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:3001/api/health
# Expected: 200 (Grafana is up and healthy)

# Docker Stats exporter (bound to 127.0.0.1:9324 on .116 — must probe from .116 localhost)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9324/metrics"
# Expected: 200 (docker-stats-exporter is up and responding)

# PVE exporter (bound to 127.0.0.1:9221 on .116 — must probe from .116 localhost)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9221/metrics"
# Expected: 200 (pve-exporter is up and responding)

Report format: Begin every report with the absolute path the probe executed from (pwd -P, or the script's absolute path) so a stale-consumer report is distinguishable from a real fault at read time. Summarize actual results from each probe. If any probe returns non-200, flag as alert.

Note: Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only) so they must be probed from .116 via SSH. Prometheus and Grafana are bound to 0.0.0.0 so they can be probed from the LAN.

restart-exporter

cd /opt/monitoring && docker compose restart pve-exporter docker-stats

rotate-pve-token

pveum user token add monitoring@pve prometheus on any PVE node → update /opt/monitoring/pve.yml → docker compose restart pve-exporter

Known Issues & Notes

• cAdvisor abandoned: v0.51 can't resolve Docker 29 containerd image-store layerdb (/var/lib/docker/image/ only has identity-cache.db). Returns 0 named containers. Replaced by custom docker-stats-exporter.
• Docker native metrics (/etc/docker/daemon.json metrics-addr: 127.0.0.1:9323, experimental:true) enabled but only gives engine_daemon_* (daemon-level), not per-container. Kept for daemon health.
• PVE exporter metric schema: NOT name-prefixed. pve_cpu_usage_ratio, pve_memory_usage_bytes, pve_disk_usage_bytes, pve_uptime_seconds are GUEST-level only (24 series, id=lxc/100 etc). Node-level host metrics come from node_exporter. Storage pool usage: pve_storage_info (info only, no usage bytes — use node_filesystem_* for actual disk usage).
• grafana piechart plugin removed from GF_INSTALL_PLUGINS (Angular, unsupported in Grafana 13).
• Single pve-exporter points at amdpve .15 — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.