root
253e19f680
- DELETE infrastructure-monitoring.prose.md (redundant — proxmox-monitor already deployed this) - FIX infrastructure-control.prose.md: remove /grafana/ nginx route (reverted Jul 2, broke gpu-fleet) - FIX infrastructure-update.prose.md: wrong CT IDs (122→112 Tanko, 123→114 Mumuni, .19→storepve Zulip) - FIX infrastructure-update.prose.md: Strix Halo :8080→router health check (firewalled to .116 only) - FIX proxmox-monitor.prose.md: stale /grafana/ URLs→:3001 (post-revert cleanup) - ADD disk-gc-threat-response.prose.md: verified run (2026-07-04, reclaimed 35.67GB from kagentz) - ADD infrastructure-update.prose.md, pi-approval-architecture.prose.md, zulip-approval-fix.prose.md, zulip-self-heal.prose.md - UPDATE hermes-config, litellm-health/self-heal, pm2-self-heal, zulip-* contracts - ADD runs/20260704-005804-51f9a9 (disk-gc-threat-response run artifacts) All contracts verified against live PVE API, nginx config, and firewall state.
116 lines
6.9 KiB
Markdown
116 lines
6.9 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: proxmox-monitor
|
|
description: >
|
|
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
|
|
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
|
|
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
|
|
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
|
|
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
|
|
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
|
|
agent: abiba
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ CT 116 (syslog-api) — monitoring compose /opt/monitoring/ │
|
|
│ Prometheus :9090 → Grafana :3001 (direct LAN, 0.0.0.0) │
|
|
└───────┬──────────────┬──────────────┬───────────────────────┘
|
|
│ │ │
|
|
▼ ▼ ▼
|
|
pve-exporter docker-stats (scrapes 5x node_exporter)
|
|
:9221 :9324
|
|
│ │
|
|
▼ ▼
|
|
PVE API Docker API
|
|
(amdpve .15, (unix socket,
|
|
cluster-wide) 10 containers)
|
|
```
|
|
|
|
## Exporters
|
|
|
|
| Exporter | Host:Port | Scope | Notes |
|
|
|----------|-----------|-------|-------|
|
|
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
|
|
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
|
|
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
|
|
|
|
## PVE API Token
|
|
|
|
- User: `monitoring@pve` (cluster-replicated)
|
|
- Role: `PVEAuditor` on `/` (read-only, whole cluster)
|
|
- Token: `monitoring@pve!prometheus` = `2c74ceb6-f905-444a-94f9-1c4f7889b68c`
|
|
- `verify_ssl: false` (proxmoxer uses `verify_ssl`, NOT `verify_tls`)
|
|
|
|
## Grafana Dashboards (file-provisioned, folder "Syslog Fleet")
|
|
|
|
| UID | Title | Panels | Source |
|
|
|-----|-------|--------|--------|
|
|
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
|
|
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
|
|
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
|
|
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
|
|
|
|
- Dashboards built by `/opt/monitoring/grafana/dashboards/build-dashboards.py` → JSON in `.../dashboards/json/`
|
|
- Provider config: `/opt/monitoring/grafana/dashboards/dashboards.yml`
|
|
- Datasource: Prometheus uid `afqpgfay4g9hce` (provisioned, `/opt/monitoring/grafana/datasources/prometheus.yml`)
|
|
- Edit dashboards in build-dashboards.py + re-run; `allowUiUpdates: true` for ad-hoc UI tweaks
|
|
|
|
## Access
|
|
|
|
- **URL**: `http://192.168.68.116:3001/` (LAN, direct — Grafana bound to `0.0.0.0:3001`)
|
|
- **Dashboards**: `http://192.168.68.116:3001/d/gpu-fleet`, `.../d/proxmox-cluster`, `.../d/proxmox-node`, `.../d/docker-containers`
|
|
- **Credentials**: admin / syslog-grafana-2026
|
|
- Grafana is NOT behind nginx — access port 3001 directly. The `harness-nginx` `/grafana/` sub-path route was tried and reverted (broke the existing `:3001` URL and gpu-fleet path). Do not re-add `GF_SERVER_SERVE_FROM_SUB_PATH` or an nginx `/grafana/` route.
|
|
- grafana compose port mapping: `"3001:3000"` (0.0.0.0, not 127.0.0.1)
|
|
|
|
## Configuration Files
|
|
|
|
| File | Host | Purpose |
|
|
|------|------|---------|
|
|
| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
|
|
| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
|
|
| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
|
|
| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
|
|
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
|
|
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
|
|
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
|
|
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
|
|
|
|
## Cluster "Tabiri" — 5 Nodes
|
|
|
|
| Node | IP | Role |
|
|
|------|----|----|
|
|
| ocupve | 192.168.68.5 | PVE |
|
|
| storepve | 192.168.68.6 | PVE |
|
|
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
|
|
| minipve | 192.168.68.12 | PVE |
|
|
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
|
|
|
|
## Operations
|
|
|
|
### view-dashboards
|
|
Open `http://192.168.68.116:3001/` → "Syslog Fleet" folder
|
|
|
|
### add-dashboard
|
|
Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
|
|
|
|
### check-targets
|
|
`curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`
|
|
|
|
### restart-exporter
|
|
`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
|
|
|
|
### rotate-pve-token
|
|
`pveum user token add monitoring@pve prometheus` on any PVE node → update `/opt/monitoring/pve.yml` → `docker compose restart pve-exporter`
|
|
|
|
## Known Issues & Notes
|
|
|
|
- **cAdvisor abandoned**: v0.51 can't resolve Docker 29 containerd image-store layerdb (`/var/lib/docker/image/` only has `identity-cache.db`). Returns 0 named containers. Replaced by custom docker-stats-exporter.
|
|
- **Docker native metrics** (`/etc/docker/daemon.json` `metrics-addr: 127.0.0.1:9323`, experimental:true) enabled but only gives `engine_daemon_*` (daemon-level), not per-container. Kept for daemon health.
|
|
- **PVE exporter metric schema**: NOT name-prefixed. `pve_cpu_usage_ratio`, `pve_memory_usage_bytes`, `pve_disk_usage_bytes`, `pve_uptime_seconds` are GUEST-level only (24 series, `id=lxc/100` etc). Node-level host metrics come from node_exporter. Storage pool usage: `pve_storage_info` (info only, no usage bytes — use node_filesystem_* for actual disk usage).
|
|
- **grafana piechart plugin removed** from `GF_INSTALL_PLUGINS` (Angular, unsupported in Grafana 13).
|
|
- **Single pve-exporter points at amdpve .15** — if amdpve API is down, cluster metrics gap (other node_exporters still report host metrics). Acceptable; amdpve is primary.
|