diff --git a/infrastructure-monitoring.prose.md b/infrastructure-monitoring.prose.md index d5cd1c6..5db12f3 100644 --- a/infrastructure-monitoring.prose.md +++ b/infrastructure-monitoring.prose.md @@ -5,8 +5,12 @@ description: > Deploys Prometheus + GPU exporters + Grafana to monitor the entire inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics - via existing /metrics Prometheus endpoint. Replaces the deprecated - harness-dashboard with a production-grade monitoring stack. + via existing /metrics Prometheus endpoint. + + STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters) + already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15) + NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed. + This contract defines the target state; proxmox-monitor is the as-built. version: 1.0.0 --- @@ -129,7 +133,7 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo) 2. Verify Prometheus targets all UP at :9090/targets 3. Verify Grafana accessible at :3001 with dashboards 4. Verify LiteLLM metrics flowing to Prometheus -5. Update nginx to proxy `/monitoring/` → Grafana (optional) +5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard) ## Verification Commands