prose: annotate infrastructure-monitoring with current deployment status
- Core stack (Prometheus + Grafana + pve/node/docker exporters) already deployed by proxmox-monitor contract (Jul 2) - GPU exporters (.8/.110/.15) NOT yet live — gpu-exporter crash-loops, sidecars never deployed - Strike nginx /monitoring/ proxy suggestion (sub-path pattern already tried and reverted for /grafana/ per proxmox-monitor) - Add cross-reference: this contract defines target state, proxmox-monitor is the as-built
This commit is contained in:
@@ -5,8 +5,12 @@ description: >
|
|||||||
Deploys Prometheus + GPU exporters + Grafana to monitor the entire
|
Deploys Prometheus + GPU exporters + Grafana to monitor the entire
|
||||||
inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
|
inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
|
||||||
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
|
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
|
||||||
via existing /metrics Prometheus endpoint. Replaces the deprecated
|
via existing /metrics Prometheus endpoint.
|
||||||
harness-dashboard with a production-grade monitoring stack.
|
|
||||||
|
STATUS: Core stack (Prometheus + Grafana + pve/node/docker exporters)
|
||||||
|
already deployed by proxmox-monitor contract. GPU exporters (.8/.110/.15)
|
||||||
|
NOT yet live — gpu-exporter crash-loops on .15, sidecars never deployed.
|
||||||
|
This contract defines the target state; proxmox-monitor is the as-built.
|
||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -129,7 +133,7 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
|
|||||||
2. Verify Prometheus targets all UP at :9090/targets
|
2. Verify Prometheus targets all UP at :9090/targets
|
||||||
3. Verify Grafana accessible at :3001 with dashboards
|
3. Verify Grafana accessible at :3001 with dashboards
|
||||||
4. Verify LiteLLM metrics flowing to Prometheus
|
4. Verify LiteLLM metrics flowing to Prometheus
|
||||||
5. Update nginx to proxy `/monitoring/` → Grafana (optional)
|
5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
|
||||||
|
|
||||||
## Verification Commands
|
## Verification Commands
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user