fix(infra): Fix Docker Stats (9324) and PVE Exporter (9221) probe ports
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s

Previously probed wrong ports:
- Docker Stats was at 9323 (dockerd metrics) but should be 9324
  (harness-docker-stats, docker_container_* metrics)
- PVE Exporter was at 9324 (harness-docker-stats) but should be 9221
  (harness-pve-exporter, 5 pve_* metrics)

Both exporters bind to 127.0.0.1 on CT 116 and must be probed via SSH.

Updated infrastructure-monitoring.prose.md to document the correct ports.
Added test assertions verifying the exact ports are probed.

Branch: fix/infra-monitoring-probe-ports-20260919
This commit is contained in:
root
2026-09-19 02:32:00 +00:00
parent ef7f90ef5a
commit a13457bcd6
3 changed files with 31 additions and 6 deletions
+21
View File
@@ -277,6 +277,27 @@ code (or failure kind with retry details). Apply the standing probe rules: any
HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not
a failure.
### Docker Stats and PVE Exporter Ports
These two exporters bind to 127.0.0.1 on CT 116 (localhost-only) and must be probed via SSH:
| Exporter | Port | Container | Metrics |
|----------|------|-----------|---------|
| **Docker Stats** | **9324** | harness-docker-stats | `docker_container_*` (per-container CPU/mem/network) |
| **PVE Exporter** | **9221** | harness-pve-exporter | `pve_*` (5 cluster-level metrics) |
**IMPORTANT**: Do not confuse with port 9323, which is owned by dockerd and serves the Docker Engine's own metrics (`builder_builds_*`, `containerd_build_info_*`).
```bash
# Docker Stats (harness-docker-stats)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9324/metrics"
# Expected: 200 or 404 (any HTTP status = ALIVE)
# PVE Exporter (harness-pve-exporter)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9221/metrics"
# Expected: 200 or 404 (any HTTP status = ALIVE)
```
### Phase 1: GPU Exporters
**NVIDIA (.8 and .110)**:
+2 -2
View File
@@ -60,8 +60,8 @@ GPU_PATH="/metrics"
GPU_EXPECTED="200"
# Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116
DOCKER_STATS_PORT="9323"
PVE_EXPORTER_PORT="9324"
DOCKER_STATS_PORT="9324" # harness-docker-stats (docker_container_* metrics)
PVE_EXPORTER_PORT="9221" # harness-pve-exporter (5 pve_* metrics)
CT116_SSH_HOST="192.168.68.116"
# Both are bare-200: 404 = container not yet started
DOCKER_STATS_EXPECTED="200|404"
+8 -4
View File
@@ -121,12 +121,16 @@ assert "Port 9405 NOT in any curl call" \
'! grep -q ":9405" "$CURL_LOG"'
# ── 5. Docker Stats / PVE Exporter: SSH-probed at correct ports ────────────
assert "Docker Stats probed at port 9323 via SSH" \
'grep -q "9323" "$SSH_LOG"'
assert "PVE Exporter probed at port 9324 via SSH" \
assert "Docker Stats probed at port 9324 via SSH" \
'grep -q "9324" "$SSH_LOG"'
assert "PVE Exporter probed at port 9221 via SSH" \
'grep -q "9221" "$SSH_LOG"'
# ── 5a. Stale port 9323 (dockerd) NOT probed ──────────────────────────────
assert "Port 9323 (dockerd) NOT in SSH log" \
'! grep -q "9323" "$SSH_LOG"'
# ── 6. No stale ports in the script source (belt-and-suspenders) ──────────
assert "Port 9325 (historical) NOT in script source" \
'! grep -q "9325" "$SCRIPT"'