fix(infra): Fix Docker Stats (9324) and PVE Exporter (9221) probe ports
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Previously probed wrong ports: - Docker Stats was at 9323 (dockerd metrics) but should be 9324 (harness-docker-stats, docker_container_* metrics) - PVE Exporter was at 9324 (harness-docker-stats) but should be 9221 (harness-pve-exporter, 5 pve_* metrics) Both exporters bind to 127.0.0.1 on CT 116 and must be probed via SSH. Updated infrastructure-monitoring.prose.md to document the correct ports. Added test assertions verifying the exact ports are probed. Branch: fix/infra-monitoring-probe-ports-20260919
This commit is contained in:
@@ -277,6 +277,27 @@ code (or failure kind with retry details). Apply the standing probe rules: any
|
||||
HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not
|
||||
a failure.
|
||||
|
||||
### Docker Stats and PVE Exporter Ports
|
||||
|
||||
These two exporters bind to 127.0.0.1 on CT 116 (localhost-only) and must be probed via SSH:
|
||||
|
||||
| Exporter | Port | Container | Metrics |
|
||||
|----------|------|-----------|---------|
|
||||
| **Docker Stats** | **9324** | harness-docker-stats | `docker_container_*` (per-container CPU/mem/network) |
|
||||
| **PVE Exporter** | **9221** | harness-pve-exporter | `pve_*` (5 cluster-level metrics) |
|
||||
|
||||
**IMPORTANT**: Do not confuse with port 9323, which is owned by dockerd and serves the Docker Engine's own metrics (`builder_builds_*`, `containerd_build_info_*`).
|
||||
|
||||
```bash
|
||||
# Docker Stats (harness-docker-stats)
|
||||
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9324/metrics"
|
||||
# Expected: 200 or 404 (any HTTP status = ALIVE)
|
||||
|
||||
# PVE Exporter (harness-pve-exporter)
|
||||
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9221/metrics"
|
||||
# Expected: 200 or 404 (any HTTP status = ALIVE)
|
||||
```
|
||||
|
||||
### Phase 1: GPU Exporters
|
||||
|
||||
**NVIDIA (.8 and .110)**:
|
||||
|
||||
@@ -60,8 +60,8 @@ GPU_PATH="/metrics"
|
||||
GPU_EXPECTED="200"
|
||||
|
||||
# Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116
|
||||
DOCKER_STATS_PORT="9323"
|
||||
PVE_EXPORTER_PORT="9324"
|
||||
DOCKER_STATS_PORT="9324" # harness-docker-stats (docker_container_* metrics)
|
||||
PVE_EXPORTER_PORT="9221" # harness-pve-exporter (5 pve_* metrics)
|
||||
CT116_SSH_HOST="192.168.68.116"
|
||||
# Both are bare-200: 404 = container not yet started
|
||||
DOCKER_STATS_EXPECTED="200|404"
|
||||
|
||||
@@ -121,12 +121,16 @@ assert "Port 9405 NOT in any curl call" \
|
||||
'! grep -q ":9405" "$CURL_LOG"'
|
||||
|
||||
# ── 5. Docker Stats / PVE Exporter: SSH-probed at correct ports ────────────
|
||||
assert "Docker Stats probed at port 9323 via SSH" \
|
||||
'grep -q "9323" "$SSH_LOG"'
|
||||
|
||||
assert "PVE Exporter probed at port 9324 via SSH" \
|
||||
assert "Docker Stats probed at port 9324 via SSH" \
|
||||
'grep -q "9324" "$SSH_LOG"'
|
||||
|
||||
assert "PVE Exporter probed at port 9221 via SSH" \
|
||||
'grep -q "9221" "$SSH_LOG"'
|
||||
|
||||
# ── 5a. Stale port 9323 (dockerd) NOT probed ──────────────────────────────
|
||||
assert "Port 9323 (dockerd) NOT in SSH log" \
|
||||
'! grep -q "9323" "$SSH_LOG"'
|
||||
|
||||
# ── 6. No stale ports in the script source (belt-and-suspenders) ──────────
|
||||
assert "Port 9325 (historical) NOT in script source" \
|
||||
'! grep -q "9325" "$SCRIPT"'
|
||||
|
||||
Reference in New Issue
Block a user