Commit Graph
7 Commits
Author SHA1 Message Date
root d07c4494b5 fix(infra): F1 - Fix Docker Stats (9324) and PVE Exporter (9221) port comments
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The leg comments were wrong:
- Line 207: Docker Stats showed :9323 (dockerd port) but should be :9324
- Line 215: PVE Exporter showed :9324 (docker-stats port) but should be :9221
These were the exact pairing this PR exists to correct.

Read back the changed lines to verify:
scripts/infra-monitoring.sh:207 shows Docker Stats (CT 116 :9324, 127.0.0.1 via SSH)
scripts/infra-monitoring.sh:215 shows PVE Exporter (CT 116 :9221, 127.0.0.1 via SSH)

Branch: fix/infra-monitoring-probe-ports-20260919
2026-09-19 02:52:33 +00:00
root a13457bcd6 fix(infra): Fix Docker Stats (9324) and PVE Exporter (9221) probe ports
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Previously probed wrong ports:
- Docker Stats was at 9323 (dockerd metrics) but should be 9324
  (harness-docker-stats, docker_container_* metrics)
- PVE Exporter was at 9324 (harness-docker-stats) but should be 9221
  (harness-pve-exporter, 5 pve_* metrics)

Both exporters bind to 127.0.0.1 on CT 116 and must be probed via SSH.

Updated infrastructure-monitoring.prose.md to document the correct ports.
Added test assertions verifying the exact ports are probed.

Branch: fix/infra-monitoring-probe-ports-20260919
2026-09-19 02:32:00 +00:00
root 933cfd223b fix(infra): PR #115 round 4 — fix TLS detection + remove duplicate probe
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
Round 3 had:
1. rc captured from wrong command (tr always exits 0)
2. Every probe issued TWICE (26 invocations instead of 13)
3. TLS branch unreachable

Round 4 fixes:
- Restructure probe_http to ONE invocation that captures both output
  and status: out=$(...); rc=$?
- Delete the duplicated block
- Fix retry classification: don't overwrite kind if already set (e.g., tls)
- Add test 7b: TLS error (000 + exit 60) → kind is tls
- Update header output shape to include (<kind>) suffix

Test: 22 passed / 0 failed (bash scripts/test_infra_monitoring.sh)
2026-09-18 18:50:39 +00:00
root 7e257ce512 fix(infra): PR #115 final round — complete F1/C1 + implement TLS detection
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Failing after 12m11s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
F1: Set LAST_KIND on unexpected-status path (was unset, causing empty
     placeholder in 7 failure lines).
F2: Implement TLS detection — capture curl exit code and map TLS
     error codes (35|51|58|59|60|77|83) to kind=tls. Previously
     TLS failures were misdiagnosed as timeout.
F3: Header comment now lists all producible kinds:
     timeout | refused | tls | unexpected:<code>.
F4: Add 2 test assertions: Grafana failure line exists + kind
     is non-empty (proves the gap that shipped in round 2).

Test: 20 passed / 0 failed (bash scripts/test_infra_monitoring.sh)
2026-09-18 18:34:39 +00:00
root 3c7f5d7d65 fix(infra+zulip): PR #115 round-2 findings F1-F4+F7
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
F1: Make kind classification real — append (<kind>) to every failure
     line, assign kind=tls on curl exit 35/60, fix :112 where
     kind=refused was set on successful retry. Update prose shape.
F3: Move credential placeholder skip from server probe to notify()
     only — server is always probed (200 without auth verified live).
F4: notify() logs ALERT SUPPRESSED when credential unusable so
     alerts from other legs are not silently dropped.
F7: Restore trailing newline in infra-monitoring.sh.

F5 (DO NOT CHANGE): Verified directly — ssh root@192.168.68.6
     'grep -n keep-daily /etc/pve/jobs.cfg' returns five
     prune-backups keep-daily=35 lines. Prose is CORRECT.

Test: 18 passed, 0 failed (bash scripts/test_infra_monitoring.sh)
2026-09-18 18:21:12 +00:00
root 385f7e0623 fix(infra-monitoring): resolve PR #115 review findings (A1-A3, B1-B2, C1-C2)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 10s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
A1: Test -k assertion now checks use_k:+-k syntax (actual bash pattern)
A2: PVE_NODES assertions now count expected nodes and verify exact array size
A3: Test now asserts liveness behavior (PVE_API_LIVENESS=1) not source text
B1: disk-gc GC schedule corrected: cron runs pbs-gc.sh (not proxmox-backup-manager),
    schedule is 20:00 LOCAL (00:00 UTC, not 20:00 UTC), host timezone America/New_York
B2: PROBE SHAPE now documents actual output shape including TLS flag notes
C1: TLS kind is now printed in PVE API failure output
C2: SSH retry logic clarified - retry is in probe_http function (not unreachable)
2026-09-18 05:54:06 +00:00
root 93f15709d1 fix(infra-monitoring): move probes to versioned script with port-drift test
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The 2026-09-17 false-verdict incident (third recurrence) showed that prose
policy is not a control: the agent probed :9325/:9405 (nonexistent ports),
CT 116 for PVE API (should be real PVE nodes), and rendered TLS failures as
connection-refused. This moves the canonical probe set into
scripts/infra-monitoring.sh (executed verbatim by the contract) and adds
scripts/test_infra_monitoring.sh which asserts every probed port matches the
documented value.

- scripts/infra-monitoring.sh: one script per contract pattern; all targets,
  ports, paths, and expected-status rules in code; -k for PVE self-signed
  certs; non-zero exit naming every failed target; no OK summary on failure
- scripts/test_infra_monitoring.sh: 20 assertions covering port drift,
  monitoring-host-as-PVE-node, and missing -k flag
- infrastructure-monitoring.prose.md: check-health section now references the
  script as executable owner; paste its raw output verbatim

Proof: all 13 legs pass (exit 0); deliberately broken Grafana port (9325)
produces 'probe-failed: 192.168.68.116:9325 (expected 200)' and exit 1.
2026-09-18 05:17:35 +00:00