Compare commits

...
Author SHA1 Message Date
root 2eb901debf fix: correct zulip-health Tanko probe to use CT 115 (amdpve) for SSH access
Fixed SSH key gap issue by probing from CT 115 (amdpve) instead of CT 117 (storepve).
Changed from 'ps aux' to 'systemctl is-active dsh-web' + localhost curl probe.

Fixes false DEGRADED status from SSH authentication failure.
2026-09-08 09:39:40 +00:00
root 801de0a25c fix: correct proxmox-monitor probe endpoints to use localhost for exporters
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only), not 0.0.0.0.
They must be probed from .116 via SSH. Prometheus and Grafana remain 0.0.0.0 (LAN-reachable).

Fixes false alarm from remote probe showing connection-refused (by design for localhost binds).
2026-09-08 08:41:21 +00:00
2 changed files with 17 additions and 10 deletions
+5 -5
View File
@@ -113,18 +113,18 @@ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9090/-/healthy
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:3001/api/health
# Expected: 200 (Grafana is up and healthy)
# Docker Stats exporter (bound to 0.0.0.0:9324 on .116)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9324/metrics
# Docker Stats exporter (bound to 127.0.0.1:9324 on .116 — must probe from .116 localhost)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9324/metrics"
# Expected: 200 (docker-stats-exporter is up and responding)
# PVE exporter (bound to 0.0.0.0:9221 on .116)
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9221/metrics
# PVE exporter (bound to 127.0.0.1:9221 on .116 — must probe from .116 localhost)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9221/metrics"
# Expected: 200 (pve-exporter is up and responding)
```
**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
**Note**: All exporters are bound to 0.0.0.0 (not 127.0.0.1) so they are reachable from the LAN.
**Note**: Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only) so they must be probed from .116 via SSH. Prometheus and Grafana are bound to 0.0.0.0 so they can be probed from the LAN.
### restart-exporter
`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
+12 -5
View File
@@ -197,15 +197,22 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
**B2: Agent Process**
```bash
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
# Probe from CT 115 (amdpve) to avoid SSH key issues on CT 117
ssh root@192.168.68.15 "pct exec 112 'systemctl is-active dsh-web'"
```
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling):
Gateway should be active. **Dual-gateway detection**: check for duplicate processes.
| Agent | Restart command | Notes |
|-------|-----------------|-------|
**B3: Gateway Health**
Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
```bash
# Probe localhost on CT 112 for DSH web service
ssh root@192.168.68.15 "pct exec 112 'curl -s -o /dev/null -w \"%{http_code}\" http://127.0.0.1:3080/'"
```
Expected: any HTTP code (200, 302, 401, etc.) = gateway is alive. Connection refused = gateway down.
**Note**: DSH web service listens on 127.0.0.1:3080 (loopback-only) by design. Public URL returns 302 through proxy-auth chain, not bare 200.
**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)