Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2eb901debf | ||
|
|
801de0a25c |
@@ -113,18 +113,18 @@ curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9090/-/healthy
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:3001/api/health
|
||||
# Expected: 200 (Grafana is up and healthy)
|
||||
|
||||
# Docker Stats exporter (bound to 0.0.0.0:9324 on .116)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9324/metrics
|
||||
# Docker Stats exporter (bound to 127.0.0.1:9324 on .116 — must probe from .116 localhost)
|
||||
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9324/metrics"
|
||||
# Expected: 200 (docker-stats-exporter is up and responding)
|
||||
|
||||
# PVE exporter (bound to 0.0.0.0:9221 on .116)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9221/metrics
|
||||
# PVE exporter (bound to 127.0.0.1:9221 on .116 — must probe from .116 localhost)
|
||||
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9221/metrics"
|
||||
# Expected: 200 (pve-exporter is up and responding)
|
||||
```
|
||||
|
||||
**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
|
||||
|
||||
**Note**: All exporters are bound to 0.0.0.0 (not 127.0.0.1) so they are reachable from the LAN.
|
||||
**Note**: Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only) so they must be probed from .116 via SSH. Prometheus and Grafana are bound to 0.0.0.0 so they can be probed from the LAN.
|
||||
|
||||
### restart-exporter
|
||||
`cd /opt/monitoring && docker compose restart pve-exporter docker-stats`
|
||||
|
||||
+12
-5
@@ -197,15 +197,22 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
|
||||
**B2: Agent Process**
|
||||
|
||||
```bash
|
||||
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
||||
# Probe from CT 115 (amdpve) to avoid SSH key issues on CT 117
|
||||
ssh root@192.168.68.15 "pct exec 112 'systemctl is-active dsh-web'"
|
||||
```
|
||||
|
||||
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling):
|
||||
Gateway should be active. **Dual-gateway detection**: check for duplicate processes.
|
||||
|
||||
| Agent | Restart command | Notes |
|
||||
|-------|-----------------|-------|
|
||||
**B3: Gateway Health**
|
||||
|
||||
Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
|
||||
```bash
|
||||
# Probe localhost on CT 112 for DSH web service
|
||||
ssh root@192.168.68.15 "pct exec 112 'curl -s -o /dev/null -w \"%{http_code}\" http://127.0.0.1:3080/'"
|
||||
```
|
||||
|
||||
Expected: any HTTP code (200, 302, 401, etc.) = gateway is alive. Connection refused = gateway down.
|
||||
|
||||
**Note**: DSH web service listens on 127.0.0.1:3080 (loopback-only) by design. Public URL returns 302 through proxy-auth chain, not bare 200.
|
||||
|
||||
**B3: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user