From d6ad016ac966ab22051124d9222daf34ef3da853 Mon Sep 17 00:00:00 2001 From: root Date: Sat, 22 Aug 2026 23:59:28 +0000 Subject: [PATCH] Add check-health section with live probes to infrastructure-monitoring contract Fixes the canned 'API UNREACHABLE (HTTP 000)' reporting. Adds an explicit check-health execution section (Zulip POST, pm2, GPU exporters, Prometheus, Grafana, LiteLLM probes) with a hard RUN LIVE, NEVER ECHO rule, mirroring the gpu-monitor contract pattern. Captain priority 2026-08-22. --- infrastructure-monitoring.prose.md | 33 ++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/infrastructure-monitoring.prose.md b/infrastructure-monitoring.prose.md index bb52406..62d8d4b 100644 --- a/infrastructure-monitoring.prose.md +++ b/infrastructure-monitoring.prose.md @@ -102,6 +102,39 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo) - Stack persists across reboots (systemd for exporters, Docker restart policy) ## Execution +### check-health + +**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.** + +```bash +# Zulip API health (POST ping) +curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u 'abiba-bot@chat.sysloggh.net:KEY' +# Expected: 200 (HTTP 000 = unreachable/cache) + +# PM2 process health +pm2 jlist +# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service) + +# GPU exporters (may be down per DEPLOYMENT STATUS) +curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL" +curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL" +curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL" + +# Prometheus targets +curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets' +# Expected: All targets UP (may show some down if exporters not deployed) + +# Grafana health +curl -s http://192.168.68.116:3001/api/health | jq '{status, version}' +# Expected: {"status":"ok","version":"..."} + +# LiteLLM metrics +curl -s http://192.168.68.116:4001/metrics | head -20 +# Expected: Prometheus-formatted metrics output +``` + +**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert. + ### Phase 1: GPU Exporters