Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 root
|
937afc0baf | ||
|
root
|
288d613876 | ||
 abiba-bot
|
1d20fbaa7f | ||
|
root
|
2f961d7e7a | ||
|
abiba-bot
|
4fe4f3621d | ||
|
root
|
86d2987ad8 | ||
|
root
|
efe9381283 | ||
|
abiba-bot
|
6a1c4967db | ||
|
root
|
5ed6f8179c | ||
|
root
|
b9322973ce | ||
|
root
|
fb1916707d | ||
|
root
|
d28df4f4de | ||
|
abiba-bot
|
cb26ee06d6 |
@@ -84,6 +84,32 @@ and escalation trail.
|
||||
- May also be invoked manually: `prose run disk-gc-threat-response`
|
||||
- Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates
|
||||
|
||||
## Scanner: scripts/disk-gc-scan.py
|
||||
|
||||
The fleet scan is executed by `scripts/disk-gc-scan.py`, which makes reachability
|
||||
verdicts deterministic:
|
||||
|
||||
1. **Retry on failure:** Each probe retries once before declaring a guest unreachable.
|
||||
2. **Named probe target:** Every rendered line names the guest, CT id, node, and
|
||||
access method actually used.
|
||||
3. **Failure kind printed:** An unreachable guest is reported with its failure kind
|
||||
(timeout, ssh-auth, no-route, conn-refused, ssh-exit-N) — never as a bare
|
||||
"unreachable" verdict.
|
||||
4. **Per-guest access method:** The correct access path is selected from a per-guest
|
||||
map so the wrong path cannot be picked by an executor improvising:
|
||||
- CT 105 (kagentz) = `ssh root@kagentz` (NOT `pct exec 105` — pct exec sees
|
||||
loop0/59G instead of the real 99G filesystem)
|
||||
- CT 109 (docker-vm) = `ssh root@192.168.68.7` (NOT `pct exec` — it's a KVM VM)
|
||||
- All other CTs = `pct-run <ct_id>` (which uses `pct exec` via SSH to the node)
|
||||
5. **Every figure traces to a named probe:** The scan output prints the exact command
|
||||
that produced each disk figure, so two different guests can never render
|
||||
identical numbers without the probe commands proving it.
|
||||
|
||||
Run: `python3 scripts/disk-gc-scan.py` (or `--json` for machine-readable output).
|
||||
|
||||
The scan feeds into `scripts/disk-gc-plan.py`, which applies the report-only gate
|
||||
from the `report_only_guests` YAML block above.
|
||||
|
||||
## Shape
|
||||
|
||||
- `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert
|
||||
|
||||
@@ -207,9 +207,11 @@ The agent picks up the new key via `infisical run --` at gateway startup.
|
||||
|
||||
**Keys are permanent and use bare agent name aliases.**
|
||||
|
||||
- **Duration**: `null` — keys never expire. NOT enforced today: CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key generated with no explicit models comes back with an empty models list. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.)
|
||||
- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug.
|
||||
- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end.
|
||||
- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job.
|
||||
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
|
||||
- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
|
||||
- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use.
|
||||
- **Max budget**: $100 per key (config default).
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -17,7 +17,7 @@ description: >
|
||||
⚠️ This contract is target-state aspirational — but GPU export + alerting
|
||||
are now as-built (verified 2026-08-09).
|
||||
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
|
||||
version: 1.0.0
|
||||
version: 1.0.1
|
||||
---
|
||||
|
||||
## Architecture
|
||||
@@ -120,132 +120,156 @@ the any-HTTP rule. On those — the authenticated Zulip POST and the router
|
||||
`/health` — an unexpected status (`401`/`403` from a bad or missing credential,
|
||||
`5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive".
|
||||
|
||||
**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):**
|
||||
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the
|
||||
service answered — report the code, never "down". A redirect is not a failure.
|
||||
Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe,
|
||||
and that is a statement about YOUR PROBE, not about the service.
|
||||
2. **A failed probe is never a service verdict.** Print
|
||||
`probe-failed: <target> <kind>` naming the exact URL/host/port and the failure
|
||||
kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only
|
||||
then report. Apply the same shape as scripts/disk-gc-scan.py.
|
||||
3. **Say which probe produced each number.** "Grafana: 000" is unusable;
|
||||
"Grafana http://192.168.68.116:3001/api/health -> connection timeout after 10s
|
||||
(retried at 25s: also timeout)" is actionable.
|
||||
|
||||
### check-health
|
||||
|
||||
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
|
||||
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real
|
||||
tool calls; never repeat a prior report unless a live probe fails.**
|
||||
|
||||
**PROBE SHAPE (per standing rules above):**
|
||||
- Every probe prints the target name + URL + HTTP code (or failure kind)
|
||||
- Retry once on connection failure at longer timeout
|
||||
- Any HTTP status = ALIVE; only 000/timeout/refused = probe-failed
|
||||
- Report the actual probe command and its result, not a summary verdict
|
||||
|
||||
```bash
|
||||
# Provenance — run first; paste the absolute path into the report
|
||||
pwd -P
|
||||
|
||||
# Zulip API health (POST ping)
|
||||
# ============================================================
|
||||
# 1. ZULIP API HEALTH (POST ping) — bare-200 probe
|
||||
# ============================================================
|
||||
# NOTE: /etc/litellm-monitor.env exists only on CT 116, retrieve keys from CT 116 via:
|
||||
zulip_key=$(ssh root@192.168.68.116 "grep ZULIP_BOT_KEY /etc/litellm-monitor.env | cut -d= -f2")
|
||||
if [ -z "$zulip_key" ]; then
|
||||
echo "credential-missing: ZULIP_BOT_KEY not found in /etc/litellm-monitor.env"
|
||||
else
|
||||
ZULIP_USER="abiba-bot@chat.sysloggh.net"
|
||||
curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${zulip_key}"
|
||||
# Expected: 200 (HTTP 000 = unreachable/cache)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${zulip_key}")
|
||||
echo "Zulip API https://chat.sysloggh.net/api/v1/messages -> $code"
|
||||
# Expected: 200 (bare-200 probe; any other status is an ALERT)
|
||||
fi
|
||||
|
||||
# PM2 process health
|
||||
# ============================================================
|
||||
# 2. PM2 PROCESS HEALTH
|
||||
# ============================================================
|
||||
pm2 jlist
|
||||
# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
|
||||
# Expected: 4/4 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner)
|
||||
# spoton-service removed 2026-09-14 (not in live set)
|
||||
|
||||
# GPU exporters (may be down per DEPLOYMENT STATUS)
|
||||
curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
|
||||
curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
|
||||
curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
|
||||
# ============================================================
|
||||
# 3. GPU EXPORTERS — any-HTTP probe (metrics endpoint)
|
||||
# ============================================================
|
||||
# Probe /metrics (the Prometheus scrape target), not bare /
|
||||
for host in 192.168.68.8 192.168.68.110 192.168.68.15; do
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 "http://$host:9400/metrics")
|
||||
if [ "$code" == "000" ]; then
|
||||
# Retry with longer timeout
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 "http://$host:9400/metrics")
|
||||
echo "GPU exporter http://$host:9400/metrics -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "GPU exporter http://$host:9400/metrics -> $code"
|
||||
fi
|
||||
done
|
||||
# Expected: 200 on all 3 hosts (RTX 3090, RTX 5070, Strix Halo)
|
||||
|
||||
# Router health (via nginx on port 80)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
|
||||
# Expected: 200 (Router is up and responding)
|
||||
# ============================================================
|
||||
# 4. ROUTER HEALTH (via nginx on port 80) — bare-200 probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116/health)
|
||||
echo "Router http://192.168.68.116/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Router http://192.168.68.116/health -> $code"
|
||||
fi
|
||||
# Expected: 200 (bare-200 probe; any other status is an ALERT)
|
||||
|
||||
# LiteLLM health (via nginx on port 80)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
|
||||
# Expected: 301 → /litellm/health/liveliness (200 after redirect) — any HTTP status = alive
|
||||
# ============================================================
|
||||
# 5. LITELLM HEALTH (via nginx on port 80) — any-HTTP probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116/litellm/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116/litellm/health)
|
||||
echo "LiteLLM http://192.168.68.116/litellm/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "LiteLLM http://192.168.68.116/litellm/health -> $code"
|
||||
fi
|
||||
# Expected: 301 → /litellm/health/liveliness (any HTTP status = ALIVE)
|
||||
|
||||
# PVE API liveness — probe the REAL PVE nodes on :8006, never the monitoring
|
||||
# host CT 116. CT 116 runs no pveproxy, so probing it on :8006 returns 000 —
|
||||
# that was the stale-vantage bug this replaces (CT 116 is the monitoring host,
|
||||
# not a cluster node). Unauthenticated GET answers 401 while the API is ALIVE
|
||||
# by design. Alive = ANY HTTP status (401 is the EXPECTED healthy response);
|
||||
# DOWN = connection refused (000) or timeout only.
|
||||
# ============================================================
|
||||
# 6. PVE API LIVENESS — any-HTTP probe (auth-gated)
|
||||
# ============================================================
|
||||
# Probe the REAL PVE nodes on :8006, never the monitoring host CT 116.
|
||||
for node in 192.168.68.9 192.168.68.5 192.168.68.15 192.168.68.6 192.168.68.12; do
|
||||
printf '%s:8006 -> %s\n' "$node" \
|
||||
"$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 "https://$node:8006/api2/json/version")"
|
||||
code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 10 "https://$node:8006/api2/json/version")
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 25 "https://$node:8006/api2/json/version")
|
||||
echo "PVE API https://$node:8006/api2/json/version -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "PVE API https://$node:8006/api2/json/version -> $code"
|
||||
fi
|
||||
done
|
||||
# Expected: 401 on every node (acerpve .9, ocupve .5, amdpve .15, storepve .6, minipve .12)
|
||||
# A node answering 000/timeout is DOWN — flag that node. 401 is NOT a fault.
|
||||
# 401 is the EXPECTED healthy response (auth-gated); 000/timeout = DOWN
|
||||
|
||||
# Prometheus targets
|
||||
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
|
||||
# Expected: All targets UP (may show some down if exporters not deployed)
|
||||
# ============================================================
|
||||
# 7. PROMETHEUS TARGETS — bare-200 probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:9090/api/v1/targets)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:9090/api/v1/targets)
|
||||
echo "Prometheus http://192.168.68.116:9090/api/v1/targets -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Prometheus http://192.168.68.116:9090/api/v1/targets -> $code"
|
||||
fi
|
||||
# Expected: 200 (bare-200 probe; any other status is an ALERT)
|
||||
|
||||
# Grafana health
|
||||
curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
|
||||
# Expected: {"status":"ok","version":"..."}
|
||||
# ============================================================
|
||||
# 8. GRAFANA HEALTH — any-HTTP probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:3001/api/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:3001/api/health)
|
||||
echo "Grafana http://192.168.68.116:3001/api/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Grafana http://192.168.68.116:3001/api/health -> $code"
|
||||
fi
|
||||
# Expected: 200 (any HTTP status = ALIVE; 000/timeout = DOWN)
|
||||
|
||||
# LiteLLM metrics (Prometheus endpoint)
|
||||
curl -s http://192.168.68.116:4000/metrics | head -20
|
||||
# Expected: Prometheus-formatted metrics output
|
||||
# ============================================================
|
||||
# 9. LITELLM METRICS (Prometheus endpoint) — any-HTTP probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:4000/metrics)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:4000/metrics)
|
||||
echo "LiteLLM metrics http://192.168.68.116:4000/metrics -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "LiteLLM metrics http://192.168.68.116:4000/metrics -> $code"
|
||||
fi
|
||||
# Expected: 200 (any HTTP status = ALIVE; 000/timeout = DOWN)
|
||||
```
|
||||
|
||||
**Report format**: Begin every report with the **absolute path the probe executed
|
||||
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
|
||||
distinguishable from a real fault at read time. Summarize actual results from
|
||||
each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE
|
||||
API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is
|
||||
DOWN; empty output is a warning. For probes whose expected result is a bare `200`
|
||||
(the authenticated Zulip POST, router `/health`), flag an alert on any unexpected
|
||||
status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200`
|
||||
expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect)
|
||||
is a stale expectation, not a fault.
|
||||
|
||||
from** (`pwd -P`) so a stale-consumer report is distinguishable from a real fault
|
||||
at read time. For each probe, print the target name, the full URL, and the HTTP
|
||||
code (or failure kind with retry details). Apply the standing probe rules: any
|
||||
HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not
|
||||
a failure.
|
||||
|
||||
### Phase 1: GPU Exporters
|
||||
|
||||
**NVIDIA (.8 and .110)**:
|
||||
1. Download `nvidia_gpu_exporter` binary
|
||||
2. Create systemd service `nvidia-gpu-exporter.service`
|
||||
3. Start and enable
|
||||
|
||||
**AMD (.15)**:
|
||||
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
|
||||
2. Parses `amdgpu_top --json -d 1000` output
|
||||
3. Exposes key metrics at `:9400/metrics` via Python http.server
|
||||
4. Create systemd service
|
||||
5. Start and enable
|
||||
|
||||
### Phase 2: Prometheus
|
||||
|
||||
1. Create `/opt/monitoring/` directory on CT 116
|
||||
2. Write `prometheus.yml` with scrape configs for all targets
|
||||
3. Add to docker-compose (or separate compose file)
|
||||
4. Start container
|
||||
|
||||
### Phase 3: Grafana
|
||||
|
||||
1. Create `/opt/monitoring/grafana/` directories
|
||||
2. Provision Prometheus datasource
|
||||
3. Provision GPU fleet dashboard JSON
|
||||
4. Provision LiteLLM dashboard JSON
|
||||
5. Add to docker-compose
|
||||
6. Start container
|
||||
|
||||
### Phase 4: Verification
|
||||
|
||||
1. Verify all 3 GPU exporters return 200 at :9400/metrics
|
||||
2. Verify Prometheus targets all UP at :9090/targets
|
||||
3. Verify Grafana accessible at :3001 with dashboards
|
||||
4. Verify LiteLLM metrics flowing to Prometheus
|
||||
5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# GPU exporters
|
||||
curl -s http://192.168.68.8:9400/metrics | grep nvidia
|
||||
curl -s http://192.168.68.110:9400/metrics | grep nvidia
|
||||
curl -s http://192.168.68.15:9400/metrics | grep amdgpu
|
||||
|
||||
# Prometheus
|
||||
curl -s http://192.168.68.116:9090/api/v1/targets
|
||||
|
||||
# Grafana
|
||||
curl -s http://192.168.68.116:3001/api/health
|
||||
|
||||
# LiteLLM metrics (already live)
|
||||
curl -s http://192.168.68.116:4000/metrics | head -20
|
||||
```
|
||||
|
||||
@@ -320,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require
|
||||
|
||||
## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
|
||||
|
||||
- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
|
||||
- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**:
|
||||
```bash
|
||||
# Read at runtime from the container's environment:
|
||||
docker exec harness-litellm printenv LITELLM_MASTER_KEY
|
||||
# Or from Infisical vault (project=infrastructure env=prod) - NOTE: --plain is broken on CLI 0.43.110 (prints nothing):
|
||||
infisical secrets get LITELLM_MASTER_KEY --project=infrastructure --env=production | awk '$1=="LITELLM_MASTER_KEY"{print $NF}'
|
||||
# Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl):
|
||||
curl -s -H "Authorization: Bearer <key>" http://127.0.0.1:4000/key/list | jq length
|
||||
```
|
||||
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
|
||||
- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
|
||||
It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
|
||||
|
||||
@@ -19,7 +19,7 @@ description: >
|
||||
Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
|
||||
- Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
|
||||
firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
|
||||
- Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100).
|
||||
- Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
|
||||
---
|
||||
|
||||
## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
|
||||
|
||||
@@ -2,6 +2,16 @@
|
||||
kind: responsibility
|
||||
name: pm2-self-heal
|
||||
description: >
|
||||
Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
|
||||
spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
|
||||
errored. Logs every action to the knowledge graph and alerts the owner via
|
||||
Zulip DM on failures.
|
||||
CRITICAL: Never restart abiba-zulip — it runs this contract.
|
||||
AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
|
||||
gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2;
|
||||
gpu-watchdog decommissioned (function folded into gpu-monitor.service);
|
||||
gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the
|
||||
2026-07-04 'removed/decommissioned' note was stale and is removed).
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
@@ -340,6 +340,36 @@ def check_gpu_ports():
|
||||
# CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
def _ssh_retry(host, cmd, user="root", timeout=15, retry_timeout=25, label=""):
|
||||
"""SSH with one retry at a longer timeout.
|
||||
|
||||
Returns (stdout_or_None, probe_failed_bool, fail_kind).
|
||||
When probe_failed is True, fail_kind is one of: timeout, ssh-failed.
|
||||
"""
|
||||
import subprocess as _sp
|
||||
def _attempt(tmo, conn_tmo):
|
||||
try:
|
||||
r = _sp.run(
|
||||
["ssh", "-o", "StrictHostKeyChecking=no", "-o", f"ConnectTimeout={conn_tmo}",
|
||||
f"{user}@{host}", cmd],
|
||||
capture_output=True, text=True, timeout=tmo)
|
||||
return r.stdout.strip() if r.returncode == 0 else None
|
||||
except _sp.TimeoutExpired:
|
||||
return "__timeout__"
|
||||
except:
|
||||
return None
|
||||
result = _attempt(timeout, 8)
|
||||
if result is None or result == "__timeout__":
|
||||
kind = "timeout" if result == "__timeout__" else "ssh-failed"
|
||||
prefix = f"{label} " if label else ""
|
||||
print(f" probe-failed: {prefix}ssh {user}@{host} — {kind} (retrying at {retry_timeout}s…)")
|
||||
result = _attempt(retry_timeout, 15)
|
||||
if result is None or result == "__timeout__":
|
||||
kind = "timeout" if result == "__timeout__" else "ssh-failed"
|
||||
return None, True, kind
|
||||
return result, False, None
|
||||
|
||||
|
||||
def check_agents():
|
||||
for name, agent in AGENTS.items():
|
||||
host = agent.get("host")
|
||||
@@ -355,42 +385,49 @@ def check_agents():
|
||||
is_dsh = agent.get("runtime") == "dsh"
|
||||
label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
|
||||
since = "since 2026-08-27" if is_dsh else "since the harness purge"
|
||||
live = ssh(host, "true", user=user)
|
||||
print(f" {'✅' if live is not None else '❌'} {name}: {label} — "
|
||||
f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
|
||||
if live is None:
|
||||
_fail(f"unreachable:{name}", name)
|
||||
live, probe_failed, fail_kind = _ssh_retry(host, "true", user=user)
|
||||
if probe_failed:
|
||||
print(f" ❌ {name}: {label} — probe-failed: ssh {user}@{host} {fail_kind} "
|
||||
f"(retried at 25s: also {fail_kind}) [CT {ct}]")
|
||||
_fail(f"probe-failed:{name}:{fail_kind}", name)
|
||||
else:
|
||||
print(f" ✅ {name}: {label} — no Hermes gateway {since} "
|
||||
f"(ssh {user}@{host} OK, CT {ct})")
|
||||
continue
|
||||
|
||||
if not host or not user:
|
||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||
continue
|
||||
|
||||
# Resolve the Hermes gateway PID once, before the report-only branch:
|
||||
# the summary line below renders `pid`, and it used to be bound only in
|
||||
# the report-only path — leaving it unbound on the abiba/koonimo path
|
||||
# raised UnboundLocalError and crashed the whole check. Agents without
|
||||
# a gateway get pid=?.
|
||||
pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid:
|
||||
pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
|
||||
if not pid:
|
||||
# Resolve the Hermes gateway PID with retry. The probe target is
|
||||
# explicit: ssh {user}@{host} pgrep -f hermes gateway.
|
||||
pid, probe_failed, fail_kind = _ssh_retry(
|
||||
host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid and not probe_failed:
|
||||
pid, probe_failed, fail_kind = _ssh_retry(
|
||||
host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
|
||||
if not pid and not probe_failed:
|
||||
pid = "?"
|
||||
|
||||
# ⛔ KOBY IS NEVER REPAIRED — diagnostic only
|
||||
if probe_failed:
|
||||
print(f" ❌ {name}: probe-failed: ssh {user}@{host} {fail_kind} "
|
||||
f"(retried at 25s: also {fail_kind}) [CT {ct}] — gateway status UNDETERMINED")
|
||||
_fail(f"probe-failed:{name}:{fail_kind}", name)
|
||||
continue
|
||||
|
||||
# ⛔ KOBY IS NEVER REPAIRED — diagnostic only (captain's 2026-08-17 ruling)
|
||||
if report_only:
|
||||
print(f" 🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
|
||||
# Still check gateway status for reporting purposes
|
||||
if pid == "?":
|
||||
print(f" ⚠️ {name}: GATEWAY NOT RUNNING (reported only)")
|
||||
print(f" 🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
|
||||
f"-> no process found (reported only, NOT counted) [CT {ct}]")
|
||||
_fail(f"gateway-down:{name}", name)
|
||||
continue
|
||||
else:
|
||||
print(f" ✅ {name}: gateway running (pid={pid}, report-only mode)")
|
||||
continue # Skip the rest of the check for Koby
|
||||
print(f" 🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
|
||||
f"-> pid={pid} (running, reported only, NOT repaired) [CT {ct}]")
|
||||
continue # Skip the rest of the check for Koby
|
||||
|
||||
# Gateway state file
|
||||
state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
|
||||
state, _, _ = _ssh_retry(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
|
||||
if state:
|
||||
try:
|
||||
st = json.loads(state)
|
||||
@@ -408,21 +445,22 @@ def check_agents():
|
||||
]
|
||||
streaming = "no"
|
||||
for p in adapter_paths:
|
||||
has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
|
||||
has_edit, _, _ = _ssh_retry(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
|
||||
if has_edit and has_edit != "0":
|
||||
streaming = "yes"
|
||||
break
|
||||
|
||||
# Recent errors
|
||||
recent_errors = ssh(host,
|
||||
recent_errors, _, _ = _ssh_retry(
|
||||
host,
|
||||
r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
|
||||
r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
|
||||
user=user)
|
||||
recent_errors = (recent_errors or "0").strip().split("\n")[-1]
|
||||
|
||||
print(f" {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
|
||||
f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
|
||||
f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
|
||||
f"{name}: probe: ssh {user}@{host} — gw={gw_state} zulip={zulip} "
|
||||
f"streaming={streaming} errors_10m={recent_errors.strip() or '0'} pid={pid} [CT {ct}]")
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
@@ -0,0 +1,339 @@
|
||||
#!/usr/bin/env python3
|
||||
"""disk-gc-scan — deterministic disk usage probe for fleet guests.
|
||||
|
||||
This is the executable scanner side of `disk-gc-threat-response.prose.md`. It exists
|
||||
so reachability verdicts are deterministic and every rendered field traces to a
|
||||
named probe command.
|
||||
|
||||
DESIGN PRINCIPLES (per task disk-gc-probe-false-unreachable-20260913):
|
||||
1. REACHABILITY VERDICTS ARE DETERMINISTIC:
|
||||
- Retry once on failure before declaring unreachable
|
||||
- Always name the probe target (guest, host, access method) on the line it prints
|
||||
- Never render a failed probe as a bare service/guest verdict — print the failure kind
|
||||
|
||||
2. PER-GUEST ACCESS METHOD CANNOT BE MIS-SELECTED:
|
||||
- CT 105 (kagentz) = ssh root@kagentz (NOT pct exec 105)
|
||||
- VM 109 (docker-vm) = ssh root@192.168.68.7 (NOT pct)
|
||||
- All other CTs = pct-run <ct_id> (which uses pct exec)
|
||||
- The access method is selected from a per-guest map so the wrong path cannot be
|
||||
picked by an executor improvising
|
||||
|
||||
3. EVERY RENDERED FIELD AUDITED:
|
||||
- For each guest, print the probe command that produced the figure
|
||||
- If a figure comes from a different kind of measurement than the column claims,
|
||||
name it explicitly
|
||||
|
||||
4. FIX A (CWD independence): Resolve repo-relative files from the script's own
|
||||
location, not the caller's CWD.
|
||||
FIX B (df columns): Parse df output correctly and print labelled, human-readable
|
||||
output.
|
||||
|
||||
Usage:
|
||||
disk-gc-scan.py # scan all guests
|
||||
disk-gc-scan.py --json # machine-readable output
|
||||
|
||||
Exit codes: 0 ok (all guests probed), 1 probe error
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
# Resolve repo-relative files from the script's own location, not the caller's CWD
|
||||
SCRIPT_DIR = pathlib.Path(__file__).resolve().parent
|
||||
HELPER_PCT_RUN = SCRIPT_DIR / "pct-run.sh"
|
||||
|
||||
# Per-guest access method map. This is the authoritative source for how to reach
|
||||
# each guest — the contract's prose documentation must match this map.
|
||||
#
|
||||
# Access methods:
|
||||
# - "pct-run": use pct-run.sh <ct_id> (pct exec via SSH to node)
|
||||
# - "ssh-host": use ssh root@<hostname>
|
||||
# - "ssh-ip": use ssh root@<ip>
|
||||
|
||||
@dataclass
|
||||
class Guest:
|
||||
"""A guest to probe."""
|
||||
ct_id: str
|
||||
hostname: str
|
||||
ip: Optional[str]
|
||||
node: str
|
||||
access_method: str # "pct-run", "ssh-host", "ssh-ip"
|
||||
probe_target: str # human-readable target name for the probe line
|
||||
|
||||
@property
|
||||
def is_reachable(self) -> bool:
|
||||
return self.probe_result is not None and self.probe_result.exit_code == 0
|
||||
|
||||
@property
|
||||
def usage_pct(self) -> Optional[float]:
|
||||
return self.probe_result.usage_pct if self.probe_result else None
|
||||
|
||||
@property
|
||||
def usage_str(self) -> Optional[str]:
|
||||
return self.probe_result.usage_str if self.probe_result else None
|
||||
|
||||
probe_result: Optional["ProbeResult"] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProbeResult:
|
||||
"""Result of probing a guest."""
|
||||
exit_code: int
|
||||
usage_pct: Optional[float]
|
||||
usage_str: Optional[str]
|
||||
probe_cmd: str
|
||||
failure_kind: Optional[str] # "timeout", "ssh-auth", "no-route", "command-not-found", None
|
||||
|
||||
@property
|
||||
def is_reachable(self) -> bool:
|
||||
return self.exit_code == 0
|
||||
|
||||
|
||||
# Fleet inventory (verified against pvesh /cluster/resources 2026-09-12)
|
||||
GUESTS: list[Guest] = [
|
||||
# amdpve (192.168.68.15)
|
||||
Guest(ct_id="105", hostname="kagentz", ip="192.168.68.105", node="amdpve",
|
||||
access_method="ssh-host", probe_target="kagentz (CT 105, amdpve)"),
|
||||
Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="amdpve",
|
||||
access_method="pct-run", probe_target="tanko (CT 112, amdpve)"),
|
||||
Guest(ct_id="113", hostname="baggy", ip="192.168.68.113", node="amdpve",
|
||||
access_method="pct-run", probe_target="baggy (CT 113, amdpve)"),
|
||||
Guest(ct_id="115", hostname="scottdenya", ip="192.168.68.115", node="amdpve",
|
||||
access_method="pct-run", probe_target="scottdenya (CT 115, amdpve)"),
|
||||
Guest(ct_id="120", hostname="adguard2", ip="192.168.68.120", node="amdpve",
|
||||
access_method="pct-run", probe_target="adguard2 (CT 120, amdpve)"),
|
||||
# minipve (192.168.68.12)
|
||||
Guest(ct_id="100", hostname="abiba", ip="192.168.68.100", node="minipve",
|
||||
access_method="pct-run", probe_target="abiba (CT 100, minipve)"),
|
||||
Guest(ct_id="102", hostname="adguard", ip="192.168.68.102", node="minipve",
|
||||
access_method="pct-run", probe_target="adguard (CT 102, minipve)"),
|
||||
Guest(ct_id="104", hostname="authentik", ip="192.168.68.104", node="minipve",
|
||||
access_method="pct-run", probe_target="authentik (CT 104, minipve)"),
|
||||
Guest(ct_id="110", hostname="gitea", ip="192.168.68.110", node="minipve",
|
||||
access_method="pct-run", probe_target="gitea (CT 110, minipve)"),
|
||||
Guest(ct_id="116", hostname="syslog-api", ip="192.168.68.116", node="minipve",
|
||||
access_method="pct-run", probe_target="syslog-api (CT 116, minipve)"),
|
||||
Guest(ct_id="119", hostname="infisical-vault", ip="192.168.68.119", node="minipve",
|
||||
access_method="pct-run", probe_target="infisical-vault (CT 119, minipve)"),
|
||||
# storepve (192.168.68.6)
|
||||
Guest(ct_id="106", hostname="ra-h-os", ip="192.168.68.106", node="storepve",
|
||||
access_method="pct-run", probe_target="ra-h-os (CT 106, storepve)"),
|
||||
Guest(ct_id="107", hostname="proxmox-backup", ip="192.168.68.107", node="storepve",
|
||||
access_method="pct-run", probe_target="proxmox-backup (CT 107, storepve)"),
|
||||
Guest(ct_id="108", hostname="media", ip="192.168.68.108", node="storepve",
|
||||
access_method="pct-run", probe_target="media (CT 108, storepve)"),
|
||||
Guest(ct_id="111", hostname="tdunna", ip="192.168.68.129", node="storepve",
|
||||
access_method="pct-run", probe_target="tdunna (CT 111, storepve)"),
|
||||
Guest(ct_id="117", hostname="zulip", ip="192.168.68.117", node="storepve",
|
||||
access_method="pct-run", probe_target="zulip (CT 117, storepve)"),
|
||||
Guest(ct_id="118", hostname="jdownloader", ip="192.168.68.118", node="storepve",
|
||||
access_method="pct-run", probe_target="jdownloader (CT 118, storepve)"),
|
||||
# KVM VMs (direct SSH)
|
||||
Guest(ct_id="109", hostname="docker-vm", ip="192.168.68.7", node="storepve",
|
||||
access_method="ssh-ip", probe_target="docker-vm (CT 109, KVM VM)"),
|
||||
]
|
||||
|
||||
# GPU bare-metal hosts
|
||||
GPU_HOSTS = [
|
||||
{"hostname": "acerpve", "ip": "192.168.68.9", "gpu": "RTX 3090",
|
||||
"probe_target": "RTX 3090 (bare metal .9)"},
|
||||
{"hostname": "ocupve", "ip": "192.168.68.110", "gpu": "RTX 5070",
|
||||
"probe_target": "RTX 5070 (bare metal .110)"},
|
||||
{"hostname": "amdpve", "ip": "192.168.68.15", "gpu": "Strix Halo",
|
||||
"probe_target": "Strix Halo (bare metal .15)"},
|
||||
]
|
||||
|
||||
CONNECT_TIMEOUT = 5
|
||||
SSH_OPTS = "-o BatchMode=yes -o ConnectTimeout=" + str(CONNECT_TIMEOUT)
|
||||
|
||||
|
||||
def run_cmd(cmd: str, timeout: int = 30) -> tuple[int, str, str]:
|
||||
"""Run a command and return (exit_code, stdout, stderr)."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd, shell=True, capture_output=True, text=True, timeout=timeout
|
||||
)
|
||||
return result.returncode, result.stdout.strip(), result.stderr.strip()
|
||||
except subprocess.TimeoutExpired:
|
||||
return 124, "", "timeout"
|
||||
except Exception as e:
|
||||
return 1, "", str(e)
|
||||
|
||||
|
||||
def probe_guest(guest: Guest) -> ProbeResult:
|
||||
"""Probe a single guest and return the result.
|
||||
|
||||
Access method is selected from guest.access_method:
|
||||
- "pct-run": pct-run.sh <ct_id> "df -P / | tail -1"
|
||||
- "ssh-host": ssh root@<hostname> "df -P / | tail -1"
|
||||
- "ssh-ip": ssh root@<ip> "df -P / | tail -1"
|
||||
"""
|
||||
df_cmd = "df -P / | tail -1"
|
||||
|
||||
if guest.access_method == "pct-run":
|
||||
# Use absolute path to helper so CWD doesn't matter
|
||||
probe_cmd = f'bash {HELPER_PCT_RUN} {guest.ct_id} "{df_cmd}"'
|
||||
elif guest.access_method == "ssh-host":
|
||||
probe_cmd = f'ssh {SSH_OPTS} root@{guest.hostname} "{df_cmd}"'
|
||||
elif guest.access_method == "ssh-ip":
|
||||
probe_cmd = f'ssh {SSH_OPTS} root@{guest.ip} "{df_cmd}"'
|
||||
else:
|
||||
raise ValueError(f"unknown access_method: {guest.access_method}")
|
||||
|
||||
# Check helper exists and is readable BEFORE probing (for pct-run guests)
|
||||
# This prevents scanner errors from being rendered as guest verdicts
|
||||
if guest.access_method == "pct-run":
|
||||
if not HELPER_PCT_RUN.exists():
|
||||
print(f"SCANNER ERROR: helper not found: {HELPER_PCT_RUN}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
if not os.access(str(HELPER_PCT_RUN), os.R_OK):
|
||||
print(f"SCANNER ERROR: helper not readable: {HELPER_PCT_RUN}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Retry once on failure before declaring unreachable
|
||||
for attempt in range(2):
|
||||
exit_code, stdout, stderr = run_cmd(probe_cmd, timeout=15)
|
||||
|
||||
if exit_code == 0:
|
||||
# Parse df output: Filesystem 1024-blocks Used Available Capacity Mounted on
|
||||
# parts[0]=Filesystem, parts[1]=Total (1K blocks), parts[2]=Used, parts[3]=Available, parts[4]=Capacity
|
||||
parts = stdout.split()
|
||||
if len(parts) >= 5:
|
||||
capacity_str = parts[4] # e.g., "34%"
|
||||
usage_pct = float(capacity_str.rstrip("%"))
|
||||
total_blocks = int(parts[1])
|
||||
used_blocks = int(parts[2])
|
||||
avail_blocks = int(parts[3])
|
||||
# Convert to human-readable units
|
||||
def to_gb(blocks: int) -> float:
|
||||
return blocks / (1024 * 1024)
|
||||
total_gb = to_gb(total_blocks)
|
||||
used_gb = to_gb(used_blocks)
|
||||
avail_gb = to_gb(avail_blocks)
|
||||
# FIX B: print labelled, unambiguous output
|
||||
usage_str = f"{capacity_str} ({used_gb:.1f}G used of {total_gb:.1f}G total, {avail_gb:.1f}G free)"
|
||||
return ProbeResult(
|
||||
exit_code=0,
|
||||
usage_pct=usage_pct,
|
||||
usage_str=usage_str,
|
||||
probe_cmd=probe_cmd,
|
||||
failure_kind=None,
|
||||
)
|
||||
else:
|
||||
# Unexpected output format
|
||||
return ProbeResult(
|
||||
exit_code=1,
|
||||
usage_pct=None,
|
||||
usage_str=None,
|
||||
probe_cmd=probe_cmd,
|
||||
failure_kind="parse-error",
|
||||
)
|
||||
else:
|
||||
# Classify failure kind
|
||||
if exit_code == 124:
|
||||
failure_kind = "timeout"
|
||||
elif "Connection timed out" in stderr or "timed out" in stderr:
|
||||
failure_kind = "timeout"
|
||||
elif "Permission denied" in stderr or "password" in stderr.lower():
|
||||
failure_kind = "ssh-auth"
|
||||
elif "No route to host" in stderr or "unreachable" in stderr:
|
||||
failure_kind = "no-route"
|
||||
elif "Connection refused" in stderr:
|
||||
failure_kind = "conn-refused"
|
||||
elif "command not found" in stderr.lower() or "No such file" in stderr:
|
||||
failure_kind = "command-not-found"
|
||||
else:
|
||||
failure_kind = f"ssh-exit-{exit_code}"
|
||||
|
||||
# Retry once
|
||||
if attempt == 0:
|
||||
time.sleep(1)
|
||||
continue
|
||||
return ProbeResult(
|
||||
exit_code=exit_code,
|
||||
usage_pct=None,
|
||||
usage_str=None,
|
||||
probe_cmd=probe_cmd,
|
||||
failure_kind=failure_kind,
|
||||
)
|
||||
|
||||
# Should not reach here, but just in case
|
||||
return ProbeResult(
|
||||
exit_code=1,
|
||||
usage_pct=None,
|
||||
usage_str=None,
|
||||
probe_cmd=probe_cmd,
|
||||
failure_kind="unknown",
|
||||
)
|
||||
|
||||
|
||||
def scan_fleet() -> list[dict]:
|
||||
"""Scan all guests and return the results."""
|
||||
results = []
|
||||
for guest in GUESTS:
|
||||
probe_result = probe_guest(guest)
|
||||
guest.probe_result = probe_result
|
||||
|
||||
row = {
|
||||
"target": guest.probe_target,
|
||||
"ct_id": guest.ct_id,
|
||||
"hostname": guest.hostname,
|
||||
"node": guest.node,
|
||||
"access_method": guest.access_method,
|
||||
"reachable": probe_result.is_reachable,
|
||||
"usage_pct": probe_result.usage_pct,
|
||||
"usage_str": probe_result.usage_str,
|
||||
"probe_cmd": probe_result.probe_cmd,
|
||||
"failure_kind": probe_result.failure_kind,
|
||||
}
|
||||
results.append(row)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def render_results(results: list[dict]) -> str:
|
||||
"""Render scan results in human-readable format."""
|
||||
lines = []
|
||||
lines.append("=== Disk GC Scan ===")
|
||||
lines.append("")
|
||||
|
||||
for row in results:
|
||||
if row["reachable"]:
|
||||
lines.append(f" ✅ {row['target']}: {row['usage_str']}")
|
||||
lines.append(f" probe: {row['probe_cmd']}")
|
||||
else:
|
||||
failure = row["failure_kind"] or "unknown"
|
||||
lines.append(f" ❌ {row['target']}: UNREACHABLE ({failure})")
|
||||
lines.append(f" probe: {row['probe_cmd']}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
import argparse
|
||||
|
||||
ap = argparse.ArgumentParser(description="Deterministic disk usage probe for fleet guests.")
|
||||
ap.add_argument("--json", action="store_true", help="machine-readable output")
|
||||
args = ap.parse_args()
|
||||
|
||||
results = scan_fleet()
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(results, indent=2))
|
||||
else:
|
||||
print(render_results(results))
|
||||
|
||||
# Exit 0 if all guests probed (reachable or not), 1 if any probe error
|
||||
# (a probe error means the probe itself failed, not just that the guest was unreachable)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+32
-4
@@ -127,20 +127,48 @@ grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
|
||||
|
||||
## Execution
|
||||
|
||||
### Liveness rule (scoped)
|
||||
|
||||
Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
|
||||
|
||||
**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):**
|
||||
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
|
||||
2. **A failed probe is never a service verdict.** Print `probe-failed: <target> <kind>` naming the exact URL/host/port and the failure kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only then report.
|
||||
3. **Say which probe produced each number.** "API: 000" is unusable; "API https://chat.sysloggh.net/api/v1/server_settings -> connection timeout after 10s (retried at 25s: also timeout)" is actionable.
|
||||
|
||||
### Step 1: Zulip Server Liveness
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
|
||||
-u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
|
||||
# Probe the Zulip API (authenticated, any HTTP status = ALIVE)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 https://chat.sysloggh.net/api/v1/server_settings -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY')
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 https://chat.sysloggh.net/api/v1/server_settings -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY')
|
||||
echo "Zulip API https://chat.sysloggh.net/api/v1/server_settings -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Zulip API https://chat.sysloggh.net/api/v1/server_settings -> $code"
|
||||
fi
|
||||
```
|
||||
|
||||
Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.
|
||||
Expected: `200` (authenticated). Any HTTP status = ALIVE; only 000/timeout = probe-failed. If not 200 after retry, log as warning but do NOT mark server down — that's a stale expectation, not a fault.
|
||||
|
||||
### Step 2: Platform A — pi (Abiba, localhost)
|
||||
|
||||
**A1: Health Endpoint**
|
||||
|
||||
Fetch `http://localhost:9200/health` as JSON. Check:
|
||||
```bash
|
||||
# Probe the Abiba extension health endpoint (loopback, any HTTP status = ALIVE)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9200/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://127.0.0.1:9200/health)
|
||||
echo "Abiba extension http://127.0.0.1:9200/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Abiba extension http://127.0.0.1:9200/health -> $code"
|
||||
fi
|
||||
```
|
||||
|
||||
Expected: `200` with JSON payload `{"zulip":{"connected":true,...}}`. Any HTTP status = ALIVE; only 000/timeout = probe-failed. **NOTE: This probe MUST run on the Abiba host (CT 100) where 127.0.0.1:9200 is the extension. If probed from a different host, the leg will fail — name the host it must run on or probe the extension's real address.**
|
||||
|
||||
Check the JSON payload:
|
||||
|
||||
| Field | Healthy | Critical |
|
||||
|-------|---------|----------|
|
||||
|
||||
Reference in New Issue
Block a user