Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 abiba-bot
|
5053a33e2c | ||
 abiba-bot
|
b386bd0c19 | ||
|
root
|
8a4dd08b05 | ||
|
root
|
88b6decb31 | ||
|
root
|
7bf9f78fc6 | ||
|
root
|
dd9e68329e | ||
|
root
|
cd9ec6a0df | ||
|
abiba-bot
|
1d20fbaa7f | ||
|
root
|
2f961d7e7a | ||
|
abiba-bot
|
4fe4f3621d | ||
|
root
|
86d2987ad8 | ||
|
root
|
efe9381283 | ||
|
abiba-bot
|
6a1c4967db |
@@ -1,19 +1,18 @@
|
||||
name: PR Pipeline — Authorize → Validate → Review → Merge
|
||||
# TRIGGER IS INTENTIONALLY UNFILTERED — DO NOT RE-ADD A `paths:` FILTER.
|
||||
#
|
||||
# This workflow previously carried `paths: ['**.prose.md', 'scripts/**.sh',
|
||||
# '**.yaml', '**.yml']` on both `push` and `pull_request`. Any PR whose diff
|
||||
# touched none of those patterns (for example a `deliverables/`-only PR, or a
|
||||
# `scripts/*.py` / `bin/*` change) therefore produced NO Gitea Actions run at
|
||||
# all: validation, lint, ai-review and the merge gate were silently skipped.
|
||||
# Validation must run for every pull request and every push to master, so the
|
||||
# trigger is deliberately unconditional.
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- '**.prose.md'
|
||||
- 'scripts/**.sh'
|
||||
- '**.yaml'
|
||||
- '**.yml'
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
paths:
|
||||
- '**.prose.md'
|
||||
- 'scripts/**.sh'
|
||||
- '**.yaml'
|
||||
- '**.yml'
|
||||
|
||||
jobs:
|
||||
auth:
|
||||
|
||||
@@ -207,9 +207,11 @@ The agent picks up the new key via `infisical run --` at gateway startup.
|
||||
|
||||
**Keys are permanent and use bare agent name aliases.**
|
||||
|
||||
- **Duration**: `null` — keys never expire. NOT enforced today: CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key generated with no explicit models comes back with an empty models list. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.)
|
||||
- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug.
|
||||
- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end.
|
||||
- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job.
|
||||
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
|
||||
- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
|
||||
- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use.
|
||||
- **Max budget**: $100 per key (config default).
|
||||
|
||||
```yaml
|
||||
|
||||
@@ -17,7 +17,7 @@ description: >
|
||||
⚠️ This contract is target-state aspirational — but GPU export + alerting
|
||||
are now as-built (verified 2026-08-09).
|
||||
As-built GPU monitoring is via gpu-monitor contract (port 9100 poll).
|
||||
version: 1.0.0
|
||||
version: 1.0.1
|
||||
---
|
||||
|
||||
## Architecture
|
||||
@@ -120,132 +120,156 @@ the any-HTTP rule. On those — the authenticated Zulip POST and the router
|
||||
`/health` — an unexpected status (`401`/`403` from a bad or missing credential,
|
||||
`5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive".
|
||||
|
||||
**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):**
|
||||
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the
|
||||
service answered — report the code, never "down". A redirect is not a failure.
|
||||
Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe,
|
||||
and that is a statement about YOUR PROBE, not about the service.
|
||||
2. **A failed probe is never a service verdict.** Print
|
||||
`probe-failed: <target> <kind>` naming the exact URL/host/port and the failure
|
||||
kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only
|
||||
then report. Apply the same shape as scripts/disk-gc-scan.py.
|
||||
3. **Say which probe produced each number.** "Grafana: 000" is unusable;
|
||||
"Grafana http://192.168.68.116:3001/api/health -> connection timeout after 10s
|
||||
(retried at 25s: also timeout)" is actionable.
|
||||
|
||||
### check-health
|
||||
|
||||
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
|
||||
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real
|
||||
tool calls; never repeat a prior report unless a live probe fails.**
|
||||
|
||||
**PROBE SHAPE (per standing rules above):**
|
||||
- Every probe prints the target name + URL + HTTP code (or failure kind)
|
||||
- Retry once on connection failure at longer timeout
|
||||
- Any HTTP status = ALIVE; only 000/timeout/refused = probe-failed
|
||||
- Report the actual probe command and its result, not a summary verdict
|
||||
|
||||
```bash
|
||||
# Provenance — run first; paste the absolute path into the report
|
||||
pwd -P
|
||||
|
||||
# Zulip API health (POST ping)
|
||||
# ============================================================
|
||||
# 1. ZULIP API HEALTH (POST ping) — bare-200 probe
|
||||
# ============================================================
|
||||
# NOTE: /etc/litellm-monitor.env exists only on CT 116, retrieve keys from CT 116 via:
|
||||
zulip_key=$(ssh root@192.168.68.116 "grep ZULIP_BOT_KEY /etc/litellm-monitor.env | cut -d= -f2")
|
||||
if [ -z "$zulip_key" ]; then
|
||||
echo "credential-missing: ZULIP_BOT_KEY not found in /etc/litellm-monitor.env"
|
||||
else
|
||||
ZULIP_USER="abiba-bot@chat.sysloggh.net"
|
||||
curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${zulip_key}"
|
||||
# Expected: 200 (HTTP 000 = unreachable/cache)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${zulip_key}")
|
||||
echo "Zulip API https://chat.sysloggh.net/api/v1/messages -> $code"
|
||||
# Expected: 200 (bare-200 probe; any other status is an ALERT)
|
||||
fi
|
||||
|
||||
# PM2 process health
|
||||
# ============================================================
|
||||
# 2. PM2 PROCESS HEALTH
|
||||
# ============================================================
|
||||
pm2 jlist
|
||||
# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
|
||||
# Expected: 4/4 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner)
|
||||
# spoton-service removed 2026-09-14 (not in live set)
|
||||
|
||||
# GPU exporters (may be down per DEPLOYMENT STATUS)
|
||||
curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
|
||||
curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
|
||||
curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
|
||||
# ============================================================
|
||||
# 3. GPU EXPORTERS — any-HTTP probe (metrics endpoint)
|
||||
# ============================================================
|
||||
# Probe /metrics (the Prometheus scrape target), not bare /
|
||||
for host in 192.168.68.8 192.168.68.110 192.168.68.15; do
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 "http://$host:9400/metrics")
|
||||
if [ "$code" == "000" ]; then
|
||||
# Retry with longer timeout
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 "http://$host:9400/metrics")
|
||||
echo "GPU exporter http://$host:9400/metrics -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "GPU exporter http://$host:9400/metrics -> $code"
|
||||
fi
|
||||
done
|
||||
# Expected: 200 on all 3 hosts (RTX 3090, RTX 5070, Strix Halo)
|
||||
|
||||
# Router health (via nginx on port 80)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
|
||||
# Expected: 200 (Router is up and responding)
|
||||
# ============================================================
|
||||
# 4. ROUTER HEALTH (via nginx on port 80) — bare-200 probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116/health)
|
||||
echo "Router http://192.168.68.116/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Router http://192.168.68.116/health -> $code"
|
||||
fi
|
||||
# Expected: 200 (bare-200 probe; any other status is an ALERT)
|
||||
|
||||
# LiteLLM health (via nginx on port 80)
|
||||
curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
|
||||
# Expected: 301 → /litellm/health/liveliness (200 after redirect) — any HTTP status = alive
|
||||
# ============================================================
|
||||
# 5. LITELLM HEALTH (via nginx on port 80) — any-HTTP probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116/litellm/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116/litellm/health)
|
||||
echo "LiteLLM http://192.168.68.116/litellm/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "LiteLLM http://192.168.68.116/litellm/health -> $code"
|
||||
fi
|
||||
# Expected: 301 → /litellm/health/liveliness (any HTTP status = ALIVE)
|
||||
|
||||
# PVE API liveness — probe the REAL PVE nodes on :8006, never the monitoring
|
||||
# host CT 116. CT 116 runs no pveproxy, so probing it on :8006 returns 000 —
|
||||
# that was the stale-vantage bug this replaces (CT 116 is the monitoring host,
|
||||
# not a cluster node). Unauthenticated GET answers 401 while the API is ALIVE
|
||||
# by design. Alive = ANY HTTP status (401 is the EXPECTED healthy response);
|
||||
# DOWN = connection refused (000) or timeout only.
|
||||
# ============================================================
|
||||
# 6. PVE API LIVENESS — any-HTTP probe (auth-gated)
|
||||
# ============================================================
|
||||
# Probe the REAL PVE nodes on :8006, never the monitoring host CT 116.
|
||||
for node in 192.168.68.9 192.168.68.5 192.168.68.15 192.168.68.6 192.168.68.12; do
|
||||
printf '%s:8006 -> %s\n' "$node" \
|
||||
"$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 "https://$node:8006/api2/json/version")"
|
||||
code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 10 "https://$node:8006/api2/json/version")
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 25 "https://$node:8006/api2/json/version")
|
||||
echo "PVE API https://$node:8006/api2/json/version -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "PVE API https://$node:8006/api2/json/version -> $code"
|
||||
fi
|
||||
done
|
||||
# Expected: 401 on every node (acerpve .9, ocupve .5, amdpve .15, storepve .6, minipve .12)
|
||||
# A node answering 000/timeout is DOWN — flag that node. 401 is NOT a fault.
|
||||
# 401 is the EXPECTED healthy response (auth-gated); 000/timeout = DOWN
|
||||
|
||||
# Prometheus targets
|
||||
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
|
||||
# Expected: All targets UP (may show some down if exporters not deployed)
|
||||
# ============================================================
|
||||
# 7. PROMETHEUS TARGETS — bare-200 probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:9090/api/v1/targets)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:9090/api/v1/targets)
|
||||
echo "Prometheus http://192.168.68.116:9090/api/v1/targets -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Prometheus http://192.168.68.116:9090/api/v1/targets -> $code"
|
||||
fi
|
||||
# Expected: 200 (bare-200 probe; any other status is an ALERT)
|
||||
|
||||
# Grafana health
|
||||
curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
|
||||
# Expected: {"status":"ok","version":"..."}
|
||||
# ============================================================
|
||||
# 8. GRAFANA HEALTH — any-HTTP probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:3001/api/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:3001/api/health)
|
||||
echo "Grafana http://192.168.68.116:3001/api/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Grafana http://192.168.68.116:3001/api/health -> $code"
|
||||
fi
|
||||
# Expected: 200 (any HTTP status = ALIVE; 000/timeout = DOWN)
|
||||
|
||||
# LiteLLM metrics (Prometheus endpoint)
|
||||
curl -s http://192.168.68.116:4000/metrics | head -20
|
||||
# Expected: Prometheus-formatted metrics output
|
||||
# ============================================================
|
||||
# 9. LITELLM METRICS (Prometheus endpoint) — any-HTTP probe
|
||||
# ============================================================
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:4000/metrics)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:4000/metrics)
|
||||
echo "LiteLLM metrics http://192.168.68.116:4000/metrics -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "LiteLLM metrics http://192.168.68.116:4000/metrics -> $code"
|
||||
fi
|
||||
# Expected: 200 (any HTTP status = ALIVE; 000/timeout = DOWN)
|
||||
```
|
||||
|
||||
**Report format**: Begin every report with the **absolute path the probe executed
|
||||
from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is
|
||||
distinguishable from a real fault at read time. Summarize actual results from
|
||||
each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE
|
||||
API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is
|
||||
DOWN; empty output is a warning. For probes whose expected result is a bare `200`
|
||||
(the authenticated Zulip POST, router `/health`), flag an alert on any unexpected
|
||||
status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200`
|
||||
expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect)
|
||||
is a stale expectation, not a fault.
|
||||
|
||||
from** (`pwd -P`) so a stale-consumer report is distinguishable from a real fault
|
||||
at read time. For each probe, print the target name, the full URL, and the HTTP
|
||||
code (or failure kind with retry details). Apply the standing probe rules: any
|
||||
HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not
|
||||
a failure.
|
||||
|
||||
### Phase 1: GPU Exporters
|
||||
|
||||
**NVIDIA (.8 and .110)**:
|
||||
1. Download `nvidia_gpu_exporter` binary
|
||||
2. Create systemd service `nvidia-gpu-exporter.service`
|
||||
3. Start and enable
|
||||
|
||||
**AMD (.15)**:
|
||||
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
|
||||
2. Parses `amdgpu_top --json -d 1000` output
|
||||
3. Exposes key metrics at `:9400/metrics` via Python http.server
|
||||
4. Create systemd service
|
||||
5. Start and enable
|
||||
|
||||
### Phase 2: Prometheus
|
||||
|
||||
1. Create `/opt/monitoring/` directory on CT 116
|
||||
2. Write `prometheus.yml` with scrape configs for all targets
|
||||
3. Add to docker-compose (or separate compose file)
|
||||
4. Start container
|
||||
|
||||
### Phase 3: Grafana
|
||||
|
||||
1. Create `/opt/monitoring/grafana/` directories
|
||||
2. Provision Prometheus datasource
|
||||
3. Provision GPU fleet dashboard JSON
|
||||
4. Provision LiteLLM dashboard JSON
|
||||
5. Add to docker-compose
|
||||
6. Start container
|
||||
|
||||
### Phase 4: Verification
|
||||
|
||||
1. Verify all 3 GPU exporters return 200 at :9400/metrics
|
||||
2. Verify Prometheus targets all UP at :9090/targets
|
||||
3. Verify Grafana accessible at :3001 with dashboards
|
||||
4. Verify LiteLLM metrics flowing to Prometheus
|
||||
5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard)
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# GPU exporters
|
||||
curl -s http://192.168.68.8:9400/metrics | grep nvidia
|
||||
curl -s http://192.168.68.110:9400/metrics | grep nvidia
|
||||
curl -s http://192.168.68.15:9400/metrics | grep amdgpu
|
||||
|
||||
# Prometheus
|
||||
curl -s http://192.168.68.116:9090/api/v1/targets
|
||||
|
||||
# Grafana
|
||||
curl -s http://192.168.68.116:3001/api/health
|
||||
|
||||
# LiteLLM metrics (already live)
|
||||
curl -s http://192.168.68.116:4000/metrics | head -20
|
||||
```
|
||||
|
||||
@@ -320,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require
|
||||
|
||||
## LiteLLM Master Key (use sparingly — agents should NOT use it directly)
|
||||
|
||||
- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY)
|
||||
- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**:
|
||||
```bash
|
||||
# PRIMARY (proven, runs on CT 116 with no extra tooling):
|
||||
docker exec harness-litellm printenv LITELLM_MASTER_KEY
|
||||
# Note: the same value is stored in /opt/inference-harness/.env on CT 116 (verified matching)
|
||||
# The master key is NOT in the Infisical vault (project=infrastructure env=production does not contain it)
|
||||
# Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl):
|
||||
curl -s -H "Authorization: Bearer <key>" http://127.0.0.1:4000/key/list | jq length
|
||||
```
|
||||
- Used for /key/generate, /key/delete, /key/list (GET), DB queries
|
||||
- **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key.
|
||||
It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`).
|
||||
|
||||
@@ -340,6 +340,36 @@ def check_gpu_ports():
|
||||
# CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents)
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
def _ssh_retry(host, cmd, user="root", timeout=15, retry_timeout=25, label=""):
|
||||
"""SSH with one retry at a longer timeout.
|
||||
|
||||
Returns (stdout_or_None, probe_failed_bool, fail_kind).
|
||||
When probe_failed is True, fail_kind is one of: timeout, ssh-failed.
|
||||
"""
|
||||
import subprocess as _sp
|
||||
def _attempt(tmo, conn_tmo):
|
||||
try:
|
||||
r = _sp.run(
|
||||
["ssh", "-o", "StrictHostKeyChecking=no", "-o", f"ConnectTimeout={conn_tmo}",
|
||||
f"{user}@{host}", cmd],
|
||||
capture_output=True, text=True, timeout=tmo)
|
||||
return r.stdout.strip() if r.returncode == 0 else None
|
||||
except _sp.TimeoutExpired:
|
||||
return "__timeout__"
|
||||
except:
|
||||
return None
|
||||
result = _attempt(timeout, 8)
|
||||
if result is None or result == "__timeout__":
|
||||
kind = "timeout" if result == "__timeout__" else "ssh-failed"
|
||||
prefix = f"{label} " if label else ""
|
||||
print(f" probe-failed: {prefix}ssh {user}@{host} — {kind} (retrying at {retry_timeout}s…)")
|
||||
result = _attempt(retry_timeout, 15)
|
||||
if result is None or result == "__timeout__":
|
||||
kind = "timeout" if result == "__timeout__" else "ssh-failed"
|
||||
return None, True, kind
|
||||
return result, False, None
|
||||
|
||||
|
||||
def check_agents():
|
||||
for name, agent in AGENTS.items():
|
||||
host = agent.get("host")
|
||||
@@ -355,42 +385,49 @@ def check_agents():
|
||||
is_dsh = agent.get("runtime") == "dsh"
|
||||
label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime"
|
||||
since = "since 2026-08-27" if is_dsh else "since the harness purge"
|
||||
live = ssh(host, "true", user=user)
|
||||
print(f" {'✅' if live is not None else '❌'} {name}: {label} — "
|
||||
f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
|
||||
if live is None:
|
||||
_fail(f"unreachable:{name}", name)
|
||||
live, probe_failed, fail_kind = _ssh_retry(host, "true", user=user)
|
||||
if probe_failed:
|
||||
print(f" ❌ {name}: {label} — probe-failed: ssh {user}@{host} {fail_kind} "
|
||||
f"(retried at 25s: also {fail_kind}) [CT {ct}]")
|
||||
_fail(f"probe-failed:{name}:{fail_kind}", name)
|
||||
else:
|
||||
print(f" ✅ {name}: {label} — no Hermes gateway {since} "
|
||||
f"(ssh {user}@{host} OK, CT {ct})")
|
||||
continue
|
||||
|
||||
if not host or not user:
|
||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||
continue
|
||||
|
||||
# Resolve the Hermes gateway PID once, before the report-only branch:
|
||||
# the summary line below renders `pid`, and it used to be bound only in
|
||||
# the report-only path — leaving it unbound on the abiba/koonimo path
|
||||
# raised UnboundLocalError and crashed the whole check. Agents without
|
||||
# a gateway get pid=?.
|
||||
pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid:
|
||||
pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
|
||||
if not pid:
|
||||
# Resolve the Hermes gateway PID with retry. The probe target is
|
||||
# explicit: ssh {user}@{host} pgrep -f hermes gateway.
|
||||
pid, probe_failed, fail_kind = _ssh_retry(
|
||||
host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid and not probe_failed:
|
||||
pid, probe_failed, fail_kind = _ssh_retry(
|
||||
host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
|
||||
if not pid and not probe_failed:
|
||||
pid = "?"
|
||||
|
||||
# ⛔ KOBY IS NEVER REPAIRED — diagnostic only
|
||||
if probe_failed:
|
||||
print(f" ❌ {name}: probe-failed: ssh {user}@{host} {fail_kind} "
|
||||
f"(retried at 25s: also {fail_kind}) [CT {ct}] — gateway status UNDETERMINED")
|
||||
_fail(f"probe-failed:{name}:{fail_kind}", name)
|
||||
continue
|
||||
|
||||
# ⛔ KOBY IS NEVER REPAIRED — diagnostic only (captain's 2026-08-17 ruling)
|
||||
if report_only:
|
||||
print(f" 🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
|
||||
# Still check gateway status for reporting purposes
|
||||
if pid == "?":
|
||||
print(f" ⚠️ {name}: GATEWAY NOT RUNNING (reported only)")
|
||||
print(f" 🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
|
||||
f"-> no process found (reported only, NOT counted) [CT {ct}]")
|
||||
_fail(f"gateway-down:{name}", name)
|
||||
continue
|
||||
else:
|
||||
print(f" ✅ {name}: gateway running (pid={pid}, report-only mode)")
|
||||
continue # Skip the rest of the check for Koby
|
||||
print(f" 🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway "
|
||||
f"-> pid={pid} (running, reported only, NOT repaired) [CT {ct}]")
|
||||
continue # Skip the rest of the check for Koby
|
||||
|
||||
# Gateway state file
|
||||
state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
|
||||
state, _, _ = _ssh_retry(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
|
||||
if state:
|
||||
try:
|
||||
st = json.loads(state)
|
||||
@@ -408,21 +445,22 @@ def check_agents():
|
||||
]
|
||||
streaming = "no"
|
||||
for p in adapter_paths:
|
||||
has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
|
||||
has_edit, _, _ = _ssh_retry(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
|
||||
if has_edit and has_edit != "0":
|
||||
streaming = "yes"
|
||||
break
|
||||
|
||||
# Recent errors
|
||||
recent_errors = ssh(host,
|
||||
recent_errors, _, _ = _ssh_retry(
|
||||
host,
|
||||
r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
|
||||
r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
|
||||
user=user)
|
||||
recent_errors = (recent_errors or "0").strip().split("\n")[-1]
|
||||
|
||||
print(f" {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
|
||||
f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
|
||||
f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
|
||||
f"{name}: probe: ssh {user}@{host} — gw={gw_state} zulip={zulip} "
|
||||
f"streaming={streaming} errors_10m={recent_errors.strip() or '0'} pid={pid} [CT {ct}]")
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════════
|
||||
|
||||
@@ -35,7 +35,12 @@ def run_command(cmd, timeout=15):
|
||||
return 1, "", str(e)
|
||||
|
||||
def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, follow_redirects=False):
|
||||
"""Probe HTTP endpoint and return status code"""
|
||||
"""Probe HTTP endpoint and return (status_code, failure_kind)
|
||||
|
||||
Returns:
|
||||
(code, None) if successful or HTTP response received
|
||||
(000, kind) if connection failed, where kind is 'timeout', 'refused', 'dns', etc.
|
||||
"""
|
||||
cmd = "curl -s -o /dev/null -w '%{http_code}' -m " + str(timeout)
|
||||
if method == "POST":
|
||||
cmd += " -X POST"
|
||||
@@ -47,15 +52,47 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
|
||||
cmd += " -L"
|
||||
cmd += " '" + url + "'"
|
||||
|
||||
rc, stdout, stderr = run_command(cmd, timeout)
|
||||
if rc != 0 and "TIMEOUT" not in stderr:
|
||||
return 000 # Connection failed
|
||||
try:
|
||||
rc, stdout, stderr = run_command(cmd, timeout)
|
||||
if rc != 0:
|
||||
# Determine failure kind from curl exit code
|
||||
# curl exit codes: 28=timeout, 7=refused, 6=dns, 35=ssl, 52=empty
|
||||
if rc == 28:
|
||||
return (000, "timeout after " + str(timeout) + "s")
|
||||
elif rc == 7:
|
||||
return (000, "connection refused")
|
||||
elif rc == 6:
|
||||
return (000, "dns failure")
|
||||
elif rc == 35:
|
||||
return (000, "ssl error")
|
||||
elif rc == 52:
|
||||
return (000, "empty response")
|
||||
else:
|
||||
return (000, "curl exit " + str(rc))
|
||||
return (int(stdout), None) if stdout.isdigit() else (000, "unparseable response")
|
||||
except subprocess.TimeoutExpired:
|
||||
return (000, "timeout after " + str(timeout) + "s")
|
||||
|
||||
def get_response_body(url, method="POST", bearer_token=None, data=None, timeout=30):
|
||||
"""Get response body for 401/403 credential faults (truncated to 200 chars)"""
|
||||
cmd = "curl -s -m " + str(timeout)
|
||||
if method == "POST":
|
||||
cmd += " -X POST"
|
||||
if bearer_token:
|
||||
cmd += " -H 'Authorization: Bearer " + bearer_token + "'"
|
||||
if data:
|
||||
cmd += " -H 'Content-Type: application/json' -d '" + data + "'"
|
||||
cmd += " '" + url + "'"
|
||||
|
||||
return int(stdout) if stdout.isdigit() else 000
|
||||
rc, stdout, stderr = run_command(cmd, timeout)
|
||||
# Return first 200 chars, single line
|
||||
body = stdout.replace('\n', ' ').replace('\t', ' ')[:200] if stdout else ""
|
||||
return body
|
||||
|
||||
|
||||
def check_liveliness():
|
||||
"""Step 1: Liveliness probe"""
|
||||
code = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness")
|
||||
code, _ = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness")
|
||||
return "Liveliness", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/health/liveliness)"
|
||||
|
||||
def check_containers():
|
||||
@@ -79,32 +116,61 @@ def check_model_probes():
|
||||
results = []
|
||||
|
||||
for model in ["gpu-dense", "gpu-vision", "strix-moe"]:
|
||||
# Single-host aliases: 30s timeout
|
||||
code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
|
||||
timeout=30)
|
||||
# Single-host aliases: 30s timeout each
|
||||
# gpu-dense (RTX 3090) may need long warmup/prefill - timeout is acceptable on cold-start
|
||||
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
|
||||
timeout=30)
|
||||
|
||||
results.append((model, code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
|
||||
if code == 000 and failure_kind:
|
||||
# Report probe failure with kind, do not assert a service verdict
|
||||
results.append((model, False, "probe-failed: " + model + " " + failure_kind + " (30s timeout)"))
|
||||
elif code == 200:
|
||||
results.append((model, True, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
|
||||
elif code in (401, 403):
|
||||
# Credential fault - capture body and key alias
|
||||
body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"' + model + '","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
|
||||
timeout=10)
|
||||
# Resolve key alias
|
||||
alias = "monitor-20260813" # Known from /etc/litellm-monitor.env on CT 116
|
||||
results.append((model, False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
|
||||
else:
|
||||
results.append((model, False, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
|
||||
|
||||
# Pool alias (syslog-auto): 60s timeout, retry once on 000
|
||||
code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
|
||||
timeout=60)
|
||||
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
|
||||
timeout=60)
|
||||
|
||||
if code == 000:
|
||||
if code == 000 and failure_kind:
|
||||
# Retry once with same timeout
|
||||
time.sleep(1)
|
||||
code = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
|
||||
timeout=60)
|
||||
|
||||
results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
|
||||
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
|
||||
timeout=60)
|
||||
if code == 000 and failure_kind:
|
||||
results.append(("syslog-auto", False, "probe-failed: syslog-auto " + failure_kind + " (60s timeout, retry)"))
|
||||
elif code in (401, 403):
|
||||
body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
|
||||
method="POST",
|
||||
bearer_token=monitor_key,
|
||||
data='{"model":"syslog-auto","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
|
||||
timeout=10)
|
||||
alias = "monitor-20260813"
|
||||
results.append(("syslog-auto", False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
|
||||
else:
|
||||
results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
|
||||
else:
|
||||
results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)"))
|
||||
|
||||
return results
|
||||
|
||||
@@ -146,18 +212,18 @@ def check_admin_key_list():
|
||||
|
||||
def check_github_status():
|
||||
"""Step 3: GitHub status - 301 redirect is acceptable for status page"""
|
||||
code = probe_http("https://status.github.com/api/status.json", timeout=15)
|
||||
code, _ = probe_http("https://status.github.com/api/status.json", timeout=15)
|
||||
# GitHub status API returns 301 redirect, which is expected behavior
|
||||
return "GitHub Status", code == 301, str(code)
|
||||
|
||||
def check_prometheus():
|
||||
"""Step 4: Prometheus health"""
|
||||
code = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy")
|
||||
code, _ = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy")
|
||||
return "Prometheus", code == 200, str(code) + " (target: " + BACKEND_HOST + ":9090/-/healthy)"
|
||||
|
||||
def check_grafana():
|
||||
"""Step 9: Grafana health"""
|
||||
code = probe_http("http://" + BACKEND_HOST + ":3001/api/health")
|
||||
code, _ = probe_http("http://" + BACKEND_HOST + ":3001/api/health")
|
||||
return "Grafana", code == 200, str(code) + " (target: " + BACKEND_HOST + ":3001/api/health)"
|
||||
|
||||
def check_docker_stats():
|
||||
|
||||
+32
-4
@@ -127,20 +127,48 @@ grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
|
||||
|
||||
## Execution
|
||||
|
||||
### Liveness rule (scoped)
|
||||
|
||||
Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
|
||||
|
||||
**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):**
|
||||
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
|
||||
2. **A failed probe is never a service verdict.** Print `probe-failed: <target> <kind>` naming the exact URL/host/port and the failure kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only then report.
|
||||
3. **Say which probe produced each number.** "API: 000" is unusable; "API https://chat.sysloggh.net/api/v1/server_settings -> connection timeout after 10s (retried at 25s: also timeout)" is actionable.
|
||||
|
||||
### Step 1: Zulip Server Liveness
|
||||
|
||||
```bash
|
||||
curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \
|
||||
-u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY'
|
||||
# Probe the Zulip API (authenticated, any HTTP status = ALIVE)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 https://chat.sysloggh.net/api/v1/server_settings -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY')
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 https://chat.sysloggh.net/api/v1/server_settings -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY')
|
||||
echo "Zulip API https://chat.sysloggh.net/api/v1/server_settings -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Zulip API https://chat.sysloggh.net/api/v1/server_settings -> $code"
|
||||
fi
|
||||
```
|
||||
|
||||
Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert.
|
||||
Expected: `200` (authenticated). Any HTTP status = ALIVE; only 000/timeout = probe-failed. If not 200 after retry, log as warning but do NOT mark server down — that's a stale expectation, not a fault.
|
||||
|
||||
### Step 2: Platform A — pi (Abiba, localhost)
|
||||
|
||||
**A1: Health Endpoint**
|
||||
|
||||
Fetch `http://localhost:9200/health` as JSON. Check:
|
||||
```bash
|
||||
# Probe the Abiba extension health endpoint (loopback, any HTTP status = ALIVE)
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9200/health)
|
||||
if [ "$code" == "000" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://127.0.0.1:9200/health)
|
||||
echo "Abiba extension http://127.0.0.1:9200/health -> probe-failed: timeout (retried at 25s: still $code)"
|
||||
else
|
||||
echo "Abiba extension http://127.0.0.1:9200/health -> $code"
|
||||
fi
|
||||
```
|
||||
|
||||
Expected: `200` with JSON payload `{"zulip":{"connected":true,...}}`. Any HTTP status = ALIVE; only 000/timeout = probe-failed. **NOTE: This probe MUST run on the Abiba host (CT 100) where 127.0.0.1:9200 is the extension. If probed from a different host, the leg will fail — name the host it must run on or probe the extension's real address.**
|
||||
|
||||
Check the JSON payload:
|
||||
|
||||
| Field | Healthy | Critical |
|
||||
|-------|---------|----------|
|
||||
|
||||
Reference in New Issue
Block a user