Compare commits

...
Author SHA1 Message Date
root c72436b406 no-mistakes(document): docs: reconcile zulip-monitor status in infrastructure-control
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-09-19 22:40:25 +00:00
root a17379676d no-mistakes(document): docs: sync zulip-health contract with C1/C3 and version 2026-09-19 22:38:53 +00:00
root 63990b84f7 no-mistakes(review): Fix review findings: test regression, contract verdict, scope trim 2026-09-19 22:31:25 +00:00
root 7a5ddb46a9 chore: Add local monitoring tooling scripts 2026-09-19 22:27:19 +00:00
root 568fec2efa test(zulip-kagentz): Replace string-presence tests with behavioural sandbox tests
- C3 502 → INCIDENT
- C3 000 → INCIDENT
- C1 401 + C3 302 → 0 issues, all healthy
- C1 000 → INCIDENT

Each test asserts from the run's own log/verdict, not from file text.
Prose assertions kept as secondary.

Proven to bite: run against pre-fix script (origin/master) shows all 4
behavioural cases fail because C3 leg doesn't exist and Result line
doesn't say 'INCIDENT'.
2026-09-19 22:25:03 +00:00
root 641a52c6da fix(zulip-monitor): Add C3 public access path, make Result line non-optimistic
- (b) Changed Result line to say 'INCIDENT' when ISSUES > 0, '0 issues (all healthy)' when ISSUES = 0
- (c) Documented C1 (no credential needed), C2 (requires LITELLM_KEY) distinction
- (d) Added C3 public access path leg for https://kagentz.sysloggh.net/
- C3 treats 200/302/401 as alive, 502/000 as incident
- Added tests/test_zulip_kagentz_legs.py to verify all changes
2026-09-19 22:14:08 +00:00
abiba-bot 58033f39c5 Merge pull request 'fix: Rule 5 accept canonical internal and public host base_url' (#122) from fix/rule5-canonical-baseurl-20260919 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-09-19 11:57:35 +00:00
abiba-bot 6c59988e7f Merge pull request 'fix(litellm-health): three-state model verdicts - busy is not a failure' (#123) from fix/litellm-health-busy-vs-down-20260919 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-19 11:57:25 +00:00
root 9a2ee6faec fix(litellm): Remove duplicate 'host healthy' from busy line
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The busy line was rendering as:
  'busy (completion timed out after retry; host healthy host healthy (200))'

because host_detail already contains 'host healthy (200)' and the prefix
also said 'host healthy'. Fixed to:
  'busy (completion timed out after retry; host healthy (200))'

F1 cosmetic fix from PR #123 verify.
2026-09-19 11:53:32 +00:00
root e71ded3c8c fix: internal /v1 WARN not FAIL; align prose
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 3s
Rule 5 now treats internal http://192.168.68.116/v1 as non-canonical
but working (authenticated via nginx), producing a WARNING instead of a
FAILURE. The canonical internal path /litellm/v1 and the public host
https://litellm.sysloggh.net/v1 both PASS. Everything else FAILS.

Prose aligned: hermes-key-enforcement.prose.md now states the canonical
internal form, notes that internal /v1 still works but is flagged as
non-canonical (WARN not FAIL), and clarifies that the public host serves
/v1 ONLY (404 on /litellm/v1). Corrected the 'unauthenticated path'
wording at line 116, which was factually wrong.

Tests updated: BASE template uses canonical internal path; new test
cases prove canonical /litellm/v1 PASSES, wrong path FAILS, public host
PASSES, and internal /v1 WARNS (not FAILS). Fixed backwards comment in
test_old_rule5_check_would_fail_canonical.
2026-09-19 11:48:48 +00:00
root a12abbeb14 fix(litellm): Implement busy vs down with proper degraded state
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Three states:
- healthy: passed, exit 0 (unchanged)
- busy (completion timed out after retry AND host /health answered):
  ⚠️ DEGRADED line, does NOT fail the run, exit 0
- host unreachable or real fault: ❌, exit 1 (unchanged)

Summary now reports degraded count:
- All pass, no degraded: '✅ All checks passed'
- All pass, 1+ degraded: '✅ All checks passed (1 degraded: gpu-dense)'
- Some failed: '❌ Some checks failed' or '❌ Some checks failed (1 degraded: ...)'

Host health mapping verified:
- gpu-dense -> 192.168.68.8:8080/health
- gpu-vision -> 192.168.68.110:8080/health
- strix-moe -> 192.168.68.15:8080/health
2026-09-19 11:43:39 +00:00
root 0b9aebca37 fix(litellm): Fix timeout kind reporting + add busy/degraded detection
1. TIMEOUT KIND FIX: run_command returns (1, '', 'TIMEOUT') when its own
   timeout fires. probe_http now checks for this before falling through to
   'curl exit <rc>', so a 30s timeout reports 'timeout after 30s' not
   'curl exit 1'.

2. BUSY/DEGRADED DETECTION: After both model probes fail, check the
   model's host health endpoint (e.g. 192.168.68.8:8080/health for
   gpu-dense). If the host answers 200, report 'busy (completion timed
   out after retry; host healthy 200)' — do NOT fail the run on that
   alone. If the host does not answer, that's a real FAIL.

3. RETRY TIMEOUT INCREASED: Single-host retry timeout raised from 45s to
   90s. Worst-case prefill on a single-slot .8 host is ~76s (observed
   83K-token prompt at 1078 tok/s), so 90s covers it.

New line shapes:
- Busy: 'gpu-dense: busy (completion timed out after retry; host healthy 200)'
- Real failure: 'probe-failed: gpu-dense timeout after 30s then timeout after 90s (2 attempts)'
2026-09-19 11:42:36 +00:00
root fa458afa26 fix: Rule 5 accept canonical internal and public host base_url
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Rule 5 in audit-hermes-config.py had an inverted check: it expected
base_url=http://192.168.68.116/v1, but the contract hermes-key-enforcement.prose.md
names http://192.168.68.116/litellm/v1 as CORRECT/CANONICAL in multiple places.
The audit script would FAIL a config using the contract's canonical internal path
and PASS one using a path the contract does not name.

Fix: Rule 5 now accepts the canonical internal base (http://192.168.68.116/litellm/v1)
AND the public base (https://litellm.sysloggh.net/v1), and FAILS anything else.
The internal nginx serves both /litellm/v1 and /v1; the public host serves /v1 only
(per 2026-09-19 probe from CT 116).

Tests aligned: BASE template updated to use the canonical internal path, and new
test cases added to prove the canonical internal path PASSES, a wrong path FAILS,
and the public host path PASSES.
2026-09-19 11:37:14 +00:00
abiba-bot aa3da83af5 Merge pull request 'fix(litellm-health): retry single-host model probes once at a longer timeout' (#121) from fix/litellm-health-retry-timeout-20260919 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-09-19 10:57:39 +00:00
root aee2de25ac fix(litellm): Report both attempts' failure kinds in probe-failed
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 4s
The failure line now preserves both attempts' failure kinds instead of
hardcoding 'timeout after retry, 45s'. If both attempts fail, the report
shows: 'probe-failed: <model> <first kind> then <retry kind> (2 attempts)'.

This fixes the self-contradictory output when the first attempt timed out
but the retry failed with connection refused, and prevents the duration
from appearing twice when both attempts were timeouts.

Example outputs:
- timeout then timeout: 'probe-failed: gpu-dense timeout after 30s then timeout after 45s (2 attempts)'
- timeout then refused: 'probe-failed: gpu-dense timeout after 30s then connection refused (2 attempts)'
- refused then refused: 'probe-failed: gpu-dense connection refused then connection refused (2 attempts)'
2026-09-19 10:53:44 +00:00
root 76653381ec fix(litellm): Add retry with longer timeout for single-host model probes
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Single-host models (gpu-dense, gpu-vision, strix-moe) now retry once at
45s on initial 30s timeout failure before declaring probe-failed. This
prevents a single transient timeout (cold prefill ~13s or concurrent
generation hold) from failing the entire health digest.

Evidence: 2026-09-19 ~06:55Z digest failed gpu-dense at 30s; 06:56Z
direct probe 200 in 1.04s.

The failed-probe-fails-the-run property is preserved: if both attempts
fail, the script still exits non-zero with the target and duration named.

Closes: daily-health-digest false negative on single transient timeout
2026-09-19 10:43:54 +00:00
abiba-bot 3b32cc9658 Merge pull request 'fix(infra-monitoring): probe the real Docker Stats and PVE exporter ports' (#120) from fix/infra-monitoring-probe-ports-20260919 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 3s
2026-09-19 02:56:31 +00:00
root d07c4494b5 fix(infra): F1 - Fix Docker Stats (9324) and PVE Exporter (9221) port comments
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
The leg comments were wrong:
- Line 207: Docker Stats showed :9323 (dockerd port) but should be :9324
- Line 215: PVE Exporter showed :9324 (docker-stats port) but should be :9221
These were the exact pairing this PR exists to correct.

Read back the changed lines to verify:
scripts/infra-monitoring.sh:207 shows Docker Stats (CT 116 :9324, 127.0.0.1 via SSH)
scripts/infra-monitoring.sh:215 shows PVE Exporter (CT 116 :9221, 127.0.0.1 via SSH)

Branch: fix/infra-monitoring-probe-ports-20260919
2026-09-19 02:52:33 +00:00
abiba-bot 66ad5ac89d Merge pull request 'feat(proxmox-monitor): add PBS GC liveness signal' (#119) from fix/pbs-gc-liveness-signal-20260919 into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-09-19 02:51:34 +00:00
root b10fd6fc98 fix(infra): F1+F2 - Fix port comments and add per-leg assertions
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
F1: Fixed leg comments to match the actual ports
- Line 207: Docker Stats now shows :9324 (was :9323)
- Line 215: PVE Exporter now shows :9221 (was :9324)
These were the exact pairing this PR exists to correct.

F2: Added per-leg assertions that prove which leg owns which port
The new assertions verify:
1. Docker Stats leg uses $DOCKER_STATS_PORT constant
2. PVE Exporter leg uses $PVE_EXPORTER_PORT constant
3. DOCKER_STATS_PORT constant is set to 9324
4. PVE_EXPORTER_PORT constant is set to 9221

Proof the new assertions bite:
Under the both-constants-swapped mutation (DOCKER_STATS_PORT=9221,
PVE_EXPORTER_PORT=9324), the suite fails with 25 passed / 2 failed
(failing exactly the two constant-value assertions). This proves the
per-leg assertions pin which leg owns which port, not just that both
ports appear somewhere in the SSH log.

Branch: fix/infra-monitoring-probe-ports-20260919
2026-09-19 02:46:30 +00:00
root a13457bcd6 fix(infra): Fix Docker Stats (9324) and PVE Exporter (9221) probe ports
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Previously probed wrong ports:
- Docker Stats was at 9323 (dockerd metrics) but should be 9324
  (harness-docker-stats, docker_container_* metrics)
- PVE Exporter was at 9324 (harness-docker-stats) but should be 9221
  (harness-pve-exporter, 5 pve_* metrics)

Both exporters bind to 127.0.0.1 on CT 116 and must be probed via SSH.

Updated infrastructure-monitoring.prose.md to document the correct ports.
Added test assertions verifying the exact ports are probed.

Branch: fix/infra-monitoring-probe-ports-20260919
2026-09-19 02:32:00 +00:00
13 changed files with 519 additions and 76 deletions
+16 -6
View File
@@ -114,12 +114,22 @@ def audit(path):
)
# --- Rule 5: Main Config Base URL ---
expected_base = "http://192.168.68.116/v1"
check(
model.get("base_url") == expected_base,
"Rule 5",
f"model.base_url must be {expected_base} (got {model.get('base_url')!r}) — /v1 not /litellm/v1",
)
# Canonical internal base (hermes-key-enforcement.prose.md:19/38/57/84/91/96)
# and public base (serves /v1 only, per 2026-09-19 probe from CT 116).
# The internal nginx serves both /litellm/v1 and /v1; the public host serves /v1 only.
# FAIL anything else (do not widen to accept any path ending in /v1).
# Internal /v1 is non-canonical but working (authenticated via nginx), so WARN not FAIL.
canonical_internal = "http://192.168.68.116/litellm/v1"
public_host = "https://litellm.sysloggh.net/v1"
non_canonical_internal = "http://192.168.68.116/v1"
allowed_bases = (canonical_internal, public_host)
actual_base = model.get("base_url")
if actual_base in allowed_bases:
check(True, "Rule 5", f"model.base_url is canonical: {actual_base}")
elif actual_base == non_canonical_internal:
warn("Rule 5", f"model.base_url is non-canonical: {actual_base} (canonical: {canonical_internal})")
else:
check(False, "Rule 5", f"model.base_url must be one of {allowed_bases} (got {actual_base!r})")
# --- Rule 6: max_tokens Is Required ---
check(
+1 -1
View File
@@ -628,7 +628,7 @@ contracts:
sensitivity: high
status: active
owner: abiba
version: 3.3.0
version: 3.4.0
trigger:
type: scheduled
cadence: '*/15 * * * *'
+4 -4
View File
@@ -16,7 +16,7 @@ author: Abiba (pi agent)
## Rule (One Sentence)
**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with canonical internal path `http://192.168.68.116/litellm/v1` (Hermes appends `/v1/responses`) or public path `https://litellm.sysloggh.net/v1` — hardcoded keys AND direct `:4000` access are both forbidden. Internal `/v1` still works but is non-canonical (WARN, not FAIL).**
## Scope
@@ -38,8 +38,8 @@ Syslog is migrating away from **unauthenticated direct access** to the shared in
| `http://192.168.68.116/litellm/v1` | Bearer `sk-*` key (nginx-fronted) | ✅ **CURRENT / CANONICAL** — captain-approved migration target; 600s proxy_read_timeout (verified) |
| `http://192.168.68.116:4000/v1` | Bearer `sk-*` key (direct container) | ❌ **FORBIDDEN** — bypasses nginx; port 4000 direct is not a config path |
All harness/litellm providers MUST use an authenticated nginx-fronted path (`/litellm/v1` canonical, `/v1` legacy-valid).
Any `base_url` pointing at `:4000` or a bare IP without nginx is a **migration violation**.
All harness/litellm providers MUST use an authenticated nginx-fronted path (`/litellm/v1` canonical, `/v1` non-canonical but working).
The public host `https://litellm.sysloggh.net` serves `/v1` ONLY (404 on `/litellm/v1`).
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
@@ -113,7 +113,7 @@ model:
model:
provider: harness
base_url: http://192.168.68.116/v1 # ← RULE VIOLATION: unauthenticated path
base_url: http://192.168.68.116/v1 # ← NON-CANONICAL but WORKING (authenticated via nginx, WARN not FAIL)
api_key_env: LITELLM_API_KEY
```
+1 -1
View File
@@ -759,4 +759,4 @@ which was kill+nohup outside systemd) are banned by policy.
| Script | Why Disabled |
|--------|-------------|
| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
| `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |
| `zulip-monitor.sh` (Abiba) | Standalone cron replaced by agent-health-check.py + PM2 auto-restart; the script itself remains active as the execution step of the `zulip-health` responsibility contract (see `zulip-health.prose.md`) |
+21
View File
@@ -277,6 +277,27 @@ code (or failure kind with retry details). Apply the standing probe rules: any
HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not
a failure.
### Docker Stats and PVE Exporter Ports
These two exporters bind to 127.0.0.1 on CT 116 (localhost-only) and must be probed via SSH:
| Exporter | Port | Container | Metrics |
|----------|------|-----------|---------|
| **Docker Stats** | **9324** | harness-docker-stats | `docker_container_*` (per-container CPU/mem/network) |
| **PVE Exporter** | **9221** | harness-pve-exporter | `pve_*` (5 cluster-level metrics) |
**IMPORTANT**: Do not confuse with port 9323, which is owned by dockerd and serves the Docker Engine's own metrics (`builder_builds_*`, `containerd_build_info_*`).
```bash
# Docker Stats (harness-docker-stats)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9324/metrics"
# Expected: 200 or 404 (any HTTP status = ALIVE)
# PVE Exporter (harness-pve-exporter)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9221/metrics"
# Expected: 200 or 404 (any HTTP status = ALIVE)
```
### Phase 1: GPU Exporters
**NVIDIA (.8 and .110)**:
+4 -4
View File
@@ -60,8 +60,8 @@ GPU_PATH="/metrics"
GPU_EXPECTED="200"
# Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116
DOCKER_STATS_PORT="9323"
PVE_EXPORTER_PORT="9324"
DOCKER_STATS_PORT="9324" # harness-docker-stats (docker_container_* metrics)
PVE_EXPORTER_PORT="9221" # harness-pve-exporter (5 pve_* metrics)
CT116_SSH_HOST="192.168.68.116"
# Both are bare-200: 404 = container not yet started
DOCKER_STATS_EXPECTED="200|404"
@@ -204,7 +204,7 @@ if [ ${#GPU_FAILED[@]} -gt 0 ]; then
FAILED+=("gpu-exporters: ${GPU_FAILED[*]}")
fi
# 6. Docker Stats (CT 116 :9323, 127.0.0.1 via SSH) — 200|404
# 6. Docker Stats (CT 116 :9324, 127.0.0.1 via SSH) — 200|404
if probe_http "127.0.0.1" "$DOCKER_STATS_PORT" "/" "$DOCKER_STATS_EXPECTED" "" "$CT116_SSH_HOST"; then
echo " ✅ Docker Stats: alive"
else
@@ -212,7 +212,7 @@ else
FAILED+=("docker-stats")
fi
# 7. PVE Exporter (CT 116 :9324, 127.0.0.1 via SSH) — 200|404
# 7. PVE Exporter (CT 116 :9221, 127.0.0.1 via SSH) — 200|404
if probe_http "127.0.0.1" "$PVE_EXPORTER_PORT" "/" "$PVE_EXPORTER_EXPECTED" "" "$CT116_SSH_HOST"; then
echo " ✅ PVE Exporter: alive"
else
+67 -13
View File
@@ -55,9 +55,12 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
try:
rc, stdout, stderr = run_command(cmd, timeout)
if rc != 0:
# Determine failure kind from curl exit code
# Check if this is a timeout from run_command (rc=1, stderr="TIMEOUT")
if rc == 1 and stderr == "TIMEOUT":
return (000, "timeout after " + str(timeout) + "s")
# Otherwise, determine failure kind from curl exit code
# curl exit codes: 28=timeout, 7=refused, 6=dns, 35=ssl, 52=empty
if rc == 28:
elif rc == 28:
return (000, "timeout after " + str(timeout) + "s")
elif rc == 7:
return (000, "connection refused")
@@ -73,6 +76,20 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
except subprocess.TimeoutExpired:
return (000, "timeout after " + str(timeout) + "s")
def check_host_health(host_ip):
"""Check if the GPU host's llama-chat-api health endpoint is reachable
Returns: (healthy: bool, detail: str)
"""
code, _ = probe_http("http://" + host_ip + ":8080/health", timeout=10)
if code == 200:
return True, "host healthy (200)"
elif code == 000:
return False, "host unreachable (timeout or refused)"
else:
return False, "host unhealthy (HTTP " + str(code) + ")"
def get_response_body(url, method="POST", bearer_token=None, data=None, timeout=30):
"""Get response body for 401/403 credential faults (truncated to 200 chars)"""
cmd = "curl -s -m " + str(timeout)
@@ -115,29 +132,53 @@ def check_model_probes():
results = []
# Host health mapping: model -> host IP
model_hosts = {
"gpu-dense": "192.168.68.8", # RTX 3090
"gpu-vision": "192.168.68.110", # RTX 5070
"strix-moe": "192.168.68.15" # Strix Halo
}
for model in ["gpu-dense", "gpu-vision", "strix-moe"]:
# Single-host aliases: 30s timeout each
# gpu-dense (RTX 3090) may need long warmup/prefill - timeout is acceptable on cold-start
host_ip = model_hosts[model]
# Single-host aliases: 30s initial timeout, retry once at 90s on failure
# Worst-case prefill ~76s, so 90s retry ensures we cover it
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
method="POST",
bearer_token=monitor_key,
data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
timeout=30)
first_kind = None
if code == 000 and failure_kind:
# Report probe failure with kind, do not assert a service verdict
results.append((model, False, "probe-failed: " + model + " " + failure_kind + " (30s timeout)"))
first_kind = failure_kind
time.sleep(1)
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
method="POST",
bearer_token=monitor_key,
data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
timeout=90)
if code == 000 and failure_kind:
# Both attempts failed - check host health to distinguish busy from down
host_healthy, host_detail = check_host_health(host_ip)
if host_healthy:
results.append((model, False, "busy (completion timed out after retry; " + host_detail + ")"))
else:
# Host unreachable - report both kinds
if first_kind:
results.append((model, False, "probe-failed: " + model + " " + first_kind + " then " + failure_kind + " (2 attempts)"))
else:
results.append((model, False, "probe-failed: " + model + " " + failure_kind))
elif code == 200:
results.append((model, True, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
elif code in (401, 403):
# Credential fault - capture body and key alias
body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
method="POST",
bearer_token=monitor_key,
data='{"model":"' + model + '","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
timeout=10)
# Resolve key alias
alias = "monitor-20260813" # Known from /etc/litellm-monitor.env on CT 116
alias = "monitor-20260813"
results.append((model, False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
else:
results.append((model, False, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
@@ -247,6 +288,7 @@ def main():
print("")
all_pass = True
degraded = [] # Track degraded (busy) checks
# Run all checks
checks = [
@@ -266,9 +308,15 @@ def main():
# Model probes
model_results = check_model_probes()
for name, passed, detail in model_results:
status = "✅" if passed else "❌"
# Check if this is a busy (degraded) verdict
if not passed and detail.startswith("busy "):
status = "⚠️"
degraded.append(name)
else:
status = "✅" if passed else "❌"
print(" " + status + " " + name + ": " + detail)
if not passed:
# Only set all_pass=False for real failures (not busy)
if not passed and not detail.startswith("busy "):
all_pass = False
# Admin key list
@@ -294,10 +342,16 @@ def main():
print("")
if all_pass:
print("✅ All checks passed")
if degraded:
print("✅ All checks passed (" + str(len(degraded)) + " degraded: " + ", ".join(degraded) + ")")
else:
print("✅ All checks passed")
return 0
else:
print("❌ Some checks failed")
if degraded:
print("❌ Some checks failed (" + str(len(degraded)) + " degraded: " + ", ".join(degraded) + ")")
else:
print("❌ Some checks failed")
return 1
if __name__ == "__main__":
+24 -4
View File
@@ -121,12 +121,32 @@ assert "Port 9405 NOT in any curl call" \
'! grep -q ":9405" "$CURL_LOG"'
# ── 5. Docker Stats / PVE Exporter: SSH-probed at correct ports ────────────
assert "Docker Stats probed at port 9323 via SSH" \
'grep -q "9323" "$SSH_LOG"'
assert "PVE Exporter probed at port 9324 via SSH" \
assert "Docker Stats probed at port 9324 via SSH" \
'grep -q "9324" "$SSH_LOG"'
assert "PVE Exporter probed at port 9221 via SSH" \
'grep -q "9221" "$SSH_LOG"'
# ── 5a. Per-leg assertions (proves which leg owns which port) ──────────────
# The script source must show Docker Stats using $DOCKER_STATS_PORT and
# PVE Exporter using $PVE_EXPORTER_PORT in the correct leg sections
assert "Docker Stats leg uses DOCKER_STATS_PORT constant" \
'grep -A 3 "# 6. Docker Stats" "$SCRIPT" | grep -q "\$DOCKER_STATS_PORT"'
assert "PVE Exporter leg uses PVE_EXPORTER_PORT constant" \
'grep -A 3 "# 7. PVE Exporter" "$SCRIPT" | grep -q "\$PVE_EXPORTER_PORT"'
# Verify the constants themselves are set to the correct values
assert "DOCKER_STATS_PORT constant set to 9324" \
'grep -q "^DOCKER_STATS_PORT=\"9324\"" "$SCRIPT"'
assert "PVE_EXPORTER_PORT constant set to 9221" \
'grep -q "^PVE_EXPORTER_PORT=\"9221\"" "$SCRIPT"'
# ── 5b. Stale port 9323 (dockerd) NOT probed ──────────────────────────────
assert "Port 9323 (dockerd) NOT in SSH log" \
'! grep -q "9323" "$SSH_LOG"'
# ── 6. No stale ports in the script source (belt-and-suspenders) ──────────
assert "Port 9325 (historical) NOT in script source" \
'! grep -q "9325" "$SCRIPT"'
+40 -9
View File
@@ -176,6 +176,11 @@ fi
# never contact her former host.
# ── Platform C: Agent Zero (kagentz) ──
# C1: A2A liveness (no credential needed) — probes the container's internal :80/a2a/
# C2: A2A response verification (needs LITELLM_KEY) — probes POST /a2a with auth
# C3: Public access path (no credential needed) — probes https://kagentz.sysloggh.net/
# C1: A2A liveness (container-internal probe)
AZ_A2A_CODE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \
"docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/ 2>/dev/null" 2>/dev/null) || AZ_A2A_CODE="000"
AZ_A2A_CODE=$(printf '%s' "$AZ_A2A_CODE" | tr -d '[:space:]')
@@ -184,27 +189,53 @@ AZ_A2A_CODE=$(printf '%s' "$AZ_A2A_CODE" | tr -d '[:space:]')
if [ "$AZ_A2A_CODE" = "000" ]; then
notify "🔴" "kagentz A2A server DOWN (connection failed)"
ISSUES=$((ISSUES + 1))
echo " kagentz: ❌ A2A down (HTTP 000)" >> "$LOG"
echo " kagentz C1: ❌ A2A down (HTTP 000)" >> "$LOG"
else
case "$AZ_A2A_CODE" in
200|401)
echo " kagentz: ✅ A2A alive (HTTP $AZ_A2A_CODE)" >> "$LOG" ;;
echo " kagentz C1: ✅ A2A alive (HTTP $AZ_A2A_CODE)" >> "$LOG" ;;
*)
notify "🟡" "kagentz A2A server answered HTTP $AZ_A2A_CODE — running, unexpected status"
ISSUES=$((ISSUES + 1))
echo " kagentz: 🟡 A2A unexpected http=$AZ_A2A_CODE (running, warning)" >> "$LOG" ;;
echo " kagentz C1: 🟡 A2A unexpected http=$AZ_A2A_CODE (running, warning)" >> "$LOG" ;;
esac
fi
# C3: Public access path (the captain's point of view)
# Probes the public URL that NetBird proxies to the container. 200/302/401 = alive,
# 502 = proxy's "upstream refused" page (incident), connection failed = incident.
# Never restarts anything — the contract forbids restarting the platform.
KAGENTZ_PUBLIC_CODE=$(curl -s -o /dev/null --connect-timeout 10 --max-time 15 \
-w '%{http_code}' https://kagentz.sysloggh.net/ 2>/dev/null) || KAGENTZ_PUBLIC_CODE="000"
KAGENTZ_PUBLIC_CODE=$(printf '%s' "$KAGENTZ_PUBLIC_CODE" | tr -d '[:space:]')
[ -n "$KAGENTZ_PUBLIC_CODE" ] || KAGENTZ_PUBLIC_CODE="000"
if [ "$KAGENTZ_PUBLIC_CODE" = "000" ]; then
notify "🔴" "kagentz public URL DOWN (connection failed)"
ISSUES=$((ISSUES + 1))
echo " kagentz C3: ❌ public URL down (HTTP 000)" >> "$LOG"
elif [ "$KAGENTZ_PUBLIC_CODE" = "502" ]; then
notify "🔴" "kagentz public URL 502 (upstream refused)"
ISSUES=$((ISSUES + 1))
echo " kagentz C3: ❌ public URL 502 (upstream refused)" >> "$LOG"
else
case "$KAGENTZ_PUBLIC_CODE" in
200|302|401)
echo " kagentz C3: ✅ public URL alive (HTTP $KAGENTZ_PUBLIC_CODE)" >> "$LOG" ;;
*)
notify "🟡" "kagentz public URL answered HTTP $KAGENTZ_PUBLIC_CODE — running, unexpected status"
ISSUES=$((ISSUES + 1))
echo " kagentz C3: 🟡 public URL unexpected http=$KAGENTZ_PUBLIC_CODE (running, warning)" >> "$LOG" ;;
esac
fi
# ── Summary ──
# The run verdict is non-optimistic: when ISSUES > 0, the run is an INCIDENT.
# The lane must quote this Result line verbatim in its status report.
if [ "$ISSUES" -eq 0 ]; then
if [ "$ZULIP_CRED_OK" -eq 0 ]; then
echo " Result: ✅ All healthy" >> "$LOG"
else
echo " Result: ✅ All healthy" >> "$LOG"
fi
echo " Result: ✅ 0 issues (all healthy)" >> "$LOG"
else
echo " Result: 🔴 $ISSUES issue(s) found" >> "$LOG"
echo " Result: 🔴 INCIDENT — $ISSUES issue(s) found" >> "$LOG"
notify "🔴" "$ISSUES issue(s) found — check /root/zulip-health-monitor.log"
fi
+72 -2
View File
@@ -24,7 +24,7 @@ BASE = """
model:
api_key: ""
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1
base_url: http://192.168.68.116/litellm/v1
max_tokens: 4096
default: syslog-auto
provider: harness
@@ -52,7 +52,7 @@ delegation:
custom_providers:
- name: harness
key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1
base_url: http://192.168.68.116/litellm/v1
"""
@@ -189,3 +189,73 @@ def test_retired_alias_in_nested_auxiliary_block_is_rejected(tmp_path):
assert code == 1, out
assert "auxiliary.tasks.summarize.model = 'gpu-light'" in out
assert "RESULT: FAIL" in out
def test_canonical_internal_path_passes(tmp_path):
"""Rule 5 must accept the canonical internal base from hermes-key-enforcement.prose.md."""
code, out = _run(
tmp_path,
"gpu-vision",
)
# Override the base_url in the config
cfg_text = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/v1",
" base_url: http://192.168.68.116/litellm/v1",
)
code, out = _run_config(tmp_path, "canonical-internal.yaml", cfg_text)
assert code == 0, out
assert "RESULT: PASS" in out
# Verify the correct message is shown
assert "model.base_url is canonical" in out
def test_wrong_base_url_fails(tmp_path):
"""Rule 5 must reject paths outside the allowed list."""
cfg_text = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/litellm/v1",
" base_url: http://192.168.68.116/litellm/v1/responses",
)
code, out = _run_config(tmp_path, "wrong-base.yaml", cfg_text)
assert code == 1, out
assert "RESULT: FAIL" in out
assert "model.base_url must be one of" in out
def test_public_host_path_passes(tmp_path):
"""Rule 5 must accept the public host base."""
cfg_text = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/v1",
" base_url: https://litellm.sysloggh.net/v1",
)
code, out = _run_config(tmp_path, "public-host.yaml", cfg_text)
assert code == 0, out
assert "RESULT: PASS" in out
def test_old_rule5_check_would_fail_canonical(tmp_path):
"""
Proof that the OLD Rule 5 check would fail the canonical internal path.
This proves the bug existed before the fix.
"""
# OLD check expected /v1, so the canonical /litellm/v1 would have failed
canonical_cfg = BASE.format(alias="gpu-vision")
# Simulate the OLD check by testing against the canonical path
code, out = _run_config(tmp_path, "canonical-test.yaml", canonical_cfg)
# NEW check: canonical /litellm/v1 SHOULD pass
assert code == 0, out
assert "RESULT: PASS" in out
# OLD check expected /v1, so the internal /v1 would have passed
# NEW check: internal /v1 is non-canonical but working (WARN not FAIL)
old_cfg = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/litellm/v1",
" base_url: http://192.168.68.116/v1",
)
code, out = _run_config(tmp_path, "old-check-test.yaml", old_cfg)
assert code == 0, out
assert "RESULT: PASS" in out
# NEW check should pass
new_cfg = BASE.format(alias="gpu-vision")
code, out = _run_config(tmp_path, "new-check-test.yaml", new_cfg)
assert code == 0, out
assert "RESULT: PASS" in out
+18 -13
View File
@@ -98,8 +98,8 @@ exit 0
"""
CURL_STUB = r"""#!/usr/bin/env bash
# Stub curl: serve the Abiba health fixture and the Zulip server 200, and
# record every call (including notify) payloads.
# Stub curl: serve the Abiba health fixture, the Zulip server 200, and the
# kagentz C3 public URL, and record every call (including notify) payloads.
printf '%s\n' "$*" >> "$RECORD_DIR/curl.calls"
case "$*" in
*:9200/health*)
@@ -109,6 +109,8 @@ case "$*" in
esac ;;
*server_settings*)
printf '%s' "$SERVER_HTTP" ;;
*kagentz.sysloggh.net*)
printf '%s' "$KAGENTZ_PUBLIC_CODE" ;;
esac
exit 0
"""
@@ -121,7 +123,8 @@ def _write_exec(path: pathlib.Path, body: str) -> None:
def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
az_a2a_code="401", az_a2a_exit=0):
az_a2a_code="401", az_a2a_exit=0,
kagentz_public_code="302"):
"""Run the shipped monitor in a sandbox; return (proc, record_dir, log_path).
Only the LOG constant is rewritten (to keep the run inside the worktree).
@@ -154,6 +157,7 @@ def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
"PI_HTTP": "200",
"PI_BODY": CONNECTED_FIXTURE.read_text(),
"SERVER_HTTP": "200",
"KAGENTZ_PUBLIC_CODE": kagentz_public_code,
})
proc = subprocess.run(["bash", str(script)], cwd=sandbox, env=env,
capture_output=True, text=True)
@@ -169,8 +173,9 @@ def test_healthy_run_is_quiet_and_never_reaches_mumuni(tmp_path):
assert "Server: ✅ HTTP 200" in log
assert "Abiba: ✅ Connected" in log
assert "Tanko: ✅ service=active http=200" in log
assert "kagentz: ✅ A2A alive (HTTP 401)" in log
assert "Result: ✅ All healthy" in log
assert "kagentz C1: ✅ A2A alive (HTTP 401)" in log
assert "kagentz C3: ✅ public URL alive (HTTP 302)" in log
assert "Result: ✅ 0 issues (all healthy)" in log
# A healthy run emits no notify at all — and certainly no Mumuni one.
assert proc.stdout == ""
@@ -207,8 +212,8 @@ def test_failing_run_alerts_on_tanko_but_never_on_mumuni(tmp_path):
# The rest of the monitor still ran alongside the failing Tanko leg.
log = log_path.read_text()
assert "Abiba: ✅ Connected" in log
assert "kagentz: ✅ A2A alive" in log
assert "Result: 🔴 1 issue(s) found" in log
assert "kagentz C1: ✅ A2A alive" in log
assert "Result: 🔴 INCIDENT — 1 issue(s) found" in log
def test_unexpected_a2a_status_is_an_issue_not_healthy(tmp_path):
@@ -216,9 +221,9 @@ def test_unexpected_a2a_status_is_an_issue_not_healthy(tmp_path):
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
assert "kagentz: 🟡 A2A unexpected http=500 (running, warning)" in log
assert "kagentz: ✅ A2A alive" not in log
assert "Result: 🔴 1 issue(s) found" in log
assert "kagentz C1: 🟡 A2A unexpected http=500 (running, warning)" in log
assert "kagentz C1: ✅ A2A alive" not in log
assert "Result: 🔴 INCIDENT — 1 issue(s) found" in log
assert "kagentz A2A server answered HTTP 500" in proc.stdout
@@ -230,10 +235,10 @@ def test_a2a_connection_failure_is_down_not_unexpected(tmp_path):
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
assert "kagentz: ❌ A2A down (HTTP 000)" in log
assert "kagentz: ✅ A2A alive" not in log
assert "kagentz C1: ❌ A2A down (HTTP 000)" in log
assert "kagentz C1: ✅ A2A alive" not in log
assert "unexpected" not in log
assert "Result: 🔴 1 issue(s) found" in log
assert "Result: 🔴 INCIDENT — 1 issue(s) found" in log
assert "kagentz A2A server DOWN (connection failed)" in proc.stdout
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/env python3
"""Behavioural tests for zulip-monitor.sh kagentz C1/C3 legs and the Result verdict.
WHY THIS FILE EXISTS: the 2026-09-19 kagentz A2A outage was correctly detected
by the monitor (C1 returned 000, the run logged an issue) but the lane's own
summarization in ops.status wrote "OK" with the note "A2A server DOWN — expected
(no credentials configured)". The optimistic verdict came from the lane, not the
script. The fix adds a C3 public-access-path leg and makes the Result line say
"INCIDENT" when issues are found, so the lane can quote it verbatim.
CONTRACT UNDER TEST:
* C1 (A2A liveness, no credential): 000 → INCIDENT.
* C3 (public access path, no credential): 502 → INCIDENT, 000 → INCIDENT,
200/302/401 → alive.
* Result verdict: when ISSUES > 0, the log's Result line says "INCIDENT",
not just "issues found".
* Healthy control: C1 401 + C3 302 → 0 issues, "all healthy".
HOW: behavioural execution using the sandbox pattern already in this repo
(tests/test_mumuni_monitor_removal.py). The sandbox copies the shipped monitor
verbatim, rewrites only its LOG constant, and runs it with stub ssh/curl on
PATH. Each test asserts from the run's own log/verdict, not from file text.
Usage: python3 -m pytest tests/test_zulip_kagentz_legs.py
"""
from __future__ import annotations
import os
import pathlib
import stat
import subprocess
ROOT = pathlib.Path(__file__).resolve().parents[1]
ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
# ── Stub ssh: answers Tanko and Agent Zero probes by env vars ──────────
SSH_STUB = r"""#!/usr/bin/env bash
# Stub ssh: record the target host, then answer by host + remote command.
printf '%s\n' "$*" >> "$RECORD_DIR/ssh.calls"
host=""
for a in "$@"; do
case "$a" in
*@192.168.*) host="${a##*@}" ;;
esac
done
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
cmd="${*: -1}"
case "$host" in
192.168.68.15)
case "$cmd" in
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
*curl*) printf '%s' "$TANKO_HTTP" ;;
esac ;;
192.168.68.14)
case "$cmd" in
*"/a2a/"*) printf '%s' "$AZ_A2A_CODE"; exit "${AZ_A2A_EXIT:-0}" ;;
esac ;;
*)
printf 'UNEXPECTED-SSH-HOST %s\n' "$host" >> "$RECORD_DIR/unexpected-ssh" ;;
esac
exit 0
"""
# ── Stub curl: serves Zulip server, Abiba health, and C3 public URL ───
CURL_STUB = r"""#!/usr/bin/env bash
# Stub curl: serve the Abiba health fixture, the Zulip server 200, and the
# C3 public URL probe (https://kagentz.sysloggh.net/). Record every call.
printf '%s\n' "$*" >> "$RECORD_DIR/curl.calls"
case "$*" in
*:9200/health*)
case " $* " in
*" -w "*) printf '%s' "$PI_HTTP" ;; # -w '%{http_code}' probe
*) printf '%s' "$PI_BODY" ;; # body probe
esac ;;
*server_settings*)
printf '%s' "$SERVER_HTTP" ;;
*kagentz.sysloggh.net*)
printf '%s' "$KAGENTZ_PUBLIC_CODE" ;;
esac
exit 0
"""
def _write_exec(path: pathlib.Path, body: str) -> None:
path.write_text(body)
path.chmod(path.stat().st_mode
| stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
az_a2a_code="401", az_a2a_exit=0,
kagentz_public_code="302"):
"""Run the shipped monitor in a sandbox; return (proc, record_dir, log_path).
Only the LOG constant is rewritten (to keep the run inside the worktree).
Everything else — legs, labels, notify logic — is the shipped script.
"""
sandbox = tmp_path / "sandbox"
bindir = sandbox / "bin"
record = sandbox / "record"
bindir.mkdir(parents=True)
record.mkdir()
_write_exec(bindir / "ssh", SSH_STUB)
_write_exec(bindir / "curl", CURL_STUB)
source = ZULIP_MONITOR.read_text()
log_line = 'LOG="/root/zulip-health-monitor.log"'
assert log_line in source, "LOG constant moved — update the sandbox harness"
log_path = sandbox / "zulip-health-monitor.log"
script = sandbox / "zulip-monitor.sh"
script.write_text(source.replace(log_line, f'LOG="{log_path}"'))
env = dict(os.environ)
env.update({
"PATH": f"{bindir}:{env['PATH']}",
"RECORD_DIR": str(record),
"TANKO_SVC": tanko_svc,
"TANKO_HTTP": tanko_http,
"AZ_A2A_CODE": az_a2a_code,
"AZ_A2A_EXIT": str(az_a2a_exit),
"PI_HTTP": "200",
"PI_BODY": CONNECTED_FIXTURE.read_text(),
"SERVER_HTTP": "200",
"KAGENTZ_PUBLIC_CODE": kagentz_public_code,
})
proc = subprocess.run(["bash", str(script)], cwd=sandbox, env=env,
capture_output=True, text=True)
return proc, record, log_path
# ── Required behavioural cases ─────────────────────────────────────────
def test_c3_502_is_incident(tmp_path):
"""C3 public leg returns 502 → the run's verdict is an INCIDENT,
the C3 line names the 502, and the run is not summarised as healthy."""
proc, record, log_path = _run_monitor(tmp_path, kagentz_public_code="502")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# The C3 line names the 502.
assert "kagentz C3: ❌ public URL 502 (upstream refused)" in log
# The verdict is an INCIDENT, not healthy.
assert "Result: 🔴 INCIDENT" in log
assert "all healthy" not in log
# The run is not summarised as healthy.
assert "✅ 0 issues" not in log
def test_c3_000_is_incident(tmp_path):
"""C3 public leg returns 000 → INCIDENT."""
proc, record, log_path = _run_monitor(tmp_path, kagentz_public_code="000")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# The C3 line reports the connection failure.
assert "kagentz C3: ❌ public URL down (HTTP 000)" in log
# The verdict is an INCIDENT.
assert "Result: 🔴 INCIDENT" in log
assert "all healthy" not in log
assert "✅ 0 issues" not in log
def test_healthy_control_c1_401_c3_302(tmp_path):
"""Healthy control: C1 401 plus C3 302 → 0 issues and a healthy verdict,
proving the new leg cannot cry wolf."""
proc, record, log_path = _run_monitor(tmp_path,
az_a2a_code="401",
kagentz_public_code="302")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# Both legs report alive.
assert "kagentz C1: ✅ A2A alive (HTTP 401)" in log
assert "kagentz C3: ✅ public URL alive (HTTP 302)" in log
# Zero issues, healthy verdict.
assert "Result: ✅ 0 issues (all healthy)" in log
# No INCIDENT.
assert "INCIDENT" not in log
# No notify fired for kagentz.
assert "kagentz public URL" not in proc.stdout
assert "kagentz A2A server" not in proc.stdout
def test_c1_000_is_incident(tmp_path):
"""C1 returns 000 → INCIDENT, keeping the leg that actually caught this
outage covered behaviourally."""
proc, record, log_path = _run_monitor(tmp_path,
az_a2a_code="000",
az_a2a_exit=7,
kagentz_public_code="302")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# The C1 line reports the A2A down.
assert "kagentz C1: ❌ A2A down (HTTP 000)" in log
# The verdict is an INCIDENT (even though C3 is healthy).
assert "Result: 🔴 INCIDENT" in log
assert "all healthy" not in log
# The notify fired for the A2A down.
assert "kagentz A2A server DOWN" in proc.stdout
+42 -19
View File
@@ -1,9 +1,9 @@
---
kind: responsibility
name: zulip-health
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. The kagentz Zulip adapter leg is retired (its code no longer exists) — Agent Zero is probed for A2A liveness only. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side.
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. The kagentz Zulip adapter leg is retired (its code no longer exists) — Agent Zero is probed for A2A liveness, A2A response, and public-path access only. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side.
title: Zulip Mesh Health Monitor — Multi-Platform
version: 3.3.0
version: 3.4.0
runtime_contract: 2
agent: abiba
report_only_agents:
@@ -30,7 +30,7 @@ session start.
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Network access** to `chat.sysloggh.net`, `kagentz.sysloggh.net` (C3 public path), `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
- **Relay access** via RA-H OS MCP for alert delivery
@@ -129,10 +129,10 @@ grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
### Liveness rule (scoped)
Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe. **Proxy-fronted exception:** a reverse proxy's `502`/`504` (upstream refused/unreachable) is not the backend answering — for proxy-fronted public endpoints such as `https://kagentz.sysloggh.net/` (C3), treat it as DOWN.
**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):**
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe (proxy `502`/`504` excepted, above).
2. **A failed probe is never a service verdict.** Print `probe-failed: <target> <kind>` naming the exact URL/host/port and the failure kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only then report.
3. **Say which probe produced each number.** "API: 000" is unusable; "API https://chat.sysloggh.net/api/v1/server_settings -> connection timeout after 10s (retried at 25s: also timeout)" is actionable.
@@ -469,9 +469,10 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null
> monitor issued a restart for something that could not start, posting a false
> kagentz-adapter-down alert on every run. Do NOT re-add an adapter-process,
> heartbeat/queue, or adapter-restart step. Agent Zero is probed for A2A
> liveness only, and a probe must never restart a platform.
> liveness/response and public-path access only, and a probe must never restart
> a platform.
**C1: A2A Server Health**
**C1: A2A Server Health (no credential needed)**
```bash
# A2A listens on :80 inside the agent-zero container (host-mapped to :50080) and
@@ -479,9 +480,9 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/"
```
Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (`000`) → A2A server down. Any other status → running but unexpected: log/report it, never restart.
Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (`000`) → A2A server down (INCIDENT). Any other status → running but unexpected: log/report it, never restart.
**C2: A2A Response Verification**
**C2: A2A Response Verification (requires LITELLM_KEY)**
```bash
# A2A listens on :80 inside the container and is auth-gated (401 expected unauthenticated).
@@ -491,15 +492,30 @@ ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:
-d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
```
Expected: task ID with "working" status. Poll for completion with `tasks/get`. If 401, check LITELLM_KEY is set.
Expected: task ID with "working" status. Poll for completion with `tasks/get`. If 401, check LITELLM_KEY is set (this is a credential issue, NOT a server-down incident).
**C3: Public Access Path (no credential needed)**
```bash
# Probes the public URL that NetBird proxies to the agent-zero container.
# This is the captain's point of view: if the captain can't reach it, it's down.
# 200/302/401 = alive, 502 = proxy's "upstream refused" page (INCIDENT),
# connection failed (000) = INCIDENT. Never restarts anything.
curl -s -o /dev/null --connect-timeout 10 --max-time 15 -w '%{http_code}' https://kagentz.sysloggh.net/
```
Expected: `200` (Agent Zero login page), `302` (redirect), or `401` (auth-gated) = alive. `502` = NetBird proxy's "upstream refused" page (the container's port 80 is not listening) = INCIDENT. `000` (connection failed) = INCIDENT. Any other status → running but unexpected: log/report it, never restart.
**Platform C Actions**
| Condition | Action |
|-----------|--------|
| A2A returns `000` (connection refused/timeout) | Alert only — never restart the platform; investigate the agent-zero container |
| A2A returns a status other than `200`/`401` | Log/report as a warning — reported, never healed on |
| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
| C1 A2A returns `000` (connection refused/timeout) | Alert only — never restart the platform; investigate the agent-zero container |
| C1 A2A returns a status other than `200`/`401` | Log/report as a warning — reported, never healed on |
| C2 A2A returns 401 | Check `LITELLM_KEY` is set (credential issue, NOT server-down) |
| C3 public URL returns `502` (upstream refused) | Alert only — the container's port 80 is not listening; investigate the agent-zero container |
| C3 public URL returns `000` (connection failed) | Alert only — the public path is down; investigate the NetBird proxy or the container |
| C3 public URL returns a status other than `200`/`302`/`401` | Log/report as a warning — reported, never healed on |
### Step 5: Global Checks
@@ -513,12 +529,19 @@ If any bot processes >50 bot-originated messages in 15min → warning.
### Step 6: Compile and Report
1. Compile all platform checks and severity
2. Determine `overall_severity` from worst per-agent severity
3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
5. If any agent critical or >2 degraded: send relay message to user
6. Update `last_check` timestamp in `### Maintains` snapshot
1. Run `scripts/zulip-monitor.sh` and take its final `Result:` line as the
authoritative run verdict. The verdict line is either
`Result: ✅ 0 issues (all healthy)` or
`Result: 🔴 INCIDENT — N issue(s) found`.
2. Quote that `Result:` line verbatim in the status report. When it says
`INCIDENT`, the run MUST be reported as an incident — never summarised as
OK/healthy and never annotated as "expected".
3. Compile all platform checks and severity
4. Determine `overall_severity` from worst per-agent severity
5. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
6. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
7. If any agent critical or >2 degraded: send relay message to user
8. Update `last_check` timestamp in `### Maintains` snapshot
### Restart Debounce