Compare commits

..
Author SHA1 Message Date
root ee57c338ec fix: correct Rule 15 wording and MCP key access contradiction
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 7s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 9s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
1. audit-hermes-config.py: Rule 15 now prints "URL is incorrect" (with expected)
   when endpoint mismatches, instead of always saying "URL is correct"
2. hermes-config-template.prose.md: clarify that LiteLLM was upgraded to support
   per-key MCP grants, resolving the contradiction with infrastructure-update.prose.md:214
2026-09-18 18:20:07 +00:00
20 changed files with 93 additions and 1463 deletions
+6 -16
View File
@@ -114,22 +114,12 @@ def audit(path):
)
# --- Rule 5: Main Config Base URL ---
# Canonical internal base (hermes-key-enforcement.prose.md:19/38/57/84/91/96)
# and public base (serves /v1 only, per 2026-09-19 probe from CT 116).
# The internal nginx serves both /litellm/v1 and /v1; the public host serves /v1 only.
# FAIL anything else (do not widen to accept any path ending in /v1).
# Internal /v1 is non-canonical but working (authenticated via nginx), so WARN not FAIL.
canonical_internal = "http://192.168.68.116/litellm/v1"
public_host = "https://litellm.sysloggh.net/v1"
non_canonical_internal = "http://192.168.68.116/v1"
allowed_bases = (canonical_internal, public_host)
actual_base = model.get("base_url")
if actual_base in allowed_bases:
check(True, "Rule 5", f"model.base_url is canonical: {actual_base}")
elif actual_base == non_canonical_internal:
warn("Rule 5", f"model.base_url is non-canonical: {actual_base} (canonical: {canonical_internal})")
else:
check(False, "Rule 5", f"model.base_url must be one of {allowed_bases} (got {actual_base!r})")
expected_base = "http://192.168.68.116/v1"
check(
model.get("base_url") == expected_base,
"Rule 5",
f"model.base_url must be {expected_base} (got {model.get('base_url')!r}) — /v1 not /litellm/v1",
)
# --- Rule 6: max_tokens Is Required ---
check(
+1 -55
View File
@@ -38,7 +38,6 @@ index:
by_category:
compliance:
- hermes-key-enforcement
- litellm-api-keys
- hermes-config-template
- hermes-agent-baseline
monitoring:
@@ -102,7 +101,6 @@ index:
proxmox:
- proxmox-monitor
litellm:
- litellm-api-keys
- litellm-health
- litellm-self-heal
memory:
@@ -630,7 +628,7 @@ contracts:
sensitivity: high
status: active
owner: abiba
version: 3.4.0
version: 3.3.0
trigger:
type: scheduled
cadence: '*/15 * * * *'
@@ -1871,58 +1869,6 @@ contracts:
drift_alerts: []
# Koby Report-Only Registry (2026-08-17 — Captain)
# ⛔ KOBY IS NEVER REPAIRED — detect + report, never fix on .129
- name: litellm-api-keys
file: litellm-api-keys.prose.md
kind: function
category: compliance
sensitivity: critical
status: active
owner: abiba
version: 1.1.0
trigger:
type: on_demand
cadence: null
description: "Manual invocation when creating/rotating/verifying agent LiteLLM keys"
cron_job_id: null
execution:
agent: abiba
timeout: 120
requires: []
protocol:
- Load contract from prose-contracts/main
- Retrieve master key from Infisical (project=infrastructure env=production)
- Read live key-scoped model roster from CT 116 /v1/models
- Create/rotate/verify the requested agent key with an EXPLICIT models list
- 'Never create a key with an empty models list or all-proxy-models (Cloud leak)'
verification:
postconditions:
- check: standard agent key is local-only
verify: 'curl -s -H "Authorization: Bearer <KEY>" http://192.168.68.116/litellm/v1/models | jq -r ''.data[].id'' | grep -c /'
expect: 0 cloud models
- check: key exists with correct alias
verify: 'curl -s -H "Authorization: Bearer <MASTER>" http://192.168.68.116/litellm/v1/key/info?key_alias=<AGENT>'
expect: 200 with matching alias
artifact: key creation/rotation report
receipt:
format: json
storage: ~/.hermes/runs/litellm-api-keys/
graph_node: true
escalation:
info:
action: log_to_receipt
notify: []
warning:
action: relay_alert
notify:
- abiba
- mumuni
critical:
action: relay_alert
notify:
- abiba
- mumuni
- ops
koby_report_only: true
koby_host: "CT 111 (tdunna)"
koby_ip: ".129"
+1 -16
View File
@@ -61,7 +61,7 @@ Host filesystems have their own risk profile and their own bands. A host root ne
|-------|-----------|----------|------------|
| **HOST-WARN** | 85% | Name the volume + % + absolute free space in the scan output | None |
| **HOST-AMBER** | 90% | Name the volume + % + absolute free space; flag for owner attention | Zulip DM to owner (state-change only) |
| **HOST-RED** | 95% | Name the volume + % + absolute free space; **media volumes: "capacity decision — owner to decide"; pbs-datastore/host-root: "immediate owner attention"** | Zulip DM + channel alert (state-change only) |
| **HOST-RED** | 95% | Name the volume + % + absolute free space; flag for immediate owner attention | Zulip DM + channel alert (state-change only) |
**Escalations are STATE-CHANGE driven, not per-run.** A volume alerts ONCE when it enters a higher band (GREEN->WARN, WARN->AMBER, AMBER->RED) and ONCE when it drops back down (a recovery notice). While a volume stays in the same band, it is reported in the scan output only — no DM, no channel alert. This prevents the same 96% easystore2 from re-DMing the owner on every 6-hour scan and drowning a real warning in noise.
@@ -228,21 +228,6 @@ call summary-reporter
plan: plan
```
## GC SCHEDULE (PBS datastore only)
The PBS GC schedule is defined in ONE authoritative place: `/etc/cron.d/pbs-gc` on storepve.
The schedule is `0 20 * * *` (20:00 LOCAL = 00:00 UTC, since host timezone is America/New_York).
This applies **only** to the PBS datastore (`/tank/pbs-backup`), NOT to media volumes.
Media volumes (/media/*) are report-only at all threat levels.
The cron runs `/usr/local/bin/pbs-gc.sh` which executes:
```bash
proxmox-backup-manager garbage-collection start storepve-datastore
```
This is NOT a `prune` operation; it is a GC pass that reclaims unreferenced chunks.
There is no `--keep-daily` flag; retention is governed by jobs.cfg (keep-daily=35).
The GC does not touch media volumes or any other filesystem.
## GC Strategies by Host Type
### Docker Hosts (kagentz 105, syslog-api 116, docker-vm 109, amdpve .15)
+2 -3
View File
@@ -243,9 +243,8 @@ MCP server entries in `mcp_servers:` must follow the format shown in the Templat
- Tested MCP initialize handshake against litellm.sysloggh.net/mcp with agent virtual key
- Confirmed: 200 response with `serverInfo.name: "litellm-mcp-server"`
- Confirmed: tools/list returns 200 (MCP endpoint accessible with virtual keys)
- Note: Per-key MCP grants are now supported (verified 2026-09-18), resolving the earlier
contradiction with infrastructure-update.prose.md (which now reflects the update)
- Key requirement: must be a valid LiteLLM virtual key (HTTP 200 on /v1/models)
- Note: This differs from infrastructure-update.prose.md:214 ("only master key has access")
— the LiteLLM version was upgraded to support per-key MCP grants
**Key rotation note:**
- MCP headers use literal keys (not env-vars), so they do NOT auto-rotate with the vault
+4 -23
View File
@@ -16,26 +16,7 @@ author: Abiba (pi agent)
## Rule (One Sentence)
**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with canonical internal path `http://192.168.68.116/litellm/v1` (Hermes appends `/v1/responses`) or public path `https://litellm.sysloggh.net/v1` — hardcoded keys AND direct `:4000` access are both forbidden. Internal `/v1` still works but is non-canonical (WARN, not FAIL).**
**Cloud provider models (OpenRouter, DeepSeek, Google AI Studio, QwenCloud PAYG/Plan, Tencent TokenHub PAYG/Plan) added to CT 116 on 2026-09-20 are reachable ONLY through the designated cloud-enabled key. Standard agent keys remain LOCAL-ONLY (`strix-moe`, `gpu-dense`, `gpu-vision`, `syslog-auto`) and MUST NOT be granted cloud models unless explicitly approved by the captain.**
## Model Access Tiers (2026-09-20)
CT 116 hosts two **access tiers** of model. Tier membership is enforced per virtual key via that key's `models` allowlist.
| Tier | Models | Who gets it |
|------|--------|-------------|
| **Local** | `strix-moe`, `gpu-dense`, `gpu-vision`, `syslog-auto` | All standard agent keys (tanko, mumuni, koby, koonimo, abiba-pi) |
| **Cloud** | 47 provider models: `openrouter/*`, `deepseek/*`, `google/*`, `qwen-payg/*`, `qwen-plan/*`, `tencent-payg/*`, `tencent-plan/*` | **Only** the designated cloud-enabled key (captain decision) |
**Rules:**
1. A key with an empty `models` list (`{}`) or `all-proxy-models` is UNSCOPED — it silently gains ALL models, including cloud. Never create or leave an agent key in this state.
2. Agent keys MUST carry an **explicit local-only** `models` list.
3. Granting a cloud model to an agent key requires explicit captain approval and a recorded reason.
4. The master key always bypasses scoping — it is admin-only, never for inference.
See `litellm-api-keys` § Cloud Provider Consolidation for the key-creation procedure.
**All harness/litellm providers MUST use `api_key_env: LITELLM_API_KEY` with authenticated path `http://192.168.68.116/litellm/v1/responses` — hardcoded keys AND unauthenticated `/v1` direct access are both forbidden.**
## Scope
@@ -57,8 +38,8 @@ Syslog is migrating away from **unauthenticated direct access** to the shared in
| `http://192.168.68.116/litellm/v1` | Bearer `sk-*` key (nginx-fronted) | ✅ **CURRENT / CANONICAL** — captain-approved migration target; 600s proxy_read_timeout (verified) |
| `http://192.168.68.116:4000/v1` | Bearer `sk-*` key (direct container) | ❌ **FORBIDDEN** — bypasses nginx; port 4000 direct is not a config path |
All harness/litellm providers MUST use an authenticated nginx-fronted path (`/litellm/v1` canonical, `/v1` non-canonical but working).
The public host `https://litellm.sysloggh.net` serves `/v1` ONLY (404 on `/litellm/v1`).
All harness/litellm providers MUST use an authenticated nginx-fronted path (`/litellm/v1` canonical, `/v1` legacy-valid).
Any `base_url` pointing at `:4000` or a bare IP without nginx is a **migration violation**.
### 🔥 CRITICAL: Double-Path Bug (2026-07-10)
@@ -132,7 +113,7 @@ model:
model:
provider: harness
base_url: http://192.168.68.116/v1 # ← NON-CANONICAL but WORKING (authenticated via nginx, WARN not FAIL)
base_url: http://192.168.68.116/v1 # ← RULE VIOLATION: unauthenticated path
api_key_env: LITELLM_API_KEY
```
+1 -1
View File
@@ -759,4 +759,4 @@ which was kill+nohup outside systemd) are banned by policy.
| Script | Why Disabled |
|--------|-------------|
| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
| `zulip-monitor.sh` (Abiba) | Standalone cron replaced by agent-health-check.py + PM2 auto-restart; the script itself remains active as the execution step of the `zulip-health` responsibility contract (see `zulip-health.prose.md`) |
| `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |
+3 -32
View File
@@ -138,19 +138,11 @@ the any-HTTP rule. On those — the authenticated Zulip POST and the router
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real
tool calls; never repeat a prior report unless a live probe fails.**
**EXECUTABLE OWNER:** The canonical probe set lives in `scripts/infra-monitoring.sh`.
A check run is a single command: `bash scripts/infra-monitoring.sh` (from the
repository root). Paste its raw output verbatim into the report. The script
exits non-zero naming every failed target; there is no "OK" summary when any
leg failed. Port drift is caught by `scripts/test_infra_monitoring.sh` which
asserts every probed port matches the documented value.
**PROBE SHAPE (per standing rules above):**
- Every probe prints: `✅ <name>: alive` on success, or `🔴 <name>: probe-failed: <host>:<port> (expected <pattern>) (<kind>)` on failure
- PVE API failures include `(any-HTTP liveness, -k for self-signed)` to distinguish TLS vs connection
- Retry once on connection failure at longer timeout (25s connect, 30s max)
- Every probe prints the target name + URL + HTTP code (or failure kind)
- Retry once on connection failure at longer timeout
- Any HTTP status = ALIVE; only 000/timeout/refused = probe-failed
- Report the actual probe output, not a summary verdict
- Report the actual probe command and its result, not a summary verdict
```bash
# Provenance — run first; paste the absolute path into the report
@@ -277,27 +269,6 @@ code (or failure kind with retry details). Apply the standing probe rules: any
HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not
a failure.
### Docker Stats and PVE Exporter Ports
These two exporters bind to 127.0.0.1 on CT 116 (localhost-only) and must be probed via SSH:
| Exporter | Port | Container | Metrics |
|----------|------|-----------|---------|
| **Docker Stats** | **9324** | harness-docker-stats | `docker_container_*` (per-container CPU/mem/network) |
| **PVE Exporter** | **9221** | harness-pve-exporter | `pve_*` (5 cluster-level metrics) |
**IMPORTANT**: Do not confuse with port 9323, which is owned by dockerd and serves the Docker Engine's own metrics (`builder_builds_*`, `containerd_build_info_*`).
```bash
# Docker Stats (harness-docker-stats)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9324/metrics"
# Expected: 200 or 404 (any HTTP status = ALIVE)
# PVE Exporter (harness-pve-exporter)
ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9221/metrics"
# Expected: 200 or 404 (any HTTP status = ALIVE)
```
### Phase 1: GPU Exporters
**NVIDIA (.8 and .110)**:
+7 -7
View File
@@ -208,19 +208,19 @@ mcp_servers:
| Key | MCP Access |
|-----|-----------|
| Master key | ✅ Full — 90 tools (vault-injected) |
| Agent keys (mumuni, tanko, etc.) | ✅ Per-key grants supported (as of 2026-09-18 verification) |
| Agent keys (mumuni, tanko, etc.) | ❌ Per-key grants not supported in v1.99.1 |
### Known Limitations
- ~~Per-key MCP server grants not functional — only master key has access~~ (resolved 2026-09-18: per-key grants now work)
- Per-key MCP server grants not functional — only master key has access
- Responses API (`/v1/responses`) with MCP tools broken on llama.cpp backends
- HTTP 307 redirect on `/mcp` → use `/mcp/` (trailing slash) or `/mcp-rest/` endpoints
- `api_mode: responses` in Hermes appends `/v1/responses` to base_url → **base_url must end at `/v1`, never `/responses`** (double-path bug)
### Migration Path (COMPLETED 2026-09-18)
Per-key MCP grants are now supported:
1. ✅ Agent keys granted MCP access via `allowed_mcp_servers` field
2. ✅ Hermes `mcp_servers.litellm.url` set to `https://litellm.sysloggh.net/mcp`
3. ✅ `headers: {x-litellm-api-key: "Bearer <literal_key>"}` added to MCP config
### Migration Path
When LiteLLM is upgraded to a version supporting per-key MCP grants:
1. Grant agent keys `mcp_servers: ["ra_h_os"]`
2. Update Hermes `mcp_servers.ra-h-os.url` from `http://192.168.68.65:3100/mcp` → `http://192.168.68.116:4000/mcp/`
3. Add `headers: {x-litellm-api-key: "Bearer $LITELLM_API_KEY"}` to MCP config
## Security-Specific Updates
-64
View File
@@ -71,10 +71,6 @@ description: >
- Set models: read the live key-scoped set rather than hardcoding one — `/v1/models` is key-scoped,
and the authoritative registry is CT 116 `/opt/inference-harness/litellm_config.yaml`. Do not add
retired names (`gemma-4-12b`, `gpu-light`, `crew-auto` — all retired 2026-09-12).
- **Standard agent keys are LOCAL-ONLY**: `["strix-moe", "gpu-dense", "gpu-vision", "syslog-auto"]`.
Cloud models are granted ONLY to the designated cloud-enabled key — see § Cloud Provider
Consolidation. NEVER create a key with an empty `models` list (`{}`) or `all-proxy-models`.
In LiteLLM Community both silently grant access to EVERY model, including cloud.
- Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed).
- Return the new key
5. **If action == "rotate"**:
@@ -91,66 +87,6 @@ description: >
- Confirm key alias matches agent_name in LiteLLM key list
- Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
## Cloud Provider Consolidation (2026-09-20)
CT 116 LiteLLM (Community v1.99.1) fronts **7 upstream providers** in addition to the local
GPU models. Added 2026-09-20 — 47 cloud deployments, 51 unique model names total.
### Provider map (per-account namespacing)
Two accounts on the same vendor get **distinct prefixes** so billing, rate limits, and keys
stay separate:
| Prefix | Upstream | Auth | Vault secret |
|--------|----------|------|--------------|
| `openrouter/` | OpenRouter | API key | `OPENROUTER_API_KEY` |
| `deepseek/` | DeepSeek direct | API key | `DEEPSEEK_API_KEY` |
| `google/` | Google AI Studio (Gemini) | API key | `GEMINI_API_KEY` |
| `qwen-payg/` | QwenCloud / DashScope (pay-as-you-go) | API key | `DASHSCOPE_PAYG_KEY` |
| `qwen-plan/` | QwenCloud / DashScope (token plan) | API key | `DASHSCOPE_PLAN_KEY` |
| `tencent-payg/` | Tencent TokenHub (PAYG) | Bearer token | `TENCENT_PAYG_KEY` |
| `tencent-plan/` | Tencent TokenHub (Plan) | Bearer token | `TENCENT_PLAN_KEY` |
All cloud `api_key` fields use `os.environ/<NAME>` — the 7 secrets live in Infisical
(project=`infrastructure`, env=`production`, folder=`root`) and are injected into the
`harness-litellm` container at start. **No literal cloud keys in `litellm_config.yaml`.**
### Access tiers (MUST be enforced per key)
| Tier | Model names | Granted to |
|------|-------------|------------|
| **Local** | `strix-moe`, `gpu-dense`, `gpu-vision`, `syslog-auto` | every standard agent key |
| **Cloud** | the 47 provider models (`<prefix>/<model>`) | **only** the designated cloud-enabled key |
> ⚠️ **Community-edition caveat:** LiteLLM Community does not restrict wildcard access groups
> the way Enterprise does. Access is decided by each key's explicit `models` list. A key with
> `models = {}` or `models = ["all-proxy-models"]` sees **all** models — a silent cloud leak.
> Every key MUST carry an explicit list. The master key always bypasses scoping (admin-only).
### Creating the cloud-enabled key
```bash
# ALWAYS read the live roster first (key-scoped):
curl -s -H "Authorization: Bearer <AGENT_KEY>" http://192.168.68.116/litellm/v1/models \
| jq -r '.data[].id'
# Then generate a key with an EXPLICIT model list (never empty, never a wildcard).
# For the cloud-enabled key, list local + cloud. For a standard agent, local only.
```
Verify after any key change: a standard agent key must return **4 models**, and must NOT return
any `<prefix>/` cloud model.
### Adding a new cloud provider
1. Add the upstream key to Infisical `infrastructure/production/root`.
2. Add the deployment(s) to `/opt/inference-harness/litellm_config.yaml` with an `os.environ/` ref
and a namespaced `model_name` (`<provider>-<account>/<model>` when a vendor has >1 account).
3. Restart the `harness-litellm` container.
4. Grant the model to the cloud-enabled key ONLY (explicit list) — never to agent keys without
captain approval.
5. Update this table and the access-tier section.
## Production Vault Access Process (canonical, 2026-07-17)
The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents
-42
View File
@@ -89,48 +89,6 @@ agent: abiba
| minipve | 192.168.68.12 | PVE |
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (strix-moe) |
## PBS GC (Proxmox Backup Server)
### Schedule
Cron `0 20 * * *` on the **storepve HOST** (192.168.68.6) = 20:00 America/New_York local = **00:00 UTC**.
**NOTE**: The closed PR #116 said "20:00 UTC" — this is WRONG by four hours. Do not copy it.
### What Actually Runs
Host script `/usr/local/bin/pbs-gc.sh` runs `pct exec 107 -- proxmox-backup-manager garbage-collection start storepve-datastore`.
**IMPORTANT**: The tool `proxmox-backup-manager` exists only inside CT 107 (where the PBS server runs). The storepve host has only `proxmox-backup-client`. This is why the job had never worked before 2026-09-19 00:00 UTC.
### Datastore Location
- **Datastore**: CT 107's `/mnt/pbs-backup` on the storepve ZFS dataset `/tank/pbs-backup` (pool `tank`, ~11T free)
- **NOT** `/media/easystore2` (media library, 3.7T, 96% used — separate volume)
### Liveness Check
The new `proxmox-monitor.sh` leg checks storepve-datastore GC health:
- Reads GC state from CT 107: `pct exec 107 -- proxmox-backup-manager garbage-collection list --output-format json`
- **FAILS** if `last-run-endtime` is older than 48 hours
- Reports age in hours and pending-bytes
**All six verdict shapes** (exactly as emitted by the script):
1. **Healthy** (fresh GC, 0 B pending):
`✅ PBS GC: healthy (last run 1h ago, pending-bytes: 0 B)`
2. **Stale** (GC ran >48h ago):
`🔴 PBS GC: stale (last run 49h ago, pending-bytes: 1048576 B)`
3. **Probe-failed: empty read** (000/timeout/unreadable):
`🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000)`
4. **Probe-failed: unparseable** (non-empty but invalid JSON — the "command not found" case):
`🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON)`
5. **Never-run: datastore absent** (valid JSON but storepve-datastore not in list):
`🔴 PBS GC: never-run (storepve-datastore not found in GC list)`
6. **Never-run: no endtime** (valid JSON with datastore present but last-run-endtime is null/0):
`🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime)`
## Operations
### view-dashboards
-234
View File
@@ -1,234 +0,0 @@
#!/bin/bash
# infrastructure-monitoring.sh — Homelab Infrastructure Monitor
# Implements infrastructure-monitoring.prose.md (check-health section)
#
# Legs: Grafana, Prometheus, LiteLLM, PVE API (5 nodes), GPU exporters,
# Docker Stats, PVE Exporter
#
# Design:
# - Every target, port, path, and expected status is defined in code
# - Liveness rule: any HTTP status = ALIVE for auth-gated/redirect endpoints;
# only connection failures (000/timeout) = probe-failed
# - Bare-200 rule: expected status must match exactly (200); anything else = alert
# - PVE API uses -k flag (self-signed certs), probes /api2/json/version
# - Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116, probed via SSH
# with one retry at longer timeout (25s connect, 30s max) to distinguish
# transient timeout from host-down
# - Non-zero exit naming every failed target; no "OK" summary when any leg failed
#
# Output shape per leg:
# ✅ <name>: alive
# 🔴 <name>: probe-failed: <host>:<port> (expected <pattern>) (<kind>)
#
# Failure kinds: timeout | refused | tls | unexpected:<code> (printed in the failure line)
set -uo pipefail
# ── Configuration (documented in infrastructure-monitoring.prose.md) ────────
# Change these in ONE place; test_infra_monitoring.sh asserts against these.
GRAFANA_HOST="192.168.68.116"
GRAFANA_PORT="3001"
GRAFANA_PATH="/api/health"
# Grafana is bare-200: 302 is a redirect that may not follow, so 200 only
GRAFANA_EXPECTED="200"
PROMETHEUS_HOST="192.168.68.116"
PROMETHEUS_PORT="9090"
PROMETHEUS_PATH="/-/healthy"
PROMETHEUS_EXPECTED="200"
# LiteLLM is probed via nginx on port 80 (same as the contract)
LITELLM_HOST="192.168.68.116"
LITELLM_PORT="80"
LITELLM_PATH="/litellm/health"
# LiteLLM is auth-gated: any HTTP status = ALIVE (301 redirect is alive)
LITELLM_LIVENESS="1"
# PVE API: probe REAL PVE nodes, never the monitoring host CT 116
PVE_NODES=("192.168.68.9" "192.168.68.12" "192.168.68.6" "192.168.68.15" "192.168.68.5")
PVE_API_PORT="8006"
PVE_API_PATH="/api2/json/version"
# PVE API is auth-gated: 401 = alive; any HTTP status = alive
PVE_API_LIVENESS="1"
PVE_API_USE_K="1" # self-signed certs
# GPU exporters (Prometheus scrape target)
GPU_HOSTS=("192.168.68.8" "192.168.68.110" "192.168.68.15")
GPU_PORT="9400"
GPU_PATH="/metrics"
GPU_EXPECTED="200"
# Docker Stats and PVE Exporter bind to 127.0.0.1 on CT 116
DOCKER_STATS_PORT="9324" # harness-docker-stats (docker_container_* metrics)
PVE_EXPORTER_PORT="9221" # harness-pve-exporter (5 pve_* metrics)
CT116_SSH_HOST="192.168.68.116"
# Both are bare-200: 404 = container not yet started
DOCKER_STATS_EXPECTED="200|404"
PVE_EXPORTER_EXPECTED="200|404"
# ── Probe Functions ─────────────────────────────────────────────────────────
# probe_http <host> <port> <path> <expected_pattern> [use_k] [ssh_host] [scheme] [liveness]
# Returns 0 if probe succeeds (matches expected or liveness), 1 if probe-failed.
# Prints the result line.
#
# FIX C1: The kind value is computed and printed in the failure line.
# FIX C2: SSH retry logic is in the first attempt branch (not unreachable).
LAST_KIND=""
probe_http() {
local host="$1" port="$2" path="$3" expected="$4"
local use_k="${5:-}" ssh_host="${6:-}" scheme="${7:-http}" liveness="${8:-0}"
local url="${scheme}://${host}:${port}${path}"
local code="" kind=""
LAST_KIND=""
# Single invocation that captures both output and status
if [ -n "$ssh_host" ]; then
out=$(ssh -o ConnectTimeout=5 -o BatchMode=yes "root@${ssh_host}" \
"curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 --max-time 15 ${use_k:+-k} ${url}" 2>/dev/null)
rc=$?
else
out=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 --max-time 15 ${use_k:+-k} "$url" 2>/dev/null)
rc=$?
fi
code=$(printf '%s' "$out" | tr -d '[:space:]')
# Classify failure kind and retry if needed
if [ -z "$code" ] || [ "$code" = "000" ]; then
# Distinguish timeout from TLS error from refused
case "$rc" in
35|51|58|59|60|77|83) kind="tls" ;;
*) kind="timeout" ;;
esac
# Retry once at longer timeout (25s connect, 30s max)
if [ -n "$ssh_host" ]; then
code=$(ssh -o ConnectTimeout=5 -o BatchMode=yes "root@${ssh_host}" \
"curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 --max-time 30 ${use_k:+-k} ${url}" 2>/dev/null)
rc=$?
code=$(printf '%s' "$code" | tr -d '[:space:]')
else
code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 --max-time 30 ${use_k:+-k} "$url" 2>/dev/null)
rc=$?
code=$(printf '%s' "$code" | tr -d '[:space:]')
# On retry, classify: still 000 = keep existing kind (or timeout if empty), unexpected status = refused
if [ -z "$code" ] || [ "$code" = "000" ]; then
[ -z "$kind" ] && kind="timeout"
elif ! echo "$code" | grep -qE "^(${expected})$"; then
kind="refused"
fi
fi
fi
# Check result
if [ -n "$code" ] && [ "$code" != "000" ]; then
if [ "$liveness" = "1" ]; then
# Any HTTP status = ALIVE for auth-gated/redirect endpoints
return 0
else
# Bare-200 or specific expected pattern
if echo "$code" | grep -qE "^(${expected})$"; then
return 0
else
kind="unexpected:$code"
LAST_KIND="$kind"
return 1
fi
fi
else
[ -z "$kind" ] && kind="refused"
LAST_KIND="$kind"
return 1
fi
}
# ── Main ────────────────────────────────────────────────────────────────────
FAILED=()
FAILED_KIND=()
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
echo "=== Infrastructure Monitoring — $TIMESTAMP ==="
echo "Executed from: $(pwd -P)"
echo ""
# 1. Grafana (CT 116 :3001 /api/health) — bare-200
if probe_http "$GRAFANA_HOST" "$GRAFANA_PORT" "$GRAFANA_PATH" "$GRAFANA_EXPECTED"; then
echo " ✅ Grafana: alive"
else
echo " 🔴 Grafana: probe-failed: ${GRAFANA_HOST}:${GRAFANA_PORT} (expected ${GRAFANA_EXPECTED}) (<${LAST_KIND}>)"
FAILED+=("grafana")
fi
# 2. Prometheus (CT 116 :9090 /-/healthy) — bare-200
if probe_http "$PROMETHEUS_HOST" "$PROMETHEUS_PORT" "$PROMETHEUS_PATH" "$PROMETHEUS_EXPECTED"; then
echo " ✅ Prometheus: alive"
else
echo " 🔴 Prometheus: probe-failed: ${PROMETHEUS_HOST}:${PROMETHEUS_PORT} (expected ${PROMETHEUS_EXPECTED}) (<${LAST_KIND}>)"
FAILED+=("prometheus")
fi
# 3. LiteLLM (CT 116 :80/litellm/health via nginx) — liveness (any HTTP = alive)
if probe_http "$LITELLM_HOST" "$LITELLM_PORT" "$LITELLM_PATH" "" "" "" "http" "$LITELLM_LIVENESS"; then
echo " ✅ LiteLLM: alive"
else
echo " 🔴 LiteLLM: probe-failed: ${LITELLM_HOST}:${LITELLM_PORT}${LITELLM_PATH} (any-HTTP liveness) (<${LAST_KIND}>)"
FAILED+=("litellm")
fi
# 4. PVE API (5 real nodes :8006 /api2/json/version, -k, liveness)
PVE_FAILED=()
for node in "${PVE_NODES[@]}"; do
if probe_http "$node" "$PVE_API_PORT" "$PVE_API_PATH" "" "$PVE_API_USE_K" "" "https" "$PVE_API_LIVENESS"; then
echo " ✅ PVE API ${node}: alive"
else
echo " 🔴 PVE API ${node}: probe-failed: ${node}:${PVE_API_PORT} (any-HTTP liveness, -k for self-signed) (<${LAST_KIND}>)"
PVE_FAILED+=("$node")
fi
done
if [ ${#PVE_FAILED[@]} -gt 0 ]; then
FAILED+=("pve-api: ${PVE_FAILED[*]}")
fi
# 5. GPU exporters (:9400/metrics) — bare-200
GPU_FAILED=()
for host in "${GPU_HOSTS[@]}"; do
if probe_http "$host" "$GPU_PORT" "$GPU_PATH" "$GPU_EXPECTED"; then
echo " ✅ GPU exporter ${host}: alive"
else
echo " 🔴 GPU exporter ${host}: probe-failed: ${host}:${GPU_PORT} (expected 200) (<${LAST_KIND}>)"
GPU_FAILED+=("$host")
fi
done
if [ ${#GPU_FAILED[@]} -gt 0 ]; then
FAILED+=("gpu-exporters: ${GPU_FAILED[*]}")
fi
# 6. Docker Stats (CT 116 :9324, 127.0.0.1 via SSH) — 200|404
if probe_http "127.0.0.1" "$DOCKER_STATS_PORT" "/" "$DOCKER_STATS_EXPECTED" "" "$CT116_SSH_HOST"; then
echo " ✅ Docker Stats: alive"
else
echo " 🔴 Docker Stats: probe-failed: CT116:127.0.0.1:${DOCKER_STATS_PORT} (expected 200|404) (<${LAST_KIND}>)"
FAILED+=("docker-stats")
fi
# 7. PVE Exporter (CT 116 :9221, 127.0.0.1 via SSH) — 200|404
if probe_http "127.0.0.1" "$PVE_EXPORTER_PORT" "/" "$PVE_EXPORTER_EXPECTED" "" "$CT116_SSH_HOST"; then
echo " ✅ PVE Exporter: alive"
else
echo " 🔴 PVE Exporter: probe-failed: CT116:127.0.0.1:${PVE_EXPORTER_PORT} (expected 200|404) (<${LAST_KIND}>)"
FAILED+=("pve-exporter")
fi
# ── Summary ─────────────────────────────────────────────────────────────────
echo ""
if [ ${#FAILED[@]} -eq 0 ]; then
echo " ✅ All legs OK"
exit 0
else
for f in "${FAILED[@]}"; do
echo " 🔴 FAILED: $f"
done
exit 1
fi
+13 -67
View File
@@ -55,12 +55,9 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
try:
rc, stdout, stderr = run_command(cmd, timeout)
if rc != 0:
# Check if this is a timeout from run_command (rc=1, stderr="TIMEOUT")
if rc == 1 and stderr == "TIMEOUT":
return (000, "timeout after " + str(timeout) + "s")
# Otherwise, determine failure kind from curl exit code
# Determine failure kind from curl exit code
# curl exit codes: 28=timeout, 7=refused, 6=dns, 35=ssl, 52=empty
elif rc == 28:
if rc == 28:
return (000, "timeout after " + str(timeout) + "s")
elif rc == 7:
return (000, "connection refused")
@@ -76,20 +73,6 @@ def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, foll
except subprocess.TimeoutExpired:
return (000, "timeout after " + str(timeout) + "s")
def check_host_health(host_ip):
"""Check if the GPU host's llama-chat-api health endpoint is reachable
Returns: (healthy: bool, detail: str)
"""
code, _ = probe_http("http://" + host_ip + ":8080/health", timeout=10)
if code == 200:
return True, "host healthy (200)"
elif code == 000:
return False, "host unreachable (timeout or refused)"
else:
return False, "host unhealthy (HTTP " + str(code) + ")"
def get_response_body(url, method="POST", bearer_token=None, data=None, timeout=30):
"""Get response body for 401/403 credential faults (truncated to 200 chars)"""
cmd = "curl -s -m " + str(timeout)
@@ -132,53 +115,29 @@ def check_model_probes():
results = []
# Host health mapping: model -> host IP
model_hosts = {
"gpu-dense": "192.168.68.8", # RTX 3090
"gpu-vision": "192.168.68.110", # RTX 5070
"strix-moe": "192.168.68.15" # Strix Halo
}
for model in ["gpu-dense", "gpu-vision", "strix-moe"]:
host_ip = model_hosts[model]
# Single-host aliases: 30s initial timeout, retry once at 90s on failure
# Worst-case prefill ~76s, so 90s retry ensures we cover it
# Single-host aliases: 30s timeout each
# gpu-dense (RTX 3090) may need long warmup/prefill - timeout is acceptable on cold-start
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
method="POST",
bearer_token=monitor_key,
data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
timeout=30)
first_kind = None
if code == 000 and failure_kind:
first_kind = failure_kind
time.sleep(1)
code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
method="POST",
bearer_token=monitor_key,
data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}',
timeout=90)
if code == 000 and failure_kind:
# Both attempts failed - check host health to distinguish busy from down
host_healthy, host_detail = check_host_health(host_ip)
if host_healthy:
results.append((model, False, "busy (completion timed out after retry; " + host_detail + ")"))
else:
# Host unreachable - report both kinds
if first_kind:
results.append((model, False, "probe-failed: " + model + " " + first_kind + " then " + failure_kind + " (2 attempts)"))
else:
results.append((model, False, "probe-failed: " + model + " " + failure_kind))
# Report probe failure with kind, do not assert a service verdict
results.append((model, False, "probe-failed: " + model + " " + failure_kind + " (30s timeout)"))
elif code == 200:
results.append((model, True, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
elif code in (401, 403):
# Credential fault - capture body and key alias
body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions",
method="POST",
bearer_token=monitor_key,
data='{"model":"' + model + '","messages":[{"role":"user","content":"health"}],"max_tokens":4}',
timeout=10)
alias = "monitor-20260813"
# Resolve key alias
alias = "monitor-20260813" # Known from /etc/litellm-monitor.env on CT 116
results.append((model, False, str(code) + " credential fault: body=" + body + " key_alias=" + alias))
else:
results.append((model, False, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")"))
@@ -288,7 +247,6 @@ def main():
print("")
all_pass = True
degraded = [] # Track degraded (busy) checks
# Run all checks
checks = [
@@ -308,15 +266,9 @@ def main():
# Model probes
model_results = check_model_probes()
for name, passed, detail in model_results:
# Check if this is a busy (degraded) verdict
if not passed and detail.startswith("busy "):
status = "⚠️"
degraded.append(name)
else:
status = "✅" if passed else "❌"
status = "✅" if passed else "❌"
print(" " + status + " " + name + ": " + detail)
# Only set all_pass=False for real failures (not busy)
if not passed and not detail.startswith("busy "):
if not passed:
all_pass = False
# Admin key list
@@ -342,16 +294,10 @@ def main():
print("")
if all_pass:
if degraded:
print("✅ All checks passed (" + str(len(degraded)) + " degraded: " + ", ".join(degraded) + ")")
else:
print("✅ All checks passed")
print("✅ All checks passed")
return 0
else:
if degraded:
print("❌ Some checks failed (" + str(len(degraded)) + " degraded: " + ", ".join(degraded) + ")")
else:
print("❌ Some checks failed")
print("❌ Some checks failed")
return 1
if __name__ == "__main__":
-138
View File
@@ -1,138 +0,0 @@
#!/bin/bash
# proxmox-monitor.sh — Proxmox Cluster + Docker Monitoring Health Check
# Implements proxmox-monitor.prose.md (check-health section)
#
# Legs: Prometheus, Grafana, Docker Stats Exporter, PVE Exporter, PBS GC
# All legs must return 200 for healthy status.
#
# Run: bash scripts/proxmox-monitor.sh
# Exits 0 if all probes pass, 1 if any fails.
set -uo pipefail
CT116_HOST="192.168.68.116"
FAILED=()
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
echo "=== Proxmox Monitor — $TIMESTAMP ==="
echo "Executed from: $(pwd -P)"
echo ""
# 1. Prometheus health (bound to 0.0.0.0:9090 on .116)
PROM_CODE=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://${CT116_HOST}:9090/-/healthy 2>/dev/null)
PROM_CODE=$(printf '%s' "$PROM_CODE" | tr -d '[:space:]')
[ -n "$PROM_CODE" ] || PROM_CODE="000"
if [ "$PROM_CODE" = "200" ]; then
echo " ✅ Prometheus: alive"
else
echo " 🔴 Prometheus: probe-failed: ${CT116_HOST}:9090 (expected 200, got ${PROM_CODE})"
FAILED+=("prometheus")
fi
# 2. Grafana health (bound to 0.0.0.0:3001 on .116)
GRAF_CODE=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://${CT116_HOST}:3001/api/health 2>/dev/null)
GRAF_CODE=$(printf '%s' "$GRAF_CODE" | tr -d '[:space:]')
[ -n "$GRAF_CODE" ] || GRAF_CODE="000"
if [ "$GRAF_CODE" = "200" ]; then
echo " ✅ Grafana: alive"
else
echo " 🔴 Grafana: probe-failed: ${CT116_HOST}:3001 (expected 200, got ${GRAF_CODE})"
FAILED+=("grafana")
fi
# 3. Docker Stats exporter (bound to 127.0.0.1:9324 on .116 — probe from .116 localhost)
DOCKER_CODE=$(ssh -o ConnectTimeout=5 -o BatchMode=yes root@${CT116_HOST} \
"curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9324/metrics" 2>/dev/null)
DOCKER_CODE=$(printf '%s' "$DOCKER_CODE" | tr -d '[:space:]')
[ -n "$DOCKER_CODE" ] || DOCKER_CODE="000"
if [ "$DOCKER_CODE" = "200" ]; then
echo " ✅ Docker Stats: alive"
else
echo " 🔴 Docker Stats: probe-failed: CT116:127.0.0.1:9324 (expected 200, got ${DOCKER_CODE})"
FAILED+=("docker-stats")
fi
# 4. PVE Exporter (bound to 127.0.0.1:9221 on .116 — probe from .116 localhost)
PVE_CODE=$(ssh -o ConnectTimeout=5 -o BatchMode=yes root@${CT116_HOST} \
"curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9221/metrics" 2>/dev/null)
PVE_CODE=$(printf '%s' "$PVE_CODE" | tr -d '[:space:]')
[ -n "$PVE_CODE" ] || PVE_CODE="000"
if [ "$PVE_CODE" = "200" ]; then
echo " ✅ PVE Exporter: alive"
else
echo " 🔴 PVE Exporter: probe-failed: CT116:127.0.0.1:9221 (expected 200, got ${PVE_CODE})"
FAILED+=("pve-exporter")
fi
# 5. PBS GC liveness (storepve-datastore GC must have run within 48h)
PBS_GC_OUTPUT=$(ssh -o ConnectTimeout=5 -o BatchMode=yes root@192.168.68.6 \
"pct exec 107 -- proxmox-backup-manager garbage-collection list --output-format json" 2>/dev/null)
PBS_GC_OUTPUT=$(printf '%s' "$PBS_GC_OUTPUT" | tr -d '[:space:]')
[ -n "$PBS_GC_OUTPUT" ] || PBS_GC_OUTPUT="000"
if [ "$PBS_GC_OUTPUT" = "000" ]; then
echo " 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (expected JSON, got 000)"
FAILED+=("pbs-gc")
else
# Parse the JSON to get storepve-datastore's last-run-endtime and pending-bytes
PBS_GC_RESULT=$(echo "$PBS_GC_OUTPUT" | python3 -c "
import sys, json
try:
data = json.load(sys.stdin)
for store in data:
if store['store'] == 'storepve-datastore':
endtime = store.get('last-run-endtime')
pending = store.get('pending-bytes', 0)
if endtime is None or endtime == 0:
print('never-run')
else:
print(f'{endtime}|{pending}')
break
else:
print('absent')
except json.JSONDecodeError:
print('unparseable')
" 2>/dev/null)
if [ -z "$PBS_GC_RESULT" ] || [ "$PBS_GC_RESULT" = "unparseable" ]; then
echo " 🔴 PBS GC: probe-failed: storepve:192.168.68.6 (unparseable JSON)"
FAILED+=("pbs-gc")
elif [ "$PBS_GC_RESULT" = "absent" ]; then
echo " 🔴 PBS GC: never-run (storepve-datastore not found in GC list)"
FAILED+=("pbs-gc")
elif [ "$PBS_GC_RESULT" = "never-run" ]; then
echo " 🔴 PBS GC: never-run (storepve-datastore has no last-run-endtime)"
FAILED+=("pbs-gc")
else
# Parse the endtime|pending format
LAST_RUN_ENDTIME=$(echo "$PBS_GC_RESULT" | cut -d'|' -f1)
PENDING_BYTES=$(echo "$PBS_GC_RESULT" | cut -d'|' -f2)
# Convert epoch to age in hours
NOW_EPOCH=$(date -u +%s)
AGE_HOURS=$(( (NOW_EPOCH - LAST_RUN_ENDTIME) / 3600 ))
if [ $AGE_HOURS -gt 48 ]; then
echo " 🔴 PBS GC: stale (last run ${AGE_HOURS}h ago, pending-bytes: ${PENDING_BYTES} B)"
FAILED+=("pbs-gc")
else
echo " ✅ PBS GC: healthy (last run ${AGE_HOURS}h ago, pending-bytes: ${PENDING_BYTES} B)"
fi
fi
fi
# ── Summary ─────────────────────────────────────────────────────────────────
echo ""
if [ ${#FAILED[@]} -eq 0 ]; then
echo " ✅ All legs OK"
exit 0
else
for f in "${FAILED[@]}"; do
echo " 🔴 FAILED: $f"
done
exit 1
fi
-228
View File
@@ -1,228 +0,0 @@
#!/bin/bash
# test_infra_monitoring.sh — Asserts that probe calls use the documented targets.
#
# Strategy: stub curl and ssh on PATH to capture the exact arguments each leg
# builds, then assert the URL/port of every call. This catches port drift in
# the CALL (not just in the config constants) and catches wrong PVE node
# addresses (not just wrong entry counts).
#
# Run: bash scripts/test_infra_monitoring.sh
# Exits 0 if all assertions pass, 1 otherwise.
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SCRIPT="${SCRIPT_DIR}/infra-monitoring.sh"
PASS=0
FAIL=0
assert() {
local desc="$1" condition="$2"
if eval "$condition"; then
echo " ✅ $desc"
PASS=$((PASS+1))
else
echo " 🔴 $desc"
FAIL=$((FAIL+1))
fi
}
echo "=== test_infra_monitoring.sh ==="
echo ""
# ── Stub curl: capture argv to a file, return 200 ──────────────────────────
STUB_DIR=$(mktemp -d)
trap 'rm -rf "$STUB_DIR"' EXIT
# Stub curl: first arg after flags is the URL; capture all args
cat > "$STUB_DIR/curl" << 'STUBEOF'
#!/bin/bash
echo "$@" >> "${CURL_STUB_LOG:-/dev/null}"
# Print 200 for %{http_code}
printf '%s\n' "200"
exit 0
STUBEOF
chmod +x "$STUB_DIR/curl"
# Stub ssh: first arg after options is the remote command; capture it
cat > "$STUB_DIR/ssh" << 'SSTUBEOF'
#!/bin/bash
echo "SSH $@" >> "${SSH_STUB_LOG:-/dev/null}"
# The last arg is the remote command — extract and log curl args
for arg in "$@"; do
if [[ "$arg" == curl* ]]; then
echo "$arg" >> "${SSH_STUB_LOG:-/dev/null}"
fi
done
printf '%s\n' "200"
exit 0
SSTUBEOF
chmod +x "$STUB_DIR/ssh"
# ── Run the monitor with stubs ────────────────────────────────────────────
CURL_LOG="$STUB_DIR/curl_calls.log"
SSH_LOG="$STUB_DIR/ssh_calls.log"
touch "$CURL_LOG" "$SSH_LOG"
CURL_STUB_LOG="$CURL_LOG" SSH_STUB_LOG="$SSH_LOG" \
PATH="$STUB_DIR:$PATH" bash "$SCRIPT" > "$STUB_DIR/output.txt" 2>&1
# ── 1. Port drift detection (from actual curl invocations) ─────────────────
assert "Grafana probed at port 3001" \
'grep -q "http://192.168.68.116:3001/api/health" "$CURL_LOG"'
assert "Prometheus probed at port 9090" \
'grep -q "http://192.168.68.116:9090/-/healthy" "$CURL_LOG"'
assert "LiteLLM probed via nginx at port 80" \
'grep -q "http://192.168.68.116:80/litellm/health" "$CURL_LOG"'
assert "PVE API probed at port 8006" \
'grep -q ":8006/api2/json/version" "$CURL_LOG"'
assert "GPU exporter probed at port 9400" \
'grep -q ":9400/metrics" "$CURL_LOG"'
# ── 2. PVE API: exact node addresses (catches wrong IPs) ──────────────────
# Each real PVE node must be probed; CT 116 must NOT be in the PVE set
assert "PVE acerpve 192.168.68.9 probed" \
'grep -q "https://192.168.68.9:8006/api2/json/version" "$CURL_LOG"'
assert "PVE minipve 192.168.68.12 probed" \
'grep -q "https://192.168.68.12:8006/api2/json/version" "$CURL_LOG"'
assert "PVE storepve 192.168.68.6 probed" \
'grep -q "https://192.168.68.6:8006/api2/json/version" "$CURL_LOG"'
assert "PVE amdpve 192.168.68.15 probed" \
'grep -q "https://192.168.68.15:8006/api2/json/version" "$CURL_LOG"'
assert "PVE ocupve 192.168.68.5 probed" \
'grep -q "https://192.168.68.5:8006/api2/json/version" "$CURL_LOG"'
# CT 116 (.116) must NOT appear as a PVE API target
assert "CT 116 (.116) NOT probed as PVE API node" \
'! grep -q "https://192.168.68.116:8006" "$CURL_LOG"'
# ── 3. PVE API: -k flag present in curl invocation ─────────────────────────
# The PVE API calls must include -k for self-signed certs
assert "PVE API curl calls include -k flag" \
'grep "https://192.168.68.9:8006" "$CURL_LOG" | grep -q -- "-k"'
# ── 4. Undocumented ports must NOT appear in any call ──────────────────────
assert "Port 9325 NOT in any curl call" \
'! grep -q ":9325" "$CURL_LOG"'
assert "Port 9405 NOT in any curl call" \
'! grep -q ":9405" "$CURL_LOG"'
# ── 5. Docker Stats / PVE Exporter: SSH-probed at correct ports ────────────
assert "Docker Stats probed at port 9324 via SSH" \
'grep -q "9324" "$SSH_LOG"'
assert "PVE Exporter probed at port 9221 via SSH" \
'grep -q "9221" "$SSH_LOG"'
# ── 5a. Per-leg assertions (proves which leg owns which port) ──────────────
# The script source must show Docker Stats using $DOCKER_STATS_PORT and
# PVE Exporter using $PVE_EXPORTER_PORT in the correct leg sections
assert "Docker Stats leg uses DOCKER_STATS_PORT constant" \
'grep -A 3 "# 6. Docker Stats" "$SCRIPT" | grep -q "\$DOCKER_STATS_PORT"'
assert "PVE Exporter leg uses PVE_EXPORTER_PORT constant" \
'grep -A 3 "# 7. PVE Exporter" "$SCRIPT" | grep -q "\$PVE_EXPORTER_PORT"'
# Verify the constants themselves are set to the correct values
assert "DOCKER_STATS_PORT constant set to 9324" \
'grep -q "^DOCKER_STATS_PORT=\"9324\"" "$SCRIPT"'
assert "PVE_EXPORTER_PORT constant set to 9221" \
'grep -q "^PVE_EXPORTER_PORT=\"9221\"" "$SCRIPT"'
# ── 5b. Stale port 9323 (dockerd) NOT probed ──────────────────────────────
assert "Port 9323 (dockerd) NOT in SSH log" \
'! grep -q "9323" "$SSH_LOG"'
# ── 6. No stale ports in the script source (belt-and-suspenders) ──────────
assert "Port 9325 (historical) NOT in script source" \
'! grep -q "9325" "$SCRIPT"'
assert "Port 9405 (historical) NOT in script source" \
'! grep -q "9405" "$SCRIPT"'
# ── 7. Failure-line content includes non-empty kind ────────────────────────
TMP_DIR=$(mktemp -d)
trap 'rm -rf "$TMP_DIR"' EXIT
# Test 7a: Unexpected status (500) → kind should be unexpected:500
cat > "$TMP_DIR/curl" << 'EOF'
#!/bin/bash
# Stub: return 500 for Grafana port, 200 otherwise
for arg in "$@"; do
if [[ "$arg" == *":3001"* ]]; then
echo "500"
exit 0
fi
done
echo "200"
exit 0
EOF
chmod +x "$TMP_DIR/curl"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
GRAFANA_FAIL=$(echo "$OUT" | grep "Grafana: probe-failed")
assert "Grafana failure line exists (unexpected status)" \
'[[ -n "$GRAFANA_FAIL" ]]'
KIND=$(echo "$GRAFANA_FAIL" | grep -oP '\(<[^>]+>\)' | tr -d '()<>')
assert "Grafana failure kind is non-empty (unexpected status)" \
'[[ -n "$KIND" ]]'
# Test 7b: TLS error (000 + exit 60) → kind should be tls
cat > "$TMP_DIR/curl" << 'EOF'
#!/bin/bash
# Stub: return 000 and exit 60 for Grafana port (TLS error) on ALL invocations
for arg in "$@"; do
if [[ "$arg" == *":3001"* ]]; then
echo "000"
exit 60
fi
done
echo "200"
exit 0
EOF
chmod +x "$TMP_DIR/curl"
# Also stub ssh to return 000 + exit 60 for the retry
cat > "$TMP_DIR/ssh" << 'EOF'
#!/bin/bash
for arg in "$@"; do
if [[ "$arg" == curl* ]]; then
echo "000"
exit 60
fi
done
echo "200"
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
export PATH="$TMP_DIR:$PATH"
OUT=$(bash "$SCRIPT" 2>&1)
GRAFANA_FAIL=$(echo "$OUT" | grep "Grafana: probe-failed")
assert "Grafana failure line exists (TLS error)" \
'[[ -n "$GRAFANA_FAIL" ]]'
KIND=$(echo "$GRAFANA_FAIL" | grep -oP '\(<[^>]+>\)' | tr -d '()<>')
assert "Grafana failure kind is tls" \
'[[ "$KIND" == "tls" ]]'
# ── Summary ─────────────────────────────────────────────────────────────────
echo ""
echo "Results: ${PASS} passed, ${FAIL} failed"
if [ $FAIL -gt 0 ]; then
echo " 🔴 TESTS FAILED"
exit 1
else
echo " ✅ ALL TESTS PASSED"
exit 0
fi
-125
View File
@@ -1,125 +0,0 @@
#!/bin/bash
# test_proxmox_monitor.sh — Stub-driven tests for PBS GC liveness leg
set -uo pipefail
SCRIPT="$(cd "$(dirname "$0")" && pwd)/proxmox-monitor.sh"
PASS=0
FAIL=0
# ── Helpers ────────────────────────────────────────────────────────────────
assert() {
local desc="$1" cond="$2"
if eval "$cond" 2>/dev/null; then
echo " ✅ $desc"
PASS=$((PASS+1))
else
echo " 🔴 $desc"
FAIL=$((FAIL+1))
fi
}
# ── 1. Healthy: fresh GC, 0 B pending ─────────────────────────────────────
TMP_DIR=$(mktemp -d)
FRESH_ENDTIME=$(( $(date -u +%s) - (1 * 3600) )) # 1 hour ago
cat > "$TMP_DIR/ssh" << EOF
#!/bin/bash
# Stub: return valid JSON with fresh endtime
echo '[{"store":"storepve-datastore","last-run-endtime":$FRESH_ENDTIME,"pending-bytes":0}]'
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
PBS_LINE=$(echo "$OUT" | grep "PBS GC:" || echo "")
assert "Healthy: PBS line exists" '[[ -n "$PBS_LINE" ]]'
assert "Healthy: shows healthy verdict" '[[ "$PBS_LINE" == *"healthy"* ]]'
assert "Healthy: shows pending-bytes 0 B" '[[ "$PBS_LINE" == *"pending-bytes: 0 B"* ]]'
rm -rf "$TMP_DIR"
# ── 2. Stale: GC >48h old ─────────────────────────────────────────────────
TMP_DIR=$(mktemp -d)
OLD_ENDTIME=$(( $(date -u +%s) - (49 * 3600) )) # 49 hours ago
cat > "$TMP_DIR/ssh" << EOF
#!/bin/bash
echo '[{"store":"storepve-datastore","last-run-endtime":$OLD_ENDTIME,"pending-bytes":1048576}]'
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
PBS_LINE=$(echo "$OUT" | grep "PBS GC:" || echo "")
assert "Stale: PBS line exists" '[[ -n "$PBS_LINE" ]]'
assert "Stale: shows stale verdict" '[[ "$PBS_LINE" == *"stale"* ]]'
assert "Stale: shows pending-bytes 1048576 B" '[[ "$PBS_LINE" == *"pending-bytes: 1048576 B"* ]]'
rm -rf "$TMP_DIR"
# ── 3. Probe-failed: empty output ─────────────────────────────────────────
TMP_DIR=$(mktemp -d)
cat > "$TMP_DIR/ssh" << 'EOF'
#!/bin/bash
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
PBS_LINE=$(echo "$OUT" | grep "PBS GC:" || echo "")
assert "Probe-failed (empty): PBS line exists" '[[ -n "$PBS_LINE" ]]'
assert "Probe-failed (empty): shows probe-failed" '[[ "$PBS_LINE" == *"probe-failed"* ]]'
rm -rf "$TMP_DIR"
# ── 4. Probe-failed: unparseable output ───────────────────────────────────
TMP_DIR=$(mktemp -d)
cat > "$TMP_DIR/ssh" << 'EOF'
#!/bin/bash
echo "proxmox-backup-manager: command not found"
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
PBS_LINE=$(echo "$OUT" | grep "PBS GC:" || echo "")
assert "Probe-failed (unparseable): PBS line exists" '[[ -n "$PBS_LINE" ]]'
assert "Probe-failed (unparseable): shows probe-failed" '[[ "$PBS_LINE" == *"probe-failed"* ]]'
rm -rf "$TMP_DIR"
# ── 5. Null endtime: never-run ────────────────────────────────────────────
TMP_DIR=$(mktemp -d)
cat > "$TMP_DIR/ssh" << 'EOF'
#!/bin/bash
echo '[{"store":"storepve-datastore","last-run-endtime":null,"pending-bytes":0}]'
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
PBS_LINE=$(echo "$OUT" | grep "PBS GC:" || echo "")
assert "Null endtime: PBS line exists" '[[ -n "$PBS_LINE" ]]'
assert "Null endtime: shows never-run" '[[ "$PBS_LINE" == *"never-run"* ]]'
rm -rf "$TMP_DIR"
# ── 6. Datastore absent ───────────────────────────────────────────────────
TMP_DIR=$(mktemp -d)
cat > "$TMP_DIR/ssh" << 'EOF'
#!/bin/bash
echo '[{"store":"s3-archive","last-run-endtime":null,"pending-bytes":0}]'
exit 0
EOF
chmod +x "$TMP_DIR/ssh"
OUT=$(PATH="$TMP_DIR:$PATH" bash "$SCRIPT" 2>&1)
PBS_LINE=$(echo "$OUT" | grep "PBS GC:" || echo "")
assert "Datastore absent: PBS line exists" '[[ -n "$PBS_LINE" ]]'
assert "Datastore absent: shows never-run" '[[ "$PBS_LINE" == *"never-run"* ]]'
rm -rf "$TMP_DIR"
# ── Summary ────────────────────────────────────────────────────────────────
echo ""
echo "Results: ${PASS} passed, ${FAIL} failed"
if [ $FAIL -gt 0 ]; then
echo " 🔴 TESTS FAILED"
exit 1
else
echo " ✅ ALL TESTS PASSED"
exit 0
fi
+21 -71
View File
@@ -8,21 +8,12 @@
set -euo pipefail
# Credentials sourced from environment variable ZULIP_API_KEY (set by vault-backed start script)
# Never fall back to a literal key.
# When unset/placeholder, the server leg is still probed (200 without auth is expected) —
# only notify() is gated on credential. The pi/Tanko/kagentz
# legs do not need the Zulip API key. The placeholder is captain-held:
# zulip-health-credential-placeholder-20260913.
ZULIP_API_KEY="${ZULIP_API_KEY:-}"
# Never fall back to a literal key
ZULIP_API_KEY="${ZULIP_API_KEY:?ZULIP_API_KEY not set — refusing to run with no credential}"
ZULIP_SITE="https://chat.sysloggh.net"
ZULIP_EMAIL="abiba-bot@chat.sysloggh.net"
OWNER_ZULIP_ID="9"
# Track whether the Zulip API credential is usable
ZULIP_CRED_OK=1
if [ -z "$ZULIP_API_KEY" ] || [[ "$ZULIP_API_KEY" == *"placeholder"* ]] || [[ "$ZULIP_API_KEY" == *"REDACTED"* ]]; then
ZULIP_CRED_OK=0
fi
LOG="/root/zulip-health-monitor.log"
TIMESTAMP=$(date -u '+%Y-%m-%d %H:%M UTC')
@@ -33,29 +24,23 @@ notify() {
local severity="$1" msg="$2"
echo "[$severity] $msg"
# Zulip DM to owner (skip if no credential)
if [ "$ZULIP_CRED_OK" -eq 1 ]; then
local content="${severity} Zulip Monitor: ${msg}"
local form
form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
-d "${form}" > /dev/null 2>&1 || true
# Zulip stream post to #agent-hub on topic 'zulip-health'
local stream_content="${severity} Zulip Monitor: ${msg}"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
-d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
> /dev/null 2>&1 \
|| echo " WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)">> "$LOG"
else
echo " ALERT SUPPRESSED (no credential): ${severity} ${msg}" >> "$LOG"
fi
# Zulip DM to owner
local content="${severity} Zulip Monitor: ${msg}"
local form
form="type=private&to=%5B${OWNER_ZULIP_ID}%5D&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote('''${content}'''))")"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
-d "${form}" > /dev/null 2>&1 || true
# Zulip stream post to #agent-hub on topic 'zulip-health'
local stream_content="${severity} Zulip Monitor: ${msg}"
curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
-u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" \
-d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
> /dev/null 2>&1 \
|| echo " WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)" >> "$LOG"
}
# ── Global: Zulip Server ──
# F3: Always probe server regardless of credential — 200 without auth is expected
# (verified live: server_settings returns 200 with no credential or wrong key).
SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \
https://chat.sysloggh.net/api/v1/server_settings \
-u "${ZULIP_EMAIL}:${ZULIP_API_KEY}" 2>/dev/null) || SERVER_CODE="000"
@@ -176,11 +161,6 @@ fi
# never contact her former host.
# ── Platform C: Agent Zero (kagentz) ──
# C1: A2A liveness (no credential needed) — probes the container's internal :80/a2a/
# C2: A2A response verification (needs LITELLM_KEY) — probes POST /a2a with auth
# C3: Public access path (no credential needed) — probes https://kagentz.sysloggh.net/
# C1: A2A liveness (container-internal probe)
AZ_A2A_CODE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \
"docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/ 2>/dev/null" 2>/dev/null) || AZ_A2A_CODE="000"
AZ_A2A_CODE=$(printf '%s' "$AZ_A2A_CODE" | tr -d '[:space:]')
@@ -189,53 +169,23 @@ AZ_A2A_CODE=$(printf '%s' "$AZ_A2A_CODE" | tr -d '[:space:]')
if [ "$AZ_A2A_CODE" = "000" ]; then
notify "🔴" "kagentz A2A server DOWN (connection failed)"
ISSUES=$((ISSUES + 1))
echo " kagentz C1: ❌ A2A down (HTTP 000)" >> "$LOG"
echo " kagentz: ❌ A2A down (HTTP 000)" >> "$LOG"
else
case "$AZ_A2A_CODE" in
200|401)
echo " kagentz C1: ✅ A2A alive (HTTP $AZ_A2A_CODE)" >> "$LOG" ;;
echo " kagentz: ✅ A2A alive (HTTP $AZ_A2A_CODE)" >> "$LOG" ;;
*)
notify "🟡" "kagentz A2A server answered HTTP $AZ_A2A_CODE — running, unexpected status"
ISSUES=$((ISSUES + 1))
echo " kagentz C1: 🟡 A2A unexpected http=$AZ_A2A_CODE (running, warning)" >> "$LOG" ;;
esac
fi
# C3: Public access path (the captain's point of view)
# Probes the public URL that NetBird proxies to the container. 200/302/401 = alive,
# 502 = proxy's "upstream refused" page (incident), connection failed = incident.
# Never restarts anything — the contract forbids restarting the platform.
KAGENTZ_PUBLIC_CODE=$(curl -s -o /dev/null --connect-timeout 10 --max-time 15 \
-w '%{http_code}' https://kagentz.sysloggh.net/ 2>/dev/null) || KAGENTZ_PUBLIC_CODE="000"
KAGENTZ_PUBLIC_CODE=$(printf '%s' "$KAGENTZ_PUBLIC_CODE" | tr -d '[:space:]')
[ -n "$KAGENTZ_PUBLIC_CODE" ] || KAGENTZ_PUBLIC_CODE="000"
if [ "$KAGENTZ_PUBLIC_CODE" = "000" ]; then
notify "🔴" "kagentz public URL DOWN (connection failed)"
ISSUES=$((ISSUES + 1))
echo " kagentz C3: ❌ public URL down (HTTP 000)" >> "$LOG"
elif [ "$KAGENTZ_PUBLIC_CODE" = "502" ]; then
notify "🔴" "kagentz public URL 502 (upstream refused)"
ISSUES=$((ISSUES + 1))
echo " kagentz C3: ❌ public URL 502 (upstream refused)" >> "$LOG"
else
case "$KAGENTZ_PUBLIC_CODE" in
200|302|401)
echo " kagentz C3: ✅ public URL alive (HTTP $KAGENTZ_PUBLIC_CODE)" >> "$LOG" ;;
*)
notify "🟡" "kagentz public URL answered HTTP $KAGENTZ_PUBLIC_CODE — running, unexpected status"
ISSUES=$((ISSUES + 1))
echo " kagentz C3: 🟡 public URL unexpected http=$KAGENTZ_PUBLIC_CODE (running, warning)" >> "$LOG" ;;
echo " kagentz: 🟡 A2A unexpected http=$AZ_A2A_CODE (running, warning)" >> "$LOG" ;;
esac
fi
# ── Summary ──
# The run verdict is non-optimistic: when ISSUES > 0, the run is an INCIDENT.
# The lane must quote this Result line verbatim in its status report.
if [ "$ISSUES" -eq 0 ]; then
echo " Result: ✅ 0 issues (all healthy)" >> "$LOG"
echo " Result: ✅ All healthy" >> "$LOG"
else
echo " Result: 🔴 INCIDENT — $ISSUES issue(s) found" >> "$LOG"
echo " Result: 🔴 $ISSUES issue(s) found" >> "$LOG"
notify "🔴" "$ISSUES issue(s) found — check /root/zulip-health-monitor.log"
fi
+2 -72
View File
@@ -24,7 +24,7 @@ BASE = """
model:
api_key: ""
api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
max_tokens: 4096
default: syslog-auto
provider: harness
@@ -52,7 +52,7 @@ delegation:
custom_providers:
- name: harness
key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/litellm/v1
base_url: http://192.168.68.116/v1
"""
@@ -189,73 +189,3 @@ def test_retired_alias_in_nested_auxiliary_block_is_rejected(tmp_path):
assert code == 1, out
assert "auxiliary.tasks.summarize.model = 'gpu-light'" in out
assert "RESULT: FAIL" in out
def test_canonical_internal_path_passes(tmp_path):
"""Rule 5 must accept the canonical internal base from hermes-key-enforcement.prose.md."""
code, out = _run(
tmp_path,
"gpu-vision",
)
# Override the base_url in the config
cfg_text = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/v1",
" base_url: http://192.168.68.116/litellm/v1",
)
code, out = _run_config(tmp_path, "canonical-internal.yaml", cfg_text)
assert code == 0, out
assert "RESULT: PASS" in out
# Verify the correct message is shown
assert "model.base_url is canonical" in out
def test_wrong_base_url_fails(tmp_path):
"""Rule 5 must reject paths outside the allowed list."""
cfg_text = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/litellm/v1",
" base_url: http://192.168.68.116/litellm/v1/responses",
)
code, out = _run_config(tmp_path, "wrong-base.yaml", cfg_text)
assert code == 1, out
assert "RESULT: FAIL" in out
assert "model.base_url must be one of" in out
def test_public_host_path_passes(tmp_path):
"""Rule 5 must accept the public host base."""
cfg_text = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/v1",
" base_url: https://litellm.sysloggh.net/v1",
)
code, out = _run_config(tmp_path, "public-host.yaml", cfg_text)
assert code == 0, out
assert "RESULT: PASS" in out
def test_old_rule5_check_would_fail_canonical(tmp_path):
"""
Proof that the OLD Rule 5 check would fail the canonical internal path.
This proves the bug existed before the fix.
"""
# OLD check expected /v1, so the canonical /litellm/v1 would have failed
canonical_cfg = BASE.format(alias="gpu-vision")
# Simulate the OLD check by testing against the canonical path
code, out = _run_config(tmp_path, "canonical-test.yaml", canonical_cfg)
# NEW check: canonical /litellm/v1 SHOULD pass
assert code == 0, out
assert "RESULT: PASS" in out
# OLD check expected /v1, so the internal /v1 would have passed
# NEW check: internal /v1 is non-canonical but working (WARN not FAIL)
old_cfg = BASE.format(alias="gpu-vision").replace(
" base_url: http://192.168.68.116/litellm/v1",
" base_url: http://192.168.68.116/v1",
)
code, out = _run_config(tmp_path, "old-check-test.yaml", old_cfg)
assert code == 0, out
assert "RESULT: PASS" in out
# NEW check should pass
new_cfg = BASE.format(alias="gpu-vision")
code, out = _run_config(tmp_path, "new-check-test.yaml", new_cfg)
assert code == 0, out
assert "RESULT: PASS" in out
+13 -18
View File
@@ -98,8 +98,8 @@ exit 0
"""
CURL_STUB = r"""#!/usr/bin/env bash
# Stub curl: serve the Abiba health fixture, the Zulip server 200, and the
# kagentz C3 public URL, and record every call (including notify) payloads.
# Stub curl: serve the Abiba health fixture and the Zulip server 200, and
# record every call (including notify) payloads.
printf '%s\n' "$*" >> "$RECORD_DIR/curl.calls"
case "$*" in
*:9200/health*)
@@ -109,8 +109,6 @@ case "$*" in
esac ;;
*server_settings*)
printf '%s' "$SERVER_HTTP" ;;
*kagentz.sysloggh.net*)
printf '%s' "$KAGENTZ_PUBLIC_CODE" ;;
esac
exit 0
"""
@@ -123,8 +121,7 @@ def _write_exec(path: pathlib.Path, body: str) -> None:
def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
az_a2a_code="401", az_a2a_exit=0,
kagentz_public_code="302"):
az_a2a_code="401", az_a2a_exit=0):
"""Run the shipped monitor in a sandbox; return (proc, record_dir, log_path).
Only the LOG constant is rewritten (to keep the run inside the worktree).
@@ -157,7 +154,6 @@ def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
"PI_HTTP": "200",
"PI_BODY": CONNECTED_FIXTURE.read_text(),
"SERVER_HTTP": "200",
"KAGENTZ_PUBLIC_CODE": kagentz_public_code,
})
proc = subprocess.run(["bash", str(script)], cwd=sandbox, env=env,
capture_output=True, text=True)
@@ -173,9 +169,8 @@ def test_healthy_run_is_quiet_and_never_reaches_mumuni(tmp_path):
assert "Server: ✅ HTTP 200" in log
assert "Abiba: ✅ Connected" in log
assert "Tanko: ✅ service=active http=200" in log
assert "kagentz C1: ✅ A2A alive (HTTP 401)" in log
assert "kagentz C3: ✅ public URL alive (HTTP 302)" in log
assert "Result: ✅ 0 issues (all healthy)" in log
assert "kagentz: ✅ A2A alive (HTTP 401)" in log
assert "Result: ✅ All healthy" in log
# A healthy run emits no notify at all — and certainly no Mumuni one.
assert proc.stdout == ""
@@ -212,8 +207,8 @@ def test_failing_run_alerts_on_tanko_but_never_on_mumuni(tmp_path):
# The rest of the monitor still ran alongside the failing Tanko leg.
log = log_path.read_text()
assert "Abiba: ✅ Connected" in log
assert "kagentz C1: ✅ A2A alive" in log
assert "Result: 🔴 INCIDENT — 1 issue(s) found" in log
assert "kagentz: ✅ A2A alive" in log
assert "Result: 🔴 1 issue(s) found" in log
def test_unexpected_a2a_status_is_an_issue_not_healthy(tmp_path):
@@ -221,9 +216,9 @@ def test_unexpected_a2a_status_is_an_issue_not_healthy(tmp_path):
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
assert "kagentz C1: 🟡 A2A unexpected http=500 (running, warning)" in log
assert "kagentz C1: ✅ A2A alive" not in log
assert "Result: 🔴 INCIDENT — 1 issue(s) found" in log
assert "kagentz: 🟡 A2A unexpected http=500 (running, warning)" in log
assert "kagentz: ✅ A2A alive" not in log
assert "Result: 🔴 1 issue(s) found" in log
assert "kagentz A2A server answered HTTP 500" in proc.stdout
@@ -235,10 +230,10 @@ def test_a2a_connection_failure_is_down_not_unexpected(tmp_path):
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
assert "kagentz C1: ❌ A2A down (HTTP 000)" in log
assert "kagentz C1: ✅ A2A alive" not in log
assert "kagentz: ❌ A2A down (HTTP 000)" in log
assert "kagentz: ✅ A2A alive" not in log
assert "unexpected" not in log
assert "Result: 🔴 INCIDENT — 1 issue(s) found" in log
assert "Result: 🔴 1 issue(s) found" in log
assert "kagentz A2A server DOWN (connection failed)" in proc.stdout
-209
View File
@@ -1,209 +0,0 @@
#!/usr/bin/env python3
"""Behavioural tests for zulip-monitor.sh kagentz C1/C3 legs and the Result verdict.
WHY THIS FILE EXISTS: the 2026-09-19 kagentz A2A outage was correctly detected
by the monitor (C1 returned 000, the run logged an issue) but the lane's own
summarization in ops.status wrote "OK" with the note "A2A server DOWN — expected
(no credentials configured)". The optimistic verdict came from the lane, not the
script. The fix adds a C3 public-access-path leg and makes the Result line say
"INCIDENT" when issues are found, so the lane can quote it verbatim.
CONTRACT UNDER TEST:
* C1 (A2A liveness, no credential): 000 → INCIDENT.
* C3 (public access path, no credential): 502 → INCIDENT, 000 → INCIDENT,
200/302/401 → alive.
* Result verdict: when ISSUES > 0, the log's Result line says "INCIDENT",
not just "issues found".
* Healthy control: C1 401 + C3 302 → 0 issues, "all healthy".
HOW: behavioural execution using the sandbox pattern already in this repo
(tests/test_mumuni_monitor_removal.py). The sandbox copies the shipped monitor
verbatim, rewrites only its LOG constant, and runs it with stub ssh/curl on
PATH. Each test asserts from the run's own log/verdict, not from file text.
Usage: python3 -m pytest tests/test_zulip_kagentz_legs.py
"""
from __future__ import annotations
import os
import pathlib
import stat
import subprocess
ROOT = pathlib.Path(__file__).resolve().parents[1]
ZULIP_MONITOR = ROOT / "scripts" / "zulip-monitor.sh"
CONNECTED_FIXTURE = ROOT / "tests" / "fixtures" / "zulip-health-connected.json"
TANKO_VANTAGE = "192.168.68.15" # amdpve — Tanko CT 112 via pct exec
AGENT_ZERO_HOST = "192.168.68.14" # kagentz host, Agent Zero docker
# ── Stub ssh: answers Tanko and Agent Zero probes by env vars ──────────
SSH_STUB = r"""#!/usr/bin/env bash
# Stub ssh: record the target host, then answer by host + remote command.
printf '%s\n' "$*" >> "$RECORD_DIR/ssh.calls"
host=""
for a in "$@"; do
case "$a" in
*@192.168.*) host="${a##*@}" ;;
esac
done
printf '%s\n' "$host" >> "$RECORD_DIR/ssh.hosts"
cmd="${*: -1}"
case "$host" in
192.168.68.15)
case "$cmd" in
*"systemctl is-active"*) printf '%s' "$TANKO_SVC" ;;
*curl*) printf '%s' "$TANKO_HTTP" ;;
esac ;;
192.168.68.14)
case "$cmd" in
*"/a2a/"*) printf '%s' "$AZ_A2A_CODE"; exit "${AZ_A2A_EXIT:-0}" ;;
esac ;;
*)
printf 'UNEXPECTED-SSH-HOST %s\n' "$host" >> "$RECORD_DIR/unexpected-ssh" ;;
esac
exit 0
"""
# ── Stub curl: serves Zulip server, Abiba health, and C3 public URL ───
CURL_STUB = r"""#!/usr/bin/env bash
# Stub curl: serve the Abiba health fixture, the Zulip server 200, and the
# C3 public URL probe (https://kagentz.sysloggh.net/). Record every call.
printf '%s\n' "$*" >> "$RECORD_DIR/curl.calls"
case "$*" in
*:9200/health*)
case " $* " in
*" -w "*) printf '%s' "$PI_HTTP" ;; # -w '%{http_code}' probe
*) printf '%s' "$PI_BODY" ;; # body probe
esac ;;
*server_settings*)
printf '%s' "$SERVER_HTTP" ;;
*kagentz.sysloggh.net*)
printf '%s' "$KAGENTZ_PUBLIC_CODE" ;;
esac
exit 0
"""
def _write_exec(path: pathlib.Path, body: str) -> None:
path.write_text(body)
path.chmod(path.stat().st_mode
| stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200",
az_a2a_code="401", az_a2a_exit=0,
kagentz_public_code="302"):
"""Run the shipped monitor in a sandbox; return (proc, record_dir, log_path).
Only the LOG constant is rewritten (to keep the run inside the worktree).
Everything else — legs, labels, notify logic — is the shipped script.
"""
sandbox = tmp_path / "sandbox"
bindir = sandbox / "bin"
record = sandbox / "record"
bindir.mkdir(parents=True)
record.mkdir()
_write_exec(bindir / "ssh", SSH_STUB)
_write_exec(bindir / "curl", CURL_STUB)
source = ZULIP_MONITOR.read_text()
log_line = 'LOG="/root/zulip-health-monitor.log"'
assert log_line in source, "LOG constant moved — update the sandbox harness"
log_path = sandbox / "zulip-health-monitor.log"
script = sandbox / "zulip-monitor.sh"
script.write_text(source.replace(log_line, f'LOG="{log_path}"'))
env = dict(os.environ)
env.update({
"PATH": f"{bindir}:{env['PATH']}",
"RECORD_DIR": str(record),
"TANKO_SVC": tanko_svc,
"TANKO_HTTP": tanko_http,
"AZ_A2A_CODE": az_a2a_code,
"AZ_A2A_EXIT": str(az_a2a_exit),
"PI_HTTP": "200",
"PI_BODY": CONNECTED_FIXTURE.read_text(),
"SERVER_HTTP": "200",
"KAGENTZ_PUBLIC_CODE": kagentz_public_code,
})
proc = subprocess.run(["bash", str(script)], cwd=sandbox, env=env,
capture_output=True, text=True)
return proc, record, log_path
# ── Required behavioural cases ─────────────────────────────────────────
def test_c3_502_is_incident(tmp_path):
"""C3 public leg returns 502 → the run's verdict is an INCIDENT,
the C3 line names the 502, and the run is not summarised as healthy."""
proc, record, log_path = _run_monitor(tmp_path, kagentz_public_code="502")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# The C3 line names the 502.
assert "kagentz C3: ❌ public URL 502 (upstream refused)" in log
# The verdict is an INCIDENT, not healthy.
assert "Result: 🔴 INCIDENT" in log
assert "all healthy" not in log
# The run is not summarised as healthy.
assert "✅ 0 issues" not in log
def test_c3_000_is_incident(tmp_path):
"""C3 public leg returns 000 → INCIDENT."""
proc, record, log_path = _run_monitor(tmp_path, kagentz_public_code="000")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# The C3 line reports the connection failure.
assert "kagentz C3: ❌ public URL down (HTTP 000)" in log
# The verdict is an INCIDENT.
assert "Result: 🔴 INCIDENT" in log
assert "all healthy" not in log
assert "✅ 0 issues" not in log
def test_healthy_control_c1_401_c3_302(tmp_path):
"""Healthy control: C1 401 plus C3 302 → 0 issues and a healthy verdict,
proving the new leg cannot cry wolf."""
proc, record, log_path = _run_monitor(tmp_path,
az_a2a_code="401",
kagentz_public_code="302")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# Both legs report alive.
assert "kagentz C1: ✅ A2A alive (HTTP 401)" in log
assert "kagentz C3: ✅ public URL alive (HTTP 302)" in log
# Zero issues, healthy verdict.
assert "Result: ✅ 0 issues (all healthy)" in log
# No INCIDENT.
assert "INCIDENT" not in log
# No notify fired for kagentz.
assert "kagentz public URL" not in proc.stdout
assert "kagentz A2A server" not in proc.stdout
def test_c1_000_is_incident(tmp_path):
"""C1 returns 000 → INCIDENT, keeping the leg that actually caught this
outage covered behaviourally."""
proc, record, log_path = _run_monitor(tmp_path,
az_a2a_code="000",
az_a2a_exit=7,
kagentz_public_code="302")
assert proc.returncode == 0, proc.stderr
log = log_path.read_text()
# The C1 line reports the A2A down.
assert "kagentz C1: ❌ A2A down (HTTP 000)" in log
# The verdict is an INCIDENT (even though C3 is healthy).
assert "Result: 🔴 INCIDENT" in log
assert "all healthy" not in log
# The notify fired for the A2A down.
assert "kagentz A2A server DOWN" in proc.stdout
+19 -42
View File
@@ -1,9 +1,9 @@
---
kind: responsibility
name: zulip-health
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. The kagentz Zulip adapter leg is retired (its code no longer exists) — Agent Zero is probed for A2A liveness, A2A response, and public-path access only. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side.
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. The kagentz Zulip adapter leg is retired (its code no longer exists) — Agent Zero is probed for A2A liveness only. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side.
title: Zulip Mesh Health Monitor — Multi-Platform
version: 3.4.0
version: 3.3.0
runtime_contract: 2
agent: abiba
report_only_agents:
@@ -30,7 +30,7 @@ session start.
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); and the Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `kagentz.sysloggh.net` (C3 public path), `localhost:9200`
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
- **Relay access** via RA-H OS MCP for alert delivery
@@ -129,10 +129,10 @@ grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
### Liveness rule (scoped)
Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe. **Proxy-fronted exception:** a reverse proxy's `502`/`504` (upstream refused/unreachable) is not the backend answering — for proxy-fronted public endpoints such as `https://kagentz.sysloggh.net/` (C3), treat it as DOWN.
Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):**
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe (proxy `502`/`504` excepted, above).
1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe.
2. **A failed probe is never a service verdict.** Print `probe-failed: <target> <kind>` naming the exact URL/host/port and the failure kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only then report.
3. **Say which probe produced each number.** "API: 000" is unusable; "API https://chat.sysloggh.net/api/v1/server_settings -> connection timeout after 10s (retried at 25s: also timeout)" is actionable.
@@ -469,10 +469,9 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null
> monitor issued a restart for something that could not start, posting a false
> kagentz-adapter-down alert on every run. Do NOT re-add an adapter-process,
> heartbeat/queue, or adapter-restart step. Agent Zero is probed for A2A
> liveness/response and public-path access only, and a probe must never restart
> a platform.
> liveness only, and a probe must never restart a platform.
**C1: A2A Server Health (no credential needed)**
**C1: A2A Server Health**
```bash
# A2A listens on :80 inside the agent-zero container (host-mapped to :50080) and
@@ -480,9 +479,9 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null
ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/"
```
Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (`000`) → A2A server down (INCIDENT). Any other status → running but unexpected: log/report it, never restart.
Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (`000`) → A2A server down. Any other status → running but unexpected: log/report it, never restart.
**C2: A2A Response Verification (requires LITELLM_KEY)**
**C2: A2A Response Verification**
```bash
# A2A listens on :80 inside the container and is auth-gated (401 expected unauthenticated).
@@ -492,30 +491,15 @@ ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:
-d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
```
Expected: task ID with "working" status. Poll for completion with `tasks/get`. If 401, check LITELLM_KEY is set (this is a credential issue, NOT a server-down incident).
**C3: Public Access Path (no credential needed)**
```bash
# Probes the public URL that NetBird proxies to the agent-zero container.
# This is the captain's point of view: if the captain can't reach it, it's down.
# 200/302/401 = alive, 502 = proxy's "upstream refused" page (INCIDENT),
# connection failed (000) = INCIDENT. Never restarts anything.
curl -s -o /dev/null --connect-timeout 10 --max-time 15 -w '%{http_code}' https://kagentz.sysloggh.net/
```
Expected: `200` (Agent Zero login page), `302` (redirect), or `401` (auth-gated) = alive. `502` = NetBird proxy's "upstream refused" page (the container's port 80 is not listening) = INCIDENT. `000` (connection failed) = INCIDENT. Any other status → running but unexpected: log/report it, never restart.
Expected: task ID with "working" status. Poll for completion with `tasks/get`. If 401, check LITELLM_KEY is set.
**Platform C Actions**
| Condition | Action |
|-----------|--------|
| C1 A2A returns `000` (connection refused/timeout) | Alert only — never restart the platform; investigate the agent-zero container |
| C1 A2A returns a status other than `200`/`401` | Log/report as a warning — reported, never healed on |
| C2 A2A returns 401 | Check `LITELLM_KEY` is set (credential issue, NOT server-down) |
| C3 public URL returns `502` (upstream refused) | Alert only — the container's port 80 is not listening; investigate the agent-zero container |
| C3 public URL returns `000` (connection failed) | Alert only — the public path is down; investigate the NetBird proxy or the container |
| C3 public URL returns a status other than `200`/`302`/`401` | Log/report as a warning — reported, never healed on |
| A2A returns `000` (connection refused/timeout) | Alert only — never restart the platform; investigate the agent-zero container |
| A2A returns a status other than `200`/`401` | Log/report as a warning — reported, never healed on |
| LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` |
### Step 5: Global Checks
@@ -529,19 +513,12 @@ If any bot processes >50 bot-originated messages in 15min → warning.
### Step 6: Compile and Report
1. Run `scripts/zulip-monitor.sh` and take its final `Result:` line as the
authoritative run verdict. The verdict line is either
`Result: ✅ 0 issues (all healthy)` or
`Result: 🔴 INCIDENT — N issue(s) found`.
2. Quote that `Result:` line verbatim in the status report. When it says
`INCIDENT`, the run MUST be reported as an incident — never summarised as
OK/healthy and never annotated as "expected".
3. Compile all platform checks and severity
4. Determine `overall_severity` from worst per-agent severity
5. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
6. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
7. If any agent critical or >2 degraded: send relay message to user
8. Update `last_check` timestamp in `### Maintains` snapshot
1. Compile all platform checks and severity
2. Determine `overall_severity` from worst per-agent severity
3. If restart action needed, check `/tmp/zulip-monitor-debounce` — apply only if >300s since last restart
4. Log full diagnostic to `/root/zulip-health-monitor.log` with timestamp
5. If any agent critical or >2 degraded: send relay message to user
6. Update `last_check` timestamp in `### Maintains` snapshot
### Restart Debounce