diff --git a/.gitea/workflows/pr-pipeline.yaml b/.gitea/workflows/pr-pipeline.yaml index 7525b10..87b7846 100644 --- a/.gitea/workflows/pr-pipeline.yaml +++ b/.gitea/workflows/pr-pipeline.yaml @@ -1,19 +1,18 @@ name: PR Pipeline — Authorize → Validate → Review → Merge +# TRIGGER IS INTENTIONALLY UNFILTERED — DO NOT RE-ADD A `paths:` FILTER. +# +# This workflow previously carried `paths: ['**.prose.md', 'scripts/**.sh', +# '**.yaml', '**.yml']` on both `push` and `pull_request`. Any PR whose diff +# touched none of those patterns (for example a `deliverables/`-only PR, or a +# `scripts/*.py` / `bin/*` change) therefore produced NO Gitea Actions run at +# all: validation, lint, ai-review and the merge gate were silently skipped. +# Validation must run for every pull request and every push to master, so the +# trigger is deliberately unconditional. on: push: branches: [master] - paths: - - '**.prose.md' - - 'scripts/**.sh' - - '**.yaml' - - '**.yml' pull_request: types: [opened, synchronize, reopened] - paths: - - '**.prose.md' - - 'scripts/**.sh' - - '**.yaml' - - '**.yml' jobs: auth: @@ -43,17 +42,21 @@ jobs: echo "=== Prose Contract Frontmatter Validation ===" FAILED=0 for f in $(find . -name "*.prose.md" -not -path "./.git/*" -not -path "./runs/*"); do + # NOTE: use herestrings, not `echo "$FM" | grep ...`. Under the runner's + # `-e -o pipefail`, `grep -q` exits on first match and can SIGPIPE the + # producer, making the pipeline report non-zero and raising a false + # "Missing name/description" whose file set varies run to run. FM=$(sed -n '/^---$/,/^---$/p' "$f" | sed '1d;$d') [ -z "$FM" ] && { echo " ❌ $f: No YAML frontmatter"; FAILED=$((FAILED+1)); continue; } - KIND=$(echo "$FM" | grep '^kind:' | awk '{print $2}') + KIND=$(grep '^kind:' <<< "$FM" | awk '{print $2}') case "$KIND" in function|responsibility|gateway|pattern|test|template|architecture|enforcement) echo " ✅ $f: kind=$KIND" ;; *) echo " ❌ $f: Invalid kind='$KIND'"; FAILED=$((FAILED+1)) ;; esac - echo "$FM" | grep -q '^name:' || { echo " ❌ $f: Missing name"; FAILED=$((FAILED+1)); } - echo "$FM" | grep -q '^description:' || { echo " ❌ $f: Missing description"; FAILED=$((FAILED+1)); } + grep -q '^name:' <<< "$FM" || { echo " ❌ $f: Missing name"; FAILED=$((FAILED+1)); } + grep -q '^description:' <<< "$FM" || { echo " ❌ $f: Missing description"; FAILED=$((FAILED+1)); } done [ $FAILED -gt 0 ] && { echo "❌ FRONTMATTER FAILED ($FAILED error(s))"; exit 1; } echo "✅ Frontmatter validation passed" diff --git a/AGENTS.md b/AGENTS.md index b9a840c..95fb2a4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -55,7 +55,7 @@ Two incidents taught us this: ### Stage 3 — AI Review - Diff is sent to `syslog-auto` model via LiteLLM - Review checks against infrastructure-control ground truth: - - CT IDs match PVE cluster (100-117, no 122/123) + - CT IDs match the PVE cluster inventory in `infrastructure-control.prose.md` Appendix B (100-120 with gaps; no 122/123) - Grafana is direct LAN :3001, NOT behind nginx - Zulip is CT 117 on storepve (bridge IP .19) - Strix Halo :8080 is firewalled to .116 only diff --git a/README.md b/README.md index 46b8968..c850817 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,7 @@ prose run memory-audit-maintenance memory_threshold=90 verify_configs=true prose run hermes-config-template agent_name=syslog-devops default_model=claude-sonnet-4 # Configure an agent with a different auxiliary model -prose run hermes-config-template agent_name=syslog-code default_model=qwen3.6-27B-code auxiliary_model=gemma-4-12b +prose run hermes-config-template agent_name=syslog-code default_model=gpu-dense auxiliary_model=gpu-vision ``` ### Option B: Manual Execution @@ -116,7 +116,7 @@ Run on trigger or schedule. Maintain persistent world-model state across runs. | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. | | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. | | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. | -| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) | +| `litellm-self-heal` | LiteLLM | Applies remediation rules for LiteLLM stack failures detected by `litellm-health` (full nginx → LiteLLM → GPU chain). 9 remediation rules. | | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. | | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. | | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. | @@ -149,7 +149,7 @@ Called on-demand as single-render tools. | Contract | Description | |---|---| | `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. | -| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. | +| `litellm-health` | LiteLLM health check: public vs backend surfaces, CT 116 containers, GPU fleet, one model per GPU host, and agent keys. Owner of the probes; `litellm-self-heal` owns remediation. | | `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. | | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. | | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. | diff --git a/abiba-zulip-restore.prose.md b/abiba-zulip-restore.prose.md index dd0f8ac..e48a24d 100644 --- a/abiba-zulip-restore.prose.md +++ b/abiba-zulip-restore.prose.md @@ -34,7 +34,7 @@ verification, and DM loopback testing. | @all-bots user ID | 20 | ✅ (config, verified by API at runtime) | | PM2 process name | abiba-zulip | ✅ | | Provider | syslog-harness (http://192.168.68.116/v1) | ✅ | -| Default model | deepseek-v4-pro | ✅ (settings.json) | +| Default model | syslog-auto | ✅ (settings.json) | ## Architecture diff --git a/audit-hermes-config.py b/audit-hermes-config.py index 773fff2..2c3d109 100644 --- a/audit-hermes-config.py +++ b/audit-hermes-config.py @@ -36,6 +36,59 @@ def warn(rule, message): WARNINGS.append(f"[{rule}] {message}") +# Derivation rule: a model name is any scalar under a mapping key named `model` or +# `model_name`, at any depth. The top-level `model:` SECTION is the exception where the model +# name lives under `default`/`model`/`model_name` inside that section, so it is descended +# specially. The only other exception is key `models` (litellm key-generation params carry a +# list of model names). EXTEND THE ALLOWLIST for a new exception; do NOT add another field by +# hand. +MODEL_KEYS = ("model", "model_name") +MODEL_SECTION_KEYS = ("default", "model", "model_name") +MODEL_LIST_KEYS = ("models",) + + +def _iter_model_values(node, path=""): + """Yield (path, value) for every model-name-bearing scalar in a config.""" + if isinstance(node, dict): + for key, value in node.items(): + child = f"{path}.{key}" if path else key + if key in MODEL_KEYS: + if isinstance(value, dict): + for subkey in MODEL_SECTION_KEYS: + subvalue = value.get(subkey) + if isinstance(subvalue, str): + yield (f"{child}.{subkey}", subvalue) + for subkey, subvalue in value.items(): + if isinstance(subvalue, (dict, list)): + yield from _iter_model_values(subvalue, f"{child}.{subkey}") + elif isinstance(value, list): + yield from _iter_model_values(value, child) + else: + yield (child, value) + elif key in MODEL_LIST_KEYS: + yield from _iter_model_list(value, child) + elif isinstance(value, (dict, list)): + yield from _iter_model_values(value, child) + elif isinstance(node, list): + for i, item in enumerate(node): + yield from _iter_model_values(item, f"{path}[{i}]") + + +def _iter_model_list(node, path): + """Yield scalars under an allowlisted `models` key (list of names or list of dicts).""" + if isinstance(node, list): + for i, item in enumerate(node): + yield from _iter_model_list(item, f"{path}[{i}]") + elif isinstance(node, dict): + for key, value in node.items(): + if key in MODEL_KEYS and isinstance(value, str): + yield (f"{path}.{key}", value) + elif isinstance(value, (dict, list)): + yield from _iter_model_list(value, f"{path}.{key}") + else: + yield (path, node) + + def audit(path): with open(path) as f: cfg = yaml.safe_load(f) @@ -89,15 +142,16 @@ def audit(path): ) # --- Rule 8: GPU Workload Distribution --- + # gpu-light (and gemma-4-12b) were retired 2026-09-12; the RTX 5070 stable alias is gpu-vision. check( - aux.get("vision", {}).get("model") == "gpu-light", + aux.get("vision", {}).get("model") == "gpu-vision", "Rule 8", - f"auxiliary.vision.model must be gpu-light (got {aux.get('vision', {}).get('model')!r}) — RTX 5070 stable alias", + f"auxiliary.vision.model must be gpu-vision (got {aux.get('vision', {}).get('model')!r}) — RTX 5070 stable alias", ) check( - aux.get("web_extract", {}).get("model") == "gpu-light", + aux.get("web_extract", {}).get("model") == "gpu-vision", "Rule 8", - f"auxiliary.web_extract.model must be gpu-light (got {aux.get('web_extract', {}).get('model')!r}) — RTX 5070 stable alias", + f"auxiliary.web_extract.model must be gpu-vision (got {aux.get('web_extract', {}).get('model')!r}) — RTX 5070 stable alias", ) # --- Rule 9: Compression Threshold --- @@ -109,7 +163,7 @@ def audit(path): check( comp.get("max_context_window") == 131072, "Rule 9", - f"compression.max_context_window must be 131072 (got {comp.get('max_context_window')!r}) — matches 128K GPU capacity", + f"compression.max_context_window must be 131072 (got {comp.get('max_context_window')!r}) — syslog-auto pool floor (NVIDIA hosts 128K; Strix Halo 256K)", ) # --- Rule 10: Default Model Must Be syslog-auto --- @@ -178,21 +232,34 @@ def audit(path): f"custom_providers[0].base_url must end with /v1 (got {cp.get('base_url')!r})", ) - # --- No raw model names (Rule 7/8 spirit) --- - raw_names = {"gemma-4-12b", "qwen3.6-27B-code", "qwen3.6-35B-udq4", "ornith-1.0-35b"} - for section_path, section_dict in [ - ("model", model), ("compression", comp), - ("auxiliary.vision", aux.get("vision", {})), - ("auxiliary.web_extract", aux.get("web_extract", {})), - ("auxiliary.compression", aux.get("compression", {})), - ("delegation", deleg), - ]: - m = section_dict.get("model", "") - if m in raw_names: + # --- Retired/raw model names (Rule 7/8 spirit) --- + # The audit's job is to catch configs that are BROKEN, not to enforce a style preference. + # NON-RESOLVING names (removed 2026-09-12, verified 400/403 via live LiteLLM) must hard-FAIL: + # gpu-light -> gpu-vision ; gemma-4-12b -> gpu-vision + # crew-auto -> syslog-auto (its 64K cap is retired; no cap in force) ; ornith-1.0-35b -> strix-moe + # RESOLVING names (verified 200) are discouraged but working, so they only WARN: + # qwen3.6-27B-code -> gpu-dense ; qwen3.6-35B-udq4 -> strix-moe + # Failing a working alias would reject valid configs - the exact defect this change fixes. + non_resolving = { + "gpu-light": "gpu-vision", + "gemma-4-12b": "gpu-vision", + "crew-auto": "syslog-auto (its 64K cap is retired; no cap in force)", + "ornith-1.0-35b": "strix-moe", + "qwen3.6-27B-code": "gpu-dense", + "qwen3.6-35B-udq4": "strix-moe", + } + raw_but_live = {} + for field_path, value in _iter_model_values(cfg): + if value in non_resolving: + check( + False, + "Rule 7/8", + f"{field_path} = {value!r} is retired and no longer resolves (2026-09-12) — use {non_resolving[value]}", + ) + elif value in raw_but_live: warn( "Rule 7/8", - f"{section_path}.model = {m!r} — raw model name, use stable alias instead " - f"(gpu-light, gpu-dense, strix-moe, syslog-auto)", + f"{field_path} = {value!r} is a raw-but-live model name — prefer the stable alias {raw_but_live[value]}", ) # --- Report --- diff --git a/contract-registry.yaml b/contract-registry.yaml index 7135893..019be2b 100644 --- a/contract-registry.yaml +++ b/contract-registry.yaml @@ -628,7 +628,7 @@ contracts: sensitivity: high status: active owner: abiba - version: 3.2.0 + version: 3.3.0 trigger: type: scheduled cadence: '*/15 * * * *' @@ -693,8 +693,8 @@ contracts: version: 1.0.0 trigger: type: scheduled - cadence: '*/10 * * * *' - description: "Every 10 minutes \u2014 LiteLLM proxy health" + cadence: '5 3,7,11,15,19,23 * * *' + description: "4-hourly staggered dispatch via fm-send (run contract litellm-health)" cron_job_id: null execution: agent: abiba diff --git a/cron-prompts-review.md b/cron-prompts-review.md index d942546..3d2151e 100644 --- a/cron-prompts-review.md +++ b/cron-prompts-review.md @@ -2,6 +2,10 @@ Generated: 2026-07-13 20:59:18 ET +> **Point-in-time snapshot.** Schedules and cadences are authoritative in +> `contract-registry.yaml`; any schedule quoted below may be stale. Do not use +> this file as the source of truth for a contract's trigger. + --- ## hermes-key-enforcement @@ -442,7 +446,7 @@ IMPORTANT: If the contract file does not exist in prose-contracts/main, report f ## litellm-health -**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** */10 * * * * +**Category:** monitoring | **Domain:** litellm | **Owner:** abiba | **Schedule:** see contract-registry.yaml (authoritative) ``` Contract Enforcement: litellm-health @@ -450,7 +454,7 @@ Contract Enforcement: litellm-health Category: monitoring Domain: litellm Owner: abiba -Schedule: Every 10 minutes — LiteLLM proxy health +Schedule: see contract-registry.yaml (authoritative) This is a monitoring contract. Execute the monitoring checks defined in the contract. Report any deviations from expected state. diff --git a/disk-gc-threat-response.prose.md b/disk-gc-threat-response.prose.md index 34243d6..77dbf11 100644 --- a/disk-gc-threat-response.prose.md +++ b/disk-gc-threat-response.prose.md @@ -1,11 +1,14 @@ --- report_only_agents: - koby # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129 +# ⛔ The guest/host-keyed report-only gate the GC executor MUST honour lives in the body +# "Hard gate" YAML block below — that block is authoritative and is the only copy. kind: responsibility name: disk-gc-threat-response description: > Recurring disk health scan, garbage collection, and threat response - across 15 Proxmox CTs + 3 GPU bare-metal hosts. Triggered by incident + across 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts + (fleet verified against `pvesh get /cluster/resources` 2026-09-12). Triggered by incident 2026-07-04 where CT 105 (kagentz) hit 87% disk (49G/59G) from Docker image bloat — 5 dangling images, 15 build cache layers. Recovered 35.67GB. Second incident 2026-07-09: amdpve (.15) Docker @@ -25,7 +28,7 @@ logged within 5 minutes of discovery. ## Scope -All 15 CTs via `pct-run` + 3 GPU bare-metal hosts via direct SSH. +All 20 Proxmox guests (17 LXC via `pct-run` + 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts via direct SSH. Docker hosts get special attention: | Host | CT | Disk Risk | GC Strategy | @@ -81,6 +84,32 @@ and escalation trail. - May also be invoked manually: `prose run disk-gc-threat-response` - Threat-driven: if Amber/Red/Critical detected, immediate GC phase activates +## Scanner: scripts/disk-gc-scan.py + +The fleet scan is executed by `scripts/disk-gc-scan.py`, which makes reachability +verdicts deterministic: + +1. **Retry on failure:** Each probe retries once before declaring a guest unreachable. +2. **Named probe target:** Every rendered line names the guest, CT id, node, and + access method actually used. +3. **Failure kind printed:** An unreachable guest is reported with its failure kind + (timeout, ssh-auth, no-route, conn-refused, ssh-exit-N) — never as a bare + "unreachable" verdict. +4. **Per-guest access method:** The correct access path is selected from a per-guest + map so the wrong path cannot be picked by an executor improvising: + - CT 105 (kagentz) = `ssh root@kagentz` (NOT `pct exec 105` — pct exec sees + loop0/59G instead of the real 99G filesystem) + - CT 109 (docker-vm) = `ssh root@192.168.68.7` (NOT `pct exec` — it's a KVM VM) + - All other CTs = `pct-run ` (which uses `pct exec` via SSH to the node) +5. **Every figure traces to a named probe:** The scan output prints the exact command + that produced each disk figure, so two different guests can never render + identical numbers without the probe commands proving it. + +Run: `python3 scripts/disk-gc-scan.py` (or `--json` for machine-readable output). + +The scan feeds into `scripts/disk-gc-plan.py`, which applies the report-only gate +from the `report_only_guests` YAML block above. + ## Shape - `self`: scan all CTs via Proxmox API + SSH exec, trigger GC, alert @@ -99,35 +128,67 @@ and escalation trail. ## Execution +### Hard gate: report-only guests (READ THIS BEFORE RUNNING GC) + +**CT 111 / hostname `tdunna` / 192.168.68.129 is DETECT-AND-REPORT-ONLY.** It belongs to Theo. +The captain ruled 2026-08-17 and re-confirmed 2026-09-10 that Theo handles CT 111 himself. +At **every** threat level — AMBER, RED, or CRITICAL — the executor must: + +- push the threat row and alert the owner, and +- **never** call `gc-executor`, and **never** run any GC command against that guest: no + `apt-get clean/autoremove`, no `journalctl --vacuum-*`, no `find /var/log -delete`, no + `/tmp`/`/var/tmp` deletion, no snap removal, no `docker system prune`. + +This gate is keyed on **guest id / hostname / IP**, not on an agent name. The frontmatter +`report_only_agents` marker (e.g. `koby`) names an AGENT while the scan unit is a GUEST, so an +agent-name marker can silently miss the guest it lives on — it must never be the only gate. + +**The authoritative machine-readable exclusion list is the YAML block below.** The executor +reads it at run time; `scripts/disk-gc-plan.py` turns a fleet scan into the action plan using it. +Extend the list here, never by hand-maintaining a second copy. The Execution loop below MUST +call that planner and MUST NOT reimplement the gate. + +```yaml +# disk-gc report-only guests — authoritative. Keyed on guest/host, not agent. +report_only_guests: + - guest: 111 + hostname: tdunna + ip: 192.168.68.129 + node: storepve + reason: "Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10" +``` + +### Loop + ```prose let fleet = call disk-scanner scope: all -let threats = [] -for ct in fleet: - if ct.usage_pct >= 95: - push threats { ct: ct.id, level: "CRITICAL", pct: ct.usage_pct } - else if ct.usage_pct >= 85: - push threats { ct: ct.id, level: "RED", pct: ct.usage_pct } - else if ct.usage_pct >= 75: - push threats { ct: ct.id, level: "AMBER", pct: ct.usage_pct } +-- The report-only gate is IMPLEMENTED IN scripts/disk-gc-plan.py and MUST NOT be +-- reimplemented here. That planner reads the contract's `report_only_guests` YAML block +-- and matches on guest id OR hostname OR IP, so the tested gate is the executed gate. +let plan = call disk-gc-plan + fleet: fleet --- sort by severity descending -sort threats by pct desc +for row in plan: + if row.action == "report-only": + -- Excluded guest: alert only. No gc-executor call is constructed for it, at any level. + call alerter + threat: row + result: { action: "report-only", reason: row.reason } + else: + let result = call gc-executor + ct: row.target + level: row.level + strategy: lookup-gc-strategy(row.target) -for threat in threats: - let result = call gc-executor - ct: threat.ct - level: threat.level - strategy: lookup-gc-strategy(threat.ct) - - call alerter - threat: threat - result: result + call alerter + threat: row + result: result call summary-reporter fleet: fleet - threats: threats + plan: plan ``` ## GC Strategies by Host Type @@ -239,7 +300,9 @@ dangling images and orphaned build cache. No automated GC was in place. ## Incident Log: 2026-07-09 — amdpve docker bloat ### Discovery -Scheduled fleet disk scan via `pct-run` across all 15 CTs + 3 GPU bare-metal hosts. +Scheduled fleet disk scan across all 20 Proxmox guests (17 LXC via `pct-run`, 3 QEMU VMs via direct SSH) + 3 GPU bare-metal hosts. +> **Report-only gate applies to this scan:** CT 111 (`tdunna`, 192.168.68.129) is alerted but never +> garbage-collected at any level. amdpve (.15) flagged at 78% (AMBER threshold: 75%). ### Diagnosis @@ -272,25 +335,35 @@ one-off GPU builds. No automated post-migration cleanup was in place. - Contract now scans GPU bare-metal hosts alongside CTs - Access via `pct-run` script for all CTs (no hardcoded IPs) -## Access Matrix (documented 2026-07-09) +## Access Matrix (verified against `pvesh get /cluster/resources` 2026-09-12) -### CT Access (via pct-run) -| CT | Name | Node | Status | -|----|------|------|--------| -| 100 | abiba | minipve | local | -| 102 | adguard | minipve | ✅ reachable | -| 104 | authentik | minipve | ✅ reachable | -| 105 | kagentz | minipve | ✅ reachable | -| 106 | ra-h-os | storepve | ✅ reachable | -| 107 | pbs | storepve | ✅ reachable | -| 108 | media | storepve | ✅ reachable | -| 110 | gitea | minipve | ✅ reachable | -| 111 | tdunna | amdpve | ✅ reachable | -| 112 | tanko | amdpve | ✅ reachable | -| 113 | baggy | amdpve | ✅ reachable | -| 115 | scottdenya | amdpve | ✅ reachable | -| 116 | syslog-api | minipve | ✅ reachable | -| 117 | zulip | storepve | ✅ reachable | +### Guest Access (via `pct-run` — CT id only, node resolved by `scripts/pct-run.sh`) +| Guest | Name | Node | Type | Status | +|------|------|------|------|--------| +| 100 | abiba | minipve | lxc | ✅ reachable (probed via pct-run like any other guest; no local shortcut) | +| 102 | adguard | minipve | lxc | ✅ reachable | +| 104 | authentik | minipve | lxc | ✅ reachable | +| 105 | kagentz | **amdpve** | lxc | ✅ reachable (was documented as minipve — corrected) | +| 106 | ra-h-os | storepve | lxc | ✅ reachable | +| 107 | pbs | storepve | lxc | ✅ reachable | +| 108 | media | storepve | lxc | ✅ reachable | +| 110 | gitea | minipve | lxc | ✅ reachable | +| 111 | tdunna | **storepve** | lxc | ⛔ **REPORT-ONLY** (192.168.68.129, Theo's box — no GC at any level) | +| 112 | tanko | amdpve | lxc | ✅ reachable | +| 113 | baggy | amdpve | lxc | ✅ reachable | +| 115 | scottdenya | amdpve | lxc | ✅ reachable | +| 116 | syslog-api | minipve | lxc | ✅ reachable | +| 117 | zulip | storepve | lxc | ✅ reachable | +| 118 | jdownloader | storepve | lxc | ✅ reachable | +| 119 | infisical-vault | minipve | lxc | ✅ reachable | +| 120 | adguard2 | amdpve | lxc | ✅ reachable | + +### QEMU VMs (via direct SSH) +| VM | Name | Node | IP | Status | +|----|------|------|-----|--------| +| 101 | llm-gpu (workload now bare metal .8) | acerpve | — | ✅ reachable | +| 103 | ocu-llm (workload now bare metal .110) | ocupve | — | ✅ reachable | +| 109 | docker-vm | storepve | 192.168.68.7 | ✅ reachable | ### GPU Bare Metal (via direct SSH) | Host | IP | GPU | Status | @@ -299,9 +372,15 @@ one-off GPU builds. No automated post-migration cleanup was in place. | ocu-llm | 192.168.68.110 | RTX 5070 | ✅ reachable | | amdpve | 192.168.68.15 | Strix Halo | ✅ reachable | -### KVM VM (via direct SSH) -| Host | IP | Role | Status | -|------|-----|------|--------| -| docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable | - -> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7. +> **Fleet count:** 20 Proxmox guests (17 LXC + 3 QEMU VMs) + 3 GPU bare-metal hosts. Corrected +> 2026-09-12: CT 105 → amdpve, CT 111 → storepve, and guests 118/119/120 were missing. +> +> **CT 100 probe gap (folded in):** CT 100 previously reported "unreachable (not reported)" every +> run. Root cause is the same stale access layer: `pct-run` resolves the guest's node from its map, +> and the map/contract must reflect `pvesh /cluster/resources`. Verified working from inside CT 100: +> `scripts/pct-run.sh 100 "df -P / | tail -1"` → `23% /`. Probe CT 100 through `pct-run` like any +> other guest — never through a local-only path, since the scanner itself runs inside CT 100 and a +> container has no `pct` binary. +> +> **KVM VM:** CT 109 (docker-vm) is a QEMU VM, not LXC — access via SSH .7. +> **NOTE:** For kagentz (CT 105), use `ssh root@kagentz` (hostname), NOT `pct exec 105` — `pct exec 105` shows loop0 (59G) while `ssh root@kagentz` shows the real filesystem (99G). For docker-vm (CT 109), use `ssh root@192.168.68.7`, not `pct exec`. diff --git a/docs/AUTHORING-GUIDE.md b/docs/AUTHORING-GUIDE.md index f73129e..adf2f78 100644 --- a/docs/AUTHORING-GUIDE.md +++ b/docs/AUTHORING-GUIDE.md @@ -13,7 +13,7 @@ the description: 1. **What system does this contract touch?** Name the hosts, CTs, containers, and services explicitly. "The inference fleet" is vague. "GPU .8 (RTX 3090, - qwen), .110 (RTX 5070, gemma), .15 (Strix Halo, strix-moe), and LiteLLM on CT + qwen), .110 (RTX 5070, gpu-vision), .15 (Strix Halo, strix-moe), and LiteLLM on CT 116" is specific. 2. **Who runs this contract, and when?** State the agent, the trigger (cron, diff --git a/docs/probe-drift-round2-evidence.md b/docs/probe-drift-round2-evidence.md index c6a5b2d..827abbb 100644 --- a/docs/probe-drift-round2-evidence.md +++ b/docs/probe-drift-round2-evidence.md @@ -261,6 +261,10 @@ repaired. not exist` on .15. `agent-health-check.py` now carries the live-verified `storepve` mapping (the script is not the topology source of truth); the CRITICAL contract itself needs an authorized correction. + **✅ Resolved 2026-09-12:** the topology was corrected in its owner, + `infrastructure-control.prose.md` (CT 111 → storepve, CT 105 → amdpve), and + `scripts/pct-run.sh` now matches. This snapshot is left as observed; treat + those owner documents as authoritative. 2. **Strix Halo `:8080` firewall claim is stale.** `prose-ai-review.sh` ground-truth rule #4 and `gpu-monitor.prose.md` say `:8080` is firewalled to `.116` only and `.24` cannot probe it. Live on .15: diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index e539ee4..f2f5db7 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -6,13 +6,14 @@ description: > registration, health checks, LiteLLM sync, agent key management, GPU saturation watchdog, Prometheus/Grafana monitoring, and self-healing. UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe, - gpu-dense, gpu-light. These never change — only the underlying model does. - Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB). - RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster). - UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability. - Strix Halo model swapped to qwen3.6-35B-udq4 (22GB, strix-moe alias). - Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling. - For larger context needs → fall back to external providers (deepseek). + gpu-dense, gpu-vision (gpu-light was superseded by gpu-vision on 2026-09-12). + These never change — only the underlying model does. + Strix Halo: strix-moe → Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (22GB, 256K ctx). + RTX 5070: gpu-vision — IQ4_NL + MTP draft (~122 tok/s, 2x faster). + UPDATED 2026-07-17: NVIDIA host context reduced from 256K to 128K for stability. + Strix Halo model: Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (alias strix-moe, 256K context). + Strix Halo runs 256K (n_ctx 262144, --kv-unified); RTX 3090 and RTX 5070 remain at 128K. + For >128K on NVIDIA hosts → fall back to external providers (deepseek). VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%. agent: abiba triggers: @@ -24,7 +25,7 @@ triggers: ## Maintains -- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models +- gpu_roster: { models: map, hosts: map } — GPU host/model roster; the authoritative alias/weight/fallback registry is CT 116 `litellm_config.yaml` - router: { status: "healthy", roster_loaded: bool, models: array } - litellm: { status: "healthy", keys: array, models: array } - agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB @@ -79,54 +80,28 @@ triggers: Agent configs, cron jobs, and workflows MUST use these aliases, never model-specific names. When a model is swapped on a GPU, ONLY the infrastructure layer changes — agent configs are untouched. -| Alias | GPU | Current Model | Will Route To | -|-------|-----|---------------|---------------| -| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo | -| `gpu-dense` | RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | Whatever runs on RTX 3090 | -| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 | +| Alias | Serves | Where | Kind | +|-------|--------|-------|------| +| `gpu-dense` | heavy reasoning | RTX 3090 (192.168.68.8) | direct alias | +| `gpu-vision` | vision / web extract / light tasks | RTX 5070 (192.168.68.110) | direct alias AND `syslog-auto` pool member | +| `strix-moe` | compression (MoE) | Strix Halo (192.168.68.15) | direct alias | +| `syslog-auto` | balanced default | weighted pool across the three GPU hosts | pool router | -**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.5-9b-it) still work -but are deprecated for agent configs. Only the stable aliases survive model swaps. +Single source of truth for models, aliases, rpm caps, weights and fallback chains: +CT 116 `/opt/inference-harness/litellm_config.yaml`. Do not duplicate those values in +contracts — read them there. -## Current Model Assignments (2026-07-15) - -| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | -|-------|-----|------|------|-----|----------|----------|-------------|--------| +**No backward compatibility**: Old model-specific names (qwen3.6-27B-code, qwen3.5-9b-it) are retired as of 2026-09-12 +and no longer resolve; do not use them in agent configs. Only the stable aliases survive model swaps. `gemma-4-12b` +is retired and returns 400 `Invalid model name`. ## Routing Configuration (LiteLLM — July 2026) -### syslog-auto Weighted Pool (Direct GPU — bypasses router) - -| Model | GPU | Weight | RPM Cap | Timeout | -|-------|-----|--------|---------|---------| -| Qwen3.8-27B-Uncensored-Q4_K_M | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | +Model, alias, rpm/weight and fallback values are owned by CT 116 +`/opt/inference-harness/litellm_config.yaml` (see § Stable Role-Based Aliases above). Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) was decommissioned 2026-09-11 and is NOT in the inference path. -### Direct Model Endpoints - -| Model | RPM Cap | Notes | -|-------|---------|-------| - -### Stable Aliases (for agent configs — never change) - -| Alias | RPM Cap | Routes To | Purpose | -|-------|---------|-----------|---------| -| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) | -| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning | -| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks | - -### Fallback Chains -- gemma → qwen -- qwen → gemma -- strix-moe → qwen → gemma -- syslog-auto → qwen → gemma → qwen3.6-35B-udq4 - -### Why Strix Halo RPM Is Capped -- Direct (strix-moe): 40 RPM (tight) — Strix Halo is shared with compression tasks -- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously -- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C - ## Operations ### add-model @@ -176,9 +151,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act 2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!") 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models` -5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml` - - Qwen3.5-9B: 120s, qwen3.6-27B-code: 300s, Carnice-Qwen3.6-MoE-35B-A3B/strix-moe: 300s (strix-moe alias retained, legacy name qwen3.6-35B-udq4 deprecated) - - global request_timeout: 300s, nginx proxy_read_timeout: 600s +5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml` — read the live values from the authority config; do not assert them from this contract. 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power) 7. Check port conflicts: verify only one llama-server on :8080 per host 8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`) @@ -217,7 +190,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server | | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) | -| `/etc/systemd/system/strix-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo) running unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | +| `/etc/systemd/system/strix-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo) running Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (256K ctx). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | ## Prometheus & Grafana @@ -234,7 +207,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first. Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster(). - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround. -- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). +- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf`, alias `strix-moe`, 256K context (n_ctx 262144), --parallel 2 --kv-unified, flash-attn + q4 KV, multimodal (mmproj loaded). - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116. @@ -243,16 +216,17 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled. - **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds. - **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down. +- **Alias-retirement sweep (2026-09-12)**: `gemma-4-12b`, `gpu-light` and `crew-auto` are retired and replaced by `gpu-vision` / no cap respectively. The agent-facing templates (`hermes-config-template.prose.md`, `hermes-agent-baseline.prose.md`), `litellm-api-keys.prose.md`, `gpu-self-heal.prose.md`, `hermes-key-enforcement.prose.md`, `inference-optimization.prose.md`, `litellm-client-timeouts.prose.md` and the executable `audit-hermes-config.py` were all updated to the live canonical alias in the same change. **koby's config on .129 still names `gpu-light` (and `gemma-4-E4B`); .129 is report-only, so that is recorded for its owner and NOT edited here.** ## GPU Inference Benchmarks (Current) | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | |-----|-------|-----------|--------------|----------|---------| | RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | **TBD** | — | — | **128K** | -| Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** | +| Strix Halo (.15) | Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (strix-moe) | **65** | 140 | — | **256K** | -Benchmarks from 2026-07-17. Strix Halo model: qwen3.6-35B-udq4. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. -All 3 GPUs now at 128K context (2026-07-17, reduced from 256K for stability). +Benchmarks from 2026-07-17. Strix Halo model: Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf (alias strix-moe), n_ctx 262144, --parallel 2 --kv-unified. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. +GPU contexts: RTX 3090 (.8) and RTX 5070 (.110) at 128K; Strix Halo (.15) at 256K (2026-09-12). Benchmarks run through LiteLLM proxy (192.168.68.116:4000) every 5 minutes. Degradation alerts fire at 30% (warning) and 50% (critical) below baseline. @@ -265,42 +239,40 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window. ### Stable Aliases — CRITICAL All agent configs MUST use stable role-based aliases, never model-specific names: -- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`) -- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`) -- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`) -- `auxiliary.web_extract.model: gpu-light` +- `compression.model: syslog-auto` +- `auxiliary.vision.model: gpu-vision` +- `delegation.model: gpu-dense` +- `auxiliary.web_extract.model: gpu-vision` When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched. ### Context Windows -- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K** -- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek) -- Compression threshold 0.60: fires at ~77K (~51K headroom before 128K ceiling) +- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **256K** (2026-09-12) +- **Agents via `syslog-auto`**: 128K ceiling — the pool's safe floor (NVIDIA hosts are 128K). For >128K workloads, use external providers (deepseek) +- Compression threshold 0.65 (audit Rule 9): fires at ~85K (~43K headroom before 128K ceiling) - **Pi agents (Abiba)**: `compaction.reserveTokens: 52739` (≈60% of 128K) -- Mumuni compression model alias: `strix-moe` with 300s timeout +- Mumuni compression model alias: `syslog-auto` ### Mumuni Agent Profile -Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the primary business assistant. This profile is the reference for all agent configs: +Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the primary business assistant. This profile is the reference for all agent configs. The compression values below are the current required values per template Rules 7/9 and `audit-hermes-config.py`; whether Mumuni's LIVE config currently complies is a separate operational question. | Setting | Value | Notes | |---------|-------|-------| -| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) | +| `model.default` | `syslog-auto` | Balanced default (pool router) | | `model.provider` | `custom:litellm` | LiteLLM on CT116 | -| `compression.model` | `strix-moe` | Stable alias — survives model swaps | -| `aux.compression.model` | `strix-moe` | Compression auxiliary model | -| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) | -| `aux.web_extract.model` | `gpu-light` | Web extraction | +| `compression.model` | `syslog-auto` | Rule 7: auto-routing, prevents Strix Halo overload | +| `aux.compression.model` | `syslog-auto` | Must match `compression.model` (Rule 7) | +| `aux.vision.model` | `gpu-vision` | Vision tasks (RTX 5070) | +| `aux.web_extract.model` | `gpu-vision` | Web extraction | | `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) | -| `context.max_context_window` | 131072 (128K) | Reduced from 256K 2026-07-17 — stable 128K ceiling | -| `compression.threshold` | 0.60 | Triggers at ~77K (~60% of 128K) — optimized for 128K context | +| `context.max_context_window` | 131072 (128K) | Conservative `syslog-auto` pool floor (NVIDIA hosts 128K; Strix Halo 256K) | +| `compression.threshold` | 0.65 | Rule 9: triggers at ~85K for a 128K window | | `compression.target_ratio` | 0.3 | Compresses to ~38K | | `compression.protect_last_n` | 40 | Preserves last 40 messages | | `memory.memory_char_limit` | 800 | Brief memory entries | | `personalities` | `creative` | Creative assistant personality | | Platforms | cli, homeassistant, signal, telegram, zulip | All Hermes platforms | -| Main model timeout | 300s | LiteLLM global timeout | -| Compression model timeout | 300s | strix-moe timeout increased from 120s | ### Agent Update Status (2026-07-15) diff --git a/gpu-monitor.prose.md b/gpu-monitor.prose.md index a6808e1..72bad0d 100644 --- a/gpu-monitor.prose.md +++ b/gpu-monitor.prose.md @@ -27,8 +27,8 @@ agent: abiba ┌──────┐ ┌──────┐ ┌────────┐ │.8:8080│ │.110 │ │.116:80 │ │RTX3090│ │:8080 │ │nginx │ -│gemma │ │RTX5070│ │router │ -└──────┘ │qwen27B│ │LiteLLM │ +│qwen │ │RTX5070│ │router │ +└──────┘ │vision │ │LiteLLM │ └──────┘ │dashboard│ └────────┘ ``` diff --git a/gpu-self-heal.prose.md b/gpu-self-heal.prose.md index 04ef8e4..c7c26ae 100644 --- a/gpu-self-heal.prose.md +++ b/gpu-self-heal.prose.md @@ -12,7 +12,7 @@ description: > Router (port 9000) DECOMMISSIONED 2026-09-11; references replaced with direct GPU routing. Benchmark baselines refreshed to live values. Prometheus exporters removed — not deployed; fall back to direct sidecar probes. - Stable role-based aliases (strix-moe, gpu-dense, gpu-light) from gpu-fleet. + Stable role-based aliases (strix-moe, gpu-dense, gpu-vision) from gpu-fleet. agent: abiba depends_on: - gpu-monitor.prose.md (live data source on .24:9100) @@ -48,12 +48,12 @@ depends_on: | Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role | |-------|-----|------|-------|------|-----|-------|------| -| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs | +| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf | ~10/64GB (16%) | 256K | 62.9 | Compression, summarization, long docs | Key notes: - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) was decommissioned 2026-09-11 and is NOT in the inference path. -- Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated. -- RTX 5070 tok/s is ~145 for Qwen3.5-9B — gpu-light is the fastest endpoint. Route vision/web/light work there first. NOTE: Qwen3.5-9B is multimodal (image+text), NOT text-only like gemma-4-12b was. +- Stable aliases (gpu-dense, gpu-vision, strix-moe) from gpu-fleet are the canonical names for agent configs — use these, not model-specific names. The retired names `gpu-light` and `gemma-4-12b` were superseded by `gpu-vision` on 2026-09-12 and no longer resolve (400 `Invalid model name`). +- The RTX 5070 is the fastest endpoint per token — `gpu-vision` is its canonical alias. Route vision/web/light work there first. The RTX 5070 model is multimodal (image+text). - Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads. - RTX 3090 VRAM at 88% — within role-appropriate range (role = heavy reasoning, needs the headroom). - RTX 5070 VRAM at 83% — role-appropriate for vision/web (smaller batch sizes). @@ -64,7 +64,7 @@ Key notes: - **Detect**: Any GPU temp >85°C sustained for 2+ consecutive polls - **Fix**: 1. Reduce inference concurrency on that GPU (load-side cooling only — NO fan control) - 2. Redirect new requests to cooler GPUs via LiteLLM fallback chains (gemma → qwen, qwen → gemma) + 2. Redirect new requests to cooler GPUs via LiteLLM fallback chains (gpu-vision → gpu-dense, gpu-dense → gpu-vision) 3. If all GPUs hot, alert about cooling infrastructure - **Verify**: Temp drops below 80°C within 5 minutes - **Escalate after**: 3 verification failures → Zulip alert @@ -142,10 +142,10 @@ Key notes: - **Escalate**: If Tier 2 triggers and temp still rising after 5 min → possible hardware failure ### Rule 9: Context Window Optimization -- **Detect**: Benchmark tok/s vs baseline for each GPU at current context (all 128K) +- **Detect**: Benchmark tok/s vs baseline for each GPU at its current context - RTX 3090 (128K ctx, ThinkingCap): baseline 74.8 tok/s — currently at 74.9 (100%) - RTX 5070 (128K ctx, HauhauCS QAT): baseline 165.2 tok/s — currently at 169.6 (103%) - - Strix Halo (128K ctx, qwen3.6-35B-udq4): baseline 70.5 tok/s — currently at 62.9 (89%) + - Strix Halo (256K ctx, Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_M.gguf): baseline 70.5 tok/s — currently at 62.9 (89%) - **Fix**: - If tok/s > baseline → context has headroom, consider increasing - If tok/s < 90% baseline → reduce context by 25% and retest @@ -157,13 +157,14 @@ Key notes: ### Rule 10: Workload Distribution Optimization (updated 2026-07-18) - **Detect**: GPU roles misaligned with hardware capabilities - **Target distribution**: - - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM). - - RTX 5070 (gpu-light, 12GB, ~145 tok/s) → Vision (image+text), web search, lightweight tasks. Weight: 0.15 (LiteLLM). - - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM). + - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). + - RTX 5070 (gpu-vision, 12GB, ~145 tok/s) → Vision (image+text), web search, lightweight tasks. + - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). - **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first. +- **Weights are not restated here** — the live `syslog-auto` pool weights and rpm caps live in CT 116 `/opt/inference-harness/litellm_config.yaml`, the single source of truth. - **Fix**: - Alert if any GPU is handling workload outside its designated role - - Recommend agent alias updates to match workload to GPU role (use stable aliases: gpu-dense, gpu-light, strix-moe) + - Recommend agent alias updates to match workload to GPU role (use stable aliases: gpu-dense, gpu-vision, strix-moe) - Track per-GPU request distribution via LiteLLM spend logs - **Verify**: Each GPU's request pattern matches its designated role within 24h - **Escalate**: If role mismatch persists >48h → agent alias audit needed @@ -311,6 +312,6 @@ Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchab If monitor response > 1MB, log a warning and skip the cycle rather than crashing. ### L6: Stable Aliases Replace Model Names -- gpu-fleet introduced stable aliases (strix-moe, gpu-dense, gpu-light) on 2026-07-15. +- gpu-fleet introduced stable aliases (strix-moe, gpu-dense, gpu-light) on 2026-07-15; `gpu-light` was superseded by `gpu-vision` on 2026-09-12. - Self-heal must use aliases for reporting and alerting, not model-specific names. - **Rule**: All alert messages and KG nodes use the stable alias as the GPU identifier. diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 5fcd5f1..99686e0 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -5,12 +5,30 @@ version: 1.0.0 description: > Canonical known-good baseline for all Syslog Hermes agents. Captures the exact configuration state, keys, workarounds, and audit procedure. When an agent's - configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 128K context (reduced from 256K for stability Jul 2026) (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo). + configuration goes sideways, restore from this baseline. Last verified 2026-07-16. RTX 3090/5070 at 128K (reduced from 256K for stability Jul 2026); Strix Halo at 256K (2026-09-12) (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo). author: Abiba (pi agent) --- # Hermes Agent Baseline — Canonical Good State +## Reachability Detection + +Before checking agent baseline, verify the host is reachable and can be audited. Use the shared reachability helper from the clone root: + +```bash +# Run on each host to check reachability (Tanko, Mumuni, Koonimo, Koby) +scripts/hermes-reachability-check.sh "api_key:" "/root/.hermes/config.yaml" +# Example: scripts/hermes-reachability-check.sh 192.168.68.122 "api_key:" "/root/.hermes/config.yaml" + +# Expected outcomes: +# - UNREACHABLE: SSH connection failed (host is down) +# - VIOLATION: SSH succeeded and found matches (report the finding) +# - COMPLIANT: SSH succeeded and found no matches (no api_key in config) +# +# NOTE: The bug this replaces was deriving reachability from the remote grep's exit code. +# The correct pattern: remote side always succeeds (grep ...; true), so ssh status = connection only. +``` + ## Quick Restore ```bash @@ -24,11 +42,12 @@ done | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | |-------|-----|------|-----|---------------|------------|----------| -| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** | +| Koby | 111 | storepve | .129 | `koby` | Infisical vault | **Hermes** | | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | > **Note**: CT hostnames (tdunna→CT111, baggy→CT113) differ from agent identities (koby, koonimo). +> CT 111 (tdunna, 192.168.68.129, storepve) is report-only — Theo's box; alert only, never garbage-collect. Access: `pct-run ` — no IPs needed. GPU hosts (.8, .110, .15) use SSH. Keys are stored in Infisical vault (project=agents, env=production) and injected at @@ -80,7 +99,7 @@ custom_providers: auxiliary: vision: provider: harness - model: gemma-4-12b # or syslog-auto + model: gpu-vision # RTX 5070 stable alias (Rule 8; do not use syslog-auto for aux) base_url: http://192.168.68.116/litellm/v1 api_key_env: LITELLM_API_KEY api_key: # ← MANDATORY workaround @@ -95,13 +114,33 @@ auxiliary: threshold: 0.65 target_ratio: 0.3 provider: harness - model: syslog-auto # or gemma-4-12b + model: syslog-auto # Rule 7: compression must be syslog-auto base_url: http://192.168.68.116/litellm/v1 api_key_env: LITELLM_API_KEY api_key: # ← MANDATORY workaround timeout: 120 ``` +## Violation Classification + +When reporting findings, separate POLICY observations from FAULT findings: + +### POLICY (observation only, not a fault) +- Agent uses a non-internal-harness provider (e.g., direct DeepSeek, Tencent, OpenRouter) +- Config text has a field that looks unusual but the agent's calls are succeeding +- Example: "POLICY: Koonimo uses deepseek directly; calls succeeding in last hour" + +### FAULT (requires request-level evidence) +- Agent's calls are failing with auth errors (401/403 in logs) +- Agent's config has no valid API key AND calls are failing +- Example: "FAULT: Koby's LiteLLM key expired; 401 observed at 2026-09-14 11:42:00" + +### Rules +1. Do NOT infer the runtime's credential resolution from config text alone. +2. Require request-level evidence before calling something a FAULT: an observed auth failure in the agent's log, or the absence of successful calls in the window. +3. If calls are succeeding, the correct output is "POLICY: uses directly; calls succeeding" - not a violation. +4. State what you OBSERVED, not what the field implies. + ## Known Bug: `api_key_env` Ignored by Auxiliary Client **Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478) @@ -185,7 +224,9 @@ Abiba (CT100) runs pi via PM2 with the Zulip extension. Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`. **models.json** — Must only list models authorized for the agent's LiteLLM key. -Key is injected via `infisical run --` wrapper at PM2 startup: +`/v1/models` is key-scoped and the live registry is CT 116 +`/opt/inference-harness/litellm_config.yaml`; treat the list below as a snapshot and re-read +the registry before applying. Key is injected via `infisical run --` wrapper at PM2 startup: ```json { "providers": { @@ -197,9 +238,7 @@ Key is injected via `infisical run --` wrapper at PM2 startup: { "id": "syslog-auto" }, { "id": "strix-moe" }, { "id": "gpu-dense" }, - { "id": "gpu-light" }, - { "id": "qwen3.6-27B-code" }, - { "id": "gemma-4-12b" } + { "id": "gpu-vision" } ] } } diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index 076ff04..0a1a6ef 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -19,6 +19,24 @@ description: > - agent_keys: map (see Agent Keys section) - infra_endpoints_verified: array +## Reachability Detection + +Before auditing the config template, verify the host is reachable and can be checked. Use the shared reachability helper from the clone root: + +```bash +# Run on each host to check reachability (Tanko, Mumuni, Koonimo, Koby) +scripts/hermes-reachability-check.sh "base_url:" "/root/.hermes/config.yaml" +# Example: scripts/hermes-reachability-check.sh 192.168.68.122 "base_url:" "/root/.hermes/config.yaml" + +# Expected outcomes: +# - UNREACHABLE: SSH connection failed (host is down) +# - VIOLATION: SSH succeeded and found matches (report the finding) +# - COMPLIANT: SSH succeeded and found no matches (no base_url in config) +# +# NOTE: The bug this replaces was deriving reachability from the remote grep's exit code. +# The correct pattern: remote side always succeeds (grep ...; true), so ssh status = connection only. +``` + ## Agent Keys (LiteLLM — Current 2026-07-11) Each agent has a unique LiteLLM API key (virtual key) generated against the LiteLLM @@ -92,17 +110,17 @@ work immediately after restart. ```yaml # ─── Model Selection ─── model: - default: # e.g., strix-moe, qwen3.6-27B-code, syslog-auto + default: # e.g., strix-moe, gpu-dense, syslog-auto provider: harness base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated path; /v1 also OK api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation - context_length: 131072 # For syslog-auto (all GPUs at 128K for stability). + context_length: 131072 # Conservative floor for syslog-auto (NVIDIA hosts 128K; Strix Halo 256K). # ⚠️ MANDATORY: Hermes probes unknown models from 256K # and falls back to 256K when /v1/models lacks a context # field (llama-server does). Without this override, agents # silently run syslog-auto at 256K (verified 2026-08-09). - # Set 65536 if using gemma-4-12b directly (tight VRAM). + # Set 65536 if pinning a single model directly (tight VRAM). fallback_providers: provider: deepseek @@ -133,7 +151,7 @@ compression: enabled: true model: syslog-auto # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name). provider: harness - max_context_window: 131072 # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17). + max_context_window: 131072 # MUST stay at the syslog-auto pool floor: NVIDIA hosts are 128K, Strix Halo 256K (2026-09-12). threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K target_ratio: 0.30 protect_last_n: 40 @@ -143,25 +161,25 @@ compression: # ─── Auxiliary Tasks (CONSISTENCY RULE) ─── # All auxiliary services MUST use identical model, base_url, and api_key_env: -# model: gpu-light # stable alias (NOT raw "gemma-4-12b") +# model: gpu-vision # stable alias (NOT a raw model name) # base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK # api_key_env: LITELLM_API_KEY # Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU. -# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning. +# gpu-vision = RTX 5070 (12B), freeing the Strix Halo for agent reasoning. # Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead. -# NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4) -# in agent configs — use the stable aliases so model swaps don't break agents. +# NEVER use retired model names (qwen3.6-27B-code, qwen3.6-35B-udq4; gemma-4-12b is retired +# and no longer resolves) in agent configs — use the stable aliases so model swaps don't break agents. auxiliary: vision: provider: harness - model: gpu-light # stable alias for RTX 5070 (was raw gemma-4-12b) + model: gpu-vision # stable alias for RTX 5070 base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK api_key_env: LITELLM_API_KEY timeout: 60 download_timeout: 30 web_extract: provider: harness - model: gpu-light # stable alias for RTX 5070 + model: gpu-vision # stable alias for RTX 5070 base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK api_key_env: LITELLM_API_KEY timeout: 30 @@ -173,10 +191,10 @@ auxiliary: timeout: 300 # gpu-fleet: 300s for large-history summarization (was 60) # ─── Delegation / Heavy Aux (use gpu-dense = RTX 3090) ─── -# delegation.model and x_search.model use gpu-dense (NOT raw qwen3.6-27B-code). +# delegation.model and x_search.model use gpu-dense (NOT retired raw name). delegation: - model: gpu-dense # stable alias for RTX 3090 (was raw qwen3.6-27B-code) + model: gpu-dense # stable alias for RTX 3090 provider: harness base_url: http://192.168.68.116/litellm/v1 # Rule 5 (2026-08-09): canonical authenticated; /v1 also OK api_key_env: LITELLM_API_KEY @@ -199,6 +217,26 @@ When LiteLLM keys are regenerated (e.g., after infrastructure changes): 3. **After update**: Restart Hermes on the agent host 4. **Verify**: `curl -H "Authorization: Bearer sk-" http://192.168.68.116/v1/models` +## Violation Classification + +When reporting findings, separate POLICY observations from FAULT findings: + +### POLICY (observation only, not a fault) +- Agent uses a non-internal-harness provider (e.g., direct DeepSeek, Tencent, OpenRouter) +- Config text has a field that looks unusual but the agent's calls are succeeding +- Example: "POLICY: Koonimo uses deepseek directly; calls succeeding in last hour" + +### FAULT (requires request-level evidence) +- Agent's calls are failing with auth errors (401/403 in logs) +- Agent's config has no valid API key AND calls are failing +- Example: "FAULT: Koby's LiteLLM key expired; 401 observed at 2026-09-14 11:42:00" + +### Rules +1. Do NOT infer the runtime's credential resolution from config text alone. +2. Require request-level evidence before calling something a FAULT: an observed auth failure in the agent's log, or the absence of successful calls in the window. +3. If calls are succeeding, the correct output is "POLICY: uses directly; calls succeeding" - not a violation. +4. State what you OBSERVED, not what the field implies. + ## Configuration Rules ### Rule 1: Shared Infra Is Locked @@ -252,10 +290,11 @@ The following MUST be identical across ALL profiles: - For agents needing longer outputs: raise to 8192, but never omit ### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16) -- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized) -- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized) -- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b` - (it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls. +- Vision and web_extract use `gpu-vision` (RTX 5070 — 12GB, vision-optimized) +- Compression uses `syslog-auto` — the Strix Halo weighted pool (64GB, 256K ctx, compression-optimized); do NOT pin `compression.model` to `strix-moe` (audit Rule 7 rejects it) +- **`ornith-1.0-35b` is NOT a valid compression model name** — LiteLLM does not serve it + (do not restate the served model list here — CT 116 `/opt/inference-harness/litellm_config.yaml` + is the single source of truth for models, aliases, weights and fallbacks). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls. - **OPERATIONAL DECISION (2026-07-23): Use `syslog-auto` for compression across all agents.** The `syslog-auto` alias routes to the Strix Halo, but uses the weighted pool instead of pinning to `strix-moe` directly. This prevents sustained Strix Halo thermal load because the pool can @@ -264,41 +303,41 @@ The following MUST be identical across ALL profiles: - All auxiliary services MUST use identical routing: - `base_url: http://192.168.68.116/litellm/v1` (Rule 5, 2026-08-09: canonical authenticated; `/v1` also OK) - `api_key_env: LITELLM_API_KEY` -- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably +- **Do NOT use `syslog-auto` for `vision`/`web_extract`** — it routes unpredictably; compression is the deliberate exception (see the OPERATIONAL DECISION above) - **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo - (64GB UMA, 128K context) — the designated compression GPU. This frees the + (64GB UMA, 256K context) — the designated compression GPU. This frees the RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning. - The `compression:` block's `model` MUST match `auxiliary: compression: model` -- The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K) +- The `compression: max_context_window: 131072` MUST stay at the syslog-auto pool floor (NVIDIA hosts 128K; Strix Halo 256K) ### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16) -- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations -- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K) -- **Strix Halo (64GB, 128K ctx, syslog-auto)**: Context compression, summarization, long docs +- **RTX 3090 (24GB, 128K ctx, gpu-dense)**: Heavy reasoning, code gen, long conversations +- **RTX 5070 (12GB, 128K ctx, gpu-vision)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K) +- **Strix Halo (64GB, 256K ctx, syslog-auto)**: Context compression, summarization, long docs - Agent profiles MUST route auxiliary tasks to the correct GPU: - - `auxiliary.vision.model: gemma-4-12b` (RTX 5070) - - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070) + - `auxiliary.vision.model: gpu-vision` (RTX 5070) + - `auxiliary.web_extract.model: gpu-vision` (RTX 5070) - `auxiliary.compression.model: syslog-auto` (Strix Halo) - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing - For 128K context window: `threshold: 0.65` (fires at ~85K tokens) - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss - Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin -- `max_context_window: 131072` MUST match the model's actual capacity (128K) +- `max_context_window: 131072` MUST stay at the pool floor (NVIDIA hosts 128K; Strix Halo 256K) - See `devops-hermes-compression` skill for full reference ### Rule 9: Compression Threshold for 128K Models - For 128K context window: `threshold: 0.65` (fires at ~85K tokens) - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss - Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin -- `max_context_window: 131072` MUST match the model's actual capacity (all GPUs = 128K) +- `max_context_window: 131072` MUST stay at the pool floor (NVIDIA hosts 128K; Strix Halo 256K) - See `devops-hermes-compression` skill for full reference ### Rule 10: Default Model Must Be `syslog-auto` (All Agents) - **Koby Exception**: Per captain ruling 2026-08-11, Koby is a DeepSeek-primary external agent; its primary model remains `deepseek-v4-flash` (via api.deepseek.com to preserve DeepSeek-specific reasoning, while other sections follow Rule 10. - **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto` - **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json` -- `syslog-auto` is the LiteLLM routing model — it load-balances between strix-moe - and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against: +- `syslog-auto` is the LiteLLM routing model — it load-balances across the live pool + (see CT 116 `/opt/inference-harness/litellm_config.yaml` for the current members and weights). Using it protects against: - Model name typos that cause 403 errors and silent worker failures - Single GPU downtime (routing falls back automatically) - Key/model authorization mismatches @@ -321,7 +360,8 @@ When an agent shows "context issues" (premature compression, 401s, 504s, DeepSee verify ALL FOUR of these against the live config. They are the only root causes found in production: 1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window` - MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used. + MUST be `131072` (the syslog-auto pool floor: NVIDIA hosts are 128K; Strix Halo is 256K). + A `262144` client window can route to a 128K NVIDIA host and fail, so it must NOT be used. ~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml` 2. **base_url uses authenticated path?** — `custom_providers[0].base_url`, `delegation.base_url`, and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/litellm/v1` (Rule 5, canonical) diff --git a/hermes-key-enforcement.prose.md b/hermes-key-enforcement.prose.md index 5e46736..a13a034 100644 --- a/hermes-key-enforcement.prose.md +++ b/hermes-key-enforcement.prose.md @@ -117,6 +117,44 @@ model: api_key_env: LITELLM_API_KEY ``` +## Reachability Detection + +Before checking for hardcoded keys, verify the host is reachable and can be audited. Use the shared reachability helper from the clone root: + +```bash +# Run on each host to check reachability (Tanko, Mumuni, Koonimo, Koby) +scripts/hermes-reachability-check.sh "api_key: sk-" "/root/.hermes/" +# Example: scripts/hermes-reachability-check.sh 192.168.68.122 "api_key: sk-" "/root/.hermes/" + +# Expected outcomes: +# - UNREACHABLE: SSH connection failed (host is down) +# - VIOLATION: SSH succeeded and found matches (report the finding) +# - COMPLIANT: SSH succeeded and found no matches (no hardcoded keys in config) +# +# NOTE: The bug this replaces was deriving reachability from the remote grep's exit code. +# The correct pattern: remote side always succeeds (grep ...; true), so ssh status = connection only. +``` + +## Violation Classification + +When reporting findings, separate POLICY observations from FAULT findings: + +### POLICY (observation only, not a fault) +- Agent uses a non-internal-harness provider (e.g., direct DeepSeek, Tencent, OpenRouter) +- Config text has a field that looks unusual but the agent's calls are succeeding +- Example: "POLICY: Koonimo uses deepseek directly; calls succeeding in last hour" + +### FAULT (requires request-level evidence) +- Agent's calls are failing with auth errors (401/403 in logs) +- Agent's config has no valid API key AND calls are failing +- Example: "FAULT: Koby's LiteLLM key expired; 401 observed at 2026-09-14 11:42:00" + +### Rules +1. Do NOT infer the runtime's credential resolution from config text alone. +2. Require request-level evidence before calling something a FAULT: an observed auth failure in the agent's log, or the absence of successful calls in the window. +3. If calls are succeeding, the correct output is "POLICY: uses directly; calls succeeding" - not a violation. +4. State what you OBSERVED, not what the field implies. + ## Detection Query Run on any Hermes host to detect violations: @@ -169,16 +207,22 @@ The agent picks up the new key via `infisical run --` at gateway startup. **Keys are permanent and use bare agent name aliases.** -- **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`. +- **Duration**: `null` — keys never expire by default. **Expiry must be set EXPLICITLY at creation** with the `duration` parameter (e.g., `90d` for 90 days). The 90-day default is the standard; however, the config default is **NOT honoured** by LiteLLM 1.99.1 (verified on CT 116: a key generated with no explicit duration returns `expires=null`). This has been recorded in `/opt/inference-harness/litellm_config.yaml` to prevent re-filing as a bug. +- **Daily Audit**: A daily audit job runs at 00:00 UTC (`/usr/local/bin/litellm-key-renewal-ct116.sh`, cron 00:00). It is **AUDIT-ONLY** and does not perform renewal. It lists every key, reports those with no expiry and those inside a 14-day warning window, explicitly EXCLUDES `abiba-pi` and `koby` (report-only, and .129 must never be touched), and logs `RENEWAL-REQUIRED-BUT-NOT-PERFORMED + NO KEY WAS CHANGED` when renewal is skipped. **Renewal is NOT implemented** — keys must not be rotated until delivery (vault injection + consumer verification) exists and is proven end-to-end. +- **Exclusions**: `abiba-pi` and every firstmate/secondmate/crewmate key stay **WITHOUT an expiry** until a proven renewal path exists. `koby` is **report-only** (never touched). These exclusions are enforced by the audit job. - **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity. -- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. +- **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven. Manual rotation is permitted only when the renewal delivery path is proven and verified on a throwaway consumer before production use. - **Max budget**: $100 per key (config default). ```yaml -# In litellm_config.yaml — ensures all future keys inherit these defaults: +# NOT currently set in the authority; recommended value. CT 116 litellm_config.yaml has no +# default_key_generate_params block today, and a key generated with no explicit models comes back +# with an EMPTY models list. `models` is a literal key-generation parameter, so this is a value to +# ADD — re-read the live registry at CT 116 /opt/inference-harness/litellm_config.yaml and +# re-verify before applying. litellm_settings: default_key_generate_params: - models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b"] + models: ["syslog-auto", "gpu-dense", "gpu-vision", "strix-moe"] duration: null # ← permanent max_budget: 100 metadata: @@ -286,13 +330,13 @@ auxiliary: api_key: sk- # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain) api_key_env: LITELLM_API_KEY base_url: http://192.168.68.116/litellm/v1 - model: gemma-4-12b + model: gpu-vision provider: harness compression: api_key: sk- # ← workaround (same as above) api_key_env: LITELLM_API_KEY base_url: http://192.168.68.116/litellm/v1 - model: gemma-4-12b + model: syslog-auto provider: harness ``` diff --git a/hermes-zulip-plugin.prose.md b/hermes-zulip-plugin.prose.md index 8dae1cf..26c8458 100644 --- a/hermes-zulip-plugin.prose.md +++ b/hermes-zulip-plugin.prose.md @@ -47,7 +47,7 @@ connectivity recovery including end-to-end DM validation. ## Requires -- SSH access to target host (direct or via amdpve for CTs) +- SSH access to target host (direct, or via the guest's Proxmox node for CTs) - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Python 3 with `httpx` installed on target @@ -56,7 +56,7 @@ connectivity recovery including end-to-end DM validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* | -| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | +| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Field | Value | Trust | @@ -72,7 +72,7 @@ connectivity recovery including end-to-end DM validation. ### Step 1: Resolve Target Map `target` to host, CT ID, hermes_home, and user from the live-state table. -For CT112 and CT111, route through `ssh root@amdpve` then `pct exec `. +For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec `. ### Step 2: Pull Latest Plugin Source diff --git a/hermes-zulip-restore.prose.md b/hermes-zulip-restore.prose.md index bafb5a3..fb6169d 100644 --- a/hermes-zulip-restore.prose.md +++ b/hermes-zulip-restore.prose.md @@ -43,7 +43,7 @@ gateway restart, and connection validation. ## Requires -- SSH access to target host (direct or via amdpve for CTs) +- SSH access to target host (direct, or via the guest's Proxmox node for CTs) - Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` - Python 3 with `httpx` installed on target - Zulip server accessible at `https://chat.sysloggh.net` @@ -52,7 +52,7 @@ gateway restart, and connection validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| -| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | +| Koby | CT111 | storepve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Field | Value | Trust | @@ -67,7 +67,7 @@ gateway restart, and connection validation. ### Step 1: Locate Target Map `target` to connectivity parameters from the live-state table above. -For CT112 and CT111, route through `ssh root@amdpve` then `pct exec `. +For CT112 route through `ssh root@amdpve`; for CT111 route through `ssh root@storepve` — then `pct exec `. ### Step 2: Deploy Zulip Adapter diff --git a/inference-optimization.prose.md b/inference-optimization.prose.md index fd28d22..ce660c5 100644 --- a/inference-optimization.prose.md +++ b/inference-optimization.prose.md @@ -4,7 +4,7 @@ kind: responsibility description: > Optimizes the full Syslog inference stack — LiteLLM routing weights, GPU model assignments, agent context management, and prompt caching — to reduce response - times to sub-15s average. All GPUs now at 128K context (stable ceiling). + times to sub-15s average. NVIDIA GPUs at 128K context; Strix Halo at 256K (2026-09-12). id: 067NC6KP02RG60S50M40E30928 --- @@ -23,7 +23,7 @@ management, and prompt caching — without sacrificing agent capability. .123, any others on .129/.122) including compression, model, context_window, prompt_caching, memory settings - `gpu-health`: health check response from all 3 GPU backends (strix-moe .15:8080, - qwen .8:8080, gemma .110:8080) + gpu-dense .8:8080, gpu-vision .110:8080) ### Maintains @@ -56,14 +56,14 @@ duration. **Context is the root cause.** Every ~46K prompt token costs ~87s of prefill time at 532 tok/s. Fix context first, routing second. -- **Route by task**: qwen for code/standard queries; gemma for - compression/auxiliary; strix-moe for compression tasks. +- **Route by task**: gpu-dense for code/standard queries; gpu-vision for + vision/web-auxiliary; syslog-auto for compression. - **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should compact at 51K, not 85K. Target 15% tail (not 30%). - **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these never change between turns. Single-digit cache hit rate is unacceptable. - **Lower context ceiling**: 128K window is the stable ceiling for agent conversations. - GPUs reduced from 256K to 128K (2026-07-17). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers. + GPUs reduced from 256K to 128K (2026-07-17) for the NVIDIA hosts; Strix Halo runs 256K (2026-09-12). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers. ### Shape @@ -96,8 +96,10 @@ call enable-prompt-caching hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110] -- Phase 5: Verify end-to-end latency +-- `models` is a literal verification parameter (a snapshot only): the authoritative registry is +-- CT 116 /opt/inference-harness/litellm_config.yaml; re-read it before use. call verify-latency host: 192.168.68.116 - models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b] + models: [syslog-auto, gpu-dense, gpu-vision, strix-moe] ``` diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index b9b682b..87f64b7 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -16,8 +16,8 @@ description: > **Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster (now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve (192.168.68.4) is a standalone PVE node + NetBird routing peer; - London relocation pending. CTs 100 (abiba) and 105 (kagentz) moved - to minipve. + London relocation pending. CT 100 (abiba) is on minipve; CT 105 + (kagentz) is on amdpve. --- # Infrastructure Control Pattern @@ -105,16 +105,16 @@ description: > | Node | IP | CPU | RAM | VMs/CTs | Role | |------|----|-----|-----|---------|------| -| minipve | .12 | 16C | 30GB | abiba, kagentz, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | -| amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute | -| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat | +| minipve | .12 | 16C | 30GB | abiba, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging | +| amdpve | .15 | 32C | 62GB | kagentz, tanko, baggy, scottdenya, adguard2 | Agents, compute | +| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna | Docker, storage, chat | | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | -> **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on -> minipve at .10, not acerpve. Abiba (CT 100) and kagentz (CT 105) are on -> minipve (moved from hwepve 2026-08-15). Mumuni runs inside Abiba CT100 -> (.24); CT 114 (mumuni) no longer exists in the cluster. +> **Note:** CTs on storepve include jdownloader (CT 118) and tdunna (CT 111). +> AdGuard (CT 102) is on minipve at .10, not acerpve. Abiba (CT 100) is on +> minipve (moved from hwepve 2026-08-15); kagentz (CT 105) is on amdpve. +> Mumuni runs inside Abiba CT100 (.24); CT 114 (mumuni) no longer exists in the cluster. > > **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):** > Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve. @@ -222,8 +222,8 @@ description: > **Prometheus targets**: - 192.168.68.8:9400 (RTX 3090 — qwen) -- 192.168.68.110:9400 (RTX 5070 — gemma) -- 192.168.68.15:9400 (Strix Halo — qwen3.6-35B-udq4) +- 192.168.68.110:9400 (RTX 5070 — gpu-vision) +- 192.168.68.15:9400 (Strix Halo — strix-moe) - harness-litellm:4000 (LiteLLM health) ### Ecosystem C: Netbird (72.61.0.17 — Hostinger srv1079750.hstgr.cloud) @@ -364,6 +364,79 @@ For docker-vm specifically: - No PBS backup in 48h → fail ``` +### Backup Safety Preconditions (2026-09-15) + +#### Background & Rationale + +Two incidents from 2026-09-13/14 demonstrate that backup operations can catastrophically fail when storage conditions are not verified first: + +1. **acerpve thin-pool VM 101** (acerpve, 192.168.68.9, 2026-09-13): A snapshot-mode vzdump of VM 101 on acerpve filled the LVM thin pool. `dmsetup status pve-data-tpool` showed `thin-pool Error` (then `Fail`), the host root remounted `emergency_ro`, ordinary commands failed with I/O errors, LVM tools returned nothing and VM 101 (the RTX 3090 host) went unreachable while the host still answered ping and ssh. It happened TWICE in one day with different modes: snapshot at 13:39Z and a `--mode stop` cold run at 18:52Z. Both times a reboot rolled the failed transaction back and the pool returned rw (~30% data, ~1.2% metadata). Pool capacity was NOT the obvious explanation - ~816G with ~572G free - which is why the metadata/snapshot-pressure hypothesis stands unproven. A full or errored thin pool fails EVERY volume on the VG at once, including the host root. + +2. **amdpve 0700 tmpdir** (amdpve, 192.168.68.15, 2026-09-14): A custom vzdump `tmpdir` created with mode 0700 broke a whole night of container backups: `fstat "/vzdumptmp_//." failed - EACCES`, because the archive step runs through an unprivileged user namespace and could not traverse a root-owned 0700 directory. Fixed with `chmod 1777` (match /var/tmp) and proved with a real backup. + +3. **acerpve GPU-host fact**: VM 101 (llm-gpu) and VM 103 (ocu-llm) are in NO scheduled job, so their only coverage is one-off runs - and for VM 101 that is deliberate until the thin-pool is understood. + +> ⚠️ **Hostname Resolution Warning (2026-09-15)**: The PVE node hostnames (acerpve, amdpve, minipve, storepve, ocupve) all resolve to the VPS (72.61.0.17, the Netbird VPS at srv1079750.hstgr.cloud) via the wildcard `*.dns.sysloggh.net` record, NOT to the actual nodes. So `ssh acerpve` lands on the VPS. **Nodes must be addressed by IP**: acerpve 192.168.68.9, amdpve 192.168.68.15, storepve 192.168.68.6, minipve 192.168.68.12, ocupve 192.168.68.5. Guest CTs are reached through their node (`pct exec`). Guest hostnames that resolve on the LAN (e.g. kagentz = 192.168.68.14) are fine. (The DNS address records are a separate decision — row: dag-daemon-node-hostnames-resolve-to-the-vps-20260915.) + +#### PREFLIGHT Preconditions (Before ANY snapshot-mode backup on thin-pool hosts) + +Before starting ANY snapshot-mode vzdump on a host whose storage is an LVM thin pool, the following checks MUST pass: + +```bash +# Check 1: Pool headroom (PRIMARY - yields percentages directly) +# Run on the NODE (e.g. ssh root@192.168.68.9 for acerpve) — NOT by bare hostname, see warning above +lvs -o lv_name,data_percent,metadata_percent,lv_size pve/data +# Example output (acerpve, 192.168.68.9): +# LV Data% Meta% LSize +# data 29.95 1.22 <816.21g +# Required thresholds (documented minimum): +# data_percent < 90% (80% recommended for safety margin) +# metadata_percent < 70% (metadata fills faster than data) + +# Check 2: Verify pool is not in error state (dmsetup shows the raw DM device) +# Run on the NODE (e.g. ssh root@192.168.68.9 for acerpve) — NOT by bare hostname +dmsetup status pve-data-tpool | grep -q "Error\|Fail" && exit 1 +# dmsetup status pve-data-tpool field order (verified on 192.168.68.9): +# $1=start $2=length $3="thin-pool" $4=transaction-id +# $5=metadata_used/metadata_total (blocks) $6=data_used/data_total (sectors) +# remaining fields are flags ("-", "rw", "discard_passdown", "queue_if_no_space", ...) +# This is only used for the ERROR-STATE check; use the lvs command above for percentages. +# metadata_percent = $5 / ($5 split by /) [second number in pair] +``` + +**Minimum thresholds**: If either `data_percent >= 90%` or `metadata_percent >= 70%`, the backup MUST NOT start. State explicitly that these are hard stops, not warnings. + +**Why this is a precondition**: A full or errored thin pool fails EVERY volume on the VG at once, including the host root. This is not a soft failure - it takes down the entire Proxmox host. + +#### Staging Directory Requirement (2026-09-14 incident) + +Any custom vzdump `tmpdir` MUST be world-traversable and writable exactly like `/var/tmp` (mode 1777). The archive step of vzdump runs in an unprivileged user namespace and cannot traverse a root-owned 0700 directory. + +**Symptom to recognize**: `fstat "/vzdumptmp_//." failed - EACCES` on every container in the backup run. + +**Fix**: `chmod 1777 ` before starting vzdump. + +#### Task Start Rule for Truncating Shells + +When starting a backup task from a shell that may truncate output (e.g., pipes, `head`), always use: + +```bash +pvesh create /storage/backup --output-format json -- ... | head -2 +# ❌ Can kill the backup task ("broken pipe" status) +``` + +Instead, capture JSON output without piping to truncating commands: + +```bash +# Use --output-format json and capture to variable +result=$(pvesh create /storage/backup --output-format json -- ...) +# Then parse result if needed +``` + +#### GPU Host Backup Status (acerpve VM 101) + +VM 101 (llm-gpu) and VM 103 (ocu-llm) have NO scheduled backup job. Coverage is manual one-off runs only. This is intentional for VM 101 until the thin-pool failure mechanism is understood and documented. + ## Section 5: Network Services — Monitoring ### 5.1 Service Inventory @@ -602,13 +675,13 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 102 | adguard | **minipve** | **.10** | DNS | ❌ | | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ | | 104 | authentik | minipve | .11 | OIDC | ❌ | -| 105 | kagentz | minipve | — | Agent Zero | ✅ | +| 105 | kagentz | amdpve | — | Agent Zero | ✅ | | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP | | 107 | pbs | storepve | — | Backups | ❌ | | 108 | media | storepve | — | Media | ❌ | | 109 | docker-vm | storepve | .7 | Docker host | ❌ | | 110 | gitea | minipve | **.17** | Git | ❌ | -| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ | +| 111 | tdunna | storepve | .129 | Hermes agent — ⛔ REPORT-ONLY (Theo's box, no GC) | ✅ | | 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ | | 113 | baggy | amdpve | .114 | Hermes agent | ✅ | | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ | @@ -616,6 +689,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server" | 117 | zulip | storepve | .19 | Chat | ❌ | | 118 | jdownloader | storepve | .20 | JDownloader LXC (dedicated, migrated from docker-vm 2026-08-01) | ✅ | | 119 | infisical-vault | minipve | — | Vault | ❌ | +| 120 | adguard2 | amdpve | — | DNS (secondary AdGuard) | ❌ | ## Appendix C: Docker Compose Files Location @@ -636,8 +710,8 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run. | CT | Name | Node | pct-run | |-----|------|------|---------| | 100 | abiba | minipve | `pct-run 100` | -| 105 | kagentz | minipve | `pct-run 105` | -| 111 | tdunna | amdpve | `pct-run 111` | +| 105 | kagentz | amdpve | `pct-run 105` | +| 111 | tdunna | storepve | `pct-run 111` (⛔ report-only — no GC) | | 112 | tanko | amdpve | `pct-run 112` | | 113 | baggy | amdpve | `pct-run 113` | | 115 | scottdenya | amdpve | `pct-run 115` | @@ -648,6 +722,9 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run. | 107 | proxmox-backup | storepve | `pct-run 107` | | 108 | media | storepve | `pct-run 108` | | 117 | zulip | storepve | `pct-run 117` | +| 118 | jdownloader | storepve | `pct-run 118` | +| 119 | infisical-vault | minipve | `pct-run 119` | +| 120 | adguard2 | amdpve | `pct-run 120` | | 102 | adguard | **minipve** | `pct-run 102` | GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly: diff --git a/infrastructure-monitoring.prose.md b/infrastructure-monitoring.prose.md index c87c35a..e3e5ea9 100644 --- a/infrastructure-monitoring.prose.md +++ b/infrastructure-monitoring.prose.md @@ -17,7 +17,7 @@ description: > ⚠️ This contract is target-state aspirational — but GPU export + alerting are now as-built (verified 2026-08-09). As-built GPU monitoring is via gpu-monitor contract (port 9100 poll). -version: 1.0.0 +version: 1.0.1 --- ## Architecture @@ -120,127 +120,156 @@ the any-HTTP rule. On those — the authenticated Zulip POST and the router `/health` — an unexpected status (`401`/`403` from a bad or missing credential, `5xx`, or anything other than the expected `200`) is an **ALERT**, not "alive". +**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):** +1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the + service answered — report the code, never "down". A redirect is not a failure. + Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe, + and that is a statement about YOUR PROBE, not about the service. +2. **A failed probe is never a service verdict.** Print + `probe-failed: ` naming the exact URL/host/port and the failure + kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only + then report. Apply the same shape as scripts/disk-gc-scan.py. +3. **Say which probe produced each number.** "Grafana: 000" is unusable; + "Grafana http://192.168.68.116:3001/api/health -> connection timeout after 10s + (retried at 25s: also timeout)" is actionable. + ### check-health -**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.** +**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real +tool calls; never repeat a prior report unless a live probe fails.** + +**PROBE SHAPE (per standing rules above):** +- Every probe prints the target name + URL + HTTP code (or failure kind) +- Retry once on connection failure at longer timeout +- Any HTTP status = ALIVE; only 000/timeout/refused = probe-failed +- Report the actual probe command and its result, not a summary verdict ```bash # Provenance — run first; paste the absolute path into the report pwd -P -# Zulip API health (POST ping) -source /etc/litellm-monitor.env -ZULIP_USER="abiba-bot@chat.sysloggh.net" -curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${ZULIP_BOT_KEY}" -# Expected: 200 (HTTP 000 = unreachable/cache) +# ============================================================ +# 1. ZULIP API HEALTH (POST ping) — bare-200 probe +# ============================================================ +# NOTE: /etc/litellm-monitor.env exists only on CT 116, retrieve keys from CT 116 via: +zulip_key=$(ssh root@192.168.68.116 "grep ZULIP_BOT_KEY /etc/litellm-monitor.env | cut -d= -f2") +if [ -z "$zulip_key" ]; then + echo "credential-missing: ZULIP_BOT_KEY not found in /etc/litellm-monitor.env" +else + ZULIP_USER="abiba-bot@chat.sysloggh.net" + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${zulip_key}") + echo "Zulip API https://chat.sysloggh.net/api/v1/messages -> $code" + # Expected: 200 (bare-200 probe; any other status is an ALERT) +fi -# PM2 process health +# ============================================================ +# 2. PM2 PROCESS HEALTH +# ============================================================ pm2 jlist -# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service) +# Expected: 4/4 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner) +# spoton-service removed 2026-09-14 (not in live set) -# GPU exporters (may be down per DEPLOYMENT STATUS) -curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL" -curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL" -curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL" +# ============================================================ +# 3. GPU EXPORTERS — any-HTTP probe (metrics endpoint) +# ============================================================ +# Probe /metrics (the Prometheus scrape target), not bare / +for host in 192.168.68.8 192.168.68.110 192.168.68.15; do + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 "http://$host:9400/metrics") + if [ "$code" == "000" ]; then + # Retry with longer timeout + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 "http://$host:9400/metrics") + echo "GPU exporter http://$host:9400/metrics -> probe-failed: timeout (retried at 25s: still $code)" + else + echo "GPU exporter http://$host:9400/metrics -> $code" + fi +done +# Expected: 200 on all 3 hosts (RTX 3090, RTX 5070, Strix Halo) -# Router health (via nginx on port 80) -curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health -# Expected: 200 (Router is up and responding) +# ============================================================ +# 4. ROUTER HEALTH (via nginx on port 80) — bare-200 probe +# ============================================================ +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116/health) +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116/health) + echo "Router http://192.168.68.116/health -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "Router http://192.168.68.116/health -> $code" +fi +# Expected: 200 (bare-200 probe; any other status is an ALERT) -# LiteLLM health (via nginx on port 80) -curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health -# Expected: 301 → /litellm/health/liveliness (200 after redirect) — any HTTP status = alive +# ============================================================ +# 5. LITELLM HEALTH (via nginx on port 80) — any-HTTP probe +# ============================================================ +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116/litellm/health) +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116/litellm/health) + echo "LiteLLM http://192.168.68.116/litellm/health -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "LiteLLM http://192.168.68.116/litellm/health -> $code" +fi +# Expected: 301 → /litellm/health/liveliness (any HTTP status = ALIVE) -# PVE API liveness — probe the REAL PVE nodes on :8006, never the monitoring -# host CT 116. CT 116 runs no pveproxy, so probing it on :8006 returns 000 — -# that was the stale-vantage bug this replaces (CT 116 is the monitoring host, -# not a cluster node). Unauthenticated GET answers 401 while the API is ALIVE -# by design. Alive = ANY HTTP status (401 is the EXPECTED healthy response); -# DOWN = connection refused (000) or timeout only. +# ============================================================ +# 6. PVE API LIVENESS — any-HTTP probe (auth-gated) +# ============================================================ +# Probe the REAL PVE nodes on :8006, never the monitoring host CT 116. for node in 192.168.68.9 192.168.68.5 192.168.68.15 192.168.68.6 192.168.68.12; do - printf '%s:8006 -> %s\n' "$node" \ - "$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 5 "https://$node:8006/api2/json/version")" + code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 10 "https://$node:8006/api2/json/version") + if [ "$code" == "000" ]; then + code=$(curl -sk -o /dev/null -w '%{http_code}' --connect-timeout 25 "https://$node:8006/api2/json/version") + echo "PVE API https://$node:8006/api2/json/version -> probe-failed: timeout (retried at 25s: still $code)" + else + echo "PVE API https://$node:8006/api2/json/version -> $code" + fi done # Expected: 401 on every node (acerpve .9, ocupve .5, amdpve .15, storepve .6, minipve .12) -# A node answering 000/timeout is DOWN — flag that node. 401 is NOT a fault. +# 401 is the EXPECTED healthy response (auth-gated); 000/timeout = DOWN -# Prometheus targets -curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets' -# Expected: All targets UP (may show some down if exporters not deployed) +# ============================================================ +# 7. PROMETHEUS TARGETS — bare-200 probe +# ============================================================ +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:9090/api/v1/targets) +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:9090/api/v1/targets) + echo "Prometheus http://192.168.68.116:9090/api/v1/targets -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "Prometheus http://192.168.68.116:9090/api/v1/targets -> $code" +fi +# Expected: 200 (bare-200 probe; any other status is an ALERT) -# Grafana health -curl -s http://192.168.68.116:3001/api/health | jq '{status, version}' -# Expected: {"status":"ok","version":"..."} +# ============================================================ +# 8. GRAFANA HEALTH — any-HTTP probe +# ============================================================ +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:3001/api/health) +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:3001/api/health) + echo "Grafana http://192.168.68.116:3001/api/health -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "Grafana http://192.168.68.116:3001/api/health -> $code" +fi +# Expected: 200 (any HTTP status = ALIVE; 000/timeout = DOWN) -# LiteLLM metrics (Prometheus endpoint) -curl -s http://192.168.68.116:4000/metrics | head -20 -# Expected: Prometheus-formatted metrics output +# ============================================================ +# 9. LITELLM METRICS (Prometheus endpoint) — any-HTTP probe +# ============================================================ +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.68.116:4000/metrics) +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://192.168.68.116:4000/metrics) + echo "LiteLLM metrics http://192.168.68.116:4000/metrics -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "LiteLLM metrics http://192.168.68.116:4000/metrics -> $code" +fi +# Expected: 200 (any HTTP status = ALIVE; 000/timeout = DOWN) ``` **Report format**: Begin every report with the **absolute path the probe executed -from** (`pwd -P`, or the script's absolute path) so a stale-consumer report is -distinguishable from a real fault at read time. Summarize actual results from -each probe. Apply the any-HTTP-response liveness rule ONLY to the auth-gated PVE -API and LiteLLM endpoints above: only connection-refused (`000`) or timeout is -DOWN; empty output is a warning. For probes whose expected result is a bare `200` -(the authenticated Zulip POST, router `/health`), flag an alert on any unexpected -status (`401`/`403`/`5xx`) — do not summarize it as alive. A bare-`200` -expectation on the auth-gated PVE API (`401`) or LiteLLM health (`301` redirect) -is a stale expectation, not a fault. - +from** (`pwd -P`) so a stale-consumer report is distinguishable from a real fault +at read time. For each probe, print the target name, the full URL, and the HTTP +code (or failure kind with retry details). Apply the standing probe rules: any +HTTP status = ALIVE; only 000/timeout/refused = probe-failed. A redirect is not +a failure. ### Phase 1: GPU Exporters **NVIDIA (.8 and .110)**: 1. Download `nvidia_gpu_exporter` binary -2. Create systemd service `nvidia-gpu-exporter.service` -3. Start and enable - -**AMD (.15)**: -1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py` -2. Parses `amdgpu_top --json -d 1000` output -3. Exposes key metrics at `:9400/metrics` via Python http.server -4. Create systemd service -5. Start and enable - -### Phase 2: Prometheus - -1. Create `/opt/monitoring/` directory on CT 116 -2. Write `prometheus.yml` with scrape configs for all targets -3. Add to docker-compose (or separate compose file) -4. Start container - -### Phase 3: Grafana - -1. Create `/opt/monitoring/grafana/` directories -2. Provision Prometheus datasource -3. Provision GPU fleet dashboard JSON -4. Provision LiteLLM dashboard JSON -5. Add to docker-compose -6. Start container - -### Phase 4: Verification - -1. Verify all 3 GPU exporters return 200 at :9400/metrics -2. Verify Prometheus targets all UP at :9090/targets -3. Verify Grafana accessible at :3001 with dashboards -4. Verify LiteLLM metrics flowing to Prometheus -5. ~~Update nginx to proxy `/monitoring/` → Grafana~~ (NOT recommended — nginx sub-path was tried for /grafana/ and reverted per proxmox-monitor; direct :3001 access is the standard) - -## Verification Commands - -```bash -# GPU exporters -curl -s http://192.168.68.8:9400/metrics | grep nvidia -curl -s http://192.168.68.110:9400/metrics | grep nvidia -curl -s http://192.168.68.15:9400/metrics | grep amdgpu - -# Prometheus -curl -s http://192.168.68.116:9090/api/v1/targets - -# Grafana -curl -s http://192.168.68.116:3001/api/health - -# LiteLLM metrics (already live) -curl -s http://192.168.68.116:4000/metrics | head -20 -``` diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index c42897d..9ea71a6 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -67,8 +67,10 @@ description: > 4. **If action == "create"**: - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date) - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" } - - Duration is null (permanent) — inherited from litellm default_key_generate_params - - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "strix-moe", "gpu-dense", "gpu-light", "qwen3.6-35B-udq4"] + - Duration is whatever the caller passes; NO default enforcement exists today (CT 116 `litellm_config.yaml` has no `default_key_generate_params` block, and a key with no explicit models returns an empty models list). Agent keys are permanent by policy, not by that block. OPEN policy question: should agent keys expire by default? (captain security-policy decision, raised separately.) + - Set models: read the live key-scoped set rather than hardcoding one — `/v1/models` is key-scoped, + and the authoritative registry is CT 116 `/opt/inference-harness/litellm_config.yaml`. Do not add + retired names (`gemma-4-12b`, `gpu-light`, `crew-auto` — all retired 2026-09-12). - Note: `ornith-1.0-35b` is NOT a valid LiteLLM model name (use `strix-moe`, the stable alias). qwen3.6-35B-A3B removed from fleet (was never deployed). - Return the new key 5. **If action == "rotate"**: @@ -318,7 +320,15 @@ directly call OpenRouter via Python's requests library. Converting would require ## LiteLLM Master Key (use sparingly — agents should NOT use it directly) -- Master key: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` (in /opt/inference-harness/.env on CT116, Infisical project=infrastructure env=production secret=LITELLM_MASTER_KEY) +- Master key: **Retrieval path (do not trust a literal value in this file — the key rotates)**: + ```bash + # PRIMARY (proven, runs on CT 116 with no extra tooling): + docker exec harness-litellm printenv LITELLM_MASTER_KEY + # Note: the same value is stored in /opt/inference-harness/.env on CT 116 (verified matching) + # The master key is NOT in the Infisical vault (project=infrastructure env=production does not contain it) + # Prove a key is live with a 200 from /key/list on the CT 116 host (the container has no curl): + curl -s -H "Authorization: Bearer " http://127.0.0.1:4000/key/list | jq length + ``` - Used for /key/generate, /key/delete, /key/list (GET), DB queries - **Known violation (RESOLVED 2026-07-16):** Abiba's LITELLM_API_KEY was previously the master key. It is now a dedicated agent key `sk-sxbphLvk1OU…` (vault secret `ABIBA_LITELLM_API_KEY`, alias `abiba-pi`). diff --git a/litellm-client-timeouts.prose.md b/litellm-client-timeouts.prose.md index 106fe37..810f1c1 100644 --- a/litellm-client-timeouts.prose.md +++ b/litellm-client-timeouts.prose.md @@ -26,9 +26,8 @@ description: > | Model | avg latency | avg TTFT | p-profile (24h) | |---|---|---|---| | syslog-auto | 28.8s | 25.4s | 68 calls took 30-120s; tail to ~300s under load | -| qwen3.6-27B-code | 23.0s | — | same backend class as syslog-auto | | strix-moe | 7.5s | — | Strix Halo, healthy | -| gemma-4-12b | 2.6s | — | RTX 5070, healthy | +| gpu-vision (retired gemma-4-12b, RTX 5070) | 2.6s | — | RTX 5070, healthy | Sep 6 incident timeline: failures 04:00-07:00 EDT (0% GPU util = wedged backend), full recovery 07:00-08:00 with ZERO client failures once requests @@ -53,13 +52,11 @@ proxy queuing. ### 2. Auxiliary tasks — keep template timeouts, one correction -- vision: 60s (keep), web_extract: 30s (keep) — gemma-4-12b averages 2.6s; - these are fine. +- vision: 60s (keep), web_extract: 30s (keep) — the 2.6s average was measured on `gemma-4-12b` (retired 2026-09-12); the live RTX 5070 alias is `gpu-vision`. - compression: 300s (keep — this was already raised from 60 per gpu-fleet). - **gpu-dense delegation/x_search: set timeout >= 120s.** The RTX 3090 - (qwen3.6-27B-code backend, 23.0s avg) is the same speed class as - syslog-auto; delegation defaults that assume fast responses will 408 the - same way. + (gpu-dense backend) is the same speed class as syslog-auto; delegation + defaults that assume fast responses will 408 the same way. ### 3. Retry policy — backoff, not repetition @@ -79,8 +76,9 @@ proxy queuing. Send a real model name and use a real (probe-designated) key. - Probe timeout: 30s. A probe that takes longer than 30s IS the alert — report "backend slow (>30s)" rather than hanging. -- Probe cadence: at most hourly. The 6h litellm-health cron cadence is the - standard; sub-hourly synthetic traffic distorts latency baselines. +- Probe cadence: at most hourly. The litellm-health cron cadence (authoritative + trigger in contract-registry.yaml) is the standard; sub-hourly synthetic + traffic distorts latency baselines. ### 5. Batch/benchmark jobs — schedule away from 04:00-07:00 EDT and chunk diff --git a/litellm-health.prose.md b/litellm-health.prose.md index ab502c8..6449828 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -1,14 +1,7 @@ --- kind: function name: litellm-health -status: deprecated -deprecated_on: 2026-07-09 -replaced_by: litellm-self-heal.prose.md -note: > - Consolidated into litellm-self-heal.prose.md to eliminate duplication - of architecture diagrams, GPU topology, timeout tables, and container - lists. Health check is now § Health Check within litellm-self-heal. - This file is retained for reference only — use litellm-self-heal instead. +status: active description: > Verifies the LiteLLM inference stack health. Current architecture (2026-07-09): nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy. @@ -51,9 +44,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) **What changed (v3.2.0 → v4.0.0 — 2026-07-08)**: - Router REMOVED from request path — LiteLLM proxies directly to GPU - All GPUs at parallel 2 (was parallel 1) -- NVIDIA context reduced 256K→128K to free VRAM — now the stable ceiling across all GPUs (2026-07-17) -- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal) -- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s +- NVIDIA context reduced 256K→128K to free VRAM — the stable NVIDIA ceiling (2026-07-17); Strix Halo runs 256K (2026-09-12) +- Timeouts and fallback chains are config state — read them from CT 116 `/opt/inference-harness/litellm_config.yaml`; they are not duplicated here. ## Parameters @@ -75,26 +67,36 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) - SSH key access to backend_host for container checks - Network access to public_url, auth_host, and gpu_dashboard_url -- LiteLLM master key for key management endpoints +- LiteLLM master key for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`) +- The dedicated `monitor` agent key on CT 116 at `/etc/litellm-monitor.env` (root-only 0600) for + model inference checks, scoped for every alias step 7 probes (`gpu-dense`, `gpu-vision`, + `strix-moe`, `syslog-auto`). Retrieve from the executor's host via: + + ``` + monitor key: ssh root@192.168.68.116 "grep LITELLM_MONITOR_KEY /etc/litellm-monitor.env | cut -d= -f2" + master key: ssh root@192.168.68.116 "docker exec harness-litellm printenv LITELLM_MASTER_KEY" + ``` + + If credentials are missing or unreadable, the probe must report `credential-missing` (not bare 401 or "0 keys"). + The master key must never be used for inference. ## GPU Fleet Topology -| Host | IP | Hardware | Models Served | Engine | Context | Parallel | -|------|-----|----------|---------------|--------|---------|----------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **128K** | 2 | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **128K** | 2 | -| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 128K | 2 | +| Host | IP | Hardware | Role | +|------|-----|----------|------| +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | heavy reasoning (`gpu-dense`) | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | vision / web extract / light tasks (`gpu-vision`) | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | compression (`strix-moe`) | -## Model Fallback Chains (LiteLLM) +Single source of truth for models, aliases, rpm caps, weights and fallback chains: +CT 116 `/opt/inference-harness/litellm_config.yaml`. Do not duplicate those values in +contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-light`, +`crew-auto`). -| Primary | Timeout | Fallback | Timeout | -|---------|---------|----------|---------| -| qwen3.6-27B-code | 300s | gemma-4-12b | 120s | -| gemma-4-12b | 120s | qwen3.6-27B-code | 300s | -| qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — | -| syslog-auto (balanced) | 300s | qwen → gemma | — | - -> Global: request_timeout=300s, nginx proxy_read_timeout=600s +> Re-scope note (2026-09-12): the earlier plan to restate the live fallback chains and +> per-model timeouts in this contract is intentionally superseded — that state is config, +> and this contract points at the CT 116 config instead. Step 7 likewise authenticates with +> the dedicated `monitor` key, not the master key, which is admin-only. ## Containers on CT 116 @@ -112,10 +114,22 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) 1. **Read parameters** — Use provided values or defaults -2. **Check public endpoints**: - - GET {{public_url}}/litellm/ui/ → expect 200 ("LiteLLM Dashboard") - - GET {{public_url}}/litellm/docs → expect 200 ("LiteLLM API - Swagger UI") - - GET {{public_url}}/ui/ and {{public_url}}/docs → expect 301 → /litellm/... → 200 (one-hop redirect helpers, added 2026-09-11) +2. **Check the end-user surfaces** — the public edge and the backend edge serve the SAME + app under DIFFERENT paths. They are not interchangeable, so every probe below must name + the surface it targets. Never point a check at a path that only resolves on the other + surface. + + **Public edge** — `{{public_url}}` (https://litellm.sysloggh.net) serves the app at the + ROOT; the `/litellm/` prefix does not exist there and 404s: + - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard") + - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI") + - GET {{public_url}}/litellm/ui/ and {{public_url}}/litellm/docs → expect 404 (not served on this edge) + + **Backend edge** — `http://{{backend_host}}` (port 80) serves the app UNDER `/litellm/`: + - GET http://{{backend_host}}/litellm/ui/ → expect 200 ("LiteLLM Dashboard") + - GET http://{{backend_host}}/litellm/docs → expect 200 ("LiteLLM API - Swagger UI") + - GET http://{{backend_host}}/ui/ and /docs → expect 301 → /litellm/... → 200 (one-hop redirect helpers, + added 2026-09-11) 3. **Check LiteLLM health (no-auth)**: - GET http://{{backend_host}}/litellm/health/liveliness → expect 200 @@ -136,16 +150,54 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) - Verify GPUs reporting status "healthy" - Check alerts array for active warnings/critical -7. **Check model inference via LiteLLM** — Test each model: - - POST /v1/chat/completions model=gemma-4-12b → expect 200 - - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 - - POST /v1/chat/completions model=strix-moe → expect 200 - - Use master key for auth +7. **Check model inference via LiteLLM** — Test one model on each GPU host. The health + check runs on the **backend edge**, not the public edge, so these paths carry the + `/litellm/` prefix: + - POST http://{{backend_host}}/litellm/v1/chat/completions model=gpu-dense → expect 200 (RTX 3090, .8) + - POST http://{{backend_host}}/litellm/v1/chat/completions model=gpu-vision → expect 200 (RTX 5070, .110) + - POST http://{{backend_host}}/litellm/v1/chat/completions model=strix-moe → expect 200 (Strix Halo, .15) + - Auth uses the dedicated `monitor` agent key. Retrieve via: + `ssh root@192.168.68.116 "grep LITELLM_MONITOR_KEY /etc/litellm-monitor.env | cut -d= -f2"` + Do NOT use the master key for inference — the master key is for admin endpoints only + (`/key/list`, `/key/generate`, `/key/info`). Retrieve master key via: + `ssh root@192.168.68.116 "docker exec harness-litellm printenv LITELLM_MASTER_KEY"` + - KEY SCOPE: the `monitor` key MUST be scoped for the three probed aliases (`gpu-dense`, + `gpu-vision`, `strix-moe`) plus the `syslog-auto` fallback pool, otherwise the probe + returns 403 and the host is not covered. + If a probe returns 403, widen the monitor key's model list on CT 116 (add the missing + alias) and re-run — never drop the host from the probe to make the check pass. + - `gemma-4-12b` was retired and returns 400 `Invalid model name` — do not re-add it to + this list. The RTX 5070 host now serves `gpu-vision`. + - `/v1/models` is **key-scoped**: a model is only visible to keys allowed to use it, so + the set depends on the key. Always state which key a model list was read with — a + snapshot without its key is not evidence. This probe uses the `monitor` key on the + backend surface (`http://{{backend_host}}/litellm/v1/models`). The authoritative model + registry is CT 116 `/opt/inference-harness/litellm_config.yaml`; read it there rather + than freezing a list here. 8. **Check agent keys**: - - GET /key/list with master key → verify all 6 agents have keys + - GET http://{{backend_host}}/litellm/key/list with master key (admin endpoint) → verify all 6 agents have keys + - **IMPORTANT**: Run the curl on the CT 116 HOST, not inside the container. The `harness-litellm` container has no curl/wget. Use: + `ssh root@192.168.68.116 "curl -s -H 'Authorization: Bearer $MASTER_KEY' http://127.0.0.1:4000/key/list"` + - If the response is empty or unparseable, report `admin-call-failed` (not "0 agent keys") 9. **Check Grafana**: - GET {{grafana_url}}/api/health → expect 200 10. **Compile and report** — Determine overall_status from individual check results + +## Executor Script (2026-09-13) + +**Run `scripts/litellm-health-check.py` from the clone.** This script implements all 11 +checks defined above and reports results in a standardized format. Paste its output in +the status line. + +- Hand-rolled probes are **not** an acceptable substitute for the script. +- Backend-edge checks (steps 2–8) must use `http://192.168.68.116` (internal IP), + **not** the public URL `https://litellm.sysloggh.net` (which returns 401 for those paths). +- Docker Stats (step 10) must be fetched from the CT 116 host itself (`127.0.0.1:9324/metrics`) + because the `harness-docker-stats` container binds to localhost on CT 116. +- Admin Key List (step 8) requires the master key expanded locally before SSH, then embedded + in the remote curl command with proper quoting. + +Expected output on a healthy fleet: 11/11 passing checks. diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index f961460..189cdb0 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -11,22 +11,22 @@ note: > Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`). Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs). GPU monitoring integrated from gpu-monitor on .24:9100. - - Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate - duplication of architecture diagrams, GPU topology, timeout tables, and container - lists. Health check is now § Health Check within this contract. - + + Health probes are owned by litellm-health.prose.md (dispatched as + `run contract: litellm-health`). This contract owns remediation only — it does not + re-specify the probes. + Source of truth for GPU topology and keys: gpu-fleet.prose.md Last verified: 2026-07-12 description: > - LiteLLM inference stack health monitoring + self-healing. Verifies the full - nginx → LiteLLM → GPU chain, 12 containers on CT 116, 3 GPU hosts, model - inference, and agent keys. Applies remediation rules for common failures. + LiteLLM inference stack remediation. Applies remediation rules for failures detected + by litellm-health.prose.md (nginx → LiteLLM → GPU chain, CT 116 containers, GPU hosts, + model inference, and agent keys). Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs). --- --- -# LiteLLM Operations — Health Check + Self-Heal +# LiteLLM Operations — Self-Heal (Remediation) ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU) @@ -58,41 +58,44 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) ## GPU Fleet Topology -| Host | IP | Hardware | Models Served | Engine | Context | Parallel | -|------|-----|----------|---------------|--------|---------|----------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 131072 --parallel 2 --ngl 99`) | **128K** | 2 | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 131072 --parallel 2`, IQ4_NL + MTP draft) | **128K** | 2 | -| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 128K | 2 | +| Host | IP | Hardware | Role | +|------|-----|----------|------| +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | heavy reasoning (`gpu-dense`) | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | vision / web extract / light tasks (`gpu-vision`) | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | compression (`strix-moe`) | -> Verified on ground 2026-07-16 via `curl /v1/models` on each host + `llama-wrapper.sh`. The AMD host's underlying model is `qwen3.6-35B-udq4`; LiteLLM exposes it under two `model_name`s: `qwen3.6-35B-udq4` and `strix-moe` (rpm 40). The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced. +> Verified on the ground 2026-07-16. The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced. -## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116) +## LiteLLM Model Surface -`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`, `crew-auto` (new 2026-08-20). +Single source of truth for models, aliases, rpm caps, weights and fallback chains: +CT 116 `/opt/inference-harness/litellm_config.yaml`. Do not duplicate those values in +contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-light`, +`crew-auto`). -### Context Cap Split (2026-08-20) +### Alias Surface + +| Alias | Serves | Where | Kind | +|-------|--------|-------|------| +| `gpu-dense` | heavy reasoning | RTX 3090 (192.168.68.8) | direct alias | +| `gpu-vision` | vision / web extract / light tasks | RTX 5070 (192.168.68.110) | direct alias AND `syslog-auto` pool member | +| `strix-moe` | compression (MoE) | Strix Halo (192.168.68.15) | direct alias | +| `syslog-auto` | balanced default | weighted pool across the three GPU hosts | pool router | + +### Context Cap Split (2026-08-20; crew cap RETIRED) - **Abiba (firstmate)**: 128K uncapped — unlimited context for primary workloads - **Hermes agents** (mumuni, tanko, koby, koonimo): 128K uncapped -- **Crewmates** (ops, tune, verify, auth-keys, build): 64K capped — alias `crew-auto` enforces 64K limit +- **Crewmates** (ops, tune, verify, auth-keys, build): the 64K cap was retired together with the `crew-auto` alias; NO context cap is currently in force. -Preferred implementation: uncap shared pool, add capped alias for crew-only. +> The 64K crew cap was retired with `crew-auto` (2026-09-12). No limit is currently in force; reinstating one would need per-key model limits as a separate, deliberately-scoped change. -- `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200). -- `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively. -- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. As of 2026-07-16 the `baggy`/`koby`/`mumuni`/`abiba-pi` keys ALSO include `qwen3.6-35B-udq4`; `abiba-pi` additionally includes `deepseek-v4-pro` (cloud fallback). `kagenz0`/`koonimo`/`pi-agents-unified` have the standard 6 only. Agents should still use the stable alias `strix-moe` (not the raw `qwen3.6-35B-udq4`) so model swaps don't break them. +- Key scoping: `/v1/models` is key-scoped, so the set a caller sees must be read with a named key rather than assumed — a monitor key, an agent key, and the master key can each return a different set. Agents should use the stable aliases (`strix-moe`, `gpu-vision`, `gpu-dense`) rather than raw model names, so model swaps don't break them. - **NEVER use `litellm_proxy_master_key` (the `sk-litellm-...` master key) for inference.** It is for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated `monitor` agent key stored at `/etc/litellm-monitor.env` on CT 116 (root-only, `chmod 600`); `/key/list` is the only call that legitimately uses `$MASTER_KEY`. -## Model Fallback Chains (LiteLLM) - -| Primary | Timeout | Fallback | Timeout | -|---------|---------|----------|---------| -| qwen3.6-27B-code | 300s | gemma-4-12b | 120s | -| gemma-4-12b | 120s | qwen3.6-27B-code | 300s | -| qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — | -| syslog-auto (balanced) | 300s | qwen → gemma | — | - -> Global: request_timeout=300s, nginx proxy_read_timeout=600s +> Re-scope note (2026-09-12): the fallback-chain and per-model timeout tables were removed +> by the single-source-of-truth re-scope; read those values from the CT 116 config named +> above rather than from this contract. ## Containers on CT 116 @@ -135,46 +138,7 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only. ## Health Check -Run this first on every cycle. Results feed into remediation rules below. - -### 1. Check public endpoints -- GET {{public_url}}/litellm/ui/ → expect 200 ("LiteLLM Dashboard") # served directly by nginx -- GET {{public_url}}/litellm/docs → expect 200 ("LiteLLM API - Swagger UI") # served directly by nginx -- GET {{public_url}}/ui/ and {{public_url}}/docs → expect 301 → /litellm/... → 200 (one-hop redirect helpers, added 2026-09-11) - -### 2. Check LiteLLM health (no-auth) -- GET http://{{backend_host}}/litellm/health/liveliness → expect 200 - -### 3. Check backend container health -- SSH to {{backend_host}} → `docker ps` → verify 12 containers healthy (11 harness + trove-agent-docker, added 2026-09-11) -- Critical: harness-litellm, harness-nginx, harness-postgres -- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, - harness-alertmanager, harness-zulip-bridge, harness-docker-stats, harness-pve-exporter, - trove-agent-docker (ghcr.io/techdox/trove-agent-docker, added 2026-09-11) -- Decommissioned 2026-09-11: harness-router (container, image and config removed) - -### 4. Check GPU fleet health (via fleet dashboard) -- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON -- Verify GPUs reporting status "healthy" -- Check alerts array for active warnings/critical - -### 5. Check model inference via LiteLLM — test each model -- POST /v1/chat/completions model=gemma-4-12b → expect 200 -- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 -- POST /v1/chat/completions model=strix-moe → expect 200 -- Use master key for auth - -### 6. Check agent keys -- GET /key/list with master key → verify all 6 agents have keys - -### 7. Check Grafana -- GET {{grafana_url}}/api/health → expect 200 - -### 8. Compile overall status -Determine overall_status from individual check results: -- "healthy" — all checks pass -- "degraded" — 1-2 non-critical checks fail -- "down" — critical checks fail +Health probes are owned by `litellm-health.prose.md` (dispatched as `run contract: litellm-health`). This contract owns remediation only — it does not re-specify the probes. --- --- diff --git a/mumuni-delegation-prose-contract.prose.md b/mumuni-delegation-prose-contract.prose.md index 38e15cb..48c425a 100644 --- a/mumuni-delegation-prose-contract.prose.md +++ b/mumuni-delegation-prose-contract.prose.md @@ -90,8 +90,8 @@ raw data never provided. | Worker | Model | Toolsets | Role | Use When | |--------|-------|----------|------|----------| -| `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | -| `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | +| `syslog-code` | gpu-dense | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | +| `syslog-devops` | gpu-dense | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | | `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | | `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | | `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index 6081167..e3df0e6 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -9,7 +9,6 @@ description: > - abiba-telegram: { status: "online", uptime: string, restarts: number } - abiba-zulip: { status: "online", uptime: string, restarts: number } - gitea-runner: { status: "online", uptime: string, restarts: number } -- spoton-service: { status: "online", uptime: string, restarts: number } - zulip-watchdog: { status: "online", uptime: string, restarts: number } - last_check: timestamp diff --git a/proxmox-monitor.prose.md b/proxmox-monitor.prose.md index 39e81d6..a7904e1 100644 --- a/proxmox-monitor.prose.md +++ b/proxmox-monitor.prose.md @@ -87,7 +87,7 @@ agent: abiba | storepve | 192.168.68.6 | PVE | | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) | | minipve | 192.168.68.12 | PVE | -| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) | +| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (strix-moe) | ## Operations diff --git a/runs/disk-gc-ct111-report-only-verification-20260912.log b/runs/disk-gc-ct111-report-only-verification-20260912.log new file mode 100644 index 0000000..641d21d --- /dev/null +++ b/runs/disk-gc-ct111-report-only-verification-20260912.log @@ -0,0 +1,44 @@ +disk-gc report-only verification — CT 111 / tdunna / 192.168.68.129 +date: 2026-09-12T19:11:36Z +host: abiba (this scanner runs INSIDE CT 100 / abiba) +branch head: 6e612ce37b1b9a9688b04e7f816848e7a4185fca +command: scripts/disk-gc-plan.py --scan + +PURPOSE: prove that on a REAL fleet scan, CT 111 is alerted and NO gc-executor action +is emitted for it at any level. No GC command was executed against .129. + +--- live fleet scan (df -P / via scripts/pct-run.sh for LXC, direct SSH for hosts) --- + tdunna 84% + acerpve 192.168.68.9 77% + amdpve 192.168.68.15 76% + ocu-llm 192.168.68.110 69% + storepve 192.168.68.6 65% + kagentz 61% + tanko 56% + minipve 192.168.68.12 49% + authentik 45% + infisical-vault 39% + adguard 38% + ocupve 192.168.68.5 38% + scottdenya 35% + syslog-api 34% + llm-gpu 192.168.68.8 25% + abiba 23% + baggy 22% + jdownloader 21% + gitea 16% + ra-h-os 13% + zulip 12% + docker-vm 192.168.68.7 11% + adguard2 10% + media 9% + proxmox-backup-server 4% + +--- planner output (action plan) --- + 111 AMBER 84.0% -> REPORT-ONLY (no GC) — Theo's box — captain ruling 2026-08-17, re-confirmed 2026-09-10 + acerpve AMBER 77.0% -> gc-executor + amdpve AMBER 76.0% -> gc-executor + +--- verdict --- + CT 111 (tdunna) 84% AMBER -> report-only; no gc-executor row emitted; no GC run on .129. + Owned hosts acerpve .9 (77%) and amdpve .15 (76%) -> gc-executor (ours). diff --git a/scripts/agent-health-check.py b/scripts/agent-health-check.py index c5d2618..5a62852 100755 --- a/scripts/agent-health-check.py +++ b/scripts/agent-health-check.py @@ -340,6 +340,36 @@ def check_gpu_ports(): # CHECK 3: Agent Gateway Liveness + Streaming (now covers all agents) # ═══════════════════════════════════════════════════════════════════ +def _ssh_retry(host, cmd, user="root", timeout=15, retry_timeout=25, label=""): + """SSH with one retry at a longer timeout. + + Returns (stdout_or_None, probe_failed_bool, fail_kind). + When probe_failed is True, fail_kind is one of: timeout, ssh-failed. + """ + import subprocess as _sp + def _attempt(tmo, conn_tmo): + try: + r = _sp.run( + ["ssh", "-o", "StrictHostKeyChecking=no", "-o", f"ConnectTimeout={conn_tmo}", + f"{user}@{host}", cmd], + capture_output=True, text=True, timeout=tmo) + return r.stdout.strip() if r.returncode == 0 else None + except _sp.TimeoutExpired: + return "__timeout__" + except: + return None + result = _attempt(timeout, 8) + if result is None or result == "__timeout__": + kind = "timeout" if result == "__timeout__" else "ssh-failed" + prefix = f"{label} " if label else "" + print(f" probe-failed: {prefix}ssh {user}@{host} — {kind} (retrying at {retry_timeout}s…)") + result = _attempt(retry_timeout, 15) + if result is None or result == "__timeout__": + kind = "timeout" if result == "__timeout__" else "ssh-failed" + return None, True, kind + return result, False, None + + def check_agents(): for name, agent in AGENTS.items(): host = agent.get("host") @@ -355,42 +385,49 @@ def check_agents(): is_dsh = agent.get("runtime") == "dsh" label = "DSH (DeepSeek Harness)" if is_dsh else "pi-only runtime" since = "since 2026-08-27" if is_dsh else "since the harness purge" - live = ssh(host, "true", user=user) - print(f" {'✅' if live is not None else '❌'} {name}: {label} — " - f"no Hermes gateway {since} (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})") - if live is None: - _fail(f"unreachable:{name}", name) + live, probe_failed, fail_kind = _ssh_retry(host, "true", user=user) + if probe_failed: + print(f" ❌ {name}: {label} — probe-failed: ssh {user}@{host} {fail_kind} " + f"(retried at 25s: also {fail_kind}) [CT {ct}]") + _fail(f"probe-failed:{name}:{fail_kind}", name) + else: + print(f" ✅ {name}: {label} — no Hermes gateway {since} " + f"(ssh {user}@{host} OK, CT {ct})") continue if not host or not user: print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check") continue - # Resolve the Hermes gateway PID once, before the report-only branch: - # the summary line below renders `pid`, and it used to be bound only in - # the report-only path — leaving it unbound on the abiba/koonimo path - # raised UnboundLocalError and crashed the whole check. Agents without - # a gateway get pid=?. - pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user) - if not pid: - pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user) - if not pid: + # Resolve the Hermes gateway PID with retry. The probe target is + # explicit: ssh {user}@{host} pgrep -f hermes gateway. + pid, probe_failed, fail_kind = _ssh_retry( + host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user) + if not pid and not probe_failed: + pid, probe_failed, fail_kind = _ssh_retry( + host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user) + if not pid and not probe_failed: pid = "?" - # ⛔ KOBY IS NEVER REPAIRED — diagnostic only + if probe_failed: + print(f" ❌ {name}: probe-failed: ssh {user}@{host} {fail_kind} " + f"(retried at 25s: also {fail_kind}) [CT {ct}] — gateway status UNDETERMINED") + _fail(f"probe-failed:{name}:{fail_kind}", name) + continue + + # ⛔ KOBY IS NEVER REPAIRED — diagnostic only (captain's 2026-08-17 ruling) if report_only: - print(f" 🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)") - # Still check gateway status for reporting purposes if pid == "?": - print(f" ⚠️ {name}: GATEWAY NOT RUNNING (reported only)") + print(f" 🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway " + f"-> no process found (reported only, NOT counted) [CT {ct}]") _fail(f"gateway-down:{name}", name) - continue else: - print(f" ✅ {name}: gateway running (pid={pid}, report-only mode)") - continue # Skip the rest of the check for Koby + print(f" 🔍 {name}: REPORT-ONLY — probe: ssh {user}@{host} pgrep hermes-gateway " + f"-> pid={pid} (running, reported only, NOT repaired) [CT {ct}]") + continue # Skip the rest of the check for Koby # Gateway state file - state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user) + state, _, _ = _ssh_retry(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user) if state: try: st = json.loads(state) @@ -408,21 +445,22 @@ def check_agents(): ] streaming = "no" for p in adapter_paths: - has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user) + has_edit, _, _ = _ssh_retry(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user) if has_edit and has_edit != "0": streaming = "yes" break # Recent errors - recent_errors = ssh(host, + recent_errors, _, _ = _ssh_retry( + host, r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null " r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0", user=user) recent_errors = (recent_errors or "0").strip().split("\n")[-1] print(f" {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} " - f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} " - f"errors_10m={recent_errors.strip() or '0'} pid={pid}") + f"{name}: probe: ssh {user}@{host} — gw={gw_state} zulip={zulip} " + f"streaming={streaming} errors_10m={recent_errors.strip() or '0'} pid={pid} [CT {ct}]") # ═══════════════════════════════════════════════════════════════════ diff --git a/scripts/disk-gc-plan.py b/scripts/disk-gc-plan.py new file mode 100755 index 0000000..ca1e867 --- /dev/null +++ b/scripts/disk-gc-plan.py @@ -0,0 +1,206 @@ +#!/usr/bin/env python3 +"""disk-gc-plan — turn a fleet disk scan into the GC action plan. + +This is the executable side of `disk-gc-threat-response.prose.md`. It exists so the +report-only gate is enforced by code that can be tested, rather than by prose the +executor might misread. + +THE HARD GATE: guests listed in the contract's `report_only_guests` block are +DETECT-AND-REPORT-ONLY at EVERY level (AMBER, RED, CRITICAL). This tool will never +emit a `gc-executor` action for one, so no GC command can be constructed for it. + +The gate is keyed on GUEST identity — guest id, hostname, or IP — never on an agent +name. An agent-name marker can silently miss the guest it lives on; a guest marker +cannot. + +The authoritative exclusion list lives in the contract itself (the fenced ```yaml +block containing `report_only_guests:`). This tool reads it from there so there is +only ever one copy. + +Usage: + disk-gc-plan.py --scan scan.json # [{"id":111,"usage_pct":84}, ...] + cat scan.json | disk-gc-plan.py # same, via stdin + disk-gc-plan.py --scan scan.json --json # machine-readable plan + +Scan entries may carry any of: id / guest / vmid / ct / ctid, hostname / name, ip. +A threshold-crossing entry with no recognizable identity is reported, never GC'd. +Exit codes: 0 ok, 1 usage/parse error. +""" +from __future__ import annotations + +import argparse +import json +import pathlib +import re +import sys + +import yaml + +REPO = pathlib.Path(__file__).resolve().parent.parent +DEFAULT_CONTRACT = REPO / "disk-gc-threat-response.prose.md" + +AMBER, RED, CRITICAL = 75, 85, 95 + +IDENTITY_FIELDS = ("id", "guest", "vmid", "ct", "ctid", "hostname", "name", "ip") +IDENTITY_TYPE_PREFIX = re.compile(r"^(?:lxc|qemu)/") +UNIDENTIFIED_REASON = "unidentified target - refusing to schedule GC" + + +def load_report_only_guests(contract_path: pathlib.Path) -> list[dict]: + """Read the authoritative report_only_guests block out of the contract. + + The contract carries it as a fenced ```yaml block. Parsing the declared, + machine-readable block is the intended interface — the contract owns the list. + """ + text = contract_path.read_text(encoding="utf-8") + for block in re.findall(r"```yaml\n(.*?)```", text, re.S): + if "report_only_guests:" in block: + data = yaml.safe_load(block) + guests = data.get("report_only_guests") or [] + if not isinstance(guests, list): + raise SystemExit("report_only_guests must be a list") + if not guests: + raise SystemExit( + "report_only_guests is empty or missing - refusing to plan GC " + "without the report-only gate" + ) + for guest in guests: + if not isinstance(guest, dict) or not _keys(guest): + raise SystemExit( + "report_only_guests entry has no recognizable identity key " + f"(expected one of: {', '.join(IDENTITY_FIELDS)}): {guest!r}" + ) + return guests + raise SystemExit( + f"no authoritative report_only_guests block found in {contract_path}" + ) + + +def _canonical_number(number: float) -> str: + if float(number).is_integer(): + return str(int(number)) + return str(number).strip().lower() + + +def _normalize_identity(value: object) -> str: + """Canonicalise a guest identity so differently-encoded ids compare equal: + numeric and numeric-string ids collapse to an integer string, Proxmox + type prefixes and leading zeros are stripped, and hostnames/IPs are only + trimmed and lowercased.""" + if isinstance(value, bool): + return str(value).strip().lower() + if isinstance(value, (int, float)): + return _canonical_number(float(value)) + text = str(value).strip().lower() + text = IDENTITY_TYPE_PREFIX.sub("", text) + try: + return _canonical_number(float(text)) + except ValueError: + return text + + +def _keys(entry: dict) -> set[str]: + """Guest/host identity keys, shared by exclusions and scan entries so the two + sides of the gate can never key on different fields.""" + out: set[str] = set() + for field in IDENTITY_FIELDS: + value = entry.get(field) + if value is None: + continue + key = _normalize_identity(value) + if key: + out.add(key) + return out + + +def level_for(pct: float) -> str | None: + if pct >= CRITICAL: + return "CRITICAL" + if pct >= RED: + return "RED" + if pct >= AMBER: + return "AMBER" + return None + + +def build_plan(scan: list[dict], report_only: list[dict]) -> list[dict]: + excluded = [(e, _keys(e)) for e in report_only] + plan: list[dict] = [] + for entry in scan: + pct = entry.get("usage_pct") + if pct is None: + continue + level = level_for(float(pct)) + if level is None: + continue # GREEN: log only, no action + scan_keys = _keys(entry) + target = next( + (entry.get(k) for k in IDENTITY_FIELDS if entry.get(k) not in (None, "")), + "?", + ) + if not scan_keys: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "report-only", + "reason": UNIDENTIFIED_REASON, + }) + continue + match = next((e for e, keys in excluded if keys & scan_keys), None) + if match is not None: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "report-only", + "reason": match.get("reason", "").strip(), + }) + else: + plan.append({ + "target": target, + "level": level, + "pct": float(pct), + "action": "gc-executor", + }) + plan.sort(key=lambda row: row["pct"], reverse=True) + return plan + + +def main() -> int: + ap = argparse.ArgumentParser(description="Plan disk GC actions with the report-only gate.") + ap.add_argument("--scan", help="JSON file: list of {id|ct|hostname|ip, usage_pct}") + ap.add_argument("--contract", default=str(DEFAULT_CONTRACT)) + ap.add_argument("--json", action="store_true", help="emit the plan as JSON") + args = ap.parse_args() + + raw = pathlib.Path(args.scan).read_text() if args.scan else sys.stdin.read() + try: + scan = json.loads(raw) + except json.JSONDecodeError as exc: + print(f"invalid scan JSON: {exc}", file=sys.stderr) + return 1 + if not isinstance(scan, list): + print("scan must be a JSON list", file=sys.stderr) + return 1 + + report_only = load_report_only_guests(pathlib.Path(args.contract)) + plan = build_plan(scan, report_only) + + if args.json: + print(json.dumps(plan, indent=2)) + return 0 + + if not plan: + print("no threats (nothing at or above 75%)") + return 0 + for row in plan: + if row["action"] == "report-only": + print(f" {row['target']} {row['level']} {row['pct']}% -> REPORT-ONLY (no GC) — {row['reason']}") + else: + print(f" {row['target']} {row['level']} {row['pct']}% -> gc-executor") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/disk-gc-scan.py b/scripts/disk-gc-scan.py new file mode 100644 index 0000000..8c5e218 --- /dev/null +++ b/scripts/disk-gc-scan.py @@ -0,0 +1,339 @@ +#!/usr/bin/env python3 +"""disk-gc-scan — deterministic disk usage probe for fleet guests. + +This is the executable scanner side of `disk-gc-threat-response.prose.md`. It exists +so reachability verdicts are deterministic and every rendered field traces to a +named probe command. + +DESIGN PRINCIPLES (per task disk-gc-probe-false-unreachable-20260913): +1. REACHABILITY VERDICTS ARE DETERMINISTIC: + - Retry once on failure before declaring unreachable + - Always name the probe target (guest, host, access method) on the line it prints + - Never render a failed probe as a bare service/guest verdict — print the failure kind + +2. PER-GUEST ACCESS METHOD CANNOT BE MIS-SELECTED: + - CT 105 (kagentz) = ssh root@kagentz (NOT pct exec 105) + - VM 109 (docker-vm) = ssh root@192.168.68.7 (NOT pct) + - All other CTs = pct-run (which uses pct exec) + - The access method is selected from a per-guest map so the wrong path cannot be + picked by an executor improvising + +3. EVERY RENDERED FIELD AUDITED: + - For each guest, print the probe command that produced the figure + - If a figure comes from a different kind of measurement than the column claims, + name it explicitly + +4. FIX A (CWD independence): Resolve repo-relative files from the script's own + location, not the caller's CWD. + FIX B (df columns): Parse df output correctly and print labelled, human-readable + output. + +Usage: + disk-gc-scan.py # scan all guests + disk-gc-scan.py --json # machine-readable output + +Exit codes: 0 ok (all guests probed), 1 probe error +""" +from __future__ import annotations + +import json +import pathlib +import subprocess +import sys +import time +import os +from dataclasses import dataclass +from typing import Optional + +# Resolve repo-relative files from the script's own location, not the caller's CWD +SCRIPT_DIR = pathlib.Path(__file__).resolve().parent +HELPER_PCT_RUN = SCRIPT_DIR / "pct-run.sh" + +# Per-guest access method map. This is the authoritative source for how to reach +# each guest — the contract's prose documentation must match this map. +# +# Access methods: +# - "pct-run": use pct-run.sh (pct exec via SSH to node) +# - "ssh-host": use ssh root@ +# - "ssh-ip": use ssh root@ + +@dataclass +class Guest: + """A guest to probe.""" + ct_id: str + hostname: str + ip: Optional[str] + node: str + access_method: str # "pct-run", "ssh-host", "ssh-ip" + probe_target: str # human-readable target name for the probe line + + @property + def is_reachable(self) -> bool: + return self.probe_result is not None and self.probe_result.exit_code == 0 + + @property + def usage_pct(self) -> Optional[float]: + return self.probe_result.usage_pct if self.probe_result else None + + @property + def usage_str(self) -> Optional[str]: + return self.probe_result.usage_str if self.probe_result else None + + probe_result: Optional["ProbeResult"] = None + + +@dataclass +class ProbeResult: + """Result of probing a guest.""" + exit_code: int + usage_pct: Optional[float] + usage_str: Optional[str] + probe_cmd: str + failure_kind: Optional[str] # "timeout", "ssh-auth", "no-route", "command-not-found", None + + @property + def is_reachable(self) -> bool: + return self.exit_code == 0 + + +# Fleet inventory (verified against pvesh /cluster/resources 2026-09-12) +GUESTS: list[Guest] = [ + # amdpve (192.168.68.15) + Guest(ct_id="105", hostname="kagentz", ip="192.168.68.105", node="amdpve", + access_method="ssh-host", probe_target="kagentz (CT 105, amdpve)"), + Guest(ct_id="112", hostname="tanko", ip="192.168.68.112", node="amdpve", + access_method="pct-run", probe_target="tanko (CT 112, amdpve)"), + Guest(ct_id="113", hostname="baggy", ip="192.168.68.113", node="amdpve", + access_method="pct-run", probe_target="baggy (CT 113, amdpve)"), + Guest(ct_id="115", hostname="scottdenya", ip="192.168.68.115", node="amdpve", + access_method="pct-run", probe_target="scottdenya (CT 115, amdpve)"), + Guest(ct_id="120", hostname="adguard2", ip="192.168.68.120", node="amdpve", + access_method="pct-run", probe_target="adguard2 (CT 120, amdpve)"), + # minipve (192.168.68.12) + Guest(ct_id="100", hostname="abiba", ip="192.168.68.100", node="minipve", + access_method="pct-run", probe_target="abiba (CT 100, minipve)"), + Guest(ct_id="102", hostname="adguard", ip="192.168.68.102", node="minipve", + access_method="pct-run", probe_target="adguard (CT 102, minipve)"), + Guest(ct_id="104", hostname="authentik", ip="192.168.68.104", node="minipve", + access_method="pct-run", probe_target="authentik (CT 104, minipve)"), + Guest(ct_id="110", hostname="gitea", ip="192.168.68.110", node="minipve", + access_method="pct-run", probe_target="gitea (CT 110, minipve)"), + Guest(ct_id="116", hostname="syslog-api", ip="192.168.68.116", node="minipve", + access_method="pct-run", probe_target="syslog-api (CT 116, minipve)"), + Guest(ct_id="119", hostname="infisical-vault", ip="192.168.68.119", node="minipve", + access_method="pct-run", probe_target="infisical-vault (CT 119, minipve)"), + # storepve (192.168.68.6) + Guest(ct_id="106", hostname="ra-h-os", ip="192.168.68.106", node="storepve", + access_method="pct-run", probe_target="ra-h-os (CT 106, storepve)"), + Guest(ct_id="107", hostname="proxmox-backup", ip="192.168.68.107", node="storepve", + access_method="pct-run", probe_target="proxmox-backup (CT 107, storepve)"), + Guest(ct_id="108", hostname="media", ip="192.168.68.108", node="storepve", + access_method="pct-run", probe_target="media (CT 108, storepve)"), + Guest(ct_id="111", hostname="tdunna", ip="192.168.68.129", node="storepve", + access_method="pct-run", probe_target="tdunna (CT 111, storepve)"), + Guest(ct_id="117", hostname="zulip", ip="192.168.68.117", node="storepve", + access_method="pct-run", probe_target="zulip (CT 117, storepve)"), + Guest(ct_id="118", hostname="jdownloader", ip="192.168.68.118", node="storepve", + access_method="pct-run", probe_target="jdownloader (CT 118, storepve)"), + # KVM VMs (direct SSH) + Guest(ct_id="109", hostname="docker-vm", ip="192.168.68.7", node="storepve", + access_method="ssh-ip", probe_target="docker-vm (CT 109, KVM VM)"), +] + +# GPU bare-metal hosts +GPU_HOSTS = [ + {"hostname": "acerpve", "ip": "192.168.68.9", "gpu": "RTX 3090", + "probe_target": "RTX 3090 (bare metal .9)"}, + {"hostname": "ocupve", "ip": "192.168.68.110", "gpu": "RTX 5070", + "probe_target": "RTX 5070 (bare metal .110)"}, + {"hostname": "amdpve", "ip": "192.168.68.15", "gpu": "Strix Halo", + "probe_target": "Strix Halo (bare metal .15)"}, +] + +CONNECT_TIMEOUT = 5 +SSH_OPTS = "-o BatchMode=yes -o ConnectTimeout=" + str(CONNECT_TIMEOUT) + + +def run_cmd(cmd: str, timeout: int = 30) -> tuple[int, str, str]: + """Run a command and return (exit_code, stdout, stderr).""" + try: + result = subprocess.run( + cmd, shell=True, capture_output=True, text=True, timeout=timeout + ) + return result.returncode, result.stdout.strip(), result.stderr.strip() + except subprocess.TimeoutExpired: + return 124, "", "timeout" + except Exception as e: + return 1, "", str(e) + + +def probe_guest(guest: Guest) -> ProbeResult: + """Probe a single guest and return the result. + + Access method is selected from guest.access_method: + - "pct-run": pct-run.sh "df -P / | tail -1" + - "ssh-host": ssh root@ "df -P / | tail -1" + - "ssh-ip": ssh root@ "df -P / | tail -1" + """ + df_cmd = "df -P / | tail -1" + + if guest.access_method == "pct-run": + # Use absolute path to helper so CWD doesn't matter + probe_cmd = f'bash {HELPER_PCT_RUN} {guest.ct_id} "{df_cmd}"' + elif guest.access_method == "ssh-host": + probe_cmd = f'ssh {SSH_OPTS} root@{guest.hostname} "{df_cmd}"' + elif guest.access_method == "ssh-ip": + probe_cmd = f'ssh {SSH_OPTS} root@{guest.ip} "{df_cmd}"' + else: + raise ValueError(f"unknown access_method: {guest.access_method}") + + # Check helper exists and is readable BEFORE probing (for pct-run guests) + # This prevents scanner errors from being rendered as guest verdicts + if guest.access_method == "pct-run": + if not HELPER_PCT_RUN.exists(): + print(f"SCANNER ERROR: helper not found: {HELPER_PCT_RUN}", file=sys.stderr) + sys.exit(1) + if not os.access(str(HELPER_PCT_RUN), os.R_OK): + print(f"SCANNER ERROR: helper not readable: {HELPER_PCT_RUN}", file=sys.stderr) + sys.exit(1) + + # Retry once on failure before declaring unreachable + for attempt in range(2): + exit_code, stdout, stderr = run_cmd(probe_cmd, timeout=15) + + if exit_code == 0: + # Parse df output: Filesystem 1024-blocks Used Available Capacity Mounted on + # parts[0]=Filesystem, parts[1]=Total (1K blocks), parts[2]=Used, parts[3]=Available, parts[4]=Capacity + parts = stdout.split() + if len(parts) >= 5: + capacity_str = parts[4] # e.g., "34%" + usage_pct = float(capacity_str.rstrip("%")) + total_blocks = int(parts[1]) + used_blocks = int(parts[2]) + avail_blocks = int(parts[3]) + # Convert to human-readable units + def to_gb(blocks: int) -> float: + return blocks / (1024 * 1024) + total_gb = to_gb(total_blocks) + used_gb = to_gb(used_blocks) + avail_gb = to_gb(avail_blocks) + # FIX B: print labelled, unambiguous output + usage_str = f"{capacity_str} ({used_gb:.1f}G used of {total_gb:.1f}G total, {avail_gb:.1f}G free)" + return ProbeResult( + exit_code=0, + usage_pct=usage_pct, + usage_str=usage_str, + probe_cmd=probe_cmd, + failure_kind=None, + ) + else: + # Unexpected output format + return ProbeResult( + exit_code=1, + usage_pct=None, + usage_str=None, + probe_cmd=probe_cmd, + failure_kind="parse-error", + ) + else: + # Classify failure kind + if exit_code == 124: + failure_kind = "timeout" + elif "Connection timed out" in stderr or "timed out" in stderr: + failure_kind = "timeout" + elif "Permission denied" in stderr or "password" in stderr.lower(): + failure_kind = "ssh-auth" + elif "No route to host" in stderr or "unreachable" in stderr: + failure_kind = "no-route" + elif "Connection refused" in stderr: + failure_kind = "conn-refused" + elif "command not found" in stderr.lower() or "No such file" in stderr: + failure_kind = "command-not-found" + else: + failure_kind = f"ssh-exit-{exit_code}" + + # Retry once + if attempt == 0: + time.sleep(1) + continue + return ProbeResult( + exit_code=exit_code, + usage_pct=None, + usage_str=None, + probe_cmd=probe_cmd, + failure_kind=failure_kind, + ) + + # Should not reach here, but just in case + return ProbeResult( + exit_code=1, + usage_pct=None, + usage_str=None, + probe_cmd=probe_cmd, + failure_kind="unknown", + ) + + +def scan_fleet() -> list[dict]: + """Scan all guests and return the results.""" + results = [] + for guest in GUESTS: + probe_result = probe_guest(guest) + guest.probe_result = probe_result + + row = { + "target": guest.probe_target, + "ct_id": guest.ct_id, + "hostname": guest.hostname, + "node": guest.node, + "access_method": guest.access_method, + "reachable": probe_result.is_reachable, + "usage_pct": probe_result.usage_pct, + "usage_str": probe_result.usage_str, + "probe_cmd": probe_result.probe_cmd, + "failure_kind": probe_result.failure_kind, + } + results.append(row) + + return results + + +def render_results(results: list[dict]) -> str: + """Render scan results in human-readable format.""" + lines = [] + lines.append("=== Disk GC Scan ===") + lines.append("") + + for row in results: + if row["reachable"]: + lines.append(f" ✅ {row['target']}: {row['usage_str']}") + lines.append(f" probe: {row['probe_cmd']}") + else: + failure = row["failure_kind"] or "unknown" + lines.append(f" ❌ {row['target']}: UNREACHABLE ({failure})") + lines.append(f" probe: {row['probe_cmd']}") + + return "\n".join(lines) + + +def main() -> int: + import argparse + + ap = argparse.ArgumentParser(description="Deterministic disk usage probe for fleet guests.") + ap.add_argument("--json", action="store_true", help="machine-readable output") + args = ap.parse_args() + + results = scan_fleet() + + if args.json: + print(json.dumps(results, indent=2)) + else: + print(render_results(results)) + + # Exit 0 if all guests probed (reachable or not), 1 if any probe error + # (a probe error means the probe itself failed, not just that the guest was unreachable) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/hermes-reachability-check.sh b/scripts/hermes-reachability-check.sh new file mode 100755 index 0000000..ba6ec1b --- /dev/null +++ b/scripts/hermes-reachability-check.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# Shared helper for Hermes contract reachability checks +# Separates SSH exit status from remote command result + +hermes_check_host() { + local host=$1 + local pattern=$2 + local path=$3 + + # Remote side always succeeds (grep ...; true), so ssh exit code = connection status only + local out + out=$(ssh -o BatchMode=yes -o ConnectTimeout=3 root@"$host" "grep -RIn '$pattern' '$path' 2>/dev/null; true" 2>/dev/null) + local status=$? + + if [ $status -ne 0 ]; then + echo "$host: UNREACHABLE (ssh exit $status)" + elif [ -n "$out" ]; then + echo "$host: VIOLATION: $out" + else + echo "$host: COMPLIANT (no matches found)" + fi +} + +# Standalone mode: scripts/hermes-reachability-check.sh +if [ "${BASH_SOURCE[0]}" = "${0}" ]; then + if [ $# -ne 3 ]; then + echo "Usage: $0 " >&2 + exit 2 + fi + hermes_check_host "$1" "$2" "$3" + exit 0 +fi diff --git a/scripts/litellm-health-check.py b/scripts/litellm-health-check.py new file mode 100755 index 0000000..03fb808 --- /dev/null +++ b/scripts/litellm-health-check.py @@ -0,0 +1,304 @@ +#!/usr/bin/env python3 +""" +LiteLLM Health Check - Contract executor +Runs all checks defined in litellm-health.prose.md and reports results. +""" + +import subprocess +import sys +import json +import time +import random + +# Configuration +BACKEND_HOST = "192.168.68.116" +GPU_HOSTS = { + "gpu-dense": "192.168.68.8", + "gpu-vision": "192.168.68.110", + "strix-moe": "192.168.68.15" +} + +def run_command(cmd, timeout=15): + """Run a command and return (exit_code, stdout, stderr)""" + try: + result = subprocess.run( + cmd, + shell=True, + capture_output=True, + text=True, + timeout=timeout + ) + return result.returncode, result.stdout.strip(), result.stderr.strip() + except subprocess.TimeoutExpired: + return 1, "", "TIMEOUT" + except Exception as e: + return 1, "", str(e) + +def probe_http(url, method="GET", bearer_token=None, data=None, timeout=10, follow_redirects=False): + """Probe HTTP endpoint and return (status_code, failure_kind) + + Returns: + (code, None) if successful or HTTP response received + (000, kind) if connection failed, where kind is 'timeout', 'refused', 'dns', etc. + """ + cmd = "curl -s -o /dev/null -w '%{http_code}' -m " + str(timeout) + if method == "POST": + cmd += " -X POST" + if bearer_token: + cmd += " -H 'Authorization: Bearer " + bearer_token + "'" + if data: + cmd += " -H 'Content-Type: application/json' -d '" + data + "'" + if follow_redirects: + cmd += " -L" + cmd += " '" + url + "'" + + try: + rc, stdout, stderr = run_command(cmd, timeout) + if rc != 0: + # Determine failure kind from curl exit code + # curl exit codes: 28=timeout, 7=refused, 6=dns, 35=ssl, 52=empty + if rc == 28: + return (000, "timeout after " + str(timeout) + "s") + elif rc == 7: + return (000, "connection refused") + elif rc == 6: + return (000, "dns failure") + elif rc == 35: + return (000, "ssl error") + elif rc == 52: + return (000, "empty response") + else: + return (000, "curl exit " + str(rc)) + return (int(stdout), None) if stdout.isdigit() else (000, "unparseable response") + except subprocess.TimeoutExpired: + return (000, "timeout after " + str(timeout) + "s") + +def get_response_body(url, method="POST", bearer_token=None, data=None, timeout=30): + """Get response body for 401/403 credential faults (truncated to 200 chars)""" + cmd = "curl -s -m " + str(timeout) + if method == "POST": + cmd += " -X POST" + if bearer_token: + cmd += " -H 'Authorization: Bearer " + bearer_token + "'" + if data: + cmd += " -H 'Content-Type: application/json' -d '" + data + "'" + cmd += " '" + url + "'" + + rc, stdout, stderr = run_command(cmd, timeout) + # Return first 200 chars, single line + body = stdout.replace('\n', ' ').replace('\t', ' ')[:200] if stdout else "" + return body + + +def check_liveliness(): + """Step 1: Liveliness probe""" + code, _ = probe_http("http://" + BACKEND_HOST + "/litellm/health/liveliness") + return "Liveliness", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/health/liveliness)" + +def check_containers(): + """Step 2: Container health via SSH""" + cmd = "ssh -o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=no root@192.168.68.116 'docker ps --format \"{{.Names}} {{.Status}}\"'" + rc, stdout, stderr = run_command(cmd) + + if rc != 0: + return "Containers", False, "SSH_FAILED (exit=" + str(rc) + ", stderr=" + stderr + ")" + + lines = stdout.split('\n') if stdout else [] + container_count = len([l for l in lines if l.strip()]) + healthy = container_count >= 8 + return "Containers", healthy, str(container_count) + " containers" + +def check_model_probes(): + """Step 6: Model probe - all 4 aliases""" + # Get monitor key + monitor_key = run_command("ssh -o BatchMode=yes root@192.168.68.116 \"grep LITELLM_MONITOR_KEY /etc/litellm-monitor.env | cut -d= -f2\"")[1] + + results = [] + + for model in ["gpu-dense", "gpu-vision", "strix-moe"]: + # Single-host aliases: 30s timeout each + # gpu-dense (RTX 3090) may need long warmup/prefill - timeout is acceptable on cold-start + code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions", + method="POST", + bearer_token=monitor_key, + data='{"model":"' + model + '","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}', + timeout=30) + + if code == 000 and failure_kind: + # Report probe failure with kind, do not assert a service verdict + results.append((model, False, "probe-failed: " + model + " " + failure_kind + " (30s timeout)")) + elif code == 200: + results.append((model, True, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")")) + elif code in (401, 403): + # Credential fault - capture body and key alias + body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions", + method="POST", + bearer_token=monitor_key, + data='{"model":"' + model + '","messages":[{"role":"user","content":"health"}],"max_tokens":4}', + timeout=10) + # Resolve key alias + alias = "monitor-20260813" # Known from /etc/litellm-monitor.env on CT 116 + results.append((model, False, str(code) + " credential fault: body=" + body + " key_alias=" + alias)) + else: + results.append((model, False, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=" + model + ")")) + + # Pool alias (syslog-auto): 60s timeout, retry once on 000 + code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions", + method="POST", + bearer_token=monitor_key, + data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}', + timeout=60) + + if code == 000 and failure_kind: + # Retry once with same timeout + time.sleep(1) + code, failure_kind = probe_http("http://" + BACKEND_HOST + "/litellm/v1/chat/completions", + method="POST", + bearer_token=monitor_key, + data='{"model":"syslog-auto","messages":[{"role":"user","content":"health ' + str(random.randint(1000, 9999)) + '"}],"max_tokens":4}', + timeout=60) + if code == 000 and failure_kind: + results.append(("syslog-auto", False, "probe-failed: syslog-auto " + failure_kind + " (60s timeout, retry)")) + elif code in (401, 403): + body = get_response_body("http://" + BACKEND_HOST + "/litellm/v1/chat/completions", + method="POST", + bearer_token=monitor_key, + data='{"model":"syslog-auto","messages":[{"role":"user","content":"health"}],"max_tokens":4}', + timeout=10) + alias = "monitor-20260813" + results.append(("syslog-auto", False, str(code) + " credential fault: body=" + body + " key_alias=" + alias)) + else: + results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)")) + else: + results.append(("syslog-auto", code == 200, str(code) + " (target: " + BACKEND_HOST + "/litellm/v1/chat/completions, model=syslog-auto)")) + + return results + +def check_admin_key_list(): + """Step 8: Admin API key list - use two-step approach""" + # Step 1: Get master key + mk_cmd = "ssh -o BatchMode=yes root@192.168.68.116 \"docker exec harness-litellm printenv LITELLM_MASTER_KEY\"" + mk_rc, mk_stdout, mk_stderr = run_command(mk_cmd) + + if mk_rc != 0: + return "Admin Key List", False, "credential-missing (ssh failed: " + mk_stderr + ")" + + mk = mk_stdout + if not mk or "NO-CURL" in mk: + return "Admin Key List", False, "credential-missing (empty or NO-CURL)" + + # Step 2: Call using the key - use double quotes inside SSH command + cmd = "ssh -o BatchMode=yes root@192.168.68.116 \"curl -s -H \\\"Authorization: Bearer " + mk + "\\\" http://127.0.0.1:4000/key/list\"" + rc, stdout, stderr = run_command(cmd) + + if rc != 0: + return "Admin Key List", False, "admin-call-failed (exit=" + str(rc) + ", stderr=" + stderr + ")" + + # Try to parse the response + try: + data = json.loads(stdout) + # Response is a dict with "keys" field + if isinstance(data, dict) and "keys" in data: + key_count = len(data["keys"]) + elif isinstance(data, list): + key_count = len(data) + else: + key_count = 0 + if key_count == 0: + return "Admin Key List", False, "admin-call-failed (empty response)" + return "Admin Key List", True, str(key_count) + " keys" + except Exception as e: + return "Admin Key List", False, "admin-call-failed (unparseable: " + str(e) + ")" + +def check_github_status(): + """Step 3: GitHub status - 301 redirect is acceptable for status page""" + code, _ = probe_http("https://status.github.com/api/status.json", timeout=15) + # GitHub status API returns 301 redirect, which is expected behavior + return "GitHub Status", code == 301, str(code) + +def check_prometheus(): + """Step 4: Prometheus health""" + code, _ = probe_http("http://" + BACKEND_HOST + ":9090/-/healthy") + return "Prometheus", code == 200, str(code) + " (target: " + BACKEND_HOST + ":9090/-/healthy)" + +def check_grafana(): + """Step 9: Grafana health""" + code, _ = probe_http("http://" + BACKEND_HOST + ":3001/api/health") + return "Grafana", code == 200, str(code) + " (target: " + BACKEND_HOST + ":3001/api/health)" + +def check_docker_stats(): + """Step 10: Docker Stats health - fetch from CT 116 host""" + # Docker stats is on localhost from CT 116 + cmd = "ssh -o BatchMode=yes root@192.168.68.116 'curl -s http://127.0.0.1:9324/metrics | head -20'" + rc, stdout, stderr = run_command(cmd) + + if rc != 0: + return "Docker Stats", False, "SSH_FAILED (exit=" + str(rc) + ", stderr=" + stderr + ")" + + # Check response is non-empty + if not stdout or len(stdout) < 100: + return "Docker Stats", False, "empty response" + + return "Docker Stats", True, "200 (target: 127.0.0.1:9324/metrics from CT 116)" + +def main(): + print("🏥 LiteLLM Health Check v1.0.0") + print("📍 Backend edge: http://" + BACKEND_HOST) + print("") + + all_pass = True + + # Run all checks + checks = [ + check_liveliness(), + check_containers(), + check_prometheus(), + check_grafana(), + ] + + for result in checks: + name, passed, detail = result + status = "✅" if passed else "❌" + print(" " + status + " " + name + ": " + detail) + if not passed: + all_pass = False + + # Model probes + model_results = check_model_probes() + for name, passed, detail in model_results: + status = "✅" if passed else "❌" + print(" " + status + " " + name + ": " + detail) + if not passed: + all_pass = False + + # Admin key list + admin_result = check_admin_key_list() + status = "✅" if admin_result[1] else "❌" + print(" " + status + " Admin Key List: " + admin_result[2]) + if not admin_result[1]: + all_pass = False + + # GitHub status + github_result = check_github_status() + status = "✅" if github_result[1] else "❌" + print(" " + status + " " + github_result[0] + ": " + github_result[2]) + if not github_result[1]: + all_pass = False + + # Docker stats + docker_stats_result = check_docker_stats() + status = "✅" if docker_stats_result[1] else "❌" + print(" " + status + " " + docker_stats_result[0] + ": " + docker_stats_result[2]) + if not docker_stats_result[1]: + all_pass = False + + print("") + if all_pass: + print("✅ All checks passed") + return 0 + else: + print("❌ Some checks failed") + return 1 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/pct-run.sh b/scripts/pct-run.sh index afbf56a..d4239d5 100755 --- a/scripts/pct-run.sh +++ b/scripts/pct-run.sh @@ -11,11 +11,13 @@ set -euo pipefail # ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ── declare -A CT_NODES=( # amdpve (192.168.68.15) - [111]=amdpve # tdunna + [105]=amdpve # kagentz (was hwepve — corrected 2026-09-12; live per pvesh) [112]=amdpve # tanko [113]=amdpve # baggy [115]=amdpve # scottdenya + [120]=amdpve # adguard2 (added 2026-09-12) # minipve (192.168.68.12) + [100]=minipve # abiba (was hwepve) [102]=minipve # adguard (was acerpve) [104]=minipve # authentik [110]=minipve # gitea @@ -25,17 +27,16 @@ declare -A CT_NODES=( [106]=storepve # ra-h-os [107]=storepve # proxmox-backup [108]=storepve # media + [111]=storepve # tdunna (was amdpve — corrected 2026-09-12; live per pvesh) [117]=storepve # zulip [118]=storepve # jdownloader # acerpve (192.168.68.9) — no CTs (bare metal GPU .8) - [100]=minipve # abiba (was hwepve) - [105]=minipve # kagentz (was hwepve) # ocupve (192.168.68.5) — no CTs (bare metal GPU .110) # # REMOVED CTs (migrated to bare metal, decommissioned, or VMs): - # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) - # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) - # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) + # 101 llm-gpu → bare metal 192.168.68.8 (RTX 3090) [QEMU VM on acerpve] + # 103 ocu-llm → bare metal 192.168.68.110 (RTX 5070) [QEMU VM on ocupve] + # 109 docker-vm → KVM VM 192.168.68.7 (use direct SSH) [QEMU VM on storepve] ) # Each node must be root-accessible via SSH hostname diff --git a/scripts/prose-ai-review.sh b/scripts/prose-ai-review.sh index cefc089..76c394a 100755 --- a/scripts/prose-ai-review.sh +++ b/scripts/prose-ai-review.sh @@ -47,17 +47,19 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol The infrastructure-control.prose.md contract is the canonical reference for the cluster topology: **Proxmox Cluster "Tabiri" (5 nodes):** -- amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya -- minipve (192.168.68.12): abiba, kagentz, adguard, authentik, gitea, syslog-api, infisical-vault -- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip +- amdpve (192.168.68.15): kagentz, tanko, baggy, scottdenya, adguard2 +- minipve (192.168.68.12): abiba, adguard, authentik, gitea, syslog-api, infisical-vault +- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip, tdunna - acerpve (192.168.68.9): llm-gpu - ocupve (192.168.68.5): ocu-llm -**CT IDs (verified 2026-07-24 against PVE API):** +**CT IDs (verified 2026-09-12 against PVE API):** 100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os 107:pbs 108:media 110:gitea 111:tdunna 112:tanko 113:baggy 115:scottdenya 116:syslog-api 117:zulip -118:jdownloader 119:infisical-vault +118:jdownloader 119:infisical-vault 120:adguard2 + +**CT 111 (tdunna, 192.168.68.129) is REPORT-ONLY — Theo's box; alert only, never garbage-collect.** **NO CT 122, CT 123, or .19 exist in the cluster.** diff --git a/scripts/zulip-monitor.sh b/scripts/zulip-monitor.sh index a6fd6f2..45838db 100755 --- a/scripts/zulip-monitor.sh +++ b/scripts/zulip-monitor.sh @@ -41,7 +41,9 @@ notify() { # ── Global: Zulip Server ── SERVER_CODE=$(curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 \ https://chat.sysloggh.net/api/v1/server_settings \ - -u 'abiba-bot@chat.sysloggh.net:cKTDMZAPW08dk3zl05sStzO7HRztzyn8' 2>/dev/null || echo "000") + -u 'abiba-bot@chat.sysloggh.net:cKTDMZAPW08dk3zl05sStzO7HRztzyn8' 2>/dev/null) || SERVER_CODE="000" +SERVER_CODE=$(printf '%s' "$SERVER_CODE" | tr -d '[:space:]') +[ -n "$SERVER_CODE" ] || SERVER_CODE="000" if [ "$SERVER_CODE" != "200" ]; then notify "🔴" "Zulip server returned HTTP $SERVER_CODE" ISSUES=$((ISSUES + 1)) @@ -59,7 +61,9 @@ fi # zulip.connected is a PROBE FAILURE: it alerts and NEVER calls pm2 restart. # pm2 restart runs ONLY on affirmative zulip.connected=false. # -- abiba-leg-start (verbatim-extracted by tests/zulip-monitor-abiba.sh) -PI_HTTP=$(curl -s -o /dev/null --connect-timeout 5 --max-time 10 -w '%{http_code}' http://localhost:9200/health 2>/dev/null || echo "000") +PI_HTTP=$(curl -s -o /dev/null --connect-timeout 5 --max-time 10 -w '%{http_code}' http://localhost:9200/health 2>/dev/null) || PI_HTTP="000" +PI_HTTP=$(printf '%s' "$PI_HTTP" | tr -d '[:space:]') +[ -n "$PI_HTTP" ] || PI_HTTP="000" PI_BODY=$(curl -s --connect-timeout 5 --max-time 10 http://localhost:9200/health 2>/dev/null || true) PI_STATE=$(printf '%s' "$PI_BODY" | python3 -c ' import sys, json @@ -156,29 +160,23 @@ fi # ── Platform C: Agent Zero (kagentz) ── AZ_A2A_CODE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ - "docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/ 2>/dev/null" 2>/dev/null || echo "000") + "docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/ 2>/dev/null" 2>/dev/null) || AZ_A2A_CODE="000" +AZ_A2A_CODE=$(printf '%s' "$AZ_A2A_CODE" | tr -d '[:space:]') +[ -n "$AZ_A2A_CODE" ] || AZ_A2A_CODE="000" if [ "$AZ_A2A_CODE" = "000" ]; then - notify "🔴" "kagentz A2A server DOWN (connection failed) — restarting" - ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ - "docker exec agent-zero bash -c 'pkill -9 -f a2a_agent; sleep 1; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &'" 2>/dev/null || true + notify "🔴" "kagentz A2A server DOWN (connection failed)" ISSUES=$((ISSUES + 1)) - echo " kagentz: ❌ A2A down — restarted" >> "$LOG" + echo " kagentz: ❌ A2A down (HTTP 000)" >> "$LOG" else - echo " kagentz: ✅ A2A alive" >> "$LOG" - - # Check adapter process - AZ_ADAPTER=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ - "docker exec agent-zero ps aux 2>/dev/null | grep adapter | grep -v grep | wc -l" 2>/dev/null || echo "0") - if [ "$AZ_ADAPTER" -lt 1 ]; then - notify "🔴" "kagentz Zulip adapter DOWN — restarting" - ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.14 \ - "docker exec agent-zero bash -c 'cd /a0/usr/kagentz-zulip && ZULIP_SITE=https://chat.sysloggh.net ZULIP_EMAIL=kagentz-bot@chat.sysloggh.net ZULIP_API_KEY=E9q9PXJTxftPYBkb5pBDWupDO7KK21ty ZULIP_AGENT_NAME=kagentz A2A_URL=http://localhost:80/a2a A2A_TOKEN=8zNgdOEXzYxjQvTl /opt/venv-a0/bin/python3 -u adapter.py > /tmp/zulip-adapter.log 2>&1 &'" 2>/dev/null || true - ISSUES=$((ISSUES + 1)) - echo " kagentz: ❌ Adapter down — restarted" >> "$LOG" - else - echo " kagentz: ✅ Adapter running" >> "$LOG" - fi + case "$AZ_A2A_CODE" in + 200|401) + echo " kagentz: ✅ A2A alive (HTTP $AZ_A2A_CODE)" >> "$LOG" ;; + *) + notify "🟡" "kagentz A2A server answered HTTP $AZ_A2A_CODE — running, unexpected status" + ISSUES=$((ISSUES + 1)) + echo " kagentz: 🟡 A2A unexpected http=$AZ_A2A_CODE (running, warning)" >> "$LOG" ;; + esac fi # ── Summary ── diff --git a/tests/test_audit_hermes_config_alias.py b/tests/test_audit_hermes_config_alias.py new file mode 100644 index 0000000..d1c479e --- /dev/null +++ b/tests/test_audit_hermes_config_alias.py @@ -0,0 +1,191 @@ +"""Regression tests for the 2026-09-12 retired-alias sweep in audit-hermes-config.py. + +WHY THIS FILE EXISTS: the executable audit pushed agent configs toward a DEAD alias. +Rule 8 required `auxiliary.vision.model == "gpu-light"` and +`auxiliary.web_extract.model == "gpu-light"`, but `gpu-light` (and its raw predecessor +`gemma-4-12b`) were retired on 2026-09-12 and now return 400 `Invalid model name`; the +live RTX 5070 alias is `gpu-vision`. A config that adopted the correct canonical alias +therefore FAILED our own audit, so the audit was actively enforcing a broken config. + +These tests execute the real CLI (`python3 audit-hermes-config.py `) and assert +observable behaviour — exit code and the emitted rule message — for the live alias and +for both retired names. No network, vault, or SSH access is required. +""" +from __future__ import annotations + +import pathlib +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +AUDIT = ROOT / "audit-hermes-config.py" + +BASE = """ +model: + api_key: "" + api_key_env: LITELLM_API_KEY + base_url: http://192.168.68.116/v1 + max_tokens: 4096 + default: syslog-auto + provider: harness +fallback_providers: + provider: deepseek + model: deepseek-v4-flash + api_key_env: DEEPSEEK_API_KEY +compression: + model: syslog-auto + provider: harness + threshold: 0.65 + max_context_window: 131072 +auxiliary: + vision: + model: {alias} + provider: harness + web_extract: + model: {alias} + provider: harness + compression: + model: syslog-auto + provider: harness +delegation: + provider: harness +custom_providers: + - name: harness + key_env: LITELLM_API_KEY + base_url: http://192.168.68.116/v1 +""" + + +def _run_config(tmp_path, name, text): + cfg = tmp_path / name + cfg.write_text(text) + proc = subprocess.run( + [sys.executable, str(AUDIT), str(cfg)], + capture_output=True, text=True, + ) + return proc.returncode, proc.stdout + + +def _run(tmp_path, alias): + return _run_config(tmp_path, f"{alias}.yaml", BASE.format(alias=alias)) + + +def test_live_canonical_alias_passes(tmp_path): + """The RTX 5070 alias that actually resolves must satisfy Rule 8.""" + code, out = _run(tmp_path, "gpu-vision") + assert code == 0, out + assert "RESULT: PASS" in out + + +def test_retired_gpu_light_is_rejected(tmp_path): + """A config pinned to the retired alias must fail, not pass.""" + code, out = _run(tmp_path, "gpu-light") + assert code == 1, out + assert "auxiliary.vision.model must be gpu-vision" in out + assert "RESULT: FAIL" in out + + +def test_retired_gemma_is_rejected(tmp_path): + """The retired raw model name must fail Rule 8 as well.""" + code, out = _run(tmp_path, "gemma-4-12b") + assert code == 1, out + assert "auxiliary.vision.model must be gpu-vision" in out + assert "RESULT: FAIL" in out + + +def test_corrected_compression_example_passes(tmp_path): + """The corrected workaround (vision=gpu-vision, compression=syslog-auto) must PASS.""" + code, out = _run(tmp_path, "gpu-vision") + assert code == 0, out + assert "[Rule 7] compression.model must be syslog-auto (got 'syslog-auto')" in out + assert "[Rule 7] auxiliary.compression.model must be syslog-auto (got 'syslog-auto')" in out + assert "RESULT: PASS" in out + + +def test_retired_alias_in_delegation_is_rejected(tmp_path): + """delegation.model has no dedicated value rule, so a retired name there used to PASS.""" + code, out = _run_config( + tmp_path, + "delegation-gpu-light.yaml", + BASE.format(alias="gpu-vision").replace( + "delegation:\n provider: harness", + "delegation:\n provider: harness\n model: gpu-light", + ), + ) + assert code == 1, out + assert "delegation.model = 'gpu-light' is retired" in out + assert "RESULT: FAIL" in out + + +def test_retired_alias_in_custom_providers_is_rejected(tmp_path): + """custom_providers[*].model is model-bearing; a retired name there must fail.""" + code, out = _run_config( + tmp_path, + "custom-provider-gpu-light.yaml", + BASE.format(alias="gpu-vision").replace( + " - name: harness\n key_env: LITELLM_API_KEY", + " - name: harness\n model: gpu-light\n key_env: LITELLM_API_KEY", + ), + ) + assert code == 1, out + assert "custom_providers[0].model = 'gpu-light'" in out + assert "RESULT: FAIL" in out + + +def test_retired_raw_name_fails(tmp_path): + """Retired raw names no longer resolve (400), so they fail; the 2026-09-12 registry change moved qwen3.6-27B-code from raw-but-live to non-resolving.""" + code, out = _run_config( + tmp_path, + "retired-qwen.yaml", + BASE.format(alias="gpu-vision").replace( + "delegation:\n provider: harness", + "delegation:\n provider: harness\n model: qwen3.6-27B-code", + ), + ) + assert code != 0, out + assert "delegation.model = 'qwen3.6-27B-code' is retired and no longer resolves" in out + assert "use gpu-dense" in out + assert "RESULT: FAIL" in out + + +def test_retired_alias_in_fallback_providers_is_rejected(tmp_path): + """fallback_providers.model is model-bearing; a retired name there must fail.""" + code, out = _run_config( + tmp_path, + "fallback-gpu-light.yaml", + BASE.format(alias="gpu-vision").replace(" model: deepseek-v4-flash", " model: gpu-light"), + ) + assert code == 1, out + assert "fallback_providers.model = 'gpu-light'" in out + assert "RESULT: FAIL" in out + + +def test_retired_alias_in_x_search_is_rejected(tmp_path): + """x_search.model was previously not enumerated; the derivation must catch it.""" + code, out = _run_config( + tmp_path, + "x-search-gpu-light.yaml", + BASE.format(alias="gpu-vision").replace( + "delegation:\n provider: harness", + "delegation:\n provider: harness\nx_search:\n model: gpu-light", + ), + ) + assert code == 1, out + assert "x_search.model = 'gpu-light'" in out + assert "RESULT: FAIL" in out + + +def test_retired_alias_in_nested_auxiliary_block_is_rejected(tmp_path): + """A nested auxiliary sub-block outside the named three must still be derived.""" + code, out = _run_config( + tmp_path, + "nested-aux-gpu-light.yaml", + BASE.format(alias="gpu-vision").replace( + " compression:\n model: syslog-auto\n provider: harness\ndelegation:", + " compression:\n model: syslog-auto\n provider: harness\n" + " tasks:\n summarize:\n model: gpu-light\ndelegation:", + ), + ) + assert code == 1, out + assert "auxiliary.tasks.summarize.model = 'gpu-light'" in out + assert "RESULT: FAIL" in out diff --git a/tests/test_disk_gc_report_only.py b/tests/test_disk_gc_report_only.py new file mode 100644 index 0000000..cd19d82 --- /dev/null +++ b/tests/test_disk_gc_report_only.py @@ -0,0 +1,185 @@ +"""Regression tests for the disk-gc report-only gate (CT 111 / tdunna / .129). + +WHY THIS FILE EXISTS: `disk-gc-threat-response.prose.md` defined AMBER as "GC scheduled +for next run" and its Execution loop called `gc-executor` for EVERY threat, with no +guest-level exclusion. CT 111 (tdunna, 192.168.68.129) belongs to Theo and is +report-only per the captain (2026-08-17, re-confirmed 2026-09-10) — so a single AMBER +reading on that guest would have scheduled GC commands (apt clean, journal vacuum, +log/tmp deletion, snap removal) against someone else's box. The only marker was +frontmatter `report_only_agents`, which names an AGENT while the scan unit is a GUEST. + +These tests execute the real planner (`scripts/disk-gc-plan.py`) and assert observable +behaviour: an excluded guest never produces a `gc-executor` action at any level, while +our own guests still do. +""" +from __future__ import annotations + +import json +import pathlib +import subprocess +import sys + +ROOT = pathlib.Path(__file__).resolve().parent.parent +PLAN = ROOT / "scripts" / "disk-gc-plan.py" + + +def _plan(scan, tmp_path): + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps(scan)) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def _actions_for(plan, target): + return [row for row in plan if str(row["target"]) == str(target)] + + +def test_excluded_guest_never_gets_gc_at_any_level(tmp_path): + """CT 111 at AMBER, RED and CRITICAL — always report-only, never gc-executor.""" + for pct, level in ((84, "AMBER"), (90, "RED"), (97, "CRITICAL")): + plan = _plan([{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", + "usage_pct": pct}], tmp_path) + rows = _actions_for(plan, 111) + assert rows, f"CT 111 must still be reported at {level}" + assert rows[0]["level"] == level + assert rows[0]["action"] == "report-only", rows + assert not any(r["action"] == "gc-executor" for r in rows) + + +def test_exclusion_matches_on_any_identity_key(tmp_path): + """The gate is keyed on guest/host, so id, ct, ctid, hostname or IP all match.""" + for entry in ({"id": 111, "usage_pct": 95}, + {"ct": 111, "usage_pct": 95}, + {"ctid": 111, "usage_pct": 95}, + {"hostname": "tdunna", "usage_pct": 95}, + {"ip": "192.168.68.129", "usage_pct": 95}): + plan = _plan([entry], tmp_path) + assert all(r["action"] == "report-only" for r in plan), (entry, plan) + + +def test_exclusion_matches_encoded_identities(tmp_path): + """A differently-encoded CT 111 id must not slip past the gate to gc-executor.""" + encodings = ({"id": 111.0}, + {"id": "111.0"}, + {"id": "lxc/111"}, + {"id": "qemu/111"}, + {"id": "0111"}, + {"id": " 111 "}) + for alias in encodings: + plan = _plan([{**alias, "usage_pct": 97}], tmp_path) + assert plan, alias + assert plan[0]["action"] == "report-only", (alias, plan) + + +def test_our_own_guests_still_get_gc(tmp_path): + """acerpve .9 and amdpve .15 are ours — they must still be acted on.""" + plan = _plan([{"hostname": "acerpve", "ip": "192.168.68.9", "usage_pct": 77}, + {"hostname": "amdpve", "ip": "192.168.68.15", "usage_pct": 76}], tmp_path) + assert len(plan) == 2 + assert all(r["action"] == "gc-executor" for r in plan), plan + + +def test_below_threshold_emits_nothing(tmp_path): + """GREEN guests produce no action at all.""" + assert _plan([{"id": 111, "usage_pct": 40}], tmp_path) == [] + + +def test_agent_name_alone_does_not_gate_a_guest(tmp_path): + """An agent-name marker must not be the gate: an unrelated guest still gets GC.""" + plan = _plan([{"id": 999, "hostname": "koby", "usage_pct": 95}], tmp_path) + assert plan and plan[0]["action"] == "gc-executor" + + +def test_unidentified_threat_fails_closed(tmp_path): + """A threshold-crossing entry with no recognized identity must not schedule GC.""" + plan = _plan([{"usage_pct": 97}], tmp_path) + assert plan, "an unidentified threat must still be reported" + assert plan[0]["action"] == "report-only", plan + assert plan[0]["reason"], plan + + +def test_exclusion_entry_without_identity_fails_closed(tmp_path): + """A mis-typed exclusion entry must break the run, never silently disable the gate.""" + contract = tmp_path / "broken.prose.md" + contract.write_text( + "```yaml\n" + "report_only_guests:\n" + " - node: storepve\n" + " reason: \"typo - no guest identity\"\n" + "```\n" + ) + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}])) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode != 0, proc.stdout + assert "gc-executor" not in proc.stdout + assert "identity" in proc.stderr.lower(), proc.stderr + + +def test_empty_exclusion_block_fails_closed(tmp_path): + """An emptied report_only_guests list must break the run, not disable the gate.""" + contract = tmp_path / "empty.prose.md" + contract.write_text("```yaml\nreport_only_guests: []\n```\n") + scan_file = tmp_path / "scan.json" + scan_file.write_text(json.dumps([{"ct": 111, "usage_pct": 97}])) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode != 0, proc.stdout + assert "gc-executor" not in proc.stdout + assert "report-only gate" in proc.stderr.lower(), proc.stderr + + +def _plan_with_contract(scan, contract_text, tmp_path, name): + contract = tmp_path / name + contract.write_text(contract_text) + scan_file = tmp_path / f"scan-{name}.json" + scan_file.write_text(json.dumps(scan)) + proc = subprocess.run( + [sys.executable, str(PLAN), "--scan", str(scan_file), + "--contract", str(contract), "--json"], + capture_output=True, text=True, + ) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +def test_gate_is_read_from_the_contract_block(tmp_path): + """The gate is data-driven by the contract block: the planner excludes the guest + when the block names it and acts on it when the block does not. Executes the real + planner interface against both fixtures so the behaviour change is observable.""" + scan = [{"id": 111, "hostname": "tdunna", "ip": "192.168.68.129", "usage_pct": 95}] + with_gate = ( + "```yaml\n" + "report_only_guests:\n" + " - guest: 111\n" + " hostname: tdunna\n" + " ip: 192.168.68.129\n" + " reason: \"fixture reason\"\n" + "```\n" + ) + without_gate = ( + "```yaml\n" + "report_only_guests:\n" + " - guest: 999\n" + " reason: \"fixture excludes a different guest\"\n" + "```\n" + ) + + gated = _plan_with_contract(scan, with_gate, tmp_path, "gated.prose.md") + assert gated[0]["action"] == "report-only", gated + assert gated[0]["reason"] == "fixture reason", gated + + ungated = _plan_with_contract(scan, without_gate, tmp_path, "ungated.prose.md") + assert ungated[0]["action"] == "gc-executor", ungated + assert ungated[0]["action"] != gated[0]["action"] diff --git a/tests/test_mumuni_monitor_removal.py b/tests/test_mumuni_monitor_removal.py index 5f9b9eb..df55352 100644 --- a/tests/test_mumuni_monitor_removal.py +++ b/tests/test_mumuni_monitor_removal.py @@ -89,8 +89,7 @@ case "$host" in esac ;; 192.168.68.14) case "$cmd" in - *agent.json*) printf '%s' "$AZ_A2A" ;; - *"ps aux"*) printf '%s\n' "$AZ_PS" ;; + *"/a2a/"*) printf '%s' "$AZ_A2A_CODE"; exit "$AZ_A2A_EXIT" ;; esac ;; *) printf 'UNEXPECTED-SSH-HOST %s\n' "$host" >> "$RECORD_DIR/unexpected-ssh" ;; @@ -122,8 +121,7 @@ def _write_exec(path: pathlib.Path, body: str) -> None: def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200", - az_a2a='{"name":"kagentz"}', - az_ps="root 111 0.1 0.2 /opt/venv-a0/bin/python3 -u adapter.py"): + az_a2a_code="401", az_a2a_exit=0): """Run the shipped monitor in a sandbox; return (proc, record_dir, log_path). Only the LOG constant is rewritten (to keep the run inside the worktree). @@ -151,8 +149,8 @@ def _run_monitor(tmp_path, *, tanko_svc="active", tanko_http="200", "RECORD_DIR": str(record), "TANKO_SVC": tanko_svc, "TANKO_HTTP": tanko_http, - "AZ_A2A": az_a2a, - "AZ_PS": az_ps, + "AZ_A2A_CODE": az_a2a_code, + "AZ_A2A_EXIT": str(az_a2a_exit), "PI_HTTP": "200", "PI_BODY": CONNECTED_FIXTURE.read_text(), "SERVER_HTTP": "200", @@ -171,8 +169,7 @@ def test_healthy_run_is_quiet_and_never_reaches_mumuni(tmp_path): assert "Server: ✅ HTTP 200" in log assert "Abiba: ✅ Connected" in log assert "Tanko: ✅ service=active http=200" in log - assert "kagentz: ✅ A2A alive" in log - assert "kagentz: ✅ Adapter running" in log + assert "kagentz: ✅ A2A alive (HTTP 401)" in log assert "Result: ✅ All healthy" in log # A healthy run emits no notify at all — and certainly no Mumuni one. @@ -214,6 +211,32 @@ def test_failing_run_alerts_on_tanko_but_never_on_mumuni(tmp_path): assert "Result: 🔴 1 issue(s) found" in log +def test_unexpected_a2a_status_is_an_issue_not_healthy(tmp_path): + proc, record, log_path = _run_monitor(tmp_path, az_a2a_code="500") + assert proc.returncode == 0, proc.stderr + log = log_path.read_text() + + assert "kagentz: 🟡 A2A unexpected http=500 (running, warning)" in log + assert "kagentz: ✅ A2A alive" not in log + assert "Result: 🔴 1 issue(s) found" in log + assert "kagentz A2A server answered HTTP 500" in proc.stdout + + +def test_a2a_connection_failure_is_down_not_unexpected(tmp_path): + # curl prints the http_code before failing, so the ssh stub exits non-zero + # with "000" on stdout — exercising the real outage path. + proc, record, log_path = _run_monitor(tmp_path, az_a2a_code="000", + az_a2a_exit=7) + assert proc.returncode == 0, proc.stderr + log = log_path.read_text() + + assert "kagentz: ❌ A2A down (HTTP 000)" in log + assert "kagentz: ✅ A2A alive" not in log + assert "unexpected" not in log + assert "Result: 🔴 1 issue(s) found" in log + assert "kagentz A2A server DOWN (connection failed)" in proc.stdout + + # ── scripts/daily-infra-report.py: behavioral digest checks ────────── @pytest.fixture(scope="module") @@ -334,7 +357,8 @@ def test_agent_health_roster_has_no_mumuni_entry(ahc): def test_health_contract_retires_mumuni_only_steps(): text = HEALTH_CONTRACT.read_text() assert MUMUNI_IP not in text - for step in ("**B4:", "**B5:", "**B6:"): + for step in ("**B4: Gateway Process**", "**B5: Heartbeat Verification**", + "**B6: Response Delivery**"): assert step not in text diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 4e9cfb6..54229f8 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -1,9 +1,9 @@ --- kind: responsibility name: zulip-health -description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side. +description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (pi/Abiba Zulip bridge), Platform B (Tanko on DSH), and Platform C (Agent Zero Docker). Verifies bot registration, DM delivery, and cross-platform connectivity. The kagentz Zulip adapter leg is retired (its code no longer exists) — Agent Zero is probed for A2A liveness only. Mumuni is no longer monitored from this host — she runs on her own container (kagentz CT 105 on minipve, .14) and is monitored on her side. title: Zulip Mesh Health Monitor — Multi-Platform -version: 3.2.0 +version: 3.3.0 runtime_contract: 2 agent: abiba report_only_agents: @@ -127,20 +127,48 @@ grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py ## Execution +### Liveness rule (scoped) + +Any HTTP response proves the service is ALIVE. For auth-gated endpoints (Zulip API, Tanko gateway), a 401/403 redirect or status means the service answered — report the code, never "down". Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe. + +**STANDING PROBE RULES (2026-09-14, from defect report 1150.msg):** +1. **Any HTTP status means ALIVE.** 200, 301, 302, 401, 403, 404 all prove the service answered — report the code, never "down". A redirect is not a failure. Only a failed CONNECTION (curl status 000, timeout, refused) is a failed probe. +2. **A failed probe is never a service verdict.** Print `probe-failed: ` naming the exact URL/host/port and the failure kind (timeout, refused, no-route, dns), retry once at a longer timeout, and only then report. +3. **Say which probe produced each number.** "API: 000" is unusable; "API https://chat.sysloggh.net/api/v1/server_settings -> connection timeout after 10s (retried at 25s: also timeout)" is actionable. + ### Step 1: Zulip Server Liveness ```bash -curl -s -o /dev/null -w "%{http_code}" https://chat.sysloggh.net/api/v1/server_settings \ - -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY' +# Probe the Zulip API (authenticated, any HTTP status = ALIVE) +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 https://chat.sysloggh.net/api/v1/server_settings -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY') +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 https://chat.sysloggh.net/api/v1/server_settings -u 'abiba-bot@chat.sysloggh.net:$ZULIP_API_KEY') + echo "Zulip API https://chat.sysloggh.net/api/v1/server_settings -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "Zulip API https://chat.sysloggh.net/api/v1/server_settings -> $code" +fi ``` -Expected: `200`. If not → mark `zulip_server_status: "down"`, skip per-platform checks, alert. +Expected: `200` (authenticated). Any HTTP status = ALIVE; only 000/timeout = probe-failed. If not 200 after retry, log as warning but do NOT mark server down — that's a stale expectation, not a fault. ### Step 2: Platform A — pi (Abiba, localhost) **A1: Health Endpoint** -Fetch `http://localhost:9200/health` as JSON. Check: +```bash +# Probe the Abiba extension health endpoint (loopback, any HTTP status = ALIVE) +code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 10 http://127.0.0.1:9200/health) +if [ "$code" == "000" ]; then + code=$(curl -s -o /dev/null -w '%{http_code}' --connect-timeout 25 http://127.0.0.1:9200/health) + echo "Abiba extension http://127.0.0.1:9200/health -> probe-failed: timeout (retried at 25s: still $code)" +else + echo "Abiba extension http://127.0.0.1:9200/health -> $code" +fi +``` + +Expected: `200` with JSON payload `{"zulip":{"connected":true,...}}`. Any HTTP status = ALIVE; only 000/timeout = probe-failed. **NOTE: This probe MUST run on the Abiba host (CT 100) where 127.0.0.1:9200 is the extension. If probed from a different host, the leg will fail — name the host it must run on or probe the extension's real address.** + +Check the JSON payload: | Field | Healthy | Critical | |-------|---------|----------| @@ -435,36 +463,29 @@ ssh root@192.168.68.15 "pct exec 112 -- curl -s -b /tmp/dsh-new.jar -o /dev/null ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) +> **The kagentz Zulip adapter leg is retired (2026-09-12).** Its code +> (`/a0/usr/kagentz-zulip/`) no longer exists in the agent-zero container, so +> the former adapter-process and heartbeat/queue checks always failed and the +> monitor issued a restart for something that could not start, posting a false +> kagentz-adapter-down alert on every run. Do NOT re-add an adapter-process, +> heartbeat/queue, or adapter-restart step. Agent Zero is probed for A2A +> liveness only, and a probe must never restart a platform. + **C1: A2A Server Health** ```bash -# A2A server is on :50080 (not :8001) and is auth-gated (401 expected for unauthenticated) -ssh root@192.168.68.14 "curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:50080/a2a/" +# A2A listens on :80 inside the agent-zero container (host-mapped to :50080) and +# is auth-gated: an unauthenticated probe gets 401, which means the server is up. +ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:80/a2a/" ``` -Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (000) → A2A server down. +Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (`000`) → A2A server down. Any other status → running but unexpected: log/report it, never restart. -**C2: Adapter Process** +**C2: A2A Response Verification** ```bash -ssh root@192.168.68.14 "docker exec agent-zero ps aux | grep adapter | grep -v grep" -``` - -Adapter should be running. Missing → restart inside container. - -**C3: Heartbeat & Queue** - -```bash -ssh root@192.168.68.14 "docker exec agent-zero grep Heartbeat /tmp/zulip-adapter.log | tail -3" -``` - -Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0. - -**C4: A2A Response Verification** - -```bash -# A2A server is on :50080 (not :8001) and is auth-gated (401 expected for unauthenticated) -ssh root@192.168.68.14 "curl -s -X POST http://127.0.0.1:50080/a2a \ +# A2A listens on :80 inside the container and is auth-gated (401 expected unauthenticated). +ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:80/a2a \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer $LITELLM_KEY' \ -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'" @@ -476,9 +497,8 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`. I | Condition | Action | |-----------|--------| -| A2A `.well-known/agent.json` fails | `docker exec agent-zero bash -c "pkill -9 -f a2a_agent; cd /a0 && /opt/venv-a0/bin/python3 -u /a0/usr/a2a_agent.py > /tmp/a2a.log 2>&1 &"` | -| Adapter process missing | Restart adapter inside container with env vars | -| Silence > 600s | Restart adapter (auto-reconnect handles BAD_EVENT_QUEUE_ID) | +| A2A returns `000` (connection refused/timeout) | Alert only — never restart the platform; investigate the agent-zero container | +| A2A returns a status other than `200`/`401` | Log/report as a warning — reported, never healed on | | LiteLLM 401 | Check API key in a2a_agent.py `LITELLM_KEY` | ### Step 5: Global Checks @@ -488,7 +508,6 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`. I Check each agent's log for excessive bot-to-bot chatter: - Abiba: `Skipped.*bot msgs` count - Tanko: Repeated DM exchanges between bots -- kagentz: Adapter log for bot DMs being processed If any bot processes >50 bot-originated messages in 15min → warning. diff --git a/zulip-mention-reliability.prose.md b/zulip-mention-reliability.prose.md index 2e0a490..21e0f2a 100644 --- a/zulip-mention-reliability.prose.md +++ b/zulip-mention-reliability.prose.md @@ -10,8 +10,9 @@ description: > > **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and > PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability -> monitoring now happens through Telegram. Agents (Mumuni on Hermes, Tanko on DSH) and -> Agent Zero (kagentz) continue to use Zulip. +> monitoring now happens through Telegram. Mumuni (Hermes) and Tanko (DSH) +> continue to use Zulip; Agent Zero's Zulip adapter is retired — see +> `zulip-health.prose.md` for current Platform C (Agent Zero) state. ## Maintains