diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index fe09854..f55cf9b 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -27,6 +27,7 @@ triggers: - benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time - grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001) - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM +- port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding ## Fleet Topology (Current — June 2026) @@ -171,6 +172,7 @@ If no SSH access, send Zulip DM via abiba-bot. - **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput. - **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV. +- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116. - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback. diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 4126fee..e13e2a2 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -179,6 +179,25 @@ EnvironmentFile=/etc/environment # sources LITELLM_API_KEY | Documents | `~/.hermes/cache/documents/` | | Audio | `~/.hermes/cache/audio/` | +## Health Verification + +Run the consolidated health check: +```bash +python3 /root/scripts/agent-health-check.py +``` +This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes), +verifies gateway liveness, confirms Zulip streaming (`edit_message` present), +and counts recent errors. Non-disruptive — never restarts anything. + +## GPU Port Conflict Detection + +All 3 GPU hosts have pre-start ghost detection in their launch wrappers: +- `.8` and `.110`: inline check in `llama-wrapper.sh` +- `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`) + +Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`. +If they differ → ghost detected → kill ghost → start fresh. + ## Related Contracts - `hermes-key-enforcement.prose.md` — key policy, rotation, detection query @@ -191,4 +210,5 @@ EnvironmentFile=/etc/environment # sources LITELLM_API_KEY | Date | Change | |------|--------| +| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. | | 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied | diff --git a/infrastructure-control.prose.md b/infrastructure-control.prose.md index 4bbf48a..df956cf 100644 --- a/infrastructure-control.prose.md +++ b/infrastructure-control.prose.md @@ -632,3 +632,29 @@ ssh root@192.168.68.8 # RTX 3090 ssh root@192.168.68.110 # RTX 5070 ssh root@192.168.68.15 # Strix Halo ``` + +## Section 7: Agent Health Check (consolidated — 2026-07-05) + +Replaces 7 scattered Zulip health scripts with a single non-disruptive check +running every 10 minutes via cron (`/root/scripts/agent-health-check.py`). + +The script is **read-only** — it never restarts, kills, or modifies anything. +Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh, +which was kill+nohup outside systemd) are banned by policy. + +### Checks Performed + +| Check | Frequency | What It Detects | +|-------|-----------|-----------------| +| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models | +| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) | +| Gateway liveness | 10 min | Gateway process running, state file readable | +| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) | +| Recent errors | 10 min | Error count in journald for last 10 min | + +### Disabled Scripts + +| Script | Why Disabled | +|--------|-------------| +| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch | +| `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart | diff --git a/zulip-health.prose.md b/zulip-health.prose.md index cf0e3e1..5dcf8de 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -77,6 +77,24 @@ Log the condition as "pending restart" with the timestamp. If the condition pers Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks. ### Severity escalation + +## Streaming Support (2026-07-05) + +Zulip agents now support progressive message editing during agent generation. +When a Hermes agent (Tanko, Mumuni) processes a message, the response is +streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API: + +- Adapter implements `edit_message()` using `_api_patch()` helper +- Gateway stream consumer progressively edits the Zulip message +- User sees real-time agent thinking instead of waiting for full response +- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active + +### Verification +```bash +# Check if agent has streaming: +grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py +# Must return 1 — streaming is active +``` - Single agent warning → log only - Single agent critical → relay message to user - Two or more agents critical → immediate relay + attempt auto-recovery