From a07ff2d96ac0214e6dc323f4d0b2a1604e6c334b Mon Sep 17 00:00:00 2001 From: root Date: Thu, 2 Jul 2026 00:29:10 +0000 Subject: [PATCH] prose: align all contracts with Vulkan rebuild + Koby key regen MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service + masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status - gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem polling (router /health/unified as source of truth), execution steps, alert threshold (sidecar-unreachable downgraded from critical) --- gpu-fleet.prose.md | 9 ++++++--- gpu-monitor.prose.md | 35 ++++++++++++++++++++--------------- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 29b172c..3fbfe24 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -130,7 +130,7 @@ Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, k | Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome | | Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root | | Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local | -| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip DM | +| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) | | Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH | | Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH | @@ -150,7 +150,8 @@ If no SSH access, send Zulip DM via abiba-bot. | `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) | | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server | | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | -| `/etc/systemd/system/llama-server.service` | .8, .110, .15 | llama-server daemons | +| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) | +| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. | ## Prometheus & Grafana @@ -184,8 +185,10 @@ If no SSH access, send Zulip DM via abiba-bot. |-----|-------|-----------|--------------|----------|---------| | RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 | | RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 | -| Strix Halo (.15) | ornith-1.0-35b | 70 | 209 | 70 | 6 | +| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 | Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes. Degradation alerts fire at 30% (warning) and 50% (critical) below baseline. History stored at `/root/data/toks-history.json` with 7-day rolling window. + +**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration. diff --git a/gpu-monitor.prose.md b/gpu-monitor.prose.md index 80fbfbd..11cb793 100644 --- a/gpu-monitor.prose.md +++ b/gpu-monitor.prose.md @@ -21,28 +21,33 @@ agent: abiba ┌─────────────────────────────────────────────────────────┐ │ GPU Monitor Server (:9100) — 192.168.68.65 │ │ Polls every 15s, serves dashboard + JSON API │ -└──┬──────────┬──────────┬──────────┬─────────────────────┘ - │ │ │ │ - ▼ ▼ ▼ ▼ -┌──────┐ ┌──────┐ ┌────────┐ ┌──────────┐ -│.8:8090│ │.110 │ │.116:80 │ │.15 (SSH) │ -│RTX3090│ │:8090 │ │nginx │ │Strix Halo│ -│gemma │ │RTX5070│ │router │ │ornith35B │ -└──────┘ │qwen27B│ │LiteLLM │ └──────────┘ +└──┬──────────┬──────────┬────────────────────────────────┘ + │ │ │ + ▼ ▼ ▼ +┌──────┐ ┌──────┐ ┌────────┐ +│.8:8080│ │.110 │ │.116:80 │ +│RTX3090│ │:8080 │ │nginx │ +│gemma │ │RTX5070│ │router │ +└──────┘ │qwen27B│ │LiteLLM │ └──────┘ │dashboard│ └────────┘ + +Note: JSON sidecar exporters at :8090 were never deployed on any +GPU host. Router falls back to GPU /health direct probe. Monitor +should use router /health/unified as source of truth for GPU status. +Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot +poll .15:8080 directly; must go through router on .116. ``` ### Subsystems Polled | Subsystem | Endpoint | Frequency | Metrics | |-----------|----------|-----------|---------| -| GPU Sidecar (RTX 3090) | `http://192.168.68.8:8090/health` | 15s | temp, VRAM, util, power, fan | -| GPU Sidecar (RTX 5070) | `http://192.168.68.110:8090/health` | 15s | temp, VRAM, util, power, fan | +| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) | | Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status | | Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness | | LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count | -| Strix Halo | SSH `192.168.68.15` + `:8080/health` | 15s | process status, CPU load, llama health | +| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) | | Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness | ### Alert Delivery @@ -61,7 +66,7 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and | GPU Temp | >80°C | >90°C | | VRAM Usage | >90% | >95% | | GPU Util | >95% | >98% | -| Sidecar Unreachable | — | critical (connectivity) | +| Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) | | Model Down | — | critical (circuit breaker open) | ### JSON API Response Schema (/gpu-data) @@ -119,10 +124,10 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly). ## Execution -1. **Poll sidecars** (every 15s): GET .8:8090/health, .110:8090/health -2. **Poll router** (every 15s): GET .116/health/unified + /health via nginx:80 +1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback) +2. **Poll router** (every 15s): GET .116/health via nginx:80 3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80 -4. **Poll Strix** (every 15s): SSH .15 for process check + GET .15:8080/health +4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only) 5. **Poll dashboard** (every 15s): GET .116/dashboard/ 6. **Check alerts**: Compare metrics against thresholds 7. **Compute summary**: Fleet-wide health aggregation