SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

prose: align all contracts with Vulkan rebuild + Koby key regen

Browse Source 
- gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service
  + masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status
- gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem
  polling (router /health/unified as source of truth), execution steps,
  alert threshold (sidecar-unreachable downgraded from critical)
This commit is contained in:
root
2026-07-02 00:29:10 +00:00
parent 49e8d968f5
commit a07ff2d96a
2 changed files with 26 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gpu-fleet.prose.md
+6 -3
View File
@@ -130,7 +130,7 @@ Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, k
| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome | | Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root | | Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local | | Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
| Koby | `sk-wb86PvcZXWkjShAIBxrpDg` | ? | Zulip DM | | Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH | | Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH | | Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
@@ -150,7 +150,8 @@ If no SSH access, send Zulip DM via abiba-bot.
| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) | | `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server | | `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard | | `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
| `/etc/systemd/system/llama-server.service` | .8, .110, .15 | llama-server daemons | | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
## Prometheus & Grafana ## Prometheus & Grafana
@@ -184,8 +185,10 @@ If no SSH access, send Zulip DM via abiba-bot.
|-----|-------|-----------|--------------|----------|---------| |-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 | | RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 | | RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
| Strix Halo (.15) | ornith-1.0-35b | 70 | 209 | 70 | 6 | | Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes. Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline. Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
History stored at `/root/data/toks-history.json` with 7-day rolling window. History stored at `/root/data/toks-history.json` with 7-day rolling window.
**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
gpu-monitor.prose.md
+20 -15
View File
@@ -21,28 +21,33 @@ agent: abiba
┌─────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────┐
│ GPU Monitor Server (:9100) — 192.168.68.65 │ │ GPU Monitor Server (:9100) — 192.168.68.65 │
│ Polls every 15s, serves dashboard + JSON API │ │ Polls every 15s, serves dashboard + JSON API │
└──┬──────────┬──────────┬──────────┬─────────────────────┘ └──┬──────────┬──────────┬────────────────────────────────┘
│ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼
┌──────┐ ┌──────┐ ┌────────┐ ┌──────────┐ ┌──────┐ ┌──────┐ ┌────────┐
│.8:8090│ │.110 │ │.116:80 │ │.15 (SSH) │ │.8:8080│ │.110 │ │.116:80 │
│RTX3090│ │:8090 │ │nginx │ │Strix Halo│ │RTX3090│ │:8080 │ │nginx │
│gemma │ │RTX5070│ │router │ │ornith35B │ │gemma │ │RTX5070│ │router │
└──────┘ │qwen27B│ │LiteLLM │ └──────────┘ └──────┘ │qwen27B│ │LiteLLM │
└──────┘ │dashboard│ └──────┘ │dashboard│
└────────┘ └────────┘
Note: JSON sidecar exporters at :8090 were never deployed on any
GPU host. Router falls back to GPU /health direct probe. Monitor
should use router /health/unified as source of truth for GPU status.
Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
poll .15:8080 directly; must go through router on .116.
``` ```
### Subsystems Polled ### Subsystems Polled
| Subsystem | Endpoint | Frequency | Metrics | | Subsystem | Endpoint | Frequency | Metrics |
|-----------|----------|-----------|---------| |-----------|----------|-----------|---------|
| GPU Sidecar (RTX 3090) | `http://192.168.68.8:8090/health` | 15s | temp, VRAM, util, power, fan | | GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
| GPU Sidecar (RTX 5070) | `http://192.168.68.110:8090/health` | 15s | temp, VRAM, util, power, fan |
| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status | | Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness | | Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count | | LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
| Strix Halo | SSH `192.168.68.15` + `:8080/health` | 15s | process status, CPU load, llama health | | Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness | | Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
### Alert Delivery ### Alert Delivery
@@ -61,7 +66,7 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
| GPU Temp | >80°C | >90°C | | GPU Temp | >80°C | >90°C |
| VRAM Usage | >90% | >95% | | VRAM Usage | >90% | >95% |
| GPU Util | >95% | >98% | | GPU Util | >95% | >98% |
| Sidecar Unreachable | — | critical (connectivity) | | Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
| Model Down | — | critical (circuit breaker open) | | Model Down | — | critical (circuit breaker open) |
### JSON API Response Schema (/gpu-data) ### JSON API Response Schema (/gpu-data)
@@ -119,10 +124,10 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
## Execution ## Execution
1. **Poll sidecars** (every 15s): GET .8:8090/health, .110:8090/health 1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
2. **Poll router** (every 15s): GET .116/health/unified + /health via nginx:80 2. **Poll router** (every 15s): GET .116/health via nginx:80
3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80 3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
4. **Poll Strix** (every 15s): SSH .15 for process check + GET .15:8080/health 4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
5. **Poll dashboard** (every 15s): GET .116/dashboard/ 5. **Poll dashboard** (every 15s): GET .116/dashboard/
6. **Check alerts**: Compare metrics against thresholds 6. **Check alerts**: Compare metrics against thresholds
7. **Compute summary**: Fleet-wide health aggregation 7. **Compute summary**: Fleet-wide health aggregation