SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

prose: align all contracts with Vulkan rebuild + Koby key regen

Browse Source 
- gpu-fleet: Koby key updated (sk-lDTsWy7H3T...), config table (ornith service
  + masked services), benchmarks (prompt 209→532 tok/s), ROCm→Vulkan status
- gpu-monitor: topology (sidecars :8090 removed, .15 via router), subsystem
  polling (router /health/unified as source of truth), execution steps,
  alert threshold (sidecar-unreachable downgraded from critical)
This commit is contained in:
root
2026-07-02 00:29:10 +00:00
parent 49e8d968f5
commit a07ff2d96a
2 changed files with 26 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
gpu-monitor.prose.md
+20 -15
View File
@@ -21,28 +21,33 @@ agent: abiba
┌─────────────────────────────────────────────────────────┐
│ GPU Monitor Server (:9100) — 192.168.68.65 │
│ Polls every 15s, serves dashboard + JSON API │
└──┬──────────┬──────────┬──────────┬─────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌──────┐ ┌──────┐ ┌────────┐ ┌──────────┐
│.8:8090│ │.110 │ │.116:80 │ │.15 (SSH) │
│RTX3090│ │:8090 │ │nginx │ │Strix Halo│
│gemma │ │RTX5070│ │router │ │ornith35B │
└──────┘ │qwen27B│ │LiteLLM │ └──────────┘
└──┬──────────┬──────────┬────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────┐ ┌──────┐ ┌────────┐
│.8:8080│ │.110 │ │.116:80 │
│RTX3090│ │:8080 │ │nginx │
│gemma │ │RTX5070│ │router │
└──────┘ │qwen27B│ │LiteLLM │
└──────┘ │dashboard│
└────────┘
Note: JSON sidecar exporters at :8090 were never deployed on any
GPU host. Router falls back to GPU /health direct probe. Monitor
should use router /health/unified as source of truth for GPU status.
Strix Halo :8080 is firewalled to .116 only — monitor on .65 cannot
poll .15:8080 directly; must go through router on .116.
```
### Subsystems Polled
| Subsystem | Endpoint | Frequency | Metrics |
|-----------|----------|-----------|---------|
| GPU Sidecar (RTX 3090) | `http://192.168.68.8:8090/health` | 15s | temp, VRAM, util, power, fan |
| GPU Sidecar (RTX 5070) | `http://192.168.68.110:8090/health` | 15s | temp, VRAM, util, power, fan |
| GPU Status (all, via router) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status (router probes each GPU /health directly) |
| Router (unified) | `http://192.168.68.116/health/unified` | 15s | models, CB, scores, GPU status |
| Router (basic) | `http://192.168.68.116/health` | 15s | basic aliveness |
| LiteLLM | `http://192.168.68.116/litellm/health` | 15s | proxy health, model count |
| Strix Halo | SSH `192.168.68.15` + `:8080/health` | 15s | process status, CPU load, llama health |
| Strix Halo | `http://192.168.68.116/health/unified` (router) | 15s | ornith status via router — cannot poll .15:8080 directly (firewalled to .116 only) |
| Dashboard | `http://192.168.68.116/dashboard/` | 15s | harness-dashboard aliveness |
### Alert Delivery
@@ -61,7 +66,7 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
| GPU Temp | >80°C | >90°C |
| VRAM Usage | >90% | >95% |
| GPU Util | >95% | >98% |
| Sidecar Unreachable | — | critical (connectivity) |
| Sidecar Unreachable | — | info (sidecars not deployed — use router /health/unified) |
| Model Down | — | critical (circuit breaker open) |
### JSON API Response Schema (/gpu-data)
@@ -119,10 +124,10 @@ The router health is accessed through nginx on port 80 (NOT port 9000 directly).
## Execution
1. **Poll sidecars** (every 15s): GET .8:8090/health, .110:8090/health
2. **Poll router** (every 15s): GET .116/health/unified + /health via nginx:80
1. **Poll router** (every 15s): GET .116/health/unified — single source of truth for all GPU status (router probes each GPU /health directly via sidecar fallback)
2. **Poll router** (every 15s): GET .116/health via nginx:80
3. **Poll LiteLLM** (every 15s): GET .116/litellm/health via nginx:80
4. **Poll Strix** (every 15s): SSH .15 for process check + GET .15:8080/health
4. **Poll Strix** (every 15s): via router /health/unified (cannot poll .15:8080 directly — firewalled to .116 only)
5. **Poll dashboard** (every 15s): GET .116/dashboard/
6. **Check alerts**: Compare metrics against thresholds
7. **Compute summary**: Fleet-wide health aggregation