SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b65116b84718fb0cfd9599bf27da53a5ec4e1661
prose-contracts/gpu-fleet.prose.md
T
root 0db95a4d1c
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped
Details
fix: correct IPs, CT IDs, and key status across all contracts 
hermes-key-enforcement.prose.md:
  - Add IP column to verified agents table
  - Fix Tanko status: was using master key via systemd drop-in (not compliant)
  - Add detection query step 2: check systemd drop-ins for master key leaks
  - Add step 3: verify running process env against dedicated key
  - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only)

gpu-fleet.prose.md:
  - Add CT ID column to agent table for cross-reference
  - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host)
  - Update Abiba LiteLLM key to match actual models.json key
  - Set Koonimo IP to unknown (was .114 — incorrect per user)
  - Fix pi-specific paths .24 → .65

hermes-config-template.prose.md:
  - Set Koonimo IP to unknown

infrastructure-control.prose.md:
  - Set Koonimo IP to unknown
  - Fix Koonimo CT reference
2026-07-05 19:58:14 +00:00

195 lines
13 KiB
Markdown
Raw Blame History

---
kind: responsibility
name: gpu-fleet
description: >
Manages the GPU inference fleet across all hosts. Handles model deployment,
registration, health checks, LiteLLM sync, agent key management, GPU
saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
router runs behind LiteLLM on :9000 (internal-only).
agent: abiba
triggers:
- on model add/remove
- on GPU health degradation
- on agent key rotation
- on router restart (roster must be loaded)
---
## Maintains
- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
- router: { status: "healthy", roster_loaded: bool, models: array }
- litellm: { status: "healthy", keys: array, models: array }
- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
- health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
- monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
- watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
- benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
- grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
## Fleet Topology (Current — June 2026)
```
┌──────────────────────────────────────────────────────────────────┐
│ CT 116 (192.168.68.116) — Inference Harness Host │
│ │
│ nginx:80 (entrypoint) │
│ ├─ /v1/* → harness-litellm:4000 (API requests) │
│ ├─ /admin/* → harness-litellm:4000 (admin endpoints) │
│ ├─ /dashboard/ → harness-dashboard:3000 (harness UI) │
│ ├─ /litellm/* → harness-litellm:4000 (LiteLLM UI + API) │
│ ├─ /health/* → harness-litellm:4000 (health probes) │
│ └─ /gpu/* → 192.168.68.24:9100 (fleet monitor) │
│ │
│ Containers: │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ LiteLLM │ │ Router │ │Dashboard │ │ Grafana │ │
│ │ :4000 │─▶│ :9000 │ │ :3000 │ │ :3000 │ │
│ │ keys+sync│ │internal │ │ harness │ │ Prometheus│ │
│ │ fallback │ │only! │ │ UI │ │ data src │ │
│ └──────────┘ └───┬──────┘ └──────────┘ └──────────┘ │
│ │ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │PostgreSQL│ │ Redis │ │Prometheus│ │
│ │ :5432 │ │ :6379 │ │ :9090 │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└────────────────────┼─────────────────────────────────────────────┘
│
┌───────────────┼───────────────┬──────────────────┐
│ │ │ │
┌────▼─────┐ ┌──────▼──────┐ ┌────▼──────┐ ┌───────▼──────┐
│ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │
│ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │
│ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │
│ │ │ │ │ llama-srv │ │ Watchdog │
│ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │
│ 27B-code │ │ :8090 │ │ :8080 │ │ exporter │
│ :8090 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │
│ :9400 │ └─────────────┘ └───────────┘ └──────────────┘
└──────────┘
```
## Current Model Assignments
| Model | GPU | Host | VRAM | Status |
|-------|-----|------|------|--------|
| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |
## Operations
### add-model
1. Download model file from Hugging Face or source
2. Check disk space + GPU VRAM compatibility
3. Start llama-server via systemd service on GPU host
4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
5. Hot-reload or restart router
6. Add model to LiteLLM config (`model_list` + `fallbacks`)
7. Generate agent keys for new model access via `/key/generate`
8. Add Prometheus scrape target for the new GPU exporter
9. Verify end-to-end: LiteLLM → Router → Model
### remove-model
1. Drain active requests (wait for active=0)
2. Remove from LiteLLM config
3. Remove from router config
4. Stop llama-server (systemd)
5. Remove Prometheus scrape target
6. Cleanup model files (optional)
### heal
1. Check all GPUs via router internal `:9000/health/unified`
2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
3. Reset stuck circuit breakers if idle (Redis)
4. Restart dead llama-server instances via SSH
5. Flush Redis active counters if stale
6. Verify GPU monitor server is running on pi (:9100)
7. Verify watchdog is running on pi
8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
9. Reload roster via `POST :9000/admin/roster/reload` if available
### sync-keys
1. List all agent keys in LiteLLM DB via `GET /key/list`
2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
3. Generate missing keys via `POST /key/generate` with unlimited budget
4. Update agent configs — `/etc/environment` LITELLM_API_KEY
5. Send Zulip DM to agents that can't be reached via SSH
6. Verify each key with test request through full chain
7. Document keys in knowledge graph
### list
Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys
## Agent Keys (LiteLLM DB — Current 2026-06-30)
| Agent | CT | IP | Key | Access |
|-------|-----|-----|-----|--------|
| Tanko | 112 | .122 | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | SSH jerome |
| Mumuni | 114 | .123 | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | SSH root |
| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
| Koby | 111 | ? | `sk-lDTsWy7H3T19UwUFEN6JSg` | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
| Koonimo | 113 | ? | `sk-Prx_vRXYb2VEzDmhPCbNeg` | no SSH |
| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
If no SSH access, send Zulip DM via abiba-bot.
## Configuration Files
| File | Host | Purpose |
|------|------|---------|
| `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
| `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |
## Prometheus & Grafana
| Component | URL | Details |
|-----------|-----|---------|
| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
| Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |
## Known Issues & Watch Points
- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.
## GPU Inference Benchmarks (Current)
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
|-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |
Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
History stored at `/root/data/toks-history.json` with 7-day rolling window.
**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
Reference in New Issue View Git Blame Copy Permalink