prose-contracts/gpu-fleet.prose.md

---
kind: responsibility
name: gpu-fleet
description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
  UPDATED 2026-07-12: Architecture is DIRECT GPU — LiteLLM routes directly
  to llama-server on each GPU host (no router in inference path). Router
  (port 9000) is running but NOT in request path. All GPUs standardized on
  api-key 'not-needed'. RTX 5070 had api-key mismatch (sk-loc...5678) that
  caused cascading 401→timeout→401 fallback loops — fixed.
  Context: RTX 3090 verified at 256K (was documented as 128K — WRONG).
  Workload: Compression moved to Strix Halo, RTX 5070 → vision/web only.
agent: abiba
triggers:
  - on model add/remove
  - on GPU health degradation
  - on agent key rotation
  - on router restart (roster must be loaded)
---

## Maintains

- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
- router: { status: "healthy", roster_loaded: bool, models: array }
- litellm: { status: "healthy", keys: array, models: array }
- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
- health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
- monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
- watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
- benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
- grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
- port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding

## Fleet Topology (Current — July 2026)

```
┌──────────────────────────────────────────────────────────────────┐
│  CT 116 (192.168.68.116) — Inference Harness Host                │
│                                                                  │
│  nginx:80 (entrypoint)                                           │
│  ├─ /v1/*       → harness-litellm:4000   (API requests)         │
│  ├─ /admin/*    → harness-litellm:4000   (admin endpoints)       │
│  ├─ /dashboard/ → harness-dashboard:3000 (harness UI)            │
│  ├─ /litellm/*  → harness-litellm:4000   (LiteLLM UI + API)     │
│  ├─ /health/*   → harness-litellm:4000   (health probes)         │
│  └─ /gpu/*      → 192.168.68.24:9100     (fleet monitor)         │
│                                                                  │
│  Containers:                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
│  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
│  │ :4000    │  │ :9000    │  │ :3000    │  │ :3000     │        │
│  │ keys+sync│  │deprecated│  │ harness  │  │ Prometheus│        │
│  │ fallback │  │not in    │  │ UI       │  │ data src  │        │
│  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
│                    │                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
│  │PostgreSQL│  │  Redis   │  │Prometheus│                       │
│  │ :5432    │  │ :6379    │  │ :9090    │                       │
│  └──────────┘  └──────────┘  └──────────┘                       │
└────────────────────┼─────────────────────────────────────────────┘
                     │
     ┌───────────────┼───────────────┬──────────────────┐
     │               │               │                  │
┌────▼─────┐  ┌──────▼──────┐  ┌────▼──────┐  ┌───────▼──────┐
│ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
│ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
│ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
│ 256K ctx │  │ 131K ctx    │  │ 256K ctx  │  │ Watchdog     │
│ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
│ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
└──────────┘
```

## Current Model Assignments (2026-07-12)

| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|-------|-----|------|------|-----|----------|----------|-------------|--------|
| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |

## Routing Configuration (LiteLLM — July 2026)

### syslog-auto Weighted Pool

| Model | GPU | Weight | RPM Cap | Purpose |
|-------|-----|--------|---------|---------|
| qwen3.6-27B-code | RTX 3090 | 0.55 | 500 | Heavy reasoning, code, long context |
| ornith-1.0-35b | Strix Halo | 0.30 | **60** | Agentic workflows, tool calling |
| gemma-4-12b | RTX 5070 | 0.15 | 200 | Overflow + vision |

### Direct Model Endpoints

| Model | RPM Cap | Notes |
|-------|---------|-------|
| ornith-1.0-35b | 40 | Tight cap — prevents Strix overload |
| qwen3.6-27B-code | 500 | High cap — primary workhorse |
| gemma-4-12b | 500 | High cap — fast 12B |

### Fallback Chains
- gemma → qwen
- qwen → gemma
- ornith → qwen → gemma
- syslog-auto → qwen → gemma → ornith

### Why ornith RPM Is Capped
- Direct: 40 RPM (tight) — Strix Halo is shared with compression tasks
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C

## Operations

### add-model
1. Download model file from Hugging Face or source
2. Check disk space + GPU VRAM compatibility
3. Start llama-server via systemd service on GPU host
4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
5. Hot-reload or restart router
6. Add model to LiteLLM config (`model_list` + `fallbacks`)
7. Generate agent keys for new model access via `/key/generate`
8. Add Prometheus scrape target for the new GPU exporter
9. Verify end-to-end: LiteLLM → Router → Model

### remove-model
1. Drain active requests (wait for active=0)
2. Remove from LiteLLM config
3. Remove from router config
4. Stop llama-server (systemd)
5. Remove Prometheus scrape target
6. Cleanup model files (optional)

### heal
1. Check all GPUs via router internal `:9000/health/unified`
2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
3. Reset stuck circuit breakers if idle (Redis)
4. Restart dead llama-server instances via SSH
5. Flush Redis active counters if stale
6. Verify GPU monitor server is running on pi (:9100)
7. Verify watchdog is running on pi
8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
9. Reload roster via `POST :9000/admin/roster/reload` if available

### sync-keys
1. List all agent keys in LiteLLM DB via `GET /key/list`
2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
3. Generate missing keys via `POST /key/generate` with unlimited budget
4. Update Infisical vault: `infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production`
5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
6. Verify each key with test request through full chain
7. Document keys in knowledge graph

### list
Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys

### health-check
1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
7. Check port conflicts: verify only one llama-server on :8080 per host
8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)

## Agent Keys (LiteLLM DB — Current 2026-07-11)

Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY).
Agent gateways inject keys at runtime via `infisical run --` wrapper.
Plaintext keys removed from this contract post-vault-migration.

| Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|-------|-----|-----|---------------|------------|--------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
| Mumuni | 114 | .123 | `mumuni` | Infisical vault | SSH root |
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
| Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | no SSH |

> **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.

**Key update procedure**: Update Infisical vault → `infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production` → restart agent gateway. Agent picks up new key via `infisical run --` wrapper at startup.
If no SSH access, send Zulip DM via abiba-bot with vault update instructions.

## Configuration Files

| File | Host | Purpose |
|------|------|---------|
| `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
| `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |

## Prometheus & Grafana

| Component | URL | Details |
|-----------|-----|---------|
| Grafana | `http://192.168.68.116:3001/` | admin / vault (`GRAFANA_ADMIN_PASSWORD`) |
| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
| Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |

## Known Issues & Watch Points

- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline).
- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2.
- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`.
- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.

## GPU Inference Benchmarks (Current)

| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
|-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |

Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
History stored at `/root/data/toks-history.json` with 7-day rolling window.

**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.

## Agent Config Implications (2026-07-12)

With RTX 3090 at 256K context (verified July 2026):
- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it
- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified)
- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072`
- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo
- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)