---
kind: responsibility
name: gpu-fleet
description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
  Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
  router runs behind LiteLLM on :9000 (internal-only).
agent: abiba
triggers:
  - on model add/remove
  - on GPU health degradation
  - on agent key rotation
  - on router restart (roster must be loaded)
---

## Maintains

- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
- router: { status: "healthy", roster_loaded: bool, models: array }
- litellm: { status: "healthy", keys: array, models: array }
- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
- health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
- monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
- watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
- benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
- grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
- prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM

## Fleet Topology (Current — June 2026)

```
┌──────────────────────────────────────────────────────────────────┐
│  CT 116 (192.168.68.116) — Inference Harness Host                │
│                                                                  │
│  nginx:80 (entrypoint)                                           │
│  ├─ /v1/*       → harness-litellm:4000   (API requests)         │
│  ├─ /admin/*    → harness-litellm:4000   (admin endpoints)       │
│  ├─ /dashboard/ → harness-dashboard:3000 (harness UI)            │
│  ├─ /litellm/*  → harness-litellm:4000   (LiteLLM UI + API)     │
│  ├─ /health/*   → harness-litellm:4000   (health probes)         │
│  └─ /gpu/*      → 192.168.68.24:9100     (fleet monitor)         │
│                                                                  │
│  Containers:                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
│  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
│  │ :4000    │─▶│ :9000    │  │ :3000    │  │ :3000     │        │
│  │ keys+sync│  │internal  │  │ harness  │  │ Prometheus│        │
│  │ fallback │  │only!     │  │ UI       │  │ data src  │        │
│  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
│                    │                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
│  │PostgreSQL│  │  Redis   │  │Prometheus│                       │
│  │ :5432    │  │ :6379    │  │ :9090    │                       │
│  └──────────┘  └──────────┘  └──────────┘                       │
└────────────────────┼─────────────────────────────────────────────┘
                     │
     ┌───────────────┼───────────────┬──────────────────┐
     │               │               │                  │
┌────▼─────┐  ┌──────▼──────┐  ┌────▼──────┐  ┌───────▼──────┐
│ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
│ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
│ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
│          │  │             │  │ llama-srv │  │ Watchdog     │
│ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
│ 27B-code │  │ :8090       │  │ :8080     │  │ exporter     │
│ :8090    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
│ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
└──────────┘
```

## Current Model Assignments

| Model | GPU | Host | VRAM | Status |
|-------|-----|------|------|--------|
| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |

## Operations

### add-model
1. Download model file from Hugging Face or source
2. Check disk space + GPU VRAM compatibility
3. Start llama-server via systemd service on GPU host
4. Add to `/opt/inference-harness/gpu_roster.yaml` (or router env vars)
5. Hot-reload or restart router
6. Add model to LiteLLM config (`model_list` + `fallbacks`)
7. Generate agent keys for new model access via `/key/generate`
8. Add Prometheus scrape target for the new GPU exporter
9. Verify end-to-end: LiteLLM → Router → Model

### remove-model
1. Drain active requests (wait for active=0)
2. Remove from LiteLLM config
3. Remove from router config
4. Stop llama-server (systemd)
5. Remove Prometheus scrape target
6. Cleanup model files (optional)

### heal
1. Check all GPUs via router internal `:9000/health/unified`
2. Check LiteLLM health via nginx `:80/litellm/health/liveliness`
3. Reset stuck circuit breakers if idle (Redis)
4. Restart dead llama-server instances via SSH
5. Flush Redis active counters if stale
6. Verify GPU monitor server is running on pi (:9100)
7. Verify watchdog is running on pi
8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
9. Reload roster via `POST :9000/admin/roster/reload` if available

### sync-keys
1. List all agent keys in LiteLLM DB via `GET /key/list`
2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
3. Generate missing keys via `POST /key/generate` with unlimited budget
4. Update agent configs — `/etc/environment` LITELLM_API_KEY
5. Send Zulip DM to agents that can't be reached via SSH
6. Verify each key with test request through full chain
7. Document keys in knowledge graph

### list
Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys

## Agent Keys (LiteLLM DB — Current 2026-06-30)

| Agent | CT | IP | Key | Access |
|-------|-----|-----|-----|--------|
| Tanko | 112 | .122 | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | SSH jerome |
| Mumuni | 114 | .123 | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | SSH root |
| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
| Koby | 111 | ? | `sk-lDTsWy7H3T19UwUFEN6JSg` | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
| Koonimo | 113 | ? | `sk-Prx_vRXYb2VEzDmhPCbNeg` | no SSH |
| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |

**Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
If no SSH access, send Zulip DM via abiba-bot.

## Configuration Files

| File | Host | Purpose |
|------|------|---------|
| `/opt/inference-harness/docker-compose.yml` | CT 116 | All containers (router, litellm, nginx, postgres, redis, dashboard) |
| `/opt/inference-harness/litellm_config.yaml` | CT 116 | LiteLLM proxy config (models, fallbacks, timeouts) |
| `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
| `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
| `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
| `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
| `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |

## Prometheus & Grafana

| Component | URL | Details |
|-----------|-----|---------|
| Grafana | `http://192.168.68.116:3001/` | admin / syslog-grafana-2026 |
| GPU Dashboard | `http://192.168.68.116:3001/d/gpu-fleet` | Gauges + time series |
| Prometheus | `http://192.168.68.116:9090/` (internal) | 5 scrape targets |
| GPU Exporters | `:9400/metrics` on .8, .110, .15 | NVIDIA/AMD GPU metrics |
| Router Exporter | `:9401/metrics` on .24 | Router + LiteLLM metrics |

## Known Issues & Watch Points

- **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
- **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
- **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
- **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
- **Router GPU_MOE_URL bug (fixed 2026-07-01)**: docker-compose had `GPU_MOE_URL=.110:8080` (gemma host) instead of `.15:8080` (amdpve). Corrected.
- **Alert migration**: All alerts now go to `#agent-hub` topics (`alerts-gpu`, `alerts-pm2`, `alerts-infra`) instead of DMs. Cross-agent visibility enabled.
- **tok/s benchmarks**: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
- **NetBird 502**: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.

## GPU Inference Benchmarks (Current)

| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
|-----|-------|-----------|--------------|----------|---------|
| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
| Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |

Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
History stored at `/root/data/toks-history.json` with 7-day rolling window.

**Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.