---
kind: responsibility
name: gpu-fleet
description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, hot-reload, health checks, LiteLLM sync, agent key management,
  and stale model cleanup. Replaces hardcoded router.py config with declarative
  gpu_roster.yaml.
agent: abiba
triggers:
  - on model add/remove
  - on GPU health degradation
  - on agent key rotation
---

## Maintains

- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
- router: { status: "healthy", models: array }
- litellm: { status: "healthy", models: array }
- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
- health: { gpus: array, circuit_breakers: array }

## Operations

### add-model
1. Download model file from Hugging Face or source
2. Check disk space + GPU VRAM compatibility
3. Build/update llama.cpp if needed (cmake, Vulkan/CUDA flags)
4. Start llama-server via systemd service
5. Add to `gpu_roster.yaml`
6. Hot-reload router: `POST /admin/roster/reload`
7. Sync LiteLLM config: add model to model_list + fallbacks
8. Generate/update agent keys for new model access
9. Verify end-to-end: LiteLLM → Router → Model

### remove-model
1. Drain active requests (wait for active=0)
2. Update `gpu_roster.yaml` — remove entry
3. Hot-reload router
4. Remove from LiteLLM config
5. Stop llama-server (systemd)
6. Cleanup model files (optional — keep if model might return)

### heal
1. Check all GPUs via router `/health/unified`
2. Reset stuck circuit breakers if idle
3. Restart dead llama-server instances
4. Flush Redis counters if stale
5. Sync LiteLLM if configs diverge

### sync-keys
1. List all agent keys in LiteLLM DB via `/key/list`
2. Compare against expected agent list
3. Generate missing keys via `/key/generate`
4. Update agent configs to use `api_key_env: LITELLM_API_KEY`
5. Set `LITELLM_API_KEY` in system-wide env + startup wrapper
6. Restart gateways

### list
Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers

## Configuration Files

| File | Host | Purpose |
|------|------|---------|
| `gpu_roster.yaml` | CT 116 (`/opt/inference-harness/`) | Model-to-GPU mapping |
| `litellm_config.yaml` | CT 116 | LiteLLM proxy config |
| `roster_loader.py` | CT 116 (router container) | Hot-reloadable roster parser |
| `/etc/systemd/system/moe-server.service` | .15 | Ornith-1.0-35B server |
| Agent configs | .122, .123, .114 | `api_key_env: LITELLM_API_KEY` |

## GPU Fleet Topology

```
┌──────────────────────────────────────────────────┐
│  LiteLLM (:4001)                                 │
│  Config: /opt/inference-harness/litellm_config.yaml │
└──────────────┬───────────────────────────────────┘
               │
┌──────────────▼───────────────────────────────────┐
│  Router (:9000)                                   │
│  Roster: /app/gpu_roster.yaml (hot-reload)       │
│  Redis: slot booking + circuit breakers          │
└──────┬──────────┬──────────┬─────────────────────┘
       │          │          │
  ┌────▼───┐ ┌───▼────┐ ┌──▼──────┐
  │ gemma  │ │ qwen   │ │ ornith  │
  │ 4-12b  │ │ 27B    │ │ 1.0-35B │
  │ .110   │ │ .8     │ │ .15     │
  │ 5070   │ │ 3090   │ │ Strix   │
  └────────┘ └────────┘ └─────────┘
```

## Agent Key Architecture

All agents use `api_key_env: LITELLM_API_KEY` in Hermes configs:

| Agent | Host | Config | Env Source |
|-------|------|--------|------------|
| Mumuni | .123 | 7 refs + 6 subagent profiles | `/etc/environment` + wrapper |
| Tanko | .122 | 4 refs | `/etc/environment` + wrapper |
| Koonimo | .114 | 5 refs | `/etc/environment` + wrapper |

To rotate all keys: update `LITELLM_API_KEY` value, restart gateways.

## Sync Protocol

1. Edit `gpu_roster.yaml` → `POST /admin/roster/reload` (hot, no restart)
2. Edit `litellm_config.yaml` → `docker restart harness-litellm`
3. Agent key changes → generate via `POST /key/generate` with master key