prose-contracts/gpu-fleet.prose.md

kind, name, description, agent, triggers

kind	name	description	agent	triggers
responsibility	gpu-fleet	Manages the GPU inference fleet across all hosts. Handles model deployment, registration, health checks, LiteLLM sync, agent key management, GPU saturation watchdog, Prometheus/Grafana monitoring, and self-healing. UPDATED 2026-07-15: Stable role-based aliases introduced: strix-moe, gpu-dense, gpu-light. These never change — only the underlying model does. Strix Halo: ornith-1.0-35b → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB). RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster). RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB). Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).	abiba	on model add/remove on GPU health degradation on agent key rotation on router restart (roster must be loaded)

Maintains

• gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
• router: { status: "healthy", roster_loaded: bool, models: array }
• litellm: { status: "healthy", keys: array, models: array }
• agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
• health: { gpus: array, circuit_breakers: array } — Fleet-wide health state
• monitor: { status: "running", version: "2.0.0" } — GPU monitor server on pi (:9100)
• watchdog: { status: "running" } — GPU saturation watchdog (restarts stuck llama-server)
• benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
• grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
• prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
• port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding

Fleet Topology (Current — July 2026)

┌──────────────────────────────────────────────────────────────────┐
│  CT 116 (192.168.68.116) — Inference Harness Host                │
│                                                                  │
│  nginx:80 (entrypoint)                                           │
│  ├─ /v1/*       → harness-litellm:4000   (API requests)         │
│  ├─ /admin/*    → harness-litellm:4000   (admin endpoints)       │
│  ├─ /dashboard/ → harness-dashboard:3000 (harness UI)            │
│  ├─ /litellm/*  → harness-litellm:4000   (LiteLLM UI + API)     │
│  ├─ /health/*   → harness-litellm:4000   (health probes)         │
│  └─ /gpu/*      → 192.168.68.24:9100     (fleet monitor)         │
│                                                                  │
│  Containers:                                                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
│  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
│  │ :4000    │  │ :9000    │  │ :3000    │  │ :3000     │        │
│  │ keys+sync│  │deprecated│  │ harness  │  │ Prometheus│        │
│  │ fallback │  │not in    │  │ UI       │  │ data src  │        │
│  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
│                    │                                             │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
│  │PostgreSQL│  │  Redis   │  │Prometheus│                       │
│  │ :5432    │  │ :6379    │  │ :9090    │                       │
│  └──────────┘  └──────────┘  └──────────┘                       │
└────────────────────┼─────────────────────────────────────────────┘
                     │
     ┌───────────────┼───────────────┬──────────────────┐
     │               │               │                  │
┌────▼─────┐  ┌──────▼──────┐  ┌────▼──────┐  ┌───────▼──────┐
│ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
│ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
│ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
│ 256K ctx │  │ 131K ctx    │  │ 256K ctx  │  │ Watchdog     │
│ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
│ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
└──────────┘

Stable Role-Based Aliases (Introduced 2026-07-15)

Agent configs, cron jobs, and workflows MUST use these aliases, never model-specific names. When a model is swapped on a GPU, ONLY the infrastructure layer changes — agent configs are untouched.

Alias	GPU	Current Model	Will Route To
strix-moe	Strix Halo (.15)	qwen3.6-35B-udq4	Whatever runs on Strix Halo
gpu-dense	RTX 3090 (.8)	qwen3.6-27B-code	Whatever runs on RTX 3090
gpu-light	RTX 5070 (.110)	gemma-4-12b	Whatever runs on RTX 5070

Backward compatibility: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work but are deprecated for agent configs. Only the stable aliases survive model swaps.

Current Model Assignments (2026-07-15)

Model	GPU	Host	VRAM	Ctx	KV Cache	Parallel	Batch/Ubatch	Status
qwen3.6-27B-code (MTP)	RTX 3090	.8 (llm-gpu)	22.2/24.6GB (90%)	256K 🚀	turbo4	2	default	✅ 63 tok/s
gemma-4-12b	RTX 5070	.110 (ocu-llm)	10.0/12.2GB (82%)	131K	q4_0	2	2048/1024	✅ healthy
qwen3.6-35B-udq4	Strix Halo Vulkan	.15 (amdpve)	~9GB/64GB	256K	q8_0	2	2048/512	✅ healthy

Routing Configuration (LiteLLM — July 2026)

syslog-auto Weighted Pool (Direct GPU — bypasses router)

Model	GPU	Weight	RPM Cap	Timeout
qwen3.6-27B-code	RTX 3090 (.8:8080)	0.55	500	90s
qwen3.6-35B-udq4	Strix Halo (.15:8080)	0.30	60	300s
gemma-4-12b	RTX 5070 (.110:8080)	0.15	200	300s

Note: All syslog-auto entries route directly to GPUs with api_key: not-needed. The router (port 9000) is NOT in the inference path.

Direct Model Endpoints

Model	RPM Cap	Notes
qwen3.6-35B-udq4	40	Tight cap — prevents Strix overload
qwen3.6-27B-code	500	High cap — primary workhorse
gemma-4-12b	500	High cap — IQ4_NL+MTP, 122 tok/s

Stable Aliases (for agent configs — never change)

Alias	RPM Cap	Routes To	Purpose
strix-moe	40	Strix Halo	Compression tasks (MoE models)
gpu-dense	500	RTX 3090	Heavy reasoning
gpu-light	500	RTX 5070	Vision, web extract, light tasks

Fallback Chains

• gemma → qwen
• qwen → gemma
• qwen3.6-35B-udq4 → qwen → gemma
• syslog-auto → qwen → gemma → qwen3.6-35B-udq4

Why Strix Halo RPM Is Capped

• Direct (qwen3.6-35B-udq4): 40 RPM (tight) — Strix Halo is shared with compression tasks
• Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
• Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C

Operations

add-model

1. Download model file from Hugging Face or source
2. Check disk space + GPU VRAM compatibility
3. Start llama-server via systemd service on GPU host
4. Add to /opt/inference-harness/gpu_roster.yaml (or router env vars)
5. Hot-reload or restart router
6. Add model to LiteLLM config (model_list + fallbacks)
7. Generate agent keys for new model access via /key/generate
8. Add Prometheus scrape target for the new GPU exporter
9. Verify end-to-end: LiteLLM → Router → Model

remove-model

1. Drain active requests (wait for active=0)
2. Remove from LiteLLM config
3. Remove from router config
4. Stop llama-server (systemd)
5. Remove Prometheus scrape target
6. Cleanup model files (optional)

heal

1. Check all GPUs via router internal :9000/health/unified
2. Check LiteLLM health via nginx :80/litellm/health/liveliness
3. Reset stuck circuit breakers if idle (Redis)
4. Restart dead llama-server instances via SSH
5. Flush Redis active counters if stale
6. Verify GPU monitor server is running on pi (:9100)
7. Verify watchdog is running on pi
8. Restart router if roster not loaded (check logs for STARTUP ROSTER)
9. Reload roster via POST :9000/admin/roster/reload if available

sync-keys

1. List all agent keys in LiteLLM DB via GET /key/list
2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
3. Generate missing keys via POST /key/generate with unlimited budget
4. Update Infisical vault: infisical secrets set LITELLM_API_KEY=<key> --project=agents --env=production
5. Send Zulip DM to agents that can't be reached via SSH (provide vault login instructions)
6. Verify each key with test request through full chain
7. Document keys in knowledge graph

list

Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys

health-check

1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
2. Check llama-server processes: ps aux | grep llama-server on all 3 hosts
3. Check LiteLLM: curl http://192.168.68.116/health (expect "I'm alive!")
4. Check LiteLLM models: curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models
5. Check LiteLLM timeouts: grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml
   • gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
   • global request_timeout: 300s, nginx proxy_read_timeout: 600s
6. Check AMD metrics: curl http://192.168.68.15:9400/metrics (Radeon 8060S, util%, VRAM, temp, power)
7. Check port conflicts: verify only one llama-server on :8080 per host
8. Verify agent keys: 9 keys in LiteLLM DB (GET /key/list)

Agent Keys (LiteLLM DB — Current 2026-07-11)

Keys stored in Infisical vault (project=agents, env=production, secret=LITELLM_API_KEY). Agent gateways inject keys at runtime via infisical run -- wrapper. Plaintext keys removed from this contract post-vault-migration.

Agent	CT	IP	LiteLLM Alias	Key Source	Access
Tanko	112	.122	tanko	Infisical vault	SSH jerome
Mumuni	114	.123	mumuni	Infisical vault	SSH root
Abiba	100	.24	abiba-pi	Infisical vault	local (pi agent)
Koby	111	?	koby	Infisical vault	Zulip DM
Koonimo	113	?	koonimo	Infisical vault (migrated 2026-07-11)	no SSH
Kagenz0	105	?	kagenz0	Infisical vault	no SSH

Note

: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113=baggy runs koonimo.

Key update procedure: Update Infisical vault → infisical secrets set LITELLM_API_KEY=sk-... --project=agents --env=production → restart agent gateway. Agent picks up new key via infisical run -- wrapper at startup. If no SSH access, send Zulip DM via abiba-bot with vault update instructions.

Configuration Files

File	Host	Purpose
/opt/inference-harness/docker-compose.yml	CT 116	All containers (router, litellm, nginx, postgres, redis, dashboard)
/opt/inference-harness/litellm_config.yaml	CT 116	LiteLLM proxy config (models, fallbacks, timeouts)
/opt/inference-harness/router/router.py	CT 116	Router source (builds via compose)
/etc/nginx/nginx.conf	CT 116 (nginx container)	Routes /v1→LiteLLM, /dashboard/, /litellm/, /health
/opt/monitoring/prometheus.yml	CT 116	Prometheus scrape config (5 targets)
/root/scripts/gpu-monitor-server.py	pi (.24)	GPU fleet monitor v2.1.0 (with benchmarks)
/root/scripts/gpu_benchmark.py	pi (.24)	GPU inference benchmark module (tok/s tracking)
/root/scripts/gpu-saturation-watchdog.py	pi (.24)	Auto-restart stuck llama-server
/root/dashboard/gpu-fleet.html	pi (.24)	Live HTML dashboard
/etc/systemd/system/llama-server.service	.8, .110	llama-server daemons (Nvidia GPUs)
/etc/systemd/system/strix-server.service	.15 (amdpve)	llama-server daemon (Vulkan, Strix Halo) running unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Note: llama-server.service and llama-server@.service are masked on .15 to prevent port 8080 collisions.

Prometheus & Grafana

Component	URL	Details
Grafana	http://192.168.68.116:3001/	admin / vault (GRAFANA_ADMIN_PASSWORD)
GPU Dashboard	http://192.168.68.116:3001/d/gpu-fleet	Gauges + time series
Prometheus	http://192.168.68.116:9090/ (internal)	5 scrape targets
GPU Exporters	:9400/metrics on .8, .110, .15	NVIDIA/AMD GPU metrics
Router Exporter	:9401/metrics on .24	Router + LiteLLM metrics

Known Issues & Watch Points

• Router startup race: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first. Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
• LiteLLM /metrics: Requires auth. Prometheus uses /health/liveliness as workaround.
• VRAM (2026-07-15): RTX 3090 at 22.2/24.6GB (90%) with 256K context (corrected from 131K). RTX 5070 at 10.8/12.2GB (88%) with 256K context + MTP. Strix Halo at ~9GB/64GB.
• RTX 3090 runs --parallel 2 with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
• RTX 3090 config: -c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: /home/llmuser/llama-wrapper.sh.
• RTX 5070 config (2026-07-15): Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: /home/llmuser/llama-wrapper.sh. Config: --model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144.
• LiteLLM timeout tuning (2026-07-08): gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at /opt/inference-harness/litellm_config.yaml.
• Strix Halo GPU: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at /root/llama.cpp/build-vk/, commit 4fc4ec5 (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: strix-server.service on port 8080 (was ornith-server.service), model changed to unsloth/Qwen3.6-35B-A3B-MTP-GGUF (UD-Q4_K_M), alias qwen3.6-35B-udq4, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference.
• Port conflict detection (2026-07-05): All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. .8 and .110 use inline pre-start check in llama-wrapper.sh; .15 uses /usr/local/bin/port-cleanup.sh ExecStartPre. Replaces the blanket pkill -9 -x llama-server on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
• Strix Halo thermal safeguard (2026-07-02): strix-server.service has -n 8192 (hard generation cap per request). Without it, --predict defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove -n without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set max_tokens.
• Port 8080 firewall: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
• Router sidecar fallback: router.py check_gpu_health() now probes GPU /health directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
• Router GPU_MOE_URL bug (fixed 2026-07-01): docker-compose had GPU_MOE_URL=.110:8080 (gemma host) instead of .15:8080 (amdpve). Corrected.
• Alert migration: All alerts now go to #agent-hub topics (alerts-gpu, alerts-pm2, alerts-infra) instead of DMs. Cross-agent visibility enabled.
• tok/s benchmarks: Measured every 5 min via LiteLLM proxy. Baselines tracked with 30%/50% degradation thresholds.
• NetBird 502: Tanko routes through NetBird for litellm.sysloggh.net. Use direct IP if NetBird down.

GPU Inference Benchmarks (Current)

GPU	Model	Gen tok/s	Prompt tok/s	Baseline	Context
RTX 3090 (.8)	qwen3.6-27B-code (MTP)	63	—	—	131K
RTX 5070 (.110)	gemma-4-12b (IQ4_NL+MTP)	191	—	—	256K
Strix Halo (.15)	qwen3.6-35B-udq4	71	—	—	256K

Benchmarks from 2026-07-15 verification run. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s. All 3 GPUs now at 256K context (2026-07-15).

Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes. Degradation alerts fire at 30% (warning) and 50% (critical) below baseline. History stored at /root/data/toks-history.json with 7-day rolling window.

Note (2026-07-01): Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.

Agent Config Implications (2026-07-15)

Stable Aliases — CRITICAL

All agent configs MUST use stable role-based aliases, never model-specific names:

• compression.model: strix-moe (NOT qwen3.6-35B-udq4)
• auxiliary.vision.model: gpu-light (NOT gemma-4-12b)
• delegation.model: gpu-dense (NOT qwen3.6-27B-code)
• auxiliary.web_extract.model: gpu-light

When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched.

Context Windows

• RTX 3090: 256K (was 131K, bumped 2026-07-15) | RTX 5070: 256K (up from 131K) | Strix Halo: 256K
• Mumuni compression context: 128K (down from 256K) — ensures compression model doesn't timeout
• Compression threshold 0.65: fires at ~85K for 128K context window
• Mumuni compression model alias: strix-moe with 300s timeout

Mumuni Agent Profile

Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile is the reference for all agent configs:

Setting	Value	Notes
model.default	syslog-auto	Weighted pool (55% qwen, 30% strix, 15% gemma)
model.provider	custom:litellm	LiteLLM on CT116
compression.model	strix-moe	Stable alias — survives model swaps
aux.compression.model	strix-moe	Compression auxiliary model
aux.vision.model	gpu-light	Vision tasks (RTX 5070)
aux.web_extract.model	gpu-light	Web extraction
delegation.model	gpu-dense	Sub-agent reasoning (RTX 3090)
context.max_context_window	131072 (128K)	Kept at 128K to avoid compression timeout
compression.threshold	0.65	Triggers at ~85K
compression.target_ratio	0.3	Compresses to ~38K
compression.protect_last_n	40	Preserves last 40 messages
memory.memory_char_limit	800	Brief memory entries
personalities	creative	Creative assistant personality
Platforms	cli, discord, homeassistant, signal, telegram, zulip	All Hermes platforms
Main model timeout	300s	LiteLLM global timeout
Compression model timeout	300s	ornith timeout increased from 120s

Agent Update Status (2026-07-15)

Agent	Host	Status
Mumuni	CT114 (.123)	✅ Updated to stable aliases
Tanko	CT112 (.122)	✅ Updated to stable aliases
Koby	CT111 (.129)	❌ SSH unreachable — needs Zulip DM
Koonimo	CT113	❌ SSH unreachable — needs Zulip DM
Kagenz0	CT105	❌ SSH unreachable — needs Zulip DM

All Hermes clients MUST set max_tokens: 4096 — first line of defense before server-side -n 8192 cap.