From be02b0e84374f91435f4f83919d607974a6d29eb Mon Sep 17 00:00:00 2001 From: root Date: Sun, 12 Jul 2026 22:54:30 +0000 Subject: [PATCH] fix: stale dates, context values, and architecture references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - gpu-fleet: Updated all context references (128K→256K for RTX 3090). Fixed parallel count (2→1 for RTX 3090). Updated architecture diagram (router labeled deprecated/not in path). Corrected VRAM values. Updated dates: June→July 2026. - hermes-config-template: Fixed description (removed incorrect '128K on NVIDIA' claim). Updated to reflect verified 256K on RTX 3090. - litellm-self-heal: Updated last-verified date. - litellm-api-keys: Updated last-verified date. --- gpu-fleet.prose.md | 38 ++++++++++++++++----------------- hermes-config-template.prose.md | 6 ++++-- litellm-api-keys.prose.md | 2 +- litellm-self-heal.prose.md | 2 +- 4 files changed, 25 insertions(+), 23 deletions(-) diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 0d0a633..37925bf 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -34,7 +34,7 @@ triggers: - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM - port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding -## Fleet Topology (Current — June 2026) +## Fleet Topology (Current — July 2026) ``` ┌──────────────────────────────────────────────────────────────────┐ @@ -51,9 +51,9 @@ triggers: │ Containers: │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ LiteLLM │ │ Router │ │Dashboard │ │ Grafana │ │ -│ │ :4000 │─▶│ :9000 │ │ :3000 │ │ :3000 │ │ -│ │ keys+sync│ │internal │ │ harness │ │ Prometheus│ │ -│ │ fallback │ │only! │ │ UI │ │ data src │ │ +│ │ :4000 │ │ :9000 │ │ :3000 │ │ :3000 │ │ +│ │ keys+sync│ │deprecated│ │ harness │ │ Prometheus│ │ +│ │ fallback │ │not in │ │ UI │ │ data src │ │ │ └──────────┘ └───┬──────┘ └──────────┘ └──────────┘ │ │ │ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ @@ -68,7 +68,7 @@ triggers: │ CT 8 │ │ CT 110 │ │ CT 15 │ │ pi (.24) │ │ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │ │ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │ -│ 128K ctx │ │ 128K ctx │ │ 256K ctx │ │ Watchdog │ +│ 256K ctx │ │ 131K ctx │ │ 256K ctx │ │ Watchdog │ │ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │ │ 27B-code │ │ :8080 │ │ :8080 │ │ exporter │ │ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │ @@ -76,13 +76,13 @@ triggers: └──────────┘ ``` -## Current Model Assignments (2026-07-08) +## Current Model Assignments (2026-07-12) | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status | |-------|-----|------|------|-----|----------|----------|-------------|--------| -| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy | -| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy | -| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy | +| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy | +| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy | +| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy | ## Operations @@ -191,10 +191,10 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first. Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster(). - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround. -- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110). -- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput. -- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`. -- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`. +- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline). +- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2. +- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`. +- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`. - **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV. - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). @@ -220,12 +220,12 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window. **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration. -## Agent Config Implications (2026-07-08) +## Agent Config Implications (2026-07-12) -With NVIDIA GPUs at 128K context: -- Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow -- Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule -- Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072` -- Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx) +With RTX 3090 at 256K context (verified July 2026): +- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it +- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified) +- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072` +- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo - All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap - Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented) diff --git a/hermes-config-template.prose.md b/hermes-config-template.prose.md index c237eb1..2292a2a 100644 --- a/hermes-config-template.prose.md +++ b/hermes-config-template.prose.md @@ -5,8 +5,10 @@ description: > Standard Hermes configuration template for Syslog Solution LLC agents. Enforces shared infrastructure setup (Firecrawl, SearXNG, local models, RA-H OS MCP) while keeping agent-specific API keys and model choices. - Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110), - parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added. + UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (ornith-1.0-35b). + RTX 3090 context verified at 256K (was incorrectly documented as 128K). + RTX 5070 stays at 131K for vision/web. Infisical .env fallback required (Rule 3). + Parallel counts corrected (RTX 3090=1, RTX 5070=2, Strix=2). --- ## Maintains diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index 60a25ff..d3fc3e7 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -16,7 +16,7 @@ description: > Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml - on CT 116. Last verified: 2026-07-09. + on CT 116. Last verified: 2026-07-12. --- ## Parameters diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index bedacaa..b92e057 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -14,7 +14,7 @@ note: > lists. Health check is now § Health Check within this contract. Source of truth for GPU topology and keys: gpu-fleet.prose.md - Last verified: 2026-07-09 + Last verified: 2026-07-12 description: > LiteLLM inference stack health monitoring + self-healing. Verifies the full nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model