2026-07-12 23:40:13 +00:00
4 changed files with 25 additions and 23 deletions
@@ -34,7 +34,7 @@ triggers:
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
 - port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding

-## Fleet Topology (Current — June 2026)
+## Fleet Topology (Current — July 2026)

 ```
 ┌──────────────────────────────────────────────────────────────────┐
@@ -51,9 +51,9 @@ triggers:
 │  Containers:                                                     │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐        │
 │  │ LiteLLM  │  │ Router   │  │Dashboard │  │ Grafana   │        │
-│  │ :4000    │─▶│ :9000    │  │ :3000    │  │ :3000     │        │
-│  │ keys+sync│  │internal  │  │ harness  │  │ Prometheus│        │
-│  │ fallback │  │only!     │  │ UI       │  │ data src  │        │
+│  │ :4000    │  │ :9000    │  │ :3000    │  │ :3000     │        │
+│  │ keys+sync│  │deprecated│  │ harness  │  │ Prometheus│        │
+│  │ fallback │  │not in    │  │ UI       │  │ data src  │        │
 │  └──────────┘  └───┬──────┘  └──────────┘  └──────────┘        │
 │                    │                                             │
 │  ┌──────────┐  ┌──────────┐  ┌──────────┐                       │
@@ -68,7 +68,7 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 128K ctx │  │ 128K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│ 256K ctx │  │ 131K ctx    │  │ 256K ctx  │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
 │ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
 │ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
@@ -76,13 +76,13 @@ triggers:
 └──────────┘
 ```

-## Current Model Assignments (2026-07-08)
+## Current Model Assignments (2026-07-12)

 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.7/24GB (84%) | **256K** | turbo4 | 1 | default | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 131K | q4_0 | 2 | 2048/1024 | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |

 ## Operations

@@ -191,10 +191,10 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110).
- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput.
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`.
+- **VRAM (2026-07-12)**: RTX 3090 at 20.7/24GB (84%), RTX 5070 at 10.0/12.2GB (82%). RTX 3090 context increased to 256K (was incorrectly documented as 128K — verified via /proc/PID/cmdline).
+- **RTX 3090 runs `--parallel 1`** (verified 2026-07-12). RTX 5070 and Strix at parallel 2.
+- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching`. Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 1024 --parallel 2`. Api-key standardized to `not-needed` (was `sk-loc...5678` causing 401 loops). Service: `/home/llmuser/llama-wrapper.sh`.
 - **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
@@ -220,12 +220,12 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.

 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.

-## Agent Config Implications (2026-07-08)
+## Agent Config Implications (2026-07-12)

-With NVIDIA GPUs at 128K context:
- Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow
- Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule
- Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072`
- Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx)
+With RTX 3090 at 256K context (verified July 2026):
+- Agents using `syslog-auto` (55/30/15 qwen+ornith+gemma): `context_length: 262144` — ornith and qwen both support it
+- Agents using `qwen3.6-27B-code` directly: `context_length: 262144` (256K ctx verified)
+- Agents using `gemma-4-12b` directly (auxiliary tasks): `context_length: 131072`
+- Compression threshold at 0.65: fires at ~170K for 262K context window on Strix Halo
 - All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
 - Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
@@ -5,8 +5,10 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110),
-  parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added.
+  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (ornith-1.0-35b).
+  RTX 3090 context verified at 256K (was incorrectly documented as 128K).
+  RTX 5070 stays at 131K for vision/web. Infisical .env fallback required (Rule 3).
+  Parallel counts corrected (RTX 3090=1, RTX 5070=2, Strix=2).
 ---

 ## Maintains
@@ -16,7 +16,7 @@ description: >
  
  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
-  on CT 116. Last verified: 2026-07-09.
+  on CT 116. Last verified: 2026-07-12.
 ---

 ## Parameters
@@ -14,7 +14,7 @@ note: >
  lists. Health check is now § Health Check within this contract.
  
  Source of truth for GPU topology and keys: gpu-fleet.prose.md
-  Last verified: 2026-07-09
+  Last verified: 2026-07-12
 description: >
  LiteLLM inference stack health monitoring + self-healing. Verifies the full
  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model