SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
Details
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 1s
Details
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped
Details
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped
Details

Browse Source 
GPU Changes:
- All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability
- Observed instability near 100K at 256K — 128K is the stable ceiling
- VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%)
- Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX
  - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement)
  - Uncensored (0/465 refusals), multimodal (mmproj F16)
  - Speed: 65 tok/s gen, 140 tok/s prompt
  - Alias strix-moe maintained

Agent Updates:
- Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65
- Tanko: max_context_window 262144→131072
- Koonimo: max_context_window + context_length 262144→131072
- CT114 SSH access confirmed (was 'Zulip only')

LiteLLM (CT116):
- Updated backend model references qwen3.6-35B-udq4→strix-moe
- Removed stale ornith-1.0-35b from model_cost
- Fallback chains updated

Contracts Updated:
- gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments
- gpu-self-heal.prose.md: Rule 9/10 context targets
- hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds
- inference-optimization.prose.md: added to repo, 128K recommendation

Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling)
For >128K workloads: route to external providers (deepseek)
This commit is contained in:
root
2026-07-17 10:59:45 +00:00
parent 65c99dab50
commit b9149bce47
4 changed files with 153 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hermes-config-template.prose.md
+20 -21
View File
@@ -6,11 +6,10 @@ description: >
Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
RA-H OS MCP) while keeping agent-specific API keys and model choices.
UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
which LiteLLM does not serve). All 3 GPUs verified 256K (RTX 5070 bumped 131K→256K on Jul 15).
which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
2026-07-16 Mumuni root-cause investigation (WAL #1300).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context
verified at 256K. Infisical .env fallback required (Rule 3/13).
UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
---
## Maintains
@@ -36,7 +35,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
| Abiba | `abiba-pi` | 192.168.68.24 | local | — |
| Koby | `koby` | CT 111 (tdunna) | Zulip | — |
| Koonimo | `koonimo` | CT 113 (baggy) | Zulip | — |
| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
| Kagenz0 | `kagenz0-*` | ? | Zulip | — |
> CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
@@ -101,8 +100,8 @@ model:
base_url: http://192.168.68.116/v1
api_key_env: LITELLM_API_KEY # Injected via infisical run -- wrapper
max_tokens: 4096 # ⚠️ CRITICAL: Prevents unbounded generation
context_length: 262144 # For syslog-auto (all GPUs support 256K).
# Set 131072 if using gemma-4-12b directly (12GB VRAM constraint).
context_length: 131072 # For syslog-auto (all GPUs at 128K for stability).
# Set 65536 if using gemma-4-12b directly (tight VRAM).
fallback_providers:
provider: deepseek
@@ -133,7 +132,7 @@ compression:
enabled: true
model: strix-moe # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
provider: harness
max_context_window: 262144 # MUST match actual GPU capacity. All 3 GPUs are 256K (Jul 15).
max_context_window: 131072 # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
threshold: 0.65 # Fires at ~170K for 262K window, ~85K for 128K
target_ratio: 0.30
protect_last_n: 40
@@ -250,7 +249,7 @@ The following MUST be identical across ALL profiles:
### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 256K ctx, compression-optimized)
- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized)
- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
(it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
- All auxiliary services MUST use identical routing:
@@ -258,31 +257,31 @@ The following MUST be identical across ALL profiles:
- `api_key_env: LITELLM_API_KEY`
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
(64GB UMA, 256K context) — the designated compression GPU. This frees the
(64GB UMA, 128K context) — the designated compression GPU. This frees the
RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
- The `compression:` block's `model` MUST match `auxiliary: compression: model`
- The `compression: max_context_window: 262144` MUST match Strix Halo's actual capacity
- The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
- **RTX 3090 (24GB, 256K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
- **RTX 5070 (12GB, 256K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (bumped 131K→256K Jul 15; IQ4_NL+MTP, 88% VRAM)
- **Strix Halo (64GB, 256K ctx, strix-moe)**: Context compression, summarization, long docs
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
- **Strix Halo (64GB, 128K ctx, strix-moe)**: Context compression, summarization, long docs
- Agent profiles MUST route auxiliary tasks to the correct GPU:
- `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
- `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
- `auxiliary.compression.model: strix-moe` (Strix Halo)
- Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
- `max_context_window: 262144` MUST match the model's actual capacity
- Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin
- `max_context_window: 131072` MUST match the model's actual capacity (128K)
- See `devops-hermes-compression` skill for full reference
### Rule 9: Compression Threshold for 256K Models
- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
### Rule 9: Compression Threshold for 128K Models
- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
- Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
- `max_context_window: 262144` MUST match the model's actual capacity (Strix Halo = 256K)
- Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin
- `max_context_window: 131072` MUST match the model's actual capacity (all GPUs = 128K)
- See `devops-hermes-compression` skill for full reference
### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
@@ -312,7 +311,7 @@ When an agent shows "context issues" (premature compression, 401s, 504s, DeepSee
verify ALL FOUR of these against the live config. They are the only root causes found in production:
1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
MUST be `262144` (all GPUs are 256K). A value of `131072` causes premature compression at
MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used.
~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
2. **base_url uses /v1 NOT /litellm/v1?** — `custom_providers[0].base_url`, `delegation.base_url`,
and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/v1` (Rule 5). nginx `/litellm/`