prose-contracts/inference-optimization.prose.md at 221f9f79f3afec7e5be5cf9f4f34c4b87f8b6126

Files

T

abiba 221f9f79f3 fix(audit): reject retired aliases; sweep gpu-light/gemma-4-12b to gpu-vision

audit-hermes-config.py Rule 8 required auxiliary.vision.model and
auxiliary.web_extract.model to equal the retired 'gpu-light', so a config
adopting the live canonical 'gpu-vision' FAILED our own audit - the audit was
enforcing a dead alias (400 Invalid model name). Rule 8 now requires
gpu-vision; retired names gpu-light/crew-auto join the raw-name rejection set;
the guidance message names the live aliases.

Sweep of the remaining references: gpu-self-heal stops canonicalizing
gpu-light; hermes-config-template, hermes-agent-baseline, hermes-key-enforcement,
inference-optimization, litellm-client-timeouts and gpu-fleet now use the live
gpu-vision alias. Where a file restated model/rpm/weight/fallback state it now
points at CT 116 /opt/inference-harness/litellm_config.yaml instead of
duplicating it. koby's .129 config is report-only and recorded, not edited.

Adds tests/test_audit_hermes_config_alias.py: executes the audit CLI and asserts
gpu-vision passes while gpu-light and gemma-4-12b fail.

2026-09-12 16:11:46 +00:00

3.6 KiB

Raw Blame History

name, kind, description, id

name	kind	description	id
inference-optimization	responsibility	Optimizes the full Syslog inference stack — LiteLLM routing weights, GPU model assignments, agent context management, and prompt caching — to reduce response times to sub-15s average. All GPUs now at 128K context (stable ceiling).	067NC6KP02RG60S50M40E30928

Goal

Syslog inference response times reduced to sub-15s average by optimizing the full stack: LiteLLM routing weights, GPU model assignments, Hermes agent context management, and prompt caching — without sacrificing agent capability.

Requires

inference-metrics: current SpendLogs from CT116 LiteLLM Postgres — avg request_duration_ms, prompt_tokens, completion_tokens, model_group breakdown, cache_hit rate over the last 3 hours
agent-configs: current config.yaml from each active Hermes agent (Mumuni .123, any others on .129/.122) including compression, model, context_window, prompt_caching, memory settings
gpu-health: health check response from all 3 GPU backends (strix-moe .15:8080, qwen .8:8080, gemma .110:8080)

Maintains

The optimized inference stack configuration — every change is applied and verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of inference calls.

liteLLM-routing

The syslog-auto routing weights, model-specific timeouts, RPM limits, and model_list entries on CT116 /opt/inference-harness/litellm_config.yaml.

agent-compression

Each Hermes agent's ~/.hermes/config.yaml compression, context_window, prompt_caching, and model sections.

prompt-caching

LiteLLM cache configuration and llama.cpp --cache-prompt flag on GPU hosts.

verification

End-to-end latency measurements after changes applied — at least 3 test inference calls per model path measuring ttft (time-to-first-token) and total duration.

Continuity

input-driven

Strategies

Context is the root cause. Every ~46K prompt token costs ~87s of prefill time at 532 tok/s. Fix context first, routing second.

Route by task: qwen for code/standard queries; gemma for compression/auxiliary; strix-moe for compression tasks.
Compress aggressively: threshold at 40% (not 65%) — a 128K window should compact at 51K, not 85K. Target 15% tail (not 30%).
Cache everything repeated: system prompts, skill docs, AGENTS.md — these never change between turns. Single-digit cache hit rate is unacceptable.
Lower context ceiling: 128K window is the stable ceiling for agent conversations. GPUs reduced from 256K to 128K (2026-07-17). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers.

Shape

self: analyze metrics, compute optimal configs, apply changes, verify
delegates:
- apply-liteLLM: update litellm_config.yaml and reload
- apply-agent-config: update hermes config.yaml per agent
- verify-latency: run test inference calls and measure response

Execution

-- Phase 1: Analyze current state (already complete)
-- Phase 2: Apply LiteLLM routing optimization

call apply-liteLLM-routing
  config_path: /opt/inference-harness/litellm_config.yaml
  host: 192.168.68.116

-- Phase 3: Apply agent context compression optimization  

call apply-agent-compression
  agent: mumuni
  host: 192.168.68.14
  config_path: /home/hermes/.hermes/config.yaml

-- Phase 4: Enable llama.cpp prompt caching on GPU hosts

call enable-prompt-caching
  hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110]

-- Phase 5: Verify end-to-end latency

call verify-latency
  host: 192.168.68.116
  models: [syslog-auto, qwen3.6-27B-code, gpu-vision]

3.6 KiB Raw Blame History