---
name: inference-optimization
kind: responsibility
description: >
  Optimizes the full Syslog inference stack — LiteLLM routing weights, GPU model
  assignments, agent context management, and prompt caching — to reduce response
  times to sub-15s average. All GPUs now at 128K context (stable ceiling).
id: 067NC6KP02RG60S50M40E30928
---

### Goal

Syslog inference response times reduced to sub-15s average by optimizing the full
stack: LiteLLM routing weights, GPU model assignments, Hermes agent context
management, and prompt caching — without sacrificing agent capability.

### Requires

- `inference-metrics`: current SpendLogs from CT116 LiteLLM Postgres — avg
  request_duration_ms, prompt_tokens, completion_tokens, model_group breakdown,
  cache_hit rate over the last 3 hours
- `agent-configs`: current config.yaml from each active Hermes agent (Mumuni
  .123, any others on .129/.122) including compression, model, context_window,
  prompt_caching, memory settings
- `gpu-health`: health check response from all 3 GPU backends (ornith .15:8080,
  qwen .8:8080, gemma .110:8080)

### Maintains

The optimized inference stack configuration — every change is applied and
verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of
non-ornith traffic; ≤ 30000 for ornith-bound agentic calls.

#### liteLLM-routing
The syslog-auto routing weights, model-specific timeouts, RPM limits, and
model_list entries on CT116 `/opt/inference-harness/litellm_config.yaml`.

#### agent-compression
Each Hermes agent's `~/.hermes/config.yaml` compression, context_window,
prompt_caching, and model sections.

#### prompt-caching
LiteLLM cache configuration and llama.cpp `--cache-prompt` flag on GPU hosts.

#### verification
End-to-end latency measurements after changes applied — at least 3 test
inference calls per model path measuring ttft (time-to-first-token) and total
duration.

### Continuity

- input-driven

### Strategies

**Context is the root cause.** Every ~46K prompt token costs ~87s of ornith
prefill time at 532 tok/s. Fix context first, routing second.

- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
  queries; gemma for compression/auxiliary. Never send simple completion to a
  35B MoE.
- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
  compact at 102K, not 166K. Target 15% tail (not 30%).
- **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these
  never change between turns. Single-digit cache hit rate is unacceptable.
- **Lower context ceiling**: 128K window is the stable ceiling for agent conversations.
  GPUs reduced from 256K to 128K (2026-07-17). For larger contexts, route to external providers.

### Shape

- `self`: analyze metrics, compute optimal configs, apply changes, verify
- `delegates`:
  - `apply-liteLLM`: update litellm_config.yaml and reload
  - `apply-agent-config`: update hermes config.yaml per agent
  - `verify-latency`: run test inference calls and measure response

### Execution

```prose
-- Phase 1: Analyze current state (already complete)
-- Phase 2: Apply LiteLLM routing optimization

call apply-liteLLM-routing
  config_path: /opt/inference-harness/litellm_config.yaml
  host: 192.168.68.116

-- Phase 3: Apply agent context compression optimization  

call apply-agent-compression
  agent: mumuni
  host: 192.168.68.123
  config_path: /root/.hermes/config.yaml

-- Phase 4: Enable llama.cpp prompt caching on GPU hosts

call enable-prompt-caching
  hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110]

-- Phase 5: Verify end-to-end latency

call verify-latency
  host: 192.168.68.116
  models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b, ornith-1.0-35b]
```