mumuni-bot
aac4c7eac3
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Third and final PR of the post-migration contract sweep (ref PRs #54, #55). Normal-sensitivity contracts per AGENTS.md (any registered agent): - gpu-fleet: key roster row, Mumuni Agent Profile header + status table (thermal-safeguard incident note left as historical record) - infrastructure-update: apt patch table row; Hermes config path block corrected to /home/hermes/.hermes + system unit /etc/systemd/system/ hermes-gateway.service (was /root/.hermes + user unit) - infrastructure-maintenance: Hermes gateway health-check target - inference-optimization: apply-agent-compression host + config_path All values verified live on kagentz 2026-08-29 (hostname, IP, user, systemd unit, /home/hermes/.hermes). Abiba-owned refs (.24 pi agent, GPU monitor :9100) and CRITICAL files intentionally untouched - flagged to Abiba via relay #739. Refs PR #54, #55; relay #738/#739.
104 lines
3.6 KiB
Markdown
104 lines
3.6 KiB
Markdown
---
|
|
name: inference-optimization
|
|
kind: responsibility
|
|
description: >
|
|
Optimizes the full Syslog inference stack — LiteLLM routing weights, GPU model
|
|
assignments, agent context management, and prompt caching — to reduce response
|
|
times to sub-15s average. All GPUs now at 128K context (stable ceiling).
|
|
id: 067NC6KP02RG60S50M40E30928
|
|
---
|
|
|
|
### Goal
|
|
|
|
Syslog inference response times reduced to sub-15s average by optimizing the full
|
|
stack: LiteLLM routing weights, GPU model assignments, Hermes agent context
|
|
management, and prompt caching — without sacrificing agent capability.
|
|
|
|
### Requires
|
|
|
|
- `inference-metrics`: current SpendLogs from CT116 LiteLLM Postgres — avg
|
|
request_duration_ms, prompt_tokens, completion_tokens, model_group breakdown,
|
|
cache_hit rate over the last 3 hours
|
|
- `agent-configs`: current config.yaml from each active Hermes agent (Mumuni
|
|
.123, any others on .129/.122) including compression, model, context_window,
|
|
prompt_caching, memory settings
|
|
- `gpu-health`: health check response from all 3 GPU backends (strix-moe .15:8080,
|
|
qwen .8:8080, gemma .110:8080)
|
|
|
|
### Maintains
|
|
|
|
The optimized inference stack configuration — every change is applied and
|
|
verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of
|
|
inference calls.
|
|
|
|
#### liteLLM-routing
|
|
The syslog-auto routing weights, model-specific timeouts, RPM limits, and
|
|
model_list entries on CT116 `/opt/inference-harness/litellm_config.yaml`.
|
|
|
|
#### agent-compression
|
|
Each Hermes agent's `~/.hermes/config.yaml` compression, context_window,
|
|
prompt_caching, and model sections.
|
|
|
|
#### prompt-caching
|
|
LiteLLM cache configuration and llama.cpp `--cache-prompt` flag on GPU hosts.
|
|
|
|
#### verification
|
|
End-to-end latency measurements after changes applied — at least 3 test
|
|
inference calls per model path measuring ttft (time-to-first-token) and total
|
|
duration.
|
|
|
|
### Continuity
|
|
|
|
- input-driven
|
|
|
|
### Strategies
|
|
|
|
**Context is the root cause.** Every ~46K prompt token costs ~87s of
|
|
prefill time at 532 tok/s. Fix context first, routing second.
|
|
|
|
- **Route by task**: qwen for code/standard queries; gemma for
|
|
compression/auxiliary; strix-moe for compression tasks.
|
|
- **Compress aggressively**: threshold at 40% (not 65%) — a 128K window should
|
|
compact at 51K, not 85K. Target 15% tail (not 30%).
|
|
- **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these
|
|
never change between turns. Single-digit cache hit rate is unacceptable.
|
|
- **Lower context ceiling**: 128K window is the stable ceiling for agent conversations.
|
|
GPUs reduced from 256K to 128K (2026-07-17). 128K window should compact at 85K (0.65 threshold). For larger contexts, route to external providers.
|
|
|
|
### Shape
|
|
|
|
- `self`: analyze metrics, compute optimal configs, apply changes, verify
|
|
- `delegates`:
|
|
- `apply-liteLLM`: update litellm_config.yaml and reload
|
|
- `apply-agent-config`: update hermes config.yaml per agent
|
|
- `verify-latency`: run test inference calls and measure response
|
|
|
|
### Execution
|
|
|
|
```prose
|
|
-- Phase 1: Analyze current state (already complete)
|
|
-- Phase 2: Apply LiteLLM routing optimization
|
|
|
|
call apply-liteLLM-routing
|
|
config_path: /opt/inference-harness/litellm_config.yaml
|
|
host: 192.168.68.116
|
|
|
|
-- Phase 3: Apply agent context compression optimization
|
|
|
|
call apply-agent-compression
|
|
agent: mumuni
|
|
host: 192.168.68.14
|
|
config_path: /home/hermes/.hermes/config.yaml
|
|
|
|
-- Phase 4: Enable llama.cpp prompt caching on GPU hosts
|
|
|
|
call enable-prompt-caching
|
|
hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110]
|
|
|
|
-- Phase 5: Verify end-to-end latency
|
|
|
|
call verify-latency
|
|
host: 192.168.68.116
|
|
models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b]
|
|
```
|