From dc572889f8d20ee7127b9a6c30a1a875b2e40662 Mon Sep 17 00:00:00 2001 From: Abiba Date: Thu, 16 Jul 2026 17:02:44 +0000 Subject: [PATCH] =?UTF-8?q?contracts:=20sync=20to=20ground=20truth=20?= =?UTF-8?q?=E2=80=94=20ornith-1.0-35b=E2=86=92strix-moe,=20256K=20all=20GP?= =?UTF-8?q?Us,=20real=20LiteLLM=20timeouts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verified on ground 2026-07-16 against CT 116 litellm_config.yaml + GPU hosts: - AMD host serves qwen3.6-35B-udq4 (LiteLLM alias strix-moe); ornith-1.0-35b does NOT exist - All 3 GPUs at 256K ctx, parallel 2 (RTX 3090 was listed 128K/parallel 1) - LiteLLM timeouts: qwen 300s, gemma 120s, strix 300s (were stale 90s/120s) - Added LiteLLM model surface + key scoping to litellm-self-heal - Patched health-check script path ref Files: litellm-self-heal, litellm-health, gpu-fleet, gpu-self-heal, zulip-adapter-lessons, abiba-zulip-restore, hermes-agent-baseline, delegation-prose-contract, mumuni-delegation-prose-contract --- abiba-zulip-restore.prose.md | 2 +- delegation-prose-contract.prose.md | 8 +++---- gpu-fleet.prose.md | 14 ++++++------ gpu-self-heal.prose.md | 2 +- hermes-agent-baseline.prose.md | 2 +- litellm-health.prose.md | 20 ++++++++--------- litellm-self-heal.prose.md | 27 ++++++++++++++++------- mumuni-delegation-prose-contract.prose.md | 8 +++---- zulip-adapter-lessons.prose.md | 4 ++-- 9 files changed, 49 insertions(+), 38 deletions(-) diff --git a/abiba-zulip-restore.prose.md b/abiba-zulip-restore.prose.md index e9fec3c..a02556d 100644 --- a/abiba-zulip-restore.prose.md +++ b/abiba-zulip-restore.prose.md @@ -276,7 +276,7 @@ ZULIP_ROLE=router pm2 start "$(which pi)" --name abiba-zulip \ **Additional fixes applied:** - Fixed `models.json`: replaced `qwen3.6-35B-A3B` (not authorized by LiteLLM) - with `syslog-auto` + `ornith-1.0-35b` (prevents silent worker failure per + with `syslog-auto` + `strix-moe` (prevents silent worker failure per Lesson #11) - Removed stale systemd unit `abiba-zulip.service` (pointed to old TS code) - PM2 saved for auto-restart on boot diff --git a/delegation-prose-contract.prose.md b/delegation-prose-contract.prose.md index c42f9fc..48141c4 100644 --- a/delegation-prose-contract.prose.md +++ b/delegation-prose-contract.prose.md @@ -72,10 +72,10 @@ reads them directly. |--------|-------|----------|------|----------| | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | -| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | -| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | -| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | -| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | +| `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | +| `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | +| `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | +| `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | ### Selection Rules diff --git a/gpu-fleet.prose.md b/gpu-fleet.prose.md index 72a6fa6..c1e7a15 100644 --- a/gpu-fleet.prose.md +++ b/gpu-fleet.prose.md @@ -68,8 +68,8 @@ triggers: │ RTX 3090 │ │ RTX 5070 │ │ Strix Halo│ │ GPU Monitor │ │ 24GB │ │ 12GB │ │ 64GB UMA │ │ :9100 │ │ 256K ctx │ │ 256K ctx │ │ 256K ctx │ │ Watchdog │ -│ qwen3.6 │ │ gemma-4-12b │ │ ornith35B │ │ Prometheus │ -│ 27B-code │ │ :8080 │ │ :8080 │ │ exporter │ +│ qwen3.6 │ │ gemma-4-12b │ │ qwen3.6 │ │ Prometheus │ +│ 27B-code │ │ :8080 │ │ -35B-udq4 │ │ exporter │ │ :8080 │ │ :9400 (exp) │ │ :9400(exp)│ │ :9401 │ │ :9400 │ └─────────────┘ └───────────┘ └──────────────┘ └──────────┘ @@ -103,9 +103,9 @@ but are deprecated for agent configs. Only the stable aliases survive model swap | Model | GPU | Weight | RPM Cap | Timeout | |-------|-----|--------|---------|---------| -| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | 90s | +| qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** | | qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** | -| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **300s** | +| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** | Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path. @@ -186,7 +186,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!") 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models` 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml` - - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s + - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (ornith-1.0-35b does NOT exist — legacy name, do not use) - global request_timeout: 300s, nginx proxy_read_timeout: 600s 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power) 7. Check port conflicts: verify only one llama-server on :8080 per host @@ -247,7 +247,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. - **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2). - **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`. - **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`. -- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`. +- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference. - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill). - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`. @@ -262,7 +262,7 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions. | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context | |-----|-------|-----------|--------------|----------|---------| -| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | 131K | +| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **256K** | | RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** | | Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** | diff --git a/gpu-self-heal.prose.md b/gpu-self-heal.prose.md index f9de786..9209219 100644 --- a/gpu-self-heal.prose.md +++ b/gpu-self-heal.prose.md @@ -120,7 +120,7 @@ depends_on: - **Detect**: Benchmark tok/s vs baseline for each GPU at current context - RTX 3090 (256K ctx, qwen3.6-27B-code): target 75+ tok/s — currently at baseline - RTX 5070 (131K ctx, gemma-4-12b): target 76+ tok/s — optimal for vision/web role - - Strix Halo (256K ctx, ornith-1.0-35b): target 70+ tok/s — currently above baseline + - Strix Halo (256K ctx, strix-moe / qwen3.6-35B-udq4): target 70+ tok/s — currently above baseline - **Fix**: - If tok/s > baseline → context has headroom, consider increasing - If tok/s < 90% baseline → reduce context by 25% and retest diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 34cc947..355273d 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -192,7 +192,7 @@ Key is injected via `infisical run --` wrapper at PM2 startup: "apiKey": "${LITELLM_API_KEY}", "models": [ { "id": "syslog-auto" }, - { "id": "ornith-1.0-35b" }, + { "id": "strix-moe" }, { "id": "qwen3.6-27B-code" }, { "id": "gemma-4-12b" } ] diff --git a/litellm-health.prose.md b/litellm-health.prose.md index 6adbb61..66cfc36 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -42,8 +42,8 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) **What changed (v3.2.0 → v4.0.0 — 2026-07-08)**: - Router REMOVED from request path — LiteLLM proxies directly to GPU - All GPUs at parallel 2 (was parallel 1) -- NVIDIA context reduced 256K→128K to free VRAM -- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s +- NVIDIA context reduced 256K→128K to free VRAM (SUPERSEDED 2026-07-16: all GPUs back to 256K — see litellm-self-heal) +- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal) - nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s ## Parameters @@ -72,18 +72,18 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) | Host | IP | Hardware | Models Served | Engine | Context | Parallel | |------|-----|----------|---------------|--------|---------|----------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 | -| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 | +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 2 | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **256K** | 2 | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 256K | 2 | ## Model Fallback Chains (LiteLLM) | Primary | Timeout | Fallback | Timeout | |---------|---------|----------|---------| -| qwen3.6-27B-code | 90s | gemma-4-12b | 120s | -| gemma-4-12b | 120s | qwen3.6-27B-code | 90s | -| ornith-1.0-35b | 120s | qwen → gemma | — | -| syslog-auto (balanced) | 90s | qwen → gemma | — | +| qwen3.6-27B-code | 300s | gemma-4-12b | 120s | +| gemma-4-12b | 120s | qwen3.6-27B-code | 300s | +| qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — | +| syslog-auto (balanced) | 300s | qwen → gemma | — | > Global: request_timeout=300s, nginx proxy_read_timeout=600s @@ -129,7 +129,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) 7. **Check model inference via LiteLLM** — Test each model: - POST /v1/chat/completions model=gemma-4-12b → expect 200 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 - - POST /v1/chat/completions model=ornith-1.0-35b → expect 200 + - POST /v1/chat/completions model=strix-moe → expect 200 - Use master key for auth 8. **Check agent keys**: diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index f39acdf..a3122cc 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -6,6 +6,7 @@ note: > DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * * Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04), now reimplemented as `litellm-health-check.sh` on CT 116. + Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`). Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph. GPU monitoring integrated from gpu-monitor on .24:9100. @@ -56,18 +57,28 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) | Host | IP | Hardware | Models Served | Engine | Context | Parallel | |------|-----|----------|---------------|--------|---------|----------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **256K** | 1 | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 131K | 2 | -| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 | +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 262144 --parallel 2 --ngl 99`) | **256K** | 2 | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 262144 --parallel 2`, IQ4_NL + MTP draft) | **256K** | 2 | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 256K | 2 | + +> Verified on ground 2026-07-16 via `curl /v1/models` on each host + `llama-wrapper.sh`. The AMD host's underlying model is `qwen3.6-35B-udq4`; LiteLLM exposes it under two `model_name`s: `qwen3.6-35B-udq4` and `strix-moe` (rpm 40). The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced. + +## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116) + +`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`. + +- `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200). +- `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively. +- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. Note `qwen3.6-35B-udq4` is NOT in the agent-key allowlist — agents must use `strix-moe` to reach the AMD model. ## Model Fallback Chains (LiteLLM) | Primary | Timeout | Fallback | Timeout | |---------|---------|----------|---------| -| qwen3.6-27B-code | 90s | gemma-4-12b | 120s | -| gemma-4-12b | 120s | qwen3.6-27B-code | 90s | -| ornith-1.0-35b | 120s | qwen → gemma | — | -| syslog-auto (balanced) | 90s | qwen → gemma | — | +| qwen3.6-27B-code | 300s | gemma-4-12b | 120s | +| gemma-4-12b | 120s | qwen3.6-27B-code | 300s | +| qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — | +| syslog-auto (balanced) | 300s | qwen → gemma | — | > Global: request_timeout=300s, nginx proxy_read_timeout=600s @@ -128,7 +139,7 @@ Run this first on every cycle. Results feed into remediation rules below. ### 5. Check model inference via LiteLLM — test each model - POST /v1/chat/completions model=gemma-4-12b → expect 200 - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 -- POST /v1/chat/completions model=ornith-1.0-35b → expect 200 +- POST /v1/chat/completions model=strix-moe → expect 200 - Use master key for auth ### 6. Check agent keys diff --git a/mumuni-delegation-prose-contract.prose.md b/mumuni-delegation-prose-contract.prose.md index 3f5f9ab..89a2919 100644 --- a/mumuni-delegation-prose-contract.prose.md +++ b/mumuni-delegation-prose-contract.prose.md @@ -92,10 +92,10 @@ raw data never provided. |--------|-------|----------|------|----------| | `syslog-code` | qwen3.6-27B-code | terminal, file, web, memory, skills | Code patches, automation, scripts | Writing/modifying code, creating scripts, debugging, reading/writing files | | `syslog-devops` | qwen3.6-27B-code | terminal, file, web, memory, skills | Infrastructure, DB, bridge, Proxmox | Server ops, SSH, Docker, Proxmox, DB queries, hardware checks | -| `syslog-email` | ornith-1.0-35b | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | -| `syslog-research` | ornith-1.0-35b | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | -| `syslog-review` | ornith-1.0-35b | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | -| `syslog-writer` | ornith-1.0-35b | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | +| `syslog-email` | strix-moe | terminal, file, web, memory, skills | Email automation, mail operations | Sending/receiving email, inbox management, SMTP operations | +| `syslog-research` | strix-moe | terminal, file, web, memory, skills, **browser** | Analysis, classification, data processing | Web research, browser tasks, data analysis, classification, reading docs | +| `syslog-review` | strix-moe | terminal, file, web, memory, skills | Verification, QA, audit validation | **ALWAYS** verify worker output before delivery — especially for infra changes, code builds, and research findings | +| `syslog-writer` | strix-moe | terminal, file, web, memory, skills | Docs, content, branding, reports | Writing docs, reports, proposals, content, markdown formatting | ### Selection Rules diff --git a/zulip-adapter-lessons.prose.md b/zulip-adapter-lessons.prose.md index 7d4e72b..33a4fa1 100644 --- a/zulip-adapter-lessons.prose.md +++ b/zulip-adapter-lessons.prose.md @@ -79,7 +79,7 @@ description: > `messages_processed` stalls. - **Root cause**: Agent's model config (`models.json` or `settings.json`) references a model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B` - configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session + configured but LiteLLM only exposes `strix-moe` (alias for qwen3.6-35B-udq4) under that key. pi's session workers emit 403 on first prompt, then never recover because the error doesn't trigger `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue. - **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer " http://192.168.68.116/v1/models` output. A stuck worker shows @@ -88,7 +88,7 @@ description: > (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process. - **Prevention**: Use `syslog-auto` as default model for all agents — it handles model - routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should + routing and fallback automatically. Direct model IDs (`strix-moe`, etc.) should only be used when explicitly requested. Validate model IDs at agent setup time. - **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider