From a2c5a14c74a1d5368ad62fdcdc618eeaaf4b305f Mon Sep 17 00:00:00 2001 From: root Date: Thu, 9 Jul 2026 05:37:16 +0000 Subject: [PATCH] =?UTF-8?q?fix:=20LiteLLM=20contracts=20=E2=80=94=20align?= =?UTF-8?q?=20with=202026-07-08=20architecture=20changes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly), GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s), Strix Halo: CPU→Vulkan, context sizes + parallel slots added - litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED (router not in path), GPU topology synced with gpu-fleet - litellm-api-keys: reference gpu-fleet as source of truth for keys, removed qwen3.6-35B-A3B from model list (never deployed) --- litellm-api-keys.prose.md | 7 +++- litellm-health.prose.md | 65 ++++++++++++++++++++++---------------- litellm-self-heal.prose.md | 23 +++++++------- 3 files changed, 55 insertions(+), 40 deletions(-) diff --git a/litellm-api-keys.prose.md b/litellm-api-keys.prose.md index 3a0e0ec..949e6fd 100644 --- a/litellm-api-keys.prose.md +++ b/litellm-api-keys.prose.md @@ -8,6 +8,10 @@ description: > Ensures agents never use the master key directly. Rotation is event-driven, not calendar-driven — rotate only on compromise, personnel change, or periodic security hygiene (quarterly/annually). + + Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys. + Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml + on CT 116. Last verified: 2026-07-09. --- ## Parameters @@ -38,7 +42,8 @@ description: > - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date) - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" } - Duration is null (permanent) — inherited from litellm default_key_generate_params - - Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"] + - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"] + - Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU) - Return the new key 5. **If action == "rotate"**: - Generate new key with same alias (LiteLLM replaces the old key) diff --git a/litellm-health.prose.md b/litellm-health.prose.md index 84c1ed3..88f0e6e 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -2,34 +2,41 @@ kind: function name: litellm-health description: > - Verifies the LiteLLM inference stack health. Current architecture (2026-06-30): - nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server). - Router runs as internal-only backend behind LiteLLM. GPU monitoring via - Prometheus/Grafana and fleet dashboard. + Verifies the LiteLLM inference stack health. Current architecture (2026-07-09): + nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy. + Router (harness-router :9000) is DEPRECATED — container still runs but + is not in the request path. GPU monitoring via Prometheus/Grafana and + fleet dashboard (gpu-monitor :9100). Designed as a reusable contract for any Syslog agent. + + Source of truth: gpu-fleet.prose.md --- -## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU) +## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU) ``` -Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server) - │ │ - Key validation Model routing - Fallback chains Slot booking - Complexity router Circuit breakers - Budget tracking Redis-backed - │ │ - Prometheus ← metrics ←───┘ +Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) + │ + Key validation + Fallback chains + Budget tracking + │ + Prometheus ← metrics │ Grafana :3001 + + harness-router :9000 — DEPRECATED, container still runs but + NOT in request path. nginx routes /v1 → LiteLLM directly. + Router slot booking + circuit breakers replaced by + LiteLLM native fallbacks + timeouts. ``` -**What changed (v3.1.0 → v3.2.0)**: -- Router is BACK — runs internally on :9000 behind LiteLLM -- nginx routes /v1/ and /admin/ → LiteLLM (not router directly) -- Router :9000 is 127.0.0.1-only, not publicly accessible -- GPU fleet dashboard: http://192.168.68.24:9100 -- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus) +**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**: +- Router REMOVED from request path — LiteLLM proxies directly to GPU +- All GPUs at parallel 2 (was parallel 1) +- NVIDIA context reduced 256K→128K to free VRAM +- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s +- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s ## Parameters @@ -55,27 +62,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se ## GPU Fleet Topology -| Host | IP | Hardware | Models Served | Engine | -|------|-----|----------|---------------|--------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker | -| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal | +| Host | IP | Hardware | Models Served | Engine | Context | Parallel | +|------|-----|----------|---------------|--------|---------|----------| +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 | ## Model Fallback Chains (LiteLLM) | Primary | Timeout | Fallback | Timeout | |---------|---------|----------|---------| -| qwen3.6-27B-code | 45s | gemma-4-12b | 30s | -| gemma-4-12b | 30s | qwen3.6-27B-code | 45s | +| qwen3.6-27B-code | 90s | gemma-4-12b | 120s | +| gemma-4-12b | 120s | qwen3.6-27B-code | 90s | | ornith-1.0-35b | 120s | qwen → gemma | — | -| syslog-auto (balanced) | — | qwen → gemma | — | +| syslog-auto (balanced) | 90s | qwen → gemma | — | + +> Global: request_timeout=300s, nginx proxy_read_timeout=600s ## Containers on CT 116 | Container | Image | Port | Health Check | |-----------|-------|------|-------------| | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness | -| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health | +| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) | | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health | | harness-postgres | postgres:16-alpine | :5432 | pg_isready | | harness-redis | redis:7-alpine | :6379 | PING | diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 8a696f9..cc470d2 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -6,6 +6,9 @@ note: > Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04). This contract is now manual-only — triggers require explicit user request. Consider reimplementing as a standalone cron job or prose contract. + + Source of truth for GPU topology: gpu-fleet.prose.md + Last verified: 2026-07-09 description: > Standing responsibility that monitors LiteLLM health and proactively fixes common issues. Reports every action via Zulip DM and RA-H OS @@ -73,11 +76,11 @@ message is sent to the responsible agent with full context. ## GPU Fleet -| Host | IP | Hardware | Models | Engine | -|------|-----|----------|--------|--------| -| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 | -| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 | -| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 | +| Host | IP | Hardware | Models | Engine | Context | +|------|-----|----------|--------|--------|---------| +| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b | llama-server :8080 (Vulkan) | 256K | +| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 (systemd) | 128K | +| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 (systemd) | 128K | ## Remediation Rules @@ -105,12 +108,10 @@ Detect → test prompt via LiteLLM returns non-200 Fix → restart llama-server on GPU host → verify Escalate → after 2 failed restarts -### Rule 7: Router Returns 503 (All GPUs Saturated) -Detect → router logs show QUEUE_TIMEOUT with GPUs idle -Root cause → roster not loaded at startup (empty TIER_MODELS) -Fix → POST /admin/roster/reload (if endpoint available) or restart router -Permanent fix → add load_roster() to router startup code -Escalate → if reload doesn't resolve within 2 min +### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED +Router is no longer in the request path (2026-07-08). LiteLLM proxies +directly to GPU. This rule is retained for reference but is inactive. +If 503 errors occur, check LiteLLM timeouts and GPU health directly. ### Rule 8: Agent Keys Invalid (401) Detect → agents report auth errors -- 2.54.0