abiba-bot/prose-contracts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: LiteLLM contracts — align with 2026-07-08 architecture changes

Browse Source 
- litellm-health: v4.0.0 — Router removed from path (nginx→LiteLLM→GPU directly),
  GPU engines corrected (Docker→systemd), timeouts updated (gemma 120s, qwen 90s),
  Strix Halo: CPU→Vulkan, context sizes + parallel slots added
- litellm-self-heal: ornith context 262K→256K, Rule 7 marked DEPRECATED
  (router not in path), GPU topology synced with gpu-fleet
- litellm-api-keys: reference gpu-fleet as source of truth for keys,
  removed qwen3.6-35B-A3B from model list (never deployed)
This commit is contained in:
root
2026-07-09 05:37:16 +00:00
parent a7fedf71c6
commit a2c5a14c74
3 changed files with 55 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
litellm-api-keys.prose.md
+6 -1
View File
@@ -8,6 +8,10 @@ description: >
Ensures agents never use the master key directly. Rotation is event-driven, Ensures agents never use the master key directly. Rotation is event-driven,
not calendar-driven — rotate only on compromise, personnel change, or not calendar-driven — rotate only on compromise, personnel change, or
periodic security hygiene (quarterly/annually). periodic security hygiene (quarterly/annually).
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
on CT 116. Last verified: 2026-07-09.
--- ---
## Parameters ## Parameters
@@ -38,7 +42,8 @@ description: >
- Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date) - Generate new key with key_alias: "{agent_name}" (e.g., "tanko" — bare name, no date)
- Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" } - Set metadata: { "agent": "{agent_name}", "purpose": "agent-inference" }
- Duration is null (permanent) — inherited from litellm default_key_generate_params - Duration is null (permanent) — inherited from litellm default_key_generate_params
- Set models: ["syslog-auto", "qwen3.6-35B-A3B", "qwen3.6-27B-code", "gemma-4-12b"] - Set models: ["syslog-auto", "qwen3.6-27B-code", "gemma-4-12b", "ornith-1.0-35b"]
- Note: qwen3.6-35B-A3B removed from fleet (was never deployed on any GPU)
- Return the new key - Return the new key
5. **If action == "rotate"**: 5. **If action == "rotate"**:
- Generate new key with same alias (LiteLLM replaces the old key) - Generate new key with same alias (LiteLLM replaces the old key)
litellm-health.prose.md
+37 -28
View File
@@ -2,34 +2,41 @@
kind: function kind: function
name: litellm-health name: litellm-health
description: > description: >
Verifies the LiteLLM inference stack health. Current architecture (2026-06-30): Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server). nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
Router runs as internal-only backend behind LiteLLM. GPU monitoring via Router (harness-router :9000) is DEPRECATED — container still runs but
Prometheus/Grafana and fleet dashboard. is not in the request path. GPU monitoring via Prometheus/Grafana and
fleet dashboard (gpu-monitor :9100).
Designed as a reusable contract for any Syslog agent. Designed as a reusable contract for any Syslog agent.
Source of truth: gpu-fleet.prose.md
--- ---
## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU) ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
``` ```
Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server) Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
│ │ │
Key validation Model routing Key validation
Fallback chains Slot booking Fallback chains
Complexity router Circuit breakers Budget tracking
Budget tracking Redis-backed │
│ │ Prometheus ← metrics
Prometheus ← metrics ←───┘
│ │
Grafana :3001 Grafana :3001
harness-router :9000 — DEPRECATED, container still runs but
NOT in request path. nginx routes /v1 → LiteLLM directly.
Router slot booking + circuit breakers replaced by
LiteLLM native fallbacks + timeouts.
``` ```
**What changed (v3.1.0 → v3.2.0)**: **What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
- Router is BACK — runs internally on :9000 behind LiteLLM - Router REMOVED from request path — LiteLLM proxies directly to GPU
- nginx routes /v1/ and /admin/ → LiteLLM (not router directly) - All GPUs at parallel 2 (was parallel 1)
- Router :9000 is 127.0.0.1-only, not publicly accessible - NVIDIA context reduced 256K→128K to free VRAM
- GPU fleet dashboard: http://192.168.68.24:9100 - LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s
- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus) - nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s
## Parameters ## Parameters
@@ -55,27 +62,29 @@ Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-se
## GPU Fleet Topology ## GPU Fleet Topology
| Host | IP | Hardware | Models Served | Engine | | Host | IP | Hardware | Models Served | Engine | Context | Parallel |
|------|-----|----------|---------------|--------| |------|-----|----------|---------------|--------|---------|----------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker | | llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker | | ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 |
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal | | amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 |
## Model Fallback Chains (LiteLLM) ## Model Fallback Chains (LiteLLM)
| Primary | Timeout | Fallback | Timeout | | Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------| |---------|---------|----------|---------|
| qwen3.6-27B-code | 45s | gemma-4-12b | 30s | | qwen3.6-27B-code | 90s | gemma-4-12b | 120s |
| gemma-4-12b | 30s | qwen3.6-27B-code | 45s | | gemma-4-12b | 120s | qwen3.6-27B-code | 90s |
| ornith-1.0-35b | 120s | qwen → gemma | — | | ornith-1.0-35b | 120s | qwen → gemma | — |
| syslog-auto (balanced) | — | qwen → gemma | — | | syslog-auto (balanced) | 90s | qwen → gemma | — |
> Global: request_timeout=300s, nginx proxy_read_timeout=600s
## Containers on CT 116 ## Containers on CT 116
| Container | Image | Port | Health Check | | Container | Image | Port | Health Check |
|-----------|-------|------|-------------| |-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness | | harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health | | harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health | | harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready | | harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING | | harness-redis | redis:7-alpine | :6379 | PING |
litellm-self-heal.prose.md
+12 -11
View File
@@ -6,6 +6,9 @@ note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04). Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
This contract is now manual-only — triggers require explicit user request. This contract is now manual-only — triggers require explicit user request.
Consider reimplementing as a standalone cron job or prose contract. Consider reimplementing as a standalone cron job or prose contract.
Source of truth for GPU topology: gpu-fleet.prose.md
Last verified: 2026-07-09
description: > description: >
Standing responsibility that monitors LiteLLM health and proactively Standing responsibility that monitors LiteLLM health and proactively
fixes common issues. Reports every action via Zulip DM and RA-H OS fixes common issues. Reports every action via Zulip DM and RA-H OS
@@ -73,11 +76,11 @@ message is sent to the responsible agent with full context.
## GPU Fleet ## GPU Fleet
| Host | IP | Hardware | Models | Engine | | Host | IP | Hardware | Models | Engine | Context |
|------|-----|----------|--------|--------| |------|-----|----------|--------|--------|---------|
| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 | | amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b | llama-server :8080 (Vulkan) | 256K |
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 | | llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 (systemd) | 128K |
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 | | ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 (systemd) | 128K |
## Remediation Rules ## Remediation Rules
@@ -105,12 +108,10 @@ Detect → test prompt via LiteLLM returns non-200
Fix → restart llama-server on GPU host → verify Fix → restart llama-server on GPU host → verify
Escalate → after 2 failed restarts Escalate → after 2 failed restarts
### Rule 7: Router Returns 503 (All GPUs Saturated) ### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
Detect → router logs show QUEUE_TIMEOUT with GPUs idle Router is no longer in the request path (2026-07-08). LiteLLM proxies
Root cause → roster not loaded at startup (empty TIER_MODELS) directly to GPU. This rule is retained for reference but is inactive.
Fix → POST /admin/roster/reload (if endpoint available) or restart router If 503 errors occur, check LiteLLM timeouts and GPU health directly.
Permanent fix → add load_roster() to router startup code
Escalate → if reload doesn't resolve within 2 min
### Rule 8: Agent Keys Invalid (401) ### Rule 8: Agent Keys Invalid (401)
Detect → agents report auth errors Detect → agents report auth errors