prose-contracts/litellm-health.prose.md

kind, name, status, deprecated_on, replaced_by, note, description

kind	name	status	deprecated_on	replaced_by	note	description
function	litellm-health	deprecated	2026-07-09	litellm-self-heal.prose.md	Consolidated into litellm-self-heal.prose.md to eliminate duplication of architecture diagrams, GPU topology, timeout tables, and container lists. Health check is now § Health Check within litellm-self-heal. This file is retained for reference only — use litellm-self-heal instead.	Verifies the LiteLLM inference stack health. Current architecture (2026-07-09): nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy. Router (harness-router :9000) is DEPRECATED — container still runs but is not in the request path. GPU monitoring via Prometheus/Grafana and fleet dashboard (gpu-monitor :9100). Designed as a reusable contract for any Syslog agent. Source of truth: gpu-fleet.prose.md

Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001

  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.

What changed (v3.2.0 → v4.0.0 — 2026-07-08):

• Router REMOVED from request path — LiteLLM proxies directly to GPU
• All GPUs at parallel 2 (was parallel 1)
• NVIDIA context reduced 256K→128K to free VRAM — now the stable ceiling across all GPUs (2026-07-17)
• LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal)
• nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s

Parameters

• public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
• backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
• auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
• gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
• gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
• grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

Returns

• overall_status: "healthy" | "degraded" | "down"
• checks: array of { name: string, status: string, detail: string }
• timestamp: string — ISO timestamp of the check run
• duration_ms: number — How long the check took

Requires

• SSH key access to backend_host for container checks
• Network access to public_url, auth_host, and gpu_dashboard_url
• LiteLLM master key for key management endpoints

GPU Fleet Topology

Host	IP	Hardware	Models Served	Engine	Context	Parallel
llm-gpu	192.168.68.8	NVIDIA RTX 3090 (24 GB)	qwen3.6-27B-code	llama-server systemd	128K	2
ocu-llm	192.168.68.110	NVIDIA RTX 5070 (12 GB)	gemma-4-12b	llama-server systemd	128K	2
amdpve	192.168.68.15	AMD Strix Halo 64GB UMA	qwen3.6-35B-udq4 (LiteLLM alias: strix-moe)	llama-server systemd (Vulkan)	128K	2

Model Fallback Chains (LiteLLM)

Primary	Timeout	Fallback	Timeout
qwen3.6-27B-code	300s	gemma-4-12b	120s
gemma-4-12b	120s	qwen3.6-27B-code	300s
qwen3.6-35B-udq4 / strix-moe	300s	qwen → gemma	—
syslog-auto (balanced)	300s	qwen → gemma	—

Global: request_timeout=300s, nginx proxy_read_timeout=600s

Containers on CT 116

Container	Image	Port	Health Check
harness-litellm	berriai/litellm:1.90.0-rc.1	:4000→:4001	/health/liveliness
harness-router	inference-harness-router	:9000 (127.0.0.1)	/health (DEPRECATED — not in path)
harness-nginx	nginx:alpine	:80	HTTP 200 on /health
harness-postgres	postgres:16-alpine	:5432	pg_isready
harness-redis	redis:7-alpine	:6379	PING
harness-dashboard	inference-harness-dashboard	:3000	/health
harness-grafana	grafana/grafana	:3000→:3001	/api/health
harness-prometheus	prom/prometheus	:9090	/-/healthy

Execution

1. Read parameters — Use provided values or defaults

2. Check public endpoints:

   • GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
   • GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")

3. Check LiteLLM health (no-auth):

   • GET http://{{backend_host}}/litellm/health/liveliness → expect 200

4. Check backend container health:

   • SSH to {{backend_host}} → docker ps → verify 8 containers healthy
   • Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
   • Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus

5. Check router roster loaded:

   • GET http://{{backend_host}}:9000/health → expect 200
   • GET http://{{backend_host}}:9000/health/unified → expect 3 models
   • If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload

6. Check GPU fleet health (via fleet dashboard):

   • GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
   • Verify GPUs reporting status "healthy"
   • Check alerts array for active warnings/critical

7. Check model inference via LiteLLM — Test each model:

   • POST /v1/chat/completions model=gemma-4-12b → expect 200
   • POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
   • POST /v1/chat/completions model=strix-moe → expect 200
   • Use master key for auth

8. Check agent keys:

   • GET /key/list with master key → verify all 6 agents have keys

9. Check Grafana:

   • GET {{grafana_url}}/api/health → expect 200

10. Compile and report — Determine overall_status from individual check results