prose-contracts/litellm-health.prose.md

kind, name, description

kind	name	description
function	litellm-health	Verifies the LiteLLM inference stack health. Current architecture (2026-06-30): nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server). Router runs as internal-only backend behind LiteLLM. GPU monitoring via Prometheus/Grafana and fleet dashboard. Designed as a reusable contract for any Syslog agent.

Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)

Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
                        │                    │
                  Key validation      Model routing
                  Fallback chains     Slot booking
                  Complexity router   Circuit breakers
                  Budget tracking     Redis-backed
                        │                    │
                  Prometheus ← metrics  ←───┘
                        │
                    Grafana :3001

What changed (v3.1.0 → v3.2.0):

• Router is BACK — runs internally on :9000 behind LiteLLM
• nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
• Router :9000 is 127.0.0.1-only, not publicly accessible
• GPU fleet dashboard: http://192.168.68.24:9100
• Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)

Parameters

• public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
• backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
• auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
• gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
• gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
• grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

Returns

• overall_status: "healthy" | "degraded" | "down"
• checks: array of { name: string, status: string, detail: string }
• timestamp: string — ISO timestamp of the check run
• duration_ms: number — How long the check took

Requires

• SSH key access to backend_host for container checks
• Network access to public_url, auth_host, and gpu_dashboard_url
• LiteLLM master key for key management endpoints

GPU Fleet Topology

Host	IP	Hardware	Models Served	Engine
llm-gpu	192.168.68.8	NVIDIA RTX 3090 (24 GB)	qwen3.6-27B-code	llama-server Docker
ocu-llm	192.168.68.110	NVIDIA RTX 5070 (12 GB)	gemma-4-12b	llama-server Docker
amdpve	192.168.68.15	AMD Strix Halo (CPU)	ornith-1.0-35b (35B)	llama-server bare-metal

Model Fallback Chains (LiteLLM)

Primary	Timeout	Fallback	Timeout
qwen3.6-27B-code	45s	gemma-4-12b	30s
gemma-4-12b	30s	qwen3.6-27B-code	45s
ornith-1.0-35b	120s	qwen → gemma	—
syslog-auto (balanced)	—	qwen → gemma	—

Containers on CT 116

Container	Image	Port	Health Check
harness-litellm	berriai/litellm:1.90.0-rc.1	:4000→:4001	/health/liveliness
harness-router	inference-harness-router	:9000 (127.0.0.1)	/health
harness-nginx	nginx:alpine	:80	HTTP 200 on /health
harness-postgres	postgres:16-alpine	:5432	pg_isready
harness-redis	redis:7-alpine	:6379	PING
harness-dashboard	inference-harness-dashboard	:3000	/health
harness-grafana	grafana/grafana	:3000→:3001	/api/health
harness-prometheus	prom/prometheus	:9090	/-/healthy

Execution

1. Read parameters — Use provided values or defaults

2. Check public endpoints:

   • GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
   • GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")

3. Check LiteLLM health (no-auth):

   • GET http://{{backend_host}}/litellm/health/liveliness → expect 200

4. Check backend container health:

   • SSH to {{backend_host}} → docker ps → verify 8 containers healthy
   • Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
   • Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus

5. Check router roster loaded:

   • GET http://{{backend_host}}:9000/health → expect 200
   • GET http://{{backend_host}}:9000/health/unified → expect 3 models
   • If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload

6. Check GPU fleet health (via fleet dashboard):

   • GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
   • Verify GPUs reporting status "healthy"
   • Check alerts array for active warnings/critical

7. Check model inference via LiteLLM — Test each model:

   • POST /v1/chat/completions model=gemma-4-12b → expect 200
   • POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
   • POST /v1/chat/completions model=ornith-1.0-35b → expect 200
   • Use master key for auth

8. Check agent keys:

   • GET /key/list with master key → verify all 6 agents have keys

9. Check Grafana:

   • GET {{grafana_url}}/api/health → expect 200

10. Compile and report — Determine overall_status from individual check results