prose-contracts/litellm-health.prose.md

---
kind: function
name: litellm-health
status: deprecated
deprecated_on: 2026-07-09
replaced_by: litellm-self-heal.prose.md
note: >
  Consolidated into litellm-self-heal.prose.md to eliminate duplication
  of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within litellm-self-heal.
  This file is retained for reference only — use litellm-self-heal instead.
description: >
  Verifies the LiteLLM inference stack health. Current architecture (2026-07-09):
  nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy.
  Router (harness-router :9000) is DEPRECATED — container still runs but
  is not in the request path. GPU monitoring via Prometheus/Grafana and
  fleet dashboard (gpu-monitor :9100).
  Designed as a reusable contract for any Syslog agent.
  
  Source of truth: gpu-fleet.prose.md
---

## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

```
Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001

  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
```

**What changed (v3.2.0 → v4.0.0 — 2026-07-08)**:
- Router REMOVED from request path — LiteLLM proxies directly to GPU
- All GPUs at parallel 2 (was parallel 1)
- NVIDIA context reduced 256K→128K to free VRAM — now the stable ceiling across all GPUs (2026-07-17)
- LiteLLM timeouts tuned: gemma 25→120s, qwen 40→90s  (SUPERSEDED 2026-07-16: qwen 300s, gemma 120s, strix 300s — see litellm-self-heal)
- nginx proxy_read_timeout: 600s, LiteLLM request_timeout: 300s

## Parameters

- public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
- auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

## Returns

- overall_status: "healthy" | "degraded" | "down"
- checks: array of { name: string, status: string, detail: string }
- timestamp: string — ISO timestamp of the check run
- duration_ms: number — How long the check took

## Requires

- SSH key access to backend_host for container checks
- Network access to public_url, auth_host, and gpu_dashboard_url
- LiteLLM master key for key management endpoints

## GPU Fleet Topology

| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
|------|-----|----------|---------------|--------|---------|----------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | **128K** | 2 |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | **128K** | 2 |
| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: strix-moe) | llama-server systemd (Vulkan) | 128K | 2 |

## Model Fallback Chains (LiteLLM)

| Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------|
| qwen3.6-27B-code | 300s | gemma-4-12b | 120s |
| gemma-4-12b | 120s | qwen3.6-27B-code | 300s |
| qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — |
| syslog-auto (balanced) | 300s | qwen → gemma | — |

> Global: request_timeout=300s, nginx proxy_read_timeout=600s

## Containers on CT 116

| Container | Image | Port | Health Check |
|-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING |
| harness-dashboard | inference-harness-dashboard | :3000 | /health |
| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
| harness-prometheus | prom/prometheus | :9090 | /-/healthy |

## Execution

1. **Read parameters** — Use provided values or defaults

2. **Check public endpoints**:
   - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
   - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")

3. **Check LiteLLM health (no-auth)**:
   - GET http://{{backend_host}}/litellm/health/liveliness → expect 200

4. **Check backend container health**:
   - SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
   - Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
   - Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus

5. **Check router roster loaded**:
   - GET http://{{backend_host}}:9000/health → expect 200
   - GET http://{{backend_host}}:9000/health/unified → expect 3 models
   - If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload

6. **Check GPU fleet health** (via fleet dashboard):
   - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
   - Verify GPUs reporting status "healthy"
   - Check alerts array for active warnings/critical

7. **Check model inference via LiteLLM** — Test each model:
   - POST /v1/chat/completions model=gemma-4-12b → expect 200
   - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
   - POST /v1/chat/completions model=strix-moe → expect 200
   - Use master key for auth

8. **Check agent keys**:
   - GET /key/list with master key → verify all 6 agents have keys

9. **Check Grafana**:
   - GET {{grafana_url}}/api/health → expect 200

10. **Compile and report** — Determine overall_status from individual check results