root
aa55634571
litellm-health: added GPU fleet topology table, model routing map, 3-tier GPU checks (reachability, VRAM, temp, test inference) litellm-self-heal: added GPU fleet reference, rules for GPU unreachable and model not responding Infrastructure topology now fully documented: - amdpve: Strix Halo CPU (35B, 16 threads, 262K ctx, llama-server) - llm-gpu: RTX 3090 24GB (Dense tier) - ocu-llm: RTX 5070 12GB (MoE + Light tiers)
84 lines
3.8 KiB
Markdown
84 lines
3.8 KiB
Markdown
---
|
|
kind: function
|
|
name: check-litellm-health
|
|
description: >
|
|
Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
|
|
OIDC auth endpoint, container status, and aggregate health on the
|
|
backend host. Designed as a reusable contract for any Syslog agent.
|
|
---
|
|
|
|
## Parameters
|
|
|
|
- public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
|
|
- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
|
|
- auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
|
|
- gpu_hosts: array — GPU inference hosts to check (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
|
|
|
|
## Returns
|
|
|
|
- overall_status: "healthy" | "degraded" | "down"
|
|
- checks: array of { name: string, status: string, detail: string }
|
|
- timestamp: string — ISO timestamp of the check run
|
|
- duration_ms: number — How long the check took
|
|
|
|
## Requires
|
|
|
|
- SSH key access to backend_host for container checks
|
|
- Network access to public_url and auth_host
|
|
- curl and openssl available on the execution host
|
|
|
|
## Ensures
|
|
|
|
- Each check returns a clear pass/fail status with detail message
|
|
- If any endpoint returns non-200, overall_status is "degraded"
|
|
- If backend host unreachable or >2 containers down, overall_status is "down"
|
|
- If /health/unified reports any non-healthy components, status reflects it
|
|
- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
|
|
|
|
## GPU Fleet Topology
|
|
|
|
| Host | IP | Hardware | Models Served | Engine |
|
|
|------|-----|----------|---------------|--------|
|
|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | gemma-4-12b (Dense) | llama-server Docker |
|
|
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | qwen3.6-27B-code (MoE), all Light tier | llama-server Docker |
|
|
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU-only, -ngl 0) | ornith-1.0-35b (35B) | llama-server bare-metal |
|
|
|
|
## Model Routing (LiteLLM → Router → GPU)
|
|
|
|
| Model Request | Router Routes To |
|
|
|--------------|-----------------|
|
|
| `gemma-4-12b` | `GPU_DENSE_URL` → 192.168.68.8:8080 |
|
|
| `qwen3.6-27B-code` | `GPU_MOE_URL` → 192.168.68.110:8080 |
|
|
| `ornith-1.0-35b` | Router routes to 192.168.68.15:8080 |
|
|
| `syslog-auto` | Auto-routed by router tier logic |
|
|
|
|
## Execution
|
|
|
|
1. **Read parameters** — Use provided values or defaults
|
|
2. **Check public endpoints**:
|
|
- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
|
|
- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
|
|
- GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
|
|
- GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
|
|
3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
|
|
- GET http://{{backend_host}}/litellm/ui/ → expect 200
|
|
- GET http://{{backend_host}}/litellm/docs → expect 200
|
|
4. **Check aggregate health endpoint**:
|
|
- GET {{backend_host}}:9000/health/unified → expect 200, check all components
|
|
5. **Check backend container health**:
|
|
- SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
|
|
- Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
|
|
6. **Check GPU fleet health**:
|
|
- For each {{gpu_hosts}}:
|
|
- SSH or nvidia-smi check → verify GPU is reachable
|
|
- Check VRAM usage (warn if >90%)
|
|
- Check temperature (warn if >80°C)
|
|
- Verify model responds to a small test prompt
|
|
7. **Check model inference** — Send test prompt to each model via LiteLLM API:
|
|
- POST /v1/chat/completions with model name → expect 200 + valid response
|
|
- Record token counts for baseline performance
|
|
8. **Check OIDC auth endpoint**:
|
|
- Verify auth.sysloggh.net resolves to {{auth_host}}
|
|
- GET https://auth.sysloggh.net/ → expect login page
|
|
9. **Compile and report** — Determine overall_status from individual check results
|