Abiba
bb20637b32
- litellm-health: 'check-litellm-health' -> 'litellm-health' - infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring' Fixes 'prose run <filename>' mismatch. No external references broken.
125 lines
5.2 KiB
Markdown
125 lines
5.2 KiB
Markdown
---
|
|
kind: function
|
|
name: litellm-health
|
|
description: >
|
|
Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
|
|
nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
|
|
Router runs as internal-only backend behind LiteLLM. GPU monitoring via
|
|
Prometheus/Grafana and fleet dashboard.
|
|
Designed as a reusable contract for any Syslog agent.
|
|
---
|
|
|
|
## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
|
|
|
|
```
|
|
Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
|
|
│ │
|
|
Key validation Model routing
|
|
Fallback chains Slot booking
|
|
Complexity router Circuit breakers
|
|
Budget tracking Redis-backed
|
|
│ │
|
|
Prometheus ← metrics ←───┘
|
|
│
|
|
Grafana :3001
|
|
```
|
|
|
|
**What changed (v3.1.0 → v3.2.0)**:
|
|
- Router is BACK — runs internally on :9000 behind LiteLLM
|
|
- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
|
|
- Router :9000 is 127.0.0.1-only, not publicly accessible
|
|
- GPU fleet dashboard: http://192.168.68.24:9100
|
|
- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
|
|
|
|
## Parameters
|
|
|
|
- public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
|
|
- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
|
|
- auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
|
|
- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
|
|
- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
|
|
- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
|
|
|
|
## Returns
|
|
|
|
- overall_status: "healthy" | "degraded" | "down"
|
|
- checks: array of { name: string, status: string, detail: string }
|
|
- timestamp: string — ISO timestamp of the check run
|
|
- duration_ms: number — How long the check took
|
|
|
|
## Requires
|
|
|
|
- SSH key access to backend_host for container checks
|
|
- Network access to public_url, auth_host, and gpu_dashboard_url
|
|
- LiteLLM master key for key management endpoints
|
|
|
|
## GPU Fleet Topology
|
|
|
|
| Host | IP | Hardware | Models Served | Engine |
|
|
|------|-----|----------|---------------|--------|
|
|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
|
|
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
|
|
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
|
|
|
|
## Model Fallback Chains (LiteLLM)
|
|
|
|
| Primary | Timeout | Fallback | Timeout |
|
|
|---------|---------|----------|---------|
|
|
| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
|
|
| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
|
|
| ornith-1.0-35b | 120s | qwen → gemma | — |
|
|
| syslog-auto (balanced) | — | qwen → gemma | — |
|
|
|
|
## Containers on CT 116
|
|
|
|
| Container | Image | Port | Health Check |
|
|
|-----------|-------|------|-------------|
|
|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
|
|
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
|
|
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
|
|
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
|
|
| harness-redis | redis:7-alpine | :6379 | PING |
|
|
| harness-dashboard | inference-harness-dashboard | :3000 | /health |
|
|
| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
|
|
| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
|
|
|
|
## Execution
|
|
|
|
1. **Read parameters** — Use provided values or defaults
|
|
|
|
2. **Check public endpoints**:
|
|
- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
|
|
- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
|
|
|
|
3. **Check LiteLLM health (no-auth)**:
|
|
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
|
|
|
|
4. **Check backend container health**:
|
|
- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
|
|
- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
|
|
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
|
|
|
|
5. **Check router roster loaded**:
|
|
- GET http://{{backend_host}}:9000/health → expect 200
|
|
- GET http://{{backend_host}}:9000/health/unified → expect 3 models
|
|
- If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
|
|
|
|
6. **Check GPU fleet health** (via fleet dashboard):
|
|
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
|
|
- Verify GPUs reporting status "healthy"
|
|
- Check alerts array for active warnings/critical
|
|
|
|
7. **Check model inference via LiteLLM** — Test each model:
|
|
- POST /v1/chat/completions model=gemma-4-12b → expect 200
|
|
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
|
|
- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
|
|
- Use master key for auth
|
|
|
|
8. **Check agent keys**:
|
|
- GET /key/list with master key → verify all 6 agents have keys
|
|
|
|
9. **Check Grafana**:
|
|
- GET {{grafana_url}}/api/health → expect 200
|
|
|
|
10. **Compile and report** — Determine overall_status from individual check results
|