SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring)

Browse Source 
- litellm-health: 'check-litellm-health' -> 'litellm-health'
- infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring'
Fixes 'prose run <filename>' mismatch. No external references broken.
This commit is contained in:
Abiba
2026-07-02 20:35:44 +00:00
parent b15771bfd9
commit bb20637b32
2 changed files with 240 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
litellm-health.prose.md
+90 -49
View File
@@ -1,18 +1,44 @@
---
kind: function
name: check-litellm-health
name: litellm-health
description: >
Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
OIDC auth endpoint, container status, and aggregate health on the
backend host. Designed as a reusable contract for any Syslog agent.
Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
Router runs as internal-only backend behind LiteLLM. GPU monitoring via
Prometheus/Grafana and fleet dashboard.
Designed as a reusable contract for any Syslog agent.
---
## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
```
Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
│ │
Key validation Model routing
Fallback chains Slot booking
Complexity router Circuit breakers
Budget tracking Redis-backed
│ │
Prometheus ← metrics ←───┘
│
Grafana :3001
```
**What changed (v3.1.0 → v3.2.0)**:
- Router is BACK — runs internally on :9000 behind LiteLLM
- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
- Router :9000 is 127.0.0.1-only, not publicly accessible
- GPU fleet dashboard: http://192.168.68.24:9100
- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
## Parameters
- public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
- auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
- gpu_hosts: array — GPU inference hosts to check (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
## Returns
@@ -24,60 +50,75 @@ description: >
## Requires
- SSH key access to backend_host for container checks
- Network access to public_url and auth_host
- curl and openssl available on the execution host
## Ensures
- Each check returns a clear pass/fail status with detail message
- If any endpoint returns non-200, overall_status is "degraded"
- If backend host unreachable or >2 containers down, overall_status is "down"
- If /health/unified reports any non-healthy components, status reflects it
- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
- Network access to public_url, auth_host, and gpu_dashboard_url
- LiteLLM master key for key management endpoints
## GPU Fleet Topology
| Host | IP | Hardware | Models Served | Engine |
|------|-----|----------|---------------|--------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | gemma-4-12b (Dense) | llama-server Docker |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | qwen3.6-27B-code (MoE), all Light tier | llama-server Docker |
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU-only, -ngl 0) | ornith-1.0-35b (35B) | llama-server bare-metal |
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
## Model Routing (LiteLLM → Router → GPU)
## Model Fallback Chains (LiteLLM)
| Model Request | Router Routes To |
|--------------|-----------------|
| `gemma-4-12b` | `GPU_DENSE_URL` → 192.168.68.8:8080 |
| `qwen3.6-27B-code` | `GPU_MOE_URL` → 192.168.68.110:8080 |
| `ornith-1.0-35b` | Router routes to 192.168.68.15:8080 |
| `syslog-auto` | Auto-routed by router tier logic |
| Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------|
| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
| ornith-1.0-35b | 120s | qwen → gemma | — |
| syslog-auto (balanced) | — | qwen → gemma | — |
## Containers on CT 116
| Container | Image | Port | Health Check |
|-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING |
| harness-dashboard | inference-harness-dashboard | :3000 | /health |
| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
## Execution
1. **Read parameters** — Use provided values or defaults
2. **Check public endpoints**:
- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
- GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
- GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
- GET http://{{backend_host}}/litellm/ui/ → expect 200
- GET http://{{backend_host}}/litellm/docs → expect 200
4. **Check aggregate health endpoint**:
- GET {{backend_host}}:9000/health/unified → expect 200, check all components
5. **Check backend container health**:
- SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
- Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
6. **Check GPU fleet health**:
- For each {{gpu_hosts}}:
- SSH or nvidia-smi check → verify GPU is reachable
- Check VRAM usage (warn if >90%)
- Check temperature (warn if >80°C)
- Verify model responds to a small test prompt
7. **Check model inference** — Send test prompt to each model via LiteLLM API:
- POST /v1/chat/completions with model name → expect 200 + valid response
- Record token counts for baseline performance
8. **Check OIDC auth endpoint**:
- Verify auth.sysloggh.net resolves to {{auth_host}}
- GET https://auth.sysloggh.net/ → expect login page
9. **Compile and report** — Determine overall_status from individual check results
3. **Check LiteLLM health (no-auth)**:
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
4. **Check backend container health**:
- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
5. **Check router roster loaded**:
- GET http://{{backend_host}}:9000/health → expect 200
- GET http://{{backend_host}}:9000/health/unified → expect 3 models
- If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
6. **Check GPU fleet health** (via fleet dashboard):
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
- Verify GPUs reporting status "healthy"
- Check alerts array for active warnings/critical
7. **Check model inference via LiteLLM** — Test each model:
- POST /v1/chat/completions model=gemma-4-12b → expect 200
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
- Use master key for auth
8. **Check agent keys**:
- GET /key/list with master key → verify all 6 agents have keys
9. **Check Grafana**:
- GET {{grafana_url}}/api/health → expect 200
10. **Compile and report** — Determine overall_status from individual check results