From bb20637b32ede25dd19fb2971dc5aa3ac9e1c752 Mon Sep 17 00:00:00 2001 From: Abiba Date: Thu, 2 Jul 2026 20:35:44 +0000 Subject: [PATCH] fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring) - litellm-health: 'check-litellm-health' -> 'litellm-health' - infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring' Fixes 'prose run ' mismatch. No external references broken. --- infrastructure-monitoring.prose.md | 150 +++++++++++++++++++++++++++++ litellm-health.prose.md | 139 ++++++++++++++++---------- 2 files changed, 240 insertions(+), 49 deletions(-) create mode 100644 infrastructure-monitoring.prose.md diff --git a/infrastructure-monitoring.prose.md b/infrastructure-monitoring.prose.md new file mode 100644 index 0000000..d5cd1c6 --- /dev/null +++ b/infrastructure-monitoring.prose.md @@ -0,0 +1,150 @@ +--- +kind: function +name: infrastructure-monitoring +description: > + Deploys Prometheus + GPU exporters + Grafana to monitor the entire + inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics + from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics + via existing /metrics Prometheus endpoint. Replaces the deprecated + harness-dashboard with a production-grade monitoring stack. +version: 1.0.0 +--- + +## Architecture + +``` +GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo) + nvidia-exporter nvidia-exporter amdgpu-exporter + :9400 :9400 :9400 + │ │ │ + └─────────────────────┼─────────────────────┘ + ▼ + ┌──────────────────────────┐ + │ Prometheus │ + │ CT 116 :9090 │ + │ │ + │ Scrape targets: │ + │ • 192.168.68.8:9400 │ + │ • 192.168.68.110:9400 │ + │ • 192.168.68.15:9400 │ + │ • litellm:4000/metrics │ + └──────────┬───────────────┘ + │ + ┌──────────▼───────────────┐ + │ Grafana │ + │ CT 116 :3001 │ + │ │ + │ Preloaded dashboards: │ + │ • GPU Fleet Overview │ + │ • LiteLLM Proxy Stats │ + └──────────────────────────┘ +``` + +## Components + +### 1. NVIDIA GPU Exporter (hosts: .8, .110) +- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary) +- Listens on `:9400`, exposes `/metrics` in Prometheus format +- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed + +### 2. AMD GPU Exporter (host: .15) +- Custom exporter: Python script wrapping `amdgpu_top --json` +- Listens on `:9400`, exposes `/metrics` in Prometheus format +- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization +- Runs as systemd service for persistence + +### 3. Prometheus (CT 116) +- Container: `prom/prometheus:latest` +- Port: `9090` (internal Docker network) +- Scrape interval: 15s +- Config: `/opt/monitoring/prometheus.yml` +- Storage: Docker volume `prometheus-data` + +### 4. Grafana (CT 116) +- Container: `grafana/grafana:latest` +- Port: `3001` (mapped to host) +- Data source: Prometheus at `http://prometheus:9090` +- Provisioned dashboards for GPU fleet + LiteLLM +- Accessible at `http://192.168.68.116:3001` + +## Parameters + +- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"] +- gpu_amd_hosts: ["192.168.68.15"] +- monitoring_host: "192.168.68.116" +- prometheus_port: 9090 +- grafana_port: 3001 +- gpu_exporter_port: 9400 + +## Requires + +- SSH access to all GPU hosts for exporter deployment +- Docker on CT 116 for Prometheus + Grafana containers +- Python 3 on AMD host for custom exporter +- nvidia-smi on NVIDIA hosts + +## Maintains + +- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format +- Prometheus scrapes all targets every 15s +- Grafana dashboards show real-time GPU utilization, temp, VRAM, power +- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics +- Stack persists across reboots (systemd for exporters, Docker restart policy) + +## Execution + +### Phase 1: GPU Exporters + +**NVIDIA (.8 and .110)**: +1. Download `nvidia_gpu_exporter` binary +2. Create systemd service `nvidia-gpu-exporter.service` +3. Start and enable + +**AMD (.15)**: +1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py` +2. Parses `amdgpu_top --json -d 1000` output +3. Exposes key metrics at `:9400/metrics` via Python http.server +4. Create systemd service +5. Start and enable + +### Phase 2: Prometheus + +1. Create `/opt/monitoring/` directory on CT 116 +2. Write `prometheus.yml` with scrape configs for all targets +3. Add to docker-compose (or separate compose file) +4. Start container + +### Phase 3: Grafana + +1. Create `/opt/monitoring/grafana/` directories +2. Provision Prometheus datasource +3. Provision GPU fleet dashboard JSON +4. Provision LiteLLM dashboard JSON +5. Add to docker-compose +6. Start container + +### Phase 4: Verification + +1. Verify all 3 GPU exporters return 200 at :9400/metrics +2. Verify Prometheus targets all UP at :9090/targets +3. Verify Grafana accessible at :3001 with dashboards +4. Verify LiteLLM metrics flowing to Prometheus +5. Update nginx to proxy `/monitoring/` → Grafana (optional) + +## Verification Commands + +```bash +# GPU exporters +curl -s http://192.168.68.8:9400/metrics | grep nvidia +curl -s http://192.168.68.110:9400/metrics | grep nvidia +curl -s http://192.168.68.15:9400/metrics | grep amdgpu + +# Prometheus +curl -s http://192.168.68.116:9090/api/v1/targets + +# Grafana +curl -s http://192.168.68.116:3001/api/health + +# LiteLLM metrics (already live) +curl -s http://192.168.68.116:4001/metrics | head -20 +``` diff --git a/litellm-health.prose.md b/litellm-health.prose.md index d45c08a..84c1ed3 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -1,18 +1,44 @@ --- kind: function -name: check-litellm-health +name: litellm-health description: > - Verifies LiteLLM deployment is healthy by checking admin UI, API docs, - OIDC auth endpoint, container status, and aggregate health on the - backend host. Designed as a reusable contract for any Syslog agent. + Verifies the LiteLLM inference stack health. Current architecture (2026-06-30): + nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server). + Router runs as internal-only backend behind LiteLLM. GPU monitoring via + Prometheus/Grafana and fleet dashboard. + Designed as a reusable contract for any Syslog agent. --- +## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU) + +``` +Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server) + │ │ + Key validation Model routing + Fallback chains Slot booking + Complexity router Circuit breakers + Budget tracking Redis-backed + │ │ + Prometheus ← metrics ←───┘ + │ + Grafana :3001 +``` + +**What changed (v3.1.0 → v3.2.0)**: +- Router is BACK — runs internally on :9000 behind LiteLLM +- nginx routes /v1/ and /admin/ → LiteLLM (not router directly) +- Router :9000 is 127.0.0.1-only, not publicly accessible +- GPU fleet dashboard: http://192.168.68.24:9100 +- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus) + ## Parameters - public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net") -- backend_host: string — Internal CT host to check containers (default: "192.168.68.116") +- backend_host: string — Internal CT host for container checks (default: "192.168.68.116") - auth_host: string — Authentik server for OIDC (default: "192.168.68.11") -- gpu_hosts: array — GPU inference hosts to check (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"]) +- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"]) +- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100") +- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001") ## Returns @@ -24,60 +50,75 @@ description: > ## Requires - SSH key access to backend_host for container checks -- Network access to public_url and auth_host -- curl and openssl available on the execution host - -## Ensures - -- Each check returns a clear pass/fail status with detail message -- If any endpoint returns non-200, overall_status is "degraded" -- If backend host unreachable or >2 containers down, overall_status is "down" -- If /health/unified reports any non-healthy components, status reflects it -- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy +- Network access to public_url, auth_host, and gpu_dashboard_url +- LiteLLM master key for key management endpoints ## GPU Fleet Topology | Host | IP | Hardware | Models Served | Engine | |------|-----|----------|---------------|--------| -| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | gemma-4-12b (Dense) | llama-server Docker | -| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | qwen3.6-27B-code (MoE), all Light tier | llama-server Docker | -| amdpve | 192.168.68.15 | AMD Strix Halo (CPU-only, -ngl 0) | ornith-1.0-35b (35B) | llama-server bare-metal | +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker | +| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal | -## Model Routing (LiteLLM → Router → GPU) +## Model Fallback Chains (LiteLLM) -| Model Request | Router Routes To | -|--------------|-----------------| -| `gemma-4-12b` | `GPU_DENSE_URL` → 192.168.68.8:8080 | -| `qwen3.6-27B-code` | `GPU_MOE_URL` → 192.168.68.110:8080 | -| `ornith-1.0-35b` | Router routes to 192.168.68.15:8080 | -| `syslog-auto` | Auto-routed by router tier logic | +| Primary | Timeout | Fallback | Timeout | +|---------|---------|----------|---------| +| qwen3.6-27B-code | 45s | gemma-4-12b | 30s | +| gemma-4-12b | 30s | qwen3.6-27B-code | 45s | +| ornith-1.0-35b | 120s | qwen → gemma | — | +| syslog-auto (balanced) | — | qwen → gemma | — | + +## Containers on CT 116 + +| Container | Image | Port | Health Check | +|-----------|-------|------|-------------| +| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness | +| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health | +| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health | +| harness-postgres | postgres:16-alpine | :5432 | pg_isready | +| harness-redis | redis:7-alpine | :6379 | PING | +| harness-dashboard | inference-harness-dashboard | :3000 | /health | +| harness-grafana | grafana/grafana | :3000→:3001 | /api/health | +| harness-prometheus | prom/prometheus | :9090 | /-/healthy | ## Execution 1. **Read parameters** — Use provided values or defaults + 2. **Check public endpoints**: - GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard") - GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI") - - GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths) - - GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc") -3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly): - - GET http://{{backend_host}}/litellm/ui/ → expect 200 - - GET http://{{backend_host}}/litellm/docs → expect 200 -4. **Check aggregate health endpoint**: - - GET {{backend_host}}:9000/health/unified → expect 200, check all components -5. **Check backend container health**: - - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy - - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard -6. **Check GPU fleet health**: - - For each {{gpu_hosts}}: - - SSH or nvidia-smi check → verify GPU is reachable - - Check VRAM usage (warn if >90%) - - Check temperature (warn if >80°C) - - Verify model responds to a small test prompt -7. **Check model inference** — Send test prompt to each model via LiteLLM API: - - POST /v1/chat/completions with model name → expect 200 + valid response - - Record token counts for baseline performance -8. **Check OIDC auth endpoint**: - - Verify auth.sysloggh.net resolves to {{auth_host}} - - GET https://auth.sysloggh.net/ → expect login page -9. **Compile and report** — Determine overall_status from individual check results + +3. **Check LiteLLM health (no-auth)**: + - GET http://{{backend_host}}/litellm/health/liveliness → expect 200 + +4. **Check backend container health**: + - SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy + - Critical: harness-litellm, harness-router, harness-nginx, harness-postgres + - Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus + +5. **Check router roster loaded**: + - GET http://{{backend_host}}:9000/health → expect 200 + - GET http://{{backend_host}}:9000/health/unified → expect 3 models + - If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload + +6. **Check GPU fleet health** (via fleet dashboard): + - GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON + - Verify GPUs reporting status "healthy" + - Check alerts array for active warnings/critical + +7. **Check model inference via LiteLLM** — Test each model: + - POST /v1/chat/completions model=gemma-4-12b → expect 200 + - POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 + - POST /v1/chat/completions model=ornith-1.0-35b → expect 200 + - Use master key for auth + +8. **Check agent keys**: + - GET /key/list with master key → verify all 6 agents have keys + +9. **Check Grafana**: + - GET {{grafana_url}}/api/health → expect 200 + +10. **Compile and report** — Determine overall_status from individual check results