SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: align contract name fields to filenames (litellm-health, infrastructure-monitoring)

Browse Source 
- litellm-health: 'check-litellm-health' -> 'litellm-health'
- infrastructure-monitoring: 'deploy-monitoring-stack' -> 'infrastructure-monitoring'
Fixes 'prose run <filename>' mismatch. No external references broken.
This commit is contained in:
Abiba
2026-07-02 20:35:44 +00:00
parent b15771bfd9
commit bb20637b32
2 changed files with 240 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infrastructure-monitoring.prose.md
+150
View File
@@ -0,0 +1,150 @@
---
kind: function
name: infrastructure-monitoring
description: >
Deploys Prometheus + GPU exporters + Grafana to monitor the entire
inference fleet (3 GPU hosts + LiteLLM) from CT 116. GPU metrics
from nvidia-smi (.8, .110) and amdgpu_top (.15). LiteLLM metrics
via existing /metrics Prometheus endpoint. Replaces the deprecated
harness-dashboard with a production-grade monitoring stack.
version: 1.0.0
---
## Architecture
```
GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
nvidia-exporter nvidia-exporter amdgpu-exporter
:9400 :9400 :9400
│ │ │
└─────────────────────┼─────────────────────┘
▼
┌──────────────────────────┐
│ Prometheus │
│ CT 116 :9090 │
│ │
│ Scrape targets: │
│ • 192.168.68.8:9400 │
│ • 192.168.68.110:9400 │
│ • 192.168.68.15:9400 │
│ • litellm:4000/metrics │
└──────────┬───────────────┘
│
┌──────────▼───────────────┐
│ Grafana │
│ CT 116 :3001 │
│ │
│ Preloaded dashboards: │
│ • GPU Fleet Overview │
│ • LiteLLM Proxy Stats │
└──────────────────────────┘
```
## Components
### 1. NVIDIA GPU Exporter (hosts: .8, .110)
- Tool: `utkuozdemir/nvidia_gpu_exporter` (Go binary, single static binary)
- Listens on `:9400`, exposes `/metrics` in Prometheus format
- Metrics: utilization, temp, VRAM, power, clock speeds, fan speed
### 2. AMD GPU Exporter (host: .15)
- Custom exporter: Python script wrapping `amdgpu_top --json`
- Listens on `:9400`, exposes `/metrics` in Prometheus format
- Metrics: power (W), temp (°C), VRAM used/total, GFX clock, utilization
- Runs as systemd service for persistence
### 3. Prometheus (CT 116)
- Container: `prom/prometheus:latest`
- Port: `9090` (internal Docker network)
- Scrape interval: 15s
- Config: `/opt/monitoring/prometheus.yml`
- Storage: Docker volume `prometheus-data`
### 4. Grafana (CT 116)
- Container: `grafana/grafana:latest`
- Port: `3001` (mapped to host)
- Data source: Prometheus at `http://prometheus:9090`
- Provisioned dashboards for GPU fleet + LiteLLM
- Accessible at `http://192.168.68.116:3001`
## Parameters
- gpu_nvidia_hosts: ["192.168.68.8", "192.168.68.110"]
- gpu_amd_hosts: ["192.168.68.15"]
- monitoring_host: "192.168.68.116"
- prometheus_port: 9090
- grafana_port: 3001
- gpu_exporter_port: 9400
## Requires
- SSH access to all GPU hosts for exporter deployment
- Docker on CT 116 for Prometheus + Grafana containers
- Python 3 on AMD host for custom exporter
- nvidia-smi on NVIDIA hosts
## Maintains
- All 3 GPU hosts export metrics at :9400/metrics in Prometheus format
- Prometheus scrapes all targets every 15s
- Grafana dashboards show real-time GPU utilization, temp, VRAM, power
- LiteLLM metrics (requests, tokens, latency, errors) visible alongside GPU metrics
- Stack persists across reboots (systemd for exporters, Docker restart policy)
## Execution
### Phase 1: GPU Exporters
**NVIDIA (.8 and .110)**:
1. Download `nvidia_gpu_exporter` binary
2. Create systemd service `nvidia-gpu-exporter.service`
3. Start and enable
**AMD (.15)**:
1. Create Python exporter script at `/opt/amdgpu-exporter/exporter.py`
2. Parses `amdgpu_top --json -d 1000` output
3. Exposes key metrics at `:9400/metrics` via Python http.server
4. Create systemd service
5. Start and enable
### Phase 2: Prometheus
1. Create `/opt/monitoring/` directory on CT 116
2. Write `prometheus.yml` with scrape configs for all targets
3. Add to docker-compose (or separate compose file)
4. Start container
### Phase 3: Grafana
1. Create `/opt/monitoring/grafana/` directories
2. Provision Prometheus datasource
3. Provision GPU fleet dashboard JSON
4. Provision LiteLLM dashboard JSON
5. Add to docker-compose
6. Start container
### Phase 4: Verification
1. Verify all 3 GPU exporters return 200 at :9400/metrics
2. Verify Prometheus targets all UP at :9090/targets
3. Verify Grafana accessible at :3001 with dashboards
4. Verify LiteLLM metrics flowing to Prometheus
5. Update nginx to proxy `/monitoring/` → Grafana (optional)
## Verification Commands
```bash
# GPU exporters
curl -s http://192.168.68.8:9400/metrics | grep nvidia
curl -s http://192.168.68.110:9400/metrics | grep nvidia
curl -s http://192.168.68.15:9400/metrics | grep amdgpu
# Prometheus
curl -s http://192.168.68.116:9090/api/v1/targets
# Grafana
curl -s http://192.168.68.116:3001/api/health
# LiteLLM metrics (already live)
curl -s http://192.168.68.116:4001/metrics | head -20
```
litellm-health.prose.md
+90 -49
View File
@@ -1,18 +1,44 @@
---
kind: function
name: check-litellm-health
name: litellm-health
description: >
Verifies LiteLLM deployment is healthy by checking admin UI, API docs,
OIDC auth endpoint, container status, and aggregate health on the
backend host. Designed as a reusable contract for any Syslog agent.
Verifies the LiteLLM inference stack health. Current architecture (2026-06-30):
nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server).
Router runs as internal-only backend behind LiteLLM. GPU monitoring via
Prometheus/Grafana and fleet dashboard.
Designed as a reusable contract for any Syslog agent.
---
## Architecture (v3.2.0 — Layered: nginx → LiteLLM → Router → GPU)
```
Request → nginx:80 → LiteLLM:4000 → Router:9000(internal) → GPU(llama-server)
│ │
Key validation Model routing
Fallback chains Slot booking
Complexity router Circuit breakers
Budget tracking Redis-backed
│ │
Prometheus ← metrics ←───┘
│
Grafana :3001
```
**What changed (v3.1.0 → v3.2.0)**:
- Router is BACK — runs internally on :9000 behind LiteLLM
- nginx routes /v1/ and /admin/ → LiteLLM (not router directly)
- Router :9000 is 127.0.0.1-only, not publicly accessible
- GPU fleet dashboard: http://192.168.68.24:9100
- Grafana: http://192.168.68.116:3001 (GPU dashboards via Prometheus)
## Parameters
- public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
- backend_host: string — Internal CT host for container checks (default: "192.168.68.116")
- auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
- gpu_hosts: array — GPU inference hosts to check (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")
## Returns
@@ -24,60 +50,75 @@ description: >
## Requires
- SSH key access to backend_host for container checks
- Network access to public_url and auth_host
- curl and openssl available on the execution host
## Ensures
- Each check returns a clear pass/fail status with detail message
- If any endpoint returns non-200, overall_status is "degraded"
- If backend host unreachable or >2 containers down, overall_status is "down"
- If /health/unified reports any non-healthy components, status reflects it
- Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
- Network access to public_url, auth_host, and gpu_dashboard_url
- LiteLLM master key for key management endpoints
## GPU Fleet Topology
| Host | IP | Hardware | Models Served | Engine |
|------|-----|----------|---------------|--------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | gemma-4-12b (Dense) | llama-server Docker |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | qwen3.6-27B-code (MoE), all Light tier | llama-server Docker |
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU-only, -ngl 0) | ornith-1.0-35b (35B) | llama-server bare-metal |
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server Docker |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server Docker |
| amdpve | 192.168.68.15 | AMD Strix Halo (CPU) | ornith-1.0-35b (35B) | llama-server bare-metal |
## Model Routing (LiteLLM → Router → GPU)
## Model Fallback Chains (LiteLLM)
| Model Request | Router Routes To |
|--------------|-----------------|
| `gemma-4-12b` | `GPU_DENSE_URL` → 192.168.68.8:8080 |
| `qwen3.6-27B-code` | `GPU_MOE_URL` → 192.168.68.110:8080 |
| `ornith-1.0-35b` | Router routes to 192.168.68.15:8080 |
| `syslog-auto` | Auto-routed by router tier logic |
| Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------|
| qwen3.6-27B-code | 45s | gemma-4-12b | 30s |
| gemma-4-12b | 30s | qwen3.6-27B-code | 45s |
| ornith-1.0-35b | 120s | qwen → gemma | — |
| syslog-auto (balanced) | — | qwen → gemma | — |
## Containers on CT 116
| Container | Image | Port | Health Check |
|-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING |
| harness-dashboard | inference-harness-dashboard | :3000 | /health |
| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
## Execution
1. **Read parameters** — Use provided values or defaults
2. **Check public endpoints**:
- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
- GET {{public_url}}/openapi.json → expect 200 (valid JSON, 497 paths)
- GET {{public_url}}/redoc → expect 200 ("LiteLLM API - ReDoc")
3. **Check nginx-proxied endpoints** (internal only — Netbird routes to LiteLLM directly):
- GET http://{{backend_host}}/litellm/ui/ → expect 200
- GET http://{{backend_host}}/litellm/docs → expect 200
4. **Check aggregate health endpoint**:
- GET {{backend_host}}:9000/health/unified → expect 200, check all components
5. **Check backend container health**:
- SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
- Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
6. **Check GPU fleet health**:
- For each {{gpu_hosts}}:
- SSH or nvidia-smi check → verify GPU is reachable
- Check VRAM usage (warn if >90%)
- Check temperature (warn if >80°C)
- Verify model responds to a small test prompt
7. **Check model inference** — Send test prompt to each model via LiteLLM API:
- POST /v1/chat/completions with model name → expect 200 + valid response
- Record token counts for baseline performance
8. **Check OIDC auth endpoint**:
- Verify auth.sysloggh.net resolves to {{auth_host}}
- GET https://auth.sysloggh.net/ → expect login page
9. **Compile and report** — Determine overall_status from individual check results
3. **Check LiteLLM health (no-auth)**:
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
4. **Check backend container health**:
- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy
- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus
5. **Check router roster loaded**:
- GET http://{{backend_host}}:9000/health → expect 200
- GET http://{{backend_host}}:9000/health/unified → expect 3 models
- If router returns "all GPUs saturated" but GPUs idle: roster not loaded → reload
6. **Check GPU fleet health** (via fleet dashboard):
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
- Verify GPUs reporting status "healthy"
- Check alerts array for active warnings/critical
7. **Check model inference via LiteLLM** — Test each model:
- POST /v1/chat/completions model=gemma-4-12b → expect 200
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
- POST /v1/chat/completions model=ornith-1.0-35b → expect 200
- Use master key for auth
8. **Check agent keys**:
- GET /key/list with master key → verify all 6 agents have keys
9. **Check Grafana**:
- GET {{grafana_url}}/api/health → expect 200
10. **Compile and report** — Determine overall_status from individual check results