From 053d30ce7cefb075d6062d38fa3faa97ee9aa90d Mon Sep 17 00:00:00 2001 From: root Date: Thu, 9 Jul 2026 06:01:48 +0000 Subject: [PATCH] feat: consolidate litellm-health into litellm-self-heal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - litellm-self-heal: merged health check as §Health Check section, includes full architecture diagram (v4.0.0), GPU topology, timeout tables, container list, and execution flow. Eliminates duplication between the two contracts. - litellm-health: marked DEPRECATED, retained for reference - README.md: updated with consolidated contract, deprecation notice, disk-gc-threat-response, and infrastructure-monitoring status --- README.md | 8 +- litellm-health.prose.md | 8 ++ litellm-self-heal.prose.md | 230 +++++++++++++++++++++++++++---------- 3 files changed, 183 insertions(+), 63 deletions(-) diff --git a/README.md b/README.md index e4b1fc0..647ecb9 100644 --- a/README.md +++ b/README.md @@ -115,10 +115,11 @@ Run on trigger or schedule. Maintain persistent world-model state across runs. | `zulip-health` | Zulip | Checks Zulip connectivity, message flow, and bot responsiveness. | | `zulip-mention-reliability` | Zulip | Diagnoses and fixes @mention detection issues in Zulip. | | `zulip-approval-fix` | Zulip | Fixes broken /approve and /deny slash commands for Hermes agents. | -| `litellm-self-heal` | LiteLLM | Standing responsibility — monitors LiteLLM health and auto-remediates: key rotation, container restart, config reload, roster fixes. | +| `litellm-self-heal` | LiteLLM | Consolidated health check + self-healing for the full nginx → LiteLLM → GPU chain. Verifies 8 containers, 3 GPUs, model inference, and agent keys. Applies 9 remediation rules. (litellm-health merged into this contract 2026-07-09.) | | `gpu-fleet` | GPU | Manages the GPU inference fleet: model deployment, registration, health checks, LiteLLM sync. | | `gpu-monitor` | GPU | Comprehensive GPU fleet monitor — polls sidecars, router, LiteLLM every 15s, renders SSE dashboard. | | `proxmox-monitor` | Infra | Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. | +| `disk-gc-threat-response` | Infra | Fleet-wide disk health scan + garbage collection across 15 CTs + 3 GPU hosts + docker-vm KVM VM. 5-tier threat levels with automated GC and Zulip alerting. Two incidents resolved: kagentz (35.67GB) and amdpve (11.56GB). | | `pm2-self-heal` | Ops | Monitors PM2 processes (abiba-zulip, abiba-telegram) and auto-restarts any that are stopped or errored. | ### Instantiable Templates (`kind: pattern` — run with `prose run`) @@ -146,8 +147,9 @@ Called on-demand as single-render tools. | Contract | Description | |---|---| -| `litellm-health` | Verifies LiteLLM inference stack health: containers, router, GPU fleet, model inference, agent keys. | -| `infrastructure-monitoring` | Deploys Prometheus + GPU exporters + Grafana to monitor the inference fleet from CT 116. | +| `litellm-api-keys` | Manages LiteLLM API keys for agent identity. Create, rotate, verify, and list agent keys. References gpu-fleet for current key inventory. | +| `litellm-health` | ⚠️ **DEPRECATED** — consolidated into `litellm-self-heal` (2026-07-09). Retained for reference only. | +| `infrastructure-monitoring` | Target-state for Prometheus + GPU exporters + Grafana. Core stack deployed, GPU exporters NOT live. | | `stirling-pdf-agent-access` | Documents the Stirling-PDF API access pattern for agents — global API key, 12 operations, curl examples. Agents use the `stirling-pdf-api` shared skill for templates. | | `hello-world` | Minimal test contract — verifies the OpenProse execution pipeline works. | diff --git a/litellm-health.prose.md b/litellm-health.prose.md index 88f0e6e..6adbb61 100644 --- a/litellm-health.prose.md +++ b/litellm-health.prose.md @@ -1,6 +1,14 @@ --- kind: function name: litellm-health +status: deprecated +deprecated_on: 2026-07-09 +replaced_by: litellm-self-heal.prose.md +note: > + Consolidated into litellm-self-heal.prose.md to eliminate duplication + of architecture diagrams, GPU topology, timeout tables, and container + lists. Health check is now § Health Check within litellm-self-heal. + This file is retained for reference only — use litellm-self-heal instead. description: > Verifies the LiteLLM inference stack health. Current architecture (2026-07-09): nginx:80 → LiteLLM:4000 → GPU(llama-server) via direct proxy. diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index cc470d2..26a5309 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -7,24 +7,89 @@ note: > This contract is now manual-only — triggers require explicit user request. Consider reimplementing as a standalone cron job or prose contract. - Source of truth for GPU topology: gpu-fleet.prose.md + Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate + duplication of architecture diagrams, GPU topology, timeout tables, and container + lists. Health check is now § Health Check within this contract. + + Source of truth for GPU topology and keys: gpu-fleet.prose.md Last verified: 2026-07-09 description: > - Standing responsibility that monitors LiteLLM health and proactively - fixes common issues. Reports every action via Zulip DM and RA-H OS - knowledge graph for full audit trail. + LiteLLM inference stack health monitoring + self-healing. Verifies the full + nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model + inference, and agent keys. Applies remediation rules for common failures. + Reports every action via Zulip DM and RA-H OS knowledge graph. --- +# LiteLLM Operations — Health Check + Self-Heal + +## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU) + +``` +Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) + │ + Key validation + Fallback chains + Budget tracking + │ + Prometheus ← metrics + │ + Grafana :3001 + + harness-router :9000 — DEPRECATED, container still runs but + NOT in request path. nginx routes /v1 → LiteLLM directly. + Router slot booking + circuit breakers replaced by + LiteLLM native fallbacks + timeouts. +``` + +## Parameters + +- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net") +- backend_host: string — Internal CT host (default: "192.168.68.116") +- auth_host: string — Authentik OIDC server (default: "192.168.68.11") +- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"]) +- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100") +- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001") + +## GPU Fleet Topology + +| Host | IP | Hardware | Models Served | Engine | Context | Parallel | +|------|-----|----------|---------------|--------|---------|----------| +| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd | 128K | 2 | +| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd | 128K | 2 | +| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | ornith-1.0-35b | llama-server systemd (Vulkan) | 256K | 2 | + +## Model Fallback Chains (LiteLLM) + +| Primary | Timeout | Fallback | Timeout | +|---------|---------|----------|---------| +| qwen3.6-27B-code | 90s | gemma-4-12b | 120s | +| gemma-4-12b | 120s | qwen3.6-27B-code | 90s | +| ornith-1.0-35b | 120s | qwen → gemma | — | +| syslog-auto (balanced) | 90s | qwen → gemma | — | + +> Global: request_timeout=300s, nginx proxy_read_timeout=600s + +## Containers on CT 116 + +| Container | Image | Port | Health Check | +|-----------|-------|------|-------------| +| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness | +| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) | +| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health | +| harness-postgres | postgres:16-alpine | :5432 | pg_isready | +| harness-redis | redis:7-alpine | :6379 | PING | +| harness-dashboard | inference-harness-dashboard | :3000 | /health | +| harness-grafana | grafana/grafana | :3000→:3001 | /api/health | +| harness-prometheus | prom/prometheus | :9090 | /-/healthy | + ## Maintains - litellm-admin-ui: { status: "healthy", last_check: timestamp } - litellm-api-docs: { status: "healthy", last_check: timestamp } - litellm-containers: { status: "healthy", last_check: timestamp } - litellm-oidc: { status: "healthy", last_check: timestamp } - -## Requires - -- litellm-health:function +- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array } +- litellm-agent-keys: { count: int, valid: int } ## Continuity @@ -32,55 +97,48 @@ description: > - Also wakes on user request - On failure: re-check after 30s, escalate after 3 consecutive failures -## Reporting +--- -Every remediation cycle produces a structured report in three channels: +## Health Check -### 1. RA-H OS Knowledge Graph Node -Created as `[LEARN] litellm-self-heal: ` with: -- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" } -- source: Full JSON report of the cycle -- description: Summary of what was found, fixed, and escalated +Run this first on every cycle. Results feed into remediation rules below. -### 2. Zulip DM to Owner -Sent immediately for: -- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied" -- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention" -- Every 10th clean cycle — "✅ All Clear (10 checks passed)" +### 1. Check public endpoints +- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard") +- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI") -### 3. Daily Digest (end of day) -A summary of the last 24 hours sent as a single Zulip DM: -``` -📋 LiteLLM Self-Heal — Daily Digest (2026-06-26) +### 2. Check LiteLLM health (no-auth) +- GET http://{{backend_host}}/litellm/health/liveliness → expect 200 -Total cycles: 288 (every 5 min) -Issues found: 3 - ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1) - └─ Escalated: 0 +### 3. Check backend container health +- SSH to {{backend_host}} → `docker ps` → verify 8 containers healthy +- Critical: harness-litellm, harness-router, harness-nginx, harness-postgres +- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus -Top actions: - • harness-nginx restarted — 2026-06-26 02:15 - • DOCS_URL corrected — 2026-06-26 07:30 - • harness-nginx restarted — 2026-06-26 14:46 +### 4. Check GPU fleet health (via fleet dashboard) +- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON +- Verify GPUs reporting status "healthy" +- Check alerts array for active warnings/critical -Uptime: 23h 47m — All healthy now ✅ -``` +### 5. Check model inference via LiteLLM — test each model +- POST /v1/chat/completions model=gemma-4-12b → expect 200 +- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200 +- POST /v1/chat/completions model=ornith-1.0-35b → expect 200 +- Use master key for auth -### 4. Weekly Digest (every Monday) -Same format as daily but covering 7 days. Sent to both Zulip DM and -logged as a `[REPORT]` knowledge graph node for long-term trending. +### 6. Check agent keys +- GET /key/list with master key → verify all 6 agents have keys -### 5. Relay Message (if cross-agent) -If a fix required another agent's help (e.g., Authentik restart), a relay -message is sent to the responsible agent with full context. +### 7. Check Grafana +- GET {{grafana_url}}/api/health → expect 200 -## GPU Fleet +### 8. Compile overall status +Determine overall_status from individual check results: +- "healthy" — all checks pass +- "degraded" — 1-2 non-critical checks fail +- "down" — critical checks fail -| Host | IP | Hardware | Models | Engine | Context | -|------|-----|----------|--------|--------|---------| -| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b | llama-server :8080 (Vulkan) | 256K | -| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 (systemd) | 128K | -| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 (systemd) | 128K | +--- ## Remediation Rules @@ -115,30 +173,82 @@ If 503 errors occur, check LiteLLM timeouts and GPU health directly. ### Rule 8: Agent Keys Invalid (401) Detect → agents report auth errors -Root cause → keys in router env var, not in LiteLLM DB +Root cause → keys not in LiteLLM DB Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts Escalate → if SSH access unavailable, send Zulip DM -### Rule 9: Stale Active Counter in Redis -Detect → active: > 0 while GPU idle -Fix → `redis-cli DEL active:` → restart router -Prevent → add TTL to active keys in router code +### Rule 9: Stale Active Counter in Redis — DEPRECATED +Router no longer in path so Redis active counters are unused. Rule retained +for reference but inactive. If Redis issues occur, check harness-redis container. + +--- + +## Reporting + +Every remediation cycle produces a structured report: + +### 1. RA-H OS Knowledge Graph Node +Created as `[LEARN] litellm-self-heal: ` with full JSON report. + +### 2. Zulip DM to Owner +- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied" +- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention" +- Every 10th clean cycle — "✅ All Clear (10 checks passed)" + +### 3. Daily Digest (end of day) +Summary of last 24 hours: total cycles, issues found/fixed/escalated, +top actions, uptime. + +### 4. Relay Message (if cross-agent) +If a fix requires another agent (e.g., Authentik restart), relay sent +to responsible agent with full context. + +--- ## Execution -1. Run health check -2. Apply remediation for each failure -3. **Generate report** with all actions taken -4. **Log to knowledge graph** — create `[LEARN]` node -5. **Notify user** via Zulip DM if anything changed -6. Wait 300s and repeat +```prose +-- Phase 1: Health Check +let health = call health-check + public_url: public_url + backend_host: backend_host + gpu_dashboard_url: gpu_dashboard_url + grafana_url: grafana_url + +-- Phase 2: Apply remediation for each failure +let actions = [] +for check in health.failed: + let fix = apply-remediation-rule + rule: lookup-rule(check.name) + target: check.target + push actions fix + +-- Phase 3: Generate report +call report-generator + health: health + actions: actions + +-- Phase 4: Log to knowledge graph +call kg-logger + run_id: run_id + health: health + actions: actions + +-- Phase 5: Notify if anything changed +if actions.length > 0: + call zulip-notifier + actions: actions + health: health + +-- Wait 300s and repeat +``` ## Audit Trail Format ```json { - "run_id": "self-heal-20260626-001", - "timestamp": "2026-06-26T14:00:00Z", + "run_id": "self-heal-20260709-001", + "timestamp": "2026-07-09T14:00:00Z", "duration_ms": 1234, "checks_passed": 7, "checks_failed": 0, @@ -152,7 +262,7 @@ Prevent → add TTL to active keys in router code With failures: ```json { - "run_id": "self-heal-20260626-002", + "run_id": "self-heal-20260709-002", "issues_found": 1, "issues_fixed": 1, "actions": [