no-mistakes(document): Make litellm-health the live owner; dedupe probes

litellm-self-heal.prose.md

@@ -11,22 +11,22 @@ note: >
   Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
   Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
   GPU monitoring integrated from gpu-monitor on .24:9100.
-  
-  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
-  duplication of architecture diagrams, GPU topology, timeout tables, and container
-  lists. Health check is now § Health Check within this contract.
-  
+
+  Health probes are owned by litellm-health.prose.md (dispatched as
+  `run contract: litellm-health`). This contract owns remediation only — it does not
+  re-specify the probes.
+
   Source of truth for GPU topology and keys: gpu-fleet.prose.md
   Last verified: 2026-07-12
 description: >
-  LiteLLM inference stack health monitoring + self-healing. Verifies the full
-  nginx → LiteLLM → GPU chain, 12 containers on CT 116, 3 GPU hosts, model
-  inference, and agent keys. Applies remediation rules for common failures.
+  LiteLLM inference stack remediation. Applies remediation rules for failures detected
+  by litellm-health.prose.md (nginx → LiteLLM → GPU chain, CT 116 containers, GPU hosts,
+  model inference, and agent keys).
   Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
 ---
 ---
 
-# LiteLLM Operations — Health Check + Self-Heal
+# LiteLLM Operations — Self-Heal (Remediation)
 
 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
 
@@ -138,55 +138,7 @@ contracts — read them there. Do not re-add retired names (`gemma-4-12b`, `gpu-
 
 ## Health Check
 
-Run this first on every cycle. Results feed into remediation rules below.
-
-### 1. Check the end-user surfaces
-The public edge and the backend edge serve the same app under different paths; they are not
-interchangeable, so every probe names the surface it targets.
-
-Public edge — `{{public_url}}` serves the app at the ROOT (the `/litellm/` prefix 404s):
-- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
-- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
-- GET {{public_url}}/litellm/ui/ and {{public_url}}/litellm/docs → expect 404 (not served on this edge)
-
-Backend edge — `http://{{backend_host}}` (port 80) serves the app UNDER `/litellm/`:
-- GET http://{{backend_host}}/litellm/ui/ → expect 200 ("LiteLLM Dashboard")
-- GET http://{{backend_host}}/litellm/docs → expect 200 ("LiteLLM API - Swagger UI")
-- GET http://{{backend_host}}/ui/ and /docs → expect 301 → /litellm/... → 200 (one-hop redirect helpers, added 2026-09-11)
-
-### 2. Check LiteLLM health (no-auth)
-- GET http://{{backend_host}}/litellm/health/liveliness → expect 200
-
-### 3. Check backend container health
-- SSH to {{backend_host}} → `docker ps` → verify 12 containers healthy (11 harness + trove-agent-docker, added 2026-09-11)
-- Critical: harness-litellm, harness-nginx, harness-postgres
-- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus,
-  harness-alertmanager, harness-zulip-bridge, harness-docker-stats, harness-pve-exporter,
-  trove-agent-docker (ghcr.io/techdox/trove-agent-docker, added 2026-09-11)
-- Decommissioned 2026-09-11: harness-router (container, image and config removed)
-
-### 4. Check GPU fleet health (via fleet dashboard)
-- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
-- Verify GPUs reporting status "healthy"
-- Check alerts array for active warnings/critical
-
-### 5. Check model inference via LiteLLM — test one model per GPU host (backend surface)
-- POST http://{{backend_host}}/litellm/v1/chat/completions model=qwen3.6-27B-code → expect 200 (RTX 3090, .8)
-- POST http://{{backend_host}}/litellm/v1/chat/completions model=gpu-vision → expect 200 (RTX 5070, .110)
-- POST http://{{backend_host}}/litellm/v1/chat/completions model=strix-moe → expect 200 (Strix Halo, .15)
-- Auth uses the dedicated `monitor` agent key, read on CT 116 from `/etc/litellm-monitor.env` (root-only 0600) — never the master key
-
-### 6. Check agent keys
-- GET http://{{backend_host}}/litellm/key/list with master key (admin endpoint) → verify all 6 agents have keys
-
-### 7. Check Grafana
-- GET {{grafana_url}}/api/health → expect 200
-
-### 8. Compile overall status
-Determine overall_status from individual check results:
-- "healthy" — all checks pass
-- "degraded" — 1-2 non-critical checks fail
-- "down" — critical checks fail
+Health probes are owned by `litellm-health.prose.md` (dispatched as `run contract: litellm-health`). This contract owns remediation only — it does not re-specify the probes.
 
 ---
 ---