prose-contracts/litellm-self-heal.prose.md

kind, name, description

kind	name	description
responsibility	litellm-self-heal	Standing responsibility that monitors LiteLLM health and proactively fixes common issues. Runs on a loop — detect drift, fix, verify, report. Escalates to the user if a fix fails or an unknown issue is detected.

Maintains

• litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" }
• litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" }
• litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" }
• litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" }

Requires

• litellm-health:function — The health check contract to run before attempting fixes

Continuity

• Self-driven: check every 300 seconds (5 min)
• Also wakes on user request (prose run litellm-self-heal)
• On failure: re-check after 30 seconds, then escalate after 3 consecutive failures

Remediation Rules

Rule 1: Container Not Healthy

• Detect: harness-litellm or harness-nginx status is not "healthy"
• Fix: docker compose -f /opt/inference-harness/docker-compose.yml up -d <container>
• Verify: Re-run health check after 15 seconds
• Escalate: If still unhealthy after 3 attempts, report to user

Rule 2: Admin UI Returns Non-200

• Detect: GET /ui/ returns anything other than 200
• Fix: Restart harness-litellm container
• Verify: Re-check /ui/ after 20 seconds
• Escalate: If still failing after 2 restarts, report to user

Rule 3: Auth Endpoint Unreachable

• Detect: Cannot reach auth.sysloggh.net or 192.168.68.11:443
• Fix: Cannot auto-fix (Authentik is external). Report to user with diagnostic info.
• Escalate: Immediately (no retry — external dependency)

Rule 4: OIDC Login Failing

• Detect: OIDC redirect returns error page instead of login
• Fix: Check if GENERIC_* env vars are present in LiteLLM container
• Verify: If env vars missing, add them and restart
• Escalate: If present but still failing, report to user with config dump

Rule 5: Docs Endpoint 404

• Detect: GET /docs returns non-200
• Fix: Check DOCS_URL env var — should be /docs (not /litellm/docs or missing)
• Verify: Fix env var and restart container

Execution

1. Load parameters — Use defaults or provided overrides
2. Run health check — Execute litellm-health.prose.md to get baseline
3. Evaluate results — Compare each check against expected status
4. Apply remediation — For each failed check, find matching rule and execute fix
5. Verify — Re-run health check after fix
6. Report — Compile: what was broken, what was fixed, what needs user attention
7. Loop — If self-driven, wait 300s and repeat from step 2

Returns

• run_id: string — Unique ID for this remediation cycle
• issues_found: number — Total issues detected
• issues_fixed: number — Issues auto-remediated
• issues_escalated: number — Issues requiring user action
• details: array of { issue: string, action: string, result: string }
• timestamp: string