---
kind: responsibility
name: litellm-self-heal
description: >
  Standing responsibility that monitors LiteLLM health and proactively
  fixes common issues. Runs on a loop — detect drift, fix, verify, report.
  Escalates to the user if a fix fails or an unknown issue is detected.
---

## Maintains

- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" }
- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" }
- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" }
- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" }

## Requires

- litellm-health:function — The health check contract to run before attempting fixes

## Continuity

- Self-driven: check every 300 seconds (5 min)
- Also wakes on user request (`prose run litellm-self-heal`)
- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures

## Remediation Rules

### Rule 1: Container Not Healthy
- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy"
- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d <container>`
- **Verify**: Re-run health check after 15 seconds
- **Escalate**: If still unhealthy after 3 attempts, report to user

### Rule 2: Admin UI Returns Non-200
- **Detect**: GET /ui/ returns anything other than 200
- **Fix**: Restart harness-litellm container
- **Verify**: Re-check /ui/ after 20 seconds
- **Escalate**: If still failing after 2 restarts, report to user

### Rule 3: Auth Endpoint Unreachable
- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443
- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info.
- **Escalate**: Immediately (no retry — external dependency)

### Rule 4: OIDC Login Failing
- **Detect**: OIDC redirect returns error page instead of login
- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container
- **Verify**: If env vars missing, add them and restart
- **Escalate**: If present but still failing, report to user with config dump

### Rule 5: Docs Endpoint 404
- **Detect**: GET /docs returns non-200
- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing)
- **Verify**: Fix env var and restart container

## Execution

1. **Load parameters** — Use defaults or provided overrides
2. **Run health check** — Execute `litellm-health.prose.md` to get baseline
3. **Evaluate results** — Compare each check against expected status
4. **Apply remediation** — For each failed check, find matching rule and execute fix
5. **Verify** — Re-run health check after fix
6. **Report** — Compile: what was broken, what was fixed, what needs user attention
7. **Loop** — If self-driven, wait 300s and repeat from step 2

## Returns

- run_id: string — Unique ID for this remediation cycle
- issues_found: number — Total issues detected
- issues_fixed: number — Issues auto-remediated
- issues_escalated: number — Issues requiring user action
- details: array of { issue: string, action: string, result: string }
- timestamp: string