feat: litellm-self-heal — proactive remediation responsibility

litellm-self-heal.prose.md

@@ -0,0 +1,74 @@
+---
+kind: responsibility
+name: litellm-self-heal
+description: >
+  Standing responsibility that monitors LiteLLM health and proactively
+  fixes common issues. Runs on a loop — detect drift, fix, verify, report.
+  Escalates to the user if a fix fails or an unknown issue is detected.
+---
+
+## Maintains
+
+- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" }
+- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" }
+- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" }
+- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" }
+
+## Requires
+
+- litellm-health:function — The health check contract to run before attempting fixes
+
+## Continuity
+
+- Self-driven: check every 300 seconds (5 min)
+- Also wakes on user request (`prose run litellm-self-heal`)
+- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures
+
+## Remediation Rules
+
+### Rule 1: Container Not Healthy
+- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy"
+- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d <container>`
+- **Verify**: Re-run health check after 15 seconds
+- **Escalate**: If still unhealthy after 3 attempts, report to user
+
+### Rule 2: Admin UI Returns Non-200
+- **Detect**: GET /ui/ returns anything other than 200
+- **Fix**: Restart harness-litellm container
+- **Verify**: Re-check /ui/ after 20 seconds
+- **Escalate**: If still failing after 2 restarts, report to user
+
+### Rule 3: Auth Endpoint Unreachable
+- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443
+- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info.
+- **Escalate**: Immediately (no retry — external dependency)
+
+### Rule 4: OIDC Login Failing
+- **Detect**: OIDC redirect returns error page instead of login
+- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container
+- **Verify**: If env vars missing, add them and restart
+- **Escalate**: If present but still failing, report to user with config dump
+
+### Rule 5: Docs Endpoint 404
+- **Detect**: GET /docs returns non-200
+- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing)
+- **Verify**: Fix env var and restart container
+
+## Execution
+
+1. **Load parameters** — Use defaults or provided overrides
+2. **Run health check** — Execute `litellm-health.prose.md` to get baseline
+3. **Evaluate results** — Compare each check against expected status
+4. **Apply remediation** — For each failed check, find matching rule and execute fix
+5. **Verify** — Re-run health check after fix
+6. **Report** — Compile: what was broken, what was fixed, what needs user attention
+7. **Loop** — If self-driven, wait 300s and repeat from step 2
+
+## Returns
+
+- run_id: string — Unique ID for this remediation cycle
+- issues_found: number — Total issues detected
+- issues_fixed: number — Issues auto-remediated
+- issues_escalated: number — Issues requiring user action
+- details: array of { issue: string, action: string, result: string }
+- timestamp: string