diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 3f6b9d1..0c105e4 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -3,72 +3,124 @@ kind: responsibility name: litellm-self-heal description: > Standing responsibility that monitors LiteLLM health and proactively - fixes common issues. Runs on a loop — detect drift, fix, verify, report. - Escalates to the user if a fix fails or an unknown issue is detected. + fixes common issues. Reports every action via Zulip DM and RA-H OS + knowledge graph for full audit trail. --- ## Maintains -- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" } -- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" } -- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" } -- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" } +- litellm-admin-ui: { status: "healthy", last_check: timestamp } +- litellm-api-docs: { status: "healthy", last_check: timestamp } +- litellm-containers: { status: "healthy", last_check: timestamp } +- litellm-oidc: { status: "healthy", last_check: timestamp } ## Requires -- litellm-health:function — The health check contract to run before attempting fixes +- litellm-health:function ## Continuity -- Self-driven: check every 300 seconds (5 min) -- Also wakes on user request (`prose run litellm-self-heal`) -- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures +- Self-driven: check every 300 seconds +- Also wakes on user request +- On failure: re-check after 30s, escalate after 3 consecutive failures + +## Reporting + +Every remediation cycle produces a structured report in three channels: + +### 1. RA-H OS Knowledge Graph Node +Created as `[LEARN] litellm-self-heal: ` with: +- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" } +- source: Full JSON report of the cycle +- description: Summary of what was found, fixed, and escalated + +### 2. Zulip DM to Owner +Sent immediately for: +- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied" +- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention" +- Every 10th clean cycle — "✅ All Clear (10 checks passed)" + +### 3. Daily Digest (end of day) +A summary of the last 24 hours sent as a single Zulip DM: +``` +📋 LiteLLM Self-Heal — Daily Digest (2026-06-26) + +Total cycles: 288 (every 5 min) +Issues found: 3 + ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1) + └─ Escalated: 0 + +Top actions: + • harness-nginx restarted — 2026-06-26 02:15 + • DOCS_URL corrected — 2026-06-26 07:30 + • harness-nginx restarted — 2026-06-26 14:46 + +Uptime: 23h 47m — All healthy now ✅ +``` + +### 4. Weekly Digest (every Monday) +Same format as daily but covering 7 days. Sent to both Zulip DM and +logged as a `[REPORT]` knowledge graph node for long-term trending. + +### 5. Relay Message (if cross-agent) +If a fix required another agent's help (e.g., Authentik restart), a relay +message is sent to the responsible agent with full context. ## Remediation Rules ### Rule 1: Container Not Healthy -- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy" -- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d ` -- **Verify**: Re-run health check after 15 seconds -- **Escalate**: If still unhealthy after 3 attempts, report to user +Detect → `docker compose up -d ` → verify → log +Escalate after 3 failures -### Rule 2: Admin UI Returns Non-200 -- **Detect**: GET /ui/ returns anything other than 200 -- **Fix**: Restart harness-litellm container -- **Verify**: Re-check /ui/ after 20 seconds -- **Escalate**: If still failing after 2 restarts, report to user +### Rule 2: Admin UI Non-200 +Detect → restart harness-litellm → verify → log +Escalate after 2 failures -### Rule 3: Auth Endpoint Unreachable -- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443 -- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info. -- **Escalate**: Immediately (no retry — external dependency) +### Rule 3: Auth Unreachable +Detect → immediate escalate (external dependency) -### Rule 4: OIDC Login Failing -- **Detect**: OIDC redirect returns error page instead of login -- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container -- **Verify**: If env vars missing, add them and restart -- **Escalate**: If present but still failing, report to user with config dump - -### Rule 5: Docs Endpoint 404 -- **Detect**: GET /docs returns non-200 -- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing) -- **Verify**: Fix env var and restart container +### Rule 4: Docs 404 +Detect → fix DOCS_URL env var → verify → log ## Execution -1. **Load parameters** — Use defaults or provided overrides -2. **Run health check** — Execute `litellm-health.prose.md` to get baseline -3. **Evaluate results** — Compare each check against expected status -4. **Apply remediation** — For each failed check, find matching rule and execute fix -5. **Verify** — Re-run health check after fix -6. **Report** — Compile: what was broken, what was fixed, what needs user attention -7. **Loop** — If self-driven, wait 300s and repeat from step 2 +1. Run health check +2. Apply remediation for each failure +3. **Generate report** with all actions taken +4. **Log to knowledge graph** — create `[LEARN]` node +5. **Notify user** via Zulip DM if anything changed +6. Wait 300s and repeat -## Returns +## Audit Trail Format -- run_id: string — Unique ID for this remediation cycle -- issues_found: number — Total issues detected -- issues_fixed: number — Issues auto-remediated -- issues_escalated: number — Issues requiring user action -- details: array of { issue: string, action: string, result: string } -- timestamp: string +```json +{ + "run_id": "self-heal-20260626-001", + "timestamp": "2026-06-26T14:00:00Z", + "duration_ms": 1234, + "checks_passed": 7, + "checks_failed": 0, + "issues_found": 0, + "issues_fixed": 0, + "issues_escalated": 0, + "actions": [] +} +``` + +With failures: +```json +{ + "run_id": "self-heal-20260626-002", + "issues_found": 1, + "issues_fixed": 1, + "actions": [ + { + "rule": "Container Not Healthy", + "target": "harness-nginx", + "action": "restarted container", + "result": "healthy", + "duration_ms": 5000 + } + ] +} +```