prose-contracts/litellm-self-heal.prose.md

kind, name, description

kind	name	description
responsibility	litellm-self-heal	Standing responsibility that monitors LiteLLM health and proactively fixes common issues. Reports every action via Zulip DM and RA-H OS knowledge graph for full audit trail.

Maintains

• litellm-admin-ui: { status: "healthy", last_check: timestamp }
• litellm-api-docs: { status: "healthy", last_check: timestamp }
• litellm-containers: { status: "healthy", last_check: timestamp }
• litellm-oidc: { status: "healthy", last_check: timestamp }

Requires

• litellm-health:function

Continuity

• Self-driven: check every 300 seconds
• Also wakes on user request
• On failure: re-check after 30s, escalate after 3 consecutive failures

Reporting

Every remediation cycle produces a structured report in three channels:

1. RA-H OS Knowledge Graph Node

Created as [LEARN] litellm-self-heal: <run_id> with:

• metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
• source: Full JSON report of the cycle
• description: Summary of what was found, fixed, and escalated

2. Zulip DM to Owner

Sent immediately for:

• issues_fixed > 0 — "🛠 LiteLLM Self-Heal — Fix Applied"
• issues_escalated > 0 — "⚠ LiteLLM Self-Heal — Needs Your Attention"
• Every 10th clean cycle — "✅ All Clear (10 checks passed)"

3. Daily Digest (end of day)

A summary of the last 24 hours sent as a single Zulip DM:

📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)

Total cycles: 288 (every 5 min)
Issues found: 3
  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
  └─ Escalated: 0

Top actions:
  • harness-nginx restarted — 2026-06-26 02:15
  • DOCS_URL corrected — 2026-06-26 07:30  
  • harness-nginx restarted — 2026-06-26 14:46

Uptime: 23h 47m — All healthy now ✅

4. Weekly Digest (every Monday)

Same format as daily but covering 7 days. Sent to both Zulip DM and logged as a [REPORT] knowledge graph node for long-term trending.

5. Relay Message (if cross-agent)

If a fix required another agent's help (e.g., Authentik restart), a relay message is sent to the responsible agent with full context.

GPU Fleet

Host	IP	Hardware	Models	Engine
amdpve	192.168.68.15	Strix Halo CPU	ornith-1.0-35b (16t, 262K ctx)	llama-server :8080
llm-gpu	192.168.68.8	RTX 3090 24GB	gemma-4-12b Dense	llama-server :8080
ocu-llm	192.168.68.110	RTX 5070 12GB	qwen3.6-27B-code MoE+Light	llama-server :8080

Remediation Rules

Rule 1: Container Not Healthy

Detect → docker compose up -d <container> → verify → log Escalate after 3 failures

Rule 2: Admin UI Non-200

Detect → restart harness-litellm → verify → log Escalate after 2 failures

Rule 3: Auth Unreachable

Detect → immediate escalate (external dependency)

Rule 4: Docs 404

Detect → fix DOCS_URL env var → verify → log

Rule 5: GPU Unreachable

Detect → SSH to GPU host fails or model not responding Fix → restart llama-server on host via SSH Escalate → immediately (downtime affects inference)

Rule 6: Model Not Responding

Detect → test prompt via LiteLLM returns non-200 Fix → restart llama-server on GPU host → verify Escalate → after 2 failed restarts

Execution

1. Run health check
2. Apply remediation for each failure
3. Generate report with all actions taken
4. Log to knowledge graph — create [LEARN] node
5. Notify user via Zulip DM if anything changed
6. Wait 300s and repeat

Audit Trail Format

{
  "run_id": "self-heal-20260626-001",
  "timestamp": "2026-06-26T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
  "issues_found": 0,
  "issues_fixed": 0,
  "issues_escalated": 0,
  "actions": []
}

With failures:

{
  "run_id": "self-heal-20260626-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
    {
      "rule": "Container Not Healthy",
      "target": "harness-nginx",
      "action": "restarted container",
      "result": "healthy",
      "duration_ms": 5000
    }
  ]
}