prose-contracts/litellm-self-heal.prose.md

kind, name, status, note, description

kind	name	status	note	description
responsibility	litellm-self-heal	manual-only	Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04). This contract is now manual-only — triggers require explicit user request. Consider reimplementing as a standalone cron job or prose contract.	Standing responsibility that monitors LiteLLM health and proactively fixes common issues. Reports every action via Zulip DM and RA-H OS knowledge graph for full audit trail.

Maintains

• litellm-admin-ui: { status: "healthy", last_check: timestamp }
• litellm-api-docs: { status: "healthy", last_check: timestamp }
• litellm-containers: { status: "healthy", last_check: timestamp }
• litellm-oidc: { status: "healthy", last_check: timestamp }

Requires

• litellm-health:function

Continuity

• Self-driven: check every 300 seconds
• Also wakes on user request
• On failure: re-check after 30s, escalate after 3 consecutive failures

Reporting

Every remediation cycle produces a structured report in three channels:

1. RA-H OS Knowledge Graph Node

Created as [LEARN] litellm-self-heal: <run_id> with:

• metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
• source: Full JSON report of the cycle
• description: Summary of what was found, fixed, and escalated

2. Zulip DM to Owner

Sent immediately for:

• issues_fixed > 0 — "🛠 LiteLLM Self-Heal — Fix Applied"
• issues_escalated > 0 — "⚠ LiteLLM Self-Heal — Needs Your Attention"
• Every 10th clean cycle — "✅ All Clear (10 checks passed)"

3. Daily Digest (end of day)

A summary of the last 24 hours sent as a single Zulip DM:

📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)

Total cycles: 288 (every 5 min)
Issues found: 3
  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
  └─ Escalated: 0

Top actions:
  • harness-nginx restarted — 2026-06-26 02:15
  • DOCS_URL corrected — 2026-06-26 07:30  
  • harness-nginx restarted — 2026-06-26 14:46

Uptime: 23h 47m — All healthy now ✅

4. Weekly Digest (every Monday)

Same format as daily but covering 7 days. Sent to both Zulip DM and logged as a [REPORT] knowledge graph node for long-term trending.

5. Relay Message (if cross-agent)

If a fix required another agent's help (e.g., Authentik restart), a relay message is sent to the responsible agent with full context.

GPU Fleet

Host	IP	Hardware	Models	Engine
amdpve	192.168.68.15	Strix Halo 64GB UMA	ornith-1.0-35b (16t, 262K ctx)	llama-server :8080
llm-gpu	192.168.68.8	RTX 3090 24GB	qwen3.6-27B-code	llama-server :8080
ocu-llm	192.168.68.110	RTX 5070 12GB	gemma-4-12b	llama-server :8080

Remediation Rules

Rule 1: Container Not Healthy

Detect → docker compose up -d <container> → verify → log Escalate after 3 failures

Rule 2: Admin UI Non-200

Detect → restart harness-litellm → verify → log Escalate after 2 failures

Rule 3: Auth Unreachable

Detect → immediate escalate (external dependency)

Rule 4: Docs 404

Detect → fix DOCS_URL env var → verify → log

Rule 5: GPU Unreachable

Detect → SSH to GPU host fails or model not responding Fix → restart llama-server on host via SSH Escalate → immediately (downtime affects inference)

Rule 6: Model Not Responding

Detect → test prompt via LiteLLM returns non-200 Fix → restart llama-server on GPU host → verify Escalate → after 2 failed restarts

Rule 7: Router Returns 503 (All GPUs Saturated)

Detect → router logs show QUEUE_TIMEOUT with GPUs idle Root cause → roster not loaded at startup (empty TIER_MODELS) Fix → POST /admin/roster/reload (if endpoint available) or restart router Permanent fix → add load_roster() to router startup code Escalate → if reload doesn't resolve within 2 min

Rule 8: Agent Keys Invalid (401)

Detect → agents report auth errors Root cause → keys in router env var, not in LiteLLM DB Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts Escalate → if SSH access unavailable, send Zulip DM

Rule 9: Stale Active Counter in Redis

Detect → active: > 0 while GPU idle Fix → redis-cli DEL active:<model> → restart router Prevent → add TTL to active keys in router code

Execution

1. Run health check
2. Apply remediation for each failure
3. Generate report with all actions taken
4. Log to knowledge graph — create [LEARN] node
5. Notify user via Zulip DM if anything changed
6. Wait 300s and repeat

Audit Trail Format

{
  "run_id": "self-heal-20260626-001",
  "timestamp": "2026-06-26T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
  "issues_found": 0,
  "issues_fixed": 0,
  "issues_escalated": 0,
  "actions": []
}

With failures:

{
  "run_id": "self-heal-20260626-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
    {
      "rule": "Container Not Healthy",
      "target": "harness-nginx",
      "action": "restarted container",
      "result": "healthy",
      "duration_ms": 5000
    }
  ]
}