prose-contracts/litellm-self-heal.prose.md

kind, name, status, note, description

kind	name	status	note	description
responsibility	litellm-self-heal	manual-only	Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04). This contract is now manual-only — triggers require explicit user request. Consider reimplementing as a standalone cron job or prose contract. Source of truth for GPU topology: gpu-fleet.prose.md Last verified: 2026-07-09	Standing responsibility that monitors LiteLLM health and proactively fixes common issues. Reports every action via Zulip DM and RA-H OS knowledge graph for full audit trail.

Maintains

• litellm-admin-ui: { status: "healthy", last_check: timestamp }
• litellm-api-docs: { status: "healthy", last_check: timestamp }
• litellm-containers: { status: "healthy", last_check: timestamp }
• litellm-oidc: { status: "healthy", last_check: timestamp }

Requires

• litellm-health:function

Continuity

• Self-driven: check every 300 seconds
• Also wakes on user request
• On failure: re-check after 30s, escalate after 3 consecutive failures

Reporting

Every remediation cycle produces a structured report in three channels:

1. RA-H OS Knowledge Graph Node

Created as [LEARN] litellm-self-heal: <run_id> with:

• metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
• source: Full JSON report of the cycle
• description: Summary of what was found, fixed, and escalated

2. Zulip DM to Owner

Sent immediately for:

• issues_fixed > 0 — "🛠 LiteLLM Self-Heal — Fix Applied"
• issues_escalated > 0 — "⚠ LiteLLM Self-Heal — Needs Your Attention"
• Every 10th clean cycle — "✅ All Clear (10 checks passed)"

3. Daily Digest (end of day)

A summary of the last 24 hours sent as a single Zulip DM:

📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)

Total cycles: 288 (every 5 min)
Issues found: 3
  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
  └─ Escalated: 0

Top actions:
  • harness-nginx restarted — 2026-06-26 02:15
  • DOCS_URL corrected — 2026-06-26 07:30  
  • harness-nginx restarted — 2026-06-26 14:46

Uptime: 23h 47m — All healthy now ✅

4. Weekly Digest (every Monday)

Same format as daily but covering 7 days. Sent to both Zulip DM and logged as a [REPORT] knowledge graph node for long-term trending.

5. Relay Message (if cross-agent)

If a fix required another agent's help (e.g., Authentik restart), a relay message is sent to the responsible agent with full context.

GPU Fleet

Host	IP	Hardware	Models	Engine	Context
amdpve	192.168.68.15	Strix Halo 64GB UMA	ornith-1.0-35b	llama-server :8080 (Vulkan)	256K
llm-gpu	192.168.68.8	RTX 3090 24GB	qwen3.6-27B-code	llama-server :8080 (systemd)	128K
ocu-llm	192.168.68.110	RTX 5070 12GB	gemma-4-12b	llama-server :8080 (systemd)	128K

Remediation Rules

Rule 1: Container Not Healthy

Detect → docker compose up -d <container> → verify → log Escalate after 3 failures

Rule 2: Admin UI Non-200

Detect → restart harness-litellm → verify → log Escalate after 2 failures

Rule 3: Auth Unreachable

Detect → immediate escalate (external dependency)

Rule 4: Docs 404

Detect → fix DOCS_URL env var → verify → log

Rule 5: GPU Unreachable

Detect → SSH to GPU host fails or model not responding Fix → restart llama-server on host via SSH Escalate → immediately (downtime affects inference)

Rule 6: Model Not Responding

Detect → test prompt via LiteLLM returns non-200 Fix → restart llama-server on GPU host → verify Escalate → after 2 failed restarts

Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED

Router is no longer in the request path (2026-07-08). LiteLLM proxies directly to GPU. This rule is retained for reference but is inactive. If 503 errors occur, check LiteLLM timeouts and GPU health directly.

Rule 8: Agent Keys Invalid (401)

Detect → agents report auth errors Root cause → keys in router env var, not in LiteLLM DB Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts Escalate → if SSH access unavailable, send Zulip DM

Rule 9: Stale Active Counter in Redis

Detect → active: > 0 while GPU idle Fix → redis-cli DEL active:<model> → restart router Prevent → add TTL to active keys in router code

Execution

1. Run health check
2. Apply remediation for each failure
3. Generate report with all actions taken
4. Log to knowledge graph — create [LEARN] node
5. Notify user via Zulip DM if anything changed
6. Wait 300s and repeat

Audit Trail Format

{
  "run_id": "self-heal-20260626-001",
  "timestamp": "2026-06-26T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
  "issues_found": 0,
  "issues_fixed": 0,
  "issues_escalated": 0,
  "actions": []
}

With failures:

{
  "run_id": "self-heal-20260626-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
    {
      "rule": "Container Not Healthy",
      "target": "harness-nginx",
      "action": "restarted container",
      "result": "healthy",
      "duration_ms": 5000
    }
  ]
}