prose-contracts/litellm-self-heal.prose.md

---
kind: responsibility
name: litellm-self-heal
status: manual-only
note: >
  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04).
  This contract is now manual-only — triggers require explicit user request.
  Consider reimplementing as a standalone cron job or prose contract.
  
  Source of truth for GPU topology: gpu-fleet.prose.md
  Last verified: 2026-07-09
description: >
  Standing responsibility that monitors LiteLLM health and proactively
  fixes common issues. Reports every action via Zulip DM and RA-H OS
  knowledge graph for full audit trail.
---

## Maintains

- litellm-admin-ui: { status: "healthy", last_check: timestamp }
- litellm-api-docs: { status: "healthy", last_check: timestamp }
- litellm-containers: { status: "healthy", last_check: timestamp }
- litellm-oidc: { status: "healthy", last_check: timestamp }

## Requires

- litellm-health:function

## Continuity

- Self-driven: check every 300 seconds
- Also wakes on user request
- On failure: re-check after 30s, escalate after 3 consecutive failures

## Reporting

Every remediation cycle produces a structured report in three channels:

### 1. RA-H OS Knowledge Graph Node
Created as `[LEARN] litellm-self-heal: <run_id>` with:
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
- source: Full JSON report of the cycle
- description: Summary of what was found, fixed, and escalated

### 2. Zulip DM to Owner
Sent immediately for:
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
- Every 10th clean cycle — "✅ All Clear (10 checks passed)"

### 3. Daily Digest (end of day)
A summary of the last 24 hours sent as a single Zulip DM:
```
📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)

Total cycles: 288 (every 5 min)
Issues found: 3
  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
  └─ Escalated: 0

Top actions:
  • harness-nginx restarted — 2026-06-26 02:15
  • DOCS_URL corrected — 2026-06-26 07:30  
  • harness-nginx restarted — 2026-06-26 14:46

Uptime: 23h 47m — All healthy now ✅
```

### 4. Weekly Digest (every Monday)
Same format as daily but covering 7 days. Sent to both Zulip DM and
logged as a `[REPORT]` knowledge graph node for long-term trending.

### 5. Relay Message (if cross-agent)
If a fix required another agent's help (e.g., Authentik restart), a relay
message is sent to the responsible agent with full context.

## GPU Fleet

| Host | IP | Hardware | Models | Engine | Context |
|------|-----|----------|--------|--------|---------|
| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b | llama-server :8080 (Vulkan) | 256K |
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 (systemd) | 128K |
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 (systemd) | 128K |

## Remediation Rules

### Rule 1: Container Not Healthy
Detect → `docker compose up -d <container>` → verify → log
Escalate after 3 failures

### Rule 2: Admin UI Non-200
Detect → restart harness-litellm → verify → log
Escalate after 2 failures

### Rule 3: Auth Unreachable
Detect → immediate escalate (external dependency)

### Rule 4: Docs 404
Detect → fix DOCS_URL env var → verify → log

### Rule 5: GPU Unreachable
Detect → SSH to GPU host fails or model not responding
Fix → restart llama-server on host via SSH
Escalate → immediately (downtime affects inference)

### Rule 6: Model Not Responding
Detect → test prompt via LiteLLM returns non-200
Fix → restart llama-server on GPU host → verify
Escalate → after 2 failed restarts

### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
Router is no longer in the request path (2026-07-08). LiteLLM proxies
directly to GPU. This rule is retained for reference but is inactive.
If 503 errors occur, check LiteLLM timeouts and GPU health directly.

### Rule 8: Agent Keys Invalid (401)
Detect → agents report auth errors
Root cause → keys in router env var, not in LiteLLM DB
Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
Escalate → if SSH access unavailable, send Zulip DM

### Rule 9: Stale Active Counter in Redis
Detect → active:<model> > 0 while GPU idle
Fix → `redis-cli DEL active:<model>` → restart router
Prevent → add TTL to active keys in router code

## Execution

1. Run health check
2. Apply remediation for each failure
3. **Generate report** with all actions taken
4. **Log to knowledge graph** — create `[LEARN]` node
5. **Notify user** via Zulip DM if anything changed
6. Wait 300s and repeat

## Audit Trail Format

```json
{
  "run_id": "self-heal-20260626-001",
  "timestamp": "2026-06-26T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
  "issues_found": 0,
  "issues_fixed": 0,
  "issues_escalated": 0,
  "actions": []
}
```

With failures:
```json
{
  "run_id": "self-heal-20260626-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
    {
      "rule": "Container Not Healthy",
      "target": "harness-nginx",
      "action": "restarted container",
      "result": "healthy",
      "duration_ms": 5000
    }
  ]
}
```