Abiba
c1d993fe01
- hello-world: Ensures -> Maintains (v0.15.0 rename). - zulip-mention-reliability: fold Success Criteria into Maintains postconditions. - litellm-self-heal: swap gemma/qwen GPU hosts to match actual deployment; add Strix Halo 64GB UMA detail; add Rules 7 (roster stuck), 8 (agent keys invalid/401), 9 (stale Redis active counter).
163 lines
4.9 KiB
Markdown
163 lines
4.9 KiB
Markdown
---
|
|
kind: responsibility
|
|
name: litellm-self-heal
|
|
description: >
|
|
Standing responsibility that monitors LiteLLM health and proactively
|
|
fixes common issues. Reports every action via Zulip DM and RA-H OS
|
|
knowledge graph for full audit trail.
|
|
---
|
|
|
|
## Maintains
|
|
|
|
- litellm-admin-ui: { status: "healthy", last_check: timestamp }
|
|
- litellm-api-docs: { status: "healthy", last_check: timestamp }
|
|
- litellm-containers: { status: "healthy", last_check: timestamp }
|
|
- litellm-oidc: { status: "healthy", last_check: timestamp }
|
|
|
|
## Requires
|
|
|
|
- litellm-health:function
|
|
|
|
## Continuity
|
|
|
|
- Self-driven: check every 300 seconds
|
|
- Also wakes on user request
|
|
- On failure: re-check after 30s, escalate after 3 consecutive failures
|
|
|
|
## Reporting
|
|
|
|
Every remediation cycle produces a structured report in three channels:
|
|
|
|
### 1. RA-H OS Knowledge Graph Node
|
|
Created as `[LEARN] litellm-self-heal: <run_id>` with:
|
|
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
|
|
- source: Full JSON report of the cycle
|
|
- description: Summary of what was found, fixed, and escalated
|
|
|
|
### 2. Zulip DM to Owner
|
|
Sent immediately for:
|
|
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
|
|
- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
|
|
- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
|
|
|
|
### 3. Daily Digest (end of day)
|
|
A summary of the last 24 hours sent as a single Zulip DM:
|
|
```
|
|
📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
|
|
|
|
Total cycles: 288 (every 5 min)
|
|
Issues found: 3
|
|
├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
|
|
└─ Escalated: 0
|
|
|
|
Top actions:
|
|
• harness-nginx restarted — 2026-06-26 02:15
|
|
• DOCS_URL corrected — 2026-06-26 07:30
|
|
• harness-nginx restarted — 2026-06-26 14:46
|
|
|
|
Uptime: 23h 47m — All healthy now ✅
|
|
```
|
|
|
|
### 4. Weekly Digest (every Monday)
|
|
Same format as daily but covering 7 days. Sent to both Zulip DM and
|
|
logged as a `[REPORT]` knowledge graph node for long-term trending.
|
|
|
|
### 5. Relay Message (if cross-agent)
|
|
If a fix required another agent's help (e.g., Authentik restart), a relay
|
|
message is sent to the responsible agent with full context.
|
|
|
|
## GPU Fleet
|
|
|
|
| Host | IP | Hardware | Models | Engine |
|
|
|------|-----|----------|--------|--------|
|
|
| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
|
|
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
|
|
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
|
|
|
|
## Remediation Rules
|
|
|
|
### Rule 1: Container Not Healthy
|
|
Detect → `docker compose up -d <container>` → verify → log
|
|
Escalate after 3 failures
|
|
|
|
### Rule 2: Admin UI Non-200
|
|
Detect → restart harness-litellm → verify → log
|
|
Escalate after 2 failures
|
|
|
|
### Rule 3: Auth Unreachable
|
|
Detect → immediate escalate (external dependency)
|
|
|
|
### Rule 4: Docs 404
|
|
Detect → fix DOCS_URL env var → verify → log
|
|
|
|
### Rule 5: GPU Unreachable
|
|
Detect → SSH to GPU host fails or model not responding
|
|
Fix → restart llama-server on host via SSH
|
|
Escalate → immediately (downtime affects inference)
|
|
|
|
### Rule 6: Model Not Responding
|
|
Detect → test prompt via LiteLLM returns non-200
|
|
Fix → restart llama-server on GPU host → verify
|
|
Escalate → after 2 failed restarts
|
|
|
|
### Rule 7: Router Returns 503 (All GPUs Saturated)
|
|
Detect → router logs show QUEUE_TIMEOUT with GPUs idle
|
|
Root cause → roster not loaded at startup (empty TIER_MODELS)
|
|
Fix → POST /admin/roster/reload (if endpoint available) or restart router
|
|
Permanent fix → add load_roster() to router startup code
|
|
Escalate → if reload doesn't resolve within 2 min
|
|
|
|
### Rule 8: Agent Keys Invalid (401)
|
|
Detect → agents report auth errors
|
|
Root cause → keys in router env var, not in LiteLLM DB
|
|
Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
|
|
Escalate → if SSH access unavailable, send Zulip DM
|
|
|
|
### Rule 9: Stale Active Counter in Redis
|
|
Detect → active:<model> > 0 while GPU idle
|
|
Fix → `redis-cli DEL active:<model>` → restart router
|
|
Prevent → add TTL to active keys in router code
|
|
|
|
## Execution
|
|
|
|
1. Run health check
|
|
2. Apply remediation for each failure
|
|
3. **Generate report** with all actions taken
|
|
4. **Log to knowledge graph** — create `[LEARN]` node
|
|
5. **Notify user** via Zulip DM if anything changed
|
|
6. Wait 300s and repeat
|
|
|
|
## Audit Trail Format
|
|
|
|
```json
|
|
{
|
|
"run_id": "self-heal-20260626-001",
|
|
"timestamp": "2026-06-26T14:00:00Z",
|
|
"duration_ms": 1234,
|
|
"checks_passed": 7,
|
|
"checks_failed": 0,
|
|
"issues_found": 0,
|
|
"issues_fixed": 0,
|
|
"issues_escalated": 0,
|
|
"actions": []
|
|
}
|
|
```
|
|
|
|
With failures:
|
|
```json
|
|
{
|
|
"run_id": "self-heal-20260626-002",
|
|
"issues_found": 1,
|
|
"issues_fixed": 1,
|
|
"actions": [
|
|
{
|
|
"rule": "Container Not Healthy",
|
|
"target": "harness-nginx",
|
|
"action": "restarted container",
|
|
"result": "healthy",
|
|
"duration_ms": 5000
|
|
}
|
|
]
|
|
}
|
|
```
|