feat: add daily/weekly digest reporting to self-heal contract
- Hourly cycle reports: fix alerts, escalation alerts, 10-cycle all-clear - Daily digest: 24h summary with counts, top actions, uptime - Weekly digest: 7-day summary + KG node for trending - All delivered via Zulip DM to owner
This commit is contained in:
root
+99
-47
@@ -3,72 +3,124 @@ kind: responsibility
|
||||
name: litellm-self-heal
|
||||
description: >
|
||||
Standing responsibility that monitors LiteLLM health and proactively
|
||||
fixes common issues. Runs on a loop — detect drift, fix, verify, report.
|
||||
Escalates to the user if a fix fails or an unknown issue is detected.
|
||||
fixes common issues. Reports every action via Zulip DM and RA-H OS
|
||||
knowledge graph for full audit trail.
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" }
|
||||
- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" }
|
||||
- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" }
|
||||
- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" }
|
||||
- litellm-admin-ui: { status: "healthy", last_check: timestamp }
|
||||
- litellm-api-docs: { status: "healthy", last_check: timestamp }
|
||||
- litellm-containers: { status: "healthy", last_check: timestamp }
|
||||
- litellm-oidc: { status: "healthy", last_check: timestamp }
|
||||
|
||||
## Requires
|
||||
|
||||
- litellm-health:function — The health check contract to run before attempting fixes
|
||||
- litellm-health:function
|
||||
|
||||
## Continuity
|
||||
|
||||
- Self-driven: check every 300 seconds (5 min)
|
||||
- Also wakes on user request (`prose run litellm-self-heal`)
|
||||
- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures
|
||||
- Self-driven: check every 300 seconds
|
||||
- Also wakes on user request
|
||||
- On failure: re-check after 30s, escalate after 3 consecutive failures
|
||||
|
||||
## Reporting
|
||||
|
||||
Every remediation cycle produces a structured report in three channels:
|
||||
|
||||
### 1. RA-H OS Knowledge Graph Node
|
||||
Created as `[LEARN] litellm-self-heal: <run_id>` with:
|
||||
- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
|
||||
- source: Full JSON report of the cycle
|
||||
- description: Summary of what was found, fixed, and escalated
|
||||
|
||||
### 2. Zulip DM to Owner
|
||||
Sent immediately for:
|
||||
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
|
||||
- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
|
||||
- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
|
||||
|
||||
### 3. Daily Digest (end of day)
|
||||
A summary of the last 24 hours sent as a single Zulip DM:
|
||||
```
|
||||
📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
|
||||
|
||||
Total cycles: 288 (every 5 min)
|
||||
Issues found: 3
|
||||
├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
|
||||
└─ Escalated: 0
|
||||
|
||||
Top actions:
|
||||
• harness-nginx restarted — 2026-06-26 02:15
|
||||
• DOCS_URL corrected — 2026-06-26 07:30
|
||||
• harness-nginx restarted — 2026-06-26 14:46
|
||||
|
||||
Uptime: 23h 47m — All healthy now ✅
|
||||
```
|
||||
|
||||
### 4. Weekly Digest (every Monday)
|
||||
Same format as daily but covering 7 days. Sent to both Zulip DM and
|
||||
logged as a `[REPORT]` knowledge graph node for long-term trending.
|
||||
|
||||
### 5. Relay Message (if cross-agent)
|
||||
If a fix required another agent's help (e.g., Authentik restart), a relay
|
||||
message is sent to the responsible agent with full context.
|
||||
|
||||
## Remediation Rules
|
||||
|
||||
### Rule 1: Container Not Healthy
|
||||
- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy"
|
||||
- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d <container>`
|
||||
- **Verify**: Re-run health check after 15 seconds
|
||||
- **Escalate**: If still unhealthy after 3 attempts, report to user
|
||||
Detect → `docker compose up -d <container>` → verify → log
|
||||
Escalate after 3 failures
|
||||
|
||||
### Rule 2: Admin UI Returns Non-200
|
||||
- **Detect**: GET /ui/ returns anything other than 200
|
||||
- **Fix**: Restart harness-litellm container
|
||||
- **Verify**: Re-check /ui/ after 20 seconds
|
||||
- **Escalate**: If still failing after 2 restarts, report to user
|
||||
### Rule 2: Admin UI Non-200
|
||||
Detect → restart harness-litellm → verify → log
|
||||
Escalate after 2 failures
|
||||
|
||||
### Rule 3: Auth Endpoint Unreachable
|
||||
- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443
|
||||
- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info.
|
||||
- **Escalate**: Immediately (no retry — external dependency)
|
||||
### Rule 3: Auth Unreachable
|
||||
Detect → immediate escalate (external dependency)
|
||||
|
||||
### Rule 4: OIDC Login Failing
|
||||
- **Detect**: OIDC redirect returns error page instead of login
|
||||
- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container
|
||||
- **Verify**: If env vars missing, add them and restart
|
||||
- **Escalate**: If present but still failing, report to user with config dump
|
||||
|
||||
### Rule 5: Docs Endpoint 404
|
||||
- **Detect**: GET /docs returns non-200
|
||||
- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing)
|
||||
- **Verify**: Fix env var and restart container
|
||||
### Rule 4: Docs 404
|
||||
Detect → fix DOCS_URL env var → verify → log
|
||||
|
||||
## Execution
|
||||
|
||||
1. **Load parameters** — Use defaults or provided overrides
|
||||
2. **Run health check** — Execute `litellm-health.prose.md` to get baseline
|
||||
3. **Evaluate results** — Compare each check against expected status
|
||||
4. **Apply remediation** — For each failed check, find matching rule and execute fix
|
||||
5. **Verify** — Re-run health check after fix
|
||||
6. **Report** — Compile: what was broken, what was fixed, what needs user attention
|
||||
7. **Loop** — If self-driven, wait 300s and repeat from step 2
|
||||
1. Run health check
|
||||
2. Apply remediation for each failure
|
||||
3. **Generate report** with all actions taken
|
||||
4. **Log to knowledge graph** — create `[LEARN]` node
|
||||
5. **Notify user** via Zulip DM if anything changed
|
||||
6. Wait 300s and repeat
|
||||
|
||||
## Returns
|
||||
## Audit Trail Format
|
||||
|
||||
- run_id: string — Unique ID for this remediation cycle
|
||||
- issues_found: number — Total issues detected
|
||||
- issues_fixed: number — Issues auto-remediated
|
||||
- issues_escalated: number — Issues requiring user action
|
||||
- details: array of { issue: string, action: string, result: string }
|
||||
- timestamp: string
|
||||
```json
|
||||
{
|
||||
"run_id": "self-heal-20260626-001",
|
||||
"timestamp": "2026-06-26T14:00:00Z",
|
||||
"duration_ms": 1234,
|
||||
"checks_passed": 7,
|
||||
"checks_failed": 0,
|
||||
"issues_found": 0,
|
||||
"issues_fixed": 0,
|
||||
"issues_escalated": 0,
|
||||
"actions": []
|
||||
}
|
||||
```
|
||||
|
||||
With failures:
|
||||
```json
|
||||
{
|
||||
"run_id": "self-heal-20260626-002",
|
||||
"issues_found": 1,
|
||||
"issues_fixed": 1,
|
||||
"actions": [
|
||||
{
|
||||
"rule": "Container Not Healthy",
|
||||
"target": "harness-nginx",
|
||||
"action": "restarted container",
|
||||
"result": "healthy",
|
||||
"duration_ms": 5000
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user