feat: add daily/weekly digest reporting to self-heal contract

- Hourly cycle reports: fix alerts, escalation alerts, 10-cycle all-clear
- Daily digest: 24h summary with counts, top actions, uptime
- Weekly digest: 7-day summary + KG node for trending
- All delivered via Zulip DM to owner

litellm-self-heal.prose.md

@@ -3,72 +3,124 @@ kind: responsibility
 name: litellm-self-heal
 description: >
   Standing responsibility that monitors LiteLLM health and proactively
-  fixes common issues. Runs on a loop — detect drift, fix, verify, report.
+  fixes common issues. Reports every action via Zulip DM and RA-H OS
-  Escalates to the user if a fix fails or an unknown issue is detected.
+  knowledge graph for full audit trail.
 ---
 
 ## Maintains
 
-- litellm-admin-ui: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/ui/" }
+- litellm-admin-ui: { status: "healthy", last_check: timestamp }
-- litellm-api-docs: { status: "healthy", last_check: timestamp, url: "https://litellm.sysloggh.net/docs" }
+- litellm-api-docs: { status: "healthy", last_check: timestamp }
-- litellm-containers: { status: "healthy", last_check: timestamp, host: "192.168.68.116" }
+- litellm-containers: { status: "healthy", last_check: timestamp }
-- litellm-oidc: { status: "healthy", last_check: timestamp, auth_host: "192.168.68.11" }
+- litellm-oidc: { status: "healthy", last_check: timestamp }
 
 ## Requires
 
-- litellm-health:function — The health check contract to run before attempting fixes
+- litellm-health:function
 
 ## Continuity
 
-- Self-driven: check every 300 seconds (5 min)
+- Self-driven: check every 300 seconds
-- Also wakes on user request (`prose run litellm-self-heal`)
+- Also wakes on user request
-- On failure: re-check after 30 seconds, then escalate after 3 consecutive failures
+- On failure: re-check after 30s, escalate after 3 consecutive failures
+
+## Reporting
+
+Every remediation cycle produces a structured report in three channels:
+
+### 1. RA-H OS Knowledge Graph Node
+Created as `[LEARN] litellm-self-heal: <run_id>` with:
+- metadata: { type: "remediation", status: "fixed" | "escalated" | "healthy" }
+- source: Full JSON report of the cycle
+- description: Summary of what was found, fixed, and escalated
+
+### 2. Zulip DM to Owner
+Sent immediately for:
+- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
+- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
+- Every 10th clean cycle — "✅ All Clear (10 checks passed)"
+
+### 3. Daily Digest (end of day)
+A summary of the last 24 hours sent as a single Zulip DM:
+```
+📋 LiteLLM Self-Heal — Daily Digest (2026-06-26)
+
+Total cycles: 288 (every 5 min)
+Issues found: 3
+  ├─ Fixed automatically: 3 (nginx restart x2, docs fix x1)
+  └─ Escalated: 0
+
+Top actions:
+  • harness-nginx restarted — 2026-06-26 02:15
+  • DOCS_URL corrected — 2026-06-26 07:30  
+  • harness-nginx restarted — 2026-06-26 14:46
+
+Uptime: 23h 47m — All healthy now ✅
+```
+
+### 4. Weekly Digest (every Monday)
+Same format as daily but covering 7 days. Sent to both Zulip DM and
+logged as a `[REPORT]` knowledge graph node for long-term trending.
+
+### 5. Relay Message (if cross-agent)
+If a fix required another agent's help (e.g., Authentik restart), a relay
+message is sent to the responsible agent with full context.
 
 ## Remediation Rules
 
 ### Rule 1: Container Not Healthy
-- **Detect**: `harness-litellm` or `harness-nginx` status is not "healthy"
+Detect → `docker compose up -d <container>` → verify → log
-- **Fix**: `docker compose -f /opt/inference-harness/docker-compose.yml up -d <container>`
+Escalate after 3 failures
-- **Verify**: Re-run health check after 15 seconds
-- **Escalate**: If still unhealthy after 3 attempts, report to user
 
-### Rule 2: Admin UI Returns Non-200
+### Rule 2: Admin UI Non-200
-- **Detect**: GET /ui/ returns anything other than 200
+Detect → restart harness-litellm → verify → log
-- **Fix**: Restart harness-litellm container
+Escalate after 2 failures
-- **Verify**: Re-check /ui/ after 20 seconds
-- **Escalate**: If still failing after 2 restarts, report to user
 
-### Rule 3: Auth Endpoint Unreachable
+### Rule 3: Auth Unreachable
-- **Detect**: Cannot reach auth.sysloggh.net or 192.168.68.11:443
+Detect → immediate escalate (external dependency)
-- **Fix**: Cannot auto-fix (Authentik is external). Report to user with diagnostic info.
-- **Escalate**: Immediately (no retry — external dependency)
 
-### Rule 4: OIDC Login Failing
+### Rule 4: Docs 404
-- **Detect**: OIDC redirect returns error page instead of login
+Detect → fix DOCS_URL env var → verify → log
-- **Fix**: Check if GENERIC_* env vars are present in LiteLLM container
-- **Verify**: If env vars missing, add them and restart
-- **Escalate**: If present but still failing, report to user with config dump
-
-### Rule 5: Docs Endpoint 404
-- **Detect**: GET /docs returns non-200
-- **Fix**: Check DOCS_URL env var — should be `/docs` (not `/litellm/docs` or missing)
-- **Verify**: Fix env var and restart container
 
 ## Execution
 
-1. **Load parameters** — Use defaults or provided overrides
+1. Run health check
-2. **Run health check** — Execute `litellm-health.prose.md` to get baseline
+2. Apply remediation for each failure
-3. **Evaluate results** — Compare each check against expected status
+3. **Generate report** with all actions taken
-4. **Apply remediation** — For each failed check, find matching rule and execute fix
+4. **Log to knowledge graph** — create `[LEARN]` node
-5. **Verify** — Re-run health check after fix
+5. **Notify user** via Zulip DM if anything changed
-6. **Report** — Compile: what was broken, what was fixed, what needs user attention
+6. Wait 300s and repeat
-7. **Loop** — If self-driven, wait 300s and repeat from step 2
 
-## Returns
+## Audit Trail Format
 
-- run_id: string — Unique ID for this remediation cycle
+```json
-- issues_found: number — Total issues detected
+{
-- issues_fixed: number — Issues auto-remediated
+  "run_id": "self-heal-20260626-001",
-- issues_escalated: number — Issues requiring user action
+  "timestamp": "2026-06-26T14:00:00Z",
-- details: array of { issue: string, action: string, result: string }
+  "duration_ms": 1234,
-- timestamp: string
+  "checks_passed": 7,
+  "checks_failed": 0,
+  "issues_found": 0,
+  "issues_fixed": 0,
+  "issues_escalated": 0,
+  "actions": []
+}
+```
+
+With failures:
+```json
+{
+  "run_id": "self-heal-20260626-002",
+  "issues_found": 1,
+  "issues_fixed": 1,
+  "actions": [
+    {
+      "rule": "Container Not Healthy",
+      "target": "harness-nginx",
+      "action": "restarted container",
+      "result": "healthy",
+      "duration_ms": 5000
+    }
+  ]
+}
+```