From c9359e180808f033518d3ccf3611b64040272951 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 28 Jul 2026 21:20:56 +0000 Subject: [PATCH] fix: redirect health check logs from knowledge graph to Gitea (hard rule) Logs (LITELLM-HEALTH, GPU-SELF-HEAL, PM2) now pushed to SyslogSolution/health-logs instead of creating orphan nodes in the shared knowledge graph. - litellm-self-heal: Phase 4 now calls gitea-logger instead of kg-logger - gpu-self-heal: Reporting section updated to Gitea path - pm2-self-heal: Log step redirected to Gitea - 271 existing orphan nodes remain in graph (no delete tool available) --- gpu-self-heal.prose.md | 4 ++-- litellm-self-heal.prose.md | 14 ++++++++------ pm2-self-heal.prose.md | 2 +- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/gpu-self-heal.prose.md b/gpu-self-heal.prose.md index 911ed08..403657a 100644 --- a/gpu-self-heal.prose.md +++ b/gpu-self-heal.prose.md @@ -251,8 +251,8 @@ call update-gpu-health ## Reporting -### 1. Knowledge Graph -Every action logged as `[GPU-SELF-HEAL] ` node with full audit trail. +### 1. Gitea Log (not knowledge graph — hard rule) +Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph. ### 2. Zulip Alerts (#agent-hub → alerts-gpu) - `issues_fixed > 0` → "🛠 GPU Self-Heal — resolved" diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 76dced7..eb99339 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -7,7 +7,7 @@ note: > Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04), now reimplemented as `litellm-health-check.sh` on CT 116. Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`). - Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph. + Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs). GPU monitoring integrated from gpu-monitor on .24:9100. Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate @@ -20,7 +20,7 @@ description: > LiteLLM inference stack health monitoring + self-healing. Verifies the full nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model inference, and agent keys. Applies remediation rules for common failures. - Reports every action via Zulip DM and RA-H OS knowledge graph. + Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs). --- # LiteLLM Operations — Health Check + Self-Heal @@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container Every remediation cycle produces a structured report: -### 1. RA-H OS Knowledge Graph Node -Created as `[LEARN] litellm-self-heal: ` with full JSON report. +### 1. Gitea Log Entry +Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph. ### 2. Zulip DM to Owner - `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied" @@ -252,9 +252,11 @@ call report-generator health: health actions: actions --- Phase 4: Log to knowledge graph -call kg-logger +-- Phase 4: Log to Gitea (not knowledge graph — hard rule) +call gitea-logger run_id: run_id + repo: SyslogSolution/health-logs + path: litellm/{run_id}.json health: health actions: actions diff --git a/pm2-self-heal.prose.md b/pm2-self-heal.prose.md index 1eb1fb5..343d9f9 100644 --- a/pm2-self-heal.prose.md +++ b/pm2-self-heal.prose.md @@ -52,7 +52,7 @@ description: > - If status is "online" → pass, log restarts count - If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately - If restarts > 5 in last hour → alert owner with full diagnostics -4. **Log results** — Create `[LEARN]` node in knowledge graph for any actions taken +4. **Log results** — Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule) 5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting) 6. **Wait 5 min** → repeat from step 1 -- 2.54.0