|
|
|
@@ -7,7 +7,7 @@ note: >
|
|
|
|
|
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
|
|
|
|
|
now reimplemented as `litellm-health-check.sh` on CT 116.
|
|
|
|
|
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
|
|
|
|
|
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
|
|
|
|
|
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
|
|
|
|
|
GPU monitoring integrated from gpu-monitor on .24:9100.
|
|
|
|
|
|
|
|
|
|
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
|
|
|
|
@@ -20,7 +20,7 @@ description: >
|
|
|
|
|
LiteLLM inference stack health monitoring + self-healing. Verifies the full
|
|
|
|
|
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
|
|
|
|
|
inference, and agent keys. Applies remediation rules for common failures.
|
|
|
|
|
Reports every action via Zulip DM and RA-H OS knowledge graph.
|
|
|
|
|
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# LiteLLM Operations — Health Check + Self-Heal
|
|
|
|
@@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container
|
|
|
|
|
|
|
|
|
|
Every remediation cycle produces a structured report:
|
|
|
|
|
|
|
|
|
|
### 1. RA-H OS Knowledge Graph Node
|
|
|
|
|
Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
|
|
|
|
|
### 1. Gitea Log Entry
|
|
|
|
|
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.
|
|
|
|
|
|
|
|
|
|
### 2. Zulip DM to Owner
|
|
|
|
|
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
|
|
|
|
@@ -252,9 +252,11 @@ call report-generator
|
|
|
|
|
health: health
|
|
|
|
|
actions: actions
|
|
|
|
|
|
|
|
|
|
-- Phase 4: Log to knowledge graph
|
|
|
|
|
call kg-logger
|
|
|
|
|
-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
|
|
|
|
|
call gitea-logger
|
|
|
|
|
run_id: run_id
|
|
|
|
|
repo: SyslogSolution/health-logs
|
|
|
|
|
path: litellm/{run_id}.json
|
|
|
|
|
health: health
|
|
|
|
|
actions: actions
|
|
|
|
|
|
|
|
|
@@ -300,3 +302,20 @@ With failures:
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## gitea-logger Implementation
|
|
|
|
|
|
|
|
|
|
When this step executes, write the report JSON to a temp file and push to Gitea:
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
REPO="https://abiba-bot:${GITEA_PAT}@git.sysloggh.net/SyslogSolution/health-logs"
|
|
|
|
|
DIR="litellm"
|
|
|
|
|
FILE="${run_id}.json"
|
|
|
|
|
echo "${report_json}" > /tmp/${FILE}
|
|
|
|
|
(cd /tmp && git clone --depth 1 "${REPO}" &&
|
|
|
|
|
cp ${FILE} health-logs/${DIR}/${FILE} &&
|
|
|
|
|
cd health-logs && git add ${DIR}/${FILE} &&
|
|
|
|
|
git commit -m "litellm-health: ${run_id}" && git push)
|
|
|
|
|
rm -rf /tmp/health-logs /tmp/${FILE}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|