Compare commits

...
Author SHA1 Message Date
root 42a1d0bd91 docs: add gitea-logger implementation example to litellm-self-heal
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
2026-07-28 21:23:04 +00:00
root c9359e1808 fix: redirect health check logs from knowledge graph to Gitea (hard rule)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Logs (LITELLM-HEALTH, GPU-SELF-HEAL, PM2) now pushed to SyslogSolution/health-logs
instead of creating orphan nodes in the shared knowledge graph.

- litellm-self-heal: Phase 4 now calls gitea-logger instead of kg-logger
- gpu-self-heal: Reporting section updated to Gitea path
- pm2-self-heal: Log step redirected to Gitea
- 271 existing orphan nodes remain in graph (no delete tool available)
2026-07-28 21:20:56 +00:00
mumuni-bot 6a0af728e9 Merge pull request 'fix: update all Mumuni IP references from .123 to .24 (inside Abiba CT100)' (#35) from fix/mumuni-ip-abiba into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-07-28 08:57:05 +00:00
mumuni-bot 835647e241 Merge pull request 'fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)' (#34) from fix/gpu-dense-q3-correction into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-07-27 20:26:13 +00:00
3 changed files with 28 additions and 9 deletions
+2 -2
View File
@@ -251,8 +251,8 @@ call update-gpu-health
## Reporting
### 1. Knowledge Graph
Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
### 1. Gitea Log (not knowledge graph — hard rule)
Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph.
### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
+25 -6
View File
@@ -7,7 +7,7 @@ note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116.
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
@@ -20,7 +20,7 @@ description: >
LiteLLM inference stack health monitoring + self-healing. Verifies the full
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
inference, and agent keys. Applies remediation rules for common failures.
Reports every action via Zulip DM and RA-H OS knowledge graph.
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
---
# LiteLLM Operations — Health Check + Self-Heal
@@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container
Every remediation cycle produces a structured report:
### 1. RA-H OS Knowledge Graph Node
Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
### 1. Gitea Log Entry
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.
### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
@@ -252,9 +252,11 @@ call report-generator
health: health
actions: actions
-- Phase 4: Log to knowledge graph
call kg-logger
-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
call gitea-logger
run_id: run_id
repo: SyslogSolution/health-logs
path: litellm/{run_id}.json
health: health
actions: actions
@@ -300,3 +302,20 @@ With failures:
]
}
```
## gitea-logger Implementation
When this step executes, write the report JSON to a temp file and push to Gitea:
```bash
REPO="https://abiba-bot:${GITEA_PAT}@git.sysloggh.net/SyslogSolution/health-logs"
DIR="litellm"
FILE="${run_id}.json"
echo "${report_json}" > /tmp/${FILE}
(cd /tmp && git clone --depth 1 "${REPO}" &&
cp ${FILE} health-logs/${DIR}/${FILE} &&
cd health-logs && git add ${DIR}/${FILE} &&
git commit -m "litellm-health: ${run_id}" && git push)
rm -rf /tmp/health-logs /tmp/${FILE}
```
+1 -1
View File
@@ -52,7 +52,7 @@ description: >
- If status is "online" → pass, log restarts count
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
- If restarts > 5 in last hour → alert owner with full diagnostics
4. **Log results**Create `[LEARN]` node in knowledge graph for any actions taken
4. **Log results**Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule)
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
6. **Wait 5 min** → repeat from step 1