Compare commits

..
Author SHA1 Message Date
root c9359e1808 fix: redirect health check logs from knowledge graph to Gitea (hard rule)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Logs (LITELLM-HEALTH, GPU-SELF-HEAL, PM2) now pushed to SyslogSolution/health-logs
instead of creating orphan nodes in the shared knowledge graph.

- litellm-self-heal: Phase 4 now calls gitea-logger instead of kg-logger
- gpu-self-heal: Reporting section updated to Gitea path
- pm2-self-heal: Log step redirected to Gitea
- 271 existing orphan nodes remain in graph (no delete tool available)
2026-07-28 21:20:56 +00:00
mumuni-bot 6a0af728e9 Merge pull request 'fix: update all Mumuni IP references from .123 to .24 (inside Abiba CT100)' (#35) from fix/mumuni-ip-abiba into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-07-28 08:57:05 +00:00
mumuni-bot 835647e241 Merge pull request 'fix: correct quant to UD-Q3_K_XL (Q4 too large for 24GB VRAM)' (#34) from fix/gpu-dense-q3-correction into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-07-27 20:26:13 +00:00
5 changed files with 13 additions and 11 deletions
+2 -2
View File
@@ -251,8 +251,8 @@ call update-gpu-health
## Reporting
### 1. Knowledge Graph
Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
### 1. Gitea Log (not knowledge graph — hard rule)
Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph.
### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
+1 -1
View File
@@ -34,7 +34,7 @@ description: >
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Abiba │ │ Tanko │ │ Mumuni │
│ (pi) │ │ (Hermes) │ │ (Hermes) │
│ CT 100 │ │ CT 112 │ │ CT 100
│ CT 100 │ │ CT 112 │ │ CT 114
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└──────────────────┼────────────────────┘
+8 -6
View File
@@ -7,7 +7,7 @@ note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116.
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
@@ -20,7 +20,7 @@ description: >
LiteLLM inference stack health monitoring + self-healing. Verifies the full
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
inference, and agent keys. Applies remediation rules for common failures.
Reports every action via Zulip DM and RA-H OS knowledge graph.
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
---
# LiteLLM Operations — Health Check + Self-Heal
@@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container
Every remediation cycle produces a structured report:
### 1. RA-H OS Knowledge Graph Node
Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
### 1. Gitea Log Entry
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.
### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
@@ -252,9 +252,11 @@ call report-generator
health: health
actions: actions
-- Phase 4: Log to knowledge graph
call kg-logger
-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
call gitea-logger
run_id: run_id
repo: SyslogSolution/health-logs
path: litellm/{run_id}.json
health: health
actions: actions
+1 -1
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Hermes Zulip gateway).
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Pi + Hermes Zulip gateway).
version: 1.0.0
---
+1 -1
View File
@@ -52,7 +52,7 @@ description: >
- If status is "online" → pass, log restarts count
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
- If restarts > 5 in last hour → alert owner with full diagnostics
4. **Log results**Create `[LEARN]` node in knowledge graph for any actions taken
4. **Log results**Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule)
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
6. **Wait 5 min** → repeat from step 1