Compare commits

..
Author SHA1 Message Date
root fb52aff6a0 fix: update Mumuni CT reference to 100 (Hermes gateway on CT 100) 2026-07-28 15:49:51 +00:00
5 changed files with 11 additions and 13 deletions
+2 -2
View File
@@ -251,8 +251,8 @@ call update-gpu-health
## Reporting
### 1. Gitea Log (not knowledge graph — hard rule)
Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph.
### 1. Knowledge Graph
Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
+1 -1
View File
@@ -34,7 +34,7 @@ description: >
┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Abiba │ │ Tanko │ │ Mumuni │
│ (pi) │ │ (Hermes) │ │ (Hermes) │
│ CT 100 │ │ CT 112 │ │ CT 114
│ CT 100 │ │ CT 112 │ │ CT 100
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└──────────────────┼────────────────────┘
+6 -8
View File
@@ -7,7 +7,7 @@ note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116.
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
@@ -20,7 +20,7 @@ description: >
LiteLLM inference stack health monitoring + self-healing. Verifies the full
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
inference, and agent keys. Applies remediation rules for common failures.
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
Reports every action via Zulip DM and RA-H OS knowledge graph.
---
# LiteLLM Operations — Health Check + Self-Heal
@@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container
Every remediation cycle produces a structured report:
### 1. Gitea Log Entry
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.
### 1. RA-H OS Knowledge Graph Node
Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
@@ -252,11 +252,9 @@ call report-generator
health: health
actions: actions
-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
call gitea-logger
-- Phase 4: Log to knowledge graph
call kg-logger
run_id: run_id
repo: SyslogSolution/health-logs
path: litellm/{run_id}.json
health: health
actions: actions
+1 -1
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Pi + Hermes Zulip gateway).
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Hermes Zulip gateway).
version: 1.0.0
---
+1 -1
View File
@@ -52,7 +52,7 @@ description: >
- If status is "online" → pass, log restarts count
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
- If restarts > 5 in last hour → alert owner with full diagnostics
4. **Log results**Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule)
4. **Log results**Create `[LEARN]` node in knowledge graph for any actions taken
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
6. **Wait 5 min** → repeat from step 1