Compare commits

..
Author SHA1 Message Date
root fb52aff6a0 fix: update Mumuni CT reference to 100 (Hermes gateway on CT 100) 2026-07-28 15:49:51 +00:00
5 changed files with 11 additions and 30 deletions
+2 -2
View File
@@ -251,8 +251,8 @@ call update-gpu-health
## Reporting ## Reporting
### 1. Gitea Log (not knowledge graph — hard rule) ### 1. Knowledge Graph
Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchable, not in graph. Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
### 2. Zulip Alerts (#agent-hub → alerts-gpu) ### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved" - `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
+1 -1
View File
@@ -34,7 +34,7 @@ description: >
┌─────────────┐ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐ ┌──────────────┐
│ Abiba │ │ Tanko │ │ Mumuni │ │ Abiba │ │ Tanko │ │ Mumuni │
│ (pi) │ │ (Hermes) │ │ (Hermes) │ │ (pi) │ │ (Hermes) │ │ (Hermes) │
│ CT 100 │ │ CT 112 │ │ CT 114 │ CT 100 │ │ CT 112 │ │ CT 100
└──────┬──────┘ └──────┬───────┘ └──────┬───────┘ └──────┬──────┘ └──────┬───────┘ └──────┬───────┘
│ │ │ │ │ │
└──────────────────┼────────────────────┘ └──────────────────┼────────────────────┘
+6 -25
View File
@@ -7,7 +7,7 @@ note: >
Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04), Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
now reimplemented as `litellm-health-check.sh` on CT 116. now reimplemented as `litellm-health-check.sh` on CT 116.
Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`). Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs). Reports to /var/log/litellm/health-*.json and RA-H OS knowledge graph.
GPU monitoring integrated from gpu-monitor on .24:9100. GPU monitoring integrated from gpu-monitor on .24:9100.
Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
@@ -20,7 +20,7 @@ description: >
LiteLLM inference stack health monitoring + self-healing. Verifies the full LiteLLM inference stack health monitoring + self-healing. Verifies the full
nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
inference, and agent keys. Applies remediation rules for common failures. inference, and agent keys. Applies remediation rules for common failures.
Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs). Reports every action via Zulip DM and RA-H OS knowledge graph.
--- ---
# LiteLLM Operations — Health Check + Self-Heal # LiteLLM Operations — Health Check + Self-Heal
@@ -211,8 +211,8 @@ for reference but inactive. If Redis issues occur, check harness-redis container
Every remediation cycle produces a structured report: Every remediation cycle produces a structured report:
### 1. Gitea Log Entry ### 1. RA-H OS Knowledge Graph Node
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph. Created as `[LEARN] litellm-self-heal: <run_id>` with full JSON report.
### 2. Zulip DM to Owner ### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied" - `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
@@ -252,11 +252,9 @@ call report-generator
health: health health: health
actions: actions actions: actions
-- Phase 4: Log to Gitea (not knowledge graph — hard rule) -- Phase 4: Log to knowledge graph
call gitea-logger call kg-logger
run_id: run_id run_id: run_id
repo: SyslogSolution/health-logs
path: litellm/{run_id}.json
health: health health: health
actions: actions actions: actions
@@ -302,20 +300,3 @@ With failures:
] ]
} }
``` ```
## gitea-logger Implementation
When this step executes, write the report JSON to a temp file and push to Gitea:
```bash
REPO="https://abiba-bot:${GITEA_PAT}@git.sysloggh.net/SyslogSolution/health-logs"
DIR="litellm"
FILE="${run_id}.json"
echo "${report_json}" > /tmp/${FILE}
(cd /tmp && git clone --depth 1 "${REPO}" &&
cp ${FILE} health-logs/${DIR}/${FILE} &&
cd health-logs && git add ${DIR}/${FILE} &&
git commit -m "litellm-health: ${run_id}" && git push)
rm -rf /tmp/health-logs /tmp/${FILE}
```
+1 -1
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns. protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Pi + Hermes Zulip gateway). Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Hermes Zulip gateway).
version: 1.0.0 version: 1.0.0
--- ---
+1 -1
View File
@@ -52,7 +52,7 @@ description: >
- If status is "online" → pass, log restarts count - If status is "online" → pass, log restarts count
- If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately - If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
- If restarts > 5 in last hour → alert owner with full diagnostics - If restarts > 5 in last hour → alert owner with full diagnostics
4. **Log results**Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule) 4. **Log results**Create `[LEARN]` node in knowledge graph for any actions taken
5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting) 5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
6. **Wait 5 min** → repeat from step 1 6. **Wait 5 min** → repeat from step 1