From 97f1cd77e41e22259b6e9f00cedfc970dc4a86e0 Mon Sep 17 00:00:00 2001 From: Abiba Date: Thu, 16 Jul 2026 17:23:37 +0000 Subject: [PATCH] litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Health-check script: critical-only gpu-fleet alerting, strix-moe model test. gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090). agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys). daily-infra-report.py: stale SYNTHETIC key → env-based master key. --- litellm-self-heal.prose.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index a3122cc..07398e3 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -97,6 +97,13 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2) | harness-docker-stats | python:3.12-alpine | — | container stats exporter | | harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus | +## Script Operations (synced 2026-07-16) + +- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`). +- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 256K context steady-state is ~96% on RTX 3090, not a fault). +- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed). +- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM. + ## Maintains - litellm-admin-ui: { status: "healthy", last_check: timestamp }