abiba-bot/prose-contracts
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

litellm-self-heal: document 2026-07-16 ops sync (script fixes, systemd gpu-monitor, live-key agent check)

Browse Source 
Health-check script: critical-only gpu-fleet alerting, strix-moe model test.
gpu-monitor: systemd unit (was bare &), Strix counted in gpu_count, VRAM
thresholds raised (93/97 — 256K ctx steady-state ~96% on RTX 3090).
agent-health-check.py: reads live LITELLM_API_KEY from gateway env (no hardcoded keys).
daily-infra-report.py: stale SYNTHETIC key → env-based master key.
This commit is contained in:
Abiba
2026-07-16 17:23:37 +00:00
parent dc572889f8
commit 97f1cd77e4
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
litellm-self-heal.prose.md
+7
View File
@@ -97,6 +97,13 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |
## Script Operations (synced 2026-07-16)
- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 256K context steady-state is ~96% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): reads each agent's **live** `LITELLM_API_KEY` from its gateway process env via SSH — never hardcodes keys (hardcoded keys rot on rotation and caused 9×401/30min). Fleet roster: abiba, tanko, mumuni, koby, koonimo (legacy `tdunna`/`baggy` removed — never existed).
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.
## Maintains
- litellm-admin-ui: { status: "healthy", last_check: timestamp }