SyslogSolution/prose-contracts
7
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2f65c38213518d662c61b5bce132c1dfa2cd2d0f
prose-contracts/litellm-self-heal.prose.md
T

15 KiB
Raw Blame History

report_only_agents, kind, name, status, note, description
report_only_agents kind name status note description
koby
responsibility litellm-self-heal deployed DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * * Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04), now reimplemented as `litellm-health-check.sh` on CT 116. Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`). Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs). GPU monitoring integrated from gpu-monitor on .24:9100. Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate duplication of architecture diagrams, GPU topology, timeout tables, and container lists. Health check is now § Health Check within this contract. Source of truth for GPU topology and keys: gpu-fleet.prose.md Last verified: 2026-07-12 LiteLLM inference stack health monitoring + self-healing. Verifies the full nginx → LiteLLM → GPU chain, 12 containers on CT 116, 3 GPU hosts, model inference, and agent keys. Applies remediation rules for common failures. Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).

LiteLLM Operations — Health Check + Self-Heal

Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001

  harness-router :9000 — DECOMMISSIONED 2026-09-11 (container, image
  and config removed). nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.

Parameters

  • public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
  • backend_host: string — Internal CT host (default: "192.168.68.116")
  • auth_host: string — Authentik OIDC server (default: "192.168.68.11")
  • gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
  • gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
  • grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

GPU Fleet Topology

Host IP Hardware Role
llm-gpu 192.168.68.8 NVIDIA RTX 3090 (24 GB) heavy reasoning (gpu-dense)
ocu-llm 192.168.68.110 NVIDIA RTX 5070 (12 GB) vision / web extract / light tasks (gpu-vision)
amdpve 192.168.68.15 AMD Strix Halo 64GB UMA compression (strix-moe)

Verified on the ground 2026-07-16. The legacy name ornith-1.0-35b does NOT exist in LiteLLM and must not be referenced.

LiteLLM Model Surface

Single source of truth for models, aliases, rpm caps, weights and fallback chains: CT 116 /opt/inference-harness/litellm_config.yaml. Do not duplicate those values in contracts — read them there. Do not re-add retired names (gemma-4-12b, gpu-light, crew-auto).

Alias Surface

Alias Serves Where Kind
gpu-dense heavy reasoning RTX 3090 (192.168.68.8) direct alias
gpu-vision vision / web extract / light tasks RTX 5070 (192.168.68.110) direct alias AND syslog-auto pool member
strix-moe compression (MoE) Strix Halo (192.168.68.15) direct alias
syslog-auto balanced default weighted pool across the three GPU hosts pool router

Context Cap Split (2026-08-20; crew cap RETIRED)

  • Abiba (firstmate): 128K uncapped — unlimited context for primary workloads
  • Hermes agents (mumuni, tanko, koby, koonimo): 128K uncapped
  • Crewmates (ops, tune, verify, auth-keys, build): the 64K cap was retired together with the crew-auto alias; NO context cap is currently in force.

The 64K crew cap was retired with crew-auto (2026-09-12). No limit is currently in force; reinstating one would need per-key model limits as a separate, deliberately-scoped change.

  • Key scoping: /v1/models is key-scoped, so the set a caller sees must be read with a named key rather than assumed — a monitor key, an agent key, and the master key can each return a different set. Agents should use the stable aliases (strix-moe, gpu-vision, gpu-dense) rather than raw model names, so model swaps don't break them.
  • NEVER use litellm_proxy_master_key (the sk-litellm-... master key) for inference. It is for admin endpoints only (/key/list, /key/generate, /key/info). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated monitor agent key stored at /etc/litellm-monitor.env on CT 116 (root-only, chmod 600); /key/list is the only call that legitimately uses $MASTER_KEY.

Containers on CT 116

Container Image Port Health Check
harness-litellm ghcr.io/docker.litellm.ai/berriai/litellm:1.99.1 :4000→:4000 /health/liveliness
harness-nginx nginx:alpine :80 HTTP 200 on /health
harness-postgres postgres:16-alpine :5432 pg_isready
harness-redis redis:7-alpine :6379 PING
harness-dashboard inference-harness-dashboard :3000 /health
harness-grafana grafana/grafana :3000→:3001 /api/health
harness-prometheus prom/prometheus :9090 /-/healthy
harness-docker-stats python:3.12-alpine — container stats exporter
harness-pve-exporter prompve/prometheus-pve-exporter — Proxmox metrics → Prometheus

Script Operations (synced 2026-07-16)

  • Health-check script (/opt/inference-harness/scripts/litellm-health-check.sh on CT 116): gpu-fleet check fails only on critical alerts (warnings are informational). Tests strix-moe (not ornith-1.0-35b).
  • GPU monitor (/root/scripts/gpu-monitor-server.py on pi .24): runs as systemd unit gpu-monitor.service (was bare & process). gpu_count includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
  • Agent key monitor (/root/scripts/agent-health-check.py on pi .24, cron */10): v4 (2026-09-10) — vault-backed agents (tanko/koby/koonimo) read their agent-specific {NAME}_LITELLM_API_KEY from Infisical vault (not the shared master key); abiba (pi agent) reads LITELLM_API_KEY from its local /root/.pi/agent/env.sh (#735 — moved out of shared /root/.bashrc), not from the vault. Abiba is pi-only since the harness purge, so its Hermes config/wrapper/gateway legs are skipped rather than reported as faults; koby is report-only (captain's 2026-08-17 ruling) — its findings go to the --json report_only array and are never counted as fleet failures or repaired, and its CT 111 liveness is probed on storepve (.6). Covers: LiteLLM keys, GPU ports, agent gateways, CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Every run/report carries the absolute execution path (script= + cwd=). The current fleet roster is owned by the script changelog (scripts/agent-health-check.py); mumuni is no longer probed from this host. Legacy tdunna/baggy replaced with canonical agent hostnames.
  • Stale keys cleaned: daily-infra-report.py SYNTHETIC_API_KEY was stale (sk-U_ydi3B → 401); now reads LITELLM_MASTER_KEY from env. Deprecated scripts (router-original.py, router-phase0-backup.py, apply-fixes.py) still reference sk-syslog-local-master-key but do not actively poll LiteLLM.

Maintains

  • litellm-admin-ui: { status: "healthy", last_check: timestamp }
  • litellm-api-docs: { status: "healthy", last_check: timestamp }
  • litellm-containers: { status: "healthy", last_check: timestamp }
  • litellm-oidc: { status: "healthy", last_check: timestamp }
  • litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
  • litellm-agent-keys: { count: int, valid: int }

Continuity

  • Self-driven: check every 300 seconds
  • Also wakes on user request
  • On failure: re-check after 30s, escalate after 3 consecutive failures

Health Check

Run this first on every cycle. Results feed into remediation rules below.

1. Check the end-user surfaces

The public edge and the backend edge serve the same app under different paths; they are not interchangeable, so every probe names the surface it targets.

Public edge — {{public_url}} serves the app at the ROOT (the /litellm/ prefix 404s):

  • GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
  • GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")
  • GET {{public_url}}/litellm/ui/ and {{public_url}}/litellm/docs → expect 404 (not served on this edge)

Backend edge — http://{{backend_host}} (port 80) serves the app UNDER /litellm/:

2. Check LiteLLM health (no-auth)

3. Check backend container health

  • SSH to {{backend_host}} → docker ps → verify 12 containers healthy (11 harness + trove-agent-docker, added 2026-09-11)
  • Critical: harness-litellm, harness-nginx, harness-postgres
  • Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-alertmanager, harness-zulip-bridge, harness-docker-stats, harness-pve-exporter, trove-agent-docker (ghcr.io/techdox/trove-agent-docker, added 2026-09-11)
  • Decommissioned 2026-09-11: harness-router (container, image and config removed)

4. Check GPU fleet health (via fleet dashboard)

  • GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
  • Verify GPUs reporting status "healthy"
  • Check alerts array for active warnings/critical

5. Check model inference via LiteLLM — test one model per GPU host (backend surface)

6. Check agent keys

7. Check Grafana

  • GET {{grafana_url}}/api/health → expect 200

8. Compile overall status

Determine overall_status from individual check results:

  • "healthy" — all checks pass
  • "degraded" — 1-2 non-critical checks fail
  • "down" — critical checks fail

Remediation Rules

Rule 1: Container Not Healthy

Detect → docker compose up -d <container> → verify → log Escalate after 3 failures

Rule 2: Admin UI Non-200

Detect → restart harness-litellm → verify → log Escalate after 2 failures

Rule 3: Auth Unreachable

Detect → immediate escalate (external dependency)

Rule 4: Docs 404

Detect → fix DOCS_URL env var → verify → log

Rule 5: GPU Unreachable

Detect → SSH to GPU host fails or model not responding Fix → restart llama-server on host via SSH Escalate → immediately (downtime affects inference)

Rule 6: Model Not Responding

Detect → test prompt via LiteLLM returns non-200 Fix → restart llama-server on GPU host → verify Escalate → after 2 failed restarts

Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED

Router is no longer in the request path (2026-07-08). LiteLLM proxies directly to GPU. This rule is retained for reference but is inactive. If 503 errors occur, check LiteLLM timeouts and GPU health directly.

Rule 8: Agent Keys Invalid (401)

Detect → agents report auth errors Root cause → keys not in LiteLLM DB Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts Escalate → if SSH access unavailable, send Zulip DM

Rule 9: Stale Active Counter in Redis — DEPRECATED

Router no longer in path so router active-slot counters are unused. Rule retained for reference but inactive. harness-redis now serves only LiteLLM cache and rate-limit state; check the container if cache errors appear.

Reporting

Every remediation cycle produces a structured report:

1. Gitea Log Entry

Pushed to SyslogSolution/health-logs/litellm/{run_id}.json — versioned, searchable, not in graph.

2. Zulip DM to Owner

  • issues_fixed > 0 — "🛠 LiteLLM Self-Heal — Fix Applied"
  • issues_escalated > 0 — "⚠ LiteLLM Self-Heal — Needs Your Attention"
  • Every 10th clean cycle — "✅ All Clear (10 checks passed)"

3. Daily Digest (end of day)

Summary of last 24 hours: total cycles, issues found/fixed/escalated, top actions, uptime.

4. Relay Message (if cross-agent)

If a fix requires another agent (e.g., Authentik restart), relay sent to responsible agent with full context.

Execution

-- Phase 1: Health Check
let health = call health-check
  public_url: public_url
  backend_host: backend_host
  gpu_dashboard_url: gpu_dashboard_url
  grafana_url: grafana_url

-- Phase 2: Apply remediation for each failure
let actions = []
for check in health.failed:
  let fix = apply-remediation-rule
    rule: lookup-rule(check.name)
    target: check.target
  push actions fix

-- Phase 3: Generate report
call report-generator
  health: health
  actions: actions

-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
call gitea-logger
  run_id: run_id
  repo: SyslogSolution/health-logs
  path: litellm/{run_id}.json
  health: health
  actions: actions

-- Phase 5: Notify if anything changed
if actions.length > 0:
  call zulip-notifier
    actions: actions
    health: health

-- Wait 300s and repeat

Audit Trail Format

{
  "run_id": "self-heal-20260709-001",
  "timestamp": "2026-07-09T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
  "issues_found": 0,
  "issues_fixed": 0,
  "issues_escalated": 0,
  "actions": []
}

With failures:

{
  "run_id": "self-heal-20260709-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
    {
      "rule": "Container Not Healthy",
      "target": "harness-nginx",
      "action": "restarted container",
      "result": "healthy",
      "duration_ms": 5000
    }
  ]
}

gitea-logger Implementation

When this step executes, write the report JSON to a temp file and push to Gitea:

REPO="https://abiba-bot:${GITEA_PAT}@git.sysloggh.net/SyslogSolution/health-logs"
DIR="litellm"
FILE="${run_id}.json"
echo "${report_json}" > /tmp/${FILE}
(cd /tmp && git clone --depth 1 "${REPO}" && 
  cp ${FILE} health-logs/${DIR}/${FILE} && 
  cd health-logs && git add ${DIR}/${FILE} && 
  git commit -m "litellm-health: ${run_id}" && git push)
rm -rf /tmp/health-logs /tmp/${FILE}
Reference in New Issue View Git Blame Copy Permalink