prose-contracts/litellm-self-heal.prose.md

---
kind: responsibility
name: litellm-self-heal
status: deployed
note: >
  DEPLOYED 2026-07-12 on CT 116 cron: 0 */6 * * *
  Auto-remediation code was removed from the pi Zulip extension (retired 2026-07-04),
  now reimplemented as `litellm-health-check.sh` on CT 116.
  Script: `/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116 (cron `0 */6 * * *`).
  Reports to /var/log/litellm/health-*.json and Gitea (SyslogSolution/health-logs).
  GPU monitoring integrated from gpu-monitor on .24:9100.
  
  Consolidated from litellm-health + litellm-self-heal on 2026-07-09 to eliminate
  duplication of architecture diagrams, GPU topology, timeout tables, and container
  lists. Health check is now § Health Check within this contract.
  
  Source of truth for GPU topology and keys: gpu-fleet.prose.md
  Last verified: 2026-07-12
description: >
  LiteLLM inference stack health monitoring + self-healing. Verifies the full
  nginx → LiteLLM → GPU chain, 8 containers on CT 116, 3 GPU hosts, model
  inference, and agent keys. Applies remediation rules for common failures.
  Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
---

# LiteLLM Operations — Health Check + Self-Heal

## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)

```
Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)
                        │
                  Key validation
                  Fallback chains
                  Budget tracking
                        │
                  Prometheus ← metrics
                        │
                    Grafana :3001

  harness-router :9000 — DEPRECATED, container still runs but
  NOT in request path. nginx routes /v1 → LiteLLM directly.
  Router slot booking + circuit breakers replaced by
  LiteLLM native fallbacks + timeouts.
```

## Parameters

- public_url: string — Public LiteLLM URL (default: "https://litellm.sysloggh.net")
- backend_host: string — Internal CT host (default: "192.168.68.116")
- auth_host: string — Authentik OIDC server (default: "192.168.68.11")
- gpu_hosts: array — GPU inference hosts (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
- gpu_dashboard_url: string — Fleet dashboard (default: "http://192.168.68.24:9100")
- grafana_url: string — Grafana dashboards (default: "http://192.168.68.116:3001")

## GPU Fleet Topology

| Host | IP | Hardware | Models Served | Engine | Context | Parallel |
|------|-----|----------|---------------|--------|---------|----------|
| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | qwen3.6-27B-code | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `-c 131072 --parallel 2 --ngl 99`) | **128K** | 2 |
| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | gemma-4-12b | llama-server systemd (`/home/llmuser/llama-wrapper.sh`, `--ctx-size 131072 --parallel 2`, IQ4_NL + MTP draft) | **128K** | 2 |
| amdpve | 192.168.68.15 | AMD Strix Halo 64GB UMA | qwen3.6-35B-udq4 (LiteLLM alias: `strix-moe`) | llama-server systemd (Vulkan) | 128K | 2 |

> Verified on ground 2026-07-16 via `curl /v1/models` on each host + `llama-wrapper.sh`. The AMD host's underlying model is `qwen3.6-35B-udq4`; LiteLLM exposes it under two `model_name`s: `qwen3.6-35B-udq4` and `strix-moe` (rpm 40). The legacy name `ornith-1.0-35b` does NOT exist in LiteLLM and must not be referenced.

## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116)

`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`.

- `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
- `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
- Key scoping: agent keys are restricted to `['syslog-auto','qwen3.6-27B-code','gemma-4-12b','strix-moe','gpu-dense','gpu-light']`. As of 2026-07-16 the `baggy`/`koby`/`mumuni`/`abiba-pi` keys ALSO include `qwen3.6-35B-udq4`; `abiba-pi` additionally includes `deepseek-v4-pro` (cloud fallback). `kagenz0`/`koonimo`/`pi-agents-unified` have the standard 6 only. Agents should still use the stable alias `strix-moe` (not the raw `qwen3.6-35B-udq4`) so model swaps don't break them.
- **NEVER use `litellm_proxy_master_key` (the `sk-litellm-...` master key) for inference.** It is for admin endpoints only (`/key/list`, `/key/generate`, `/key/info`). All inference — agent traffic, health-check model tests, monitor scripts — uses agent-specific keys. The health-check script's model tests use a dedicated `monitor` agent key stored at `/etc/litellm-monitor.env` on CT 116 (root-only, `chmod 600`); `/key/list` is the only call that legitimately uses `$MASTER_KEY`.

## Model Fallback Chains (LiteLLM)

| Primary | Timeout | Fallback | Timeout |
|---------|---------|----------|---------|
| qwen3.6-27B-code | 300s | gemma-4-12b | 120s |
| gemma-4-12b | 120s | qwen3.6-27B-code | 300s |
| qwen3.6-35B-udq4 / strix-moe | 300s | qwen → gemma | — |
| syslog-auto (balanced) | 300s | qwen → gemma | — |

> Global: request_timeout=300s, nginx proxy_read_timeout=600s

## Containers on CT 116

| Container | Image | Port | Health Check |
|-----------|-------|------|-------------|
| harness-litellm | berriai/litellm:1.90.0-rc.1 | :4000→:4001 | /health/liveliness |
| harness-router | inference-harness-router | :9000 (127.0.0.1) | /health (DEPRECATED — not in path) |
| harness-nginx | nginx:alpine | :80 | HTTP 200 on /health |
| harness-postgres | postgres:16-alpine | :5432 | pg_isready |
| harness-redis | redis:7-alpine | :6379 | PING |
| harness-dashboard | inference-harness-dashboard | :3000 | /health |
| harness-grafana | grafana/grafana | :3000→:3001 | /api/health |
| harness-prometheus | prom/prometheus | :9090 | /-/healthy |
| harness-docker-stats | python:3.12-alpine | — | container stats exporter |
| harness-pve-exporter | prompve/prometheus-pve-exporter | — | Proxmox metrics → Prometheus |

## Script Operations (synced 2026-07-16)

- **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
- **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
- **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.

## Maintains

- litellm-admin-ui: { status: "healthy", last_check: timestamp }
- litellm-api-docs: { status: "healthy", last_check: timestamp }
- litellm-containers: { status: "healthy", last_check: timestamp }
- litellm-oidc: { status: "healthy", last_check: timestamp }
- litellm-gpu-fleet: { status: "healthy", models: int, alerts: array }
- litellm-agent-keys: { count: int, valid: int }

## Continuity

- Self-driven: check every 300 seconds
- Also wakes on user request
- On failure: re-check after 30s, escalate after 3 consecutive failures

---

## Health Check

Run this first on every cycle. Results feed into remediation rules below.

### 1. Check public endpoints
- GET {{public_url}}/ui/ → expect 200 ("LiteLLM Dashboard")
- GET {{public_url}}/docs → expect 200 ("LiteLLM API - Swagger UI")

### 2. Check LiteLLM health (no-auth)
- GET http://{{backend_host}}/litellm/health/liveliness → expect 200

### 3. Check backend container health
- SSH to {{backend_host}} → `docker ps` → verify 10 containers healthy
- Critical: harness-litellm, harness-nginx, harness-postgres
- Monitoring: harness-redis, harness-dashboard, harness-grafana, harness-prometheus, harness-docker-stats, harness-pve-exporter
- Deprecated but running: harness-router (not in path, reference only)

### 4. Check GPU fleet health (via fleet dashboard)
- GET {{gpu_dashboard_url}}/gpu-data → expect 200 with GPU metrics JSON
- Verify GPUs reporting status "healthy"
- Check alerts array for active warnings/critical

### 5. Check model inference via LiteLLM — test each model
- POST /v1/chat/completions model=gemma-4-12b → expect 200
- POST /v1/chat/completions model=qwen3.6-27B-code → expect 200
- POST /v1/chat/completions model=strix-moe → expect 200
- Use master key for auth

### 6. Check agent keys
- GET /key/list with master key → verify all 6 agents have keys

### 7. Check Grafana
- GET {{grafana_url}}/api/health → expect 200

### 8. Compile overall status
Determine overall_status from individual check results:
- "healthy" — all checks pass
- "degraded" — 1-2 non-critical checks fail
- "down" — critical checks fail

---

## Remediation Rules

### Rule 1: Container Not Healthy
Detect → `docker compose up -d <container>` → verify → log
Escalate after 3 failures

### Rule 2: Admin UI Non-200
Detect → restart harness-litellm → verify → log
Escalate after 2 failures

### Rule 3: Auth Unreachable
Detect → immediate escalate (external dependency)

### Rule 4: Docs 404
Detect → fix DOCS_URL env var → verify → log

### Rule 5: GPU Unreachable
Detect → SSH to GPU host fails or model not responding
Fix → restart llama-server on host via SSH
Escalate → immediately (downtime affects inference)

### Rule 6: Model Not Responding
Detect → test prompt via LiteLLM returns non-200
Fix → restart llama-server on GPU host → verify
Escalate → after 2 failed restarts

### Rule 7: Router Returns 503 (All GPUs Saturated) — DEPRECATED
Router is no longer in the request path (2026-07-08). LiteLLM proxies
directly to GPU. This rule is retained for reference but is inactive.
If 503 errors occur, check LiteLLM timeouts and GPU health directly.

### Rule 8: Agent Keys Invalid (401)
Detect → agents report auth errors
Root cause → keys not in LiteLLM DB
Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
Escalate → if SSH access unavailable, send Zulip DM

### Rule 9: Stale Active Counter in Redis — DEPRECATED
Router no longer in path so Redis active counters are unused. Rule retained
for reference but inactive. If Redis issues occur, check harness-redis container.

---

## Reporting

Every remediation cycle produces a structured report:

### 1. Gitea Log Entry
Pushed to `SyslogSolution/health-logs/litellm/{run_id}.json` — versioned, searchable, not in graph.

### 2. Zulip DM to Owner
- `issues_fixed > 0` — "🛠 LiteLLM Self-Heal — Fix Applied"
- `issues_escalated > 0` — "⚠ LiteLLM Self-Heal — Needs Your Attention"
- Every 10th clean cycle — "✅ All Clear (10 checks passed)"

### 3. Daily Digest (end of day)
Summary of last 24 hours: total cycles, issues found/fixed/escalated,
top actions, uptime.

### 4. Relay Message (if cross-agent)
If a fix requires another agent (e.g., Authentik restart), relay sent
to responsible agent with full context.

---

## Execution

```prose
-- Phase 1: Health Check
let health = call health-check
  public_url: public_url
  backend_host: backend_host
  gpu_dashboard_url: gpu_dashboard_url
  grafana_url: grafana_url

-- Phase 2: Apply remediation for each failure
let actions = []
for check in health.failed:
  let fix = apply-remediation-rule
    rule: lookup-rule(check.name)
    target: check.target
  push actions fix

-- Phase 3: Generate report
call report-generator
  health: health
  actions: actions

-- Phase 4: Log to Gitea (not knowledge graph — hard rule)
call gitea-logger
  run_id: run_id
  repo: SyslogSolution/health-logs
  path: litellm/{run_id}.json
  health: health
  actions: actions

-- Phase 5: Notify if anything changed
if actions.length > 0:
  call zulip-notifier
    actions: actions
    health: health

-- Wait 300s and repeat
```

## Audit Trail Format

```json
{
  "run_id": "self-heal-20260709-001",
  "timestamp": "2026-07-09T14:00:00Z",
  "duration_ms": 1234,
  "checks_passed": 7,
  "checks_failed": 0,
  "issues_found": 0,
  "issues_fixed": 0,
  "issues_escalated": 0,
  "actions": []
}
```

With failures:
```json
{
  "run_id": "self-heal-20260709-002",
  "issues_found": 1,
  "issues_fixed": 1,
  "actions": [
    {
      "rule": "Container Not Healthy",
      "target": "harness-nginx",
      "action": "restarted container",
      "result": "healthy",
      "duration_ms": 5000
    }
  ]
}
```