---
kind: responsibility
name: gpu-self-heal
description: >
  GPU fleet self-healing — detects anomalies, applies remediation, tracks
  benchmarks, and predicts failures before they happen. Extends gpu-monitor
  (v2.1.0) with active remediation rules, Prometheus metrics consumption,
  VRAM trend analysis, and predictive alerting.
  UPDATED 2026-07-18: Model assignments synced to 2026-07-17 swaps.
  Router (port 9000) references replaced with direct GPU routing.
  Benchmark baselines refreshed to live values.
  Prometheus exporters removed — not deployed; fall back to direct sidecar probes.
  Stable role-based aliases (strix-moe, gpu-dense, gpu-light) from gpu-fleet.
agent: abiba
depends_on:
  - gpu-monitor.prose.md (live data source on .24:9100)
  - gpu-fleet.prose.md (source of truth for topology, aliases, model assignments)
---

## Maintains

- gpu-health: { status: "healthy"|"degraded"|"down", issues: array, actions: array }
- gpu-self-heal-log: array of { timestamp, gpu, issue, action, result } — audit trail
- benchmark-regression: { gpu, baseline_tok_sec, current_tok_sec, trend, alerts }
- vram-trend: { gpu, current_mb, rate_mb_per_hour, projected_full_in_hours }
- circuit-breaker-status: { gpu, open, auto_reset_attempted, last_reset }

## Requires

- gpu-monitor:function — Live fleet data from localhost:9100/gpu-data
- Direct sidecar probe access to all GPU hosts (:8080/health)
- SSH access to GPU hosts for restart operations

## Continuity

- Self-driven: check every 60 seconds against GPU monitor data
- Also wakes on gpu-fleet health degradation
- On fix: verify with benchmark inference test before declaring resolved
- Escalate: after 3 failed remediation attempts → Zulip #agent-hub alert

---

## Current Fleet Baseline (2026-07-18)

| Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role |
|-------|-----|------|-------|------|-----|-------|------|
| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |

Key notes:
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
- Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated.
- RTX 5070 tok/s is 2.3x faster than RTX 3090 for its model — gpu-light is the fastest endpoint. Route vision/web/light work there first.
- Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads.
- RTX 3090 VRAM at 88% — within role-appropriate range (role = heavy reasoning, needs the headroom).
- RTX 5070 VRAM at 83% — role-appropriate for vision/web (smaller batch sizes).

## Remediation Rules

### Rule 1: GPU Temperature Critical (>85°C for >2 min)
- **Detect**: Any GPU temp >85°C sustained for 2+ consecutive polls
- **Fix**: 
  1. Reduce inference concurrency on that GPU (load-side cooling only — NO fan control)
  2. Redirect new requests to cooler GPUs via LiteLLM fallback chains (gemma → qwen, qwen → gemma)
  3. If all GPUs hot, alert about cooling infrastructure
- **Verify**: Temp drops below 80°C within 5 minutes
- **Escalate after**: 3 verification failures → Zulip alert

### Rule 2: VRAM Leak Detection (tiered by GPU capacity)
- **Detect**: VRAM growing at sustained rate over 6+ hour window
  - RTX 3090 (24GB): ≥300MB/hour
  - RTX 5070 (12GB): ≥300MB/hour
  - Strix Halo (64GB UMA): ≥200MB/hour
- **Fix**: 
  1. Log VRAM snapshot with process list (nvidia-smi/rocm-smi + ps aux)
  2. If llama-server is the growth source → restart with memory cap flag
  3. If unknown process → kill and alert
- **Verify**: VRAM growth rate drops below threshold
- **Escalate after**: persistent leak after restart → hardware investigation

### Rule 3: Model Inference Timeout / GPU Stuck
- **Detect**: >50% failure rate over 60s window + 30s grace period (not just single stuck request)
- **Fix**: 
  1. Restart llama-server on affected GPU host
  2. Wait 15s for model to reload
  3. Run benchmark inference test
- **Verify**: Model returns 200 with <30s response, failure rate drops to 0%
- **Escalate after**: 3 restarts in 1 hour → GPU hardware check

### Rule 4: Benchmark Regression (>20% drop)
- **Detect**: gen_tok_per_sec drops >20% below baseline over 3+ benchmarks
  - RTX 3090 baseline: 74.8 tok/s → alert at <59.8 tok/s
  - RTX 5070 baseline: 165.2 tok/s → alert at <132.2 tok/s
  - Strix Halo baseline: 70.5 tok/s → alert at <56.4 tok/s
- **Fix**:
  1. Check GPU utilization — if >90%, other process is competing
  2. Check power limit — if throttled, restore to max
  3. Check thermal — if hot, apply Rule 1
- **Verify**: Benchmark returns to within 10% of baseline
- **Escalate after**: persistent regression → possible hardware degradation

### Rule 5: Circuit Breaker Stuck Open
- **Detect**: Circuit breaker open >10 minutes with GPU reporting healthy
- **Note**: Router (port 9000) is deprecated. If circuit breakers are reported by gpu-monitor, they come from LiteLLM's internal tracking, not the old router.
- **Fix**: 
  1. Verify GPU /health returns 200 on direct port (:8080)
  2. If GPU healthy, alert but do NOT reset via router API (deprecated)
  3. Check LiteLLM health directly: http://192.168.68.116/litellm/health/liveliness
  4. Restart LiteLLM container on CT 116 if circuit breakers are stuck
- **Verify**: LiteLLM returns healthy, circuit breaker clears within 60s
- **Escalate after**: LiteLLM restart doesn't clear → human investigation

### Rule 6: Strix Halo Unreachable
- **Detect**: Strix not responding — probe .15:8080 directly (firewall opened .24→.15)
- **Fix**: 
  1. SSH to .15 → check llama-server process
  2. Restart llama-server if not running
  3. Verify through both direct probe AND LiteLLM health
- **Verify**: Direct health probe returns 200, LiteLLM reports model healthy
- **Escalate**: If host .15 itself is unreachable → infrastructure alert

### Rule 7: GPU Data Source Unreachable (replaces old Prometheus rule)
- **Detect**: gpu-monitor endpoint (localhost:9100/gpu-data) or sidecar port (:8080) on any GPU unreachable for >2 polls
- **Fix**:
  1. If gpu-monitor is down: restart systemd service `gpu-monitor.service` on this host
  2. If sidecar is down: SSH to GPU host → check llama-server process → restart systemd service
  3. Fall back to direct nvidia-smi/rocm-smi probe via SSH if all API paths fail
- **Verify**: gpu-monitor returns healthy + all sidecars reachable
- **Escalate after**: 3 failed restarts → networking issue

### Rule 8: Predictive Thermal Warning (two-tier)
- **Detect**: 
  - Tier 1 (warning): temp >70°C AND rising >2°C/min → reduce concurrency, no alert
  - Tier 2 (critical): temp >80°C AND still rising → full alert + aggressive load shedding
- **Fix**: 
  - Tier 1: silently reduce parallel requests to that GPU by 50%
  - Tier 2: redirect all new requests away, alert #agent-hub, apply Rule 1 logic
- **Verify**: Temp rise rate drops below 1°C/min (Tier 1) or temp drops below 80°C (Tier 2)
- **Escalate**: If Tier 2 triggers and temp still rising after 5 min → possible hardware failure

### Rule 9: Context Window Optimization
- **Detect**: Benchmark tok/s vs baseline for each GPU at current context (all 128K)
  - RTX 3090 (128K ctx, ThinkingCap): baseline 74.8 tok/s — currently at 74.9 (100%)
  - RTX 5070 (128K ctx, HauhauCS QAT): baseline 165.2 tok/s — currently at 169.6 (103%)
  - Strix Halo (128K ctx, qwen3.6-35B-udq4): baseline 70.5 tok/s — currently at 62.9 (89%)
- **Fix**:
  - If tok/s > baseline → context has headroom, consider increasing
  - If tok/s < 90% baseline → reduce context by 25% and retest
  - If tok/s within 10% of baseline → optimal, no change
  - Strix Halo at 89% of baseline → MONITOR but do not reduce yet (recent model swap may still be settling)
- **Verify**: Re-benchmark after context change, confirm within 10% of target
- **Escalate**: If context can't be adjusted without significant perf loss

### Rule 10: Workload Distribution Optimization (updated 2026-07-18)
- **Detect**: GPU roles misaligned with hardware capabilities
- **Target distribution**:
  - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
  - RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
  - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
- **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
- **Fix**:
  - Alert if any GPU is handling workload outside its designated role
  - Recommend agent alias updates to match workload to GPU role (use stable aliases: gpu-dense, gpu-light, strix-moe)
  - Track per-GPU request distribution via LiteLLM spend logs
- **Verify**: Each GPU's request pattern matches its designated role within 24h
- **Escalate**: If role mismatch persists >48h → agent alias audit needed

---

## Execution

```prose
-- Phase 1: Fetch live GPU data
let fleet = call gpu-monitor
  endpoint: "http://localhost:9100/gpu-data"

-- Phase 2: Evaluate each GPU against remediation rules
let actions = []
for gpu in fleet.gpus:
  -- Rule 1: Thermal critical
  if gpu.temp_c > 85 and sustained_for(gpu, 120):
    push actions apply-thermal-fix(gpu)
  
  -- Rule 2: VRAM leak
  let vram_rate = calculate-vram-trend(gpu, hours=6)
  if vram_rate > 50:
    push actions apply-vram-fix(gpu, vram_rate)
  
  -- Rule 4: Benchmark regression
  let bench = fleet.benchmarks[gpu.hostname]
  if bench.current_tok_s < bench.baseline_tok_s * 0.8:
    push actions apply-benchmark-fix(gpu, bench)

-- Rule 3: Model stuck
for model in fleet.summary.available_models:
  if model.consecutive_timeouts >= 3:
    push actions apply-model-restart(model)

-- Rule 5: Circuit breaker check via LiteLLM (router deprecated)
if fleet.summary.circuit_breakers_open > 0:
  push actions check-litellm-circuit-breakers()

-- Rule 6: Strix Halo
if not fleet.strix.running and pingable("192.168.68.15"):
  push actions apply-strix-restart()

-- Rule 7: GPU data source
if not fleet.gpus or len(fleet.gpus) < 2:
  push actions check-gpu-monitor-service()

-- Rule 8: Predictive thermal
for gpu in fleet.gpus:
  let rise_rate = calculate-temp-rise(gpu, minutes=5)
  if rise_rate > 2.0 and gpu.temp_c < 80:
    push actions apply-proactive-cooling(gpu)

-- Phase 3: Execute actions, verify, log
for action in actions:
  let result = execute-with-verify(action)
  log-to-kg(action, result)
  if result.failed:
    escalate-if-needed(action)

-- Phase 4: Update health state
call update-gpu-health
  gpus: fleet.gpus
  actions: actions
  status: derive-overall-status(fleet, actions)

-- Wait 60s and repeat
```

## Audit Trail Format

```json
{
  "run_id": "gpu-self-heal-20260718-001",
  "timestamp": "2026-07-18T08:00:00Z",
  "gpu": "ct8-rtx3090",
  "issue": "thermal-critical",
  "detected": { "temp_c": 87, "duration_s": 180 },
  "action": "load-shedding",
  "result": "resolved",
  "verification": { "temp_c": 76, "after_s": 300 },
  "escalated": false
}
```

---

## Reporting

### 1. Knowledge Graph
Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.

### 2. Zulip Alerts (#agent-hub → alerts-gpu)
- `issues_fixed > 0` → "🛠 GPU Self-Heal — <gpu> <issue> resolved"
- `issues_escalated > 0` → "⚠ GPU Self-Heal — <gpu> needs attention"
- Every 100th clean cycle → "✅ GPU Fleet: All Clear"

### 3. Weekly Benchmark Report
- Per-GPU tok/s trend over 7 days
- Regression alerts if any GPU degrades >10% week-over-week

---

## Design Decisions (Verified 2026-07-12, Reaffirmed 2026-07-18)

1. **Fan control**: ❌ NO auto fan control. Load-side cooling only (reduce concurrency, redirect).
2. **Model restart**: ✅ Only if >50% failure rate over 60s + 30s grace period. Not on single stuck request.
3. **Strix direct access**: ✅ Open firewall .15:8080 → .24 for direct health probe + restart.
4. **VRAM thresholds**: Tiered — **300MB/h** (RTX 3090), **300MB/h** (RTX 5070), 200MB/h (Strix). Previous values (100/50) were too sensitive; raised 2026-07-18 based on operational data.
5. **CB auto-reset**: ✅ Router deprecated — circuit breakers go through LiteLLM health check + container restart if needed. No per-GPU auto-reset.
6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Current baselines live in gpu-monitor.
7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
8. **Prometheus**: ❌ Not deployed. Use direct sidecar probes (:8080/health) and gpu-monitor API. Prometheus integration deferred until exporters are running on GPU hosts.

## Lessons Learned (2026-07-12, Updated 2026-07-18)

### L1: API Key Standardization Is Critical
- All GPU llama-servers MUST use the same api-key as the LiteLLM config.
- RTX 5070 had `--api-key sk-loc...5678` while LiteLLM sent `not-needed`.
  This caused cascading 401 → fallback → timeout → 401 loops.
- **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config (`not-needed` for direct routing).

### L2: Fallback Chain Cascading Failures
- When one model returns 401 (auth) and another is slow (timeout), the fallback
  chain creates an infinite loop.
- **Rule**: If a model returns 401 (auth error), do NOT fall back to it again.
  Mark it as permanently failed for this request.

### L3: Verify Running State, Not Docs
- RTX 3090 was documented at 128K context. Running at 128K (verified 2026-07-18).
- Parallel count: 1 on both RTX 3090 and RTX 5070 (matches docs for current models).
- **Rule**: Before making decisions, check `/proc/PID/cmdline` on GPU hosts.

### L4: Infisical Is Not Always Available
- Keep a local `.env` fallback for `LITELLM_API_KEY`.
- **Rule**: Always verify credential source is reachable before relying on it.

### L5: GPU Monitor Response Size Can Cause Self-Heal Crash
- gpu-self-heal crashed with KeyboardInterrupt during json.loads() of 20MB response.
- Root cause: router poll returns accumulated data → cache balloons.
- **Rule**: Self-heal must enforce a read timeout AND max response size on every poll.
  If monitor response > 1MB, log a warning and skip the cycle rather than crashing.

### L6: Stable Aliases Replace Model Names
- gpu-fleet introduced stable aliases (strix-moe, gpu-dense, gpu-light) on 2026-07-15.
- Self-heal must use aliases for reporting and alerting, not model-specific names.
- **Rule**: All alert messages and KG nodes use the stable alias as the GPU identifier.