feat: tighten contracts with GPU fleet topology, model routing, Strix Halo details
litellm-health: added GPU fleet topology table, model routing map, 3-tier GPU checks (reachability, VRAM, temp, test inference) litellm-self-heal: added GPU fleet reference, rules for GPU unreachable and model not responding Infrastructure topology now fully documented: - amdpve: Strix Halo CPU (35B, 16 threads, 262K ctx, llama-server) - llm-gpu: RTX 3090 24GB (Dense tier) - ocu-llm: RTX 5070 12GB (MoE + Light tiers)
This commit is contained in:
@@ -66,6 +66,14 @@ logged as a `[REPORT]` knowledge graph node for long-term trending.
|
||||
If a fix required another agent's help (e.g., Authentik restart), a relay
|
||||
message is sent to the responsible agent with full context.
|
||||
|
||||
## GPU Fleet
|
||||
|
||||
| Host | IP | Hardware | Models | Engine |
|
||||
|------|-----|----------|--------|--------|
|
||||
| amdpve | 192.168.68.15 | Strix Halo CPU | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
|
||||
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | gemma-4-12b Dense | llama-server :8080 |
|
||||
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | qwen3.6-27B-code MoE+Light | llama-server :8080 |
|
||||
|
||||
## Remediation Rules
|
||||
|
||||
### Rule 1: Container Not Healthy
|
||||
@@ -82,6 +90,16 @@ Detect → immediate escalate (external dependency)
|
||||
### Rule 4: Docs 404
|
||||
Detect → fix DOCS_URL env var → verify → log
|
||||
|
||||
### Rule 5: GPU Unreachable
|
||||
Detect → SSH to GPU host fails or model not responding
|
||||
Fix → restart llama-server on host via SSH
|
||||
Escalate → immediately (downtime affects inference)
|
||||
|
||||
### Rule 6: Model Not Responding
|
||||
Detect → test prompt via LiteLLM returns non-200
|
||||
Fix → restart llama-server on GPU host → verify
|
||||
Escalate → after 2 failed restarts
|
||||
|
||||
## Execution
|
||||
|
||||
1. Run health check
|
||||
|
||||
Reference in New Issue
Block a user