fix: GPU table correction + OpenProse v0.15 alignment + self-heal rules 7-9
- hello-world: Ensures -> Maintains (v0.15.0 rename). - zulip-mention-reliability: fold Success Criteria into Maintains postconditions. - litellm-self-heal: swap gemma/qwen GPU hosts to match actual deployment; add Strix Halo 64GB UMA detail; add Rules 7 (roster stuck), 8 (agent keys invalid/401), 9 (stale Redis active counter).
This commit is contained in:
@@ -70,9 +70,9 @@ message is sent to the responsible agent with full context.
|
||||
|
||||
| Host | IP | Hardware | Models | Engine |
|
||||
|------|-----|----------|--------|--------|
|
||||
| amdpve | 192.168.68.15 | Strix Halo CPU | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
|
||||
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | gemma-4-12b Dense | llama-server :8080 |
|
||||
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | qwen3.6-27B-code MoE+Light | llama-server :8080 |
|
||||
| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
|
||||
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
|
||||
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
|
||||
|
||||
## Remediation Rules
|
||||
|
||||
@@ -100,6 +100,24 @@ Detect → test prompt via LiteLLM returns non-200
|
||||
Fix → restart llama-server on GPU host → verify
|
||||
Escalate → after 2 failed restarts
|
||||
|
||||
### Rule 7: Router Returns 503 (All GPUs Saturated)
|
||||
Detect → router logs show QUEUE_TIMEOUT with GPUs idle
|
||||
Root cause → roster not loaded at startup (empty TIER_MODELS)
|
||||
Fix → POST /admin/roster/reload (if endpoint available) or restart router
|
||||
Permanent fix → add load_roster() to router startup code
|
||||
Escalate → if reload doesn't resolve within 2 min
|
||||
|
||||
### Rule 8: Agent Keys Invalid (401)
|
||||
Detect → agents report auth errors
|
||||
Root cause → keys in router env var, not in LiteLLM DB
|
||||
Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
|
||||
Escalate → if SSH access unavailable, send Zulip DM
|
||||
|
||||
### Rule 9: Stale Active Counter in Redis
|
||||
Detect → active:<model> > 0 while GPU idle
|
||||
Fix → `redis-cli DEL active:<model>` → restart router
|
||||
Prevent → add TTL to active keys in router code
|
||||
|
||||
## Execution
|
||||
|
||||
1. Run health check
|
||||
|
||||
Reference in New Issue
Block a user