fix: GPU table correction + OpenProse v0.15 alignment + self-heal rules 7-9

- hello-world: Ensures -> Maintains (v0.15.0 rename).
- zulip-mention-reliability: fold Success Criteria into Maintains postconditions.
- litellm-self-heal: swap gemma/qwen GPU hosts to match actual deployment;
  add Strix Halo 64GB UMA detail; add Rules 7 (roster stuck), 8 (agent keys
  invalid/401), 9 (stale Redis active counter).
This commit is contained in:
Abiba
2026-07-02 20:39:17 +00:00
parent dfacab12f5
commit c1d993fe01
3 changed files with 29 additions and 12 deletions
+1 -1
View File
@@ -10,6 +10,6 @@ description: A simple hello world contract to test OpenProse on pi
## Returns ## Returns
- greeting: string — The generated greeting message - greeting: string — The generated greeting message
## Ensures ## Maintains
- The greeting includes the provided name - The greeting includes the provided name
- The greeting is friendly and warm - The greeting is friendly and warm
+21 -3
View File
@@ -70,9 +70,9 @@ message is sent to the responsible agent with full context.
| Host | IP | Hardware | Models | Engine | | Host | IP | Hardware | Models | Engine |
|------|-----|----------|--------|--------| |------|-----|----------|--------|--------|
| amdpve | 192.168.68.15 | Strix Halo CPU | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 | | amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
| llm-gpu | 192.168.68.8 | RTX 3090 24GB | gemma-4-12b Dense | llama-server :8080 | | llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 |
| ocu-llm | 192.168.68.110 | RTX 5070 12GB | qwen3.6-27B-code MoE+Light | llama-server :8080 | | ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 |
## Remediation Rules ## Remediation Rules
@@ -100,6 +100,24 @@ Detect → test prompt via LiteLLM returns non-200
Fix → restart llama-server on GPU host → verify Fix → restart llama-server on GPU host → verify
Escalate → after 2 failed restarts Escalate → after 2 failed restarts
### Rule 7: Router Returns 503 (All GPUs Saturated)
Detect → router logs show QUEUE_TIMEOUT with GPUs idle
Root cause → roster not loaded at startup (empty TIER_MODELS)
Fix → POST /admin/roster/reload (if endpoint available) or restart router
Permanent fix → add load_roster() to router startup code
Escalate → if reload doesn't resolve within 2 min
### Rule 8: Agent Keys Invalid (401)
Detect → agents report auth errors
Root cause → keys in router env var, not in LiteLLM DB
Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts
Escalate → if SSH access unavailable, send Zulip DM
### Rule 9: Stale Active Counter in Redis
Detect → active:<model> > 0 while GPU idle
Fix → `redis-cli DEL active:<model>` → restart router
Prevent → add TTL to active keys in router code
## Execution ## Execution
1. Run health check 1. Run health check
+1 -2
View File
@@ -15,8 +15,7 @@ description: >
- known_failure_modes: array — History of what went wrong and how it was fixed - known_failure_modes: array — History of what went wrong and how it was fixed
- queue_health: "healthy" | "expired" | "reconnecting" - queue_health: "healthy" | "expired" | "reconnecting"
## Success Criteria **Postconditions**
- test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s - test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s
- stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat - stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat
- queue_active == true — PM2's event queue is registered and polling - queue_active == true — PM2's event queue is registered and polling