From c1d993fe01fe5c2abb65f1be6732477670edf87a Mon Sep 17 00:00:00 2001 From: Abiba Date: Thu, 2 Jul 2026 20:39:17 +0000 Subject: [PATCH] fix: GPU table correction + OpenProse v0.15 alignment + self-heal rules 7-9 - hello-world: Ensures -> Maintains (v0.15.0 rename). - zulip-mention-reliability: fold Success Criteria into Maintains postconditions. - litellm-self-heal: swap gemma/qwen GPU hosts to match actual deployment; add Strix Halo 64GB UMA detail; add Rules 7 (roster stuck), 8 (agent keys invalid/401), 9 (stale Redis active counter). --- hello-world.prose.md | 2 +- litellm-self-heal.prose.md | 24 +++++++++++++++++++++--- zulip-mention-reliability.prose.md | 15 +++++++-------- 3 files changed, 29 insertions(+), 12 deletions(-) diff --git a/hello-world.prose.md b/hello-world.prose.md index b960d5a..8b4edd6 100644 --- a/hello-world.prose.md +++ b/hello-world.prose.md @@ -10,6 +10,6 @@ description: A simple hello world contract to test OpenProse on pi ## Returns - greeting: string — The generated greeting message -## Ensures +## Maintains - The greeting includes the provided name - The greeting is friendly and warm diff --git a/litellm-self-heal.prose.md b/litellm-self-heal.prose.md index 1713f23..bffd6be 100644 --- a/litellm-self-heal.prose.md +++ b/litellm-self-heal.prose.md @@ -70,9 +70,9 @@ message is sent to the responsible agent with full context. | Host | IP | Hardware | Models | Engine | |------|-----|----------|--------|--------| -| amdpve | 192.168.68.15 | Strix Halo CPU | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 | -| llm-gpu | 192.168.68.8 | RTX 3090 24GB | gemma-4-12b Dense | llama-server :8080 | -| ocu-llm | 192.168.68.110 | RTX 5070 12GB | qwen3.6-27B-code MoE+Light | llama-server :8080 | +| amdpve | 192.168.68.15 | Strix Halo 64GB UMA | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 | +| llm-gpu | 192.168.68.8 | RTX 3090 24GB | qwen3.6-27B-code | llama-server :8080 | +| ocu-llm | 192.168.68.110 | RTX 5070 12GB | gemma-4-12b | llama-server :8080 | ## Remediation Rules @@ -100,6 +100,24 @@ Detect → test prompt via LiteLLM returns non-200 Fix → restart llama-server on GPU host → verify Escalate → after 2 failed restarts +### Rule 7: Router Returns 503 (All GPUs Saturated) +Detect → router logs show QUEUE_TIMEOUT with GPUs idle +Root cause → roster not loaded at startup (empty TIER_MODELS) +Fix → POST /admin/roster/reload (if endpoint available) or restart router +Permanent fix → add load_roster() to router startup code +Escalate → if reload doesn't resolve within 2 min + +### Rule 8: Agent Keys Invalid (401) +Detect → agents report auth errors +Root cause → keys in router env var, not in LiteLLM DB +Fix → generate keys in LiteLLM via /key/generate → update /etc/environment on agent hosts +Escalate → if SSH access unavailable, send Zulip DM + +### Rule 9: Stale Active Counter in Redis +Detect → active: > 0 while GPU idle +Fix → `redis-cli DEL active:` → restart router +Prevent → add TTL to active keys in router code + ## Execution 1. Run health check diff --git a/zulip-mention-reliability.prose.md b/zulip-mention-reliability.prose.md index b7c8513..9b28ef7 100644 --- a/zulip-mention-reliability.prose.md +++ b/zulip-mention-reliability.prose.md @@ -14,14 +14,13 @@ description: > - last_test_result: "pass" | "fail" | "degraded" - known_failure_modes: array — History of what went wrong and how it was fixed - queue_health: "healthy" | "expired" | "reconnecting" - -## Success Criteria - -- test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s -- stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat -- queue_active == true — PM2's event queue is registered and polling -- no_echo_loop == true — Bot doesn't respond to its own messages -- dm_always_works == true — DMs still respond even if streams are broken + + **Postconditions** + - test_mention_gets_response == true — Send @Abiba Bot test, confirm response within 30s + - stream_subscription_active == true — Bot is subscribed to #agent-hub and #general chat + - queue_active == true — PM2's event queue is registered and polling + - no_echo_loop == true — Bot doesn't respond to its own messages + - dm_always_works == true — DMs still respond even if streams are broken ## Known Failure Modes (from History)