Merge branch 'master' of https://git.sysloggh.net/SyslogSolution/prose-contracts

litellm-health: added GPU fleet topology table, model routing map,
3-tier GPU checks (reachability, VRAM, temp, test inference)

litellm-self-heal: added GPU fleet reference, rules for GPU
unreachable and model not responding

Infrastructure topology now fully documented:
- amdpve: Strix Halo CPU (35B, 16 threads, 262K ctx, llama-server)
- llm-gpu: RTX 3090 24GB (Dense tier)
- ocu-llm: RTX 5070 12GB (MoE + Light tiers)

gpu-fleet.prose.md

@@ -0,0 +1,111 @@
+---
+kind: responsibility
+name: gpu-fleet
+description: >
+  Manages the GPU inference fleet across all hosts. Handles model deployment,
+  registration, hot-reload, health checks, LiteLLM sync, agent key management,
+  and stale model cleanup. Replaces hardcoded router.py config with declarative
+  gpu_roster.yaml.
+agent: abiba
+triggers:
+  - on model add/remove
+  - on GPU health degradation
+  - on agent key rotation
+---
+
+## Maintains
+
+- gpu_roster: { models: map, hosts: map } — Single source of truth for all GPU models
+- router: { status: "healthy", models: array }
+- litellm: { status: "healthy", models: array }
+- agent_keys: { agent: api_key } — All agent API keys registered in LiteLLM DB
+- health: { gpus: array, circuit_breakers: array }
+
+## Operations
+
+### add-model
+1. Download model file from Hugging Face or source
+2. Check disk space + GPU VRAM compatibility
+3. Build/update llama.cpp if needed (cmake, Vulkan/CUDA flags)
+4. Start llama-server via systemd service
+5. Add to `gpu_roster.yaml`
+6. Hot-reload router: `POST /admin/roster/reload`
+7. Sync LiteLLM config: add model to model_list + fallbacks
+8. Generate/update agent keys for new model access
+9. Verify end-to-end: LiteLLM → Router → Model
+
+### remove-model
+1. Drain active requests (wait for active=0)
+2. Update `gpu_roster.yaml` — remove entry
+3. Hot-reload router
+4. Remove from LiteLLM config
+5. Stop llama-server (systemd)
+6. Cleanup model files (optional — keep if model might return)
+
+### heal
+1. Check all GPUs via router `/health/unified`
+2. Reset stuck circuit breakers if idle
+3. Restart dead llama-server instances
+4. Flush Redis counters if stale
+5. Sync LiteLLM if configs diverge
+
+### sync-keys
+1. List all agent keys in LiteLLM DB via `/key/list`
+2. Compare against expected agent list
+3. Generate missing keys via `/key/generate`
+4. Update agent configs to use `api_key_env: LITELLM_API_KEY`
+5. Set `LITELLM_API_KEY` in system-wide env + startup wrapper
+6. Restart gateways
+
+### list
+Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers
+
+## Configuration Files
+
+| File | Host | Purpose |
+|------|------|---------|
+| `gpu_roster.yaml` | CT 116 (`/opt/inference-harness/`) | Model-to-GPU mapping |
+| `litellm_config.yaml` | CT 116 | LiteLLM proxy config |
+| `roster_loader.py` | CT 116 (router container) | Hot-reloadable roster parser |
+| `/etc/systemd/system/moe-server.service` | .15 | Ornith-1.0-35B server |
+| Agent configs | .122, .123, .114 | `api_key_env: LITELLM_API_KEY` |
+
+## GPU Fleet Topology
+
+```
+┌──────────────────────────────────────────────────┐
+│  LiteLLM (:4001)                                 │
+│  Config: /opt/inference-harness/litellm_config.yaml │
+└──────────────┬───────────────────────────────────┘
+               │
+┌──────────────▼───────────────────────────────────┐
+│  Router (:9000)                                   │
+│  Roster: /app/gpu_roster.yaml (hot-reload)       │
+│  Redis: slot booking + circuit breakers          │
+└──────┬──────────┬──────────┬─────────────────────┘
+       │          │          │
+  ┌────▼───┐ ┌───▼────┐ ┌──▼──────┐
+  │ gemma  │ │ qwen   │ │ ornith  │
+  │ 4-12b  │ │ 27B    │ │ 1.0-35B │
+  │ .110   │ │ .8     │ │ .15     │
+  │ 5070   │ │ 3090   │ │ Strix   │
+  └────────┘ └────────┘ └─────────┘
+```
+
+## Agent Key Architecture
+
+All agents use `api_key_env: LITELLM_API_KEY` in Hermes configs:
+
+| Agent | Host | Config | Env Source |
+|-------|------|--------|------------|
+| Mumuni | .123 | 7 refs + 6 subagent profiles | `/etc/environment` + wrapper |
+| Tanko | .122 | 4 refs | `/etc/environment` + wrapper |
+| Koonimo | .114 | 5 refs | `/etc/environment` + wrapper |
+
+To rotate all keys: update `LITELLM_API_KEY` value, restart gateways.
+
+## Sync Protocol
+
+1. Edit `gpu_roster.yaml` → `POST /admin/roster/reload` (hot, no restart)
+2. Edit `litellm_config.yaml` → `docker restart harness-litellm`
+3. Agent key changes → generate via `POST /key/generate` with master key

litellm-health.prose.md

@@ -12,6 +12,7 @@ description: >
 - public_url: string — The public LiteLLM URL (default: "https://litellm.sysloggh.net")
 - backend_host: string — Internal CT host to check containers (default: "192.168.68.116")
 - auth_host: string — Authentik server for OIDC (default: "192.168.68.11")
+- gpu_hosts: array — GPU inference hosts to check (default: ["192.168.68.8", "192.168.68.110", "192.168.68.15"])
 
 ## Returns
 
@@ -34,6 +35,23 @@ description: >
 - If /health/unified reports any non-healthy components, status reflects it
 - Checks cover at minimum: admin UI, API docs, OIDC, containers, unified health, nginx proxy
 
+## GPU Fleet Topology
+
+| Host | IP | Hardware | Models Served | Engine |
+|------|-----|----------|---------------|--------|
+| llm-gpu | 192.168.68.8 | NVIDIA RTX 3090 (24 GB) | gemma-4-12b (Dense) | llama-server Docker |
+| ocu-llm | 192.168.68.110 | NVIDIA RTX 5070 (12 GB) | qwen3.6-27B-code (MoE), all Light tier | llama-server Docker |
+| amdpve | 192.168.68.15 | AMD Strix Halo (CPU-only, -ngl 0) | ornith-1.0-35b (35B) | llama-server bare-metal |
+
+## Model Routing (LiteLLM → Router → GPU)
+
+| Model Request | Router Routes To |
+|--------------|-----------------|
+| `gemma-4-12b` | `GPU_DENSE_URL` → 192.168.68.8:8080 |
+| `qwen3.6-27B-code` | `GPU_MOE_URL` → 192.168.68.110:8080 |
+| `ornith-1.0-35b` | Router routes to 192.168.68.15:8080 |
+| `syslog-auto` | Auto-routed by router tier logic |
+
 ## Execution
 
 1. **Read parameters** — Use provided values or defaults
@@ -50,7 +68,16 @@ description: >
 5. **Check backend container health**:
    - SSH to {{backend_host}} → `docker ps` → verify all 6 containers are healthy
    - Containers: harness-litellm, harness-nginx, harness-router, harness-postgres, harness-redis, harness-dashboard
-6. **Check OIDC auth endpoint**:
+6. **Check GPU fleet health**:
+   - For each {{gpu_hosts}}:
+     - SSH or nvidia-smi check → verify GPU is reachable
+     - Check VRAM usage (warn if >90%)
+     - Check temperature (warn if >80°C)
+     - Verify model responds to a small test prompt
+7. **Check model inference** — Send test prompt to each model via LiteLLM API:
+   - POST /v1/chat/completions with model name → expect 200 + valid response
+   - Record token counts for baseline performance
+8. **Check OIDC auth endpoint**:
    - Verify auth.sysloggh.net resolves to {{auth_host}}
    - GET https://auth.sysloggh.net/ → expect login page
-7. **Compile and report** — Determine overall_status from individual check results
+9. **Compile and report** — Determine overall_status from individual check results

litellm-self-heal.prose.md

@@ -66,6 +66,14 @@ logged as a `[REPORT]` knowledge graph node for long-term trending.
 If a fix required another agent's help (e.g., Authentik restart), a relay
 message is sent to the responsible agent with full context.
 
+## GPU Fleet
+
+| Host | IP | Hardware | Models | Engine |
+|------|-----|----------|--------|--------|
+| amdpve | 192.168.68.15 | Strix Halo CPU | ornith-1.0-35b (16t, 262K ctx) | llama-server :8080 |
+| llm-gpu | 192.168.68.8 | RTX 3090 24GB | gemma-4-12b Dense | llama-server :8080 |
+| ocu-llm | 192.168.68.110 | RTX 5070 12GB | qwen3.6-27B-code MoE+Light | llama-server :8080 |
+
 ## Remediation Rules
 
 ### Rule 1: Container Not Healthy
@@ -82,6 +90,16 @@ Detect → immediate escalate (external dependency)
 ### Rule 4: Docs 404
 Detect → fix DOCS_URL env var → verify → log
 
+### Rule 5: GPU Unreachable
+Detect → SSH to GPU host fails or model not responding
+Fix → restart llama-server on host via SSH
+Escalate → immediately (downtime affects inference)
+
+### Rule 6: Model Not Responding
+Detect → test prompt via LiteLLM returns non-200
+Fix → restart llama-server on GPU host → verify
+Escalate → after 2 failed restarts
+
 ## Execution
 
 1. Run health check

pm2-self-heal.sh

@@ -21,6 +21,10 @@ send_alert() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] Alert sent" >> "$LOG"
 }
 
+# Log-only mode: replaced by prose contract pm2-self-heal.prose.md
+# Only sends DM on actual failure (status != online), not just high restarts
+# This prevents the process from receiving alerts about itself
+
 # Parse PM2 status (--no-color to avoid ANSI escape codes in awk columns)
 STATUS=$(pm2 status --no-color 2>/dev/null)
 
@@ -51,7 +55,7 @@ ZUL_RESTARTS=$(echo "$ZUL_LINE" | awk -F'│' '{print $9}' | xargs)
 if [ "$ZUL_STATUS" != "online" ]; then
     ALERTS="${ALERTS}🚨 **abiba-zulip is $ZUL_STATUS** — needs investigation!\n"
     ALERTS="${ALERTS}   NOT auto-restarting (runs this contract)\n"
-elif [ "$ZUL_RESTARTS" -gt 5 ] 2>/dev/null; then
+elif [ "$ZUL_RESTARTS" -gt 8 ] 2>/dev/null; then
     ALERTS="${ALERTS}⚠️ abiba-zulip has **$ZUL_RESTARTS restarts** — may need attention\n"
 fi
 

zulip-health.prose.md

@@ -45,6 +45,8 @@ const health = await fetch("http://localhost:9200/health").then(r => r.json());
 | Field | Healthy | Critical |
 |-------|---------|----------|
 | `connected` | `true` | `false` |
+| `stuck` | `false` | `true` (Gen 3) |
+| `idle_seconds` | < 1800 | >= 1800 |
 | `last_error` | `null` | non-null string |
 | `retry_count` | 0-2 | 3+ |
 | `queue_id` | non-null string | `null` |
@@ -78,6 +80,7 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | Condition | Action |
 |-----------|--------|
 | `connected: false` | `pm2 restart abiba-zulip` |
+| `stuck: true` | `pm2 restart abiba-zulip` (Gen 3) |
 | `retry_count >= 3` | `pm2 restart abiba-zulip` |
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |
@@ -137,6 +140,11 @@ ssh root@192.168.68.122 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.he
 | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
 | Response empty/short | Check A2A endpoint / LiteLLM model |
 
+## Restart Debounce (Gen 3)
+
+All restart actions MUST debounce: minimum 300s between restarts.
+Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
+
 ---
 
 ## Platform C: Agent Zero (kagentz — CT 105)
@@ -233,3 +241,17 @@ If any bot is processing >50 bot-originated messages in 15min → 🟡 Warning.
 
 All diagnostics logged to `/root/zulip-health-monitor.log` with timestamps.
 Critical alerts sent as relay messages to user.
+
+## Gen 3 Changes (2026-06-28)
+
+### Problem
+- Bot shows `connected: true` but silently stops processing messages (Zulip event queue expires without BAD_EVENT_QUEUE_ID error)
+- `messages_processed=0` for 45+ min while health endpoint reports healthy
+- Multiple overlapping monitors cause restart death spiral
+
+### Fixes
+1. **Stuck detection**: Health endpoint now reports `stuck: bool`, `idle_seconds`, `last_activity_time`
+2. **Proactive queue recovery**: Re-register queue after 30min of no events (even without BAD_EVENT_QUEUE_ID)
+3. **Monitor stuck awareness**: `zulip-monitor.sh` restarts on `stuck: true` not just `connected: false`
+4. **Restart debounce**: 300s minimum between restarts (via `/tmp/zulip-monitor-debounce`)
+5. **Broader queue expiry detection**: Matches multiple error message patterns