Fix zulip-monitor.sh LOG variable scope bug

Move LOG, TIMESTAMP, and ISSUES declarations from inside notify() to script scope so they are available throughout the script.
Fix zulip-monitor.sh email spam: Replace Gmail SMTP with Zulip stream post to #agent-hub on topic 'zulip-health'
2026-08-29 22:09:46 +00:00 · 2026-08-29 21:30:54 +00:00
47 changed files with 293 additions and 1384 deletions
@@ -67,10 +67,10 @@ Two incidents taught us this:

 | Contract | Sensitivity | Who can change |
 |----------|------------|----------------|
-| `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba, Tanko (Tanko maintains its own CT row) |
+| `infrastructure-control.prose.md` | **CRITICAL** — topology source of truth | Abiba only (after live verification) |
 | `proxmox-monitor.prose.md` | **CRITICAL** — deployed monitoring | Abiba only |
 | `hermes-config-template.prose.md` | **HIGH** — all agent configs | Abiba, Mumuni, Tanko |
-| `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni, Tanko |
+| `zulip-health.prose.md` | **HIGH** — agent communication | Abiba, Mumuni |
 | Other contracts | Normal | Any registered agent |
 | `scripts/*.sh` | **HIGH** — runtime scripts | Abiba only |

@@ -1,2 +0,0 @@
-<!-- Points Claude at AGENTS.md via import; edit AGENTS.md, not this file. -->
-@AGENTS.md
@@ -0,0 +1 @@
+AGENTS.md
@@ -20,8 +20,7 @@ An agent ran `pct set` without checking `pct config` first and changed the IP
 to the wrong value, breaking Zulip. The staleness was harmless until acted on.

 **Layer 3 enforcement — the `safe-mutate` wrapper** (deployed on all 5 agents:
-Abiba, Tanko, Mumuni, Koby, Koonimo — Tanko runs on DSH/DeepSeek Harness since
-2026-08-27 but safe-mutate enforcement still applies). ALL infrastructure mutations MUST go
+Abiba, Tanko, Mumuni, Koby, Koonimo). ALL infrastructure mutations MUST go
 through `safe-mutate`. Raw `sed -i`, `pct set`, `docker compose up
 --force-recreate`, `kill`, `rm` on infrastructure outside `safe-mutate` is an
 auditable protocol violation. The wrapper runs a verify command, optionally
@@ -162,7 +161,7 @@ Companion shell scripts that contracts delegate to.
 |---|---|
 | `daily-infra-report.py` | Generates the daily infrastructure dashboard (HTML email to jerome@sysloggh.com). |
 | `pm2-self-heal.sh` | Shell companion to the pm2-self-heal contract — restarts crashed PM2 processes (runs every 5 min). |
-| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). |
+| `agent-health-check.py` | Consolidated agent health: LiteLLM key validation + GPU port conflict + streaming checks (every 10 min). Replaced zulip-monitor.sh. |

 ## Contract Structure

@@ -1,6 +1,4 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 kind: function
 name: abiba-zulip-restore
 description: >
@@ -13,7 +11,6 @@ version: 1.0.0
 status: active
 runtime_contract: 2
 ---
---

 # Abiba Zulip Restore — Resume pi Zulip Communication

@@ -307,7 +304,6 @@ module.exports = {
 };
 ```

---
 ---

 **Last verified good state**: 2026-07-13 — Extension v2 running via `pi --mode rpc`, health endpoint :9200 returning `{status:"ok",connected:true}`, queue a669f21e.
@@ -1,186 +0,0 @@
-# Agent Zero Issue Fix Summary
-
-**Date**: 2026-09-01  
-**Agent**: Agent Zero (Docker container on kagentz CT105)  
-**Issue**: AuthenticationError + Telegram conflicts  
-**Status**: ✅ RESOLVED
-
---
-
-## Problems Identified
-
-### 1. OpenRouter Authentication Error (CRITICAL)
-```
-litellm.exceptions.AuthenticationError: OpenrouterException - 
-{"error":{"message":"User not found.","code":401}}
-```
-**Root Cause**: The OpenRouter API key in `/a0/usr/.env` belonged to a different OpenRouter user.
-
-**Old Key**: `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`  
-**New Key**: `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`  
-**New User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
-
-### 2. Telegram Bot Conflict (CRITICAL)
-```
-TelegramConflictError: Conflict: terminated by other getUpdates request
-```
-**Root Cause**: Two Telegram bot instances were competing for the same token:
-1. Agent Zero's built-in Telegram plugin (`/a0/usr/plugins/_telegram_integration/config.json`)
-2. Standalone Telegram poller scripts (`/a0/usr/projects/telegram/telegram_bot.py`)
-
-Both were using token `8476855065:***` in polling mode.
-
-**Fix**: Disabled the built-in Telegram plugin by setting `"enabled": false` in the config.
-
-### 3. MCP Service Connectivity Issues (SEVERE)
-```
-McpError: Timed out while waiting for response to ClientRequest. Waited 10.0 seconds.
-```
-**Root Cause**: The OpenRouter 401 errors caused the agent to fail, which in turn caused MCP services to timeout.
-
-**Status**: ✅ RESOLVED with OpenRouter key fix.
-
---
-
-## Fixes Applied
-
-### Fix 1: Update OpenRouter Key
-```bash
-# Container .env update
-sudo docker exec agent-zero bash -c '
-sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=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|" /a0/usr/.env
-'
-```
-
-**Verification**:
-```bash
-curl -s https://openrouter.ai/api/v1/auth/key \
-  -H "Authorization: Bearer sk-or-v1-0af3f3..." | python3 -m json.tool
-```
-Result: HTTP 200, user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`, not free tier.
-
-### Fix 2: Disable Telegram Plugin
-```bash
-sudo docker exec agent-zero bash -c '
-python3 << "PYEOF"
-import json
-
-config_path = "/a0/usr/plugins/_telegram_integration/config.json"
-with open(config_path) as f:
-    config = json.load(f)
-
-config["bots"][0]["enabled"] = False
-
-with open(config_path, "w") as f:
-    json.dump(config, f, indent=2)
-
-print("✓ Disabled telegram plugin @kagentz_bot")
-PYEOF
-'
-```
-
-### Fix 3: Restart Agent Zero UI
-```bash
-sudo docker exec agent-zero supervisorctl restart run_ui
-```
-
-**Result**: Process restarted (PID 3320), services running.
-
-### Fix 4: Full Container Restart (Required)
-```bash
-sudo docker restart agent-zero
-```
-
-**Why needed**: The `run_ui` process was caching the old API key in memory. A full container restart was required to force Agent Zero to reload the `.env` file with the new OpenRouter key.
-
-**Result**: All services restarted cleanly, no more 401 errors.
-
-### Fix 5: Update Stale `.env.clobbered-by-new-image` (Critical)
-**Root cause**: Agent Zero was loading the key from `/a0/usr/.env.clobbered-by-new-image` (line 28) instead of the main `/a0/usr/.env` (line 72). The clobbered file still had the old, stale key.
-
-**Fix**:
-```bash
-KEY=$(grep "^API_KEY_OPENROUTER=" /a0/usr/.env | cut -d"=" -f2-)
-sed -i "s|^API_KEY_OPENROUTER=.*|API_KEY_OPENROUTER=$KEY|" /a0/usr/.env.clobbered-by-new-image
-```
-
-**Lesson**: When updating Agent Zero's `.env`, check BOTH files:
- `/a0/usr/.env` (main)
- `/a0/usr/.env.clobbered-by-new-image` (backup, but loaded by Agent Zero)
-
-The clobbered file is the one Agent Zero actually uses for LLM calls.
-
---
-
-## Infrastructure Documentation
-
-### New Contract Created
-**File**: `/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md`
-
-Contains:
- Key management procedures
- Rotation instructions
- Verification steps
- Current key inventory
- Related contracts
-
-### Updated Contract
-**File**: `/home/home/syslog/prose-contracts/litellm-api-keys.prose.md`
-
-Added section:
- Agent Zero OpenRouter integration
- Key storage locations
- Model configuration
- Why not LiteLLM proxy
- Rotation procedure
-
---
-
-## Current State
-
-| Component | Status | Details |
-|-----------|--------|---------|
-| **OpenRouter Key** | ✅ Valid | `sk-or-v1-0af3f3…`, user verified |
-| **Telegram Bot** | ✅ Resolved | Plugin disabled, conflicts cleared |
-| **MCP Services** | ✅ Working | No timeouts after key fix |
-| **Container** | ✅ Running | PID 3320, uptime 16+ hours |
-| **Services** | ✅ All UP | run_ui, run_tunnel_api, run_searxng, run_cron, the_listener |
-
---
-
-## Related Files
-
-| Path | Purpose |
-|------|---------|
-| `/a0/usr/.env` | Container key storage |
-| `/a0/usr/plugins/_telegram_integration/config.json` | Telegram plugin config |
-| `/a0/usr/plugins/_model_config/presets.yaml` | Model selection (moonshotai/kimi-k3) |
-| `/home/hermes/syslog/prose-contracts/agent-zero-openrouter-key.prose.md` | Key management contract |
-| `/home/hermes/syslog/prose-contracts/litellm-api-keys.prose.md` | Fleet key inventory |
-
---
-
-## Next Steps
-
-1. **Sync key to Infisical vault** (optional, currently .env fallback only)
-2. **Monitor usage** — Check OpenRouter dashboard for daily/weekly spend
-3. **Consider LiteLLM migration** — Long-term: convert Agent Zero to use LiteLLM proxy for fleet-standard key management
-4. **Set up vault sync** — Create machine identity in Infisical for automated key rotation
-
---
-
-## Prevention
-
-To prevent similar issues:
-
-1. **Always verify API keys** against their providers before using
-2. **Keep fleet-wide key inventory** updated in prose contracts
-3. **Rotate keys on schedule** (quarterly hygiene, not on-demand only)
-4. **Test key changes** in staging before production rollout
-5. **Document key locations** in both code and prose contracts
-
---
-
-**Verified by**: Mumuni 🦅  
-**Last updated**: 2026-09-01  
-**Session**: 1
@@ -1,129 +0,0 @@
---
-kind: function
-name: agent-zero-openrouter-key
-description: >
-  Manages the OpenRouter API key for Agent Zero (Docker container on kagentz .14).
-  Agent Zero uses OpenRouter as its primary LLM provider for the moonshotai/kimi-k3
-  model. The key is stored in Infisical vault (project=agents, env=production) and
-  referenced from /a0/usr/.env in the container. Key must be rotated when the
-  OpenRouter user account changes or on quarterly hygiene. Last verified: 2026-09-01.
---
-
-## Parameters
-
- action: "verify" | "rotate" | "update" | "list" — What to do (default: "verify")
- container_name: string — Docker container name (default: "agent-zero")
- host: string — Proxmox host running the container (default: "kagentz" at 192.168.68.14)
- env_path: string — Path to .env file in container (default: "/a0/usr/.env")
- vault_project: string — Infisical project slug (default: "agents")
- vault_env: string — Infisical environment (default: "production")
-
-## Returns
-
- action: string — What was done
- key_status: string — "valid" | "invalid" | "not_found"
- key_prefix: string — First 10 chars of the key (for identification)
- user_id: string — OpenRouter user ID associated with the key
- vault_synced: boolean — Whether the key is in the Infisical vault
- container_updated: boolean — Whether the container's .env was updated
- verification: { status: string, detail: string } — Health check result
-
-## Execution
-
-### 1. Verify the key
-
-1. **Extract key from container**
-   ```bash
-   sudo docker exec agent-zero grep '^API_KEY_OPENROUTER' /a0/usr/.env | cut -d'=' -f2-
-   ```
-
-2. **Test against OpenRouter API**
-   ```bash
-   curl -s https://openrouter.ai/api/v1/auth/key \
-     -H "Authorization: Bearer <key>" | python3 -m json.tool
-   ```
-   Expected: HTTP 200, JSON with `data.label` and `data.is_free_tier`
-
-3. **Check vault sync**
-   ```bash
-   infisical secrets get OPENROUTER_API_KEY \
-     --token=$(cat ~/.infisical-token) \
-     --projectId=agents \
-     --env=production \
-     --domain=https://vault.sysloggh.net
-   ```
-
-4. **Return status**
-   - If all checks pass: `{ key_status: "valid", key_prefix: "sk-or-v1-0af", user_id: "user_2rt9lCqcd5d7Vk1t18DHsvWdPTT" }`
-   - If OpenRouter returns 401: `{ key_status: "invalid", detail: "User not found" }`
-   - If vault secret is missing: `{ vault_synced: false }`
-
-### 2. Rotate the key
-
-1. **Generate new key** in OpenRouter UI or via API
-2. **Update container .env**
-   ```bash
-   sudo docker exec agent-zero sed -i 's/^API_KEY_OPENROUTER=.*/API_KEY_OPENROUTER=<new_key>/' /a0/usr/.env
-   ```
-3. **Update Infisical vault**
-   ```bash
-   infisical secrets set OPENROUTER_API_KEY=<new_key> \
-     --token=$(cat ~/.infisical-token) \
-     --projectId=agents \
-     --env=production \
-     --domain=https://vault.sysloggh.net
-   ```
-4. **Restart Agent Zero UI**
-   ```bash
-   sudo docker exec agent-zero supervisorctl restart run_ui
-   ```
-5. **Verify** — Run "verify" action again
-
-### 3. Update (key changed but no rotation)
-
-1. **Update container .env** (same as rotate step 2)
-2. **Sync vault** (same as rotate step 3)
-3. **Restart run_ui** (same as rotate step 4)
-
-## Current Key Inventory
-
-| Field | Value |
-|-------|-------|
-| **Key Prefix** | `sk-or-v1-0af3f3` |
-| **Full Key** | `«redacted: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»` (in vault + /a0/usr/.env) |
-| **OpenRouter User** | `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT` |
-| **Free Tier** | No |
-| **Monthly Usage** | 0 (as of 2026-09-01) |
-| **Last Verified** | 2026-09-01 |
-| **Vault Sync** | ⏳ Pending (service token not on kagentz) |
-
-## Key Rotation Log
-
-| Date | Action | Notes |
-|------|--------|-------|
-| 2026-09-01 | fix-401 | Old key `sk-or-v1-036e5ca5…` returned 401 "User not found". Replaced with new key `sk-or-v1-0af3f3…` for user `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`. Verified OpenRouter 200. Container .env updated, run_ui restarted. |
-
-## Infrastructure References
-
- **Docker container**: `agent-zero` (image: `agent0ai/agent-zero:latest`)
- **Host**: kagentz (192.168.68.14, Proxmox LXC CT105)
- **Volume**: `/var/lib/docker/volumes/agent_zero/_data` → `/a0/usr`
- **Config path**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=…`)
- **Model preset**: "Cost Efficient" (uses `openrouter/moonshotai/kimi-k3`)
- **Model config**: `/a0/usr/plugins/_model_config/config.json`
-
-## Verification Before Acting
-
-**Key is a lead, not a fact.** Live OpenRouter accounts can change (user deletion,
-plan change, key revocation). Before acting on this contract:
-
-1. Verify the key against OpenRouter's `/auth/key` endpoint
-2. Check the user ID matches the expected account
-3. Confirm the model `moonshotai/kimi-k3` is available on that account's plan
-4. Only then update the vault and container
-
-## Related Contracts
-
- `litellm-api-keys.prose.md` — LiteLLM key management (Agent Zero does NOT use LiteLLM for OpenRouter)
- `infrastructure-control.prose.md` — Proxmox topology, container locations
- `gpu-fleet.prose.md` — Fleet-wide agent key inventory (add Agent Zero here)
@@ -45,10 +45,9 @@ description: >
 ## Status

 **Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
-for Hermes agents that connect to Zulip (Mumuni, Koby, Koonimo). The pi
+for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
 Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
-plugin system, which is unaffected. **Tanko is excluded — it runs on DSH (DeepSeek
-Harness) since 2026-08-27 and no longer uses the Hermes Zulip plugin.**
+plugin system, which is unaffected.

 ## Parameters

@@ -1155,7 +1155,7 @@ contracts:
    type: scheduled
    cadence: 0 3 * * *
    description: Daily at 3am ET
-    cron_job_id: b59f3cc21f4c  # provisioned on kagentz 2026-09-08 (okyeame-memory-audit, glm-5.3-flash)
+    cron_job_id: null
  execution:
    agent: mumuni
    timeout: 600
@@ -1867,73 +1867,3 @@ contracts:
  last_run: null
  last_status: null
  drift_alerts: []
-# Koby Report-Only Registry (2026-08-17 — Captain)
-# ⛔ KOBY IS NEVER REPAIRED — detect + report, never fix on .129
-koby_report_only: true
-koby_host: "CT 111 (tdunna)"
-koby_ip: ".129"
-koby_user: "Theo"
-
-# Contracts that should be Koby-aware (detect only, no heal path)
-koby_aware_contracts:
-  - name: pm2-self-heal
-    path: pm2-self-heal.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby PM2 processes reported to Zulip, never auto-restarted on .129"
-    
-  - name: zulip-health
-    path: zulip-health.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby Zulip bridge issues reported to Zulip, never repaired on .129"
-    
-  - name: hermes-zulip-restore
-    path: hermes-zulip-restore.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby Zulip restoration skipped, only diagnostic alerts"
-    
-  - name: abiba-zulip-restore
-    path: abiba-zulip-restore.prose.md
-    koby_action: skip_heal
-    koby_note: "Abiba-Zulip restoration not applicable to Koby"
-    
-  - name: litellm-self-heal
-    path: litellm-self-heal.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby LiteLLM issues reported, never fixed on .129"
-    
-  - name: disk-gc-threat-response
-    path: disk-gc-threat-response.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby disk GC threats reported, never executed on .129"
-    
-  - name: memory-fixer
-    path: memory-fixer.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby memory issues reported, never fixed on .129"
-    
-  - name: memory-audit-maintenance
-    path: memory-audit-maintenance.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby memory audits reported, never performed on .129"
-    
-  - name: gpu-self-heal
-    path: gpu-self-heal.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby GPU issues reported, never fixed on .129"
-    
-  - name: gpu-monitor
-    path: gpu-monitor.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby GPU monitoring reports only, never repairs on .129"
-    
-  - name: agent-health-check
-    path: agent-health-check.prose.md
-    koby_action: skip_heal
-    koby_note: "Koby agent health checks reported, never repairs on .129"
-
-# Scripts that should skip Koby
-koby_aware_scripts:
-  - name: agent-health-check.py
-    path: scripts/agent-health-check.py
-    koby_action: skip_heal
-    koby_note: "Script should only run diagnostics on Koby, not repairs"
@@ -1,6 +1,4 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 kind: responsibility
 name: disk-gc-threat-response
 description: >
@@ -13,7 +11,6 @@ description: >
 id: 067NV8KJ03ZG71S44N41F31022
 version: 1.0.0
 ---
---

 # Disk GC & Threat Response

@@ -277,10 +274,10 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 ### CT Access (via pct-run)
 | CT | Name | Node | Status |
 |----|------|------|--------|
-| 100 | abiba | minipve | local |
+| 100 | abiba | hwepve | local |
 | 102 | adguard | minipve | ✅ reachable |
 | 104 | authentik | minipve | ✅ reachable |
-| 105 | kagentz | minipve | ✅ reachable |
+| 105 | kagentz | hwepve | ✅ reachable |
 | 106 | ra-h-os | storepve | ✅ reachable |
 | 107 | pbs | storepve | ✅ reachable |
 | 108 | media | storepve | ✅ reachable |
@@ -288,6 +285,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 | 111 | tdunna | amdpve | ✅ reachable |
 | 112 | tanko | amdpve | ✅ reachable |
 | 113 | baggy | amdpve | ✅ reachable |
+| 114 | mumuni | hwepve | ✅ reachable |
 | 115 | scottdenya | amdpve | ✅ reachable |
 | 116 | syslog-api | minipve | ✅ reachable |
 | 117 | zulip | storepve | ✅ reachable |
@@ -304,4 +302,4 @@ one-off GPU builds. No automated post-migration cleanup was in place.
 |------|-----|------|--------|
 | docker-vm | 192.168.68.7 | 16 Docker containers, 4 stacks | ✅ reachable |

-> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
+> **Note:** CT 118 is now jdownloader (active on storepve). CT 119 (infisical-vault) added on minipve.\n> **Migrated:** CT 101 → .8, CT 103 → .110 (bare metal GPU).\n> **KVM VM:** CT 109 (docker-vm) is a KVM VM, not LXC — access via SSH .7.
@@ -185,7 +185,7 @@ what, and why should I care?

 ```
 ❌ "Monitors infrastructure health"
-✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker
+✅ "Scans all 6 Proxmox nodes and 19 CTs for disk pressure, checks Docker
   container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
 ```

@@ -14,6 +14,10 @@ description: >
  Instability observed near 100K at 256K (now all GPUs at 128K). 128K is the stable ceiling.
  For larger context needs → fall back to external providers (deepseek).
  VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
+  UPDATED 2026-07-27: gpu-dense swapped to SmartCode-Fable-5-CoT-Reasoning-QKVO-Qwen-3.6-27B-Distilled
+  (UD-Q3_K_XL, ~14.7GB — Q4 was too large for 24GB VRAM with 128K KV cache). ~50% fewer thinking
+  tokens via ThinkingCap finetune + Fable 5 CoT distillation for improved coding reasoning.
+  VRAM ~22.4/24.6GB (91%).
 agent: abiba
 triggers:
  - on model add/remove
@@ -71,7 +75,7 @@ triggers:
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
 │ 128K ctx │  │ 128K ctx    │  │ 128K ctx  │  │ Watchdog     │
-│ qwen3.6  │  │ Qwen3.5-9B  │  │ qwen3.5   │  │ Prometheus   │
+│ qwen3.6  │  │ gemma-4-12b │  │ qwen3.6   │  │ Prometheus   │
 │ 27B-code │  │ :8080       │  │ -35B-udq4 │  │ exporter     │
 │ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
@@ -86,16 +90,19 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
 | Alias | GPU | Current Model | Will Route To |
 |-------|-----|---------------|---------------|
 | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
-| `gpu-dense` | RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | Whatever runs on RTX 3090 |
+| `gpu-dense` | RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q4_K_XL | Whatever runs on RTX 3090 |
 | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |

-**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.5-9b-it) still work
+**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
 but are deprecated for agent configs. Only the stable aliases survive model swaps.

 ## Current Model Assignments (2026-07-15)

 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
+| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 | .8 (llm-gpu) | ~22.4/24.6GB (91%) | **128K** | q4_0 | 1 | 2048/1024 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
+| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~22GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |

 ## Routing Configuration (LiteLLM — July 2026)

@@ -103,7 +110,9 @@ but are deprecated for agent configs. Only the stable aliases survive model swap

 | Model | GPU | Weight | RPM Cap | Timeout |
 |-------|-----|--------|---------|---------|
-| Qwen3.8-27B-Uncensored-Q4_K_M | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
+| SmartCode-Fable-5-27B-UD-Q3_K_XL | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
+| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
+| gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |

 Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.

@@ -111,6 +120,9 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.

 | Model | RPM Cap | Notes |
 |-------|---------|-------|
+| strix-moe (qwen3.6-35B-udq4) | 40 | Tight cap — prevents Strix overload |
+| SmartCode-Fable-5-27B-UD-Q3_K_XL | 500 | High cap — primary workhorse (replaces qwen3.6-27B-code) |
+| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |

 ### Stable Aliases (for agent configs — never change)

@@ -181,7 +193,7 @@ Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, act
 3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
 4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
 5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
-   - Qwen3.5-9B: 120s, qwen3.6-27B-code: 300s, Carnice-Qwen3.6-MoE-35B-A3B/strix-moe: 300s (strix-moe alias retained, legacy name qwen3.6-35B-udq4 deprecated)
+   - gemma-4-12b: 120s, qwen3.6-27B-code: 300s, qwen3.6-35B-udq4/strix-moe: 300s (strix-moe does NOT exist — legacy name, do not use)
   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
 6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
 7. Check port conflicts: verify only one llama-server on :8080 per host
@@ -196,7 +208,7 @@ Plaintext keys removed from this contract post-vault-migration.
 | Agent | CT | IP | LiteLLM Alias | Key Source | Access |
 |-------|-----|-----|---------------|------------|--------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
-| Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | SSH root |
+| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | SSH root |
 | Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
 | Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
 | Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
@@ -238,6 +250,10 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
+- **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
+- **RTX 3090 (2026-07-27)**: Swapped to SmartCode-Fable-5-27B-UD-Q3_K_XL (14.7GB). Q4 was too large for 24GB VRAM with 128K context + KV cache overhead. Q3 fits at ~22.4GB (91%). Uses standard llama.cpp build b9190 (turboquant b9150 incompatible with qwen3_5 arch). Config: `-c 131072 -ctk q4_0 -ctv q4_0 --flash-attn on --cont-batching`. Sampler: `--temp 0.9 --top-p 0.95 --top-k 60 --min-p 0.0 --repeat-penalty 1.0`. Service: `/home/llmuser/llama-fable-wrapper.sh`.
+- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
+- **LiteLLM timeout tuning (verified 2026-07-27)**: SmartCode-Fable-5-27B 300s, gemma-4-12b 120s, qwen3.6-27B-code 300s (legacy), qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `qwen3.6-35B-udq4`, alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded).
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (old Mumuni CT114 — now inside Abiba CT100 at .24) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
@@ -252,7 +268,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.

 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | Qwen3.8-27B-Uncensored-Q4_K_M | **TBD** | — | — | **128K** |
+| RTX 3090 (.8) | SmartCode-Fable-5-27B-UD-Q3_K_XL | **TBD** | — | — | **128K** |
+| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
 | Strix Halo (.15) | qwen3.6-35B-udq4 | **65** | 140 | — | **128K** |

 Benchmarks from 2026-07-17. Strix Halo model: qwen3.6-35B-udq4. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
@@ -279,13 +296,12 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
 ### Context Windows
 - RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
 - **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
- Compression threshold 0.60: fires at ~77K (~51K headroom before 128K ceiling)
- **Pi agents (Abiba)**: `compaction.reserveTokens: 52739` (≈60% of 128K)
+- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
 - Mumuni compression model alias: `strix-moe` with 300s timeout

 ### Mumuni Agent Profile

-Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the primary business assistant. This profile is the reference for all agent configs:
+Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This profile is the reference for all agent configs:

 | Setting | Value | Notes |
 |---------|-------|-------|
@@ -297,7 +313,7 @@ Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the
 | `aux.web_extract.model` | `gpu-light` | Web extraction |
 | `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
 | `context.max_context_window` | 131072 (128K) | Reduced from 256K 2026-07-17 — stable 128K ceiling |
-| `compression.threshold` | 0.60 | Triggers at ~77K (~60% of 128K) — optimized for 128K context |
+| `compression.threshold` | 0.65 | Triggers at ~85K |
 | `compression.target_ratio` | 0.3 | Compresses to ~38K |
 | `compression.protect_last_n` | 40 | Preserves last 40 messages |
 | `memory.memory_char_limit` | 800 | Brief memory entries |
@@ -310,7 +326,7 @@ Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the

 | Agent | Host | Status |
 |-------|------|--------|
-| **Mumuni** | CT105 (.14) | ✅ Updated to stable aliases |
+| **Mumuni** | CT100 (.24) | ✅ Updated to stable aliases |
 | **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
 | **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
 | **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
@@ -97,28 +97,7 @@ This replaces the previous DM-only delivery. All agents on the mesh can see and
 `curl http://localhost:9100/gpu-data | jq` — Full fleet status

 ### check-health
-
-**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
-
-```bash
-# GPU Monitor health
- curl http://localhost:9100/health | jq
-# Expected: 200 with {"status": "healthy", "cache_age_seconds": <n>}
-
-# Router health (via nginx on port 80)
-curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
-# Expected: 200 (Router is up and responding)
-
-# LiteLLM health (via nginx on port 80)
-curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
-# Expected: 200 (LiteLLM is up and responding)
-
-# Dashboard
- curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/dashboard/
-# Expected: 200 (Dashboard is up and responding)
-```
-
-**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
+`curl http://localhost:9100/health` — Monitor self-check

 ### view-dashboard
 Open `http://localhost:9100/` in browser — Live HTML dashboard
@@ -1,6 +1,4 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 kind: responsibility
 name: gpu-self-heal
 description: >
@@ -18,7 +16,6 @@ depends_on:
  - gpu-monitor.prose.md (live data source on .24:9100)
  - gpu-fleet.prose.md (source of truth for topology, aliases, model assignments)
 ---
---

 ## Maintains

@@ -41,19 +38,20 @@ depends_on:
 - On fix: verify with benchmark inference test before declaring resolved
 - Escalate: after 3 failed remediation attempts → Zulip #agent-hub alert

---
 ---

 ## Current Fleet Baseline (2026-07-18)

 | Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role |
 |-------|-----|------|-------|------|-----|-------|------|
+| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
+| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
 | `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | qwen3.6-35B-udq4 | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |

 Key notes:
 - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
 - Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated.
- RTX 5070 tok/s is ~145 for Qwen3.5-9B — gpu-light is the fastest endpoint. Route vision/web/light work there first. NOTE: Qwen3.5-9B is multimodal (image+text), NOT text-only like gemma-4-12b was.
+- RTX 5070 tok/s is 2.3x faster than RTX 3090 for its model — gpu-light is the fastest endpoint. Route vision/web/light work there first.
 - Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads.
 - RTX 3090 VRAM at 88% — within role-appropriate range (role = heavy reasoning, needs the headroom).
 - RTX 5070 VRAM at 83% — role-appropriate for vision/web (smaller batch sizes).
@@ -158,7 +156,7 @@ Key notes:
 - **Detect**: GPU roles misaligned with hardware capabilities
 - **Target distribution**:
  - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
-  - RTX 5070 (gpu-light, 12GB, ~145 tok/s) → Vision (image+text), web search, lightweight tasks. Weight: 0.15 (LiteLLM).
+  - RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
  - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
 - **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
 - **Fix**:
@@ -168,7 +166,6 @@ Key notes:
 - **Verify**: Each GPU's request pattern matches its designated role within 24h
 - **Escalate**: If role mismatch persists >48h → agent alias audit needed

---
 ---

 ## Execution
@@ -250,7 +247,6 @@ call update-gpu-health
 }
 ```

---
 ---

 ## Reporting
@@ -267,7 +263,6 @@ Pushed to `SyslogSolution/health-logs/gpu/{run_id}.json` — versioned, searchab
 - Per-GPU tok/s trend over 7 days
 - Regression alerts if any GPU degrades >10% week-over-week

---
 ---

 ## Design Decisions (Verified 2026-07-12, Reaffirmed 2026-07-18)
@@ -24,6 +24,8 @@ done

 | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
 |-------|-----|------|-----|---------------|------------|----------|
+| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
+| Mumuni | 100 | hwepve | .24 | `mumuni` | Infisical vault | Hermes |
 | Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** |
 | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
 | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
@@ -51,10 +53,9 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) →

 ## Config Pattern — Mandatory Fields

-### For Hermes Agents (Mumuni, Koonimo)
+### For Hermes Agents (Tanko, Mumuni, Koonimo)

-Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
-(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
+Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:

 ### 1. Main Model
 ```yaml
@@ -70,7 +71,7 @@ model:
 custom_providers:
  - name: harness
    model: syslog-auto
-    base_url: http://192.168.68.116/litellm/v1
+    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
 ```
@@ -81,7 +82,7 @@ auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b              # or syslog-auto
-    base_url: http://192.168.68.116/litellm/v1
+    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 60
@@ -96,7 +97,7 @@ auxiliary:
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # or gemma-4-12b
-    base_url: http://192.168.68.116/litellm/v1
+    base_url: http://192.168.68.116/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 120
@@ -168,15 +169,11 @@ pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
 # Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
 ```

-### For Koby (CT 111 / tdunna) — **REPORT-ONLY MODE**
+### For Koby (CT 111 / tdunna)

 Koby runs Hermes on CT 111 (tdunna). Config files at `/root/.hermes/config.yaml`.
 Same Hermes pattern as Tanko/Mumuni/Koonimo — see config sections above.

-**⛔ KOBY IS NEVER REPAIRED (2026-08-17, Captain)**: Diagnostic only — detect and report, never fix on .129.
-No heal step, no restart, no key rotation, no config edit, no memory rewrite, no disk GC, no service touch, no process kill — ever.
-If a health check shows Koby degraded, **DO NOT** execute any repair action. Instead, report to Zulip and let Theo fix it.
-
 **LiteLLM key**: alias `koby` in LiteLLM DB, injected via `infisical run --` wrapper.

 ### For pi Agents (Abiba)
@@ -15,7 +15,7 @@ description: >

 - template_version: "2.1.0"
 - last_applied: timestamp
- agents_configured: ["mumuni", "abiba", "koby", "koonimo", "kagenz0"]  # tanko removed 2026-08-27 (now on DSH/DeepSeek Harness)
+- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
 - agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array

@@ -30,6 +30,8 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed

 | Agent | Key Alias | Host | SSH | Sub-Agents |
 |-------|-----------|------|-----|-----------|
+| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
+| Mumuni | `mumuni` | 192.168.68.24 | root@.24 | 6 profiles ✱ |
 | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
 | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
 | Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — |
@@ -346,17 +348,6 @@ curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $

 ### Rule 13: API Key Injection — Two Patterns (UPDATED 2026-07-16, WAL #1300)

-### Rule 14: Hermes Context Detection Uses `max_model_tokens`, NOT `max_input_tokens`
-
-**CRITICAL**: Hermes context detection reads `max_model_tokens` (128K), NOT `max_input_tokens` (64K cap).
-
- **Abiba and Hermes agents**: `max_model_tokens: 131072` (128K) — unlimited context
- **Crewmates (ops, tune, verify, auth-keys, build)**: `max_input_tokens: 64000` (64K) — capped
- If you see `max_input_tokens: 64000` in an Abiba/Hermes config, that's a mistake
- Using `max_input_tokens` for Hermes agents causes premature context loss
- Check: `grep -n 'max_model_tokens\|max_input_tokens' ~/.hermes/config.yaml`
- Expected output: `max_model_tokens: 131072` (not max_input_tokens)
-
 Agents inject `LITELLM_API_KEY` via ONE of two mechanisms. Both are valid; the contract
 requirement is that the key is a **valid LiteLLM virtual key** (HTTP 200 on /v1/models).

@@ -433,3 +424,13 @@ curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.
 5. **Set model choice** — Per agent's workload
 6. **Verify** — curl all shared endpoints, test the model with the new key
 7. **Report** — What was changed, preserved, custom
+### Rule 16: Koby Configuration (DeepSeek-primary)
+(Ref: See Rule 10 for default model behavior, with Koby exception)
+
+Koby uses a split-model architecture:
+
+- Primary Model: `deepseek-v4-flash` via `api.deepseek.com` (for reasoning)
+- Auxiliary Models: `gpu-light` (vision/web_extract) and `syslog-auto` (compression)
+- Key Hygiene: `api_key_env` is strictly `LITELLM_API_KEY` or `DEEPSEEK_API_KEY`
+- Constraint: Do NOT touch Koby's primary model/provider/compression settings unless explicitly ruled by the captain.
+
@@ -190,7 +190,7 @@ litellm_settings:
 | Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
 |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
-| Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | ✅ Fixed | systemd Hermes gateway | 2026-08-29 |
+| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
 | Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed (DeepSeek-primary) | `infisical run` | 23:30 UTC Jul 5 |
 | Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 |
 | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
@@ -285,13 +285,13 @@ auxiliary:
  vision:
    api_key: sk-<agent-key-from-vault>           # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/litellm/v1
+    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
  compression:
    api_key: sk-<agent-key-from-vault>           # ← workaround (same as above)
    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/litellm/v1
+    base_url: http://192.168.68.116/v1
    model: gemma-4-12b
    provider: harness
 ```
@@ -28,7 +28,7 @@ connectivity recovery including end-to-end DM validation.

 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — on DSH since 2026-08-27, no Hermes plugin) |
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |
 | `branch` | string | no | `master` | Git branch to pull (overridable for pinning) |

 ## Maintains
@@ -55,7 +55,8 @@ connectivity recovery including end-to-end DM validation.

 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | *(DSH since 2026-08-27 — historical, plugin retired on this host)* |
+| Mumuni | CT100 | — | 192.168.68.24 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |

@@ -120,8 +121,7 @@ cp plugins/platforms/zulip/adapter.py \
   plugins/platforms/zulip/plugin.yaml \
   {{hermes_home}}/hermes-agent/plugins/platforms/zulip/

-# Fix ownership (was Tanko-only, runs as jerome user)
-# RETIRED 2026-08-27: tanko no longer uses the Hermes Zulip plugin (DSH).
+# Fix ownership (Tanko only — runs as jerome user)
 [ "{{target}}" = "tanko" ] && chown -R jerome:jerome \
  {{hermes_home}}/hermes-agent/plugins/platforms/zulip/

@@ -1,9 +1,9 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 kind: function
 name: hermes-zulip-restore
 description: >
+  Restores Zulip connectivity for any Hermes agent (Mumuni CT100, Tanko CT112,
+  Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
  path, verifies env credentials, restarts the gateway, and confirms Zulip
  connects. Run this whenever a Hermes agent stops responding on Zulip or after
  a fresh agent deployment.
@@ -12,7 +12,6 @@ version: 1.0.0
 status: active
 runtime_contract: 2
 ---
---

 # Hermes Zulip Restore — Bring Any Agent Back to Good State

@@ -24,7 +23,7 @@ gateway restart, and connection validation.

 | Param | Type | Required | Default | Description |
 |-------|------|----------|---------|-------------|
-| `target` | string | yes | — | Agent name: `mumuni`, `koby`, or `shumba` (Tanko excluded — DSH since 2026-08-27) |
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, `koby`, or `shumba` |

 ## Maintains

@@ -37,7 +36,7 @@ gateway restart, and connection validation.

 - `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
 - All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko, historical — DSH since 2026-08-27)
+- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
 - Gateway restarted and zulip platform reports state `connected`
 - HTML stripping enabled for `/approve` and `/deny` slash command support

@@ -52,6 +51,8 @@ gateway restart, and connection validation.

 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
+| Mumuni | CT100 (abiba) | hwepve | 192.168.68.24 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |

@@ -93,8 +94,8 @@ cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/

-# Fix ownership (was Tanko-only; RETIRED 2026-08-27 — tanko on DSH, no Hermes plugin)
-chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only (historical)
+# Fix ownership (Tanko only)
+chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only

 # Clean up
 rm -rf /tmp/zulip-deploy
@@ -183,7 +184,6 @@ https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/z
 Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
 Pull request #33 is the primary integration branch.

---
 ---

 **Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
@@ -87,8 +87,8 @@ call apply-liteLLM-routing

 call apply-agent-compression
  agent: mumuni
-  host: 192.168.68.14
-  config_path: /home/hermes/.hermes/config.yaml
+  host: 192.168.68.24
+  config_path: /root/.hermes/config.yaml

 -- Phase 4: Enable llama.cpp prompt caching on GPU hosts

@@ -3,7 +3,7 @@ kind: pattern
 name: infrastructure-control
 description: >
  Full infrastructure monitoring and control pattern covering the
-  5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
+  6-node Proxmox cluster, 3 Docker ecosystems (22 containers),
  NFS storage, and network services. Defines monitors, remediations,
  and the access matrix for all environments.

@@ -13,11 +13,10 @@ description: >
  against the live system. Policy fields are authoritative. See the
  `verify-before-mutate` skill.

-  **Last verified:** 2026-08-15 — hwepve removed from Tabiri cluster
-  (now 5 nodes: minipve, amdpve, storepve, acerpve, ocupve). hwepve
-  (192.168.68.4) is a standalone PVE node + NetBird routing peer;
-  London relocation pending. CTs 100 (abiba) and 105 (kagentz) moved
-  to minipve.
+  **Last verified:** 2026-07-24 — corrected Gitea IP (.17 not .110),
+  AdGuard IP (.10 not .102), AdGuard placement (minipve not acerpve),
+  Abiba placement (hwepve not amdpve), added hwepve as 6th node,
+  added dns.sysloggh.net route.
 ---

 # Infrastructure Control Pattern
@@ -35,21 +34,21 @@ description: >
   ┌─────────────┐    ┌──────────────┐    ┌──────────────┐
   │   Abiba     │    │   Tanko      │    │   Mumuni     │
   │   (pi)      │    │   (Hermes)   │    │   (Hermes)   │
-   │   CT 100    │    │   CT 112     │    │   CT 100     │
+   │   CT 100    │    │   CT 112     │    │   CT 114     │
   └──────┬──────┘    └──────┬───────┘    └──────┬───────┘
          │                  │                    │
          └──────────────────┼────────────────────┘
                             ▼
-          ┌────────────────────────────────────┐
-          │         Proxmox Cluster API        │
-          │    minipve.sysloggh.net:443        │
-          │   (monitoring@pve!mumuni token)    │
-          └────┬──────┬──────┬──────┬──────────┘
-               │      │      │      │
-          ┌────┘ ┌────┘ ┌────┘ ┌────┘
-          ▼      ▼      ▼      ▼
-       minipve amdpve storepve acerpve ocupve
-        (.12)   (.15)   (.6)    (.9)    (.5)
+          ┌──────────────────────────────────────┐
+          │         Proxmox Cluster API          │
+          │    minipve.sysloggh.net:443          │
+          │   (monitoring@pve!mumuni token)      │
+          └────┬──────┬──────┬──────┬──────┬─────┘
+               │      │      │      │      │
+          ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
+          ▼      ▼      ▼      ▼      ▼
+       minipve amdpve storepve acerpve ocupve hwepve
+        (.12)   (.15)   (.6)    (.9)    (.5)    (.4)

          ▼
   ┌─────────────────────────────────────────────┐
@@ -101,29 +100,21 @@ description: >

 ## Section 2: Proxmox Cluster — Monitoring

-### Nodes (5)
+### Nodes (6)

 | Node | IP | CPU | RAM | VMs/CTs | Role |
 |------|----|-----|-----|---------|------|
-| minipve | .12 | 16C | 30GB | abiba, kagentz, authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
+| minipve | .12 | 16C | 30GB | authentik, gitea, syslog-api, infisical-vault, jitsi | Auth, git, messaging |
 | amdpve | .15 | 32C | 62GB | tanko, tdunna, baggy, scottdenya | Agents, compute |
 | storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, jdownloader, zulip | Docker, storage, chat |
 | acerpve | .9 | 28C | 31GB | llm-gpu | GPU VMs |
 | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
+| hwepve | .4 | 12C | 15GB | abiba, kagentz, (mumuni CT 114 stopped) | Agents (new node) |

 > **Note:** CTs on storepve include jdownloader (CT 118). AdGuard (CT 102) is on
-> minipve at .10, not acerpve. Abiba (CT 100) and kagentz (CT 105) are on
-> minipve (moved from hwepve 2026-08-15). Mumuni runs inside Abiba CT100
-> (.24); CT 114 (mumuni) no longer exists in the cluster.
->
-> **hwepve (192.168.68.4) — STANDALONE (removed from Tabiri 2026-08-15):**
-> Huawei MateBook 16 (KLVL-WXX9), pve-manager/9.2.10, kernel 7.0.14-8-pve.
-> Zero VMs/CTs. Being relocated to London as a standalone PVE node + NetBird
-> routing peer (relocation pending). Localizations applied: timezone
-> Europe/London, lid-switch ignore, sleep/suspend/hibernate targets masked,
-> cluster-shared storage removed (remaining: local, local-lvm, storage,
-> mediastore). prometheus-node-exporter active on :9100; net.ipv4.ip_forward=1;
-> NetBird client not yet installed (enrollment pending setup key).
+> minipve at .10, not acerpve. Abiba (CT 100) is on hwepve, not amdpve. Mumuni
+> (CT 114) is on hwepve (currently stopped), not minipve. Mumuni also has a
+> second instance on minipve at .123 — distinguish by CT ID, not hostname.

 ### Checks (every 5 min)

@@ -601,20 +592,21 @@ ssh root@192.168.68.110 "systemctl restart llama-server"

 | CT | Name | Node | IP | Role | Agent |
 |----|------|------|----|------|-------|
-| 100 | abiba | minipve | .24 | Pi agent | ✅ pi |
+| 100 | abiba | **hwepve** | .24 | Pi agent | ✅ pi |
 | 101 | llm-gpu | acerpve | .8 | GPU RTX 3090 | ❌ |
 | 102 | adguard | **minipve** | **.10** | DNS | ❌ |
 | 103 | ocu-llm | ocupve | .110 | GPU RTX 5070 | ❌ |
 | 104 | authentik | minipve | .11 | OIDC | ❌ |
-| 105 | kagentz | minipve | — | Agent Zero | ✅ |
+| 105 | kagentz | **hwepve** | — | Agent Zero | ✅ |
 | 106 | ra-h-os | storepve | .65 | KG bridge | ✅ MCP |
 | 107 | pbs | storepve | — | Backups | ❌ |
 | 108 | media | storepve | — | Media | ❌ |
 | 109 | docker-vm | storepve | .7 | Docker host | ❌ |
 | 110 | gitea | minipve | **.17** | Git | ❌ |
 | 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
-| 112 | tanko | amdpve | .122 | DSH (DeepSeek Harness) agent | ✅ |
+| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
 | 113 | baggy | amdpve | .114 | Hermes agent | ✅ |
+| 114 | mumuni | **hwepve** | .123 | Hermes agent (stopped) | ✅ |
 | 115 | scottdenya | amdpve | .75 | Denya OneCare | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
 | 117 | zulip | storepve | .19 | Chat | ❌ |
@@ -639,14 +631,15 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.

 | CT | Name | Node | pct-run |
 |-----|------|------|---------|
-| 100 | abiba | minipve | `pct-run 100` |
-| 105 | kagentz | minipve | `pct-run 105` |
+| 100 | abiba | hwepve | `pct-run 100` |
+| 105 | kagentz | hwepve | `pct-run 105` |
 | 111 | tdunna | amdpve | `pct-run 111` |
 | 112 | tanko | amdpve | `pct-run 112` |
 | 113 | baggy | amdpve | `pct-run 113` |
 | 115 | scottdenya | amdpve | `pct-run 115` |
 | 104 | authentik | minipve | `pct-run 104` |
 | 110 | gitea | minipve | `pct-run 110` |
+| 114 | mumuni | hwepve | `pct-run 114` |
 | 116 | syslog-api | minipve | `pct-run 116` |
 | 106 | ra-h-os | storepve | `pct-run 106` |
 | 107 | proxmox-backup | storepve | `pct-run 107` |
@@ -659,7 +652,7 @@ GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use S
 ssh root@192.168.68.8   # RTX 3090
 ssh root@192.168.68.110 # RTX 5070
 ssh root@192.168.68.15  # Strix Halo
-ssh root@192.168.68.4   # hwepve — standalone London node + NetBird routing peer (relocation pending)
+ssh root@192.168.68.4   # hwepve (abiba, kagentz, mumuni)
 ```

 ## Section 7: Agent Health Check (consolidated — 2026-07-05)
@@ -140,6 +140,7 @@ After ALL updates (apt + images + restarts), verify every critical service is ba
 | Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
 | Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
 | PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
+| Hermes gateways | SSH to Mumuni CT 100, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |

 Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.

@@ -102,53 +102,6 @@ GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
 - Stack persists across reboots (systemd for exporters, Docker restart policy)

 ## Execution
-### check-health
-
-**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
-
-```bash
-# Zulip API health (POST ping)
-source /etc/litellm-monitor.env
-ZULIP_USER="abiba-bot@chat.sysloggh.net"
-curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u "${ZULIP_USER}:${ZULIP_BOT_KEY}"
-# Expected: 200 (HTTP 000 = unreachable/cache)
-
-# PM2 process health
-pm2 jlist
-# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
-
-# GPU exporters (may be down per DEPLOYMENT STATUS)
-curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
-curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
-curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
-
-# Router health (via nginx on port 80)
-curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/health
-# Expected: 200 (Router is up and responding)
-
-# LiteLLM health (via nginx on port 80)
-curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116/litellm/health
-# Expected: 200 (LiteLLM is up and responding)
-
-# PVE API (401 expected for unauthenticated probe — API is up over https)
-curl -s -o /dev/null -w '%{http_code}' https://192.168.68.116:8006/api2/json
-# Expected: 401 (unauthorized — API is up; 000 = unreachable, 500 = API down)
-
-# Prometheus targets
-curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
-# Expected: All targets UP (may show some down if exporters not deployed)
-
-# Grafana health
-curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
-# Expected: {"status":"ok","version":"..."}
-
-# LiteLLM metrics (Prometheus endpoint)
-curl -s http://192.168.68.116:4001/metrics | head -20
-# Expected: Prometheus-formatted metrics output
-```
-
-**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
-

 ### Phase 1: GPU Exporters

@@ -2,7 +2,7 @@
 kind: responsibility
 name: infrastructure-update
 description: >
-  Autonomous system-wide update contract covering all 5 Proxmox nodes,
+  Autonomous system-wide update contract covering all 6 Proxmox nodes,
  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
@@ -56,10 +56,11 @@ Before ANY update wave:
 | amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
+| hwepve (.4) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
 | CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
-| CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
+| CT 100 (mumuni/abiba, hwepve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
 | VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
 | VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |

@@ -162,9 +163,9 @@ Before Wave 1, snapshot these files:
 /etc/systemd/system/strix-server.service                (amdpve .15 — strix-moe)
 /etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
 # Hermes agent configs (key enforcement — 2026-07-10)
-/home/hermes/.hermes/config.yaml                        (Mumuni kagentz CT105; Tanko CT112 uses /home/jerome/.hermes)
-/etc/systemd/system/hermes-gateway.service              (Mumuni kagentz CT105 — system unit, User=hermes)
-/etc/environment                                        (LITELLM_API_KEY — legacy path, Mumuni now keys via Infisical)
+/root/.hermes/config.yaml                               (Mumuni CT 114, Tanko CT 112, etc.)
+/root/.config/systemd/user/hermes-gateway.service       (Mumuni CT 114 — EnvironmentFile fixed)
+/etc/environment                                        (Mumuni CT 114 — LITELLM_API_KEY)
 ```

 ## MCP Gateway (2026-07-10)
@@ -218,7 +219,7 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants:

 ## Success Criteria

- [ ] All 5 PVE nodes updated, no reboot-loop
+- [ ] All 6 PVE nodes updated, no reboot-loop
 - [ ] All VMs/CTs running post-update
 - [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
 - [ ] LiteLLM inference passing (syslog-auto test)
@@ -234,7 +235,7 @@ After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD

-Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
+Updated: 6 PVE nodes, 12 CTs/VMs, 30+ containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>
@@ -178,7 +178,7 @@ through its agent wrapper.
 | Agent | Host | Pattern | Keys | Status |
 |-------|------|---------|------|--------|
 | abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
-| mumuni | .14 (kagentz CT105) | systemd unit hermes-gateway.service (user hermes) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
+| mumuni | .24 (CT100 abiba) | Pi Hermes gateway (no systemd) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
 | tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
 | koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
 | koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
@@ -257,51 +257,9 @@ reads use per-agent identities. This eliminates the single shared token risk.
 | Agent | .env Keys |
 |-------|-----------|
 | Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY |
-|| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
-|| Koby | (wrapper injects from vault — .env has Telegram token) |
-|| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |
-|| Agent Zero (kagentz .14) | OPENROUTER_API_KEY (direct OpenRouter access) |
-
-### Agent Zero (kagentz .14) — OpenRouter Integration (2026-09-01)
-
-Agent Zero runs in Docker on kagentz (CT105) and uses **direct OpenRouter API access**,
-not via the LiteLLM proxy. This is because Agent Zero's workflow (self-update manager,
-UI bootstrap, model selection) is built around OpenRouter's native authentication.
-
-**Key Storage:**
- **Container**: `/a0/usr/.env` (line ~72: `API_KEY_OPENROUTER=sk-or-v1-…`)
- **Vault**: Infisical secret `OPENROUTER_API_KEY` (project=agents, env=production)
- **Fallback**: The container's .env is the primary source; vault sync is optional
-  (unlike fleet agents which require vault injection)
-
-**Current Key (2026-09-01):**
- **Prefix**: `sk-or-v1-0af3f3…`
- **User**: `user_2rt9lCqcd5d7Vk1t18DHsvWdPTT`
- **Plan**: Paid (not free tier)
- **Usage**: 0 (as of 2026-09-01)
-
-**Model Configuration:**
- **Preset**: "Cost Efficient" (`/a0/usr/plugins/_model_config/presets.yaml`)
- **Model**: `openrouter/moonshotai/kimi-k3`
- **API Base**: (empty — uses OpenRouter default)
-
-**Why not LiteLLM proxy?**
-Agent Zero's architecture was designed before the fleet adopted the LiteLLM proxy
-standard. The container runs `/exe/self_update_manager.py` and `/a0/run_ui.py` which
-directly call OpenRouter via Python's requests library. Converting would require:
-1. Refactoring all LLM calls to use `litellm` library
-2. Adding vault wrapper injection
-3. Updating self_update_manager to use proxy-aware key handling
-
-**Rotation Procedure:**
-1. Generate new key in OpenRouter UI
-2. Update container: `sed -i 's/^API_KEY_OPENROUTER=.*/API_KEY_OPENROUTER=<new_key>/' /a0/usr/.env`
-3. Update vault: `infisical secrets set OPENROUTER_API_KEY=<new_key> --projectId=agents --env=production`
-4. Restart container: `sudo docker exec agent-zero supervisorctl restart run_ui`
-5. Verify: `curl -s https://openrouter.ai/api/v1/auth/key -H "Authorization: Bearer <new_key>"`
-
-**Related Contract:**
- `agent-zero-openrouter-key.prose.md` — Full agent-zero key management contract
+| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
+| Koby | (wrapper injects from vault — .env has Telegram token) |
+| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |

 ## Key Rotation Log

@@ -1,117 +0,0 @@
---
-kind: pattern
-name: litellm-client-timeouts
-description: >
-  Standard client timeout and retry policy for ALL agents calling LiteLLM
-  (CT 116, http://192.168.68.116). Created 2026-09-08 after the Sep 6 incident:
-  a backend stall 04:00-06:30 EDT produced 157 client-abandoned 408 failures
-  (82% from abiba-pi, 13 from mumuni) against a backend that was actually
-  succeeding at 20-70s per call once clients stopped giving up. Grounded in
-  measured data: syslog-auto (LiteLLM virtual model, no dedicated GPU — routes
-  to backends) averages 28.8s/request with 25.4s TTFT over 888 calls/24h;
-  nginx already allows 600s (Rule 5, verified 2026-08-09); the gap is entirely
-  client-side. Blast radius if wrong: agents fall back to DeepSeek silently
-  (key/timeout failures present as model degradation, not errors) or abandon
-  healthy-but-slow reasoning calls, fragmenting long tasks.
---
-
-## Maintains
-
- client_timeout_standard: "litellm-client-timeouts v1.0 (2026-09-08)"
- applies_to: ALL agents and scripts calling http://192.168.68.116 (main model path, auxiliary tasks, health probes, benchmark jobs)
- verified_against: Prometheus litellm_* metrics 24h window ending 2026-09-08 ~10:00 EDT; live probe (syslog-auto tiny call 0.56s TTFB, 200 OK); nginx 600s proxy_read_timeout (Rule 5)
-
-## The measured numbers these values come from
-
-| Model | avg latency | avg TTFT | p-profile (24h) |
-|---|---|---|---|
-| syslog-auto | 28.8s | 25.4s | 68 calls took 30-120s; tail to ~300s under load |
-| qwen3.6-27B-code | 23.0s | — | same backend class as syslog-auto |
-| strix-moe | 7.5s | — | Strix Halo, healthy |
-| gemma-4-12b | 2.6s | — | RTX 5070, healthy |
-
-Sep 6 incident timeline: failures 04:00-07:00 EDT (0% GPU util = wedged
-backend), full recovery 07:00-08:00 with ZERO client failures once requests
-tolerated 20-70s — 392 successful slow calls in the three hours after recovery.
-LiteLLM's internal queue time is ~0s; the latency is model inference, not
-proxy queuing.
-
-## Parameters
-
-### 1. Primary model path (model.default / custom_providers) — NO client timeout below 300s
-
- The default Hermes HTTP timeout (~60s) is TOO SHORT for syslog-auto's healthy
-  28.8s average + 120-300s tail. Every 408 in the incident was a client
-  abandoning a request the backend would have answered.
- If the transport exposes a timeout setting for the main model, set it to
-  **300s or more**. If it does not (current Hermes custom-provider path has no
-  timeout knob), that is acceptable ONLY because nginx holds the request for
-  600s — but any wrapper, script, or direct API call you write MUST set its own
-  timeout >= 300s for syslog-auto/qwen-class calls.
- Never hardcode a shorter timeout "to fail fast" on this path — failing fast
-  here is what caused the incident.
-
-### 2. Auxiliary tasks — keep template timeouts, one correction
-
- vision: 60s (keep), web_extract: 30s (keep) — gemma-4-12b averages 2.6s;
-  these are fine.
- compression: 300s (keep — this was already raised from 60 per gpu-fleet).
- **gpu-dense delegation/x_search: set timeout >= 120s.** The RTX 3090
-  (qwen3.6-27B-code backend, 23.0s avg) is the same speed class as
-  syslog-auto; delegation defaults that assume fast responses will 408 the
-  same way.
-
-### 3. Retry policy — backoff, not repetition
-
- On timeout (408) or 5xx: retry up to **2 times** with exponential backoff
-  (**15s, 45s**) before giving up.
- Do NOT retry in a tight loop. The Sep 6 spike shape (65 failures in one hour
-  from one key) was a batch job retrying without backoff while the backend was
-  down — it multiplied load during recovery.
- On 401/403: do NOT retry — that is a key/permission problem (see
-  litellm-api-keys.prose.md and Rule 11); retrying just spams the log.
- On 429: honor the retry-after header if present, else back off 60s.
-
-### 4. Health probes — identify yourself and time out sanely
-
- Probes MUST NOT appear as keyless, model-less failures in the metrics (12
-  such orphans appeared in the incident window and cost investigation time).
-  Send a real model name and use a real (probe-designated) key.
- Probe timeout: 30s. A probe that takes longer than 30s IS the alert —
-  report "backend slow (>30s)" rather than hanging.
- Probe cadence: at most hourly. The 6h litellm-health cron cadence is the
-  standard; sub-hourly synthetic traffic distorts latency baselines.
-
-### 5. Batch/benchmark jobs — schedule away from 04:00-07:00 EDT and chunk
-
- The incident window showed bulk clients amplifying a backend stall 5:1.
- Batch jobs that can tolerate delay: schedule 09:00-17:00 EDT.
- Any batch loop over N requests MUST sleep >= 5s between requests and honor
-  the retry policy in section 3.
-
-## Returns
-
- A single standard any agent or script can cite: timeouts >= 300s on the
-  syslog-auto path, >= 120s on gpu-dense delegation, backoff retries (2x,
-  15s/45s), identified probes at 30s/hourly, batch jobs chunked and
-  day-scheduled.
- Failure signature recognition: bulk 408s from multiple keys in one window =
-  backend event (check gpu_utilization_percent: 0% = wedged, ~100% = saturated);
-  single-key 408s = that client's timeout is too short.
- Cross-references: hermes-config-template.prose.md (Rule 5 nginx 600s,
-  auxiliary timeouts), gpu-fleet.prose.md (compression 300s precedent,
-  stable aliases), litellm-api-keys.prose.md (key/permission failures).
-
-## Intentionally NOT changed
-
- No server-side LiteLLM timeout/cooldown changes proposed — the incident
-  self-recovered and the server is healthy (0.56s live probe); changing
-  server behavior without process-level root cause (CT116 requires root;
-  not reachable from kagentz) would be guessing.
- No change to the template's vision/web_extract/compression timeouts —
-  measured data says they are correct.
- No per-agent key permission changes — those are litellm-api-keys.prose.md
-  territory (and the open gpu-vision/gemma 403 items are already filed with
-  the key owners).
- No model routing changes — syslog-auto's weighted pool behaved correctly
-  throughout the incident.
@@ -26,7 +26,7 @@ description: >
    Scraped by Prometheus with Bearer master key; endpoint returns 307 → /metrics/.
  - Alertmanager (harness-alertmanager :9093) + zulip-bridge (:9102) deliver
    firing alerts to #agent-hub > alerts-infra via abiba-bot. Added 2026-08-09.
-  - Prometheus node job covers ALL 5 PVE nodes (.5/.6/.9/.12/.15:9100).
+  - Prometheus node job covers ALL 6 PVE nodes (.4/.5/.6/.9/.12/.15:9100).
 ---

 ## Architecture (v4.0.0 — Direct: nginx → LiteLLM → GPU)
@@ -1,6 +1,4 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 kind: responsibility
 name: litellm-self-heal
 status: deployed
@@ -24,7 +22,6 @@ description: >
  inference, and agent keys. Applies remediation rules for common failures.
  Reports every action via Zulip DM and Gitea (SyslogSolution/health-logs).
 ---
---

 # LiteLLM Operations — Health Check + Self-Heal

@@ -68,15 +65,7 @@ Request → nginx:80 → LiteLLM:4000 → GPU(llama-server, parallel 2)

 ## LiteLLM Model Surface (ground truth — `/opt/inference-harness/litellm_config.yaml` on CT 116)

-`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`, `crew-auto` (new 2026-08-20).
-
-### Context Cap Split (2026-08-20)
-
- **Abiba (firstmate)**: 128K uncapped — unlimited context for primary workloads
- **Hermes agents** (mumuni, tanko, koby, koonimo): 128K uncapped
- **Crewmates** (ops, tune, verify, auth-keys, build): 64K capped — alias `crew-auto` enforces 64K limit
-
-Preferred implementation: uncap shared pool, add capped alias for crew-only.
+`model_name`s served: `qwen3.6-27B-code`, `gemma-4-12b`, `qwen3.6-35B-udq4`, `strix-moe`, `gpu-dense`, `gpu-light`, `syslog-auto`.

 - `syslog-auto` is a weighted router model: qwen3.6-27B-code (0.55, rpm 500) + qwen3.6-35B-udq4 (0.30, rpm 60) + gemma-4-12b (0.15, rpm 200).
 - `gpu-dense` / `gpu-light` are high-rpm aliases (rpm 500) onto qwen3.6-27B-code / gemma-4-12b respectively.
@@ -113,7 +102,7 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.

 - **Health-check script** (`/opt/inference-harness/scripts/litellm-health-check.sh` on CT 116): `gpu-fleet` check fails only on **critical** alerts (warnings are informational). Tests `strix-moe` (not `ornith-1.0-35b`).
 - **GPU monitor** (`/root/scripts/gpu-monitor-server.py` on pi .24): runs as **systemd unit `gpu-monitor.service`** (was bare `&` process). `gpu_count` includes Strix Halo (was 2, now 3). VRAM alert thresholds: warning 93%, critical 97% (raised from 90/95 — 128K context steady-state is ~70% on RTX 3090, not a fault).
- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v3 (2026-09-08) — vault-backed agents (tanko/koby/koonimo) read their **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key); abiba (pi agent) reads `LITELLM_API_KEY` from its local `/root/.pi/agent/env.sh` (#735 — moved out of shared `/root/.bashrc`), not from the vault. Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
+- **Agent key monitor** (`/root/scripts/agent-health-check.py` on pi .24, cron `*/10`): v2 (2026-07-26) — reads each agent's **agent-specific** `{NAME}_LITELLM_API_KEY` from Infisical vault (not the shared master key). Covers: LiteLLM keys, GPU ports, agent gateways (all 5 agents now SSHa ble), CT liveness (pct status on PVE nodes), config.yaml YAML integrity, wrapper/CLI integrity, vault secret non-emptiness checks. Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114), abiba (.24). Legacy `tdunna`/`baggy` replaced with canonical agent hostnames.
 - **Stale keys cleaned**: `daily-infra-report.py` SYNTHETIC_API_KEY was stale (`sk-U_ydi3B` → 401); now reads `LITELLM_MASTER_KEY` from env. Deprecated scripts (`router-original.py`, `router-phase0-backup.py`, `apply-fixes.py`) still reference `sk-syslog-local-master-key` but do not actively poll LiteLLM.

 ## Maintains
@@ -131,7 +120,6 @@ Preferred implementation: uncap shared pool, add capped alias for crew-only.
 - Also wakes on user request
 - On failure: re-check after 30s, escalate after 3 consecutive failures

---
 ---

 ## Health Check
@@ -174,7 +162,6 @@ Determine overall_status from individual check results:
 - "degraded" — 1-2 non-critical checks fail
 - "down" — critical checks fail

---
 ---

 ## Remediation Rules
@@ -218,7 +205,6 @@ Escalate → if SSH access unavailable, send Zulip DM
 Router no longer in path so Redis active counters are unused. Rule retained
 for reference but inactive. If Redis issues occur, check harness-redis container.

---
 ---

 ## Reporting
@@ -241,7 +227,6 @@ top actions, uptime.
 If a fix requires another agent (e.g., Authentik restart), relay sent
 to responsible agent with full context.

---
 ---

 ## Execution
@@ -1,12 +1,9 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 name: memory-audit-maintenance
 kind: responsibility
-description: Shared memory audit and maintenance contract for Hermes agents (Mumuni, Koby, Koonimo). Tanko is no longer a Hermes agent (now on DSH/DeepSeek Harness since 2026-08-27) and uses DSH-native memory, so it is excluded from this Hermes roster. Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---
---

 ### Goal

@@ -18,10 +15,11 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,

 ### Scope

-This contract is the **Hermes Agent standard** for memory maintenance. It is shared across Hermes agents (Mumuni, Koby, Koonimo). **Tanko is excluded — it migrated to DSH (DeepSeek Harness) on 2026-08-27 and now uses DSH-native memory (mnemon), not `~/.hermes/memories/`.** Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.

-**Agent Roster (Hermes):**
+**Agent Roster:**
 - Mumuni
+- Tanko
 - Koby (CT 111 / tdunna)
 - Koonimo (CT 113 / baggy)

@@ -345,3 +343,4 @@ return {

 ### Per-Agent Notes

+Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
@@ -1,6 +1,4 @@
 ---
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 kind: pattern
 name: memory-fixer
 description: >
@@ -8,7 +6,6 @@ description: >
  Escalate anything that needs Kwame's input. Executes confirmed Kwame decisions to completion (state + updated_at).
 version: 2.0.0
 ---
---

 # Memory Fixer

@@ -6,7 +6,7 @@ description: >
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
-  Runs on Mumuni (kagentz CT105, minipve, .14) via Hermes agent (Zulip gateway via systemd).
+  Runs on Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent (Pi + Hermes Zulip gateway).
 version: 1.0.0
 ---

@@ -19,8 +19,8 @@ version: 1.0.0

 ## Topology

-**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
-**Manager:** Mumuni (kagentz CT105, minipve, .14) via Hermes agent
+**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
+**Manager:** Mumuni (inside Abiba CT100, hwepve, .24) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed

 This contract is infrastructure-agnostic in terms of which nodes are used.
@@ -31,7 +31,7 @@ Workers execute tasks on whatever infrastructure they're given — SSH to .6,
 ## Why This Matters

 Without enforced delegation, the manager consumes the full iteration budget
-(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
+(60 calls) on single-turn tasks — SSH to 6 nodes, check each VM, read logs —
 leaving no capacity for actual coordination. The result: context overflow
 (59K tokens in system prompt), iteration exhaustion, and degraded response
 quality. This contract exists because I blew through my budget checking
@@ -82,7 +82,7 @@ it asks the manager (via relay) — it doesn't go find it on its own.

 **This is a hard rule, not a recommendation.** Violating it produces the exact
 type of discrepancy the kanban pipeline exists to prevent: a review worker finds
-"5 nodes present" in the raw data but "5/5 online" in the report — even though
+"6 nodes present" in the raw data but "6/6 online" in the report — even though
 one of those nodes was unreachable. The report lied because it used data the
 raw data never provided.

@@ -137,7 +137,7 @@ delegate_task(
 ```
 delegate_task(
    tasks=[
-        {"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
+        {"goal": "Check all 6 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
        {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
    ]
 )
@@ -191,7 +191,7 @@ Only verified results reach Kwame. Format per channel:
    {
      "lane_id": "devops-check",
      "worker": "syslog-devops",
-      "goal": "Check all 5 Proxmox nodes",
+      "goal": "Check all 6 Proxmox nodes",
      "status": "dispatched|completed|failed",
      "output_file": "/tmp/node-report.md"
    }
@@ -54,7 +54,7 @@ garbled input. When a user types these commands to Abiba:

 - **`/approve`** → "Pi doesn't have pending approvals. Commands execute immediately."
 - **`/approve session`** → Same response
- **`/deny`** → Same response + "For agents (Mumuni on Hermes, Tanko on DSH), these work with their built-in approval system."
+- **`/deny`** → Same response + "For Hermes agents (Tanko, Mumuni), these work with their built-in approval system."

 This keeps the UX consistent across agents — users can type `/approve` anywhere
 without getting confused by LLM responses.
@@ -2,6 +2,17 @@
 kind: responsibility
 name: pm2-self-heal
 description: >
+  Monitors critical PM2 processes (abiba-zulip, abiba-telegram, gitea-runner,
+  spoton-service, zulip-watchdog) and auto-restarts any that are stopped or
+  errored. Logs every action to Gitea (SyslogSolution/health-logs — not
+  knowledge graph, hard rule) and alerts the owner via
+  Zulip DM on failures.
+  CRITICAL: Never restart abiba-zulip — it runs this contract.
+  AS-BUILT 2026-08-09 (captain ruling, ecosystem is authoritative):
+  gpu-monitor is systemd-managed (gpu-monitor.service) — NOT PM2;
+  gpu-watchdog decommissioned (function folded into gpu-monitor.service);
+  gitea-runner KEPT (online in PM2); abiba-zulip KEPT (online 4d+, the
+  2026-07-04 'removed/decommissioned' note was stale and is removed).
 ---

 ## Maintains
@@ -13,6 +24,9 @@ description: >
 - zulip-watchdog: { status: "online", uptime: string, restarts: number }
 - last_check: timestamp

+> **Note (2026-07-04, SUPERSEDED 2026-08-09):** `abiba-zulip` remains ONLINE and
+> is monitored — the decommission note was stale (process re-added; do not treat
+> it as removed).

 ## Continuity

@@ -27,15 +41,10 @@ description: >
 - **Verify**: Re-check status after 5 seconds
 - **Escalate**: If still failed after 2 retries, send Zulip DM to owner

-### Rule 2: Process Restarting Too Often (crash-loop guard)
- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter) **or** a
-  process reporting a restart count > 1000 while showing "online" (a crash-loop mask)
+### Rule 2: Process Restarting Too Often
+- **Detect**: `pm2 status` shows restarts > 30 (cumulative lifetime counter)
 - **Note**: PM2 counter never decrements; only full delete+re-add resets it
 - **Fix**: `pm2 delete <name> && pm2 start <ecosystem> --only <name>`
- **Script guard (2026-08-16)**: `scripts/pm2-self-heal.sh` now restarts `abiba-telegram`
-  when `TEL_RESTARTS > 1000` even if the process reports "online" — catches a quiet
-  crash-loop that never toggles status to "stopped"/"errored" (e.g. the 10k-restarts
-  spoton incident). Alerts include the restart count.
 - **Escalate**: Only when restarts > 30 — alerts to Zulip DM
 - **Historical fix**: Previous cycles were caused by abiba-zulip extension's
  stuck detection (STUCK_THRESHOLD_MS was 30min, raised to 4h in v2).
@@ -49,9 +58,9 @@ description: >
   - If status is "online" → pass
   - If status is "stopped" or "errored" → apply Rule 1
   - If restarts > 5 → alert owner
-3. **Check abiba-zulip** (live Zulip bridge, heartbeating):
+3. **Check abiba-zulip** (self-process, read-only):
   - If status is "online" → pass, log restarts count
-   - If status is "stopped" or "errored" → restart (`pm2 restart abiba-zulip` — fully restored)
+   - If status is "stopped" or "errored" → **DO NOT RESTART** — alert owner immediately
   - If restarts > 5 in last hour → alert owner with full diagnostics
 4. **Log results** — Append to `SyslogSolution/health-logs/pm2/{timestamp}.md` in Gitea (not knowledge graph — hard rule)
 5. **Alert** — Send Zulip DM to owner if escalation needed (do NOT run pm2 commands during alerting)
@@ -5,7 +5,7 @@ description: >
  Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
  on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
  exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
-  instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
+  instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
  (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
  the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
 agent: abiba
@@ -33,8 +33,8 @@ agent: abiba

 | Exporter | Host:Port | Scope | Notes |
 |----------|-----------|-------|-------|
-| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
-| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
+| prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
+| node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
 | docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |

 ## PVE API Token
@@ -48,7 +48,7 @@ agent: abiba

 | UID | Title | Panels | Source |
 |-----|-------|--------|--------|
-| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
+| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
 | proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
 | docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
 | gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
@@ -77,9 +77,9 @@ agent: abiba
 | `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
 | `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
 | `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
-| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
+| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |

-## Cluster "Tabiri" — 5 Nodes
+## Cluster "Tabiri" — 6 Nodes

 | Node | IP | Role |
 |------|----|----|
@@ -88,6 +88,7 @@ agent: abiba
 | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
 | minipve | 192.168.68.12 | PVE |
 | amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) |
+| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts abiba (lxc/100), kagentz (lxc/105), mumuni (lxc/114). CTs 100/105 migrated from amdpve, CT 114 from minipve 2026-07-20 |

 ## Operations

@@ -100,32 +101,6 @@ Edit `build-dashboards.py`, run it, `docker restart harness-grafana`
 ### check-targets
 `curl http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job,health}'`

-### check-health
-
-**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
-
-```bash
-# Prometheus health (bound to 0.0.0.0:9090 on .116)
-curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:9090/-/healthy
-# Expected: 200 (Prometheus is up and healthy)
-
-# Grafana health (bound to 0.0.0.0:3001 on .116)
-curl -s -o /dev/null -w '%{http_code}' http://192.168.68.116:3001/api/health
-# Expected: 200 (Grafana is up and healthy)
-
-# Docker Stats exporter (bound to 127.0.0.1:9324 on .116 — must probe from .116 localhost)
-ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9324/metrics"
-# Expected: 200 (docker-stats-exporter is up and responding)
-
-# PVE exporter (bound to 127.0.0.1:9221 on .116 — must probe from .116 localhost)
-ssh root@192.168.68.116 "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:9221/metrics"
-# Expected: 200 (pve-exporter is up and responding)
-```
-
-**Report format**: Summarize actual results from each probe. If any probe returns non-200, flag as alert.
-
-**Note**: Docker Stats and PVE Exporter are bound to 127.0.0.1 (localhost-only) so they must be probed from .116 via SSH. Prometheus and Grafana are bound to 0.0.0.0 so they can be probed from the LAN.
-
 ### restart-exporter
 `cd /opt/monitoring && docker compose restart pve-exporter docker-stats`

@@ -19,13 +19,6 @@ Changelog:
    name format ({NAME}_LITELLM_API_KEY not LITELLM_API_KEY_{NAME}).
    Fleet roster: tanko (.122), mumuni (.24, inside abiba CT100), koby (.129), koonimo (.114),
    abiba (.24).
-  v3 (2026-09-08): GPU unit repoint verified live (.8 llama-chat-api.service,
-    .110 llama-server.service, .15 strix-server.service) — .8 was probing a stale
-    llama-server unit that reads inactive, producing false UNREACHABLE legs.
-    systemctl is-active no longer swallows non-zero exit as SSH failure.
-    Fixed UnboundLocalError on the abiba/koonimo gateway leg (pid unbound in the
-    summary f-string). Abiba's LiteLLM key now comes from /root/.pi/agent/env.sh
-    (#735 agent separation; creds moved out of shared /root/.bashrc).
 """

 import subprocess, json, sys, os, time
@@ -37,6 +30,7 @@ INFISICAL_ENV = "prod"

 # PVE node IPs for CT liveness checks
 PVE_NODES = {
+    "hwepve":  "192.168.68.4",
    "amdpve":  "192.168.68.15",
    "minipve": "192.168.68.12",
    "storepve": "192.168.68.6",
@@ -46,26 +40,16 @@ PVE_NODES = {

 # Agent definitions: ct, host, user, pve_node, vault_key_name
 AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve",  "vault_key": "TANKO_LITELLM_API_KEY", "runtime": "dsh"},
-    # abiba = pi agent (.24) — no vault key; its LiteLLM key is read from its
-    # local env file (key_env below), not from the shared vault or .bashrc.
-    "abiba":   {"ct": 100, "host": "192.168.68.24", "user": "root", "pve": "minipve",
-               "vault_key": None,
-               "key_env": {"file": "/root/.pi/agent/env.sh", "var": "LITELLM_API_KEY"}},
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "user": "jerome", "pve": "amdpve",  "vault_key": "TANKO_LITELLM_API_KEY"},
+    "abiba":   {"ct": 100, "host": "192.168.68.24",  "user": "root",   "pve": "hwepve",  "vault_key": None},  # Pi agent + Mumuni Zulip, no vault key
    "koby":    {"ct": 111, "host": "192.168.68.129", "user": "root",   "pve": "amdpve",  "vault_key": "KOBY_LITELLM_API_KEY"},
    "koonimo": {"ct": 113, "host": "192.168.68.114", "user": "root",   "pve": "amdpve",  "vault_key": "KOONIMO_LITELLM_API_KEY"},
 }

-# Systemd units verified live 2026-09-08 (systemctl list-units on each host):
-#   .8   rtx3090 (gpu-dense)  -> llama-chat-api.service  (active; the old
-#        llama-server.service unit file is stale/inactive — probing it read as
-#        UNREACHABLE for a healthy process)
-#   .110 rtx5070 (ocu-llm VM) -> llama-server.service    (active)
-#   .15  strixhalo (amdpve)   -> strix-server.service    (active)
 GPU_HOSTS = {
-    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-chat-api.service"},
-    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server.service"},
-    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "strix-server.service"},
+    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
+    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
+    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "strix-server"},
 }

 FAIL = []
@@ -178,41 +162,10 @@ def _get_agent_key(agent_name, vault_key_name):

    return None

-
-def _read_env_export(path, var):
-    """Parse `export VAR=value` (or `VAR=value`) out of a local env file.
-
-    #735 agent separation (2026-09-06): agent creds moved out of the shared
-    /root/.bashrc into per-agent env files under /root/.pi/agent/ (bashrc's
-    source line keeps abiba shells resolving them, but the file of record is
-    env.sh). Do NOT fall back to /root/.bashrc here: desktop (.200) SSH
-    sessions override LITELLM_API_KEY with mumuni's key, so sourcing bashrc
-    would validate the wrong identity.
-    """
-    try:
-        with open(os.path.expanduser(path)) as _f:
-            for line in _f:
-                line = line.strip()
-                if not (line.startswith("export " + var + "=") or line.startswith(var + "=")):
-                    continue
-                value = line.split("=", 1)[1].strip().strip('"').strip("'")
-                if value:
-                    return value
-    except (OSError, UnicodeDecodeError):
-        pass
-    return None
-
-
-# Inject keys for each agent:
-#   - vault-backed agents (tanko/koby/koonimo): {NAME}_LITELLM_API_KEY from
-#     Infisical (project 322fceab-39da-4854-a55a-568e76c0f13f, env prod).
-#   - abiba (pi agent, no vault key): LITELLM_API_KEY from its local env file
-#     /root/.pi/agent/env.sh (moved there from /root/.bashrc in #735).
+# Inject keys from vault for each agent
 for agent_name in AGENTS:
    info = AGENTS[agent_name]
    key = _get_agent_key(agent_name, info.get("vault_key"))
-    if not key and info.get("key_env"):
-        key = _read_env_export(info["key_env"]["file"], info["key_env"]["var"])
    AGENTS[agent_name]["key"] = key


@@ -224,7 +177,7 @@ def check_keys():
    for name, agent in AGENTS.items():
        key = agent.get("key")
        if not key:
-            print(f"  ❌ {name}: NO KEY FOUND (vault/env empty or unreachable)")
+            print(f"  ❌ {name}: NO KEY FOUND (vault empty or unreachable)")
            FAIL.append(f"key:{name}:no-key")
            continue
        data = http_json(f"{LITELLM}/v1/models",
@@ -238,7 +191,7 @@ def check_keys():


 # ═══════════════════════════════════════════════════════════════════
-# CHECK 2: GPU Port Conflict Detection (unit names verified live 2026-09-08)
+# CHECK 2: GPU Port Conflict Detection (unchanged)
 # ═══════════════════════════════════════════════════════════════════

 def check_gpu_ports():
@@ -247,11 +200,7 @@ def check_gpu_ports():
        port = gpu["port"]
        svc = gpu["service"]

-        # `systemctl is-active` exits non-zero when the unit is inactive or
-        # missing, which the ssh() helper would swallow as an SSH failure and
-        # report as UNREACHABLE. `|| true` keeps the real state word so we can
-        # tell "unit inactive" from "host unreachable".
-        svc_status = ssh(host, f"systemctl is-active {svc} || true")
+        svc_status = ssh(host, f"systemctl is-active {svc}")
        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")

        if not svc_status:
@@ -290,44 +239,20 @@ def check_agents():
        host = agent.get("host")
        user = agent.get("user")
        ct = agent["ct"]
-        report_only = agent.get("report_only", False)
-
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — it no longer runs a
-        # Hermes gateway, so skip the Hermes gateway/state/streaming/journal checks.
-        if agent.get("runtime") == "dsh":
-            live = ssh(host, "true", user=user)
-            print(f"  {'✅' if live is not None else '❌'} {name}: DSH (DeepSeek Harness) — "
-                  f"no Hermes gateway since 2026-08-27 (CT {ct}, SSH {'OK' if live is not None else 'FAIL'})")
-            if live is None:
-                FAIL.append(f"unreachable:{name}")
-            continue

        if not host or not user:
            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
            continue

-        # Resolve the Hermes gateway PID once, before the report-only branch:
-        # the summary line below renders `pid`, and it used to be bound only in
-        # the report-only path — leaving it unbound on the abiba/koonimo path
-        # raised UnboundLocalError and crashed the whole check. Agents without
-        # a gateway get pid=?.
-        pid = ssh(host, "pgrep -f '[h]ermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
+        # Gateway process
+        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
        if not pid:
-            pid = ssh(host, "pgrep -f '[h]ermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
+            # Try alternate binary name
+            pid = ssh(host, "pgrep -f 'hermes.*gateway' | grep -v infisical | grep -v bash | head -1", user=user)
        if not pid:
-            pid = "?"
-
-        # ⛔ KOBY IS NEVER REPAIRED — diagnostic only
-        if report_only:
-            print(f"  🔍 {name}: REPORT-ONLY mode (diagnostic only, no repairs on .129)")
-            # Still check gateway status for reporting purposes
-            if pid == "?":
-                print(f"  ⚠️  {name}: GATEWAY NOT RUNNING (reported only)")
-                FAIL.append(f"gateway-down:{name}")
-                continue
-            else:
-                print(f"  ✅ {name}: gateway running (pid={pid}, report-only mode)")
-                continue  # Skip the rest of the check for Koby
+            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
+            FAIL.append(f"gateway-down:{name}")
+            continue

        # Gateway state file
        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
@@ -403,10 +328,6 @@ def check_ct_liveness():
 def check_config_integrity():
    """Verify agent config.yaml parses as valid YAML."""
    for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no Hermes config.yaml.
-        if agent.get("runtime") == "dsh":
-            print(f"  ⏭️ {name}: DSH — no Hermes config.yaml since 2026-08-27")
-            continue
        host = agent.get("host")
        user = agent.get("user")
        if not host or not user:
@@ -436,10 +357,6 @@ def check_config_integrity():
 def check_wrapper_integrity():
    """Verify the hermes CLI wrapper exists and can reach hermes-real."""
    for name, agent in AGENTS.items():
-        # Tanko runs on DSH (DeepSeek Harness) since 2026-08-27 — no hermes CLI wrapper.
-        if agent.get("runtime") == "dsh":
-            print(f"  ⏭️ {name}: DSH — no hermes CLI wrapper since 2026-08-27")
-            continue
        host = agent.get("host")
        user = agent.get("user")
        if not host or not user:
@@ -296,16 +296,21 @@ def collect():
        "pm2_uptime": pm2.get("uptime", "?"),
    }
    
-    # Tanko (CT 112, IP 192.168.68.122) — DSH (DeepSeek Harness), no Hermes gateway
-    # since 2026-08-27. There is no ~/.hermes/gateway_state.json on CT 112 anymore;
-    # Zulip/Telegram connectivity is managed by the DSH harness, not the Hermes gateway.
+    # Tanko (CT 122)
+    tanko_state = ssh_jerome("192.168.68.122", "cat ~/.hermes/gateway_state.json 2>/dev/null")
+    tanko_data = {}
+    try:
+        tanko_data = json.loads(tanko_state) if tanko_state else {}
+    except:
+        tanko_data = {}
+    platforms = tanko_data.get("platforms", {})
    report["agents"]["tanko"] = {
-        "platform": "dsh", "ct": 112, "ip": "192.168.68.122",
-        "gateway_state": "n/a (DSH)",
-        "zulip_state": "unknown",
-        "telegram_state": "unknown",
-        "gateway_pid": None,
-        "updated_at": "",
+        "platform": "hermes", "ct": 112, "ip": "192.168.68.122",
+        "gateway_state": tanko_data.get("gateway_state", "unknown"),
+        "zulip_state": platforms.get("zulip", {}).get("state", "unknown"),
+        "telegram_state": platforms.get("telegram", {}).get("state", "unknown"),
+        "gateway_pid": tanko_data.get("pid"),
+        "updated_at": tanko_data.get("updated_at"),
    }
    
    # Mumuni (CT 100, IP 192.168.68.24)
@@ -317,7 +322,7 @@ def collect():
        mumuni_data = {}
    mumuni_platforms = mumuni_data.get("platforms", {})
    report["agents"]["mumuni"] = {
-        "platform": "hermes", "ct": 100, "ip": "192.168.68.24",
+        "platform": "hermes", "ct": 114, "ip": "192.168.68.24",
        "gateway_state": mumuni_data.get("gateway_state", "unknown"),
        "telegram_state": mumuni_platforms.get("telegram", {}).get("state", "unknown"),
        "zulip_state": mumuni_platforms.get("zulip", {}).get("state", "not_installed"),
@@ -538,7 +543,7 @@ th {{ color: #8b949e; font-weight: normal; }}
        elif name == "tanko":
            zulip_state = "✅" if agent.get("zulip_state") == "connected" else ("❌" if agent.get("zulip_state") == "disconnected" else "⬜")
            gateway = agent.get("gateway_state", "?")
-            processed = "DSH"
+            processed = agent.get("updated_at", "")[:10]
        elif name == "mumuni":
            zulip_state = "⬜" if agent.get("zulip_state") == "not_installed" else ("✅" if agent.get("zulip_state") == "connected" else "⬜")
            gateway = agent.get("gateway_state", "?")
@@ -28,8 +28,10 @@ declare -A CT_NODES=(
    [117]=storepve  # zulip
    [118]=storepve  # jdownloader
    # acerpve (192.168.68.9) — no CTs (bare metal GPU .8)
-    [100]=minipve   # abiba (was hwepve)
-    [105]=minipve   # kagentz (was hwepve)
+    # hwepve (192.168.68.4)
+    [100]=hwepve    # abiba (was amdpve)
+    [105]=hwepve    # kagentz (was amdpve)
+    [114]=hwepve    # mumuni (was minipve)
    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
    #
    # REMOVED CTs (migrated to bare metal, decommissioned, or VMs):
@@ -48,6 +50,7 @@ declare -A NODE_IPS=(
    [storepve]=192.168.68.6
    [acerpve]=192.168.68.9
    [ocupve]=192.168.68.5
+    [hwepve]=192.168.68.4
 )

 resolve_node() {
@@ -74,7 +77,7 @@ main() {
    if [[ $# -lt 1 ]]; then
        echo "Usage: pct-run <CT_ID> [command...]" >&2
        echo "  pct-run 112 cat /etc/hostname" >&2
-        echo "  pct-run 100 systemctl status hermes-gateway" >&2
+        echo "  pct-run 114 systemctl status hermes-gateway" >&2
        echo ""
        echo "Known CTs:" >&2
        for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
@@ -33,7 +33,7 @@ TEL_LINE=$(echo "$STATUS" | grep "abiba-telegram")
 TEL_STATUS=$(echo "$TEL_LINE" | awk -F'│' '{print $10}' | xargs)
 TEL_RESTARTS=$(echo "$TEL_LINE" | awk -F'│' '{print $9}' | xargs)

-if [ "$TEL_STATUS" != "online" ] || [ "$TEL_RESTARTS" -gt 1000 ]; then
+if [ "$TEL_STATUS" != "online" ]; then
    pm2 restart abiba-telegram > /dev/null 2>&1
    sleep 3
    TEL_LINE2=$(pm2 status --no-color 2>/dev/null | grep "abiba-telegram")
@@ -46,17 +46,18 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol

 The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:

-**Proxmox Cluster "Tabiri" (5 nodes):**
+**Proxmox Cluster "Tabiri" (6 nodes):**
 - amdpve (192.168.68.15): tanko, tdunna, baggy, scottdenya
- minipve (192.168.68.12): abiba, kagentz, adguard, authentik, gitea, syslog-api, infisical-vault
+- minipve (192.168.68.12): adguard, authentik, gitea, syslog-api, infisical-vault
 - storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, jdownloader, zulip
 - acerpve (192.168.68.9): llm-gpu
 - ocupve (192.168.68.5): ocu-llm
+- hwepve (192.168.68.4): abiba, kagentz, mumuni

 **CT IDs (verified 2026-07-24 against PVE API):**
 100:abiba  102:adguard  104:authentik  105:kagentz  106:ra-h-os
 107:pbs    108:media    110:gitea      111:tdunna   112:tanko
-113:baggy  115:scottdenya  116:syslog-api 117:zulip
+113:baggy  114:mumuni   115:scottdenya  116:syslog-api 117:zulip
 118:jdownloader  119:infisical-vault

 **NO CT 122, CT 123, or .19 exist in the cluster.**
@@ -64,7 +65,7 @@ The infrastructure-control.prose.md contract is the canonical reference for the
 **CRITICAL RULES (never regress):**
 1. NO /grafana/ nginx route — it was tried and reverted on 2026-07-02. Grafana is direct LAN at :3001.
 2. NO .19 IP — Zulip is CT 117 on storepve.
-3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 100 (inside Abiba). No CT 114 anywhere.
+3. NO CT 122/123 — Tanko=CT 112, Mumuni=CT 114.
 4. Strix Halo :8080 is FIREWALLED to .116 only — cannot be probed from abiba (.24).
 5. abiba-zulip PM2 process is DECOMMISSIONED (2026-07-04) — abiba uses Telegram only.

@@ -12,10 +12,10 @@ echo ""
 # Authorized agents for restricted contracts
 # Format: contract_pattern|authorized_agents (comma-separated)
 declare -A RESTRICTED
-RESTRICTED["infrastructure-control.prose.md"]="abiba,abiba-bot,tanko,tanko-bot,mumuni,mumuni-bot"
-RESTRICTED["proxmox-monitor.prose.md"]="abiba,abiba-bot"
-RESTRICTED["hermes-config-template.prose.md"]="abiba,abiba-bot,mumuni,mumuni-bot,tanko,tanko-bot"
-RESTRICTED["zulip-health.prose.md"]="abiba,abiba-bot,mumuni,mumuni-bot,tanko,tanko-bot"
+RESTRICTED["infrastructure-control.prose.md"]="abiba"
+RESTRICTED["proxmox-monitor.prose.md"]="abiba"
+RESTRICTED["hermes-config-template.prose.md"]="abiba,mumuni,tanko"
+RESTRICTED["zulip-health.prose.md"]="abiba,mumuni"
 RESTRICTED["scripts/pm2-self-heal.sh"]="abiba"
 RESTRICTED["scripts/prose-lint.sh"]="abiba"
 RESTRICTED["scripts/prose-ai-review.sh"]="abiba"
@@ -1,7 +1,7 @@
 #!/bin/bash
 # /root/scripts/zulip-monitor.sh — Zulip Mesh Health Monitor
 # Implements zulip-health.prose.md v2
-# Runs every 15 min via cron. Alerts: Zulip private DM to the owner plus a stream post to #agent-hub on topic 'zulip-health'.
+# Runs every 15 min via cron. Alerts via Telegram.
 set -euo pipefail

 ZULIP_SITE="https://chat.sysloggh.net"
@@ -29,9 +29,8 @@ notify() {
  local stream_content="${severity} Zulip Monitor: ${msg}"
  curl -sf -X POST "${ZULIP_SITE}/api/v1/messages" \
    -u "${ZULIP_EMAIL}:${ZULIP_KEY}" \
-    -d "type=stream&to=%5B7%5D&topic=zulip-health&content=$(printf '%s' "${stream_content}" | python3 -c "import sys,urllib.parse; print(urllib.parse.quote_from_bytes(sys.stdin.buffer.read()))")" \
-    > /dev/null 2>&1 \
-    || echo "  WARN: stream alert to #agent-hub (zulip-health) delivery failed (curl exit $?)" >> "$LOG"
+    -d "type=stream\&to=%5B7%5D\&topic=zulip-health\&content=$(python3 -c "import urllib.parse; print(urllib.parse.quote(str()))")" \
+    > /dev/null 2>&1 || true
 }

 # ── Global: Zulip Server ──
@@ -46,99 +45,43 @@ else
 fi

 # ── Platform A: pi (Abiba) ──
-# Probes the pi Zulip extension health endpoint (:9200/health, served by the
-# extension's startHealthServer; shape documented in zulip-health.prose.md).
-# FAIL-SAFE contract (pinned by tests/zulip-monitor-abiba.sh): connection state
-# lives NESTED at zulip.connected / zulip.last_error — there is no top-level
-# `connected` and no retry counter in the payload. A fetch error, non-2xx
-# response, empty/unparseable body, or payload missing a boolean
-# zulip.connected is a PROBE FAILURE: it alerts and NEVER calls pm2 restart.
-# pm2 restart runs ONLY on affirmative zulip.connected=false.
-# -- abiba-leg-start (verbatim-extracted by tests/zulip-monitor-abiba.sh)
-PI_HTTP=$(curl -s -o /dev/null --connect-timeout 5 --max-time 10 -w '%{http_code}' http://localhost:9200/health 2>/dev/null || echo "000")
-PI_BODY=$(curl -s --connect-timeout 5 --max-time 10 http://localhost:9200/health 2>/dev/null || true)
-PI_STATE=$(printf '%s' "$PI_BODY" | python3 -c '
-import sys, json
-code = sys.argv[1]
-body = sys.stdin.read()
-try:
-    d = json.loads(body)
-except Exception:
-    sys.stdout.write("probe-failed|unparseable body")
-    sys.exit(0)
-if not code.startswith("2"):
-    sys.stdout.write("probe-failed|HTTP %s" % code)
-    sys.exit(0)
-if not isinstance(d, dict) or not isinstance(d.get("zulip"), dict):
-    sys.stdout.write("probe-failed|missing zulip.connected")
-    sys.exit(0)
-z = d["zulip"]
-if "connected" not in z or not isinstance(z["connected"], bool):
-    sys.stdout.write("probe-failed|missing or non-boolean zulip.connected")
-    sys.exit(0)
-err = z.get("last_error") or ""
-if z["connected"]:
-    if err:
-        sys.stdout.write("degraded|%s" % err)
-    else:
-        sys.stdout.write("healthy|%s" % z.get("messages_processed", 0))
-else:
-    sys.stdout.write("disconnected|")
-' "$PI_HTTP" 2>/dev/null) || PI_STATE="probe-failed|python error"
-PI_VERDICT=${PI_STATE%%|*}
-PI_DETAIL=${PI_STATE#*|}
+PI_HEALTH=$(curl -sf --connect-timeout 5 http://localhost:9200/health 2>/dev/null || echo "{}")
+PI_CONNECTED=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('connected',False))" 2>/dev/null)
+PI_ERROR=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('last_error') or '')" 2>/dev/null)
+PI_RETRIES=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('retry_count',0))" 2>/dev/null)

-case "$PI_VERDICT" in
-  healthy)
-    echo "  Abiba: ✅ Connected (processed=$PI_DETAIL)" >> "$LOG" ;;
-  degraded)
-    notify "🟡" "Abiba pi extension error: ${PI_DETAIL:0:100}"
-    echo "  Abiba: 🟡 Error: ${PI_DETAIL:0:100}" >> "$LOG" ;;
-  disconnected)
-    notify "🔴" "Abiba pi extension DISCONNECTED — restarting"
-    pm2 restart abiba-zulip 2>/dev/null || true
-    ISSUES=$((ISSUES + 1))
-    echo "  Abiba: ❌ Disconnected — restarted" >> "$LOG" ;;
-  probe-failed)
-    notify "🟠" "Abiba pi extension health probe FAILED (${PI_DETAIL}; HTTP $PI_HTTP) — NOT restarting, manual check needed"
-    ISSUES=$((ISSUES + 1))
-    echo "  Abiba: ⚠️ Probe failed (${PI_DETAIL}; HTTP $PI_HTTP) — NOT restarted" >> "$LOG" ;;
-  *)
-    notify "🟠" "Abiba pi extension health probe returned unexpected verdict (${PI_STATE}) — NOT restarting, manual check needed"
-    ISSUES=$((ISSUES + 1))
-    echo "  Abiba: ⚠️ Unexpected probe verdict (${PI_STATE}) — NOT restarted" >> "$LOG" ;;
-esac
-# -- abiba-leg-end
-
-# ── Platform B: Tanko (DSH dsh-web on amdpve CT 112) ──
-# Direct SSH to 192.168.68.122 is not a dependency of this monitor — per-worker
-# key availability varies — so probes run from the amdpve vantage via `pct exec`.
-# Tanko's Zulip gateway runs as the dsh-web systemd unit inside CT 112 on amdpve
-# (192.168.68.15). The gateway binds 127.0.0.1:3080 loopback-only by design — a
-# remote :3080 probe is refused and is NOT a fault.
-TANKO_SVC=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
-  "pct exec 112 -- systemctl is-active dsh-web" 2>/dev/null || true)
-[ -n "$TANKO_SVC" ] || TANKO_SVC="unknown"
-TANKO_HTTP=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 root@192.168.68.15 \
-  "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/" 2>/dev/null || true)
-[ -n "$TANKO_HTTP" ] || TANKO_HTTP="000"
-
-if [ "$TANKO_SVC" != "active" ]; then
-  notify "🔴" "Tanko (DSH dsh-web) service state: $TANKO_SVC — needs restart"
+if [ "$PI_CONNECTED" != "True" ]; then
+  notify "🔴" "Abiba pi extension DISCONNECTED — restarting"
+  pm2 restart abiba-zulip 2>/dev/null || true
  ISSUES=$((ISSUES + 1))
-  echo "  Tanko: ❌ service=$TANKO_SVC" >> "$LOG"
-elif [ "$TANKO_HTTP" = "000" ]; then
-  notify "🔴" "Tanko (DSH dsh-web) HTTP :3080 connection refused/timeout — needs restart"
-  ISSUES=$((ISSUES + 1))
-  echo "  Tanko: ❌ http=000 (refused/timeout)" >> "$LOG"
+  echo "  Abiba: ❌ Disconnected — restarted" >> "$LOG"
+elif [ -n "$PI_ERROR" ]; then
+  notify "🟡" "Abiba pi extension error: ${PI_ERROR:0:100}"
+  echo "  Abiba: 🟡 Error: ${PI_ERROR:0:100}" >> "$LOG"
+elif [ "$PI_RETRIES" -ge 3 ]; then
+  notify "🟡" "Abiba pi extension: $PI_RETRIES retries — restarting"
+  pm2 restart abiba-zulip 2>/dev/null || true
+  echo "  Abiba: 🟡 $PI_RETRIES retries — restarted" >> "$LOG"
 else
-  case "$TANKO_HTTP" in
-    200|301|302|307|308|401|403)
-      echo "  Tanko: ✅ service=active http=$TANKO_HTTP" >> "$LOG" ;;
-    *)
-      notify "🟡" "Tanko (DSH dsh-web) HTTP :3080 answered $TANKO_HTTP — running, unexpected status"
-      echo "  Tanko: 🟡 service=active http=$TANKO_HTTP (running, warning)" >> "$LOG" ;;
-  esac
+  echo "  Abiba: ✅ Connected (processed=$(echo "$PI_HEALTH" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('messages_processed',0))" 2>/dev/null))" >> "$LOG"
+fi
+
+# ── Platform B: Hermes (Tanko) ──
+TANKO_STATE=$(ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 jerome@192.168.68.122 \
+  "cat ~/.hermes/gateway_state.json 2>/dev/null" 2>/dev/null || echo "{}")
+TANKO_ZULIP=$(echo "$TANKO_STATE" | python3 -c "
+import sys,json
+d=json.load(sys.stdin)
+p=d.get('platforms',{}).get('zulip',{})
+print(p.get('state','unknown'))
+" 2>/dev/null)
+
+if [ "$TANKO_ZULIP" != "connected" ]; then
+  notify "🔴" "Tanko (Hermes) Zulip state: $TANKO_ZULIP — needs restart"
+  ISSUES=$((ISSUES + 1))
+  echo "  Tanko: ❌ state=$TANKO_ZULIP" >> "$LOG"
+else
+  echo "  Tanko: ✅ Zulip connected" >> "$LOG"
 fi

 # ── Platform B: Hermes (Mumuni) ──
@@ -1,25 +0,0 @@
-{
-    "status": "ok",
-    "platform": "pi",
-    "agent": "abiba",
-    "zulip": {
-        "connected": true,
-        "site": "https://chat.sysloggh.net",
-        "email": "abiba-bot@chat.sysloggh.net",
-        "queue_id": "ee7f8b6d-9d53-48a7-ad58-f6e999771001",
-        "bot_user_id": 21,
-        "messages_processed": 0,
-        "skipped": 0,
-        "last_error": null
-    },
-    "circuit_breaker": {
-        "state": "CLOSED",
-        "failures": 0,
-        "successes": 5,
-        "totalRequests": 5,
-        "failureRate": "0.000",
-        "openedAt": null
-    },
-    "workers": [],
-    "worker_count": 0
-}
@@ -1,25 +0,0 @@
-{
-    "status": "down",
-    "platform": "pi",
-    "agent": "abiba",
-    "zulip": {
-        "connected": false,
-        "site": "https://chat.sysloggh.net",
-        "email": "abiba-bot@chat.sysloggh.net",
-        "queue_id": null,
-        "bot_user_id": null,
-        "messages_processed": 0,
-        "skipped": 0,
-        "last_error": "Zulip API error 401: queue registration failed"
-    },
-    "circuit_breaker": {
-        "state": "CLOSED",
-        "failures": 0,
-        "successes": 0,
-        "totalRequests": 0,
-        "failureRate": "0.000",
-        "openedAt": null
-    },
-    "workers": [],
-    "worker_count": 0
-}
@@ -1,211 +0,0 @@
-#!/bin/bash
-# tests/zulip-monitor-abiba.sh — regression test pinning the producer→consumer
-# contract between the pi Zulip extension's :9200/health payload and the Abiba
-# leg of scripts/zulip-monitor.sh.
-#
-# WHY THIS TEST EXISTS: 2026-09-09 live incident. The monitor parsed the health
-# payload at the WRONG nesting level (d.get('connected') at top level, while the
-# extension serves zulip.connected) so PI_CONNECTED was always False and every
-# monitor run restarted a healthy bot: pm2 showed restarts=8 with the process
-# created 2026-09-09T09:35:09Z, the monitor log recorded four ❌ Abiba verdicts
-# (04:23, 05:35, 06:55, 09:35 UTC) and zero ✅, while the Zulip server answered
-# HTTP 200 and the bot logged a clean connect plus continuing heartbeats. The
-# watchdog was the fault, not the connection. This test makes that class of
-# regression fail loudly instead of silently restarting healthy services.
-#
-# CONTRACT UNDER TEST (must hold for scripts/zulip-monitor.sh):
-#   * Connection state is NESTED: zulip.connected (boolean) and zulip.last_error
-#     live inside the `zulip` object. There is NO top-level `connected` and NO
-#     retry counter anywhere in the payload (verified against the extension's
-#     startHealthServer handler) — the old retry_count branch was dropped.
-#   * zulip.connected=true                  -> log "✅ Connected", NO pm2 restart.
-#   * zulip.connected=false                 -> alert, pm2 restart abiba-zulip.
-#   * fetch error / non-2xx / empty body / unparseable body / missing or
-#     non-boolean zulip.connected           -> "⚠️ Probe failed" alert with a
-#     "NOT restarting" label, NO pm2 restart. A parse miss must never kill a
-#     healthy service.
-#   * zulip.connected=true with last_error  -> degraded 🟡 warning, no restart.
-#
-# HOW: the Abiba leg of the shipped script sits between the
-# `# -- abiba-leg-start` / `# -- abiba-leg-end` marker comments. This runner
-# extracts that block verbatim and executes it with a stubbed curl (fixture body
-# + HTTP code), recorded notify()/pm2 shims, and a temp $LOG. If the markers
-# disappear (fix reverted or renamed) extraction yields nothing and the suite
-# fails — the bug cannot return silently.
-#
-# Usage: bash tests/zulip-monitor-abiba.sh [path/to/zulip-monitor.sh]
-# Exit 0 iff every check passes.
-#
-# shellcheck disable=SC2034,SC2329,SC1090
-# LOG/ISSUES and the notify/pm2/curl stubs below are consumed at runtime by
-# the leg extracted between the marker comments and `source`d in each case;
-# the static analyzer cannot see across that dynamic source, so it flags them.
-set -uo pipefail
-
-ROOT=$(cd "$(dirname "$0")/.." && pwd)
-SCRIPT=${1:-"$ROOT/scripts/zulip-monitor.sh"}
-FIXTURES="$ROOT/tests/fixtures"
-TMP=$(mktemp -d)
-trap 'rm -rf "$TMP"' EXIT
-
-PASS=0
-FAIL=0
-ok()  { PASS=$((PASS + 1)); printf '  \033[32m✔\033[0m %s\n' "$1"; }
-bad() { FAIL=$((FAIL + 1)); printf '  \033[31m✘\033[0m %s\n' "$1"; }
-
-echo "== tests/zulip-monitor-abiba.sh — Abiba leg vs :9200/health producer contract =="
-echo "target script: $SCRIPT"
-
-# --- structural guards -------------------------------------------------------
-if ! grep -q '^# -- abiba-leg-start' "$SCRIPT"; then
-  echo "✘ FATAL: $SCRIPT has no '# -- abiba-leg-start' marker — the fix has been reverted or renamed."
-  exit 1
-fi
-if ! grep -q '^# -- abiba-leg-end' "$SCRIPT"; then
-  echo "✘ FATAL: $SCRIPT has no '# -- abiba-leg-end' marker."
-  exit 1
-fi
-
-LEG="$TMP/leg.sh"
-awk '/^# -- abiba-leg-start/{f=1; next}
-     /^# -- abiba-leg-end/{f=0; next}
-     f' "$SCRIPT" > "$LEG"
-if [ ! -s "$LEG" ]; then
-  echo "✘ FATAL: extracted Abiba leg is empty."
-  exit 1
-fi
-echo "== structural =="
-if bash -n "$SCRIPT"; then ok "syntax: bash -n $SCRIPT"; else bad "syntax: bash -n $SCRIPT failed"; fi
-if bash -n "$LEG"; then ok "syntax: extracted leg parses (bash -n)"; else bad "syntax: extracted leg fails bash -n"; fi
-
-# --- per-case harness ---------------------------------------------------------
-CURRENT_NAME=""
-CURRENT_DIR=""
-
-# $1 case name, $2 http-code, $3 body (file path or literal)
-run_case() {
-  local name="$1" http="$2" body_src="$3" body
-  CURRENT_NAME="$name"
-  CURRENT_DIR=$(mktemp -d "$TMP/case.XXXXXX")
-  if [ -f "$body_src" ]; then
-    body=$(cat "$body_src")
-  else
-    body="$body_src"
-  fi
-  (
-    LOG="$CURRENT_DIR/log"; ISSUES=0
-    notify() { printf 'ALERT [%s] %s\n' "$1" "$2" >> "$CURRENT_DIR/alerts"; }
-    pm2()    { printf 'PM2 %s\n' "$*"    >> "$CURRENT_DIR/pm2"; }
-    curl() {
-      local url=""
-      for a in "$@"; do case "$a" in http*) url="$a";; esac; done
-      case "$url" in
-        *:9200/health*)
-          case " $* " in
-            *"-w"*) printf '%s' "$http" ;;   # -w '%{http_code}' code probe
-            *)      printf '%s' "$body"  ;;   # body probe
-          esac ;;
-        *)
-          printf 'UNEXPECTED-CURL %s\n' "$*" >> "$CURRENT_DIR/unexpected-curl"
-          return 7 ;;
-      esac
-      return 0
-    }
-    source "$LEG"
-  )
-}
-
-assert_log_has() {
-  if grep -qF -- "$1" "$CURRENT_DIR/log"; then ok "$CURRENT_NAME — log has: $1"; else bad "$CURRENT_NAME — log MISSING: $1"; fi
-}
-assert_log_lacks() {
-  if grep -qF -- "$1" "$CURRENT_DIR/log"; then bad "$CURRENT_NAME — log must NOT contain: $1"; else ok "$CURRENT_NAME — log correctly lacks: $1"; fi
-}
-assert_alert_has() {
-  if grep -qF -- "$1" "$CURRENT_DIR/alerts"; then ok "$CURRENT_NAME — alert sent: $1"; else bad "$CURRENT_NAME — alert MISSING: $1"; fi
-}
-assert_alert_empty() {
-  if [ ! -s "$CURRENT_DIR/alerts" ]; then ok "$CURRENT_NAME — no alert sent (quiet healthy path)"; else bad "$CURRENT_NAME — unexpected alert: $(cat "$CURRENT_DIR/alerts")"; fi
-}
-assert_pm2_restarted() {
-  if grep -qF "PM2 restart abiba-zulip" "$CURRENT_DIR/pm2"; then ok "$CURRENT_NAME — pm2 restart abiba-zulip was called"; else bad "$CURRENT_NAME — expected pm2 restart abiba-zulip, pm2 log: $(cat "$CURRENT_DIR/pm2" 2>/dev/null)"; fi
-}
-assert_no_restart() {
-  if [ ! -s "$CURRENT_DIR/pm2" ]; then ok "$CURRENT_NAME — NO pm2 restart (fail-safe holds)"; else bad "$CURRENT_NAME — pm2 was called but must NOT be: $(cat "$CURRENT_DIR/pm2")"; fi
-}
-assert_no_unexpected_curl() {
-  if [ ! -s "$CURRENT_DIR/unexpected-curl" ]; then ok "$CURRENT_NAME — only :9200/health was probed"; else bad "$CURRENT_NAME — unexpected curl: $(cat "$CURRENT_DIR/unexpected-curl")"; fi
-}
-
-# --- case 1: real payload shape, zulip.connected=true -> healthy, no restart --
-echo "== case 1: connected (real producer payload: nested zulip.connected=true) =="
-run_case "connected" 200 "$FIXTURES/zulip-health-connected.json"
-assert_log_has "Abiba: ✅ Connected (processed=0)"
-assert_log_lacks "Disconnected"
-assert_alert_empty
-assert_no_restart
-assert_no_unexpected_curl
-
-# --- case 2: zulip.connected=false -> disconnected, restart -------------------
-echo "== case 2: disconnected (nested zulip.connected=false triggers restart) =="
-run_case "disconnected" 200 "$FIXTURES/zulip-health-disconnected.json"
-assert_log_has "Abiba: ❌ Disconnected — restarted"
-assert_alert_has "DISCONNECTED — restarting"
-assert_pm2_restarted
-assert_no_unexpected_curl
-
-# --- cases 3-9: probe failures must alert and MUST NOT restart ----------------
-echo "== probe-failure cases: alert 'NOT restarting', zero pm2 restarts =="
-
-run_case "empty body" 200 ""
-assert_log_has "Abiba: ⚠️ Probe failed"
-assert_log_lacks "❌ Disconnected"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-run_case "garbage body" 200 '{not valid json!!'
-assert_log_has "Abiba: ⚠️ Probe failed"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-run_case "missing zulip key" 200 '{"status":"ok","platform":"pi","agent":"abiba"}'
-assert_log_has "Probe failed"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-run_case "zulip without connected" 200 '{"status":"ok","zulip":{"last_error":null}}'
-assert_log_has "Probe failed"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-run_case "non-boolean connected" 200 '{"status":"ok","zulip":{"connected":"true"}}'
-assert_log_has "Probe failed"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-run_case "fetch failure http 000" 000 ""
-assert_log_has "Probe failed"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-run_case "non-2xx http 500" 500 '{"error":"boom"}'
-assert_log_has "Probe failed"
-assert_alert_has "NOT restarting"
-assert_no_restart
-
-# --- case 10: connected but last_error set -> degraded 🟡, no restart ---------
-echo "== case 10: degraded (connected=true but last_error set) warns, no restart =="
-run_case "degraded" 200 '{"status":"ok","zulip":{"connected":true,"last_error":"transient queue hiccup","messages_processed":3}}'
-assert_log_has "Abiba: 🟡 Error: transient queue hiccup"
-assert_log_lacks "❌ Disconnected"
-assert_no_restart
-
-# --- summary -------------------------------------------------------------------
-echo ""
-if [ "$FAIL" -eq 0 ]; then
-  echo "✅ ALL CHECKS PASSED ($PASS/$PASS) — tests/zulip-monitor-abiba.sh"
-  exit 0
-else
-  echo "❌ $FAIL CHECK(S) FAILED ($PASS passed) — tests/zulip-monitor-abiba.sh"
-  exit 1
-fi
@@ -1,24 +1,22 @@
 ---
 kind: responsibility
 name: zulip-health
-description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Tanko on DSH / Mumuni on Hermes), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
+description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
 title: Zulip Mesh Health Monitor — Multi-Platform
 version: 3.0.0
 runtime_contract: 2
 agent: abiba
-report_only_agents:
-  - koby  # ⛔ KOBY IS NEVER REPAIRED (Rule 17, 2026-08-17) — detect + report, never fix on .129
 ---

 # Zulip Mesh Health Monitor

-Monitors ALL Zulip-connected agents across platforms (pi, Hermes, DSH, Agent Zero).
+Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
 Runs every 15 minutes in the background. Also triggers on session start.

 ## Requires

 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to amdpve (192.168.68.15) for Tanko — CT 112 reached via `pct exec` (direct SSH to .122 is not a dependency of this contract: per-worker key availability varies); Mumuni (192.168.68.14, kagentz CT105 on minipve); and Agent Zero Docker host (192.168.68.14)
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.24, inside Abiba CT100 on hwepve), and Agent Zero Docker host (192.168.68.14)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -47,9 +45,11 @@ Runs every 15 minutes in the background. Also triggers on session start.
    "severity": "healthy"
  },
  "tanko": {
-    "platform": "dsh",
-    "service_state": "active",
-    "http_status": 200,
+    "platform": "hermes",
+    "zulip_state": "connected",
+    "heartbeat_age_seconds": 45,
+    "gateway_pid": 1234,
+    "edit_fail_rate_pct": 0,
    "severity": "healthy"
  }
 }
@@ -81,13 +81,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
 ## Streaming Support (2026-07-05)

 Zulip agents now support progressive message editing during agent generation.
-When a Zulip agent (Tanko on DSH, Mumuni on Hermes) processes a message, the response is
+When a Hermes agent (Tanko, Mumuni) processes a message, the response is
 streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:

 - Adapter implements `edit_message()` using `_api_patch()` helper
 - Gateway stream consumer progressively edits the Zulip message
 - User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112) and Mumuni (kagentz CT 105) both have streaming active
+- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active

 ### Verification
 ```bash
@@ -182,90 +182,45 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |

+### Step 3: Platform B — Hermes (Tanko .122, Mumuni .24)

-### Step 3: Platform B — Tanko (DSH on amdpve CT 112) & Mumuni (Hermes)
-
-Tanko runs on DSH (DeepSeek Harness) — it no longer runs a Hermes gateway, so
-there is no `~/.hermes/gateway_state.json` on CT 112. Tanko's Zulip gateway runs
-as the `dsh-web` systemd unit inside **CT 112**, which resides on the **amdpve**
-PVE host (**192.168.68.15**). Direct SSH to 192.168.68.122 is not a dependency
-of this contract — per-worker key availability varies — so CT 112 probes run
-from the amdpve vantage via `pct exec`:
+**B1: Gateway State**

 ```bash
-ssh root@192.168.68.15 "pct exec 112 -- <command>"
+ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
+ssh root@192.168.68.24 "cat ~/.hermes/gateway_state.json"  # Mumuni inside Abiba CT100
 ```

-> **By design (verified 2026-09-08):** the `dsh-web` gateway binds
-> `127.0.0.1:3080` **loopback-only**. A remote probe against
-> `192.168.68.122:3080` gets connection-refused — that is EXPECTED, NOT a fault,
-> and must never be raised as Tanko down. Only loopback probes from inside
-> CT 112 (or the public-URL fallback below) are valid health signals.
+Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.

-**B1: Gateway Service State (Tanko)**
-
-```bash
-ssh root@192.168.68.15 "pct exec 112 -- systemctl is-active dsh-web"
-```
-
-Expected: `active`. Anything else → gateway service down → apply the Tanko heal
-(restart via DSH service, Platform B Actions table below).
-
-**B2: Gateway HTTP Liveness (Tanko — loopback-only :3080)**
-
-```bash
-ssh root@192.168.68.15 "pct exec 112 -- curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' http://127.0.0.1:3080/"
-```
-
-Alive = **ANY** HTTP status response from the endpoint — the expected set is
-`200`/`301`/`302`/`307`/`308`/`401`/`403` (the gateway UI is token-gated and
-legitimately answers with redirects/auth-challenges, so never require a bare
-`200`), and any other status, including `404`/`5xx`, also counts alive: a
-process answering `503` is running and self-heal must NOT restart-loop it.
-Down = connection refused (`000`) or timeout only. Statuses outside the
-expected set are logged/reported as a warning — reported, never healed on.
-
-**B3: Public-URL Fallback Probe (Tanko — for nodes without pct/ssh access to amdpve)**
-
-```bash
-curl -s --connect-timeout 10 --max-time 15 -o /dev/null -w '%{http_code}' https://tankodhs.sysloggh.net/
-```
-
-Fallback only — used when the monitoring node has no pct/SSH path to amdpve.
-Alive = **ANY** HTTP status response from the endpoint — healthy signals are
-`302` (authentik proxy-auth redirect) and `401` (auth-gated), and any other
-status, including `404`/`5xx`, also counts alive: the endpoint is up and
-answering and must NOT be restart-looped. Down = connection refused (`000`) or
-timeout only. Never expect a bare `200` — the public URL terminates in the
-token-gated authentik chain. Statuses outside the healthy set are
-logged/reported as a warning — reported, never healed on.
-
-**B4: Gateway Process** (Hermes agent Mumuni only — Tanko runs no Hermes gateway)
+**B2: Agent Process**

 ```bash
 ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
 ```

-Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
-than one `gateway run` process is found, the gateway has a collision (typically
-one `--force` and one `--replace` process). Kill the newer/duplicate process,
-then restart the remaining gateway per-agent (parameterized 2026-08-09, captain
-ruling). Check the gateway log for "Gateway running with 2 platform(s)" (not 1)
-to confirm Zulip reloaded.
+Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway per-agent (parameterized 2026-08-09, captain ruling):

-**B5: Heartbeat Verification** (Hermes agent Mumuni only — Tanko has no Hermes gateway)
+| Agent | Restart command | Notes |
+|-------|-----------------|-------|
+| Mumuni (.24) | `pm2 restart abiba-zulip` | Hermes gateway runs under PM2 as `abiba-zulip` |
+| Tanko (.122) | `bash /opt/hermes-zulip-plugin/run.sh` (or the agent's systemd/user unit) | Tanko does NOT use PM2 — never run `pm2 restart mumuni-zulip` for Tanko (process does not exist) |
+
+Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
+
+**B3: Heartbeat Verification**

 ```bash
-ssh root@192.168.68.24 "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
+ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
 ```

 Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
 Silence > 300s → warning. Silence > 600s → critical.

-**B6: Response Delivery** (Hermes agent Mumuni only)
+**B4: Response Delivery**

 ```bash
-ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
+ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
 ```

 > 50% fail rate → critical.
@@ -274,7 +229,7 @@ ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.her

 | Condition | Action |
 |-----------|--------|
-| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` (Mumuni) / restart Tanko via DSH service |
+| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
 | No heartbeat in 10min | Same as above |
 | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
 | Response empty/short | Check A2A endpoint / LiteLLM model |
@@ -284,11 +239,10 @@ ssh root@192.168.68.24 "grep -E 'Finalized|Failed to finalize|Replied to' ~/.her
 **C1: A2A Server Health**

 ```bash
-# A2A server is on :50080 (not :8001) and is auth-gated (401 expected for unauthenticated)
-ssh root@192.168.68.14 "curl -s --connect-timeout 5 -o /dev/null -w '%{http_code}' http://127.0.0.1:50080/a2a/"
+ssh root@192.168.68.14 "docker exec agent-zero curl -s --connect-timeout 5 http://127.0.0.1:8001/.well-known/agent.json"
 ```

-Expected: `401` (auth-gated, A2A server is up and responding) or `200` (if no auth required). Connection refused (000) → A2A server down.
+Expected: `{"name":"kagentz",...}`. Connection refused → A2A server down.

 **C2: Adapter Process**

@@ -309,14 +263,12 @@ Check: `processed=N` incrementing, `silence < 600s`, `reconnects` ≈ 0.
 **C4: A2A Response Verification**

 ```bash
-# A2A server is on :50080 (not :8001) and is auth-gated (401 expected for unauthenticated)
-ssh root@192.168.68.14 "curl -s -X POST http://127.0.0.1:50080/a2a \
+ssh root@192.168.68.14 "docker exec agent-zero curl -s -X POST http://127.0.0.1:8001/a2a \
  -H 'Content-Type: application/json' \
-  -H 'Authorization: Bearer $LITELLM_KEY' \
  -d '{\"jsonrpc\":\"2.0\",\"method\":\"tasks/send\",\"params\":{\"message\":{\"role\":\"user\",\"parts\":[{\"text\":\"ping\"}]}},\"id\":1}'"
 ```

-Expected: task ID with "working" status. Poll for completion with `tasks/get`. If 401, check LITELLM_KEY is set.
+Expected: task ID with "working" status. Poll for completion with `tasks/get`.

 **Platform C Actions**

@@ -372,6 +324,7 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
   - **Persistent failure threshold**: Only restart after 3 consecutive failed checks (45 min) AND no self-healing in progress
   - **Debounce gate**: Skip all checks entirely if recently restarted

+4. **Zulip stream delivery** — Alerts posted to #agent-hub on topic 'zulip-health' (stream_id 7) instead of email, keeping DM to captain for critical (🔴) alerts
 ### Gen 4 (2026-06-29) — Response Pipeline Fix

 **Root Cause**: LLM hangs due to GPU saturation (503 QUEUE_TIMEOUT) → pi's `agent_end` never fires → pending Zulip replies accumulate with no timeout. Health endpoint showed `connected: true, stuck: false` while user experienced complete silence.
@@ -10,7 +10,7 @@ description: >

 > **⚠️ RETIRED** — The pi Zulip extension (`~/.pi/agent/extensions/zulip/`) and
 > PM2 process (`abiba-zulip`) have been decommissioned. All mention/reliability
-> monitoring now happens through Telegram. Agents (Mumuni on Hermes, Tanko on DSH) and
+> monitoring now happens through Telegram. Hermes agents (Tanko, Mumuni) and
 > Agent Zero (kagentz) continue to use Zulip.

 ## Maintains
@@ -420,7 +420,7 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
 |-------|----------|-------------|--------------|-------------|
 | **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
 | **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
-| **Mumuni** | Hermes (kagentz CT 105, migrated 2026-08-29) | ✅ Connected | No issues found | None needed |
+| **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |

 ### Key Fixes Applied

@@ -13,8 +13,7 @@ triggers:

 > **⚠️ RETIRED** — This contract was embedded in the pi Zulip extension code
 > (`performHealthCheck()`). That code has been removed. Zulip self-healing for
-> agents (Mumuni on Hermes, Tanko on DSH) continues through their own platform
-> monitoring.
+> Hermes agents (Tanko, Mumuni) continues through their own gateway monitoring.

 ## Maintains

@@ -66,7 +65,8 @@ triggers:
 |------|----|------|---------|
 | Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
 | Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
-| Tanko | 192.168.68.122 (CT 112) | jerome | DSH (DeepSeek Harness) — restart via DSH service, not `hermes gateway restart` (no longer a Hermes agent since 2026-08-27) |
+| Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` |
+| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |

 ## Debounce
Author	SHA1	Message	Date
root	27bc6bafba	Fix zulip-monitor.sh LOG variable scope bug Move LOG, TIMESTAMP, and ISSUES declarations from inside notify() to script scope so they are available throughout the script.	2026-08-29 22:09:46 +00:00
root	97dd261123	Fix zulip-monitor.sh email spam: Replace Gmail SMTP with Zulip stream post to #agent-hub on topic 'zulip-health'	2026-08-29 21:30:54 +00:00