feat: hermes-zulip-restore contract — restore Zulip on any Hermes agent

Deploys zulip-platform adapter to bundled plugin path, verifies env credentials, restarts gateway, and validates Zulip connection. Covers Mumuni (CT114), Tanko (CT112), Koby (CT111). Includes _strip_html slash-command fix, known failure modes, and owner-aware restart paths.
2026-07-08 03:19:01 -04:00
14 changed files with 284 additions and 900 deletions
@@ -21,7 +21,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -34,7 +34,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -64,7 +64,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -77,7 +77,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -21,11 +21,9 @@ description: >
  - connected == true — Establishes and maintains Zulip event queue
  - reconnect_on_failure == true — Reconnects after queue expiry
  - recovers_from_network_loss == true — Handles temporary network drops
-  - stream_subscription_check == true — Bot is subscribed to primary streams (Gen 5+)
  
  *Message Flow*
  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
-  - dm_response_time_offline_verifiable == true — simulate_dm_latency() passes without live Zulip (Gen 6+)
  - stream_mention_detection == true — @mentions in streams are detected and routed
  - all_bots_detection == true — @all-bots mentions are detected
  - echo_loop_prevention == true — Never replies to its own messages
@@ -40,14 +38,6 @@ description: >
  - graceful_disconnect == true — shutdown doesn't crash
  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
  - typing_indicators_work == true — Sends typing notifications
-  - known_issues_tracked == true — add_known_issue() persists problems across generations (Gen 6+)
-
-## Status
-
-**Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
-for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
-Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
-plugin system, which is unaffected.

 ## Parameters

@@ -100,28 +90,13 @@ Each generation stores:

 ## Execution

-1. **Read state** — Check previous generations from knowledge graph node `build-zulip-plugin:state`
+1. **Read state** — Check previous generations from knowledge graph
 2. **Generate or improve** — Based on generation count:
   - Generation 1: Scaffold full plugin from scratch
   - Generation N: Read previous code, apply improvements
 3. **Validate** — Syntax check, structure check, dependency check
-4. **Run offline tests** — Always run `simulate_dm_latency()` and selftest() if no live Zulip
-5. **Log** — Save generation result to knowledge graph as [LEARN] node. Update the
-   `build-zulip-plugin:state` node with current generation_count, plugin_version,
-   and known_issues.
-6. **Report** — Output what was generated, what changed, what needs review
-
-## State Persistence (Gen 6+)
-
-All generation state is stored in a knowledge graph node titled `build-zulip-plugin:state`:
- `generation_count` — incremented on each verified run
- `plugin_version` — current version string
- `last_generation` — ISO timestamp of most recent run
- `known_issues` — array of unresolved issues carried forward
- `last_postcondition_results` — pass/fail map from most recent verification
-
-Agents executing this contract MUST read this node on startup and update it after
-validation. If the node does not exist, create it with generation_count=1.
+4. **Log** — Save generation result to knowledge graph as [LEARN]
+5. **Report** — Output what was generated, what changed, what needs review

 ## Example Output

@@ -5,9 +5,8 @@ description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
-  Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2
-  on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully
-  deprecated — nginx routes /v1 → LiteLLM directly.
+  Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
+  router runs behind LiteLLM on :9000 (internal-only).
 agent: abiba
 triggers:
  - on model add/remove
@@ -28,7 +27,6 @@ triggers:
 - benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
 - grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
- port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding

 ## Fleet Topology (Current — June 2026)

@@ -64,21 +62,21 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 128K ctx │  │ 128K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│          │  │             │  │ llama-srv │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
-│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
-│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
+│ 27B-code │  │ :8090       │  │ :8080     │  │ exporter     │
+│ :8090    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
 └──────────┘
 ```

-## Current Model Assignments (2026-07-08)
+## Current Model Assignments

-| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
-|-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+| Model | GPU | Host | VRAM | Status |
+|-------|-----|------|------|--------|
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |

 ## Operations

@@ -114,7 +112,7 @@ triggers:

 ### sync-keys
 1. List all agent keys in LiteLLM DB via `GET /key/list`
-2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
+2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
 3. Generate missing keys via `POST /key/generate` with unlimited budget
 4. Update agent configs — `/etc/environment` LITELLM_API_KEY
 5. Send Zulip DM to agents that can't be reached via SSH
@@ -122,30 +120,18 @@ triggers:
 7. Document keys in knowledge graph

 ### list
-Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys
-
-### health-check
-1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
-2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
-3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
-4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
-5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
-   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
-   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
-6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
-7. Check port conflicts: verify only one llama-server on :8080 per host
-8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)
+Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys

 ## Agent Keys (LiteLLM DB — Current 2026-06-30)

-| Agent | CT | IP | Key | Access |
-|-------|-----|-----|-----|--------|
-| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
-| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
-| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
-| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
-| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
-| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |
+| Agent | Key | Host | Access |
+|-------|-----|------|--------|
+| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
+| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
+| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
+| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
+| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
+| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |

 **Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
 If no SSH access, send Zulip DM via abiba-bot.
@@ -159,10 +145,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
+| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
+| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |

@@ -181,13 +167,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110).
- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput.
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`.
- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
+- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
+- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
+- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
 - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
 - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
@@ -200,8 +183,8 @@ If no SSH access, send Zulip DM via abiba-bot.

 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
-| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
+| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
+| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
 | Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |

 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
@@ -209,13 +192,3 @@ Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
 History stored at `/root/data/toks-history.json` with 7-day rolling window.

 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
-
-## Agent Config Implications (2026-07-08)
-
-With NVIDIA GPUs at 128K context:
- Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow
- Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule
- Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072`
- Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx)
- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
@@ -1,262 +0,0 @@
---
-kind: template
-name: hermes-agent-baseline
-version: 1.0.0
-description: >
-  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
-  configuration state, keys, workarounds, and audit procedure. When an agent's
-  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
-author: Abiba (pi agent)
---
-
-# Hermes Agent Baseline — Canonical Good State
-
-## Quick Restore
-
-```bash
-# Verify all agents against baseline in one command:
-for ct in 112 114 111 113; do
-    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
-done
-```
-
-## Agent Map
-
-| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
-|-------|-----|------|-----|-------------|---------------|----------|
-| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
-| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
-| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
-| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |
-
-Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
-
-## Key Architecture
-
-```
-Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
-                                                     └── Key DB (Postgres)
-```
-
- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
-
-## Config Pattern — Mandatory Fields
-
-### For Hermes Agents (Tanko, Mumuni, Baggy)
-
-Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
-
-### 1. Main Model
-```yaml
-model:
-  default: syslog-auto
-  provider: custom:harness       # or: harness
-  api_key_env: LITELLM_API_KEY
-  max_tokens: 4096
-```
-
-### 2. Custom Provider
-```yaml
-custom_providers:
-  - name: harness
-    model: syslog-auto
-    base_url: http://192.168.68.116/v1
-    api_key_env: LITELLM_API_KEY
-    api_mode: chat_completions
-```
-
-### 3. Vision (CRITICAL — must have api_key directly!)
-```yaml
-auxiliary:
-  vision:
-    provider: harness
-    model: gemma-4-12b              # or syslog-auto
-    base_url: http://192.168.68.116/v1
-    api_key_env: LITELLM_API_KEY
-    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
-    timeout: 60
-    download_timeout: 30
-```
-
-### 4. Compression (must have api_key!)
-```yaml
-  compression:
-    enabled: true
-    threshold: 0.65
-    target_ratio: 0.3
-    provider: harness
-    model: syslog-auto               # or gemma-4-12b
-    base_url: http://192.168.68.116/v1
-    api_key_env: LITELLM_API_KEY
-    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
-    timeout: 120
-```
-
-## Known Bug: `api_key_env` Ignored by Auxiliary Client
-
-**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
-
-**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
-resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
-to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
-to `"no-key-required"` → 401 from LiteLLM.
-
-**Impact**: Vision analysis, compression, and any other auxiliary task calling
-LiteLLM/harness will fail with:
-```
-401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
-```
-
-**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
-`api_key_env` in every auxiliary task config that uses the harness provider.
-
-**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
-`api_key` is empty:
-```python
-cfg_api_key = str(task_config.get("api_key", "")).strip() or None
-if not cfg_api_key:
-    key_env = str(task_config.get("api_key_env", "")).strip()
-    if key_env:
-        cfg_api_key = os.getenv(key_env, "").strip() or None
-```
-
-## Audit Procedure
-
-### Full Audit (all agents)
-```bash
-for ct in 112 114 111 113; do
-    echo "=== CT $ct ==="
-    pct-run $ct grep LITELLM_API_KEY /etc/environment
-    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
-    echo ""
-done
-```
-
-### Master Key Leak Check
-```bash
-# On every agent:
-pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
-# Must return empty
-```
-
-### Verify Key Works
-```bash
-curl -s http://192.168.68.116:80/v1/models \
-  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
-# Must return model list
-```
-
-### Verify Vision/Compression
-```bash
-# Check both api_key and api_key_env are present:
-pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
-# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
-```
-
-### For pi Agents (Tdunna)
-
-Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
-Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
-
-**models.json** — Must only list models authorized for the agent's LiteLLM key:
-```json
-{
-  "providers": {
-    "syslog-harness": {
-      "baseUrl": "http://192.168.68.116/v1",
-      "api": "openai-completions",
-      "apiKey": "sk-...",
-      "models": [
-        { "id": "syslog-auto" },
-        { "id": "ornith-1.0-35b" },
-        { "id": "qwen3.6-27B-code" },
-        { "id": "gemma-4-12b" }
-      ]
-    }
-  }
-}
-```
-
-**settings.json** — Always use `syslog-auto` as default:
-```json
-{
-  "defaultProvider": "syslog-harness",
-  "defaultModel": "syslog-auto"
-}
-```
-
-**Validation**: Verify models match LiteLLM's authorized list:
-```bash
-curl -s http://192.168.68.116:4000/v1/models \
-  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
-  | jq '.data[].id'
-```
-
-**Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
-a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
-stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.
-
-**Service**: `pm2 restart koby-zulip`, health at `:9201/health`.
-
-## Systemd Pattern
-
-For agents where the gateway runs as a user service:
-```ini
-# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
-[Service]
-Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
-```
-
-For agents where the gateway runs as root:
-```ini
-# /root/.config/systemd/user/hermes-gateway.service
-EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
-```
-
-**No drop-in that hardcodes the master key. Ever.**
-
-## Attachment Cache Locations
-
-| Type | Path |
-|------|------|
-| Images | `~/.hermes/cache/images/` |
-| Documents | `~/.hermes/cache/documents/` |
-| Audio | `~/.hermes/cache/audio/` |
-
-## Health Verification
-
-Run the consolidated health check:
-```bash
-python3 /root/scripts/agent-health-check.py
-```
-This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
-verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
-and counts recent errors. Non-disruptive — never restarts anything.
-
-## GPU Port Conflict Detection
-
-All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
- `.8` and `.110`: inline check in `llama-wrapper.sh`
- `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)
-
-Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
-If they differ → ghost detected → kill ghost → start fresh.
-
-## Related Contracts
-
- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
- `hermes-config-template.prose.md` — full configuration template
- `gpu-fleet.prose.md` — GPU fleet and agent key table
- `infrastructure-control.prose.md` — CT inventory with pct-run access
- `litellm-health.prose.md` — LiteLLM stack health verification
-
-## Change Log
-
-| Date | Change |
-|------|--------|
-| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
-| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
-| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
@@ -5,15 +5,14 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110),
-  parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added.
+  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
 ---

 ## Maintains

 - template_version: "2.1.0"
 - last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "tdunna", "baggy", "kagenz0"]
+- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
 - agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array

@@ -30,8 +29,8 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
 | Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
 | Abiba | `abiba-*` | 192.168.68.24 | local | — |
-| Tdunna | `tdunna-*` | ? | Zulip | — |
-| Baggy | `baggy-*` | ? | Zulip | — |
+| Koby | `koby-*` | ? | Zulip | — |
+| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |

 ✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
@@ -93,8 +92,7 @@ model:
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
-  context_length: 262144          # For syslog-auto (ornith route supports 256K).
-                                  # Set 131072 if using qwen3.6-27B-code or gemma-4-12b directly.
+  context_length: 262144          # Must match model's max context window

 fallback_providers:
  provider: deepseek
@@ -125,9 +123,8 @@ compression:
  enabled: true
  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
  provider: harness
-  max_context_window: 262144      # For syslog-auto (ornith supports 256K).
-                                  # Set 131072 if using qwen or gemma directly.
-  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
+  max_context_window: 262144      # Must match model's actual capacity
+  threshold: 0.65                 # Fires at ~170K for 262K window (not 0.25!)
  target_ratio: 0.30
  protect_last_n: 40
  hygiene_hard_message_limit: 350
@@ -239,28 +236,6 @@ The following MUST be identical across ALL profiles:
 - `max_context_window: 262144` MUST match the model's actual capacity
 - See `devops-hermes-compression` skill for full reference

-### Rule 9: Default Model Must Be `syslog-auto` (All Agents)
- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
- `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b
-  and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
-  - Model name typos that cause 403 errors and silent worker failures
-  - Single GPU downtime (routing falls back automatically)
-  - Key/model authorization mismatches
- **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
-  for specialized tasks, but MUST validate those models exist in the key's authorized list
-
-### Rule 10: Validate Model IDs Before Deployment (pi Agents)
- After configuring a pi agent's `models.json`, verify every model ID:
-  ```bash
-  curl -s http://192.168.68.116:4000/v1/models \
-    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
-  ```
- All model IDs in `models.json` MUST appear in the LiteLLM response
- The agent's API key may have a SUBSET of the full model catalog — check per-key
- A non-existent model ID causes 403 errors that silently break the pi RPC worker
-  (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
-
 ## Execution

 1. **Check current config** — Read the target agent's config.yaml
@@ -74,21 +74,12 @@ model:
 Run on any Hermes host to detect violations:

 ```bash
-# 1. Check config.yaml for hardcoded harness keys
-grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
+grep -rn 'api_key: sk-' /root/.hermes/ \
  --include='config.yaml' \
  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
-
-# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
-grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
-
-# 3. Verify running process env matches dedicated key
-cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
-  | tr '\0' '\n' | grep LITELLM_API_KEY
 ```

-If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
+If any output — violation. Fix immediately.

 ## Rotation Procedure

@@ -111,7 +102,7 @@ curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
 **Keys are permanent and use bare agent name aliases.**

 - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
 - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
 - **Max budget**: $100 per key (config default).

@@ -126,16 +117,16 @@ litellm_settings:
      purpose: "agent-inference"
 ```

-## Verified Agents (2026-07-05 update)
+## Verified Agents (2026-07-04 final)

-| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
-|-------|-----|-----|---------------|-----|--------|----------------|---------------|
-| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
-| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
-| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
-| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
-| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
-| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
+| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified |
+|-------|-----|--------|---------------|----------|----------------|---------------|
+| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT |
+| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT |
+| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT |
+| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT |
+| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT |
+| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT |

 ### Systemd Service Pattern (2026-07-04 fix)

@@ -189,37 +180,3 @@ auth → validate → lint → ai-review → gate
 - **Status check**: `PR Pipeline — Authorize → Validate → Review → Merge` required before merge
 - **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
 - **Config**: `.gitea/workflows/pr-pipeline.yaml`
-
-## Known Bug: auxiliary_client ignores api_key_env (2026-07-05)
-
-**Bug**: `_resolve_task_provider_model()` in `agent/auxiliary_client.py` reads
-`api_key` from auxiliary task configs (vision, compression, etc.) but does NOT
-resolve `api_key_env`. The custom provider resolution path handles `api_key_env`,
-but auxiliary tasks take a different code path that ignores it.
-
-**Impact**: Vision analysis and compression calls fall through to the `"no-key-required"`
-placeholder, causing 401 errors on LiteLLM/harness (which require `sk-*` keys).
-
-**Workaround**: Set `api_key` directly alongside `api_key_env` in each auxiliary
-task config:
-
-```yaml
-auxiliary:
-  vision:
-    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
-    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/v1
-    model: gemma-4-12b
-    provider: harness
-  compression:
-    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
-    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/v1
-    model: gemma-4-12b
-    provider: harness
-```
-
-**Affected agents**: All Hermes agents with harness/LiteLLM provider and
-`api_key_env` in auxiliary configs (all 4 Hermes agents patched 2026-07-05).
-
-**Source location**: `agent/auxiliary_client.py` line 5478
@@ -0,0 +1,188 @@
+---
+kind: function
+name: hermes-zulip-restore
+description: >
+  Restores Zulip connectivity for any Hermes agent (Mumuni CT114, Tanko CT112,
+  Koby CT111). Deploys the zulip-platform adapter to the correct bundled plugin
+  path, verifies env credentials, restarts the gateway, and confirms Zulip
+  connects. Run this whenever a Hermes agent stops responding on Zulip or after
+  a fresh agent deployment.
+agent: abiba
+version: 1.0.0
+status: active
+runtime_contract: 2
+---
+
+# Hermes Zulip Restore — Bring Any Agent Back to Good State
+
+Single-shot function that restores full Zulip connectivity for a Hermes agent.
+Covers adapter deployment, HTML stripping (slash command fix), env verification,
+gateway restart, and connection validation.
+
+## Parameters
+
+| Param | Type | Required | Default | Description |
+|-------|------|----------|---------|-------------|
+| `target` | string | yes | — | Agent name: `mumuni`, `tanko`, or `koby` |
+
+## Maintains
+
+- adapter_deployed: bool — Whether `_strip_html` adapter is at correct bundled path
+- zulip_connected: bool — Whether gateway_state shows zulip.state = "connected"
+- env_valid: bool — Whether .env has ZULIP_SITE, ZULIP_EMAIL, ZULIP_API_KEY
+- gateway_running: bool — Whether gateway process is running
+
+### Postconditions
+
+- `_strip_html` function present in `<hermes-agent>/plugins/platforms/zulip/adapter.py`
+- All three adapter files (__init__.py, adapter.py, plugin.yaml) present at bundled path
+- Zulip env vars set in `~/.hermes/.env` (or `/home/jerome/.hermes/.env` for Tanko)
+- Gateway restarted and zulip platform reports state `connected`
+- HTML stripping enabled for `/approve` and `/deny` slash command support
+
+## Requires
+
+- SSH access to target host (direct or via amdpve for CTs)
+- Git repo at `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git`
+- Python 3 with `httpx` installed on target
+- Zulip server accessible at `https://chat.sysloggh.net`
+
+## Live-State Fields
+
+| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
+|------|-----|---------|-------------|-------------|------|
+| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
+| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
+| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
+
+| Field | Value | Trust |
+|-------|-------|-------|
+| Zulip server | https://chat.sysloggh.net | ✅ Verified |
+| Git repo (zulip-platform) | `https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git` | ✅ Verified |
+| Bundled adapter path | `<HERMES_HOME>/hermes-agent/plugins/platforms/zulip/` | ✅ Verified |
+| Git branch | `feat/zulip-streaming` | ✅ Verified (contains _strip_html fix) |
+
+## Execution
+
+### Step 1: Locate Target
+
+Map `target` to connectivity parameters from the live-state table above.
+For CT112 and CT111, route through `ssh root@amdpve` then `pct exec <id>`.
+
+### Step 2: Deploy Zulip Adapter
+
+On the target host:
+
+```bash
+# Clone or update the plugin repo
+mkdir -p /tmp/zulip-deploy
+cd /tmp/zulip-deploy
+if [ -d zulip-platform-plugins ]; then
+  cd zulip-platform-plugins && git pull origin feat/zulip-streaming
+else
+  git clone --branch feat/zulip-streaming \
+    https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins.git
+fi
+
+# Ensure bundled plugin directory exists
+mkdir -p <HERMES_HOME>/hermes-agent/plugins/platforms/zulip
+
+# Copy adapter files
+cp zulip-platform-plugins/plugins/platforms/zulip/adapter.py \
+   zulip-platform-plugins/plugins/platforms/zulip/__init__.py \
+   zulip-platform-plugins/plugins/platforms/zulip/plugin.yaml \
+   <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/
+
+# Fix ownership (Tanko only)
+chown -R jerome:jerome <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/  # Tanko only
+
+# Clean up
+rm -rf /tmp/zulip-deploy
+```
+
+### Step 3: Verify _strip_html is Present
+
+```bash
+grep -q "_strip_html" <HERMES_HOME>/hermes-agent/plugins/platforms/zulip/adapter.py
+```
+Expected: exit code 0. If not found → adapter is stale, re-run Step 2 with fresh clone.
+
+### Step 4: Verify Env Credentials
+
+```bash
+grep -E "ZULIP_SITE|ZULIP_EMAIL|ZULIP_API_KEY" <HERMES_HOME>/.env
+```
+
+Expected: all three variables set with non-empty values. If any missing:
+- ZULIP_SITE: `https://chat.sysloggh.net`
+- ZULIP_EMAIL: `<agent>-bot@chat.sysloggh.net`
+- ZULIP_API_KEY: obtain from Zulip admin panel (Bots → show API key)
+
+### Step 5: Restart Gateway
+
+```bash
+cd <HERMES_HOME>/hermes-agent
+# Use venv if available
+python3 -m hermes_cli.main gateway restart  # or: venv/bin/python -m hermes_cli.main gateway restart
+```
+
+Wait for the restart to complete (up to 45s). Check:
+
+```bash
+grep "Gateway running" <HERMES_HOME>/logs/gateway.log | tail -1
+```
+
+Expected: "Gateway running with N platform(s)" where N > 1 (includes zulip).
+
+### Step 6: Validate Zulip Connection
+
+```bash
+python3 -c "
+import json
+d = json.load(open('$HERMES_HOME/gateway_state.json'))
+print('zulip:', d.get('platforms', {}).get('zulip', {}).get('state', 'NOT FOUND'))
+"
+```
+
+Expected: `zulip: connected`. If not connected, check gateway log:
+
+```bash
+grep -E "zulip|Zulip|ZULIP" <HERMES_HOME>/logs/gateway.log | tail -10
+```
+
+### Step 7: Report
+
+Compile results: `{ adapter_deployed, zulip_connected, env_valid, gateway_running }`.
+
+| State | Action |
+|-------|--------|
+| All true | ✅ Agent restored — relay success to user |
+| `adapter_deployed: false` | Re-run Step 2 |
+| `env_valid: false` | Prompt for missing credentials |
+| `zulip_connected: false` | Check Zulip server reachability, verify API key |
+| `gateway_running: false` | Check process logs for crash cause |
+
+## Known Failure Modes
+
+| Symptom | Root Cause | Recovery |
+|---------|-----------|----------|
+| Gateway running with 1 platform(s) | Adapter at wrong path (user plugins vs bundled) | Deploy to `<hermes-agent>/plugins/platforms/zulip/` not `~/.hermes/plugins/` |
+| Queue expired / BAD_EVENT_QUEUE_ID | Idle for 10+ minutes → normal | Auto-reconnects — no action needed |
+| No events received for N seconds | No DMs or @mentions sent to this bot | Normal if nobody messaged the agent |
+| `httpx` not found | Missing dependency | `pip install httpx` in the Hermes venv or system Python |
+| Slash commands not matching | Missing `_strip_html` — Zulip sends `<p>/approve</p>` | Verify `_strip_html` in adapter (Step 3) |
+| Permission denied on gateway restart | Running as wrong user | Use `su - jerome` for Tanko; root for others |
+
+## Git Branch Reference
+
+The `_strip_html` fix lives on `feat/zulip-streaming` branch:
+```
+https://git.sysloggh.net/SyslogSolution/zulip-platform-plugins/src/branch/feat/zulip-streaming
+```
+
+Commit `55ca15d` — `fix(zulip): add _strip_html for slash command matching`
+Pull request #33 is the primary integration branch.
+
+---
+
+**Last verified good state**: 2026-07-08 — Mumuni, Tanko, Koby all connected with `_strip_html` applied.
@@ -68,7 +68,7 @@ description: >
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
 | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
-| Baggy CT (113) | SSH jerome | ❌ no key access |
+| Koonimo CT (.114) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
 | Gitea | API token | abiba-bot token | ✅ |
 | Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
@@ -85,7 +85,7 @@ description: >

 ### Reachability Matrix

-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Koonimo (.114) |
 |-----------|---------|----------------|--------|-------------|---------------|----------------|
 | **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
 | **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
@@ -585,7 +585,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 110 | gitea | minipve | — | Git | ❌ |
 | 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
+| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
 | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | — | ? | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
@@ -602,59 +602,3 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
 | CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |
-
-## CT Access (pct-run — no IPs needed)
-
-All CTs are accessible via `pct-run <CT_ID> <command>`. No IP addresses required.
-Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.sh`.
-
-| CT | Name | Node | pct-run |
-|-----|------|------|---------|
-| 100 | abiba | amdpve | `pct-run 100` |
-| 105 | kagentz | amdpve | `pct-run 105` |
-| 111 | tdunna | amdpve | `pct-run 111` |
-| 112 | tanko | amdpve | `pct-run 112` |
-| 113 | baggy | amdpve | `pct-run 113` |
-| 115 | scottdenya | amdpve | `pct-run 115` |
-| 104 | authentik | minipve | `pct-run 104` |
-| 110 | gitea | minipve | `pct-run 110` |
-| 114 | mumuni | minipve | `pct-run 114` |
-| 116 | syslog-api | minipve | `pct-run 116` |
-| 106 | ra-h-os | storepve | `pct-run 106` |
-| 107 | proxmox-backup | storepve | `pct-run 107` |
-| 108 | media | storepve | `pct-run 108` |
-| 117 | zulip | storepve | `pct-run 117` |
-| 102 | adguard | acerpve | `pct-run 102` |
-
-GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
-```bash
-ssh root@192.168.68.8   # RTX 3090
-ssh root@192.168.68.110 # RTX 5070
-ssh root@192.168.68.15  # Strix Halo
-```
-
-## Section 7: Agent Health Check (consolidated — 2026-07-05)
-
-Replaces 7 scattered Zulip health scripts with a single non-disruptive check
-running every 10 minutes via cron (`/root/scripts/agent-health-check.py`).
-
-The script is **read-only** — it never restarts, kills, or modifies anything.
-Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
-which was kill+nohup outside systemd) are banned by policy.
-
-### Checks Performed
-
-| Check | Frequency | What It Detects |
-|-------|-----------|-----------------|
-| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models |
-| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
-| Gateway liveness | 10 min | Gateway process running, state file readable |
-| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
-| Recent errors | 10 min | Error count in journald for last 10 min |
-
-### Disabled Scripts
-
-| Script | Why Disabled |
-|--------|-------------|
-| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
-| `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |
@@ -3,7 +3,7 @@ kind: responsibility
 name: infrastructure-update
 description: >
  Autonomous system-wide update contract covering all 5 Proxmox nodes,
-  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
+  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
 agent: abiba
@@ -11,7 +11,7 @@ triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
-version: 1.1.0
+version: 1.0.0
 ---

 ## Maintains
@@ -24,11 +24,11 @@ version: 1.1.0

 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
-3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
+2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
+3. ✅ Docker healthy on .7, .116
 4. ✅ LiteLLM health check passing
 5. ✅ Zulip server reachable
-6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
+6. ✅ GPU fleet healthy (all 3 GPUs)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

@@ -38,7 +38,7 @@ Before ANY update wave:
 |--------|------|---------|---------|
 | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
-| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
+| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
 | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
 | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |

@@ -55,16 +55,16 @@ Before ANY update wave:
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
-| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
-| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
-| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
+| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
+| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |

 **Verify after Wave 2:**
- All VMs/CTs running: check via Proxmox API
- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- All CTs running: check via Proxmox API
+- LiteLLM healthy: `curl :4001/health/liveliness`
+- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`

@@ -72,20 +72,20 @@ Before ANY update wave:

 | Host | Stack | Command |
 |------|-------|---------|
-| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
-| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
-| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
 | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |

 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
+- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
 - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl localhost:3001/` (via CT 116)
- Firecrawl test: `curl :3002/`
- SearXNG test: `curl :8888`
+- Dashboard loading: `curl :3001/`
+- Firecrawl test: `curl :3002/health`
+- SearXNG test: `curl storepve:8888`

 ## Wave 4: Proxmox Kernel Reboot (if needed)

@@ -111,20 +111,16 @@ If ANY verification fails:

 Before Wave 1, snapshot these files:
 ```
-/opt/inference-harness/docker-compose.yml               (CT 116 .116)
-/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
-/opt/monitoring/prometheus.yml                          (CT 116 .116)
-/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
-/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
-/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
-/opt/home_stack/docker-compose.yml                      (VM 109 .7)
-/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
-/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
-/etc/systemd/system/ornith-server.service               (amdpve .15)
-/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
+/opt/inference-harness/litellm_config.yaml        (.116)
+/opt/inference-harness/docker-compose.yml          (.116)
+/etc/nginx/nginx.conf                              (.116 container)
+/opt/search-stack/*/docker-compose.yaml            (.7)
+/root/.pi/agent/extensions/config.yaml             (.24)
+/etc/systemd/system/ornith-server.service          (.15)
+/etc/systemd/system/llama-server.service           (.8, .110)
 ```

-Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`

 ## Security-Specific Updates

@@ -138,8 +134,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`
 ## Success Criteria

 - [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
+- [ ] All CTs running post-update
+- [ ] All 22 Docker containers healthy
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
@@ -152,7 +148,7 @@ After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD

-Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
+Updated: 5 PVE nodes, 12 CTs, 22 containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>
@@ -1,7 +1,7 @@
 ---
 name: memory-audit-maintenance
 kind: responsibility
-description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---

@@ -15,13 +15,13 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,

 ### Scope

-This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.

 **Agent Roster:**
 - Mumuni
 - Tanko
- Tdunna
- Baggy
+- Koby
+- Koonimo

 **Isolation Principle:** Each agent has its own:
 - `MEMORY.md` and `USER.md`
@@ -343,4 +343,4 @@ return {

 ### Per-Agent Notes

-Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
+Each Hermes agent (Mumuni, Tanko, Koby, Koonimo) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
@@ -1,228 +0,0 @@
-#!/usr/bin/env python3
-"""
-/root/scripts/agent-health-check.py — Consolidated Agent Health Verification
-
-Single non-disruptive health check replacing 7 scattered scripts.
-Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming,
-gateway liveness, and gateway log health. NEVER restarts anything.
-
-Usage:
-  python3 /root/scripts/agent-health-check.py          # Full check
-  python3 /root/scripts/agent-health-check.py --json   # Machine-readable
-  python3 /root/scripts/agent-health-check.py --quiet  # Only output on failure
-
-Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
-"""
-
-import subprocess, json, sys, os, time
-from datetime import datetime
-
-LITELLM = "http://192.168.68.116:80"
-
-AGENTS = {
-    "tanko":   {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"},
-    "mumuni":  {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"},
-    "tdunna":  {"ct": 111, "host": None,              "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None},
-    "baggy":   {"ct": 113, "host": None,              "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None},
-}
-
-GPU_HOSTS = {
-    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
-    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
-    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "ornith-server"},
-}
-
-FAIL = []
-
-def ssh(host, cmd, user="root"):
-    """Execute a command on a remote host, return stdout or None."""
-    try:
-        result = subprocess.run(
-            ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
-             f"{user}@{host}", cmd],
-            capture_output=True, text=True, timeout=15
-        )
-        return result.stdout.strip() if result.returncode == 0 else None
-    except:
-        return None
-
-def http_get(url, headers=None, timeout=5):
-    """Return HTTP status code as string."""
-    try:
-        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout), "-o", "/dev/null", "-w", "%{http_code}"]
-        if headers:
-            for k, v in headers.items():
-                cmd.extend(["-H", f"{k}: {v}"])
-        cmd.append(url)
-        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
-        return result.stdout.strip() or "000"
-    except:
-        return "timeout"
-
-def http_json(url, headers=None, timeout=5):
-    """Return parsed JSON from URL, or None."""
-    try:
-        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout)]
-        if headers:
-            for k, v in headers.items():
-                cmd.extend(["-H", f"{k}: {v}"])
-        cmd.append(url)
-        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
-        return json.loads(result.stdout) if result.returncode == 0 and result.stdout else None
-    except:
-        return None
-
-
-# ═══════════════════════════════════════════════════════════════════
-# CHECK 1: LiteLLM Key Validation
-# ═══════════════════════════════════════════════════════════════════
-
-def check_keys():
-    for name, agent in AGENTS.items():
-        data = http_json(f"{LITELLM}/v1/models",
-                         headers={"Authorization": f"Bearer {agent['key']}"})
-        if data and data.get("data"):
-            model = data["data"][0].get("id", "?")
-            print(f"  ✅ {name}: key valid → {model}")
-        else:
-            print(f"  ❌ {name}: KEY FAILURE — auth rejected or unreachable")
-            FAIL.append(f"key:{name}")
-
-
-# ═══════════════════════════════════════════════════════════════════
-# CHECK 2: GPU Port Conflict Detection
-# ═══════════════════════════════════════════════════════════════════
-
-def check_gpu_ports():
-    for label, gpu in GPU_HOSTS.items():
-        host = gpu["host"]
-        port = gpu["port"]
-        svc = gpu["service"]
-
-        svc_status = ssh(host, f"systemctl is-active {svc}")
-        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
-
-        if not svc_status:
-            print(f"  ❌ {label}: UNREACHABLE")
-            FAIL.append(f"gpu-unreachable:{host}")
-            continue
-
-        if not port_owner:
-            print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
-            FAIL.append(f"gpu-no-port:{label}")
-        elif svc_status != "active":
-            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
-            if svc_pid and port_owner != svc_pid:
-                print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
-                FAIL.append(f"gpu-ghost:{label}:{port_owner}")
-            else:
-                print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
-        else:
-            # Verify health endpoint
-            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
-            if health and '"status":"ok"' in health:
-                print(f"  ✅ {label}: healthy (pid={port_owner})")
-            elif health and '"status":"no slot available"' in health:
-                print(f"  ✅ {label}: healthy (loading, pid={port_owner})")
-            elif health and '"error"' in health.lower():
-                print(f"  ⚠️  {label}: error response (pid={port_owner}): {health[:80]}")
-            else:
-                print(f"  ⚠️  {label}: unknown health (pid={port_owner}): {str(health)[:80]}")
-
-
-# ═══════════════════════════════════════════════════════════════════
-# CHECK 3: Agent Gateway Liveness + Streaming
-# ═══════════════════════════════════════════════════════════════════
-
-def check_agents():
-    for name, agent in AGENTS.items():
-        host = agent.get("host")
-        user = agent.get("user")
-        ct = agent["ct"]
-
-        if not host or not user:
-            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
-            continue
-
-        # Gateway process
-        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
-        if not pid:
-            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
-            FAIL.append(f"gateway-down:{name}")
-            continue
-
-        # Gateway state file
-        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
-        if state:
-            try:
-                st = json.loads(state)
-                gw_state = st.get("gateway_state", "?")
-                zulip = st.get("platforms", {}).get("zulip", {}).get("state", "?")
-            except:
-                gw_state, zulip = "corrupt", "?"
-        else:
-            gw_state, zulip = "no-state-file", "?"
-
-        # Zulip streaming: does adapter have edit_message?
-        adapter_paths = [
-            "~/.hermes/plugins/zulip-platform/adapter.py",
-            "~/.hermes/plugins/platforms/zulip/adapter.py",
-        ]
-        streaming = "no"
-        for p in adapter_paths:
-            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
-            if has_edit and has_edit != "0":
-                streaming = "yes"
-                break
-
-        # Recent errors
-        recent_errors = ssh(host,
-            r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
-            r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
-            user=user)
-        recent_errors = (recent_errors or "0").strip().split("\n")[-1]  # take last line
-
-        print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
-              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
-              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
-
-
-# ═══════════════════════════════════════════════════════════════════
-# MAIN
-# ═══════════════════════════════════════════════════════════════════
-
-def main():
-    quiet = "--quiet" in sys.argv
-    as_json = "--json" in sys.argv
-
-    if not quiet:
-        print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
-        print()
-
-    print("🔑 LiteLLM Keys:")
-    check_keys()
-    print()
-
-    print("🎮 GPU Port Health:")
-    check_gpu_ports()
-    print()
-
-    print("🤖 Agent Gateways:")
-    check_agents()
-
-    if FAIL:
-        print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
-        if quiet:
-            # In quiet mode, only print failures as a single alert line
-            print(f"ALERT agent-health:{','.join(FAIL)}")
-    elif not quiet:
-        print("\n✅ All checks passed")
-
-    if as_json:
-        print(json.dumps({"timestamp": datetime.now().isoformat(),
-                          "failures": FAIL, "healthy": len(FAIL) == 0}))
-
-    sys.exit(1 if FAIL else 0)
-
-if __name__ == "__main__":
-    main()
@@ -1,96 +0,0 @@
-#!/usr/bin/env bash
-# pct-run — Run commands inside Proxmox CTs by CT ID only. No IPs needed.
-# Usage: pct-run <CT_ID> [command...]
-# If no command given, opens a shell.
-#
-# Source of truth: Proxmox pct config on each node.
-# No hardcoded IPs. No prose contract drift.
-
-set -euo pipefail
-
-# ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
-declare -A CT_NODES=(
-    # amdpve (192.168.68.15)
-    [100]=amdpve    # abiba
-    [105]=amdpve    # kagentz
-    [111]=amdpve    # tdunna
-    [112]=amdpve    # tanko
-    [113]=amdpve    # baggy
-    [115]=amdpve    # scottdenya
-    # minipve (192.168.68.12)
-    [104]=minipve   # authentik
-    [110]=minipve   # gitea
-    [114]=minipve   # mumuni
-    [116]=minipve   # syslog-api
-    # storepve (192.168.68.6)
-    [106]=storepve  # ra-h-os
-    [107]=storepve  # proxmox-backup
-    [108]=storepve  # media
-    [117]=storepve  # zulip
-    # acerpve (192.168.68.9)
-    [102]=acerpve   # adguard
-    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
-)
-
-# Each node must be root-accessible via SSH hostname
-# (Ensure ~/.ssh/config or /etc/hosts has these resolvable)
-
-# ── PVE node → IP (needed only because SSH needs an address) ──
-declare -A NODE_IPS=(
-    [amdpve]=192.168.68.15
-    [minipve]=192.168.68.12
-    [storepve]=192.168.68.6
-    [acerpve]=192.168.68.9
-    [ocupve]=192.168.68.5
-)
-
-resolve_node() {
-    local ct_id="$1"
-    local node="${CT_NODES[$ct_id]:-}"
-    if [[ -z "$node" ]]; then
-        echo "ERROR: CT $ct_id not found in mapping" >&2
-        exit 1
-    fi
-    echo "$node"
-}
-
-node_ip() {
-    local node="$1"
-    local ip="${NODE_IPS[$node]:-}"
-    if [[ -z "$ip" ]]; then
-        echo "ERROR: Node $node not found in IP mapping" >&2
-        exit 1
-    fi
-    echo "$ip"
-}
-
-main() {
-    if [[ $# -lt 1 ]]; then
-        echo "Usage: pct-run <CT_ID> [command...]" >&2
-        echo "  pct-run 112 cat /etc/hostname" >&2
-        echo "  pct-run 114 systemctl status hermes-gateway" >&2
-        echo ""
-        echo "Known CTs:" >&2
-        for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
-            echo "  CT $ct → ${CT_NODES[$ct]}" >&2
-        done
-        exit 1
-    fi
-
-    local ct_id="$1"
-    shift
-    local node
-    node=$(resolve_node "$ct_id")
-    local ip
-    ip=$(node_ip "$node")
-
-    if [[ $# -eq 0 ]]; then
-        # Interactive shell
-        exec ssh -t "root@${ip}" "pct enter ${ct_id}"
-    else
-        # One-shot command
-        ssh "root@${ip}" "pct exec ${ct_id} -- $*"
-    fi
-}
-
-main "$@"
@@ -73,25 +73,6 @@ description: >
 - **Fix**: If edit fails, send the response as a new message instead
 - **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)

-### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
- **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
-  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
-  `messages_processed` stalls.
- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
-  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
-  configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session
-  workers emit 403 on first prompt, then never recover because the error doesn't trigger
-  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
-  `workers=[<id>:busy:N]` with growing N in extension logs.
- **Fix**: (1) Update `models.json` to only include models from the authorized list.
-  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
-  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
-  routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should
-  only be used when explicitly requested. Validate model IDs at agent setup time.
- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
-
 ## Deployment Checklist

 When deploying a new Zulip adapter, verify:
@@ -105,4 +86,3 @@ When deploying a new Zulip adapter, verify:
 - [ ] `@all-bots` detected via configurable user_id
 - [ ] Poll uses long-poll (not `dont_block=true` polling)
 - [ ] Stuck/idle detection accounts for quiet periods
- [ ] Model IDs in config validated against `GET /v1/models` with actual API key
@@ -77,24 +77,6 @@ Log the condition as "pending restart" with the timestamp. If the condition pers
 Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.

 ### Severity escalation
-
-## Streaming Support (2026-07-05)
-
-Zulip agents now support progressive message editing during agent generation.
-When a Hermes agent (Tanko, Mumuni) processes a message, the response is
-streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
-
- Adapter implements `edit_message()` using `_api_patch()` helper
- Gateway stream consumer progressively edits the Zulip message
- User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
-
-### Verification
-```bash
-# Check if agent has streaming:
-grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
-# Must return 1 — streaming is active
-```
 - Single agent warning → log only
 - Single agent critical → relay message to user
 - Two or more agents critical → immediate relay + attempt auto-recovery