merge: resolve PR #26 conflict — keep qwen model ref + add hwepve node

- proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement
- zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve
- hermes-agent-baseline: fix Mumuni node minipve→hwepve
- hermes-zulip-restore: add hwepve Proxmox column for Mumuni
- mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes
- pm2-self-heal: verified — no Mumuni references, no changes needed

Post-migration discoveries:
- Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4)
- IP preserved (192.168.68.123), hostname 'mumuni' unchanged
- Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by
  container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip
  queue (errors=0, reconnects=0)

hermes-agent-baseline.prose.md

@@ -25,7 +25,7 @@ done
 | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
 |-------|-----|------|-----|---------------|------------|----------|
 | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
-| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
+| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
 | Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
 | Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
 | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |

hermes-key-enforcement.prose.md

@@ -61,31 +61,21 @@ base_url: http://192.168.68.116/litellm/v1/responses
 
 This applies to ALL sections using the harness provider: `custom_providers`, `delegation`, `auxiliary.*`.
 
-## DeepSeek Harness Exemption
+## Exemptions
 
-**External providers are explicitly exempt** and may use hardcoded keys:
-- DeepSeek (`api.deepseek.com`) — **HARNESS EXEMPTION**
+External providers are **explicitly exempt** and may use hardcoded keys:
+- DeepSeek (`api.deepseek.com`)
 - OpenAI (`api.openai.com`)
 - Anthropic (`api.anthropic.com`)
 - OpenRouter
 - Any provider whose base_url does NOT match `192.168.68.116` or `litellm.sysloggh.net`
 
-### Example: DeepSeek Hardcoded Key
-
-```yaml
-fallback_providers:
-  - provider: deepseek
-    base_url: https://api.deepseek.com
-    api_key: sk-b7d9...               # ← HARDCODED OK (external)
-    api_key_env: DEEPSEEK_API_KEY    # ← also OK if set in environment
-```
-
 ## Standard Pattern
 
 > **Canonical vault process (2026-07-16):** see `litellm-api-keys` § Production Vault Access Process.
 > All agents MUST use the `infisical-gateway.sh` wrapper (live vault injection). Hardcoded systemd
 > drop-ins / config.yaml keys are DEPRECATED — they rot on rotation (root cause of the 2026-07-16 401 storm).
-> Fleet-wide standardization completed 2026-07-17. All 4 Hermes agents migrated.
+> 4/5 agents migrated; tanko (user jerome) pending.
 
 ```yaml
 # ✅ CORRECT — all harness/litellm providers (authenticated path, NO /responses suffix)
@@ -106,12 +96,12 @@ auxiliary:
     base_url: http://192.168.68.116/litellm/v1           # ← NO /responses suffix!
     api_key_env: LITELLM_API_KEY
 
-# ✅ CORRECT — DeepSeek harness exemption (external provider)
+# ✅ ALSO CORRECT — external providers
 fallback_providers:
   - provider: deepseek
     base_url: https://api.deepseek.com
-    api_key: sk-b7d9...               # ← HARDCODED OK (external provider)
-    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment
+    api_key: sk-b7d9...               # ← hardcoded OK (external)
+    api_key_env: DEEPSEEK_API_KEY      # ← also OK if set in environment (vault or /etc/environment)
 ```
 
 ```yaml
@@ -194,31 +184,16 @@ litellm_settings:
       purpose: "agent-inference"
 ```
 
-## Verified Agents
+## Verified Agents (2026-07-05 update)
 
-All 4 Hermes agents (Mumuni, Tanko, Koby, Koonimo) are now verified and standardized on the canonical pattern as of 2026-07-17.
-
-| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | .env Fallback |
+| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
 |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
-| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Verified | `infisical run` | ✅ |
-| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Verified | `infisical run` | ✅ |
-| Koby | 129 | .129 | `koby` | Infisical vault | ✅ Verified | `infisical run` | ✅ |
-| Koonimo | 114 | .114 | `koonimo` | Infisical vault | ✅ Verified | `infisical run` | ✅ |
-| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | ✅ N/A (pi agent) | — | ✅ |
-| Kagenz0 | 105 | ? | `kagenz0` | Infisical vault | ❌ DOWN | — | — |
-
-> **Note**: CT hostnames differ from agent identities. CT111=tdunna runs koby; CT113/114=baggy runs koonimo.
-
-### Migration Status: Authenticated Path
-
-All agents have migrated to the authenticated `/litellm/v1/responses` path:
-
-| Agent | `/litellm/v1/responses` | Deprecated `/v1` | Status |
-|-------|--------------------------|--------------------|--------|
-| Mumuni | ✅ 5 sections | 0 | ✅ Authenticated |
-| Tanko | ✅ Verified | 0 | ✅ Authenticated |
-| Koby | ✅ Verified | 0 | ✅ Authenticated |
-| Koonimo | ✅ Verified | 0 | ✅ Authenticated |
+| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
+| Mumuni | 114 | .123 | `mumuni` | Infisical vault | ✅ Fixed | `infisical run` | 01:46 EDT Jul 10 |
+| Koby | 111 | ? | `koby` | Infisical vault | ✅ Fixed | `infisical run` | 23:30 UTC Jul 5 |
+| Koonimo | 113 | ? | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-07-11 |
+| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |
 
 > **Note**: CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
 > LiteLLM key aliases use agent identity, not CT hostname.
@@ -270,26 +245,6 @@ EnvironmentFile=/etc/environment
 6. **Update** — bump the verified table above
 7. **Use safe-mutate** — if the fix requires updating vault secrets or restarting the gateway on a remote host, use `safe-mutate` to verify current state before mutating.
 
-## PR #46 Verification
-
-**PR #46** (Hermes Key Enforcement) has been implemented and verified. The PR established:
-
-1. **Standardized API key configuration** across all Hermes agents
-2. **Infisical vault** as the single source of truth (project=agents, env=production)
-3. **Runtime key injection** via `infisical run --` wrapper
-4. **DeepSeek harness exemption** for external providers
-5. **Detection queries** for compliance checking
-6. **CI pipeline integration** for automated validation
-
-### Verification Status
-
-- ✅ All 4 Hermes agents (Mumuni, Tanko, Koby, Koonimo) verified
-- ✅ No hardcoded harness keys in configs
-- ✅ All agents using `api_key_env: LITELLM_API_KEY`
-- ✅ DeepSeek harness exemption properly documented
-- ✅ Detection queries validated
-- ✅ CI pipeline in place
-
 ## Related Contracts
 
 - `hermes-config-template.prose.md` — full configuration template
@@ -311,7 +266,7 @@ auth → validate → lint → ai-review → gate
 - **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
 - **Config**: `.gitea/workflows/pr-pipeline.yaml`
 
-## Known Bug: auxiliary_client ignores api_key_env
+## Known Bug: auxiliary_client ignores api_key_env (2026-07-05)
 
 **Bug**: `_resolve_task_provider_model()` in `agent/auxiliary_client.py` reads
 `api_key` from auxiliary task configs (vision, compression, etc.) but does NOT
@@ -327,29 +282,19 @@ task config:
 ```yaml
 auxiliary:
   vision:
-    api_key: sk-<agent-key-from-vault>           # ← HARDCODED WORKAROUND
+    api_key: sk-<agent-key-from-vault>           # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
     api_key_env: LITELLM_API_KEY
     base_url: http://192.168.68.116/v1
     model: gemma-4-12b
     provider: harness
   compression:
-    api_key: sk-<agent-key-from-vault>           # ← HARDCODED WORKAROUND
+    api_key: sk-<agent-key-from-vault>           # ← workaround (same as above)
     api_key_env: LITELLM_API_KEY
     base_url: http://192.168.68.116/v1
     model: gemma-4-12b
     provider: harness
 ```
 
-**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
-`api_key` is empty:
-```python
-cfg_api_key = str(task_config.get("api_key", "")).strip() or None
-if not cfg_api_key:
-    key_env = str(task_config.get("api_key_env", "")).strip()
-    if key_env:
-        cfg_api_key = os.getenv(key_env, "").strip() or None
-```
-
 **Affected agents**: All Hermes agents with harness/LiteLLM provider and
 `api_key_env` in auxiliary configs (all 4 Hermes agents patched 2026-07-05).
 

hermes-zulip-restore.prose.md

@@ -51,7 +51,7 @@ gateway restart, and connection validation.
 
 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
+| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |

mumuni-delegation-prose-contract.prose.md

@@ -6,7 +6,7 @@ description: >
   delegation, verification, and delivery. Defines when to delegate, which
   worker to use for what, how to handle failures, and the kanban board
   protocol. Enforces context-window discipline and separation of concerns.
-  Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
+  Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
 version: 1.0.0
 ---
 
@@ -19,8 +19,8 @@ version: 1.0.0
 
 ## Topology
 
-**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
-**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
+**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
+**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
 
 This contract is infrastructure-agnostic in terms of which nodes are used.

proxmox-monitor.prose.md

@@ -5,7 +5,7 @@ description: >
   Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
   on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
   exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
-  instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
+  instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
   (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
   the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
 agent: abiba
@@ -33,8 +33,8 @@ agent: abiba
 
 | Exporter | Host:Port | Scope | Notes |
 |----------|-----------|-------|-------|
-| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
-| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
+| prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
+| node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
 | docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
 
 ## PVE API Token
@@ -48,7 +48,7 @@ agent: abiba
 
 | UID | Title | Panels | Source |
 |-----|-------|--------|--------|
-| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
+| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
 | proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
 | docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
 | gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
@@ -77,9 +77,9 @@ agent: abiba
 | `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
 | `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
 | `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
-| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
+| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
 
-## Cluster "Tabiri" — 5 Nodes
+## Cluster "Tabiri" — 6 Nodes
 
 | Node | IP | Role |
 |------|----|----|
@@ -88,6 +88,7 @@ agent: abiba
 | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
 | minipve | 192.168.68.12 | PVE |
 | amdpve | 192.168.68.15 | PVE + Strix Halo LLM (qwen3.6-35B-udq4, strix-moe) |
+| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
 
 ## Operations
 

zulip-health.prose.md

@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
 ## Requires
 
 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
-- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -199,7 +199,7 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
 ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
 ```
 
-Gateway PID should exist with uptime > 60s.
+Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
 
 **B3: Heartbeat Verification**