fix: correct Mumuni gateway lifecycle path (no sudo on kagentz) + 2 leftover CT100 mentions

Post-merge verification of PR #54 found: 1. Lifecycle commands in zulip-health/zulip-self-heal used 'sudo systemctl' for hermes-gateway - sudo is NOT installed on kagentz (no sudoers, no polkit user rules). Correct path: root SSH invocation (root@192.168.68.14), matching how the Proxmox host actually manages the unit. 2. hermes-zulip-restore.prose.md line 5 + zulip-resilience-v3 line 423 still said 'Mumuni CT100' in prose - repointed. Found via independent review + live runtime check (whoami, which sudo, journalctl, systemctl show). Refs PR #54, relay #738/#739.
Merge pull request 'fix: repoint Mumuni refs from CT100/.24 to kagentz CT105/.14 post-migration' (#54 ) from fix/mumuni-kagentz-repoint into master
2026-08-30 01:06:30 +00:00 · 2026-08-29 21:44:37 +00:00 · 2026-08-29 21:41:07 +00:00 · 2026-08-28 17:16:18 +00:00 · 2026-08-28 15:32:24 +00:00 · 2026-08-23 00:01:39 +00:00
11 changed files with 54 additions and 21 deletions
@@ -25,7 +25,7 @@ done
 | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
 |-------|-----|------|-----|---------------|------------|----------|
 | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
-| Mumuni | 100 | minipve | .24 | `mumuni` | Infisical vault | Hermes |
+| Mumuni | 105 | minipve | .14 | `mumuni` | Infisical vault | Hermes |
 | Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** |
 | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
 | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
@@ -71,7 +71,7 @@ model:
 custom_providers:
  - name: harness
    model: syslog-auto
-    base_url: http://192.168.68.116/v1
+    base_url: http://192.168.68.116/litellm/v1
    api_key_env: LITELLM_API_KEY
    api_mode: chat_completions
 ```
@@ -82,7 +82,7 @@ auxiliary:
  vision:
    provider: harness
    model: gemma-4-12b              # or syslog-auto
-    base_url: http://192.168.68.116/v1
+    base_url: http://192.168.68.116/litellm/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 60
@@ -97,7 +97,7 @@ auxiliary:
    target_ratio: 0.3
    provider: harness
    model: syslog-auto               # or gemma-4-12b
-    base_url: http://192.168.68.116/v1
+    base_url: http://192.168.68.116/litellm/v1
    api_key_env: LITELLM_API_KEY
    api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production>   # ← MANDATORY workaround
    timeout: 120
@@ -31,7 +31,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 | Agent | Key Alias | Host | SSH | Sub-Agents |
 |-------|-----------|------|-----|-----------|
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
-| Mumuni | `mumuni` | 192.168.68.24 | root@.24 | 6 profiles ✱ |
+| Mumuni | `mumuni` | 192.168.68.14 | hermes@.14 | 6 profiles ✱ |
 | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
 | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
 | Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — |
@@ -190,7 +190,7 @@ litellm_settings:
 | Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
 |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
 | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
-| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 |
+| Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | ✅ Fixed | systemd Hermes gateway | 2026-08-29 |
 | Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed (DeepSeek-primary) | `infisical run` | 23:30 UTC Jul 5 |
 | Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 |
 | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
@@ -285,13 +285,13 @@ auxiliary:
  vision:
    api_key: sk-<agent-key-from-vault>           # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/v1
+    base_url: http://192.168.68.116/litellm/v1
    model: gemma-4-12b
    provider: harness
  compression:
    api_key: sk-<agent-key-from-vault>           # ← workaround (same as above)
    api_key_env: LITELLM_API_KEY
-    base_url: http://192.168.68.116/v1
+    base_url: http://192.168.68.116/litellm/v1
    model: gemma-4-12b
    provider: harness
 ```
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.

 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Mumuni | CT100 | — | 192.168.68.24 | /root/.hermes | root |
+| Mumuni | CT105 | minipve | 192.168.68.14 | /home/hermes/.hermes | hermes |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
@@ -2,7 +2,7 @@
 kind: function
 name: hermes-zulip-restore
 description: >
-  Restores Zulip connectivity for any Hermes agent (Mumuni CT100, Tanko CT112,
+  Restores Zulip connectivity for any Hermes agent (Mumuni CT105 kagentz, Tanko CT112,
  Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
  path, verifies env credentials, restarts the gateway, and confirms Zulip
  connects. Run this whenever a Hermes agent stops responding on Zulip or after
@@ -51,7 +51,7 @@ gateway restart, and connection validation.

 | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
 |------|-----|---------|-------------|-------------|------|
-| Mumuni | CT100 (abiba) | minipve | 192.168.68.24 | /root/.hermes | root |
+| Mumuni | CT105 (kagentz) | minipve | 192.168.68.14 | /home/hermes/.hermes | hermes |
 | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
 | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
 | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
@@ -102,6 +102,39 @@ GPU .8 (RTX 3090)    GPU .110 (RTX 5070)   GPU .15 (Strix Halo)
 - Stack persists across reboots (systemd for exporters, Docker restart policy)

 ## Execution
+### check-health
+
+**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
+
+```bash
+# Zulip API health (POST ping)
+curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u 'abiba-bot@chat.sysloggh.net:KEY'
+# Expected: 200 (HTTP 000 = unreachable/cache)
+
+# PM2 process health
+pm2 jlist
+# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
+
+# GPU exporters (may be down per DEPLOYMENT STATUS)
+curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
+curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
+curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
+
+# Prometheus targets
+curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
+# Expected: All targets UP (may show some down if exporters not deployed)
+
+# Grafana health
+curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
+# Expected: {"status":"ok","version":"..."}
+
+# LiteLLM metrics
+curl -s http://192.168.68.116:4001/metrics | head -20
+# Expected: Prometheus-formatted metrics output
+```
+
+**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
+

 ### Phase 1: GPU Exporters

@@ -178,7 +178,7 @@ through its agent wrapper.
 | Agent | Host | Pattern | Keys | Status |
 |-------|------|---------|------|--------|
 | abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
-| mumuni | .24 (CT100 abiba) | Pi Hermes gateway (no systemd) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
+| mumuni | .14 (kagentz CT105) | systemd unit hermes-gateway.service (user hermes) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
 | tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
 | koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
 | koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
@@ -6,7 +6,7 @@ description: >
  delegation, verification, and delivery. Defines when to delegate, which
  worker to use for what, how to handle failures, and the kanban board
  protocol. Enforces context-window discipline and separation of concerns.
-  Runs on Mumuni (inside Abiba CT100, minipve, .24) via Hermes agent (Pi + Hermes Zulip gateway).
+  Runs on Mumuni (kagentz CT105, minipve, .14) via Hermes agent (Zulip gateway via systemd).
 version: 1.0.0
 ---

@@ -20,7 +20,7 @@ version: 1.0.0
 ## Topology

 **Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
-**Manager:** Mumuni (inside Abiba CT100, minipve, .24) via Hermes agent
+**Manager:** Mumuni (kagentz CT105, minipve, .14) via Hermes agent
 **Workers:** 6 profiles, all running on the same agent — no separate hosts needed

 This contract is infrastructure-agnostic in terms of which nodes are used.
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
 ## Requires

 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.24, inside Abiba CT100 on minipve), and Agent Zero Docker host (192.168.68.14)
+- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -87,7 +87,7 @@ streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
 - Adapter implements `edit_message()` using `_api_patch()` helper
 - Gateway stream consumer progressively edits the Zulip message
 - User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112) and Mumuni (inside Abiba CT 100) both have streaming active
+- Verified: Tanko (CT 112) and Mumuni (kagentz CT 105) both have streaming active

 ### Verification
 ```bash
@@ -182,13 +182,13 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |

-### Step 3: Platform B — Hermes (Tanko .122, Mumuni .24)
+### Step 3: Platform B — Hermes (Tanko .122, Mumuni .14)

 **B1: Gateway State**

 ```bash
 ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
-ssh root@192.168.68.24 "cat ~/.hermes/gateway_state.json"  # Mumuni inside Abiba CT100
+ssh hermes@192.168.68.14 "cat ~/.hermes/gateway_state.json"  # Mumuni on kagentz CT105
 ```

 Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
@@ -203,7 +203,7 @@ Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more

 | Agent | Restart command | Notes |
 |-------|-----------------|-------|
-| Mumuni (.24) | `pm2 restart abiba-zulip` | Hermes gateway runs under PM2 as `abiba-zulip` |
+| Mumuni (.14) | `ssh root@192.168.68.14 "systemctl restart hermes-gateway"` | systemd unit `hermes-gateway.service` (user hermes); sudo not installed on kagentz — restart via root SSH from Proxmox host or monitoring box |
 | Tanko (.122) | `bash /opt/hermes-zulip-plugin/run.sh` (or the agent's systemd/user unit) | Tanko does NOT use PM2 — never run `pm2 restart mumuni-zulip` for Tanko (process does not exist) |

 Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
@@ -420,7 +420,7 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
 |-------|----------|-------------|--------------|-------------|
 | **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
 | **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
-| **Mumuni** | Hermes (inside Abiba CT 100) | ✅ Connected | No issues found | None needed |
+| **Mumuni** | Hermes (kagentz CT 105, migrated 2026-08-29) | ✅ Connected | No issues found | None needed |

 ### Key Fixes Applied

@@ -65,7 +65,7 @@ triggers:
 |------|----|------|---------|
 | Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
 | Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
-| Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` |
+| Mumuni | 192.168.68.14 (kagentz CT105) | root (SSH) | `ssh root@192.168.68.14 "systemctl restart hermes-gateway"` |
 | Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |

 ## Debounce
Author	SHA1	Message	Date
mumuni-bot	ca39fead74	fix: correct Mumuni gateway lifecycle path (no sudo on kagentz) + 2 leftover CT100 mentions PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details Post-merge verification of PR #54 found: 1. Lifecycle commands in zulip-health/zulip-self-heal used 'sudo systemctl' for hermes-gateway - sudo is NOT installed on kagentz (no sudoers, no polkit user rules). Correct path: root SSH invocation (root@192.168.68.14), matching how the Proxmox host actually manages the unit. 2. hermes-zulip-restore.prose.md line 5 + zulip-resilience-v3 line 423 still said 'Mumuni CT100' in prose - repointed. Found via independent review + live runtime check (whoami, which sudo, journalctl, systemctl show). Refs PR #54, relay #738/#739.	2026-08-30 01:06:30 +00:00
mumuni-bot	9b280060b7	Merge pull request 'fix: repoint Mumuni refs from CT100/.24 to kagentz CT105/.14 post-migration' (#54 ) from fix/mumuni-kagentz-repoint into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details	2026-08-29 21:44:37 +00:00
mumuni-bot	e898048baf	fix: repoint Mumuni refs from CT100/.24 to kagentz CT105/.14 post-migration PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details Mumuni migrated off Abiba CT100 (192.168.68.24, purged) to dedicated CT kagentz (192.168.68.14) on minipve. Verified live on kagentz: - Gateway: systemd unit hermes-gateway.service (User=hermes), active - Hermes home: /home/hermes/.hermes - gateway_state.json present at ~/.hermes/gateway_state.json Updates Mumuni rows/references in 9 contracts. Abiba-owned refs (.24 pi agent, GPU monitor, infrastructure-control CRITICAL file) and historical runs/ logs intentionally left unchanged. Per relay #738 follow-ups. Refs #735-#738.	2026-08-29 21:41:07 +00:00
abiba-bot	b01469ba18	Merge pull request 'fix: standardize Hermes contract base URLs to authenticated path' (#53 ) from fix/hermes-key-enforcement-update into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 3s Details	2026-08-28 17:16:18 +00:00
root	994ae1b7ac	fix: standardize base URLs to authenticated /litellm/v1 path in remaining contracts PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s Details - Update hermes-agent-baseline.prose.md: change /v1 → /litellm/v1 in all base_url references - Update hermes-key-enforcement.prose.md: change workaround section base_url to /litellm/v1 - Ensures all contracts consistently use the authenticated LiteLLM path - DeepSeek harness exemption remains documented for external providers	2026-08-28 15:32:24 +00:00
abiba-bot	b835986d44	Merge pull request 'Add check-health section with live probes to infrastructure-monitoring contract' (#51 ) from fix/infra-check-health into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Merged by captain approval 2026-08-22: add check-health section with live probes to infrastructure-monitoring contract	2026-08-23 00:01:39 +00:00
root	d6ad016ac9	Add check-health section with live probes to infrastructure-monitoring contract PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 11s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 16s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 13s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details Fixes the canned 'API UNREACHABLE (HTTP 000)' reporting. Adds an explicit check-health execution section (Zulip POST, pm2, GPU exporters, Prometheus, Grafana, LiteLLM probes) with a hard RUN LIVE, NEVER ECHO rule, mirroring the gpu-monitor contract pattern. Captain priority 2026-08-22.	2026-08-22 23:59:28 +00:00
mumuni-bot	c3306e87e4	Merge pull request 'fix: ship pm2-self-heal crash-loop guard + align gpu-dense docs to Qwen3.8-27B' (#49 ) from fix/pm2-guard-gpu-doc into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 8s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details	2026-08-17 02:34:49 +00:00