From 9f0e04f22c3b9a3adefffe3fea461e61ac2fc058 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 20 Jul 2026 17:08:58 +0000 Subject: [PATCH] fix: update contracts for hwepve 6th node + Mumuni migration (lxc/114 on hwepve) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement - zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve - hermes-agent-baseline: fix Mumuni node minipve→hwepve - hermes-zulip-restore: add hwepve Proxmox column for Mumuni - mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes - pm2-self-heal: verified — no Mumuni references, no changes needed Post-migration discoveries: - Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4) - IP preserved (192.168.68.123), hostname 'mumuni' unchanged - Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip queue (errors=0, reconnects=0) --- hermes-agent-baseline.prose.md | 2 +- hermes-zulip-restore.prose.md | 2 +- mumuni-delegation-prose-contract.prose.md | 6 +++--- proxmox-monitor.prose.md | 13 +++++++------ zulip-health.prose.md | 4 ++-- 5 files changed, 14 insertions(+), 13 deletions(-) diff --git a/hermes-agent-baseline.prose.md b/hermes-agent-baseline.prose.md index 02b9f17..e0a35c9 100644 --- a/hermes-agent-baseline.prose.md +++ b/hermes-agent-baseline.prose.md @@ -25,7 +25,7 @@ done | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | |-------|-----|------|-----|---------------|------------|----------| | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes | -| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes | +| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes | | Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** | | Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | diff --git a/hermes-zulip-restore.prose.md b/hermes-zulip-restore.prose.md index c5df8cf..d48991c 100644 --- a/hermes-zulip-restore.prose.md +++ b/hermes-zulip-restore.prose.md @@ -51,7 +51,7 @@ gateway restart, and connection validation. | Host | CT | Proxmox | IP (direct) | Hermes Home | User | |------|-----|---------|-------------|-------------|------| -| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root | +| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | diff --git a/mumuni-delegation-prose-contract.prose.md b/mumuni-delegation-prose-contract.prose.md index 89a2919..91d8121 100644 --- a/mumuni-delegation-prose-contract.prose.md +++ b/mumuni-delegation-prose-contract.prose.md @@ -6,7 +6,7 @@ description: > delegation, verification, and delivery. Defines when to delegate, which worker to use for what, how to handle failures, and the kanban board protocol. Enforces context-window discipline and separation of concerns. - Runs on Mumuni (CT 118, storepve, .6) via Hermes agent. + Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent. version: 1.0.0 --- @@ -19,8 +19,8 @@ version: 1.0.0 ## Topology -**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve) -**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent +**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve) +**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent **Workers:** 6 profiles, all running on the same agent — no separate hosts needed This contract is infrastructure-agnostic in terms of which nodes are used. diff --git a/proxmox-monitor.prose.md b/proxmox-monitor.prose.md index 33a0151..61ecdfc 100644 --- a/proxmox-monitor.prose.md +++ b/proxmox-monitor.prose.md @@ -5,7 +5,7 @@ description: > Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single - instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter + instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx). agent: abiba @@ -33,8 +33,8 @@ agent: abiba | Exporter | Host:Port | Scope | Notes | |----------|-----------|-------|-------| -| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). | -| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. | +| prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). | +| node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. | | docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. | ## PVE API Token @@ -48,7 +48,7 @@ agent: abiba | UID | Title | Panels | Source | |-----|-------|--------|--------| -| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries | +| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries | | proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) | | docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) | | gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) | @@ -77,9 +77,9 @@ agent: abiba | `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator | | `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions | | `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning | -| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config | +| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config | -## Cluster "Tabiri" — 5 Nodes +## Cluster "Tabiri" — 6 Nodes | Node | IP | Role | |------|----|----| @@ -88,6 +88,7 @@ agent: abiba | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) | | minipve | 192.168.68.12 | PVE | | amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) | +| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 | ## Operations diff --git a/zulip-health.prose.md b/zulip-health.prose.md index 5dcf8de..876fd2e 100644 --- a/zulip-health.prose.md +++ b/zulip-health.prose.md @@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start. ## Requires - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` -- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14) +- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14) - **PM2** on localhost for pi process management - **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` @@ -199,7 +199,7 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ssh root@ "ps aux | grep 'gateway run' | grep -v grep" ``` -Gateway PID should exist with uptime > 60s. +Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded. **B3: Heartbeat Verification**