Compare commits

...
Author SHA1 Message Date
mumuni-bot 3dcbbf1d76 ci: retrigger pipeline (validate job failed in 2s on clean tree; local repro of validation logic = 0 failures)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
2026-08-30 01:15:08 +00:00
mumuni-bot aac4c7eac3 fix: repoint Mumuni refs in Normal-sensitivity contracts to kagentz CT105/.14
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Skipped
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Skipped
Third and final PR of the post-migration contract sweep (ref PRs #54, #55).
Normal-sensitivity contracts per AGENTS.md (any registered agent):

- gpu-fleet: key roster row, Mumuni Agent Profile header + status table
  (thermal-safeguard incident note left as historical record)
- infrastructure-update: apt patch table row; Hermes config path block
  corrected to /home/hermes/.hermes + system unit /etc/systemd/system/
  hermes-gateway.service (was /root/.hermes + user unit)
- infrastructure-maintenance: Hermes gateway health-check target
- inference-optimization: apply-agent-compression host + config_path

All values verified live on kagentz 2026-08-29 (hostname, IP, user,
systemd unit, /home/hermes/.hermes). Abiba-owned refs (.24 pi agent,
GPU monitor :9100) and CRITICAL files intentionally untouched - flagged
to Abiba via relay #739.

Refs PR #54, #55; relay #738/#739.
2026-08-30 01:12:22 +00:00
mumuni-bot 031ad814a0 Merge pull request 'fix: correct Mumuni gateway lifecycle path (no sudo on kagentz) + 2 leftover CT100 mentions' (#55) from fix/restart-cmd-and-leftovers into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-08-30 01:08:13 +00:00
mumuni-bot ca39fead74 fix: correct Mumuni gateway lifecycle path (no sudo on kagentz) + 2 leftover CT100 mentions
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Post-merge verification of PR #54 found:
1. Lifecycle commands in zulip-health/zulip-self-heal used
   'sudo systemctl' for hermes-gateway - sudo is NOT installed
   on kagentz (no sudoers, no polkit user rules). Correct path:
   root SSH invocation (root@192.168.68.14), matching how the Proxmox
   host actually manages the unit.
2. hermes-zulip-restore.prose.md line 5 + zulip-resilience-v3 line 423
   still said 'Mumuni CT100' in prose - repointed.

Found via independent review + live runtime check (whoami, which sudo,
journalctl, systemctl show). Refs PR #54, relay #738/#739.
2026-08-30 01:06:30 +00:00
mumuni-bot 9b280060b7 Merge pull request 'fix: repoint Mumuni refs from CT100/.24 to kagentz CT105/.14 post-migration' (#54) from fix/mumuni-kagentz-repoint into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
2026-08-29 21:44:37 +00:00
mumuni-bot e898048baf fix: repoint Mumuni refs from CT100/.24 to kagentz CT105/.14 post-migration
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
Mumuni migrated off Abiba CT100 (192.168.68.24, purged) to dedicated
CT kagentz (192.168.68.14) on minipve. Verified live on kagentz:
- Gateway: systemd unit hermes-gateway.service (User=hermes), active
- Hermes home: /home/hermes/.hermes
- gateway_state.json present at ~/.hermes/gateway_state.json

Updates Mumuni rows/references in 9 contracts. Abiba-owned refs
(.24 pi agent, GPU monitor, infrastructure-control CRITICAL file)
and historical runs/ logs intentionally left unchanged.

Per relay #738 follow-ups. Refs #735-#738.
2026-08-29 21:41:07 +00:00
abiba-bot b01469ba18 Merge pull request 'fix: standardize Hermes contract base URLs to authenticated path' (#53) from fix/hermes-key-enforcement-update into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 3s
2026-08-28 17:16:18 +00:00
root 994ae1b7ac fix: standardize base URLs to authenticated /litellm/v1 path in remaining contracts
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s
- Update hermes-agent-baseline.prose.md: change /v1 → /litellm/v1 in all base_url references
- Update hermes-key-enforcement.prose.md: change workaround section base_url to /litellm/v1
- Ensures all contracts consistently use the authenticated LiteLLM path
- DeepSeek harness exemption remains documented for external providers
2026-08-28 15:32:24 +00:00
abiba-bot b835986d44 Merge pull request 'Add check-health section with live probes to infrastructure-monitoring contract' (#51) from fix/infra-check-health into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 6s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s
Merged by captain approval 2026-08-22: add check-health section with live probes to infrastructure-monitoring contract
2026-08-23 00:01:39 +00:00
root d6ad016ac9 Add check-health section with live probes to infrastructure-monitoring contract
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 16s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 13s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Fixes the canned 'API UNREACHABLE (HTTP 000)' reporting. Adds an explicit
check-health execution section (Zulip POST, pm2, GPU exporters, Prometheus,
Grafana, LiteLLM probes) with a hard RUN LIVE, NEVER ECHO rule, mirroring
the gpu-monitor contract pattern. Captain priority 2026-08-22.
2026-08-22 23:59:28 +00:00
mumuni-bot c3306e87e4 Merge pull request 'fix: ship pm2-self-heal crash-loop guard + align gpu-dense docs to Qwen3.8-27B' (#49) from fix/pm2-guard-gpu-doc into master
PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 5s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s
2026-08-17 02:34:49 +00:00
15 changed files with 64 additions and 31 deletions
+3 -3
View File
@@ -207,7 +207,7 @@ Plaintext keys removed from this contract post-vault-migration.
| Agent | CT | IP | LiteLLM Alias | Key Source | Access | | Agent | CT | IP | LiteLLM Alias | Key Source | Access |
|-------|-----|-----|---------------|------------|--------| |-------|-----|-----|---------------|------------|--------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome | | Tanko | 112 | .122 | `tanko` | Infisical vault | SSH jerome |
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | SSH root | | Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | SSH root |
| Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) | | Abiba | 100 | .24 | `abiba-pi` | Infisical vault | local (pi agent) |
| Koby | 111 | ? | `koby` | Infisical vault | Zulip DM | | Koby | 111 | ? | `koby` | Infisical vault | Zulip DM |
| Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH | | Koonimo | 113 | ? | `koonimo` | Infisical vault (migrated 2026-07-11) | no SSH |
@@ -304,7 +304,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
### Mumuni Agent Profile ### Mumuni Agent Profile
Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This profile is the reference for all agent configs: Mumuni (kagentz CT105, 192.168.68.14 — migrated from CT100 2026-08-29) is the primary business assistant. This profile is the reference for all agent configs:
| Setting | Value | Notes | | Setting | Value | Notes |
|---------|-------|-------| |---------|-------|-------|
@@ -329,7 +329,7 @@ Mumuni (CT100/abiba, 192.168.68.24) is the primary business assistant. This prof
| Agent | Host | Status | | Agent | Host | Status |
|-------|------|--------| |-------|------|--------|
| **Mumuni** | CT100 (.24) | ✅ Updated to stable aliases | | **Mumuni** | CT105 (.14) | ✅ Updated to stable aliases |
| **Tanko** | CT112 (.122) | ✅ Updated to stable aliases | | **Tanko** | CT112 (.122) | ✅ Updated to stable aliases |
| **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM | | **Koby** | CT111 (.129) | ❌ SSH unreachable — needs Zulip DM |
| **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM | | **Koonimo** | CT113 | ❌ SSH unreachable — needs Zulip DM |
+4 -4
View File
@@ -25,7 +25,7 @@ done
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------| |-------|-----|------|-----|---------------|------------|----------|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes | | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 100 | minipve | .24 | `mumuni` | Infisical vault | Hermes | | Mumuni | 105 | minipve | .14 | `mumuni` | Infisical vault | Hermes |
| Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** | | Koby | 111 | amdpve | .129 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes | | Koonimo | 113 | amdpve | .114 | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
@@ -71,7 +71,7 @@ model:
custom_providers: custom_providers:
- name: harness - name: harness
model: syslog-auto model: syslog-auto
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/litellm/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
api_mode: chat_completions api_mode: chat_completions
``` ```
@@ -82,7 +82,7 @@ auxiliary:
vision: vision:
provider: harness provider: harness
model: gemma-4-12b # or syslog-auto model: gemma-4-12b # or syslog-auto
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/litellm/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 60 timeout: 60
@@ -97,7 +97,7 @@ auxiliary:
target_ratio: 0.3 target_ratio: 0.3
provider: harness provider: harness
model: syslog-auto # or gemma-4-12b model: syslog-auto # or gemma-4-12b
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/litellm/v1
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround api_key: <value from: infisical secrets get LITELLM_API_KEY --project=agents --env=production> # ← MANDATORY workaround
timeout: 120 timeout: 120
+1 -1
View File
@@ -31,7 +31,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
| Agent | Key Alias | Host | SSH | Sub-Agents | | Agent | Key Alias | Host | SSH | Sub-Agents |
|-------|-----------|------|-----|-----------| |-------|-----------|------|-----|-----------|
| Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — | | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
| Mumuni | `mumuni` | 192.168.68.24 | root@.24 | 6 profiles ✱ | | Mumuni | `mumuni` | 192.168.68.14 | hermes@.14 | 6 profiles ✱ |
| Abiba | `abiba-pi` | 192.168.68.24 | local | — | | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
| Koby | `koby` | CT 111 (tdunna) | Zulip | — | | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
| Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — | | Koonimo | `koonimo` | CT 113 (baggy) | SSH root | — |
+3 -3
View File
@@ -190,7 +190,7 @@ litellm_settings:
| Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified | | Agent | CT | IP | LiteLLM Alias | Key Source | Status | Gateway Wrapper | Last Verified |
|-------|-----|-----|---------------|------------|--------|-----------------|---------------| |-------|-----|-----|---------------|------------|--------|-----------------|---------------|
| Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 | | Tanko | 112 | .122 | `tanko` | Infisical vault | ✅ Fixed | `infisical run` | 20:17 UTC Jul 5 |
| Mumuni | 100 (abiba) | .24 | `mumuni` | Infisical vault | ✅ Fixed | Pi Hermes gateway | 2026-07-27 | | Mumuni | 105 (kagentz) | .14 | `mumuni` | Infisical vault | ✅ Fixed | systemd Hermes gateway | 2026-08-29 |
| Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed (DeepSeek-primary) | `infisical run` | 23:30 UTC Jul 5 | | Koby | 111 | .129 | `koby` | Infisical vault | ✅ Fixed (DeepSeek-primary) | `infisical run` | 23:30 UTC Jul 5 |
| Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 | | Koonimo | 113 | .114 | `koonimo` | Infisical vault | ✅ Fixed | `infisical run` (migrated 2026-07-11) | 2026-08-09 |
| Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 | | Abiba | 100 | .65 | `abiba-pi` | Infisical vault | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
@@ -285,13 +285,13 @@ auxiliary:
vision: vision:
api_key: sk-<agent-key-from-vault> # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain) api_key: sk-<agent-key-from-vault> # ← workaround (get via: infisical secrets get LITELLM_API_KEY --project=agents --env=production --plain)
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/litellm/v1
model: gemma-4-12b model: gemma-4-12b
provider: harness provider: harness
compression: compression:
api_key: sk-<agent-key-from-vault> # ← workaround (same as above) api_key: sk-<agent-key-from-vault> # ← workaround (same as above)
api_key_env: LITELLM_API_KEY api_key_env: LITELLM_API_KEY
base_url: http://192.168.68.116/v1 base_url: http://192.168.68.116/litellm/v1
model: gemma-4-12b model: gemma-4-12b
provider: harness provider: harness
``` ```
+1 -1
View File
@@ -55,7 +55,7 @@ connectivity recovery including end-to-end DM validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User | | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------| |------|-----|---------|-------------|-------------|------|
| Mumuni | CT100 | — | 192.168.68.24 | /root/.hermes | root | | Mumuni | CT105 | minipve | 192.168.68.14 | /home/hermes/.hermes | hermes |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
+2 -2
View File
@@ -2,7 +2,7 @@
kind: function kind: function
name: hermes-zulip-restore name: hermes-zulip-restore
description: > description: >
Restores Zulip connectivity for any Hermes agent (Mumuni CT100, Tanko CT112, Restores Zulip connectivity for any Hermes agent (Mumuni CT105 kagentz, Tanko CT112,
Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin Koby CT111, Shumba on Lucky's mini PC). Deploys the zulip-platform adapter to the correct bundled plugin
path, verifies env credentials, restarts the gateway, and confirms Zulip path, verifies env credentials, restarts the gateway, and confirms Zulip
connects. Run this whenever a Hermes agent stops responding on Zulip or after connects. Run this whenever a Hermes agent stops responding on Zulip or after
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User | | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------| |------|-----|---------|-------------|-------------|------|
| Mumuni | CT100 (abiba) | minipve | 192.168.68.24 | /root/.hermes | root | | Mumuni | CT105 (kagentz) | minipve | 192.168.68.14 | /home/hermes/.hermes | hermes |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
+2 -2
View File
@@ -87,8 +87,8 @@ call apply-liteLLM-routing
call apply-agent-compression call apply-agent-compression
agent: mumuni agent: mumuni
host: 192.168.68.24 host: 192.168.68.14
config_path: /root/.hermes/config.yaml config_path: /home/hermes/.hermes/config.yaml
-- Phase 4: Enable llama.cpp prompt caching on GPU hosts -- Phase 4: Enable llama.cpp prompt caching on GPU hosts
+1 -1
View File
@@ -140,7 +140,7 @@ After ALL updates (apt + images + restarts), verify every critical service is ba
| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK | | Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK | | Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` | | PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
| Hermes gateways | SSH to Mumuni CT 100, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each | | Hermes gateways | SSH to Mumuni kagentz CT105 (root@.14) and Tanko CT112 (jerome@.122); `systemctl is-active hermes-gateway` | `active` for each |
Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here. Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
+33
View File
@@ -102,6 +102,39 @@ GPU .8 (RTX 3090) GPU .110 (RTX 5070) GPU .15 (Strix Halo)
- Stack persists across reboots (systemd for exporters, Docker restart policy) - Stack persists across reboots (systemd for exporters, Docker restart policy)
## Execution ## Execution
### check-health
**RUN LIVE, NEVER ECHO — every dispatch must execute the probes below with real tool calls; never repeat a prior report unless a live probe fails.**
```bash
# Zulip API health (POST ping)
curl -s -o /dev/null -w '%{http_code}' -X POST https://chat.sysloggh.net/api/v1/messages -u 'abiba-bot@chat.sysloggh.net:KEY'
# Expected: 200 (HTTP 000 = unreachable/cache)
# PM2 process health
pm2 jlist
# Expected: 5/5 online (abiba-telegram, abiba-zulip, zulip-watchdog, gitea-runner, spoton-service)
# GPU exporters (may be down per DEPLOYMENT STATUS)
curl -s http://192.168.68.8:9400/metrics && echo " - OK" || echo " - FAIL"
curl -s http://192.168.68.110:9400/metrics && echo " - OK" || echo " - FAIL"
curl -s http://192.168.68.15:9400/metrics && echo " - OK" || echo " - FAIL"
# Prometheus targets
curl -s http://192.168.68.116:9090/api/v1/targets | jq '.data.activeTargets'
# Expected: All targets UP (may show some down if exporters not deployed)
# Grafana health
curl -s http://192.168.68.116:3001/api/health | jq '{status, version}'
# Expected: {"status":"ok","version":"..."}
# LiteLLM metrics
curl -s http://192.168.68.116:4001/metrics | head -20
# Expected: Prometheus-formatted metrics output
```
**Report format**: Summarize actual results from each probe. If any probe returns non-200 or empty output, flag as alert.
### Phase 1: GPU Exporters ### Phase 1: GPU Exporters
+4 -4
View File
@@ -59,7 +59,7 @@ Before ANY update wave:
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min | | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min | | CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min | | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 100 (mumuni/abiba, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min | | CT 105 (kagentz, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min | | VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min | | VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
@@ -162,9 +162,9 @@ Before Wave 1, snapshot these files:
/etc/systemd/system/strix-server.service (amdpve .15 — strix-moe) /etc/systemd/system/strix-server.service (amdpve .15 — strix-moe)
/etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110) /etc/systemd/system/llama-server.service (VM 101 .8, VM 103 .110)
# Hermes agent configs (key enforcement — 2026-07-10) # Hermes agent configs (key enforcement — 2026-07-10)
/root/.hermes/config.yaml (Mumuni inside CT 100, Tanko CT 112, etc.) /home/hermes/.hermes/config.yaml (Mumuni kagentz CT105; Tanko CT112 uses /home/jerome/.hermes)
/root/.config/systemd/user/hermes-gateway.service (Mumuni inside CT 100 — EnvironmentFile fixed) /etc/systemd/system/hermes-gateway.service (Mumuni kagentz CT105 — system unit, User=hermes)
/etc/environment (Mumuni inside CT 100 — LITELLM_API_KEY) /etc/environment (LITELLM_API_KEY — legacy path, Mumuni now keys via Infisical)
``` ```
## MCP Gateway (2026-07-10) ## MCP Gateway (2026-07-10)
+1 -1
View File
@@ -178,7 +178,7 @@ through its agent wrapper.
| Agent | Host | Pattern | Keys | Status | | Agent | Host | Pattern | Keys | Status |
|-------|------|---------|------|--------| |-------|------|---------|------|--------|
| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed | | abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
| mumuni | .24 (CT100 abiba) | Pi Hermes gateway (no systemd) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback | | mumuni | .14 (kagentz CT105) | systemd unit hermes-gateway.service (user hermes) | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback | | tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed | | koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed | | koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
+2 -2
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns. protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (inside Abiba CT100, minipve, .24) via Hermes agent (Pi + Hermes Zulip gateway). Runs on Mumuni (kagentz CT105, minipve, .14) via Hermes agent (Zulip gateway via systemd).
version: 1.0.0 version: 1.0.0
--- ---
@@ -20,7 +20,7 @@ version: 1.0.0
## Topology ## Topology
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve) **Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
**Manager:** Mumuni (inside Abiba CT100, minipve, .24) via Hermes agent **Manager:** Mumuni (kagentz CT105, minipve, .14) via Hermes agent
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
This contract is infrastructure-agnostic in terms of which nodes are used. This contract is infrastructure-agnostic in terms of which nodes are used.
+5 -5
View File
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
## Requires ## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.24, inside Abiba CT100 on minipve), and Agent Zero Docker host (192.168.68.14) - **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.14, kagentz CT105 on minipve), and Agent Zero Docker host (192.168.68.14)
- **PM2** on localhost for pi process management - **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -87,7 +87,7 @@ streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
- Adapter implements `edit_message()` using `_api_patch()` helper - Adapter implements `edit_message()` using `_api_patch()` helper
- Gateway stream consumer progressively edits the Zulip message - Gateway stream consumer progressively edits the Zulip message
- User sees real-time agent thinking instead of waiting for full response - User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112) and Mumuni (inside Abiba CT 100) both have streaming active - Verified: Tanko (CT 112) and Mumuni (kagentz CT 105) both have streaming active
### Verification ### Verification
```bash ```bash
@@ -182,13 +182,13 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| `last_error` set | Log and monitor | | `last_error` set | Log and monitor |
| Crash loop >10/h | Alert user | | Crash loop >10/h | Alert user |
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .24) ### Step 3: Platform B — Hermes (Tanko .122, Mumuni .14)
**B1: Gateway State** **B1: Gateway State**
```bash ```bash
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.24 "cat ~/.hermes/gateway_state.json" # Mumuni inside Abiba CT100 ssh hermes@192.168.68.14 "cat ~/.hermes/gateway_state.json" # Mumuni on kagentz CT105
``` ```
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed. Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
@@ -203,7 +203,7 @@ Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more
| Agent | Restart command | Notes | | Agent | Restart command | Notes |
|-------|-----------------|-------| |-------|-----------------|-------|
| Mumuni (.24) | `pm2 restart abiba-zulip` | Hermes gateway runs under PM2 as `abiba-zulip` | | Mumuni (.14) | `ssh root@192.168.68.14 "systemctl restart hermes-gateway"` | systemd unit `hermes-gateway.service` (user hermes); sudo not installed on kagentz — restart via root SSH from Proxmox host or monitoring box |
| Tanko (.122) | `bash /opt/hermes-zulip-plugin/run.sh` (or the agent's systemd/user unit) | Tanko does NOT use PM2 — never run `pm2 restart mumuni-zulip` for Tanko (process does not exist) | | Tanko (.122) | `bash /opt/hermes-zulip-plugin/run.sh` (or the agent's systemd/user unit) | Tanko does NOT use PM2 — never run `pm2 restart mumuni-zulip` for Tanko (process does not exist) |
Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded. Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
+1 -1
View File
@@ -420,7 +420,7 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
|-------|----------|-------------|--------------|-------------| |-------|----------|-------------|--------------|-------------|
| **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s | | **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
| **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) | | **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
| **Mumuni** | Hermes (inside Abiba CT 100) | ✅ Connected | No issues found | None needed | | **Mumuni** | Hermes (kagentz CT 105, migrated 2026-08-29) | ✅ Connected | No issues found | None needed |
### Key Fixes Applied ### Key Fixes Applied
+1 -1
View File
@@ -65,7 +65,7 @@ triggers:
|------|----|------|---------| |------|----|------|---------|
| Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` | | Zulip server | 192.168.68.19 | root | Docker: `zulip-zulip-1` |
| Abiba (pi) | localhost | root | PM2: `abiba-zulip` | | Abiba (pi) | localhost | root | PM2: `abiba-zulip` |
| Mumuni | 192.168.68.24 (CT100 abiba) | root | `hermes gateway restart` | | Mumuni | 192.168.68.14 (kagentz CT105) | root (SSH) | `ssh root@192.168.68.14 "systemctl restart hermes-gateway"` |
| Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` | | Tanko | 192.168.68.122 | jerome | `PATH=$PATH:/home/jerome/.hermes/hermes-agent hermes gateway restart` |
## Debounce ## Debounce