Compare commits

..
Author SHA1 Message Date
root 6cc1b7d492 no-mistakes(document): Update 5→6 node references and Mumuni placement across all docs 2026-07-20 17:27:28 +00:00
root 35d09ba69a no-mistakes(review): Fix 5→6 node examples and parameterize restart commands 2026-07-20 17:19:18 +00:00
root df836b115c no-mistakes(review): fix review findings: 5→6 node labels, parameterize dual-gateway restart 2026-07-20 17:13:55 +00:00
root 9f0e04f22c fix: update contracts for hwepve 6th node + Mumuni migration (lxc/114 on hwepve)
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s
- proxmox-monitor: 5→6 nodes, add hwepve (.4, Huawei Matebook 16), note Mumuni (lxc/114) placement
- zulip-health: add dual-gateway collision detection to B2, note Mumuni on hwepve
- hermes-agent-baseline: fix Mumuni node minipve→hwepve
- hermes-zulip-restore: add hwepve Proxmox column for Mumuni
- mumuni-delegation: fix CT 118/storepve/.6 → lxc/114/hwepve/.123, 5→6 nodes
- pm2-self-heal: verified — no Mumuni references, no changes needed

Post-migration discoveries:
- Mumuni container migrated from minipve to hwepve (6th node, 192.168.68.4)
- IP preserved (192.168.68.123), hostname 'mumuni' unchanged
- Dual gateway collision (PID 1442 --replace + PID 2924 --force) resolved by
  container restart at 10:07; gateway restarted clean at 13:06 with fresh Zulip
  queue (errors=0, reconnects=0)
2026-07-20 17:08:58 +00:00
14 changed files with 59 additions and 246 deletions
+1 -1
View File
@@ -137,7 +137,7 @@ Encode topology, architectural decisions, and lessons. You read them before plan
| Contract | Description | | Contract | Description |
|---|---| |---|---|
| `infrastructure-control` | Full topology and control pattern: 5-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. | | `infrastructure-control` | Full topology and control pattern: 6-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
| `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. | | `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
| `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. | | `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
+2 -2
View File
@@ -1,5 +1,5 @@
registry_version: 0.1.0 registry_version: 0.1.0
last_updated: '2026-07-23T00:00:00Z' last_updated: '2026-07-13T00:00:00Z'
updated_by: mumuni updated_by: mumuni
categories: categories:
- compliance - compliance
@@ -628,7 +628,7 @@ contracts:
sensitivity: high sensitivity: high
status: active status: active
owner: abiba owner: abiba
version: 3.1.0 version: 3.0.0
trigger: trigger:
type: scheduled type: scheduled
cadence: '*/15 * * * *' cadence: '*/15 * * * *'
+1 -1
View File
@@ -286,7 +286,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
| 111 | tdunna | amdpve | ✅ reachable | | 111 | tdunna | amdpve | ✅ reachable |
| 112 | tanko | amdpve | ✅ reachable | | 112 | tanko | amdpve | ✅ reachable |
| 113 | baggy | amdpve | ✅ reachable | | 113 | baggy | amdpve | ✅ reachable |
| 114 | mumuni | minipve | ✅ reachable | | 114 | mumuni | hwepve | ✅ reachable |
| 115 | scottdenya | amdpve | ✅ reachable | | 115 | scottdenya | amdpve | ✅ reachable |
| 116 | syslog-api | minipve | ✅ reachable | | 116 | syslog-api | minipve | ✅ reachable |
| 117 | zulip | storepve | ✅ reachable | | 117 | zulip | storepve | ✅ reachable |
+1 -1
View File
@@ -185,7 +185,7 @@ what, and why should I care?
``` ```
❌ "Monitors infrastructure health" ❌ "Monitors infrastructure health"
✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker ✅ "Scans all 6 Proxmox nodes and 19 CTs for disk pressure, checks Docker
container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL" container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
``` ```
+1 -1
View File
@@ -25,7 +25,7 @@ done
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform | | Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|-------|-----|------|-----|---------------|------------|----------| |-------|-----|------|-----|---------------|------------|----------|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes | | Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes | | Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** | | Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes | | Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) | | Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
+1 -1
View File
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
| Host | CT | Proxmox | IP (direct) | Hermes Home | User | | Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|------|-----|---------|-------------|-------------|------| |------|-----|---------|-------------|-------------|------|
| Mumuni | CT114 | | 192.168.68.123 | /root/.hermes | root | | Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome | | Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root | | Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky | | Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
+9 -8
View File
@@ -3,7 +3,7 @@ kind: pattern
name: infrastructure-control name: infrastructure-control
description: > description: >
Full infrastructure monitoring and control pattern covering the Full infrastructure monitoring and control pattern covering the
5-node Proxmox cluster, 3 Docker ecosystems (22 containers), 6-node Proxmox cluster, 3 Docker ecosystems (22 containers),
NFS storage, and network services. Defines monitors, remediations, NFS storage, and network services. Defines monitors, remediations,
and the access matrix for all environments. and the access matrix for all environments.
@@ -42,8 +42,8 @@ description: >
│ │ │ │ │ │ │ │ │ │
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
minipve amdpve storepve acerpve ocupve minipve amdpve storepve acerpve ocupve hwepve
(.12) (.15) (.6) (.9) (.5) (.12) (.15) (.6) (.9) (.5) (.4)
┌─────────────────────────────────────────────┐ ┌─────────────────────────────────────────────┐
@@ -95,15 +95,16 @@ description: >
## Section 2: Proxmox Cluster — Monitoring ## Section 2: Proxmox Cluster — Monitoring
### Nodes (5) ### Nodes (6)
| Node | IP | CPU | RAM | VMs/CTs | Role | | Node | IP | CPU | RAM | VMs/CTs | Role |
|------|----|-----|-----|---------|------| |------|----|-----|-----|---------|------|
| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging | | minipve | .12 | 16C | 30GB | authentik, gitea, syslog-api, jitsi | Auth, git, messaging |
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute | | amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat | | storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs | | acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs | | ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
| hwepve | .4 | 12C | 15GB | mumuni | Huawei Matebook 16, agent host |
### Checks (every 5 min) ### Checks (every 5 min)
@@ -117,7 +118,7 @@ description: >
## Checks ## Checks
- For each node in [minipve, amdpve, storepve, acerpve, ocupve]: - For each node in [minipve, amdpve, storepve, acerpve, ocupve, hwepve]:
- GET /api2/json/nodes/{node}/status → check status == "online" - GET /api2/json/nodes/{node}/status → check status == "online"
- GET /api2/json/nodes/{node}/status → cpu < 0.80 - GET /api2/json/nodes/{node}/status → cpu < 0.80
- GET /api2/json/nodes/{node}/status → free_mem > 10% - GET /api2/json/nodes/{node}/status → free_mem > 10%
@@ -589,7 +590,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ | | 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ | | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
| 113 | baggy | amdpve | ? | Hermes agent | ✅ | | 113 | baggy | amdpve | ? | Hermes agent | ✅ |
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ | | 114 | mumuni | hwepve | .123 | Hermes agent | ✅ |
| 115 | scottdenya | amdpve | — | ? | ❌ | | 115 | scottdenya | amdpve | — | ? | ❌ |
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ | | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
| 117 | zulip | storepve | — | Chat | ❌ | | 117 | zulip | storepve | — | Chat | ❌ |
@@ -621,7 +622,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
| 115 | scottdenya | amdpve | `pct-run 115` | | 115 | scottdenya | amdpve | `pct-run 115` |
| 104 | authentik | minipve | `pct-run 104` | | 104 | authentik | minipve | `pct-run 104` |
| 110 | gitea | minipve | `pct-run 110` | | 110 | gitea | minipve | `pct-run 110` |
| 114 | mumuni | minipve | `pct-run 114` | | 114 | mumuni | hwepve | `pct-run 114` |
| 116 | syslog-api | minipve | `pct-run 116` | | 116 | syslog-api | minipve | `pct-run 116` |
| 106 | ra-h-os | storepve | `pct-run 106` | | 106 | ra-h-os | storepve | `pct-run 106` |
| 107 | proxmox-backup | storepve | `pct-run 107` | | 107 | proxmox-backup | storepve | `pct-run 107` |
+5 -4
View File
@@ -2,7 +2,7 @@
kind: responsibility kind: responsibility
name: infrastructure-update name: infrastructure-update
description: > description: >
Autonomous system-wide update contract covering all 5 Proxmox nodes, Autonomous system-wide update contract covering all 6 Proxmox nodes,
15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages, 15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
Docker images, and container stacks in safe waves with health checks Docker images, and container stacks in safe waves with health checks
and automatic rollback on failure. and automatic rollback on failure.
@@ -56,10 +56,11 @@ Before ANY update wave:
| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min | | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| hwepve (.4) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min | | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min | | CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min | | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min | | CT 114 (mumuni, hwepve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min | | VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min | | VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
@@ -183,7 +184,7 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants:
## Success Criteria ## Success Criteria
- [ ] All 5 PVE nodes updated, no reboot-loop - [ ] All 6 PVE nodes updated, no reboot-loop
- [ ] All VMs/CTs running post-update - [ ] All VMs/CTs running post-update
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117) - [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
- [ ] LiteLLM inference passing (syslog-auto test) - [ ] LiteLLM inference passing (syslog-auto test)
@@ -199,7 +200,7 @@ After completion, send Zulip DM:
``` ```
📋 Infrastructure Update — YYYY-MM-DD 📋 Infrastructure Update — YYYY-MM-DD
Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers Updated: 6 PVE nodes, 12 CTs/VMs, 30+ containers
Security fixes: N CVEs patched Security fixes: N CVEs patched
Downtime: <service> <duration> Downtime: <service> <duration>
Failures: none / <details> Failures: none / <details>
+7 -7
View File
@@ -6,7 +6,7 @@ description: >
delegation, verification, and delivery. Defines when to delegate, which delegation, verification, and delivery. Defines when to delegate, which
worker to use for what, how to handle failures, and the kanban board worker to use for what, how to handle failures, and the kanban board
protocol. Enforces context-window discipline and separation of concerns. protocol. Enforces context-window discipline and separation of concerns.
Runs on Mumuni (CT 118, storepve, .6) via Hermes agent. Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
version: 1.0.0 version: 1.0.0
--- ---
@@ -19,8 +19,8 @@ version: 1.0.0
## Topology ## Topology
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve) **Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent **Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed **Workers:** 6 profiles, all running on the same agent — no separate hosts needed
This contract is infrastructure-agnostic in terms of which nodes are used. This contract is infrastructure-agnostic in terms of which nodes are used.
@@ -31,7 +31,7 @@ Workers execute tasks on whatever infrastructure they're given — SSH to .6,
## Why This Matters ## Why This Matters
Without enforced delegation, the manager consumes the full iteration budget Without enforced delegation, the manager consumes the full iteration budget
(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs — (60 calls) on single-turn tasks — SSH to 6 nodes, check each VM, read logs —
leaving no capacity for actual coordination. The result: context overflow leaving no capacity for actual coordination. The result: context overflow
(59K tokens in system prompt), iteration exhaustion, and degraded response (59K tokens in system prompt), iteration exhaustion, and degraded response
quality. This contract exists because I blew through my budget checking quality. This contract exists because I blew through my budget checking
@@ -82,7 +82,7 @@ it asks the manager (via relay) — it doesn't go find it on its own.
**This is a hard rule, not a recommendation.** Violating it produces the exact **This is a hard rule, not a recommendation.** Violating it produces the exact
type of discrepancy the kanban pipeline exists to prevent: a review worker finds type of discrepancy the kanban pipeline exists to prevent: a review worker finds
"5 nodes present" in the raw data but "5/5 online" in the report — even though "6 nodes present" in the raw data but "5/6 online" in the report — even though
one of those nodes was unreachable. The report lied because it used data the one of those nodes was unreachable. The report lied because it used data the
raw data never provided. raw data never provided.
@@ -137,7 +137,7 @@ delegate_task(
``` ```
delegate_task( delegate_task(
tasks=[ tasks=[
{"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"}, {"goal": "Check all 6 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
{"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"}, {"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
] ]
) )
@@ -191,7 +191,7 @@ Only verified results reach Kwame. Format per channel:
{ {
"lane_id": "devops-check", "lane_id": "devops-check",
"worker": "syslog-devops", "worker": "syslog-devops",
"goal": "Check all 5 Proxmox nodes", "goal": "Check all 6 Proxmox nodes",
"status": "dispatched|completed|failed", "status": "dispatched|completed|failed",
"output_file": "/tmp/node-report.md" "output_file": "/tmp/node-report.md"
} }
+9 -8
View File
@@ -5,7 +5,7 @@ description: >
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve (Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx). the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
agent: abiba agent: abiba
@@ -20,7 +20,7 @@ agent: abiba
└───────┬──────────────┬──────────────┬───────────────────────┘ └───────┬──────────────┬──────────────┬───────────────────────┘
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
pve-exporter docker-stats (scrapes 5x node_exporter) pve-exporter docker-stats (scrapes 6x node_exporter)
:9221 :9324 :9221 :9324
│ │ │ │
▼ ▼ ▼ ▼
@@ -33,8 +33,8 @@ agent: abiba
| Exporter | Host:Port | Scope | Notes | | Exporter | Host:Port | Scope | Notes |
|----------|-----------|-------|-------| |----------|-----------|-------|-------|
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). | | prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. | | node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. | | docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
## PVE API Token ## PVE API Token
@@ -48,7 +48,7 @@ agent: abiba
| UID | Title | Panels | Source | | UID | Title | Panels | Source |
|-----|-------|--------|--------| |-----|-------|--------|--------|
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries | | proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) | | proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) | | docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) | | gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
@@ -71,15 +71,15 @@ agent: abiba
| File | Host | Purpose | | File | Host | Purpose |
|------|------|---------| |------|------|---------|
| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) | | `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) | | `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x6, docker-stats) |
| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) | | `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter | | `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator | | `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions | | `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning | | `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config | | `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
## Cluster "Tabiri" — 5 Nodes ## Cluster "Tabiri" — 6 Nodes
| Node | IP | Role | | Node | IP | Role |
|------|----|----| |------|----|----|
@@ -88,6 +88,7 @@ agent: abiba
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) | | acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
| minipve | 192.168.68.12 | PVE | | minipve | 192.168.68.12 | PVE |
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) | | amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
## Operations ## Operations
+3 -1
View File
@@ -17,10 +17,11 @@ declare -A CT_NODES=(
[112]=amdpve # tanko [112]=amdpve # tanko
[113]=amdpve # baggy [113]=amdpve # baggy
[115]=amdpve # scottdenya [115]=amdpve # scottdenya
# hwepve (192.168.68.4) — Huawei Matebook 16
[114]=hwepve # mumuni (migrated from minipve 2026-07-20)
# minipve (192.168.68.12) # minipve (192.168.68.12)
[104]=minipve # authentik [104]=minipve # authentik
[110]=minipve # gitea [110]=minipve # gitea
[114]=minipve # mumuni
[116]=minipve # syslog-api [116]=minipve # syslog-api
# storepve (192.168.68.6) # storepve (192.168.68.6)
[106]=storepve # ra-h-os [106]=storepve # ra-h-os
@@ -48,6 +49,7 @@ declare -A NODE_IPS=(
[storepve]=192.168.68.6 [storepve]=192.168.68.6
[acerpve]=192.168.68.9 [acerpve]=192.168.68.9
[ocupve]=192.168.68.5 [ocupve]=192.168.68.5
[hwepve]=192.168.68.4
) )
resolve_node() { resolve_node() {
+3 -2
View File
@@ -46,12 +46,13 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology: The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
**Proxmox Cluster "Tabiri" (5 nodes):** **Proxmox Cluster "Tabiri" (6 nodes):**
- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya - amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
- minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi - minipve (192.168.68.12): authentik, gitea, syslog-api, jitsi
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip - storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
- acerpve (192.168.68.9): llm-gpu, adguard - acerpve (192.168.68.9): llm-gpu, adguard
- ocupve (192.168.68.5): ocu-llm - ocupve (192.168.68.5): ocu-llm
- hwepve (192.168.68.4): mumuni (migrated from minipve 2026-07-20, Huawei Matebook 16, 12C/15GB)
**CT IDs (verified 2026-07-04 against PVE API):** **CT IDs (verified 2026-07-04 against PVE API):**
100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os 100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os
+3 -7
View File
@@ -90,16 +90,12 @@ echo "── 3. Cross-contract consistency ──"
# Check that contracts referencing each other have correct names # Check that contracts referencing each other have correct names
if [ -f "infrastructure-control.prose.md" ]; then if [ -f "infrastructure-control.prose.md" ]; then
# Any contract that claims to check "all 5 PVE nodes" should name them # Any contract that claims to check "all 6 PVE nodes" should name them
for f in *.prose.md; do for f in *.prose.md; do
[ -f "$f" ] || continue [ -f "$f" ] || continue
if grep -q "5-node\|5 node\|5 Proxmox\|all.*PVE.*node" "$f" 2>/dev/null; then if grep -qE "\b5-node\b|\b5 node\b|\b5 Proxmox\b|all 5 PVE" "$f" 2>/dev/null; then
for node in amdpve minipve storepve acerpve ocupve; do echo " ⚠️ $f: still references 5-node cluster (migrated to 6 nodes 2026-07-20)"
grep -q "$node" "$f" || {
echo " ⚠️ $f: references 5 nodes but '$node' not mentioned"
WARNINGS=$((WARNINGS + 1)) WARNINGS=$((WARNINGS + 1))
}
done
fi fi
done done
fi fi
+12 -201
View File
@@ -1,9 +1,9 @@
--- ---
kind: responsibility kind: responsibility
name: zulip-health name: zulip-health
description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Abiba pi), Platform B (Hermes agents Tanko/Mumuni/Koonimo/Koby), and Platform C (Agent Zero). Verifies bot registration, DM delivery, cross-platform connectivity, secret injection, and YAML config integrity. description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
title: Zulip Mesh Health Monitor — Multi-Platform title: Zulip Mesh Health Monitor — Multi-Platform
version: 3.1.0 version: 3.0.0
runtime_contract: 2 runtime_contract: 2
agent: abiba agent: abiba
--- ---
@@ -12,15 +12,11 @@ agent: abiba
Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero). Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
Runs every 15 minutes in the background. Also triggers on session start. Runs every 15 minutes in the background. Also triggers on session start.
v3.1.0 adds Koonimo+Koby to Platform B, Infisical dependency checks, config YAML
validation, stale PID/lock detection, Telegram adapter health, and the
cli_agent_setup_mixin patch verification.
## Requires ## Requires
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY` - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14) - **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
- **SSH access to amdpve (192.168.68.15)** for `pct exec` access to Koonimo (CT 113) and Koby (CT 111)
- **PM2** on localhost for pi process management - **PM2** on localhost for pi process management
- **Network access** to `chat.sysloggh.net`, `localhost:9200` - **Network access** to `chat.sysloggh.net`, `localhost:9200`
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce` - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -53,22 +49,12 @@ cli_agent_setup_mixin patch verification.
"zulip_state": "connected", "zulip_state": "connected",
"heartbeat_age_seconds": 45, "heartbeat_age_seconds": 45,
"gateway_pid": 1234, "gateway_pid": 1234,
"infisical_present": true,
"config_valid": true,
"telegram_state": "connected",
"no_key_required_count": 0,
"edit_fail_rate_pct": 0, "edit_fail_rate_pct": 0,
"severity": "healthy" "severity": "healthy"
} }
} }
``` ```
New fields in v3.1.0:
- `infisical_present` — /usr/local/bin/infisical exists on the agent CT
- `config_valid` — /root/.hermes/config.yaml passes YAML validation
- `telegram_state` — Telegram adapter status from gateway_state.json
- `no_key_required_count` — count of `no-key-required` in gateway logs
### Postconditions ### Postconditions
- Every platform is independently checked; one failure doesn't block others - Every platform is independently checked; one failure doesn't block others
@@ -95,13 +81,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
## Streaming Support (2026-07-05) ## Streaming Support (2026-07-05)
Zulip agents now support progressive message editing during agent generation. Zulip agents now support progressive message editing during agent generation.
When a Hermes agent (Tanko, Mumuni, Koonimo, Koby) processes a message, the When a Hermes agent (Tanko, Mumuni) processes a message, the response is
response is streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API: streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
- Adapter implements `edit_message()` using `_api_patch()` helper - Adapter implements `edit_message()` using `_api_patch()` helper
- Gateway stream consumer progressively edits the Zulip message - Gateway stream consumer progressively edits the Zulip message
- User sees real-time agent thinking instead of waiting for full response - User sees real-time agent thinking instead of waiting for full response
- Verified: Tanko (CT 112), Mumuni (CT 114), Koonimo (CT 113), Koby (CT 111) - Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
### Verification ### Verification
```bash ```bash
@@ -196,217 +182,50 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
| `last_error` set | Log and monitor | | `last_error` set | Log and monitor |
| Crash loop >10/h | Alert user | | Crash loop >10/h | Alert user |
### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123, Koonimo .114, Koby .129) ### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
Platform B now monitors four Hermes agents:
- Tanko (CT 112, 192.168.68.122) — Zulip + Telegram
- Mumuni (CT 114, 192.168.68.123) — Zulip + Telegram + Email
- Koonimo (CT 113, 192.168.68.114, hostname "baggy") — Zulip + Telegram
- Koby (CT 111, 192.168.68.129, hostname "tdunna") — Zulip + Telegram
SSH access: Koonimo is reachable at .114; Koby has no direct SSH. Use `pct exec`
from amdpve as the primary access method for both:
```bash
ssh root@192.168.68.15 "pct exec 113 -- <command>" # Koonimo (or ssh .114)
ssh root@192.168.68.15 "pct exec 111 -- <command>" # Koby (pct exec only)
```
**B1: Gateway State** **B1: Gateway State**
```bash ```bash
ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json" ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
``` ```
Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌. Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` | missing → not installed.
**B1.5: Infisical Dependency Check**
```bash
ssh root@<CT> "test -f /usr/local/bin/infisical && echo OK || echo MISSING"
# pct exec variant for Koonimo/Koby:
ssh root@192.168.68.15 "pct exec 113 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
ssh root@192.168.68.15 "pct exec 111 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
```
If MISSING → flag `infisical_present: false`, note as degraded — gateway cannot
auto-start on reboot without the infisical binary.
**B2: Agent Process** **B2: Agent Process**
```bash ```bash
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep" ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- ps aux | grep 'gateway run' | grep -v grep"
ssh root@192.168.68.15 "pct exec 111 -- ps aux | grep 'gateway run' | grep -v grep"
``` ```
Gateway PID should exist with uptime > 60s. Check for stale PIDs: Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway: for Mumuni (PM2-managed), use `pm2 restart mumuni-zulip`; for Tanko (systemd/non-PM2), use `hermes gateway restart`. Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
- `gateway.pid` and `gateway.lock` files that reference a dead process
- Multiple gateway processes (duplicate PIDs)
**B2.5: Config YAML Validation**
```bash
ssh root@<CT> "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
ssh root@192.168.68.15 "pct exec 111 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
```
Expected: no output (clean parse). If parse fails → flag `config_valid: false`,
degraded — gateway is running on stale in-memory config.
Check specifically for:
- Stray `api_key: sk-...` lines indented under `api_key_env` entries in
`custom_providers` section (hardcoded keys violate hermes-key-enforcement)
- Indentation errors in `custom_providers`, `auxiliary`, or `compression` blocks
**B3: Heartbeat Verification** **B3: Heartbeat Verification**
```bash ```bash
ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3" ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
ssh root@192.168.68.15 "pct exec 111 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
``` ```
Expected: recent heartbeat (within 5 min), `polls=N` incrementing. Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
Silence > 300s → warning. Silence > 600s → critical. Silence > 300s → warning. Silence > 600s → critical.
**B3.5: Stale PID/Lock Detection**
Before any restart action, check for stale pid/lock files:
```bash
ssh root@<CT> "ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
ssh root@192.168.68.15 "pct exec 111 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
```
If gateway process is dead (no PID) but pid/lock files exist:
```bash
ssh root@<CT> "rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
ssh root@192.168.68.15 "pct exec 111 -- rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
```
Pid/lock files blocking restart → clear them before restart attempt.
**B4: Response Delivery** **B4: Response Delivery**
```bash ```bash
ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10" ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
# pct exec variant for Koonimo/Koby:
ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
``` ```
> 50% fail rate → critical. > 50% fail rate → critical.
**B4.5: LiteLLM Key Injection Verification**
Check gateway logs for `no-key-required` failure pattern (indicates the
cli_agent_setup_mixin.py patch is missing):
```bash
ssh root@<CT> "grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
# pct exec variant for Koonimo/Koby:
ssh root@192.168.68.15 "pct exec 113 -- grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
ssh root@192.168.68.15 "pct exec 111 -- grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
```
If > 0 → flag `no_key_required_count: <count>`, note as degraded — provider
requests silently fall back to `no-key-required` when LITELLM_API_KEY env var
resolves empty.
**B5: Telegram Adapter Health**
Check Telegram connectivity in gateway state or logs:
```bash
# From gateway_state.json (all agents):
ssh root@<CT> "cat ~/.hermes/gateway_state.json | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d[\"platforms\"].get(\"telegram\",{}).get(\"state\",\"missing\"))'"
# pct exec variant for Koonimo/Koby (cat the file; read platforms.telegram.state):
ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
# From logs (check for stuck DNS resolution):
ssh root@<CT> "grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
```
Telegram states: `connected` ✅ | `disconnected` ❌ | `retrying` ⚠️ | `fatal` ❌ | `paused` ⚠️
If stuck on "attempt 1/8" for > 60s → flag Telegram as degraded (Zulip may
still be fine — do NOT treat as Zulip outage).
**B6: cli_agent_setup_mixin.py Patch Verification**
Check whether the `no-key-required` fallback string exists without the
LiteLLM-specific guard (only needed when LiteLLM key injection failures are
suspected). A bare count of the fallback string cannot distinguish a guarded
occurrence from an unguarded one, so count both the fallback string and the
LiteLLM guard token:
```bash
ssh root@<CT> "f=\$(grep -c 'no-key-required' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c 'LITELLM_API_KEY' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
ssh root@192.168.68.15 "pct exec 111 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
```
If `fallback > 0` and `guard == 0` → the fallback string is present without
the LiteLLM-specific guard, so the patch is missing. If unsure, inspect each
occurrence with `grep -n -B2 -A2 'no-key-required'` to confirm the guard
wraps it.
**Platform B Actions** **Platform B Actions**
| Condition | Action | | Condition | Action |
|-----------|--------| |-----------|--------|
| `zulip.state != "connected"` | Restart gateway (see B2 restart commands below) | | `zulip.state != "connected"` | Mumuni (PM2): `ssh root@<CT> "pm2 restart mumuni-zulip"`<br>Tanko (systemd): `ssh root@<CT> "hermes gateway restart"` |
| No heartbeat in 10min | Restart gateway | | No heartbeat in 10min | Same as above |
| `Failed to finalize` > 50% | Check PATCH API, Zulip server | | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
| Response empty/short | Check A2A endpoint / LiteLLM model | | Response empty/short | Check A2A endpoint / LiteLLM model |
| `infisical_present: false` | Flag as degraded — log and alert; do NOT auto-restart via the standard command (it cannot inject the key without infisical). Operator may use the Infisical-missing fallback restart below. |
| `config_valid: false` | Flag as degraded — alert user, gateway running on stale config |
| Stale pid/lock files detected | Clean files before restart |
| `no_key_required_count > 0` | Flag as degraded — check LITELLM_API_KEY injection |
| Telegram stuck on attempt 1/8 | Flag Telegram as degraded, no Zulip action needed |
**Restart Commands**
Standard restart (Infisical present):
```bash
ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
ssh root@192.168.68.15 "pct exec 111 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
```
The mechanisms differ by design: Tanko/Mumuni launch the gateway through a
wrapper loop, so the `hermes gateway restart` CLI is the correct entry point
(it re-arms the wrapper). Koonimo/Koby run the gateway as a systemd unit
(`hermes-gateway.service`), so `systemctl restart hermes-gateway` is the
correct entry point under `pct exec`. Do not swap the two — using the CLI on
Koonimo/Koby would bypass the unit, and using `systemctl` on Tanko/Mumuni
would miss the wrapper loop.
Fallback: Infisical missing → start gateway directly from venv with env vars:
```bash
ssh root@<CT> "source /usr/local/lib/hermes-agent/venv/bin/activate && \
export LITELLM_API_KEY=\$(grep -E '^LITELLM_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
export ZULIP_API_KEY=\$(grep -E '^ZULIP_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
cd /root/.hermes && nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &"
# pct exec variant:
ssh root@192.168.68.15 "pct exec 113 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
ssh root@192.168.68.15 "pct exec 111 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
```
### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14) ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
@@ -459,7 +278,7 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`.
Check each agent's log for excessive bot-to-bot chatter: Check each agent's log for excessive bot-to-bot chatter:
- Abiba: `Skipped.*bot msgs` count - Abiba: `Skipped.*bot msgs` count
- Tanko/Mumuni/Koonimo/Koby: Repeated DM exchanges between bots - Tanko/Mumuni: Repeated DM exchanges between bots
- kagentz: Adapter log for bot DMs being processed - kagentz: Adapter log for bot DMs being processed
If any bot processes >50 bot-originated messages in 15min → warning. If any bot processes >50 bot-originated messages in 15min → warning.
@@ -482,14 +301,6 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
## History ## History
### v3.1.0 (2026-07-23) — Koonimo Outage Lessons
Added Koonimo (CT 113) and Koby (CT 111) to Platform B monitoring.
Added Infisical dependency check, config YAML validation, stale PID/lock
detection, Telegram adapter health, LiteLLM key injection verification, and
cli_agent_setup_mixin.py patch verification. Updated restart commands with
Infisical-missing fallback path.
### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix ### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
**Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours. **Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.