Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 root
|
9f0e04f22c | ||
 jerome
|
fc88265e76 | ||
|
jerome
|
51da19d92d | ||
|
jerome
|
5d2ecbace6 | ||
|
root
|
99789a00a1 | ||
|
root
|
a68879e904 | ||
|
jerome
|
06d2bcbc9e | ||
|
jerome
|
c4a8c45835 | ||
|
root
|
1d027f71f6 | ||
|
jerome
|
a74229ee74 | ||
|
root
|
aebc98ead6 | ||
|
root
|
17a77e6b3f | ||
|
root
|
23f3f378c5 |
@@ -128,3 +128,10 @@ safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
|
||||
Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
|
||||
It covers the full process: verify → draft → lint → review → ship, with templates
|
||||
and style rules.
|
||||
|
||||
## Maintaining this file
|
||||
|
||||
Keep this file for knowledge useful to almost every future agent session in this project.
|
||||
Do not repeat what the codebase already shows; point to the authoritative file or command instead.
|
||||
Prefer rewriting or pruning existing entries over appending new ones.
|
||||
When updating this file, preserve this bar for all agents and keep entries concise.
|
||||
|
||||
+104
-2
@@ -22,6 +22,7 @@ owners:
|
||||
- abiba
|
||||
- mumuni
|
||||
- kwame
|
||||
- ops
|
||||
trigger_types:
|
||||
- scheduled
|
||||
- event_driven
|
||||
@@ -58,6 +59,7 @@ index:
|
||||
- memory-audit-maintenance
|
||||
- gpu-fleet
|
||||
- infrastructure-update
|
||||
- infrastructure-maintenance
|
||||
reference:
|
||||
- infrastructure-control
|
||||
- ra-h-os-custodianship-contract
|
||||
@@ -90,6 +92,7 @@ index:
|
||||
- infrastructure-control
|
||||
- infrastructure-monitoring
|
||||
- infrastructure-update
|
||||
- infrastructure-maintenance
|
||||
- pm2-self-heal
|
||||
- disk-gc-threat-response
|
||||
gpu:
|
||||
@@ -130,7 +133,6 @@ index:
|
||||
- build-zulip-plugin
|
||||
- stirling-pdf-agent-access
|
||||
- gpu-fleet
|
||||
- infrastructure-update
|
||||
- infrastructure-control
|
||||
- zulip-adapter-lessons
|
||||
- pi-approval-architecture
|
||||
@@ -144,6 +146,9 @@ index:
|
||||
- mumuni-delegation
|
||||
kwame:
|
||||
- hello-world
|
||||
ops:
|
||||
- infrastructure-maintenance
|
||||
- infrastructure-update
|
||||
by_trigger:
|
||||
scheduled:
|
||||
- hermes-key-enforcement
|
||||
@@ -156,6 +161,7 @@ index:
|
||||
- litellm-health
|
||||
- memory-audit-maintenance
|
||||
- infrastructure-update
|
||||
- infrastructure-maintenance
|
||||
event_driven:
|
||||
- litellm-self-heal
|
||||
- pm2-self-heal
|
||||
@@ -197,6 +203,7 @@ index:
|
||||
- hermes-zulip-plugin
|
||||
- build-zulip-plugin
|
||||
- infrastructure-update
|
||||
- infrastructure-maintenance
|
||||
- ra-h-os-custodianship-contract
|
||||
- mumuni-delegation
|
||||
normal:
|
||||
@@ -1256,13 +1263,108 @@ contracts:
|
||||
last_run: null
|
||||
last_status: null
|
||||
drift_alerts: []
|
||||
- name: infrastructure-maintenance
|
||||
file: infrastructure-maintenance.prose.md
|
||||
kind: responsibility
|
||||
category: maintenance
|
||||
sensitivity: high
|
||||
status: active
|
||||
owner: ops
|
||||
version: 1.0.0
|
||||
trigger:
|
||||
type: scheduled
|
||||
cadence: 0 2 * * 0
|
||||
description: Weekly host-level maintenance Sunday at 2am ET (replaces infrastructure-update
|
||||
build-phase role; infra-update moves to ops)
|
||||
cron_job_id: null
|
||||
execution:
|
||||
agent: ops
|
||||
timeout: 3600
|
||||
requires:
|
||||
- infrastructure-monitoring run within last 30 minutes (pre-update health baseline)
|
||||
- Proxmox snapshot of primary host OR /tmp backup dir created this run
|
||||
- LiteLLM master key from Infisical vault for health verification
|
||||
protocol:
|
||||
- Load contract from prose-contracts/main
|
||||
- Phase 0 preflight — capture health baseline, backup check, record image baseline
|
||||
- Phase 1 apt update && apt upgrade -y on primary host
|
||||
- Phase 2 docker compose pull for LiteLLM, SearXNG, and other running containers
|
||||
- Phase 3 restart stacks one at a time with per-stack health verification
|
||||
- Phase 4 verify every critical service (LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways)
|
||||
- On failure — rollback per protocol, escalate, do not loop beyond circuit breaker
|
||||
- Log actions to ~/.hermes/runs/infrastructure-maintenance/
|
||||
verification:
|
||||
postconditions:
|
||||
- check: all critical services running after update
|
||||
verify: 'curl -sf http://192.168.68.116/litellm/v1/models && curl -sf https://chat.sysloggh.net/api/v1/server_settings && curl -sf https://git.sysloggh.net/api/v1/version && curl -sf http://192.168.68.7:8888 && pm2 jlist'
|
||||
expect: all probes 200 OK / processes online
|
||||
- check: no regressions from pre-update health baseline
|
||||
verify: diff Phase 0 health-baseline against Phase 4 results
|
||||
expect: no GREEN service turned RED
|
||||
- check: docker containers on latest stable tags
|
||||
verify: docker inspect --format '{{.Config.Image}}' <container> per service matches image-baseline.pulled_tag
|
||||
expect: all containers running pulled tags
|
||||
- check: APT packages up to date with no held broken packages
|
||||
verify: apt list --upgradable 2>/dev/null | wc -l and apt-get -s upgrade | grep -ci broken
|
||||
expect: upgradable == 0, broken == 0
|
||||
artifact: maintenance run report with phase results and any rollback/escalation
|
||||
verify_commands:
|
||||
- curl -sf http://192.168.68.116/litellm/v1/models
|
||||
- curl -sf http://192.168.68.7:8888
|
||||
- curl -sf https://chat.sysloggh.net/api/v1/server_settings
|
||||
- curl -sf https://git.sysloggh.net/api/v1/version
|
||||
- pm2 jlist
|
||||
receipt:
|
||||
format: json
|
||||
storage: ~/.hermes/runs/infrastructure-maintenance/
|
||||
graph_node: true
|
||||
schema:
|
||||
contract: string
|
||||
run_id: string
|
||||
timestamp: ISO 8601
|
||||
agent: string
|
||||
status: pass|fail|escalated
|
||||
phase: preflight|apt|images|restarts|verify|rollback|done|failed
|
||||
actions_taken: array
|
||||
postconditions: array
|
||||
drift_alerts: array
|
||||
evidence_path: string
|
||||
escalation:
|
||||
info:
|
||||
action: log_to_receipt
|
||||
notify: []
|
||||
warning:
|
||||
action: relay_alert
|
||||
notify:
|
||||
- abiba
|
||||
- mumuni
|
||||
critical:
|
||||
action: relay_alert
|
||||
notify:
|
||||
- abiba
|
||||
- mumuni
|
||||
fatal:
|
||||
action: relay_alert + pause + human_required
|
||||
notify:
|
||||
- abiba
|
||||
- mumuni
|
||||
- kwame
|
||||
circuit_breaker:
|
||||
max_retries: 2
|
||||
window: 7200
|
||||
trip_action: escalate_to_fatal
|
||||
depends_on:
|
||||
- infrastructure-monitoring
|
||||
last_run: null
|
||||
last_status: null
|
||||
drift_alerts: []
|
||||
- name: infrastructure-update
|
||||
file: infrastructure-update.prose.md
|
||||
kind: responsibility
|
||||
category: maintenance
|
||||
sensitivity: high
|
||||
status: active
|
||||
owner: abiba
|
||||
owner: ops
|
||||
version: 1.0.0
|
||||
trigger:
|
||||
type: scheduled
|
||||
|
||||
+14
-26
@@ -9,9 +9,9 @@ description: >
|
||||
gpu-dense, gpu-light. These never change — only the underlying model does.
|
||||
Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
|
||||
RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
|
||||
UPDATED 2026-07-17: Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored,
|
||||
UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability.
|
||||
Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored,
|
||||
Hermes agent fine-tune, tensor repair, multimodal with mmproj).
|
||||
RTX 5070 swapped to HauhauCS Gemma4-12B QAT Uncensored Balanced (Q4_K_M, 87 tok/s, 0/465 refusals).
|
||||
Instability observed near 100K at 256K. 128K is the stable ceiling.
|
||||
For larger context needs → fall back to external providers (deepseek).
|
||||
VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
|
||||
@@ -87,32 +87,20 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
|
||||
| Alias | GPU | Current Model | Will Route To |
|
||||
|-------|-----|---------------|---------------|
|
||||
| `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
|
||||
| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code (ThinkingCap) | Whatever runs on RTX 3090 |
|
||||
| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
|
||||
| `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
|
||||
|
||||
**Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
|
||||
but are deprecated for agent configs. Only the stable aliases survive model swaps.
|
||||
|
||||
## Current Model Assignments (2026-07-17)
|
||||
## Current Model Assignments (2026-07-15)
|
||||
|
||||
| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
|
||||
|-------|-----|------|------|-----|----------|----------|-------------|--------|
|
||||
| qwen3.6-27B-code (ThinkingCap) | RTX 3090 | .8 (llm-gpu) | ~20.9/24.6GB (85%) | **128K** | turbo4 | 1 | default | ✅ 68 tok/s |
|
||||
| gemma-4-12b (HauhauCS QAT) | RTX 5070 | .110 (ocu-llm) | ~10.0/12.2GB (82%) | 128K | q4_0 | 1 | 2048/1024 | ✅ 87 tok/s |
|
||||
| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s |
|
||||
| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
|
||||
| Genesis Hermes V3 APEX | Strix Halo Vulkan | .15 (amdpve) | ~10GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
|
||||
|
||||
> **RTX 5070 model swap (2026-07-17)**: Switched from `gemma-4-12b-it-IQ4_NL` (Unsloth, 191 tok/s)
|
||||
> to `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` (Q4_K_M QAT, 87 tok/s).
|
||||
> Trade: 54% slower generation for QAT quality, 0/465 refusals, and agent-optimized tuning.
|
||||
> MTP draft also swapped: Q8_0 (444MB) → tuned draft (242MB), saving 200MB VRAM.
|
||||
> Role unchanged: gpu-light (vision, web extract, light auxiliary tasks).
|
||||
|
||||
> **RTX 3090 model swap (2026-07-17)**: Switched from `Qwopus3.6-27B-v2-MTP-Q4_K_M` (63 tok/s)
|
||||
> to `bottlecapai/ThinkingCap-Qwen3.6-27B` (Q4_K_M QAT, 68 tok/s).
|
||||
> RL-finetuned: 50% fewer thinking tokens, MMLU-Pro 0.85 vs 0.83 base.
|
||||
> Self-spec MTP REQUIRED (crashes without it on turboquant build).
|
||||
> Added vision via mmproj (0.9GB). VRAM 85%.
|
||||
|
||||
## Routing Configuration (LiteLLM — July 2026)
|
||||
|
||||
### syslog-auto Weighted Pool (Direct GPU — bypasses router)
|
||||
@@ -130,16 +118,16 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
|
||||
| Model | RPM Cap | Notes |
|
||||
|-------|---------|-------|
|
||||
| strix-moe (Hermes V3) | 40 | Tight cap — prevents Strix overload |
|
||||
| qwen3.6-27B-code | 500 | ThinkingCap Q4_K_M + MTP self-spec + vision, 68 tok/s |
|
||||
| gemma-4-12b | 500 | HauhauCS QAT Uncensored Balanced + MTP, 87 tok/s |
|
||||
| qwen3.6-27B-code | 500 | High cap — primary workhorse |
|
||||
| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
|
||||
|
||||
### Stable Aliases (for agent configs — never change)
|
||||
|
||||
| Alias | RPM Cap | Routes To | Purpose |
|
||||
|-------|---------|-----------|---------|
|
||||
| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
|
||||
| `gpu-dense` | 500 | RTX 3090 (ThinkingCap) | Heavy reasoning, code gen, delegation |
|
||||
| `gpu-light` | 500 | RTX 5070 (HauhauCS QAT) | Vision, web extract, light tasks |
|
||||
| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
|
||||
| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
|
||||
|
||||
### Fallback Chains
|
||||
- gemma → qwen
|
||||
@@ -261,8 +249,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
- **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
|
||||
- **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
|
||||
- **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
|
||||
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching --spec-type draft-mtp --spec-draft-n-max 4`. ThinkingCap Qwen3.6-27B Q4_K_M (15.7GB) + mmproj (0.9GB) + MTP self-spec. VRAM: ~85%. Service: `/home/llmuser/llama-wrapper.sh`. ⚠️ MTP REQUIRED for stability on this turboquant build — model segfaults without `--spec-type draft-mtp`. Outputs reasoning_content (hidden from agent, improves answer quality).
|
||||
- **RTX 5070 config (2026-07-17)**: HauhauCS Gemma4-12B QAT Uncensored Balanced (Q4_K_M) + tuned MTP draft (242MB) at 128K context, single slot. Gen speed: 87 tok/s (vs 191 IQ4_NL). VRAM: ~10.0/12.2GB (~82%). Service: `/home/llmuser/llama-wrapper.sh`. Model: `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`, MTP: `mtp-gemma-4-12B-it.gguf`, mmproj: `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`. Recommended sampling: temp 0.6, top_k 64, top_p 0.9, min_p 0.05, repeat_penalty 1.1.
|
||||
- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
|
||||
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
|
||||
- **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
|
||||
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V3-GGUF` (APEX quant), alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). Hermes agent fine-tune, tensor repair (SSM layers fixed via SVD), uncensored (0/465 refusals).
|
||||
- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
|
||||
@@ -278,8 +266,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
|
||||
|
||||
| GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
|
||||
|-----|-------|-----------|--------------|----------|---------|
|
||||
| RTX 3090 (.8) | ThinkingCap-Qwen3.6-27B Q4_K_M | **68** | — | — | **128K** |
|
||||
| RTX 5070 (.110) | HauhauCS QAT Uncensored Balanced | **87** | — | — | **128K** |
|
||||
| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** |
|
||||
| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
|
||||
| Strix Halo (.15) | Genesis Hermes V3 APEX | **65** | 140 | — | **128K** |
|
||||
|
||||
Benchmarks from 2026-07-17. Strix Halo swapped to Genesis Hermes V3 APEX (LuffyTheFox). RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
|
||||
|
||||
@@ -25,7 +25,7 @@ done
|
||||
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|
||||
|-------|-----|------|-----|---------------|------------|----------|
|
||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
|
||||
"models": [
|
||||
{ "id": "syslog-auto" },
|
||||
{ "id": "strix-moe" },
|
||||
{ "id": "gpu-dense" },
|
||||
{ "id": "gpu-light" },
|
||||
{ "id": "qwen3.6-27B-code" },
|
||||
{ "id": "gemma-4-12b" }
|
||||
]
|
||||
|
||||
@@ -332,7 +332,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
|
||||
|
||||
One-line agent health check (run on the agent host):
|
||||
```bash
|
||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
||||
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||
PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||
cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
|
||||
curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
|
||||
```
|
||||
@@ -358,9 +359,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
|
||||
The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
|
||||
See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
|
||||
|
||||
**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
|
||||
the wrapper will inject an empty key and the gateway will silently get 401 errors
|
||||
on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
|
||||
is present but the vault takes precedence when the secret key exists (even if empty).
|
||||
|
||||
**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
|
||||
is empty or shorter than 20 chars, log a warning and either fail with a clear error
|
||||
message or fall back to the `.env` value before starting the gateway.
|
||||
|
||||
**Verification (all agents):**
|
||||
```bash
|
||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
|
||||
# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
|
||||
GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
|
||||
K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
|
||||
curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models # must be 200
|
||||
```
|
||||
|
||||
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
|
||||
|
||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||
|------|-----|---------|-------------|-------------|------|
|
||||
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
|
||||
| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
|
||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
||||
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
||||
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
---
|
||||
kind: responsibility
|
||||
name: infrastructure-maintenance
|
||||
description: >
|
||||
Weekly system-level maintenance for the Syslog inference fleet: OS package
|
||||
updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
|
||||
running containers, container restarts with health verification, post-update
|
||||
verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
|
||||
PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
|
||||
raw shell scripts that previously did this piecemeal. This contract owns the
|
||||
HOST-LEVEL weekly maintenance loop on the primary host plus Docker image
|
||||
pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET
|
||||
cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker
|
||||
image Wave 3 across all stacks). Runs Sunday 2am ET. Owner:
|
||||
ops (firstmate secondmate). Blast radius: an unverified image pull can break
|
||||
LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
|
||||
upgrade can leave the host in a half-upgraded state. Pre-update backup check
|
||||
and rollback are mandatory for this reason.
|
||||
agent: ops
|
||||
triggers:
|
||||
- weekly (Sunday 02:00 ET) via cron
|
||||
- on demand when ops/abiba triggers "infra maintenance"
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
## Maintains
|
||||
|
||||
- maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
|
||||
- image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
|
||||
- apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
|
||||
- health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
|
||||
- rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
|
||||
- maintenance-history: array of past runs with phase results and any escalations
|
||||
|
||||
## Scope
|
||||
|
||||
Primary host is the maintenance host where apt updates apply. Docker image pulls
|
||||
span the two Docker ecosystems that run critical services. infrastructure-update
|
||||
runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across
|
||||
all stacks/hosts); this contract runs a narrower host-level weekly pull limited
|
||||
to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against
|
||||
`infrastructure-control.prose.md` (the source of truth) and the live system
|
||||
before mutating.
|
||||
|
||||
| Host | IP | Role | Trust |
|
||||
|------|----|------|-------|
|
||||
| CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
|
||||
| VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
|
||||
| CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
|
||||
| Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
|
||||
| CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
|
||||
|
||||
> "Primary host" for the apt phase is the host the ops agent runs maintenance
|
||||
> from. Confirm which host that is against infrastructure-control before
|
||||
> running; do not assume. If the ops agent is containerized/CT-based, apt runs
|
||||
> inside that CT.
|
||||
|
||||
## Requires
|
||||
|
||||
- SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
|
||||
- `apt`, `docker`, `docker compose` available on target hosts
|
||||
- LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
|
||||
- `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
|
||||
- Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
|
||||
- Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
|
||||
|
||||
## Continuity
|
||||
|
||||
- Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
|
||||
- Also wakes on: explicit "infra maintenance" trigger from ops/abiba
|
||||
- Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
|
||||
|
||||
## Execution
|
||||
|
||||
### Phase 0 — Preflight (snapshot/backup check + health baseline)
|
||||
|
||||
1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
|
||||
2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
|
||||
- `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
|
||||
- `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
|
||||
3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
|
||||
4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
|
||||
|
||||
### Phase 1 — OS package updates (primary host)
|
||||
|
||||
```bash
|
||||
# On the primary host only (VERIFY host against infrastructure-control first)
|
||||
apt update
|
||||
apt upgrade -y
|
||||
```
|
||||
|
||||
- Capture `apt list --upgradable` before and after → store in `apt-state`.
|
||||
- If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
|
||||
- If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
|
||||
|
||||
### Phase 2 — Docker image pulls
|
||||
|
||||
Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
|
||||
|
||||
```bash
|
||||
# CT 116 (.116) — inference harness
|
||||
cd /opt/inference-harness && docker compose pull
|
||||
|
||||
# VM 109 (.7) — search + home stacks
|
||||
cd /opt/search-stack/searxng && docker compose pull
|
||||
cd /opt/search-stack/firecrawl-source && docker compose pull
|
||||
# any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
|
||||
```
|
||||
|
||||
- LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
|
||||
- Record pulled tag + digest per service in `image-baseline`.
|
||||
|
||||
### Phase 3 — Container restarts with health verification
|
||||
|
||||
Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
|
||||
|
||||
```bash
|
||||
# CT 116
|
||||
cd /opt/inference-harness && docker compose up -d
|
||||
# VM 109
|
||||
cd /opt/search-stack/searxng && docker compose up -d
|
||||
cd /opt/search-stack/firecrawl-source && docker compose up -d
|
||||
```
|
||||
|
||||
After each stack comes up, wait for health (max 120s):
|
||||
- `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
|
||||
- Service-specific probe passes (see Phase 4 probes)
|
||||
|
||||
If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
|
||||
|
||||
### Phase 4 — Post-update service verification
|
||||
|
||||
After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
|
||||
|
||||
| Service | Probe | Expect |
|
||||
|---------|-------|--------|
|
||||
| LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
|
||||
| LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
|
||||
| SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
|
||||
| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
|
||||
| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
|
||||
| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
|
||||
| Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
|
||||
|
||||
Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
|
||||
|
||||
## Rollback Protocol
|
||||
|
||||
If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
|
||||
|
||||
1. **Image rollback** — for the failing stack, restore the previous image:
|
||||
```bash
|
||||
# Restore from recorded image-baseline digest
|
||||
docker compose down
|
||||
# Pin the service image to the recorded digest in compose, then recreate
|
||||
# image: <name>@sha256:<previous_digest>
|
||||
docker compose pull && docker compose up -d
|
||||
```
|
||||
2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
|
||||
3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
|
||||
4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
|
||||
5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
|
||||
|
||||
## Circuit Breaker
|
||||
|
||||
- `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
|
||||
- `window: 7200` seconds — no more than 2 retries within a 2-hour window.
|
||||
- `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
|
||||
|
||||
## Report
|
||||
|
||||
After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
|
||||
|
||||
```
|
||||
🛠 Infrastructure Maintenance — YYYY-MM-DD
|
||||
|
||||
Phase: apt | images | restarts | verify | rollback
|
||||
Primary host: <host>
|
||||
APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
|
||||
Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
|
||||
Services: all GREEN | <service> FAILED (rolled back)
|
||||
Baseline regression: none | <details>
|
||||
Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
|
||||
Escalation: none | warning | critical | fatal
|
||||
```
|
||||
|
||||
## Verification Postconditions
|
||||
|
||||
- All critical services running after update (Phase 4 all GREEN)
|
||||
- No regressions from pre-update health baseline (Phase 0 baseline)
|
||||
- Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
|
||||
- APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
|
||||
|
||||
## Related Contracts
|
||||
|
||||
- `infrastructure-update.prose.md` — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
|
||||
- `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
|
||||
- `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
|
||||
- `litellm-health.prose.md` — LiteLLM probe details.
|
||||
- `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.
|
||||
@@ -8,18 +8,18 @@ description: >
|
||||
Ensures agents never use the master key directly. Rotation is event-driven,
|
||||
not calendar-driven — rotate only on compromise, personnel change, or
|
||||
periodic security hygiene (quarterly/annually).
|
||||
|
||||
|
||||
UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
|
||||
BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
|
||||
expire/404. The .env fallback prevents agents from running without keys.
|
||||
Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
|
||||
|
||||
|
||||
UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
|
||||
token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
|
||||
infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
|
||||
Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
|
||||
Abiba's key is now a proper agent key (NOT the master key — stale note removed).
|
||||
|
||||
|
||||
UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
|
||||
standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
|
||||
infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
|
||||
@@ -30,7 +30,7 @@ description: >
|
||||
Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
|
||||
— hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
|
||||
(3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
|
||||
|
||||
|
||||
Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
|
||||
Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
|
||||
on CT 116. Last verified: 2026-07-17.
|
||||
@@ -154,7 +154,7 @@ through its agent wrapper.
|
||||
The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
|
||||
then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
|
||||
regeneration** by `hermes gateway install` — the drop-in always wins.
|
||||
|
||||
|
||||
**Why a drop-in instead of editing the unit file:** `hermes gateway install`
|
||||
(called during Hermes updates and some self-heal operations) regenerates the
|
||||
systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
|
||||
@@ -171,7 +171,7 @@ through its agent wrapper.
|
||||
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
|
||||
- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
|
||||
- **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
|
||||
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
|
||||
- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
|
||||
|
||||
### Migration status (2026-07-17)
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ description: >
|
||||
delegation, verification, and delivery. Defines when to delegate, which
|
||||
worker to use for what, how to handle failures, and the kanban board
|
||||
protocol. Enforces context-window discipline and separation of concerns.
|
||||
Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
|
||||
Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
|
||||
version: 1.0.0
|
||||
---
|
||||
|
||||
@@ -19,8 +19,8 @@ version: 1.0.0
|
||||
|
||||
## Topology
|
||||
|
||||
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
|
||||
**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
|
||||
**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
|
||||
**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
|
||||
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
||||
|
||||
This contract is infrastructure-agnostic in terms of which nodes are used.
|
||||
|
||||
@@ -5,7 +5,7 @@ description: >
|
||||
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
|
||||
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
|
||||
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
|
||||
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
|
||||
instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
|
||||
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
|
||||
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
|
||||
agent: abiba
|
||||
@@ -33,8 +33,8 @@ agent: abiba
|
||||
|
||||
| Exporter | Host:Port | Scope | Notes |
|
||||
|----------|-----------|-------|-------|
|
||||
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
|
||||
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
|
||||
| prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
|
||||
| node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
|
||||
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
|
||||
|
||||
## PVE API Token
|
||||
@@ -48,7 +48,7 @@ agent: abiba
|
||||
|
||||
| UID | Title | Panels | Source |
|
||||
|-----|-------|--------|--------|
|
||||
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
|
||||
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
|
||||
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
|
||||
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
|
||||
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
|
||||
@@ -77,9 +77,9 @@ agent: abiba
|
||||
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
|
||||
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
|
||||
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
|
||||
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
|
||||
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
|
||||
|
||||
## Cluster "Tabiri" — 5 Nodes
|
||||
## Cluster "Tabiri" — 6 Nodes
|
||||
|
||||
| Node | IP | Role |
|
||||
|------|----|----|
|
||||
@@ -88,6 +88,7 @@ agent: abiba
|
||||
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
|
||||
| minipve | 192.168.68.12 | PVE |
|
||||
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
|
||||
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
|
||||
|
||||
## Operations
|
||||
|
||||
|
||||
@@ -184,8 +184,8 @@ def check_agents():
|
||||
print(f" ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
|
||||
continue
|
||||
|
||||
# Gateway process
|
||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
|
||||
# Gateway process (exclude the infisical bash wrapper that contains the same string)
|
||||
pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
|
||||
if not pid:
|
||||
print(f" ❌ {name}: GATEWAY NOT RUNNING")
|
||||
FAIL.append(f"gateway-down:{name}")
|
||||
|
||||
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
|
||||
## Requires
|
||||
|
||||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
|
||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
|
||||
- **PM2** on localhost for pi process management
|
||||
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
||||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||||
@@ -199,7 +199,7 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
|
||||
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
||||
```
|
||||
|
||||
Gateway PID should exist with uptime > 60s.
|
||||
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway via PM2 (`pm2 restart mumuni-zulip`). Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
|
||||
|
||||
**B3: Heartbeat Verification**
|
||||
|
||||
|
||||
@@ -409,3 +409,68 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
|
||||
| Queue expiry handling | Crash | Auto re-register |
|
||||
| Busy worker deadlock | Router death | Worker SIGKILL + error DM |
|
||||
| PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |
|
||||
|
||||
---
|
||||
|
||||
## Incident Log — 2026-07-18 Fleet-Wide Audit
|
||||
|
||||
### Fleet State After Audit
|
||||
|
||||
| Agent | Platform | Zulip State | Issues Found | Fix Applied |
|
||||
|-------|----------|-------------|--------------|-------------|
|
||||
| **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
|
||||
| **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
|
||||
| **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |
|
||||
|
||||
### Key Fixes Applied
|
||||
|
||||
**1. Abiba — Credential Fallback (L4 Pattern)**
|
||||
- Root cause: `zulip.api_key` in config.yaml is `""` (expected from Infisical). Infisical vault `ABIBA_ZULIP_API_KEY` wasn't being injected into the process environment.
|
||||
- Fix: Added `.env` file fallback at `/root/.pi/agent/extensions/zulip/.env` with known-working key, sourced before the Infisical `exec`.
|
||||
- Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local `.env` fallback.
|
||||
|
||||
**2. Abiba — Poll Timeout Handling**
|
||||
- Root cause: Zulip long-poll uses `AbortSignal.timeout(65000)`. Zulip's default `event_queue_longpoll_timeout_seconds` can exceed 65s. When the signal fires, an `AbortError` is thrown and caught by the circuit breaker as a failure.
|
||||
- Fix: Caught `AbortError` inside `poll()` and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default.
|
||||
- Reference: [Zulip Events System — long-poll timeout](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html)
|
||||
|
||||
**3. Tanko — Gateway Restart**
|
||||
- Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (`infisical-gateway.sh`) restarts on crash but the gateway wasn't re-establishing Zulip on restart.
|
||||
- Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully.
|
||||
|
||||
### Fleet-Wide Zulip Health Metrics (as of 2026-07-18)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Zulip server | ✅ HTTP 200 |
|
||||
| Agents connected | 3/3 (Abiba, Tanko, Mumuni) |
|
||||
| Abiba circuit breaker | CLOSED (0 failures) |
|
||||
| Abiba uptime | 2D (post-restart) |
|
||||
| Tanko gateway uptime | Ongoing |
|
||||
| Mumuni gateway uptime | Ongoing |
|
||||
| Watchdog status | ✅ Online (2D uptime) |
|
||||
|
||||
### Hermes Agent Zulip Plugin Improvements
|
||||
|
||||
Based on the audit, improvements that should be ported to all Hermes Zulip adapters:
|
||||
|
||||
1. **Circuit breaker pattern** — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff.
|
||||
2. **Credential fallback** — All Hermes agents use Infisical for credentials. Add `.env` local fallback per L4 pattern for `ZULIP_API_KEY`.
|
||||
3. **Queue re-registration** — Handle `BAD_EVENT_QUEUE_ID` with automatic re-registration instead of gateway restart.
|
||||
4. **Supervisor watchdog** — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks.
|
||||
5. **Streaming** — All agents have `streaming: true` in their zulip config. Verify `edit_message()` is implemented in each adapter.
|
||||
|
||||
### Abiba pi Zulip Extension v2 — Implemented Resilience Summary
|
||||
|
||||
| Feature | Status | Notes |
|
||||
|---------|--------|-------|
|
||||
| Circuit breaker | ✅ | CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout |
|
||||
| Retry with jitter | ✅ | 2 attempts, 200ms base, 50-100% jitter |
|
||||
| Queue lifecycle | ✅ | 10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling |
|
||||
| Crash prevention | ✅ | uncaughtException + unhandledRejection recovery |
|
||||
| Worker timeout | ✅ | 5min busy timeout → SIGKILL + error DM |
|
||||
| Health endpoint | ✅ | :9200 with circuit breaker metrics |
|
||||
| Echo prevention | ✅ | Dynamic bot user resolution |
|
||||
| Poll timeout (AbortError) | ✅ v2.1 | Normal timeout returns [] instead of error |
|
||||
| Credential fallback | ✅ v2.1 | .env file before Infisical exec |
|
||||
| Provider auto-fix | ✅ | Detects reasoning_content models, switches to compatible |
|
||||
|
||||
Reference in New Issue
Block a user