Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6d975360d8 | ||
|
|
3e246835d5 |
@@ -128,10 +128,3 @@ safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
|
|||||||
Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
|
Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
|
||||||
It covers the full process: verify → draft → lint → review → ship, with templates
|
It covers the full process: verify → draft → lint → review → ship, with templates
|
||||||
and style rules.
|
and style rules.
|
||||||
|
|
||||||
## Maintaining this file
|
|
||||||
|
|
||||||
Keep this file for knowledge useful to almost every future agent session in this project.
|
|
||||||
Do not repeat what the codebase already shows; point to the authoritative file or command instead.
|
|
||||||
Prefer rewriting or pruning existing entries over appending new ones.
|
|
||||||
When updating this file, preserve this bar for all agents and keep entries concise.
|
|
||||||
|
|||||||
@@ -137,7 +137,7 @@ Encode topology, architectural decisions, and lessons. You read them before plan
|
|||||||
|
|
||||||
| Contract | Description |
|
| Contract | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `infrastructure-control` | Full topology and control pattern: 6-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
|
| `infrastructure-control` | Full topology and control pattern: 5-node Proxmox cluster, 3 Docker ecosystems, NFS storage, network verification, IP-first configuration doctrine. Live-state fields marked `VERIFY-BEFORE-USE`. |
|
||||||
| `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
|
| `pi-approval-architecture` | pi's approval model vs Hermes, available commands, architectural constraints. |
|
||||||
| `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
|
| `zulip-adapter-lessons` | Failure modes, fixes, and patterns from building the pi Zulip extension and Hermes Zulip plugin. |
|
||||||
|
|
||||||
|
|||||||
+2
-104
@@ -22,7 +22,6 @@ owners:
|
|||||||
- abiba
|
- abiba
|
||||||
- mumuni
|
- mumuni
|
||||||
- kwame
|
- kwame
|
||||||
- ops
|
|
||||||
trigger_types:
|
trigger_types:
|
||||||
- scheduled
|
- scheduled
|
||||||
- event_driven
|
- event_driven
|
||||||
@@ -59,7 +58,6 @@ index:
|
|||||||
- memory-audit-maintenance
|
- memory-audit-maintenance
|
||||||
- gpu-fleet
|
- gpu-fleet
|
||||||
- infrastructure-update
|
- infrastructure-update
|
||||||
- infrastructure-maintenance
|
|
||||||
reference:
|
reference:
|
||||||
- infrastructure-control
|
- infrastructure-control
|
||||||
- ra-h-os-custodianship-contract
|
- ra-h-os-custodianship-contract
|
||||||
@@ -92,7 +90,6 @@ index:
|
|||||||
- infrastructure-control
|
- infrastructure-control
|
||||||
- infrastructure-monitoring
|
- infrastructure-monitoring
|
||||||
- infrastructure-update
|
- infrastructure-update
|
||||||
- infrastructure-maintenance
|
|
||||||
- pm2-self-heal
|
- pm2-self-heal
|
||||||
- disk-gc-threat-response
|
- disk-gc-threat-response
|
||||||
gpu:
|
gpu:
|
||||||
@@ -133,6 +130,7 @@ index:
|
|||||||
- build-zulip-plugin
|
- build-zulip-plugin
|
||||||
- stirling-pdf-agent-access
|
- stirling-pdf-agent-access
|
||||||
- gpu-fleet
|
- gpu-fleet
|
||||||
|
- infrastructure-update
|
||||||
- infrastructure-control
|
- infrastructure-control
|
||||||
- zulip-adapter-lessons
|
- zulip-adapter-lessons
|
||||||
- pi-approval-architecture
|
- pi-approval-architecture
|
||||||
@@ -146,9 +144,6 @@ index:
|
|||||||
- mumuni-delegation
|
- mumuni-delegation
|
||||||
kwame:
|
kwame:
|
||||||
- hello-world
|
- hello-world
|
||||||
ops:
|
|
||||||
- infrastructure-maintenance
|
|
||||||
- infrastructure-update
|
|
||||||
by_trigger:
|
by_trigger:
|
||||||
scheduled:
|
scheduled:
|
||||||
- hermes-key-enforcement
|
- hermes-key-enforcement
|
||||||
@@ -161,7 +156,6 @@ index:
|
|||||||
- litellm-health
|
- litellm-health
|
||||||
- memory-audit-maintenance
|
- memory-audit-maintenance
|
||||||
- infrastructure-update
|
- infrastructure-update
|
||||||
- infrastructure-maintenance
|
|
||||||
event_driven:
|
event_driven:
|
||||||
- litellm-self-heal
|
- litellm-self-heal
|
||||||
- pm2-self-heal
|
- pm2-self-heal
|
||||||
@@ -203,7 +197,6 @@ index:
|
|||||||
- hermes-zulip-plugin
|
- hermes-zulip-plugin
|
||||||
- build-zulip-plugin
|
- build-zulip-plugin
|
||||||
- infrastructure-update
|
- infrastructure-update
|
||||||
- infrastructure-maintenance
|
|
||||||
- ra-h-os-custodianship-contract
|
- ra-h-os-custodianship-contract
|
||||||
- mumuni-delegation
|
- mumuni-delegation
|
||||||
normal:
|
normal:
|
||||||
@@ -1263,108 +1256,13 @@ contracts:
|
|||||||
last_run: null
|
last_run: null
|
||||||
last_status: null
|
last_status: null
|
||||||
drift_alerts: []
|
drift_alerts: []
|
||||||
- name: infrastructure-maintenance
|
|
||||||
file: infrastructure-maintenance.prose.md
|
|
||||||
kind: responsibility
|
|
||||||
category: maintenance
|
|
||||||
sensitivity: high
|
|
||||||
status: active
|
|
||||||
owner: ops
|
|
||||||
version: 1.0.0
|
|
||||||
trigger:
|
|
||||||
type: scheduled
|
|
||||||
cadence: 0 2 * * 0
|
|
||||||
description: Weekly host-level maintenance Sunday at 2am ET (replaces infrastructure-update
|
|
||||||
build-phase role; infra-update moves to ops)
|
|
||||||
cron_job_id: null
|
|
||||||
execution:
|
|
||||||
agent: ops
|
|
||||||
timeout: 3600
|
|
||||||
requires:
|
|
||||||
- infrastructure-monitoring run within last 30 minutes (pre-update health baseline)
|
|
||||||
- Proxmox snapshot of primary host OR /tmp backup dir created this run
|
|
||||||
- LiteLLM master key from Infisical vault for health verification
|
|
||||||
protocol:
|
|
||||||
- Load contract from prose-contracts/main
|
|
||||||
- Phase 0 preflight — capture health baseline, backup check, record image baseline
|
|
||||||
- Phase 1 apt update && apt upgrade -y on primary host
|
|
||||||
- Phase 2 docker compose pull for LiteLLM, SearXNG, and other running containers
|
|
||||||
- Phase 3 restart stacks one at a time with per-stack health verification
|
|
||||||
- Phase 4 verify every critical service (LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways)
|
|
||||||
- On failure — rollback per protocol, escalate, do not loop beyond circuit breaker
|
|
||||||
- Log actions to ~/.hermes/runs/infrastructure-maintenance/
|
|
||||||
verification:
|
|
||||||
postconditions:
|
|
||||||
- check: all critical services running after update
|
|
||||||
verify: 'curl -sf http://192.168.68.116/litellm/v1/models && curl -sf https://chat.sysloggh.net/api/v1/server_settings && curl -sf https://git.sysloggh.net/api/v1/version && curl -sf http://192.168.68.7:8888 && pm2 jlist'
|
|
||||||
expect: all probes 200 OK / processes online
|
|
||||||
- check: no regressions from pre-update health baseline
|
|
||||||
verify: diff Phase 0 health-baseline against Phase 4 results
|
|
||||||
expect: no GREEN service turned RED
|
|
||||||
- check: docker containers on latest stable tags
|
|
||||||
verify: docker inspect --format '{{.Config.Image}}' <container> per service matches image-baseline.pulled_tag
|
|
||||||
expect: all containers running pulled tags
|
|
||||||
- check: APT packages up to date with no held broken packages
|
|
||||||
verify: apt list --upgradable 2>/dev/null | wc -l and apt-get -s upgrade | grep -ci broken
|
|
||||||
expect: upgradable == 0, broken == 0
|
|
||||||
artifact: maintenance run report with phase results and any rollback/escalation
|
|
||||||
verify_commands:
|
|
||||||
- curl -sf http://192.168.68.116/litellm/v1/models
|
|
||||||
- curl -sf http://192.168.68.7:8888
|
|
||||||
- curl -sf https://chat.sysloggh.net/api/v1/server_settings
|
|
||||||
- curl -sf https://git.sysloggh.net/api/v1/version
|
|
||||||
- pm2 jlist
|
|
||||||
receipt:
|
|
||||||
format: json
|
|
||||||
storage: ~/.hermes/runs/infrastructure-maintenance/
|
|
||||||
graph_node: true
|
|
||||||
schema:
|
|
||||||
contract: string
|
|
||||||
run_id: string
|
|
||||||
timestamp: ISO 8601
|
|
||||||
agent: string
|
|
||||||
status: pass|fail|escalated
|
|
||||||
phase: preflight|apt|images|restarts|verify|rollback|done|failed
|
|
||||||
actions_taken: array
|
|
||||||
postconditions: array
|
|
||||||
drift_alerts: array
|
|
||||||
evidence_path: string
|
|
||||||
escalation:
|
|
||||||
info:
|
|
||||||
action: log_to_receipt
|
|
||||||
notify: []
|
|
||||||
warning:
|
|
||||||
action: relay_alert
|
|
||||||
notify:
|
|
||||||
- abiba
|
|
||||||
- mumuni
|
|
||||||
critical:
|
|
||||||
action: relay_alert
|
|
||||||
notify:
|
|
||||||
- abiba
|
|
||||||
- mumuni
|
|
||||||
fatal:
|
|
||||||
action: relay_alert + pause + human_required
|
|
||||||
notify:
|
|
||||||
- abiba
|
|
||||||
- mumuni
|
|
||||||
- kwame
|
|
||||||
circuit_breaker:
|
|
||||||
max_retries: 2
|
|
||||||
window: 7200
|
|
||||||
trip_action: escalate_to_fatal
|
|
||||||
depends_on:
|
|
||||||
- infrastructure-monitoring
|
|
||||||
last_run: null
|
|
||||||
last_status: null
|
|
||||||
drift_alerts: []
|
|
||||||
- name: infrastructure-update
|
- name: infrastructure-update
|
||||||
file: infrastructure-update.prose.md
|
file: infrastructure-update.prose.md
|
||||||
kind: responsibility
|
kind: responsibility
|
||||||
category: maintenance
|
category: maintenance
|
||||||
sensitivity: high
|
sensitivity: high
|
||||||
status: active
|
status: active
|
||||||
owner: ops
|
owner: abiba
|
||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
trigger:
|
trigger:
|
||||||
type: scheduled
|
type: scheduled
|
||||||
|
|||||||
@@ -286,7 +286,7 @@ one-off GPU builds. No automated post-migration cleanup was in place.
|
|||||||
| 111 | tdunna | amdpve | ✅ reachable |
|
| 111 | tdunna | amdpve | ✅ reachable |
|
||||||
| 112 | tanko | amdpve | ✅ reachable |
|
| 112 | tanko | amdpve | ✅ reachable |
|
||||||
| 113 | baggy | amdpve | ✅ reachable |
|
| 113 | baggy | amdpve | ✅ reachable |
|
||||||
| 114 | mumuni | hwepve | ✅ reachable |
|
| 114 | mumuni | minipve | ✅ reachable |
|
||||||
| 115 | scottdenya | amdpve | ✅ reachable |
|
| 115 | scottdenya | amdpve | ✅ reachable |
|
||||||
| 116 | syslog-api | minipve | ✅ reachable |
|
| 116 | syslog-api | minipve | ✅ reachable |
|
||||||
| 117 | zulip | storepve | ✅ reachable |
|
| 117 | zulip | storepve | ✅ reachable |
|
||||||
|
|||||||
@@ -185,7 +185,7 @@ what, and why should I care?
|
|||||||
|
|
||||||
```
|
```
|
||||||
❌ "Monitors infrastructure health"
|
❌ "Monitors infrastructure health"
|
||||||
✅ "Scans all 6 Proxmox nodes and 19 CTs for disk pressure, checks Docker
|
✅ "Scans all 5 Proxmox nodes and 19 CTs for disk pressure, checks Docker
|
||||||
container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
|
container health on .7/.116/.17, alerts via Telegram DM on RED/CRITICAL"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
+7
-7
@@ -125,7 +125,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
|
|||||||
|
|
||||||
| Alias | RPM Cap | Routes To | Purpose |
|
| Alias | RPM Cap | Routes To | Purpose |
|
||||||
|-------|---------|-----------|---------|
|
|-------|---------|-----------|---------|
|
||||||
| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
|
| `strix-moe` | 40 | Strix Halo | Agent reasoning, compression (~30% via syslog-auto pool) (MoE models) |
|
||||||
| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
|
| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
|
||||||
| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
|
| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
|
||||||
|
|
||||||
@@ -136,7 +136,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
|
|||||||
- syslog-auto → qwen → gemma → qwen3.6-35B-udq4
|
- syslog-auto → qwen → gemma → qwen3.6-35B-udq4
|
||||||
|
|
||||||
### Why Strix Halo RPM Is Capped
|
### Why Strix Halo RPM Is Capped
|
||||||
- Direct (strix-moe): 40 RPM (tight) — Strix Halo is shared with compression tasks
|
- Direct (strix-moe): 40 RPM (tight) — Strix Halo handles agent reasoning + compression via syslog-auto pool
|
||||||
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
|
- Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
|
||||||
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
|
- Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
|
||||||
|
|
||||||
@@ -284,7 +284,7 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.
|
|||||||
### Stable Aliases — CRITICAL
|
### Stable Aliases — CRITICAL
|
||||||
|
|
||||||
All agent configs MUST use stable role-based aliases, never model-specific names:
|
All agent configs MUST use stable role-based aliases, never model-specific names:
|
||||||
- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`)
|
- `compression.model: syslog-auto` — switched from strix-moe 2026-07-18 to distribute across weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070); relieves Strix Halo thermal pressure
|
||||||
- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
|
- `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
|
||||||
- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
|
- `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
|
||||||
- `auxiliary.web_extract.model: gpu-light`
|
- `auxiliary.web_extract.model: gpu-light`
|
||||||
@@ -295,7 +295,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
|
|||||||
- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
|
- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
|
||||||
- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
|
- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
|
||||||
- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
|
- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
|
||||||
- Mumuni compression model alias: `strix-moe` with 300s timeout
|
- Mumuni compression model alias: `syslog-auto` (switched from strix-moe 2026-07-18) with 300s timeout
|
||||||
|
|
||||||
### Mumuni Agent Profile
|
### Mumuni Agent Profile
|
||||||
|
|
||||||
@@ -305,8 +305,8 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
|
|||||||
|---------|-------|-------|
|
|---------|-------|-------|
|
||||||
| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
|
| `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
|
||||||
| `model.provider` | `custom:litellm` | LiteLLM on CT116 |
|
| `model.provider` | `custom:litellm` | LiteLLM on CT116 |
|
||||||
| `compression.model` | `strix-moe` | Stable alias — survives model swaps |
|
| `compression.model` | `syslog-auto` | Switched from strix-moe 2026-07-18 — distributes across weighted pool |
|
||||||
| `aux.compression.model` | `strix-moe` | Compression auxiliary model |
|
| `aux.compression.model` | `syslog-auto` | Compression auxiliary — switched from strix-moe 2026-07-18 |
|
||||||
| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
|
| `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
|
||||||
| `aux.web_extract.model` | `gpu-light` | Web extraction |
|
| `aux.web_extract.model` | `gpu-light` | Web extraction |
|
||||||
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
|
| `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
|
||||||
@@ -318,7 +318,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
|
|||||||
| `personalities` | `creative` | Creative assistant personality |
|
| `personalities` | `creative` | Creative assistant personality |
|
||||||
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
| Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
|
||||||
| Main model timeout | 300s | LiteLLM global timeout |
|
| Main model timeout | 300s | LiteLLM global timeout |
|
||||||
| Compression model timeout | 300s | strix-moe timeout increased from 120s |
|
| Compression model timeout | 300s | syslog-auto timeout (switched from strix-moe 2026-07-18) |
|
||||||
|
|
||||||
### Agent Update Status (2026-07-15)
|
### Agent Update Status (2026-07-15)
|
||||||
|
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ depends_on:
|
|||||||
|-------|-----|------|-------|------|-----|-------|------|
|
|-------|-----|------|-------|------|-----|-------|------|
|
||||||
| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
|
| `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
|
||||||
| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
|
| `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
|
||||||
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
|
| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Agent reasoning, compression (~30% via syslog-auto pool), summarization, long docs |
|
||||||
|
|
||||||
Key notes:
|
Key notes:
|
||||||
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
|
- All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
|
||||||
@@ -157,7 +157,7 @@ Key notes:
|
|||||||
- **Target distribution**:
|
- **Target distribution**:
|
||||||
- RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
|
- RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
|
||||||
- RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
|
- RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
|
||||||
- Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
|
- Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Agent reasoning, compression (~30% via syslog-auto pool), summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
|
||||||
- **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
|
- **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
|
||||||
- **Fix**:
|
- **Fix**:
|
||||||
- Alert if any GPU is handling workload outside its designated role
|
- Alert if any GPU is handling workload outside its designated role
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ version: 1.0.0
|
|||||||
description: >
|
description: >
|
||||||
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
|
||||||
configuration state, keys, workarounds, and audit procedure. When an agent's
|
configuration state, keys, workarounds, and audit procedure. When an agent's
|
||||||
configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
|
configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 128K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (compression via syslog-auto pool — switched from strix-moe 2026-07-18).
|
||||||
author: Abiba (pi agent)
|
author: Abiba (pi agent)
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -25,7 +25,7 @@ done
|
|||||||
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|
| Agent | CT | Node | IP | LiteLLM Alias | Key Source | Platform |
|
||||||
|-------|-----|------|-----|---------------|------------|----------|
|
|-------|-----|------|-----|---------------|------------|----------|
|
||||||
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
| Tanko | 112 | amdpve | .122 | `tanko` | Infisical vault | Hermes |
|
||||||
| Mumuni | 114 | hwepve | .123 | `mumuni` | Infisical vault | Hermes |
|
| Mumuni | 114 | minipve | .123 | `mumuni` | Infisical vault | Hermes |
|
||||||
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
| Koby | 129 | amdpve | srv1079750 | `koby` | Infisical vault | **Hermes** |
|
||||||
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
| Koonimo | 114 | amdpve | ? | `koonimo` | Infisical vault | Hermes |
|
||||||
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
| Shumba | — | 192.168.68.119 | N/A | N/A (DeepSeek) | Hermes (RETIRED — CT119 now Infisical vault) |
|
||||||
|
|||||||
@@ -272,7 +272,7 @@ The following MUST be identical across ALL profiles:
|
|||||||
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
|
### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
|
||||||
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
|
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
|
||||||
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
|
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
|
||||||
- **Strix Halo (64GB, 128K ctx, Geneis Hermes V3 APEX)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
|
- **Strix Halo (64GB, 128K ctx, Genesis Hermes V3 APEX)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
|
||||||
- Agent profiles MUST route auxiliary tasks to the correct GPU:
|
- Agent profiles MUST route auxiliary tasks to the correct GPU:
|
||||||
- `auxiliary.vision.model: gpu-light` (RTX 5070)
|
- `auxiliary.vision.model: gpu-light` (RTX 5070)
|
||||||
- `auxiliary.web_extract.model: gpu-light` (RTX 5070)
|
- `auxiliary.web_extract.model: gpu-light` (RTX 5070)
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ gateway restart, and connection validation.
|
|||||||
|
|
||||||
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
| Host | CT | Proxmox | IP (direct) | Hermes Home | User |
|
||||||
|------|-----|---------|-------------|-------------|------|
|
|------|-----|---------|-------------|-------------|------|
|
||||||
| Mumuni | CT114 | hwepve | 192.168.68.123 | /root/.hermes | root |
|
| Mumuni | CT114 | — | 192.168.68.123 | /root/.hermes | root |
|
||||||
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
| Tanko | CT112 | amdpve | 192.168.68.122 | /home/jerome/.hermes | jerome |
|
||||||
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
| Koby | CT111 | amdpve | 192.168.68.129 | /root/.hermes | root |
|
||||||
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
| Shumba | — | — | 192.168.68.119 | /home/lucky/.hermes | lucky |
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ duration.
|
|||||||
prefill time at 532 tok/s. Fix context first, routing second.
|
prefill time at 532 tok/s. Fix context first, routing second.
|
||||||
|
|
||||||
- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
|
- **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
|
||||||
queries; gemma for compression/auxiliary. Never send simple completion to a
|
queries; gemma for vision/web extraction; syslog-auto for compression (via weighted pool). Never send simple completion to a
|
||||||
35B MoE.
|
35B MoE.
|
||||||
- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
|
- **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
|
||||||
compact at 102K, not 166K. Target 15% tail (not 30%).
|
compact at 102K, not 166K. Target 15% tail (not 30%).
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ kind: pattern
|
|||||||
name: infrastructure-control
|
name: infrastructure-control
|
||||||
description: >
|
description: >
|
||||||
Full infrastructure monitoring and control pattern covering the
|
Full infrastructure monitoring and control pattern covering the
|
||||||
6-node Proxmox cluster, 3 Docker ecosystems (22 containers),
|
5-node Proxmox cluster, 3 Docker ecosystems (22 containers),
|
||||||
NFS storage, and network services. Defines monitors, remediations,
|
NFS storage, and network services. Defines monitors, remediations,
|
||||||
and the access matrix for all environments.
|
and the access matrix for all environments.
|
||||||
|
|
||||||
@@ -42,8 +42,8 @@ description: >
|
|||||||
│ │ │ │ │
|
│ │ │ │ │
|
||||||
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
|
┌────┘ ┌────┘ ┌────┘ ┌────┘ ┌────┘
|
||||||
▼ ▼ ▼ ▼ ▼
|
▼ ▼ ▼ ▼ ▼
|
||||||
minipve amdpve storepve acerpve ocupve hwepve
|
minipve amdpve storepve acerpve ocupve
|
||||||
(.12) (.15) (.6) (.9) (.5) (.4)
|
(.12) (.15) (.6) (.9) (.5)
|
||||||
|
|
||||||
▼
|
▼
|
||||||
┌─────────────────────────────────────────────┐
|
┌─────────────────────────────────────────────┐
|
||||||
@@ -95,16 +95,15 @@ description: >
|
|||||||
|
|
||||||
## Section 2: Proxmox Cluster — Monitoring
|
## Section 2: Proxmox Cluster — Monitoring
|
||||||
|
|
||||||
### Nodes (6)
|
### Nodes (5)
|
||||||
|
|
||||||
| Node | IP | CPU | RAM | VMs/CTs | Role |
|
| Node | IP | CPU | RAM | VMs/CTs | Role |
|
||||||
|------|----|-----|-----|---------|------|
|
|------|----|-----|-----|---------|------|
|
||||||
| minipve | .12 | 16C | 30GB | authentik, gitea, syslog-api, jitsi | Auth, git, messaging |
|
| minipve | .12 | 16C | 30GB | authentik, gitea, mumuni, syslog-api, jitsi | Auth, git, messaging |
|
||||||
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
|
| amdpve | .15 | 32C | 62GB | abiba, kagentz, tanko, tdunna, baggy, scottdenya | Agents, compute |
|
||||||
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
|
| storepve | .6 | 28C | 31GB | docker-vm, ra-h-os, PBS, media, zulip | Docker, storage, chat |
|
||||||
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
|
| acerpve | .9 | 28C | 31GB | llm-gpu, adguard | GPU VMs |
|
||||||
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
| ocupve | .5 | 12C | 14GB | ocu-llm | GPU VMs |
|
||||||
| hwepve | .4 | 12C | 15GB | mumuni | Huawei Matebook 16, agent host |
|
|
||||||
|
|
||||||
### Checks (every 5 min)
|
### Checks (every 5 min)
|
||||||
|
|
||||||
@@ -118,7 +117,7 @@ description: >
|
|||||||
|
|
||||||
## Checks
|
## Checks
|
||||||
|
|
||||||
- For each node in [minipve, amdpve, storepve, acerpve, ocupve, hwepve]:
|
- For each node in [minipve, amdpve, storepve, acerpve, ocupve]:
|
||||||
- GET /api2/json/nodes/{node}/status → check status == "online"
|
- GET /api2/json/nodes/{node}/status → check status == "online"
|
||||||
- GET /api2/json/nodes/{node}/status → cpu < 0.80
|
- GET /api2/json/nodes/{node}/status → cpu < 0.80
|
||||||
- GET /api2/json/nodes/{node}/status → free_mem > 10%
|
- GET /api2/json/nodes/{node}/status → free_mem > 10%
|
||||||
@@ -590,7 +589,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
|
|||||||
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
|
| 111 | tdunna | amdpve | .129 | Hermes agent | ✅ |
|
||||||
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
| 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
|
||||||
| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
|
| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
|
||||||
| 114 | mumuni | hwepve | .123 | Hermes agent | ✅ |
|
| 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
|
||||||
| 115 | scottdenya | amdpve | — | ? | ❌ |
|
| 115 | scottdenya | amdpve | — | ? | ❌ |
|
||||||
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
| 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
|
||||||
| 117 | zulip | storepve | — | Chat | ❌ |
|
| 117 | zulip | storepve | — | Chat | ❌ |
|
||||||
@@ -622,7 +621,7 @@ Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.
|
|||||||
| 115 | scottdenya | amdpve | `pct-run 115` |
|
| 115 | scottdenya | amdpve | `pct-run 115` |
|
||||||
| 104 | authentik | minipve | `pct-run 104` |
|
| 104 | authentik | minipve | `pct-run 104` |
|
||||||
| 110 | gitea | minipve | `pct-run 110` |
|
| 110 | gitea | minipve | `pct-run 110` |
|
||||||
| 114 | mumuni | hwepve | `pct-run 114` |
|
| 114 | mumuni | minipve | `pct-run 114` |
|
||||||
| 116 | syslog-api | minipve | `pct-run 116` |
|
| 116 | syslog-api | minipve | `pct-run 116` |
|
||||||
| 106 | ra-h-os | storepve | `pct-run 106` |
|
| 106 | ra-h-os | storepve | `pct-run 106` |
|
||||||
| 107 | proxmox-backup | storepve | `pct-run 107` |
|
| 107 | proxmox-backup | storepve | `pct-run 107` |
|
||||||
|
|||||||
@@ -1,200 +0,0 @@
|
|||||||
---
|
|
||||||
kind: responsibility
|
|
||||||
name: infrastructure-maintenance
|
|
||||||
description: >
|
|
||||||
Weekly system-level maintenance for the Syslog inference fleet: OS package
|
|
||||||
updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
|
|
||||||
running containers, container restarts with health verification, post-update
|
|
||||||
verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
|
|
||||||
PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
|
|
||||||
raw shell scripts that previously did this piecemeal. This contract owns the
|
|
||||||
HOST-LEVEL weekly maintenance loop on the primary host plus Docker image
|
|
||||||
pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET
|
|
||||||
cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker
|
|
||||||
image Wave 3 across all stacks). Runs Sunday 2am ET. Owner:
|
|
||||||
ops (firstmate secondmate). Blast radius: an unverified image pull can break
|
|
||||||
LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
|
|
||||||
upgrade can leave the host in a half-upgraded state. Pre-update backup check
|
|
||||||
and rollback are mandatory for this reason.
|
|
||||||
agent: ops
|
|
||||||
triggers:
|
|
||||||
- weekly (Sunday 02:00 ET) via cron
|
|
||||||
- on demand when ops/abiba triggers "infra maintenance"
|
|
||||||
version: 1.0.0
|
|
||||||
---
|
|
||||||
|
|
||||||
## Maintains
|
|
||||||
|
|
||||||
- maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
|
|
||||||
- image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
|
|
||||||
- apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
|
|
||||||
- health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
|
|
||||||
- rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
|
|
||||||
- maintenance-history: array of past runs with phase results and any escalations
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
Primary host is the maintenance host where apt updates apply. Docker image pulls
|
|
||||||
span the two Docker ecosystems that run critical services. infrastructure-update
|
|
||||||
runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across
|
|
||||||
all stacks/hosts); this contract runs a narrower host-level weekly pull limited
|
|
||||||
to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against
|
|
||||||
`infrastructure-control.prose.md` (the source of truth) and the live system
|
|
||||||
before mutating.
|
|
||||||
|
|
||||||
| Host | IP | Role | Trust |
|
|
||||||
|------|----|------|-------|
|
|
||||||
| CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
|
|
||||||
| VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
|
|
||||||
| CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
|
|
||||||
| Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
|
|
||||||
| CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
|
|
||||||
|
|
||||||
> "Primary host" for the apt phase is the host the ops agent runs maintenance
|
|
||||||
> from. Confirm which host that is against infrastructure-control before
|
|
||||||
> running; do not assume. If the ops agent is containerized/CT-based, apt runs
|
|
||||||
> inside that CT.
|
|
||||||
|
|
||||||
## Requires
|
|
||||||
|
|
||||||
- SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
|
|
||||||
- `apt`, `docker`, `docker compose` available on target hosts
|
|
||||||
- LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
|
|
||||||
- `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
|
|
||||||
- Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
|
|
||||||
- Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
|
|
||||||
|
|
||||||
## Continuity
|
|
||||||
|
|
||||||
- Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
|
|
||||||
- Also wakes on: explicit "infra maintenance" trigger from ops/abiba
|
|
||||||
- Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
|
|
||||||
|
|
||||||
## Execution
|
|
||||||
|
|
||||||
### Phase 0 — Preflight (snapshot/backup check + health baseline)
|
|
||||||
|
|
||||||
1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
|
|
||||||
2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
|
|
||||||
- `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
|
|
||||||
- `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
|
|
||||||
3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
|
|
||||||
4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
|
|
||||||
|
|
||||||
### Phase 1 — OS package updates (primary host)
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# On the primary host only (VERIFY host against infrastructure-control first)
|
|
||||||
apt update
|
|
||||||
apt upgrade -y
|
|
||||||
```
|
|
||||||
|
|
||||||
- Capture `apt list --upgradable` before and after → store in `apt-state`.
|
|
||||||
- If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
|
|
||||||
- If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
|
|
||||||
|
|
||||||
### Phase 2 — Docker image pulls
|
|
||||||
|
|
||||||
Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# CT 116 (.116) — inference harness
|
|
||||||
cd /opt/inference-harness && docker compose pull
|
|
||||||
|
|
||||||
# VM 109 (.7) — search + home stacks
|
|
||||||
cd /opt/search-stack/searxng && docker compose pull
|
|
||||||
cd /opt/search-stack/firecrawl-source && docker compose pull
|
|
||||||
# any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
|
|
||||||
```
|
|
||||||
|
|
||||||
- LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
|
|
||||||
- Record pulled tag + digest per service in `image-baseline`.
|
|
||||||
|
|
||||||
### Phase 3 — Container restarts with health verification
|
|
||||||
|
|
||||||
Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# CT 116
|
|
||||||
cd /opt/inference-harness && docker compose up -d
|
|
||||||
# VM 109
|
|
||||||
cd /opt/search-stack/searxng && docker compose up -d
|
|
||||||
cd /opt/search-stack/firecrawl-source && docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
After each stack comes up, wait for health (max 120s):
|
|
||||||
- `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
|
|
||||||
- Service-specific probe passes (see Phase 4 probes)
|
|
||||||
|
|
||||||
If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
|
|
||||||
|
|
||||||
### Phase 4 — Post-update service verification
|
|
||||||
|
|
||||||
After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
|
|
||||||
|
|
||||||
| Service | Probe | Expect |
|
|
||||||
|---------|-------|--------|
|
|
||||||
| LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
|
|
||||||
| LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
|
|
||||||
| SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
|
|
||||||
| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
|
|
||||||
| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
|
|
||||||
| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
|
|
||||||
| Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
|
|
||||||
|
|
||||||
Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
|
|
||||||
|
|
||||||
## Rollback Protocol
|
|
||||||
|
|
||||||
If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
|
|
||||||
|
|
||||||
1. **Image rollback** — for the failing stack, restore the previous image:
|
|
||||||
```bash
|
|
||||||
# Restore from recorded image-baseline digest
|
|
||||||
docker compose down
|
|
||||||
# Pin the service image to the recorded digest in compose, then recreate
|
|
||||||
# image: <name>@sha256:<previous_digest>
|
|
||||||
docker compose pull && docker compose up -d
|
|
||||||
```
|
|
||||||
2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
|
|
||||||
3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
|
|
||||||
4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
|
|
||||||
5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
|
|
||||||
|
|
||||||
## Circuit Breaker
|
|
||||||
|
|
||||||
- `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
|
|
||||||
- `window: 7200` seconds — no more than 2 retries within a 2-hour window.
|
|
||||||
- `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
|
|
||||||
|
|
||||||
## Report
|
|
||||||
|
|
||||||
After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
|
|
||||||
|
|
||||||
```
|
|
||||||
🛠 Infrastructure Maintenance — YYYY-MM-DD
|
|
||||||
|
|
||||||
Phase: apt | images | restarts | verify | rollback
|
|
||||||
Primary host: <host>
|
|
||||||
APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
|
|
||||||
Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
|
|
||||||
Services: all GREEN | <service> FAILED (rolled back)
|
|
||||||
Baseline regression: none | <details>
|
|
||||||
Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
|
|
||||||
Escalation: none | warning | critical | fatal
|
|
||||||
```
|
|
||||||
|
|
||||||
## Verification Postconditions
|
|
||||||
|
|
||||||
- All critical services running after update (Phase 4 all GREEN)
|
|
||||||
- No regressions from pre-update health baseline (Phase 0 baseline)
|
|
||||||
- Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
|
|
||||||
- APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
|
|
||||||
|
|
||||||
## Related Contracts
|
|
||||||
|
|
||||||
- `infrastructure-update.prose.md` — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
|
|
||||||
- `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
|
|
||||||
- `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
|
|
||||||
- `litellm-health.prose.md` — LiteLLM probe details.
|
|
||||||
- `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.
|
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
kind: responsibility
|
kind: responsibility
|
||||||
name: infrastructure-update
|
name: infrastructure-update
|
||||||
description: >
|
description: >
|
||||||
Autonomous system-wide update contract covering all 6 Proxmox nodes,
|
Autonomous system-wide update contract covering all 5 Proxmox nodes,
|
||||||
15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
|
15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
|
||||||
Docker images, and container stacks in safe waves with health checks
|
Docker images, and container stacks in safe waves with health checks
|
||||||
and automatic rollback on failure.
|
and automatic rollback on failure.
|
||||||
@@ -56,11 +56,10 @@ Before ANY update wave:
|
|||||||
| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
| amdpve (.15) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||||
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
| acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||||
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
| ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
||||||
| hwepve (.4) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
|
|
||||||
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
|
| CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
|
||||||
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
|
| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
|
||||||
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
|
| CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
|
||||||
| CT 114 (mumuni, hwepve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
| CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
|
||||||
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
|
| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
|
||||||
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
|
| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |
|
||||||
|
|
||||||
@@ -184,7 +183,7 @@ When LiteLLM is upgraded to a version supporting per-key MCP grants:
|
|||||||
|
|
||||||
## Success Criteria
|
## Success Criteria
|
||||||
|
|
||||||
- [ ] All 6 PVE nodes updated, no reboot-loop
|
- [ ] All 5 PVE nodes updated, no reboot-loop
|
||||||
- [ ] All VMs/CTs running post-update
|
- [ ] All VMs/CTs running post-update
|
||||||
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
|
- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
|
||||||
- [ ] LiteLLM inference passing (syslog-auto test)
|
- [ ] LiteLLM inference passing (syslog-auto test)
|
||||||
@@ -200,7 +199,7 @@ After completion, send Zulip DM:
|
|||||||
```
|
```
|
||||||
📋 Infrastructure Update — YYYY-MM-DD
|
📋 Infrastructure Update — YYYY-MM-DD
|
||||||
|
|
||||||
Updated: 6 PVE nodes, 12 CTs/VMs, 30+ containers
|
Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
|
||||||
Security fixes: N CVEs patched
|
Security fixes: N CVEs patched
|
||||||
Downtime: <service> <duration>
|
Downtime: <service> <duration>
|
||||||
Failures: none / <details>
|
Failures: none / <details>
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ description: >
|
|||||||
delegation, verification, and delivery. Defines when to delegate, which
|
delegation, verification, and delivery. Defines when to delegate, which
|
||||||
worker to use for what, how to handle failures, and the kanban board
|
worker to use for what, how to handle failures, and the kanban board
|
||||||
protocol. Enforces context-window discipline and separation of concerns.
|
protocol. Enforces context-window discipline and separation of concerns.
|
||||||
Runs on Mumuni (lxc/114, hwepve, .123) via Hermes agent.
|
Runs on Mumuni (CT 118, storepve, .6) via Hermes agent.
|
||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -19,8 +19,8 @@ version: 1.0.0
|
|||||||
|
|
||||||
## Topology
|
## Topology
|
||||||
|
|
||||||
**Cluster:** 6 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve, hwepve)
|
**Cluster:** 5 Proxmox nodes (ocupve, acerpve, minipve, amdpve, storepve)
|
||||||
**Manager:** Mumuni (lxc/114, hwepve, .123) via Hermes agent
|
**Manager:** Mumuni (CT 118, storepve, .6) via Hermes agent
|
||||||
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
**Workers:** 6 profiles, all running on the same agent — no separate hosts needed
|
||||||
|
|
||||||
This contract is infrastructure-agnostic in terms of which nodes are used.
|
This contract is infrastructure-agnostic in terms of which nodes are used.
|
||||||
@@ -31,7 +31,7 @@ Workers execute tasks on whatever infrastructure they're given — SSH to .6,
|
|||||||
## Why This Matters
|
## Why This Matters
|
||||||
|
|
||||||
Without enforced delegation, the manager consumes the full iteration budget
|
Without enforced delegation, the manager consumes the full iteration budget
|
||||||
(60 calls) on single-turn tasks — SSH to 6 nodes, check each VM, read logs —
|
(60 calls) on single-turn tasks — SSH to 5 nodes, check each VM, read logs —
|
||||||
leaving no capacity for actual coordination. The result: context overflow
|
leaving no capacity for actual coordination. The result: context overflow
|
||||||
(59K tokens in system prompt), iteration exhaustion, and degraded response
|
(59K tokens in system prompt), iteration exhaustion, and degraded response
|
||||||
quality. This contract exists because I blew through my budget checking
|
quality. This contract exists because I blew through my budget checking
|
||||||
@@ -82,7 +82,7 @@ it asks the manager (via relay) — it doesn't go find it on its own.
|
|||||||
|
|
||||||
**This is a hard rule, not a recommendation.** Violating it produces the exact
|
**This is a hard rule, not a recommendation.** Violating it produces the exact
|
||||||
type of discrepancy the kanban pipeline exists to prevent: a review worker finds
|
type of discrepancy the kanban pipeline exists to prevent: a review worker finds
|
||||||
"6 nodes present" in the raw data but "5/6 online" in the report — even though
|
"5 nodes present" in the raw data but "5/5 online" in the report — even though
|
||||||
one of those nodes was unreachable. The report lied because it used data the
|
one of those nodes was unreachable. The report lied because it used data the
|
||||||
raw data never provided.
|
raw data never provided.
|
||||||
|
|
||||||
@@ -137,7 +137,7 @@ delegate_task(
|
|||||||
```
|
```
|
||||||
delegate_task(
|
delegate_task(
|
||||||
tasks=[
|
tasks=[
|
||||||
{"goal": "Check all 6 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
|
{"goal": "Check all 5 Proxmox nodes for VM status", "context": "SSH to each node via 192.168.68.x, run 'qm list'"},
|
||||||
{"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
|
{"goal": "Check Docker container health on .7/.116/.17", "context": "SSH to each host, check container status"},
|
||||||
]
|
]
|
||||||
)
|
)
|
||||||
@@ -191,7 +191,7 @@ Only verified results reach Kwame. Format per channel:
|
|||||||
{
|
{
|
||||||
"lane_id": "devops-check",
|
"lane_id": "devops-check",
|
||||||
"worker": "syslog-devops",
|
"worker": "syslog-devops",
|
||||||
"goal": "Check all 6 Proxmox nodes",
|
"goal": "Check all 5 Proxmox nodes",
|
||||||
"status": "dispatched|completed|failed",
|
"status": "dispatched|completed|failed",
|
||||||
"output_file": "/tmp/node-report.md"
|
"output_file": "/tmp/node-report.md"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ description: >
|
|||||||
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
|
Proxmox cluster + Docker monitoring via the existing Grafana/Prometheus stack
|
||||||
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
|
on CT 116. Replaces Pulse with file-provisioned Grafana dashboards. Three
|
||||||
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
|
exporters feed Prometheus: prometheus-pve-exporter (cluster-aware, single
|
||||||
instance), node_exporter (all 6 PVE nodes), and a custom docker-stats-exporter
|
instance), node_exporter (all 5 PVE nodes), and a custom docker-stats-exporter
|
||||||
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
|
(Docker 29 / containerd image-store compatible, since cAdvisor cannot resolve
|
||||||
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
|
the layerdb). Dashboards exposed at http://192.168.68.116:3001/ (direct LAN, not behind nginx).
|
||||||
agent: abiba
|
agent: abiba
|
||||||
@@ -20,7 +20,7 @@ agent: abiba
|
|||||||
└───────┬──────────────┬──────────────┬───────────────────────┘
|
└───────┬──────────────┬──────────────┬───────────────────────┘
|
||||||
│ │ │
|
│ │ │
|
||||||
▼ ▼ ▼
|
▼ ▼ ▼
|
||||||
pve-exporter docker-stats (scrapes 6x node_exporter)
|
pve-exporter docker-stats (scrapes 5x node_exporter)
|
||||||
:9221 :9324
|
:9221 :9324
|
||||||
│ │
|
│ │
|
||||||
▼ ▼
|
▼ ▼
|
||||||
@@ -33,8 +33,8 @@ agent: abiba
|
|||||||
|
|
||||||
| Exporter | Host:Port | Scope | Notes |
|
| Exporter | Host:Port | Scope | Notes |
|
||||||
|----------|-----------|-------|-------|
|
|----------|-----------|-------|-------|
|
||||||
| prometheus-pve-exporter | .116:9221 (container) | All 6 nodes + guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
|
| prometheus-pve-exporter | .116:9221 (container) | All 5 nodes + 14 guests + 36 storage pools | Single instance, cluster-aware via amdpve API. Config `/opt/monitoring/pve.yml` (token `monitoring@pve!prometheus`, PVEAuditor role). Metric schema is label-based (`id=node/amdpve`, `id=lxc/100`). |
|
||||||
| node_exporter | .5/.6/.9/.12/.15/.4:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 6 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. hwepve (.4) added 2026-07-19. |
|
| node_exporter | .5/.6/.9/.12/.15:9100 (systemd) | Per-node CPU/mem/disk/net/temp | Installed via apt on all 5 PVE nodes, enabled (reboot-persistent). Collectors: textfile, systemd, tcpstat, ethtool. |
|
||||||
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
|
| docker-stats-exporter | .116:9324 (container) | 10 Docker containers on .116 | **Custom** (cAdvisor v0.51 incompatible with Docker 29 containerd image store — layerdb gone). Uses Docker Engine API over unix socket. Script `/opt/monitoring/docker-stats-exporter.py`. |
|
||||||
|
|
||||||
## PVE API Token
|
## PVE API Token
|
||||||
@@ -48,7 +48,7 @@ agent: abiba
|
|||||||
|
|
||||||
| UID | Title | Panels | Source |
|
| UID | Title | Panels | Source |
|
||||||
|-----|-------|--------|--------|
|
|-----|-------|--------|--------|
|
||||||
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 6-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
|
| proxmox-cluster | Proxmox Cluster Overview | 16 | cluster status, 5-node CPU/mem/disk/load gauges, guests table, storage pools, guest CPU/mem timeseries |
|
||||||
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
|
| proxmox-node | Proxmox Node Detail | 13 | per-node CPU per-core, memory, network, disk IO/IOPS/latency, temperature, disk space (variable: $node) |
|
||||||
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
|
| docker-containers | Docker Containers | 10 | per-container CPU/mem/network, restarts, memory limit ratio (variable: $container) |
|
||||||
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
|
| gpu-fleet | GPU Fleet | 7 | (existing, preserved in DB, not provisioned) |
|
||||||
@@ -71,15 +71,15 @@ agent: abiba
|
|||||||
| File | Host | Purpose |
|
| File | Host | Purpose |
|
||||||
|------|------|---------|
|
|------|------|---------|
|
||||||
| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
|
| `/opt/monitoring/docker-compose.yml` | .116 | monitoring stack (prometheus, grafana, pve-exporter, docker-stats) |
|
||||||
| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x6, docker-stats) |
|
| `/opt/monitoring/prometheus.yml` | .116 | 6 scrape jobs (3 GPU, pve, node x5, docker-stats) |
|
||||||
| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
|
| `/opt/monitoring/pve.yml` | .116 | PVE API credentials (chmod 644, contains token) |
|
||||||
| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
|
| `/opt/monitoring/docker-stats-exporter.py` | .116 | custom Docker metrics exporter |
|
||||||
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
|
| `/opt/monitoring/grafana/dashboards/build-dashboards.py` | .116 | dashboard JSON generator |
|
||||||
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
|
| `/opt/monitoring/grafana/dashboards/json/*.json` | .116 | provisioned dashboard definitions |
|
||||||
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
|
| `/opt/monitoring/grafana/datasources/prometheus.yml` | .116 | datasource provisioning |
|
||||||
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15/.4 | node_exporter collector config |
|
| `/etc/default/prometheus-node-exporter` | .5/.6/.9/.12/.15 | node_exporter collector config |
|
||||||
|
|
||||||
## Cluster "Tabiri" — 6 Nodes
|
## Cluster "Tabiri" — 5 Nodes
|
||||||
|
|
||||||
| Node | IP | Role |
|
| Node | IP | Role |
|
||||||
|------|----|----|
|
|------|----|----|
|
||||||
@@ -88,7 +88,6 @@ agent: abiba
|
|||||||
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
|
| acerpve | 192.168.68.9 | PVE (hosts llm-gpu qemu/101) |
|
||||||
| minipve | 192.168.68.12 | PVE |
|
| minipve | 192.168.68.12 | PVE |
|
||||||
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
|
| amdpve | 192.168.68.15 | PVE + Strix Halo LLM (ornith) |
|
||||||
| hwepve | 192.168.68.4 | PVE (Huawei Matebook 16, 12C/15GB) — hosts Mumuni (lxc/114) migrated from minipve 2026-07-20 |
|
|
||||||
|
|
||||||
## Operations
|
## Operations
|
||||||
|
|
||||||
|
|||||||
+1
-3
@@ -17,11 +17,10 @@ declare -A CT_NODES=(
|
|||||||
[112]=amdpve # tanko
|
[112]=amdpve # tanko
|
||||||
[113]=amdpve # baggy
|
[113]=amdpve # baggy
|
||||||
[115]=amdpve # scottdenya
|
[115]=amdpve # scottdenya
|
||||||
# hwepve (192.168.68.4) — Huawei Matebook 16
|
|
||||||
[114]=hwepve # mumuni (migrated from minipve 2026-07-20)
|
|
||||||
# minipve (192.168.68.12)
|
# minipve (192.168.68.12)
|
||||||
[104]=minipve # authentik
|
[104]=minipve # authentik
|
||||||
[110]=minipve # gitea
|
[110]=minipve # gitea
|
||||||
|
[114]=minipve # mumuni
|
||||||
[116]=minipve # syslog-api
|
[116]=minipve # syslog-api
|
||||||
# storepve (192.168.68.6)
|
# storepve (192.168.68.6)
|
||||||
[106]=storepve # ra-h-os
|
[106]=storepve # ra-h-os
|
||||||
@@ -49,7 +48,6 @@ declare -A NODE_IPS=(
|
|||||||
[storepve]=192.168.68.6
|
[storepve]=192.168.68.6
|
||||||
[acerpve]=192.168.68.9
|
[acerpve]=192.168.68.9
|
||||||
[ocupve]=192.168.68.5
|
[ocupve]=192.168.68.5
|
||||||
[hwepve]=192.168.68.4
|
|
||||||
)
|
)
|
||||||
|
|
||||||
resolve_node() {
|
resolve_node() {
|
||||||
|
|||||||
@@ -46,13 +46,12 @@ You are a code reviewer for OpenProse infrastructure contracts in the Syslog Sol
|
|||||||
|
|
||||||
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
|
The infrastructure-control.prose.md contract is the canonical reference for the cluster topology:
|
||||||
|
|
||||||
**Proxmox Cluster "Tabiri" (6 nodes):**
|
**Proxmox Cluster "Tabiri" (5 nodes):**
|
||||||
- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
|
- amdpve (192.168.68.15): abiba, kagentz, tanko, tdunna, baggy, scottdenya
|
||||||
- minipve (192.168.68.12): authentik, gitea, syslog-api, jitsi
|
- minipve (192.168.68.12): authentik, gitea, mumuni, syslog-api, jitsi
|
||||||
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
|
- storepve (192.168.68.6): docker-vm, ra-h-os, PBS, media, zulip
|
||||||
- acerpve (192.168.68.9): llm-gpu, adguard
|
- acerpve (192.168.68.9): llm-gpu, adguard
|
||||||
- ocupve (192.168.68.5): ocu-llm
|
- ocupve (192.168.68.5): ocu-llm
|
||||||
- hwepve (192.168.68.4): mumuni (migrated from minipve 2026-07-20, Huawei Matebook 16, 12C/15GB)
|
|
||||||
|
|
||||||
**CT IDs (verified 2026-07-04 against PVE API):**
|
**CT IDs (verified 2026-07-04 against PVE API):**
|
||||||
100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os
|
100:abiba 102:adguard 104:authentik 105:kagentz 106:ra-h-os
|
||||||
|
|||||||
@@ -90,12 +90,16 @@ echo "── 3. Cross-contract consistency ──"
|
|||||||
|
|
||||||
# Check that contracts referencing each other have correct names
|
# Check that contracts referencing each other have correct names
|
||||||
if [ -f "infrastructure-control.prose.md" ]; then
|
if [ -f "infrastructure-control.prose.md" ]; then
|
||||||
# Any contract that claims to check "all 6 PVE nodes" should name them
|
# Any contract that claims to check "all 5 PVE nodes" should name them
|
||||||
for f in *.prose.md; do
|
for f in *.prose.md; do
|
||||||
[ -f "$f" ] || continue
|
[ -f "$f" ] || continue
|
||||||
if grep -qE "\b5-node\b|\b5 node\b|\b5 Proxmox\b|all 5 PVE" "$f" 2>/dev/null; then
|
if grep -q "5-node\|5 node\|5 Proxmox\|all.*PVE.*node" "$f" 2>/dev/null; then
|
||||||
echo " ⚠️ $f: still references 5-node cluster (migrated to 6 nodes 2026-07-20)"
|
for node in amdpve minipve storepve acerpve ocupve; do
|
||||||
|
grep -q "$node" "$f" || {
|
||||||
|
echo " ⚠️ $f: references 5 nodes but '$node' not mentioned"
|
||||||
WARNINGS=$((WARNINGS + 1))
|
WARNINGS=$((WARNINGS + 1))
|
||||||
|
}
|
||||||
|
done
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ Runs every 15 minutes in the background. Also triggers on session start.
|
|||||||
## Requires
|
## Requires
|
||||||
|
|
||||||
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
- **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
|
||||||
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123, lxc/114 on hwepve since 2026-07-20), and Agent Zero Docker host (192.168.68.14)
|
- **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
|
||||||
- **PM2** on localhost for pi process management
|
- **PM2** on localhost for pi process management
|
||||||
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
- **Network access** to `chat.sysloggh.net`, `localhost:9200`
|
||||||
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
- **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
|
||||||
@@ -199,7 +199,7 @@ Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error`
|
|||||||
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
|
||||||
```
|
```
|
||||||
|
|
||||||
Gateway PID should exist with uptime > 60s. **Dual-gateway detection**: if more than one `gateway run` process is found, the gateway has a collision (typically one `--force` and one `--replace` process). Kill the newer/duplicate process, then restart the remaining gateway: for Mumuni (PM2-managed), use `pm2 restart mumuni-zulip`; for Tanko (systemd/non-PM2), use `hermes gateway restart`. Check gateway log for "Gateway running with 2 platform(s)" (not 1) to confirm Zulip reloaded.
|
Gateway PID should exist with uptime > 60s.
|
||||||
|
|
||||||
**B3: Heartbeat Verification**
|
**B3: Heartbeat Verification**
|
||||||
|
|
||||||
@@ -222,7 +222,7 @@ ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/
|
|||||||
|
|
||||||
| Condition | Action |
|
| Condition | Action |
|
||||||
|-----------|--------|
|
|-----------|--------|
|
||||||
| `zulip.state != "connected"` | Mumuni (PM2): `ssh root@<CT> "pm2 restart mumuni-zulip"`<br>Tanko (systemd): `ssh root@<CT> "hermes gateway restart"` |
|
| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
|
||||||
| No heartbeat in 10min | Same as above |
|
| No heartbeat in 10min | Same as above |
|
||||||
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
|
| `Failed to finalize` > 50% | Check PATCH API, Zulip server |
|
||||||
| Response empty/short | Check A2A endpoint / LiteLLM model |
|
| Response empty/short | Check A2A endpoint / LiteLLM model |
|
||||||
|
|||||||
Reference in New Issue
Block a user