no-mistakes(document): Fix stale compression-model docs across 4 contracts

tune: switch compression model from strix-moe to syslog-auto (v2)
Relieves Strix Halo thermal pressure by routing compression through syslog-auto weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070). - compression.model: strix-moe -> syslog-auto - auxiliary.compression.model: strix-moe -> syslog-auto - Rule 7: Updated for syslog-auto compression distribution - Rule 8: Updated GPU workload distribution with syslog-auto pool - Description/version banners: Added 2026-07-18 change record
2026-07-19 00:31:54 +00:00 · 2026-07-18 23:57:07 +00:00
9 changed files with 42 additions and 345 deletions
@@ -128,10 +128,3 @@ safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
 Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
 It covers the full process: verify → draft → lint → review → ship, with templates
 and style rules.
 ## Maintaining this file
 Keep this file for knowledge useful to almost every future agent session in this project.
 Do not repeat what the codebase already shows; point to the authoritative file or command instead.
 Prefer rewriting or pruning existing entries over appending new ones.
 When updating this file, preserve this bar for all agents and keep entries concise.
@@ -1 +0,0 @@
 AGENTS.md
@@ -22,7 +22,6 @@ owners:
 - abiba
 - mumuni
 - kwame
 - ops
 trigger_types:
 - scheduled
 - event_driven
@@ -59,7 +58,6 @@ index:
    - memory-audit-maintenance
    - gpu-fleet
    - infrastructure-update
    - infrastructure-maintenance
    reference:
    - infrastructure-control
    - ra-h-os-custodianship-contract
@@ -92,7 +90,6 @@ index:
    - infrastructure-control
    - infrastructure-monitoring
    - infrastructure-update
    - infrastructure-maintenance
    - pm2-self-heal
    - disk-gc-threat-response
    gpu:
@@ -133,6 +130,7 @@ index:
    - build-zulip-plugin
    - stirling-pdf-agent-access
    - gpu-fleet
    - infrastructure-update
    - infrastructure-control
    - zulip-adapter-lessons
    - pi-approval-architecture
@@ -146,9 +144,6 @@ index:
    - mumuni-delegation
    kwame:
    - hello-world
    ops:
    - infrastructure-maintenance
    - infrastructure-update
  by_trigger:
    scheduled:
    - hermes-key-enforcement
@@ -161,7 +156,6 @@ index:
    - litellm-health
    - memory-audit-maintenance
    - infrastructure-update
    - infrastructure-maintenance
    event_driven:
    - litellm-self-heal
    - pm2-self-heal
@@ -203,7 +197,6 @@ index:
    - hermes-zulip-plugin
    - build-zulip-plugin
    - infrastructure-update
    - infrastructure-maintenance
    - ra-h-os-custodianship-contract
    - mumuni-delegation
    normal:
@@ -1263,108 +1256,13 @@ contracts:
  last_run: null
  last_status: null
  drift_alerts: []
 - name: infrastructure-maintenance
  file: infrastructure-maintenance.prose.md
  kind: responsibility
  category: maintenance
  sensitivity: high
  status: active
  owner: ops
  version: 1.0.0
  trigger:
    type: scheduled
    cadence: 0 2 * * 0
    description: Weekly host-level maintenance Sunday at 2am ET (replaces infrastructure-update
      build-phase role; infra-update moves to ops)
    cron_job_id: null
  execution:
    agent: ops
    timeout: 3600
    requires:
    - infrastructure-monitoring run within last 30 minutes (pre-update health baseline)
    - Proxmox snapshot of primary host OR /tmp backup dir created this run
    - LiteLLM master key from Infisical vault for health verification
    protocol:
    - Load contract from prose-contracts/main
    - Phase 0 preflight — capture health baseline, backup check, record image baseline
    - Phase 1 apt update && apt upgrade -y on primary host
    - Phase 2 docker compose pull for LiteLLM, SearXNG, and other running containers
    - Phase 3 restart stacks one at a time with per-stack health verification
    - Phase 4 verify every critical service (LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways)
    - On failure — rollback per protocol, escalate, do not loop beyond circuit breaker
    - Log actions to ~/.hermes/runs/infrastructure-maintenance/
  verification:
    postconditions:
    - check: all critical services running after update
      verify: 'curl -sf http://192.168.68.116/litellm/v1/models && curl -sf https://chat.sysloggh.net/api/v1/server_settings && curl -sf https://git.sysloggh.net/api/v1/version && curl -sf http://192.168.68.7:8888 && pm2 jlist'
      expect: all probes 200 OK / processes online
    - check: no regressions from pre-update health baseline
      verify: diff Phase 0 health-baseline against Phase 4 results
      expect: no GREEN service turned RED
    - check: docker containers on latest stable tags
      verify: docker inspect --format '{{.Config.Image}}' <container> per service matches image-baseline.pulled_tag
      expect: all containers running pulled tags
    - check: APT packages up to date with no held broken packages
      verify: apt list --upgradable 2>/dev/null | wc -l and apt-get -s upgrade | grep -ci broken
      expect: upgradable == 0, broken == 0
    artifact: maintenance run report with phase results and any rollback/escalation
    verify_commands:
    - curl -sf http://192.168.68.116/litellm/v1/models
    - curl -sf http://192.168.68.7:8888
    - curl -sf https://chat.sysloggh.net/api/v1/server_settings
    - curl -sf https://git.sysloggh.net/api/v1/version
    - pm2 jlist
  receipt:
    format: json
    storage: ~/.hermes/runs/infrastructure-maintenance/
    graph_node: true
    schema:
      contract: string
      run_id: string
      timestamp: ISO 8601
      agent: string
      status: pass|fail|escalated
      phase: preflight|apt|images|restarts|verify|rollback|done|failed
      actions_taken: array
      postconditions: array
      drift_alerts: array
      evidence_path: string
  escalation:
    info:
      action: log_to_receipt
      notify: []
    warning:
      action: relay_alert
      notify:
      - abiba
      - mumuni
    critical:
      action: relay_alert
      notify:
      - abiba
      - mumuni
    fatal:
      action: relay_alert + pause + human_required
      notify:
      - abiba
      - mumuni
      - kwame
  circuit_breaker:
    max_retries: 2
    window: 7200
    trip_action: escalate_to_fatal
  depends_on:
  - infrastructure-monitoring
  last_run: null
  last_status: null
  drift_alerts: []
 - name: infrastructure-update
  file: infrastructure-update.prose.md
  kind: responsibility
  category: maintenance
  sensitivity: high
  status: active
-  owner: ops
+  owner: abiba
  version: 1.0.0
  trigger:
    type: scheduled
@@ -125,7 +125,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 | Alias | RPM Cap | Routes To | Purpose |
 |-------|---------|-----------|---------|
-| `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
+| `strix-moe` | 40 | Strix Halo | Agent reasoning, compression (~30% via syslog-auto pool) (MoE models) |
 | `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
 | `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
@@ -136,7 +136,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 - syslog-auto → qwen → gemma → qwen3.6-35B-udq4
 ### Why Strix Halo RPM Is Capped
- Direct (strix-moe): 40 RPM (tight) — Strix Halo is shared with compression tasks
+- Direct (strix-moe): 40 RPM (tight) — Strix Halo handles agent reasoning + compression via syslog-auto pool
 - Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
 - Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
@@ -284,7 +284,7 @@ History stored at `/root/data/toks-history.json` with 7-day rolling window.
 ### Stable Aliases — CRITICAL
 All agent configs MUST use stable role-based aliases, never model-specific names:
- `compression.model: strix-moe` (NOT `qwen3.6-35B-udq4`)
+- `compression.model: syslog-auto` — switched from strix-moe 2026-07-18 to distribute across weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070); relieves Strix Halo thermal pressure
 - `auxiliary.vision.model: gpu-light` (NOT `gemma-4-12b`)
 - `delegation.model: gpu-dense` (NOT `qwen3.6-27B-code`)
 - `auxiliary.web_extract.model: gpu-light`
@@ -295,7 +295,7 @@ When the underlying model is swapped, only the LiteLLM config changes — agent
 - RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
 - **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
 - Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
- Mumuni compression model alias: `strix-moe` with 300s timeout
+- Mumuni compression model alias: `syslog-auto` (switched from strix-moe 2026-07-18) with 300s timeout
 ### Mumuni Agent Profile
@@ -305,8 +305,8 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
 |---------|-------|-------|
 | `model.default` | `syslog-auto` | Weighted pool (55% qwen, 30% strix, 15% gemma) |
 | `model.provider` | `custom:litellm` | LiteLLM on CT116 |
-| `compression.model` | `strix-moe` | Stable alias — survives model swaps |
+| `compression.model` | `syslog-auto` | Switched from strix-moe 2026-07-18 — distributes across weighted pool |
-| `aux.compression.model` | `strix-moe` | Compression auxiliary model |
+| `aux.compression.model` | `syslog-auto` | Compression auxiliary — switched from strix-moe 2026-07-18 |
 | `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
 | `aux.web_extract.model` | `gpu-light` | Web extraction |
 | `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
@@ -318,7 +318,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
 | `personalities` | `creative` | Creative assistant personality |
 | Platforms | cli, discord, homeassistant, signal, telegram, zulip | All Hermes platforms |
 | Main model timeout | 300s | LiteLLM global timeout |
-| Compression model timeout | 300s | strix-moe timeout increased from 120s |
+| Compression model timeout | 300s | syslog-auto timeout (switched from strix-moe 2026-07-18) |
 ### Agent Update Status (2026-07-15)
@@ -46,7 +46,7 @@ depends_on:
 |-------|-----|------|-------|------|-----|-------|------|
 | `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
 | `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
-| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
+| `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Agent reasoning, compression (~30% via syslog-auto pool), summarization, long docs |
 Key notes:
 - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
@@ -157,7 +157,7 @@ Key notes:
 - **Target distribution**:
  - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
  - RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
-  - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
+  - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Agent reasoning, compression (~30% via syslog-auto pool), summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
 - **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
 - **Fix**:
  - Alert if any GPU is handling workload outside its designated role
@@ -5,7 +5,7 @@ version: 1.0.0
 description: >
  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
  configuration state, keys, workarounds, and audit procedure. When an agent's
-  configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 256K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (Strix Halo handles compression solo).
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-16. All GPUs 128K context (RTX 3090 .8, RTX 5070 .110, Strix Halo .15). Parallel 1 fleet-wide (compression via syslog-auto pool — switched from strix-moe 2026-07-18).
 author: Abiba (pi agent)
 ---
@@ -5,11 +5,14 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
+  UPDATED 2026-07-18: Compression model switched to `syslog-auto` (was `strix-moe`)
  to relieve Strix Halo pressure. syslog-auto distributes compression across the
  weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
  UPDATED 2026-07-16: Compression model was the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
  which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
  Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
  2026-07-16 Mumuni root-cause investigation (WAL #1300).
-  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
+  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (later switched to syslog-auto 2026-07-18). RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
 ---
 ## Maintains
@@ -130,7 +133,9 @@ mcp_servers:
 # ─── Compression ───
 compression:
  enabled: true
-  model: strix-moe               # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
+  model: syslog-auto              # ⚠️ Switched from strix-moe 2026-07-18 to relieve Strix Halo.
                                  # syslog-auto distributes across weighted pool (55% RTX 3090,
                                  # 30% Strix Halo, 15% RTX 5070). All GPUs at 128K.
  provider: harness
  max_context_window: 131072      # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
@@ -145,8 +150,9 @@ compression:
 #   model: gpu-light                 # stable alias (NOT raw "gemma-4-12b")
 #   base_url: http://192.168.68.116/v1
 #   api_key_env: LITELLM_API_KEY
-# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
+# Compression uses syslog-auto (switched from strix-moe 2026-07-18) to distribute
-# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
+# load across the weighted pool and relieve Strix Halo pressure.
 # Vision and web_extract use gpu-light = RTX 5070 (12B).
 # Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
 # NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
 # in agent configs — use the stable aliases so model swaps don't break agents.
@@ -166,7 +172,7 @@ auxiliary:
    timeout: 30
  compression:
    provider: harness
-    model: strix-moe              # MUST match compression.model above. Stable alias for Strix Halo.
+    model: syslog-auto             # Switched from strix-moe 2026-07-18. Relieves Strix Halo pressure.
    base_url: http://192.168.68.116/v1   # Rule 5: /v1 NOT /litellm/v1
    api_key_env: LITELLM_API_KEY
    timeout: 300                 # gpu-fleet: 300s for large-history summarization (was 60)
@@ -247,29 +253,30 @@ The following MUST be identical across ALL profiles:
 - Apply to BOTH main config AND all sub-agent profiles
 - For agents needing longer outputs: raise to 8192, but never omit
-### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
+### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-18)
- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
+- Vision and web_extract use `gpu-light` (stable alias, RTX 5070 — 12GB, vision-optimized)
- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized)
+- Compression now uses `syslog-auto` (switched from `strix-moe` 2026-07-18) to distribute
- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
+  compression load across the weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
-  (it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
+  This relieves Strix Halo pressure while keeping compression functional on all GPUs.
 - **`syslog-auto` is the valid compression model** — LiteLLM serves it as the weighted pool.
  Old configs with `strix-moe` for compression should be updated to `syslog-auto`.
 - All auxiliary services MUST use identical routing:
  - `base_url: http://192.168.68.116/v1`  (Rule 5: `/v1`, NOT `/litellm/v1`)
  - `api_key_env: LITELLM_API_KEY`
- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
+- **Compression via syslog-auto**: Routes through the weighted pool. Strix Halo still handles
- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
+  ~30% of compression calls (at 60 RPM via pool vs 40 RPM direct), but the bulk (55%)
-  (64GB UMA, 128K context) — the designated compression GPU. This frees the
+  goes to RTX 3090 which has ample spare capacity.
  RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
 - The `compression:` block's `model` MUST match `auxiliary: compression: model`
 - The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
-### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
+### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
+- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
+- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
- **Strix Halo (64GB, 128K ctx, strix-moe)**: Context compression, summarization, long docs
+- **Strix Halo (64GB, 128K ctx, Genesis Hermes V3 APEX)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
 - Agent profiles MUST route auxiliary tasks to the correct GPU:
-  - `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
+  - `auxiliary.vision.model: gpu-light` (RTX 5070)
-  - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
+  - `auxiliary.web_extract.model: gpu-light` (RTX 5070)
-  - `auxiliary.compression.model: strix-moe` (Strix Halo)
+  - `auxiliary.compression.model: syslog-auto` (distributed pool, switched from strix-moe 2026-07-18)
 - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
 - For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
@@ -57,7 +57,7 @@ duration.
 prefill time at 532 tok/s. Fix context first, routing second.
 - **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
-  queries; gemma for compression/auxiliary. Never send simple completion to a
+  queries; gemma for vision/web extraction; syslog-auto for compression (via weighted pool). Never send simple completion to a
  35B MoE.
 - **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
  compact at 102K, not 166K. Target 15% tail (not 30%).
@@ -1,200 +0,0 @@
 ---
 kind: responsibility
 name: infrastructure-maintenance
 description: >
  Weekly system-level maintenance for the Syslog inference fleet: OS package
  updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
  running containers, container restarts with health verification, post-update
  verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
  PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
  raw shell scripts that previously did this piecemeal. This contract owns the
  HOST-LEVEL weekly maintenance loop on the primary host plus Docker image
  pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET
  cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker
  image Wave 3 across all stacks). Runs Sunday 2am ET. Owner:
  ops (firstmate secondmate). Blast radius: an unverified image pull can break
  LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
  upgrade can leave the host in a half-upgraded state. Pre-update backup check
  and rollback are mandatory for this reason.
 agent: ops
 triggers:
  - weekly (Sunday 02:00 ET) via cron
  - on demand when ops/abiba triggers "infra maintenance"
 version: 1.0.0
 ---
 ## Maintains
 - maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
 - image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
 - apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
 - health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
 - rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
 - maintenance-history: array of past runs with phase results and any escalations
 ## Scope
 Primary host is the maintenance host where apt updates apply. Docker image pulls
 span the two Docker ecosystems that run critical services. infrastructure-update
 runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across
 all stacks/hosts); this contract runs a narrower host-level weekly pull limited
 to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against
 `infrastructure-control.prose.md` (the source of truth) and the live system
 before mutating.
 | Host | IP | Role | Trust |
 |------|----|------|-------|
 | CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
 | VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
 | CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
 | Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
 | CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
 > "Primary host" for the apt phase is the host the ops agent runs maintenance
 > from. Confirm which host that is against infrastructure-control before
 > running; do not assume. If the ops agent is containerized/CT-based, apt runs
 > inside that CT.
 ## Requires
 - SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
 - `apt`, `docker`, `docker compose` available on target hosts
 - LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
 - `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
 - Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
 - Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
 ## Continuity
 - Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
 - Also wakes on: explicit "infra maintenance" trigger from ops/abiba
 - Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
 ## Execution
 ### Phase 0 — Preflight (snapshot/backup check + health baseline)
 1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
 2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
   - `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
   - `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
 3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
 4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
 ### Phase 1 — OS package updates (primary host)
 ```bash
 # On the primary host only (VERIFY host against infrastructure-control first)
 apt update
 apt upgrade -y
 ```
 - Capture `apt list --upgradable` before and after → store in `apt-state`.
 - If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
 - If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
 ### Phase 2 — Docker image pulls
 Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
 ```bash
 # CT 116 (.116) — inference harness
 cd /opt/inference-harness && docker compose pull
 # VM 109 (.7) — search + home stacks
 cd /opt/search-stack/searxng && docker compose pull
 cd /opt/search-stack/firecrawl-source && docker compose pull
 # any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
 ```
 - LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
 - Record pulled tag + digest per service in `image-baseline`.
 ### Phase 3 — Container restarts with health verification
 Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
 ```bash
 # CT 116
 cd /opt/inference-harness && docker compose up -d
 # VM 109
 cd /opt/search-stack/searxng && docker compose up -d
 cd /opt/search-stack/firecrawl-source && docker compose up -d
 ```
 After each stack comes up, wait for health (max 120s):
 - `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
 - Service-specific probe passes (see Phase 4 probes)
 If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
 ### Phase 4 — Post-update service verification
 After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
 | Service | Probe | Expect |
 |---------|-------|--------|
 | LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
 | LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
 | SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
 | Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
 | Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
 | PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
 | Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
 Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
 ## Rollback Protocol
 If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
 1. **Image rollback** — for the failing stack, restore the previous image:
   ```bash
   # Restore from recorded image-baseline digest
   docker compose down
   # Pin the service image to the recorded digest in compose, then recreate
   #   image: <name>@sha256:<previous_digest>
   docker compose pull && docker compose up -d
   ```
 2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
 3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
 4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
 5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
 ## Circuit Breaker
 - `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
 - `window: 7200` seconds — no more than 2 retries within a 2-hour window.
 - `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
 ## Report
 After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
 ```
 🛠 Infrastructure Maintenance — YYYY-MM-DD
 Phase: apt | images | restarts | verify | rollback
 Primary host: <host>
 APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
 Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
 Services: all GREEN | <service> FAILED (rolled back)
 Baseline regression: none | <details>
 Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
 Escalation: none | warning | critical | fatal
 ```
 ## Verification Postconditions
 - All critical services running after update (Phase 4 all GREEN)
 - No regressions from pre-update health baseline (Phase 0 baseline)
 - Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
 - APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
 ## Related Contracts
 - `infrastructure-update.prose.md` — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
 - `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
 - `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
 - `litellm-health.prose.md` — LiteLLM probe details.
 - `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.