no-mistakes(document): Sync zulip-health v3.1.0 in registry and fix Koby IP to .129

- Add Koonimo (CT 113, .114) and Koby (CT 111) to Platform B monitoring
- Add Infisical dependency check (/usr/local/bin/infisical presence)
- Add config YAML validation (yaml.safe_load check)
- Add stale PID/lock detection and cleanup before restart
- Add LiteLLM key injection verification (no-key-required pattern)
- Add Telegram adapter health check
- Add cli_agent_setup_mixin.py patch verification
- Update restart commands with Infisical-missing fallback path
- Update requires/maintains schema with new fields
- Use pct exec from amdpve as primary access for Koonimo/Koby
- Verified correct IPs: Koonimo=.114 (not .113), Koby=pct exec only

AGENTS.md

@@ -128,3 +128,10 @@ safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
 Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
 It covers the full process: verify → draft → lint → review → ship, with templates
 and style rules.
+
+## Maintaining this file
+
+Keep this file for knowledge useful to almost every future agent session in this project.
+Do not repeat what the codebase already shows; point to the authoritative file or command instead.
+Prefer rewriting or pruning existing entries over appending new ones.
+When updating this file, preserve this bar for all agents and keep entries concise.

CLAUDE.md

@@ -0,0 +1 @@
+AGENTS.md

contract-registry.yaml

@@ -1,5 +1,5 @@
 registry_version: 0.1.0
-last_updated: '2026-07-13T00:00:00Z'
+last_updated: '2026-07-23T00:00:00Z'
 updated_by: mumuni
 categories:
 - compliance
@@ -22,6 +22,7 @@ owners:
 - abiba
 - mumuni
 - kwame
+- ops
 trigger_types:
 - scheduled
 - event_driven
@@ -58,6 +59,7 @@ index:
     - memory-audit-maintenance
     - gpu-fleet
     - infrastructure-update
+    - infrastructure-maintenance
     reference:
     - infrastructure-control
     - ra-h-os-custodianship-contract
@@ -90,6 +92,7 @@ index:
     - infrastructure-control
     - infrastructure-monitoring
     - infrastructure-update
+    - infrastructure-maintenance
     - pm2-self-heal
     - disk-gc-threat-response
     gpu:
@@ -130,7 +133,6 @@ index:
     - build-zulip-plugin
     - stirling-pdf-agent-access
     - gpu-fleet
-    - infrastructure-update
     - infrastructure-control
     - zulip-adapter-lessons
     - pi-approval-architecture
@@ -144,6 +146,9 @@ index:
     - mumuni-delegation
     kwame:
     - hello-world
+    ops:
+    - infrastructure-maintenance
+    - infrastructure-update
   by_trigger:
     scheduled:
     - hermes-key-enforcement
@@ -156,6 +161,7 @@ index:
     - litellm-health
     - memory-audit-maintenance
     - infrastructure-update
+    - infrastructure-maintenance
     event_driven:
     - litellm-self-heal
     - pm2-self-heal
@@ -197,6 +203,7 @@ index:
     - hermes-zulip-plugin
     - build-zulip-plugin
     - infrastructure-update
+    - infrastructure-maintenance
     - ra-h-os-custodianship-contract
     - mumuni-delegation
     normal:
@@ -621,7 +628,7 @@ contracts:
   sensitivity: high
   status: active
   owner: abiba
-  version: 3.0.0
+  version: 3.1.0
   trigger:
     type: scheduled
     cadence: '*/15 * * * *'
@@ -1256,13 +1263,108 @@ contracts:
   last_run: null
   last_status: null
   drift_alerts: []
+- name: infrastructure-maintenance
+  file: infrastructure-maintenance.prose.md
+  kind: responsibility
+  category: maintenance
+  sensitivity: high
+  status: active
+  owner: ops
+  version: 1.0.0
+  trigger:
+    type: scheduled
+    cadence: 0 2 * * 0
+    description: Weekly host-level maintenance Sunday at 2am ET (replaces infrastructure-update
+      build-phase role; infra-update moves to ops)
+    cron_job_id: null
+  execution:
+    agent: ops
+    timeout: 3600
+    requires:
+    - infrastructure-monitoring run within last 30 minutes (pre-update health baseline)
+    - Proxmox snapshot of primary host OR /tmp backup dir created this run
+    - LiteLLM master key from Infisical vault for health verification
+    protocol:
+    - Load contract from prose-contracts/main
+    - Phase 0 preflight — capture health baseline, backup check, record image baseline
+    - Phase 1 apt update && apt upgrade -y on primary host
+    - Phase 2 docker compose pull for LiteLLM, SearXNG, and other running containers
+    - Phase 3 restart stacks one at a time with per-stack health verification
+    - Phase 4 verify every critical service (LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways)
+    - On failure — rollback per protocol, escalate, do not loop beyond circuit breaker
+    - Log actions to ~/.hermes/runs/infrastructure-maintenance/
+  verification:
+    postconditions:
+    - check: all critical services running after update
+      verify: 'curl -sf http://192.168.68.116/litellm/v1/models && curl -sf https://chat.sysloggh.net/api/v1/server_settings && curl -sf https://git.sysloggh.net/api/v1/version && curl -sf http://192.168.68.7:8888 && pm2 jlist'
+      expect: all probes 200 OK / processes online
+    - check: no regressions from pre-update health baseline
+      verify: diff Phase 0 health-baseline against Phase 4 results
+      expect: no GREEN service turned RED
+    - check: docker containers on latest stable tags
+      verify: docker inspect --format '{{.Config.Image}}' <container> per service matches image-baseline.pulled_tag
+      expect: all containers running pulled tags
+    - check: APT packages up to date with no held broken packages
+      verify: apt list --upgradable 2>/dev/null | wc -l and apt-get -s upgrade | grep -ci broken
+      expect: upgradable == 0, broken == 0
+    artifact: maintenance run report with phase results and any rollback/escalation
+    verify_commands:
+    - curl -sf http://192.168.68.116/litellm/v1/models
+    - curl -sf http://192.168.68.7:8888
+    - curl -sf https://chat.sysloggh.net/api/v1/server_settings
+    - curl -sf https://git.sysloggh.net/api/v1/version
+    - pm2 jlist
+  receipt:
+    format: json
+    storage: ~/.hermes/runs/infrastructure-maintenance/
+    graph_node: true
+    schema:
+      contract: string
+      run_id: string
+      timestamp: ISO 8601
+      agent: string
+      status: pass|fail|escalated
+      phase: preflight|apt|images|restarts|verify|rollback|done|failed
+      actions_taken: array
+      postconditions: array
+      drift_alerts: array
+      evidence_path: string
+  escalation:
+    info:
+      action: log_to_receipt
+      notify: []
+    warning:
+      action: relay_alert
+      notify:
+      - abiba
+      - mumuni
+    critical:
+      action: relay_alert
+      notify:
+      - abiba
+      - mumuni
+    fatal:
+      action: relay_alert + pause + human_required
+      notify:
+      - abiba
+      - mumuni
+      - kwame
+  circuit_breaker:
+    max_retries: 2
+    window: 7200
+    trip_action: escalate_to_fatal
+  depends_on:
+  - infrastructure-monitoring
+  last_run: null
+  last_status: null
+  drift_alerts: []
 - name: infrastructure-update
   file: infrastructure-update.prose.md
   kind: responsibility
   category: maintenance
   sensitivity: high
   status: active
-  owner: abiba
+  owner: ops
   version: 1.0.0
   trigger:
     type: scheduled

hermes-config-template.prose.md

@@ -5,11 +5,14 @@ description: >
   Standard Hermes configuration template for Syslog Solution LLC agents.
   Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
   RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
+  UPDATED 2026-07-18: Compression model switched to `syslog-auto` (was `strix-moe`)
+  to relieve Strix Halo pressure. syslog-auto distributes compression across the
+  weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
+  UPDATED 2026-07-16: Compression model was the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
   which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
   Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
   2026-07-16 Mumuni root-cause investigation (WAL #1300).
-  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
+  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo (later switched to syslog-auto 2026-07-18). RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
 ---
 
 ## Maintains
@@ -130,7 +133,9 @@ mcp_servers:
 # ─── Compression ───
 compression:
   enabled: true
-  model: strix-moe               # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
+  model: syslog-auto              # ⚠️ Switched from strix-moe 2026-07-18 to relieve Strix Halo.
+                                  # syslog-auto distributes across weighted pool (55% RTX 3090,
+                                  # 30% Strix Halo, 15% RTX 5070). All GPUs at 128K.
   provider: harness
   max_context_window: 131072      # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
   threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
@@ -145,8 +150,9 @@ compression:
 #   model: gpu-light                 # stable alias (NOT raw "gemma-4-12b")
 #   base_url: http://192.168.68.116/v1
 #   api_key_env: LITELLM_API_KEY
-# Do NOT use syslog-auto for auxiliary tasks — it routes to the primary GPU.
-# gpu-light = RTX 5070 (12B), freeing the Strix Halo for agent reasoning.
+# Compression uses syslog-auto (switched from strix-moe 2026-07-18) to distribute
+# load across the weighted pool and relieve Strix Halo pressure.
+# Vision and web_extract use gpu-light = RTX 5070 (12B).
 # Heavy aux (delegation, x_search) use gpu-dense (RTX 3090) instead.
 # NEVER use raw model names (gemma-4-12b, qwen3.6-27B-code, qwen3.6-35B-udq4)
 # in agent configs — use the stable aliases so model swaps don't break agents.
@@ -166,7 +172,7 @@ auxiliary:
     timeout: 30
   compression:
     provider: harness
-    model: strix-moe              # MUST match compression.model above. Stable alias for Strix Halo.
+    model: syslog-auto             # Switched from strix-moe 2026-07-18. Relieves Strix Halo pressure.
     base_url: http://192.168.68.116/v1   # Rule 5: /v1 NOT /litellm/v1
     api_key_env: LITELLM_API_KEY
     timeout: 300                 # gpu-fleet: 300s for large-history summarization (was 60)
@@ -247,29 +253,30 @@ The following MUST be identical across ALL profiles:
 - Apply to BOTH main config AND all sub-agent profiles
 - For agents needing longer outputs: raise to 8192, but never omit
 
-### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
-- Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
-- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized)
-- **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
-  (it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
+### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-18)
+- Vision and web_extract use `gpu-light` (stable alias, RTX 5070 — 12GB, vision-optimized)
+- Compression now uses `syslog-auto` (switched from `strix-moe` 2026-07-18) to distribute
+  compression load across the weighted pool (55% RTX 3090, 30% Strix Halo, 15% RTX 5070).
+  This relieves Strix Halo pressure while keeping compression functional on all GPUs.
+- **`syslog-auto` is the valid compression model** — LiteLLM serves it as the weighted pool.
+  Old configs with `strix-moe` for compression should be updated to `syslog-auto`.
 - All auxiliary services MUST use identical routing:
   - `base_url: http://192.168.68.116/v1`  (Rule 5: `/v1`, NOT `/litellm/v1`)
   - `api_key_env: LITELLM_API_KEY`
-- **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
-- **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
-  (64GB UMA, 128K context) — the designated compression GPU. This frees the
-  RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
+- **Compression via syslog-auto**: Routes through the weighted pool. Strix Halo still handles
+  ~30% of compression calls (at 60 RPM via pool vs 40 RPM direct), but the bulk (55%)
+  goes to RTX 3090 which has ample spare capacity.
 - The `compression:` block's `model` MUST match `auxiliary: compression: model`
 - The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
 
-### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
-- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
-- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
-- **Strix Halo (64GB, 128K ctx, strix-moe)**: Context compression, summarization, long docs
+### Rule 8: GPU Workload Distribution (UPDATED 2026-07-18)
+- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations — also handles ~55% of compression via syslog-auto pool
+- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract — handles ~15% of compression via syslog-auto pool
+- **Strix Halo (64GB, 128K ctx, Geneis Hermes V3 APEX)**: Agent reasoning, compression (~30% via syslog-auto pool), fallback for other GPUs
 - Agent profiles MUST route auxiliary tasks to the correct GPU:
-  - `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
-  - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
-  - `auxiliary.compression.model: strix-moe` (Strix Halo)
+  - `auxiliary.vision.model: gpu-light` (RTX 5070)
+  - `auxiliary.web_extract.model: gpu-light` (RTX 5070)
+  - `auxiliary.compression.model: syslog-auto` (distributed pool, switched from strix-moe 2026-07-18)
 - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
 - For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss

infrastructure-maintenance.prose.md

@@ -0,0 +1,200 @@
+---
+kind: responsibility
+name: infrastructure-maintenance
+description: >
+  Weekly system-level maintenance for the Syslog inference fleet: OS package
+  updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
+  running containers, container restarts with health verification, post-update
+  verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
+  PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
+  raw shell scripts that previously did this piecemeal. This contract owns the
+  HOST-LEVEL weekly maintenance loop on the primary host plus Docker image
+  pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET
+  cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker
+  image Wave 3 across all stacks). Runs Sunday 2am ET. Owner:
+  ops (firstmate secondmate). Blast radius: an unverified image pull can break
+  LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
+  upgrade can leave the host in a half-upgraded state. Pre-update backup check
+  and rollback are mandatory for this reason.
+agent: ops
+triggers:
+  - weekly (Sunday 02:00 ET) via cron
+  - on demand when ops/abiba triggers "infra maintenance"
+version: 1.0.0
+---
+
+## Maintains
+
+- maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
+- image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
+- apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
+- health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
+- rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
+- maintenance-history: array of past runs with phase results and any escalations
+
+## Scope
+
+Primary host is the maintenance host where apt updates apply. Docker image pulls
+span the two Docker ecosystems that run critical services. infrastructure-update
+runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across
+all stacks/hosts); this contract runs a narrower host-level weekly pull limited
+to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against
+`infrastructure-control.prose.md` (the source of truth) and the live system
+before mutating.
+
+| Host | IP | Role | Trust |
+|------|----|------|-------|
+| CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
+| VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
+| CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
+| Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
+| CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
+
+> "Primary host" for the apt phase is the host the ops agent runs maintenance
+> from. Confirm which host that is against infrastructure-control before
+> running; do not assume. If the ops agent is containerized/CT-based, apt runs
+> inside that CT.
+
+## Requires
+
+- SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
+- `apt`, `docker`, `docker compose` available on target hosts
+- LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
+- `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
+- Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
+- Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
+
+## Continuity
+
+- Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
+- Also wakes on: explicit "infra maintenance" trigger from ops/abiba
+- Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
+
+## Execution
+
+### Phase 0 — Preflight (snapshot/backup check + health baseline)
+
+1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
+2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
+   - `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
+   - `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
+3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
+4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
+
+### Phase 1 — OS package updates (primary host)
+
+```bash
+# On the primary host only (VERIFY host against infrastructure-control first)
+apt update
+apt upgrade -y
+```
+
+- Capture `apt list --upgradable` before and after → store in `apt-state`.
+- If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
+- If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
+
+### Phase 2 — Docker image pulls
+
+Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
+
+```bash
+# CT 116 (.116) — inference harness
+cd /opt/inference-harness && docker compose pull
+
+# VM 109 (.7) — search + home stacks
+cd /opt/search-stack/searxng && docker compose pull
+cd /opt/search-stack/firecrawl-source && docker compose pull
+# any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
+```
+
+- LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
+- Record pulled tag + digest per service in `image-baseline`.
+
+### Phase 3 — Container restarts with health verification
+
+Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
+
+```bash
+# CT 116
+cd /opt/inference-harness && docker compose up -d
+# VM 109
+cd /opt/search-stack/searxng && docker compose up -d
+cd /opt/search-stack/firecrawl-source && docker compose up -d
+```
+
+After each stack comes up, wait for health (max 120s):
+- `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
+- Service-specific probe passes (see Phase 4 probes)
+
+If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
+
+### Phase 4 — Post-update service verification
+
+After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
+
+| Service | Probe | Expect |
+|---------|-------|--------|
+| LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
+| LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
+| SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
+| Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
+| Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
+| PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
+| Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
+
+Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
+
+## Rollback Protocol
+
+If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
+
+1. **Image rollback** — for the failing stack, restore the previous image:
+   ```bash
+   # Restore from recorded image-baseline digest
+   docker compose down
+   # Pin the service image to the recorded digest in compose, then recreate
+   #   image: <name>@sha256:<previous_digest>
+   docker compose pull && docker compose up -d
+   ```
+2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
+3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
+4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
+5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
+
+## Circuit Breaker
+
+- `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
+- `window: 7200` seconds — no more than 2 retries within a 2-hour window.
+- `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
+
+## Report
+
+After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
+
+```
+🛠 Infrastructure Maintenance — YYYY-MM-DD
+
+Phase: apt | images | restarts | verify | rollback
+Primary host: <host>
+APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
+Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
+Services: all GREEN | <service> FAILED (rolled back)
+Baseline regression: none | <details>
+Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
+Escalation: none | warning | critical | fatal
+```
+
+## Verification Postconditions
+
+- All critical services running after update (Phase 4 all GREEN)
+- No regressions from pre-update health baseline (Phase 0 baseline)
+- Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
+- APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
+
+## Related Contracts
+
+- `infrastructure-update.prose.md` — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
+- `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
+- `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
+- `litellm-health.prose.md` — LiteLLM probe details.
+- `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.

zulip-health.prose.md

@@ -1,9 +1,9 @@
 ---
 kind: responsibility
 name: zulip-health
-description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Agent Zero Docker), Platform B (Hermes agents Tanko/Mumuni), and the Zulip bridge. Verifies bot registration, DM delivery, and cross-platform connectivity.
+description: Multi-platform health monitor for the Zulip messaging mesh spanning Platform A (Abiba pi), Platform B (Hermes agents Tanko/Mumuni/Koonimo/Koby), and Platform C (Agent Zero). Verifies bot registration, DM delivery, cross-platform connectivity, secret injection, and YAML config integrity.
 title: Zulip Mesh Health Monitor — Multi-Platform
-version: 3.0.0
+version: 3.1.0
 runtime_contract: 2
 agent: abiba
 ---
@@ -12,11 +12,15 @@ agent: abiba
 
 Monitors ALL Zulip-connected agents across three platforms (pi, Hermes, Agent Zero).
 Runs every 15 minutes in the background. Also triggers on session start.
+v3.1.0 adds Koonimo+Koby to Platform B, Infisical dependency checks, config YAML
+validation, stale PID/lock detection, Telegram adapter health, and the
+cli_agent_setup_mixin patch verification.
 
 ## Requires
 
 - **Zulip API key** for `abiba-bot@chat.sysloggh.net` in `$ZULIP_API_KEY`
 - **SSH access** to Tanko (192.168.68.122), Mumuni (192.168.68.123), and Agent Zero Docker host (192.168.68.14)
+- **SSH access to amdpve (192.168.68.15)** for `pct exec` access to Koonimo (CT 113) and Koby (CT 111)
 - **PM2** on localhost for pi process management
 - **Network access** to `chat.sysloggh.net`, `localhost:9200`
 - **Write access** to `/root/zulip-health-monitor.log` and `/tmp/zulip-monitor-debounce`
@@ -49,12 +53,22 @@ Runs every 15 minutes in the background. Also triggers on session start.
     "zulip_state": "connected",
     "heartbeat_age_seconds": 45,
     "gateway_pid": 1234,
+    "infisical_present": true,
+    "config_valid": true,
+    "telegram_state": "connected",
+    "no_key_required_count": 0,
     "edit_fail_rate_pct": 0,
     "severity": "healthy"
   }
 }
 ```
 
+New fields in v3.1.0:
+- `infisical_present` — /usr/local/bin/infisical exists on the agent CT
+- `config_valid` — /root/.hermes/config.yaml passes YAML validation
+- `telegram_state` — Telegram adapter status from gateway_state.json
+- `no_key_required_count` — count of `no-key-required` in gateway logs
+
 ### Postconditions
 
 - Every platform is independently checked; one failure doesn't block others
@@ -81,13 +95,13 @@ Log as "unreachable" — don't treat as critical unless it persists for 3+ conse
 ## Streaming Support (2026-07-05)
 
 Zulip agents now support progressive message editing during agent generation.
-When a Hermes agent (Tanko, Mumuni) processes a message, the response is
-streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
+When a Hermes agent (Tanko, Mumuni, Koonimo, Koby) processes a message, the
+response is streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
 
 - Adapter implements `edit_message()` using `_api_patch()` helper
 - Gateway stream consumer progressively edits the Zulip message
 - User sees real-time agent thinking instead of waiting for full response
-- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
+- Verified: Tanko (CT 112), Mumuni (CT 114), Koonimo (CT 113), Koby (CT 111)
 
 ### Verification
 ```bash
@@ -182,50 +196,217 @@ grep -a "Finalized\|Failed to finalize" /root/.pm2/logs/abiba-zulip-out.log | ta
 | `last_error` set | Log and monitor |
 | Crash loop >10/h | Alert user |
 
-### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123)
+### Step 3: Platform B — Hermes (Tanko .122, Mumuni .123, Koonimo .114, Koby .129)
+
+Platform B now monitors four Hermes agents:
+- Tanko (CT 112, 192.168.68.122) — Zulip + Telegram
+- Mumuni (CT 114, 192.168.68.123) — Zulip + Telegram + Email
+- Koonimo (CT 113, 192.168.68.114, hostname "baggy") — Zulip + Telegram
+- Koby (CT 111, 192.168.68.129, hostname "tdunna") — Zulip + Telegram
+
+SSH access: Koonimo is reachable at .114; Koby has no direct SSH. Use `pct exec`
+from amdpve as the primary access method for both:
+```bash
+ssh root@192.168.68.15 "pct exec 113 -- <command>"   # Koonimo (or ssh .114)
+ssh root@192.168.68.15 "pct exec 111 -- <command>"   # Koby (pct exec only)
+```
 
 **B1: Gateway State**
 
 ```bash
 ssh root@192.168.68.122 "cat ~/.hermes/gateway_state.json"
 ssh root@192.168.68.123 "cat ~/.hermes/gateway_state.json"
+ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
+ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
 ```
 
-Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌ | missing → not installed.
+Check `platforms.zulip.state`: `connected` ✅ | `disconnected` ❌ | `error` ❌.
+
+**B1.5: Infisical Dependency Check**
+
+```bash
+ssh root@<CT> "test -f /usr/local/bin/infisical && echo OK || echo MISSING"
+# pct exec variant for Koonimo/Koby:
+ssh root@192.168.68.15 "pct exec 113 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
+ssh root@192.168.68.15 "pct exec 111 -- test -f /usr/local/bin/infisical && echo OK || echo MISSING"
+```
+
+If MISSING → flag `infisical_present: false`, note as degraded — gateway cannot
+auto-start on reboot without the infisical binary.
 
 **B2: Agent Process**
 
 ```bash
 ssh root@<CT> "ps aux | grep 'gateway run' | grep -v grep"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- ps aux | grep 'gateway run' | grep -v grep"
+ssh root@192.168.68.15 "pct exec 111 -- ps aux | grep 'gateway run' | grep -v grep"
 ```
 
-Gateway PID should exist with uptime > 60s.
+Gateway PID should exist with uptime > 60s. Check for stale PIDs:
+- `gateway.pid` and `gateway.lock` files that reference a dead process
+- Multiple gateway processes (duplicate PIDs)
+
+**B2.5: Config YAML Validation**
+
+```bash
+ssh root@<CT> "python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
+ssh root@192.168.68.15 "pct exec 111 -- python3 -c 'import yaml; yaml.safe_load(open(\"/root/.hermes/config.yaml\"))' 2>&1"
+```
+
+Expected: no output (clean parse). If parse fails → flag `config_valid: false`,
+degraded — gateway is running on stale in-memory config.
+
+Check specifically for:
+- Stray `api_key: sk-...` lines indented under `api_key_env` entries in
+  `custom_providers` section (hardcoded keys violate hermes-key-enforcement)
+- Indentation errors in `custom_providers`, `auxiliary`, or `compression` blocks
 
 **B3: Heartbeat Verification**
 
 ```bash
 ssh root@<CT> "grep Heartbeat ~/.hermes/logs/agent.log | tail -3"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
+ssh root@192.168.68.15 "pct exec 111 -- grep Heartbeat /root/.hermes/logs/agent.log | tail -3"
 ```
 
 Expected: recent heartbeat (within 5 min), `polls=N` incrementing.
 Silence > 300s → warning. Silence > 600s → critical.
 
+**B3.5: Stale PID/Lock Detection**
+
+Before any restart action, check for stale pid/lock files:
+
+```bash
+ssh root@<CT> "ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
+ssh root@192.168.68.15 "pct exec 111 -- ls -la /root/.hermes/gateway.pid /root/.hermes/gateway.lock 2>/dev/null"
+```
+
+If gateway process is dead (no PID) but pid/lock files exist:
+```bash
+ssh root@<CT> "rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
+ssh root@192.168.68.15 "pct exec 111 -- rm -f /root/.hermes/gateway.pid /root/.hermes/gateway.lock"
+```
+
+Pid/lock files blocking restart → clear them before restart attempt.
+
 **B4: Response Delivery**
 
 ```bash
 ssh root@<CT> "grep -E 'Finalized|Failed to finalize|Replied to' ~/.hermes/logs/agent.log | tail -10"
+# pct exec variant for Koonimo/Koby:
+ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
+ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Finalized|Failed to finalize|Replied to' /root/.hermes/logs/agent.log | tail -10"
 ```
 
 > 50% fail rate → critical.
 
+**B4.5: LiteLLM Key Injection Verification**
+
+Check gateway logs for `no-key-required` failure pattern (indicates the
+cli_agent_setup_mixin.py patch is missing):
+
+```bash
+ssh root@<CT> "grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
+# pct exec variant for Koonimo/Koby:
+ssh root@192.168.68.15 "pct exec 113 -- grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
+ssh root@192.168.68.15 "pct exec 111 -- grep -c 'no-key-required' /root/.hermes/logs/gateway.log 2>/dev/null || echo 0"
+```
+
+If > 0 → flag `no_key_required_count: <count>`, note as degraded — provider
+requests silently fall back to `no-key-required` when LITELLM_API_KEY env var
+resolves empty.
+
+**B5: Telegram Adapter Health**
+
+Check Telegram connectivity in gateway state or logs:
+
+```bash
+# From gateway_state.json (all agents):
+ssh root@<CT> "cat ~/.hermes/gateway_state.json | python3 -c 'import json,sys;d=json.load(sys.stdin);print(d[\"platforms\"].get(\"telegram\",{}).get(\"state\",\"missing\"))'"
+# pct exec variant for Koonimo/Koby (cat the file; read platforms.telegram.state):
+ssh root@192.168.68.15 "pct exec 113 -- cat /root/.hermes/gateway_state.json"
+ssh root@192.168.68.15 "pct exec 111 -- cat /root/.hermes/gateway_state.json"
+# From logs (check for stuck DNS resolution):
+ssh root@<CT> "grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
+ssh root@192.168.68.15 "pct exec 113 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
+ssh root@192.168.68.15 "pct exec 111 -- grep -E 'Telegram.*Connecting|Telegram.*Connected|attempt 1/8' /root/.hermes/logs/gateway.log | tail -5"
+```
+
+Telegram states: `connected` ✅ | `disconnected` ❌ | `retrying` ⚠️ | `fatal` ❌ | `paused` ⚠️
+
+If stuck on "attempt 1/8" for > 60s → flag Telegram as degraded (Zulip may
+still be fine — do NOT treat as Zulip outage).
+
+**B6: cli_agent_setup_mixin.py Patch Verification**
+
+Check whether the `no-key-required` fallback string exists without the
+LiteLLM-specific guard (only needed when LiteLLM key injection failures are
+suspected). A bare count of the fallback string cannot distinguish a guarded
+occurrence from an unguarded one, so count both the fallback string and the
+LiteLLM guard token:
+
+```bash
+ssh root@<CT> "f=\$(grep -c 'no-key-required' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c 'LITELLM_API_KEY' /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
+ssh root@192.168.68.15 "pct exec 111 -- bash -c 'f=\$(grep -c no-key-required /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); g=\$(grep -c LITELLM_API_KEY /usr/local/lib/hermes-agent/hermes_cli/cli_agent_setup_mixin.py 2>/dev/null || echo 0); echo fallback=\$f guard=\$g'"
+```
+
+If `fallback > 0` and `guard == 0` → the fallback string is present without
+the LiteLLM-specific guard, so the patch is missing. If unsure, inspect each
+occurrence with `grep -n -B2 -A2 'no-key-required'` to confirm the guard
+wraps it.
+
 **Platform B Actions**
 
 | Condition | Action |
 |-----------|--------|
-| `zulip.state != "connected"` | `ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"` |
-| No heartbeat in 10min | Same as above |
+| `zulip.state != "connected"` | Restart gateway (see B2 restart commands below) |
+| No heartbeat in 10min | Restart gateway |
 | `Failed to finalize` > 50% | Check PATCH API, Zulip server |
 | Response empty/short | Check A2A endpoint / LiteLLM model |
+| `infisical_present: false` | Flag as degraded — log and alert; do NOT auto-restart via the standard command (it cannot inject the key without infisical). Operator may use the Infisical-missing fallback restart below. |
+| `config_valid: false` | Flag as degraded — alert user, gateway running on stale config |
+| Stale pid/lock files detected | Clean files before restart |
+| `no_key_required_count > 0` | Flag as degraded — check LITELLM_API_KEY injection |
+| Telegram stuck on attempt 1/8 | Flag Telegram as degraded, no Zulip action needed |
+
+**Restart Commands**
+
+Standard restart (Infisical present):
+```bash
+ssh root@<CT> "pkill -f 'gateway run'; sleep 2; hermes gateway restart"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
+ssh root@192.168.68.15 "pct exec 111 -- bash -c 'pkill -f \"gateway run\"; sleep 2; systemctl restart hermes-gateway'"
+```
+
+The mechanisms differ by design: Tanko/Mumuni launch the gateway through a
+wrapper loop, so the `hermes gateway restart` CLI is the correct entry point
+(it re-arms the wrapper). Koonimo/Koby run the gateway as a systemd unit
+(`hermes-gateway.service`), so `systemctl restart hermes-gateway` is the
+correct entry point under `pct exec`. Do not swap the two — using the CLI on
+Koonimo/Koby would bypass the unit, and using `systemctl` on Tanko/Mumuni
+would miss the wrapper loop.
+
+Fallback: Infisical missing → start gateway directly from venv with env vars:
+```bash
+ssh root@<CT> "source /usr/local/lib/hermes-agent/venv/bin/activate && \
+  export LITELLM_API_KEY=\$(grep -E '^LITELLM_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
+  export ZULIP_API_KEY=\$(grep -E '^ZULIP_API_KEY=' /root/.hermes/.env | cut -d= -f2-) && \
+  cd /root/.hermes && nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &"
+# pct exec variant:
+ssh root@192.168.68.15 "pct exec 113 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
+ssh root@192.168.68.15 "pct exec 111 -- bash -c 'source /usr/local/lib/hermes-agent/venv/bin/activate; export LITELLM_API_KEY=\$(grep -E ^LITELLM_API_KEY= /root/.hermes/.env | cut -d= -f2-); export ZULIP_API_KEY=\$(grep -E ^ZULIP_API_KEY= /root/.hermes/.env | cut -d= -f2-); cd /root/.hermes; nohup hermes gateway run > logs/gateway-manual-start.log 2>&1 &'"
+```
 
 ### Step 4: Platform C — Agent Zero (kagentz, CT 105 via Docker host .14)
 
@@ -278,7 +459,7 @@ Expected: task ID with "working" status. Poll for completion with `tasks/get`.
 
 Check each agent's log for excessive bot-to-bot chatter:
 - Abiba: `Skipped.*bot msgs` count
-- Tanko/Mumuni: Repeated DM exchanges between bots
+- Tanko/Mumuni/Koonimo/Koby: Repeated DM exchanges between bots
 - kagentz: Adapter log for bot DMs being processed
 
 If any bot processes >50 bot-originated messages in 15min → warning.
@@ -301,6 +482,14 @@ Track via `/tmp/zulip-monitor-debounce` (unix timestamp of last restart).
 
 ## History
 
+### v3.1.0 (2026-07-23) — Koonimo Outage Lessons
+
+Added Koonimo (CT 113) and Koby (CT 111) to Platform B monitoring.
+Added Infisical dependency check, config YAML validation, stale PID/lock
+detection, Telegram adapter health, LiteLLM key injection verification, and
+cli_agent_setup_mixin.py patch verification. Updated restart commands with
+Infisical-missing fallback path.
+
 ### Gen 5 (2026-07-02) — Rate Limit Death Spiral Fix
 
 **Root Cause**: Proactive Queue Rotation at 25 min triggered queue re-registration every cycle. Each re-registration + retry loop (3 attempts) + monitor restart = 8-12 API calls per cycle. Combined with monitor's own API calls (server check, stream alerts), `abiba-bot` hit Zulip's rate limit (429 RATE_LIMIT_HIT). Each restart reset the cycle, creating a death spiral: 111 restarts in 24 hours.

zulip-resilience-v3.prose.md

@@ -409,3 +409,68 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
 | Queue expiry handling | Crash | Auto re-register |
 | Busy worker deadlock | Router death | Worker SIGKILL + error DM |
 | PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |
+
+---
+
+## Incident Log — 2026-07-18 Fleet-Wide Audit
+
+### Fleet State After Audit
+
+| Agent | Platform | Zulip State | Issues Found | Fix Applied |
+|-------|----------|-------------|--------------|-------------|
+| **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
+| **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
+| **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |
+
+### Key Fixes Applied
+
+**1. Abiba — Credential Fallback (L4 Pattern)**
+- Root cause: `zulip.api_key` in config.yaml is `""` (expected from Infisical). Infisical vault `ABIBA_ZULIP_API_KEY` wasn't being injected into the process environment.
+- Fix: Added `.env` file fallback at `/root/.pi/agent/extensions/zulip/.env` with known-working key, sourced before the Infisical `exec`.
+- Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local `.env` fallback.
+
+**2. Abiba — Poll Timeout Handling**
+- Root cause: Zulip long-poll uses `AbortSignal.timeout(65000)`. Zulip's default `event_queue_longpoll_timeout_seconds` can exceed 65s. When the signal fires, an `AbortError` is thrown and caught by the circuit breaker as a failure.
+- Fix: Caught `AbortError` inside `poll()` and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default.
+- Reference: [Zulip Events System — long-poll timeout](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html)
+
+**3. Tanko — Gateway Restart**
+- Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (`infisical-gateway.sh`) restarts on crash but the gateway wasn't re-establishing Zulip on restart.
+- Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully.
+
+### Fleet-Wide Zulip Health Metrics (as of 2026-07-18)
+
+| Metric | Value |
+|--------|-------|
+| Zulip server | ✅ HTTP 200 |
+| Agents connected | 3/3 (Abiba, Tanko, Mumuni) |
+| Abiba circuit breaker | CLOSED (0 failures) |
+| Abiba uptime | 2D (post-restart) |
+| Tanko gateway uptime | Ongoing |
+| Mumuni gateway uptime | Ongoing |
+| Watchdog status | ✅ Online (2D uptime) |
+
+### Hermes Agent Zulip Plugin Improvements
+
+Based on the audit, improvements that should be ported to all Hermes Zulip adapters:
+
+1. **Circuit breaker pattern** — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff.
+2. **Credential fallback** — All Hermes agents use Infisical for credentials. Add `.env` local fallback per L4 pattern for `ZULIP_API_KEY`.
+3. **Queue re-registration** — Handle `BAD_EVENT_QUEUE_ID` with automatic re-registration instead of gateway restart.
+4. **Supervisor watchdog** — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks.
+5. **Streaming** — All agents have `streaming: true` in their zulip config. Verify `edit_message()` is implemented in each adapter.
+
+### Abiba pi Zulip Extension v2 — Implemented Resilience Summary
+
+| Feature | Status | Notes |
+|---------|--------|-------|
+| Circuit breaker | ✅ | CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout |
+| Retry with jitter | ✅ | 2 attempts, 200ms base, 50-100% jitter |
+| Queue lifecycle | ✅ | 10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling |
+| Crash prevention | ✅ | uncaughtException + unhandledRejection recovery |
+| Worker timeout | ✅ | 5min busy timeout → SIGKILL + error DM |
+| Health endpoint | ✅ | :9200 with circuit breaker metrics |
+| Echo prevention | ✅ | Dynamic bot user resolution |
+| Poll timeout (AbortError) | ✅ v2.1 | Normal timeout returns [] instead of error |
+| Credential fallback | ✅ v2.1 | .env file before Infisical exec |
+| Provider auto-fix | ✅ | Detects reasoning_content models, switches to compatible |