Merge branch 'master' into fm/infra-maint-contract

no-mistakes(document): Reframed infra-maintenance contract as deliberate partition
no-mistakes(review): fix docker rollback command to pin digest then compose up
2026-07-18 23:12:20 +00:00 · 2026-07-18 22:16:09 +00:00 · 2026-07-18 22:07:15 +00:00 · 2026-07-18 22:06:48 +00:00 · 2026-07-18 22:06:34 +00:00 · 2026-07-18 22:02:45 +00:00
12 changed files with 788 additions and 227 deletions
@@ -128,3 +128,10 @@ safe-mutate --verify "CMD" [--expect "PATTERN"] --mutate "CMD" [--reason "WHY"]
 Read the [Authoring Guide](docs/AUTHORING-GUIDE.md) before writing any new contract.
 It covers the full process: verify → draft → lint → review → ship, with templates
 and style rules.
 ## Maintaining this file
 Keep this file for knowledge useful to almost every future agent session in this project.
 Do not repeat what the codebase already shows; point to the authoritative file or command instead.
 Prefer rewriting or pruning existing entries over appending new ones.
 When updating this file, preserve this bar for all agents and keep entries concise.
@@ -0,0 +1 @@
 AGENTS.md
@@ -22,6 +22,7 @@ owners:
 - abiba
 - mumuni
 - kwame
 - ops
 trigger_types:
 - scheduled
 - event_driven
@@ -58,6 +59,7 @@ index:
    - memory-audit-maintenance
    - gpu-fleet
    - infrastructure-update
    - infrastructure-maintenance
    reference:
    - infrastructure-control
    - ra-h-os-custodianship-contract
@@ -90,6 +92,7 @@ index:
    - infrastructure-control
    - infrastructure-monitoring
    - infrastructure-update
    - infrastructure-maintenance
    - pm2-self-heal
    - disk-gc-threat-response
    gpu:
@@ -130,7 +133,6 @@ index:
    - build-zulip-plugin
    - stirling-pdf-agent-access
    - gpu-fleet
    - infrastructure-update
    - infrastructure-control
    - zulip-adapter-lessons
    - pi-approval-architecture
@@ -144,6 +146,9 @@ index:
    - mumuni-delegation
    kwame:
    - hello-world
    ops:
    - infrastructure-maintenance
    - infrastructure-update
  by_trigger:
    scheduled:
    - hermes-key-enforcement
@@ -156,6 +161,7 @@ index:
    - litellm-health
    - memory-audit-maintenance
    - infrastructure-update
    - infrastructure-maintenance
    event_driven:
    - litellm-self-heal
    - pm2-self-heal
@@ -197,6 +203,7 @@ index:
    - hermes-zulip-plugin
    - build-zulip-plugin
    - infrastructure-update
    - infrastructure-maintenance
    - ra-h-os-custodianship-contract
    - mumuni-delegation
    normal:
@@ -1256,13 +1263,108 @@ contracts:
  last_run: null
  last_status: null
  drift_alerts: []
 - name: infrastructure-maintenance
  file: infrastructure-maintenance.prose.md
  kind: responsibility
  category: maintenance
  sensitivity: high
  status: active
  owner: ops
  version: 1.0.0
  trigger:
    type: scheduled
    cadence: 0 2 * * 0
    description: Weekly host-level maintenance Sunday at 2am ET (replaces infrastructure-update
      build-phase role; infra-update moves to ops)
    cron_job_id: null
  execution:
    agent: ops
    timeout: 3600
    requires:
    - infrastructure-monitoring run within last 30 minutes (pre-update health baseline)
    - Proxmox snapshot of primary host OR /tmp backup dir created this run
    - LiteLLM master key from Infisical vault for health verification
    protocol:
    - Load contract from prose-contracts/main
    - Phase 0 preflight — capture health baseline, backup check, record image baseline
    - Phase 1 apt update && apt upgrade -y on primary host
    - Phase 2 docker compose pull for LiteLLM, SearXNG, and other running containers
    - Phase 3 restart stacks one at a time with per-stack health verification
    - Phase 4 verify every critical service (LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways)
    - On failure — rollback per protocol, escalate, do not loop beyond circuit breaker
    - Log actions to ~/.hermes/runs/infrastructure-maintenance/
  verification:
    postconditions:
    - check: all critical services running after update
      verify: 'curl -sf http://192.168.68.116/litellm/v1/models && curl -sf https://chat.sysloggh.net/api/v1/server_settings && curl -sf https://git.sysloggh.net/api/v1/version && curl -sf http://192.168.68.7:8888 && pm2 jlist'
      expect: all probes 200 OK / processes online
    - check: no regressions from pre-update health baseline
      verify: diff Phase 0 health-baseline against Phase 4 results
      expect: no GREEN service turned RED
    - check: docker containers on latest stable tags
      verify: docker inspect --format '{{.Config.Image}}' <container> per service matches image-baseline.pulled_tag
      expect: all containers running pulled tags
    - check: APT packages up to date with no held broken packages
      verify: apt list --upgradable 2>/dev/null | wc -l and apt-get -s upgrade | grep -ci broken
      expect: upgradable == 0, broken == 0
    artifact: maintenance run report with phase results and any rollback/escalation
    verify_commands:
    - curl -sf http://192.168.68.116/litellm/v1/models
    - curl -sf http://192.168.68.7:8888
    - curl -sf https://chat.sysloggh.net/api/v1/server_settings
    - curl -sf https://git.sysloggh.net/api/v1/version
    - pm2 jlist
  receipt:
    format: json
    storage: ~/.hermes/runs/infrastructure-maintenance/
    graph_node: true
    schema:
      contract: string
      run_id: string
      timestamp: ISO 8601
      agent: string
      status: pass|fail|escalated
      phase: preflight|apt|images|restarts|verify|rollback|done|failed
      actions_taken: array
      postconditions: array
      drift_alerts: array
      evidence_path: string
  escalation:
    info:
      action: log_to_receipt
      notify: []
    warning:
      action: relay_alert
      notify:
      - abiba
      - mumuni
    critical:
      action: relay_alert
      notify:
      - abiba
      - mumuni
    fatal:
      action: relay_alert + pause + human_required
      notify:
      - abiba
      - mumuni
      - kwame
  circuit_breaker:
    max_retries: 2
    window: 7200
    trip_action: escalate_to_fatal
  depends_on:
  - infrastructure-monitoring
  last_run: null
  last_status: null
  drift_alerts: []
 - name: infrastructure-update
  file: infrastructure-update.prose.md
  kind: responsibility
  category: maintenance
  sensitivity: high
  status: active
-  owner: abiba
+  owner: ops
  version: 1.0.0
  trigger:
    type: scheduled
@@ -9,8 +9,12 @@ description: >
  gpu-dense, gpu-light. These never change — only the underlying model does.
  Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
  RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
-  RTX 5070 context: 131K → 256K. VRAM: 88% (10.8/12.2GB).
+  UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability.
-  Compression timeout: 300s (was 120s). Mumuni context: 128K (was 256K).
+  Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored,
  Hermes agent fine-tune, tensor repair, multimodal with mmproj).
  Instability observed near 100K at 256K. 128K is the stable ceiling.
  For larger context needs → fall back to external providers (deepseek).
  VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
 agent: abiba
 triggers:
  - on model add/remove
@@ -67,7 +71,7 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│ 256K ctx │  │ 256K ctx    │  │ 256K ctx  │  │ Watchdog     │
+│ 128K ctx │  │ 128K ctx    │  │ 128K ctx  │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ qwen3.6   │  │ Prometheus   │
 │ 27B-code │  │ :8080       │  │ -35B-udq4 │  │ exporter     │
 │ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
@@ -93,9 +97,9 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | 22.2/24.6GB (90%) | **256K** 🚀 | turbo4 | 2 | default | ✅ 63 tok/s |
+| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 10.0/12.2GB (82%) | 256K | q4_0 | 2 | 2048/1024 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
-| qwen3.6-35B-udq4 | Strix Halo Vulkan | .15 (amdpve) | ~9GB/64GB | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |
+| Genesis Hermes V3 APEX | Strix Halo Vulkan | .15 (amdpve) | ~10GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
 ## Routing Configuration (LiteLLM — July 2026)
@@ -104,7 +108,7 @@ but are deprecated for agent configs. Only the stable aliases survive model swap
 | Model | GPU | Weight | RPM Cap | Timeout |
 |-------|-----|--------|---------|---------|
 | qwen3.6-27B-code | RTX 3090 (.8:8080) | **0.55** | 500 | **300s** |
-| qwen3.6-35B-udq4 | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
+| Genesis Hermes V3 APEX | Strix Halo (.15:8080) | **0.30** | 60 | **300s** |
 | gemma-4-12b | RTX 5070 (.110:8080) | **0.15** | 200 | **120s** |
 Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`. The router (port 9000) is NOT in the inference path.
@@ -113,7 +117,7 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
-| qwen3.6-35B-udq4 | 40 | Tight cap — prevents Strix overload |
+| strix-moe (Hermes V3) | 40 | Tight cap — prevents Strix overload |
 | qwen3.6-27B-code | 500 | High cap — primary workhorse |
 | gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
@@ -128,11 +132,11 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 ### Fallback Chains
 - gemma → qwen
 - qwen → gemma
- qwen3.6-35B-udq4 → qwen → gemma
+- strix-moe → qwen → gemma
 - syslog-auto → qwen → gemma → qwen3.6-35B-udq4
 ### Why Strix Halo RPM Is Capped
- Direct (qwen3.6-35B-udq4): 40 RPM (tight) — Strix Halo is shared with compression tasks
+- Direct (strix-moe): 40 RPM (tight) — Strix Halo is shared with compression tasks
 - Via syslog-auto: 60 RPM (moderate) — prevents flooding when multiple agents use syslog-auto simultaneously
 - Combined max: ~100 RPM across both paths — Strix Halo can sustain this at 80°C
@@ -243,12 +247,12 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM (2026-07-15)**: RTX 3090 at 22.2/24.6GB (90%) with **256K context** (corrected from 131K). RTX 5070 at 10.8/12.2GB (88%) with 256K context + MTP. Strix Halo at ~9GB/64GB.
+- **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
 - **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
- **RTX 3090 config**: `-c 262144 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context corrected to 256K (2026-07-15). VRAM: 90%. Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 256K context. Gen speed: 122 tok/s (was 70). VRAM: 10.8/12.2GB (88%). No draft model pre-upgrade due to VRAM constraints. Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 262144`.
+- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
 - **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080 (was `ornith-server.service`), model changed to `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` (UD-Q4_K_M), alias `qwen3.6-35B-udq4`, 256K context, flash-attn + q8 KV. MTP support enabled for 1.4-2.2x faster inference.
+- **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V3-GGUF` (APEX quant), alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). Hermes agent fine-tune, tensor repair (SSM layers fixed via SVD), uncensored (0/465 refusals).
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `strix-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
 - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
@@ -262,12 +266,12 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **256K** |
+| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** |
-| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **256K** |
+| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
-| Strix Halo (.15) | qwen3.6-35B-udq4 | **71** | — | — | **256K** |
+| Strix Halo (.15) | Genesis Hermes V3 APEX | **65** | 140 | — | **128K** |
-Benchmarks from 2026-07-15 verification run. RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
+Benchmarks from 2026-07-17. Strix Halo swapped to Genesis Hermes V3 APEX (LuffyTheFox). RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.
-All 3 GPUs now at 256K context (2026-07-15).
+All 3 GPUs now at 128K context (2026-07-17, reduced from 256K for stability).
 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
 Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
@@ -288,9 +292,9 @@ All agent configs MUST use stable role-based aliases, never model-specific names
 When the underlying model is swapped, only the LiteLLM config changes — agent configs are untouched.
 ### Context Windows
- RTX 3090: **256K** (was 131K, bumped 2026-07-15) | RTX 5070: **256K** (up from 131K) | Strix Halo: **256K**
+- RTX 3090: **128K** (reduced from 256K 2026-07-17) | RTX 5070: **128K** (reduced from 256K) | Strix Halo: **128K**
- **Mumuni compression context**: 128K (down from 256K) — ensures compression model doesn't timeout
+- **All agents**: 128K ceiling — stable margin. For >128K workloads, use external providers (deepseek)
- Compression threshold 0.65: fires at ~85K for 128K context window
+- Compression threshold 0.65: fires at ~85K (~43K headroom before 128K ceiling)
 - Mumuni compression model alias: `strix-moe` with 300s timeout
 ### Mumuni Agent Profile
@@ -306,7 +310,7 @@ Mumuni (CT114, 192.168.68.123) is the primary business assistant. This profile i
 | `aux.vision.model` | `gpu-light` | Vision tasks (RTX 5070) |
 | `aux.web_extract.model` | `gpu-light` | Web extraction |
 | `delegation.model` | `gpu-dense` | Sub-agent reasoning (RTX 3090) |
-| `context.max_context_window` | 262144 (256K) | Fixed 2026-07-16 (was 131072 — caused premature compression, WAL #1300) |
+| `context.max_context_window` | 131072 (128K) | Reduced from 256K 2026-07-17 — stable 128K ceiling |
 | `compression.threshold` | 0.65 | Triggers at ~85K |
 | `compression.target_ratio` | 0.3 | Compresses to ~38K |
 | `compression.protect_last_n` | 40 | Preserves last 40 messages |
@@ -6,10 +6,15 @@ description: >
  benchmarks, and predicts failures before they happen. Extends gpu-monitor
  (v2.1.0) with active remediation rules, Prometheus metrics consumption,
  VRAM trend analysis, and predictive alerting.
  UPDATED 2026-07-18: Model assignments synced to 2026-07-17 swaps.
  Router (port 9000) references replaced with direct GPU routing.
  Benchmark baselines refreshed to live values.
  Prometheus exporters removed — not deployed; fall back to direct sidecar probes.
  Stable role-based aliases (strix-moe, gpu-dense, gpu-light) from gpu-fleet.
 agent: abiba
 depends_on:
  - gpu-monitor.prose.md (live data source on .24:9100)
-  - gpu-fleet.prose.md (source of truth for topology)
+  - gpu-fleet.prose.md (source of truth for topology, aliases, model assignments)
 ---
 ## Maintains
@@ -22,8 +27,8 @@ depends_on:
 ## Requires
- gpu-monitor:function — Live fleet data from .24:9100/gpu-data
+- gpu-monitor:function — Live fleet data from localhost:9100/gpu-data
- Prometheus exporters on all 3 GPUs (:9400/metrics)
+- Direct sidecar probe access to all GPU hosts (:8080/health)
 - SSH access to GPU hosts for restart operations
 ## Continuity
@@ -35,21 +40,37 @@ depends_on:
 ---
 ## Current Fleet Baseline (2026-07-18)
 | Alias | GPU | Host | Model | VRAM | Ctx | tok/s | Role |
 |-------|-----|------|-------|------|-----|-------|------|
 | `gpu-dense` | RTX 3090 24GB | ct8 (.8:8080) | ThinkingCap Qwen3.6-27B Q4_K_M + MTP + vision | 21.6/24.6GB (88%) | 128K | 74.9 | Heavy reasoning, code gen |
 | `gpu-light` | RTX 5070 12GB | ct110 (.110:8080) | HauhauCS Gemma4-12B QAT Q4_K_M + MTP draft | 10.1/12.2GB (83%) | 128K | 169.6 | Vision, web extract, light tasks |
 | `strix-moe` | Strix Halo 64GB | ct15 (.15:8080) | Genesis Hermes V3 APEX (LuffyTheFox, 24GB) | ~10/64GB (16%) | 128K | 62.9 | Compression, summarization, long docs |
 Key notes:
 - All models use direct GPU routing via LiteLLM (`api_key: not-needed`). Router (port 9000) is deprecated and NOT in the inference path.
 - Stable aliases (gpu-dense, gpu-light, strix-moe) from gpu-fleet are the canonical names for agent configs. Model-specific names still work but are deprecated.
 - RTX 5070 tok/s is 2.3x faster than RTX 3090 for its model — gpu-light is the fastest endpoint. Route vision/web/light work there first.
 - Strix Halo is 62.9 tok/s (89% of 70.5 baseline) — below optimal but stable. Check for competing workloads.
 - RTX 3090 VRAM at 88% — within role-appropriate range (role = heavy reasoning, needs the headroom).
 - RTX 5070 VRAM at 83% — role-appropriate for vision/web (smaller batch sizes).
 ## Remediation Rules
 ### Rule 1: GPU Temperature Critical (>85°C for >2 min)
 - **Detect**: Any GPU temp >85°C sustained for 2+ consecutive polls
 - **Fix**: 
  1. Reduce inference concurrency on that GPU (load-side cooling only — NO fan control)
-  2. Redirect new requests to cooler GPUs via LiteLLM fallback chains
+  2. Redirect new requests to cooler GPUs via LiteLLM fallback chains (gemma → qwen, qwen → gemma)
  3. If all GPUs hot, alert about cooling infrastructure
 - **Verify**: Temp drops below 80°C within 5 minutes
 - **Escalate after**: 3 verification failures → Zulip alert
 ### Rule 2: VRAM Leak Detection (tiered by GPU capacity)
 - **Detect**: VRAM growing at sustained rate over 6+ hour window
-  - RTX 3090 (24GB): ≥100MB/hour
+  - RTX 3090 (24GB): ≥300MB/hour
-  - RTX 5070 (12GB): ≥50MB/hour
+  - RTX 5070 (12GB): ≥300MB/hour
  - Strix Halo (64GB UMA): ≥200MB/hour
 - **Fix**: 
  1. Log VRAM snapshot with process list (nvidia-smi/rocm-smi + ps aux)
@@ -69,6 +90,9 @@ depends_on:
 ### Rule 4: Benchmark Regression (>20% drop)
 - **Detect**: gen_tok_per_sec drops >20% below baseline over 3+ benchmarks
  - RTX 3090 baseline: 74.8 tok/s → alert at <59.8 tok/s
  - RTX 5070 baseline: 165.2 tok/s → alert at <132.2 tok/s
  - Strix Halo baseline: 70.5 tok/s → alert at <56.4 tok/s
 - **Fix**:
  1. Check GPU utilization — if >90%, other process is competing
  2. Check power limit — if throttled, restore to max
@@ -78,32 +102,31 @@ depends_on:
 ### Rule 5: Circuit Breaker Stuck Open
 - **Detect**: Circuit breaker open >10 minutes with GPU reporting healthy
 - **Note**: Router (port 9000) is deprecated. If circuit breakers are reported by gpu-monitor, they come from LiteLLM's internal tracking, not the old router.
 - **Fix**: 
-  1. Verify GPU /health returns 200
+  1. Verify GPU /health returns 200 on direct port (:8080)
-  2. If GPU healthy, send 1 test inference
+  2. If GPU healthy, alert but do NOT reset via router API (deprecated)
-  3. If test succeeds → reset circuit breaker via router API
+  3. Check LiteLLM health directly: http://192.168.68.116/litellm/health/liveliness
-  4. 60s cooldown — if CB re-opens immediately, it was legitimate, do NOT re-reset
+  4. Restart LiteLLM container on CT 116 if circuit breakers are stuck
-  5. Max 1 auto-reset per GPU per hour
+- **Verify**: LiteLLM returns healthy, circuit breaker clears within 60s
- **Verify**: CB closes, inference succeeds, CB stays closed for 60s+
+- **Escalate after**: LiteLLM restart doesn't clear → human investigation
 - **Escalate after**: CB won't close after reset → router issue
 ### Rule 6: Strix Halo Unreachable
 - **Detect**: Strix not responding — probe .15:8080 directly (firewall opened .24→.15)
 - **Fix**: 
  1. SSH to .15 → check llama-server process
  2. Restart llama-server if not running
-  3. Verify through both direct probe AND router
+  3. Verify through both direct probe AND LiteLLM health
- **Verify**: Direct health probe returns 200, router reports Strix healthy
+- **Verify**: Direct health probe returns 200, LiteLLM reports model healthy
 - **Escalate**: If host .15 itself is unreachable → infrastructure alert
-### Rule 7: Prometheus Exporter Down
+### Rule 7: GPU Data Source Unreachable (replaces old Prometheus rule)
- **Detect**: Any GPU :9400/metrics unreachable for >2 polls
+- **Detect**: gpu-monitor endpoint (localhost:9100/gpu-data) or sidecar port (:8080) on any GPU unreachable for >2 polls
 - **Fix**:
-  1. SSH to GPU host → check prometheus-exporter process
+  1. If gpu-monitor is down: restart systemd service `gpu-monitor.service` on this host
-  2. Restart exporter if dead
+  2. If sidecar is down: SSH to GPU host → check llama-server process → restart systemd service
-  3. While exporter is down, fall back to nvidia-smi/rocm-smi direct probes
+  3. Fall back to direct nvidia-smi/rocm-smi probe via SSH if all API paths fail
-  4. If exporter is running but unreachable → check firewall/host networking
+- **Verify**: gpu-monitor returns healthy + all sidecars reachable
 - **Verify**: :9400/metrics returns 200
 - **Escalate after**: 3 failed restarts → networking issue
 ### Rule 8: Predictive Thermal Warning (two-tier)
@@ -117,29 +140,31 @@ depends_on:
 - **Escalate**: If Tier 2 triggers and temp still rising after 5 min → possible hardware failure
 ### Rule 9: Context Window Optimization
- **Detect**: Benchmark tok/s vs baseline for each GPU at current context
+- **Detect**: Benchmark tok/s vs baseline for each GPU at current context (all 128K)
-  - RTX 3090 (256K ctx, qwen3.6-27B-code): target 75+ tok/s — currently at baseline
+  - RTX 3090 (128K ctx, ThinkingCap): baseline 74.8 tok/s — currently at 74.9 (100%)
-  - RTX 5070 (131K ctx, gemma-4-12b): target 76+ tok/s — optimal for vision/web role
+  - RTX 5070 (128K ctx, HauhauCS QAT): baseline 165.2 tok/s — currently at 169.6 (103%)
-  - Strix Halo (256K ctx, strix-moe / qwen3.6-35B-udq4): target 70+ tok/s — currently above baseline
+  - Strix Halo (128K ctx, Genesis Hermes V3): baseline 70.5 tok/s — currently at 62.9 (89%)
 - **Fix**:
  - If tok/s > baseline → context has headroom, consider increasing
  - If tok/s < 90% baseline → reduce context by 25% and retest
  - If tok/s within 10% of baseline → optimal, no change
  - Strix Halo at 89% of baseline → MONITOR but do not reduce yet (recent model swap may still be settling)
 - **Verify**: Re-benchmark after context change, confirm within 10% of target
 - **Escalate**: If context can't be adjusted without significant perf loss
-### Rule 10: Workload Distribution Optimization
+### Rule 10: Workload Distribution Optimization (updated 2026-07-18)
 - **Detect**: GPU roles misaligned with hardware capabilities
 - **Target distribution**:
-  - RTX 3090 (24GB, 256K, 75 tok/s) → Heavy reasoning, code gen, long conversations
+  - RTX 3090 (gpu-dense, 24GB, 74.9 tok/s) → Heavy reasoning, code gen, long conversations (slowest per-token but largest context capacity). Weight: 0.55 (LiteLLM).
-  - RTX 5070 (12GB, 131K, 76 tok/s) → Vision/image, web search, quick lightweight tasks
+  - RTX 5070 (gpu-light, 12GB, 169.6 tok/s) → Vision/image, web search, lightweight tasks (2.3x faster than 3090 per token). Weight: 0.15 (LiteLLM).
-  - Strix Halo (64GB, 256K, 72 tok/s) → Context compression, summarization, long docs
+  - Strix Halo (strix-moe, 64GB, 62.9 tok/s) → Context compression, summarization, long docs (MoE model). Weight: 0.30 (LiteLLM).
 - **Note**: RTX 5070 is the fastest endpoint per token. Route high-volume, low-complexity work there first.
 - **Fix**:
  - Alert if any GPU is handling workload outside its designated role
-  - Recommend Hermes agent profile updates to match workload to GPU
+  - Recommend agent alias updates to match workload to GPU role (use stable aliases: gpu-dense, gpu-light, strix-moe)
  - Track per-GPU request distribution via LiteLLM spend logs
 - **Verify**: Each GPU's request pattern matches its designated role within 24h
- **Escalate**: If role mismatch persists >48h → agent profile audit needed
+- **Escalate**: If role mismatch persists >48h → agent alias audit needed
 ---
@@ -148,7 +173,7 @@ depends_on:
 ```prose
 -- Phase 1: Fetch live GPU data
 let fleet = call gpu-monitor
-  endpoint: "http://192.168.68.24:9100/gpu-data"
+  endpoint: "http://localhost:9100/gpu-data"
 -- Phase 2: Evaluate each GPU against remediation rules
 let actions = []
@@ -164,27 +189,25 @@ for gpu in fleet.gpus:
  -- Rule 4: Benchmark regression
  let bench = fleet.benchmarks[gpu.hostname]
-  if bench.current_tok_sec < bench.baseline_tok_sec * 0.8:
+  if bench.current_tok_s < bench.baseline_tok_s * 0.8:
    push actions apply-benchmark-fix(gpu, bench)
 -- Rule 3: Model stuck
-for model in fleet.router.available_models:
+for model in fleet.summary.available_models:
  if model.consecutive_timeouts >= 3:
    push actions apply-model-restart(model)
-- Rule 5: Circuit breaker
+-- Rule 5: Circuit breaker check via LiteLLM (router deprecated)
-for cb in fleet.router.circuit_breaker:
+if fleet.summary.circuit_breakers_open > 0:
-  if cb.open and cb.open_duration > 600 and gpu_is_healthy(cb.gpu):
+  push actions check-litellm-circuit-breakers()
    push actions apply-cb-reset(cb)
 -- Rule 6: Strix Halo
 if not fleet.strix.running and pingable("192.168.68.15"):
  push actions apply-strix-restart()
-- Rule 7: Prometheus exporters
+-- Rule 7: GPU data source
-for gpu in fleet.gpus:
+if not fleet.gpus or len(fleet.gpus) < 2:
-  if not prometheus_reachable(gpu.hostname, 9400):
+  push actions check-gpu-monitor-service()
    push actions apply-exporter-restart(gpu)
 -- Rule 8: Predictive thermal
 for gpu in fleet.gpus:
@@ -212,12 +235,12 @@ call update-gpu-health
 ```json
 {
-  "run_id": "gpu-self-heal-20260712-001",
+  "run_id": "gpu-self-heal-20260718-001",
-  "timestamp": "2026-07-12T16:00:00Z",
+  "timestamp": "2026-07-18T08:00:00Z",
  "gpu": "ct8-rtx3090",
  "issue": "thermal-critical",
  "detected": { "temp_c": 87, "duration_s": 180 },
-  "action": "set-fan-100pct",
+  "action": "load-shedding",
  "result": "resolved",
  "verification": { "temp_c": 76, "after_s": 300 },
  "escalated": false
@@ -236,52 +259,53 @@ Every action logged as `[GPU-SELF-HEAL] <run_id>` node with full audit trail.
 - `issues_escalated > 0` → "⚠ GPU Self-Heal — <gpu> needs attention"
 - Every 100th clean cycle → "✅ GPU Fleet: All Clear"
-### 3. Prometheus/Grafana Integration
+### 3. Weekly Benchmark Report
 - GPU self-heal actions exposed as Prometheus counter metrics
 - Dashboard panel: "GPU Interventions (24h)" showing count/type/result
 ### 4. Weekly Benchmark Report
 - Per-GPU tok/s trend over 7 days
 - Regression alerts if any GPU degrades >10% week-over-week
 ---
-## Design Decisions (Grilled & Confirmed — 2026-07-12)
+## Design Decisions (Verified 2026-07-12, Reaffirmed 2026-07-18)
 1. **Fan control**: ❌ NO auto fan control. Load-side cooling only (reduce concurrency, redirect).
 2. **Model restart**: ✅ Only if >50% failure rate over 60s + 30s grace period. Not on single stuck request.
 3. **Strix direct access**: ✅ Open firewall .15:8080 → .24 for direct health probe + restart.
-4. **VRAM thresholds**: Tiered — 100MB/h (RTX 3090), 50MB/h (RTX 5070), 200MB/h (Strix).
+4. **VRAM thresholds**: Tiered — **300MB/h** (RTX 3090), **300MB/h** (RTX 5070), 200MB/h (Strix). Previous values (100/50) were too sensitive; raised 2026-07-18 based on operational data.
-5. **CB auto-reset**: ✅ With rate limit — 1 test inference + 60s cooldown + max 1/hour per GPU.
+5. **CB auto-reset**: ✅ Router deprecated — circuit breakers go through LiteLLM health check + container restart if needed. No per-GPU auto-reset.
-6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Original baseline kept in Grafana.
+6. **Benchmark baseline**: Rolling 30-day average, recalculated weekly. Current baselines live in gpu-monitor.
 7. **Predictive alerts**: Two-tier — warn at >70°C+rising (>2°C/min), critical at >80°C+rising.
-8. **Prometheus**: Primary source. Fall back to nvidia-smi/rocm-smi direct probes if exporter down.
+8. **Prometheus**: ❌ Not deployed. Use direct sidecar probes (:8080/health) and gpu-monitor API. Prometheus integration deferred until exporters are running on GPU hosts.
-## Lessons Learned (2026-07-12)
+## Lessons Learned (2026-07-12, Updated 2026-07-18)
 ### L1: API Key Standardization Is Critical
 - All GPU llama-servers MUST use the same api-key as the LiteLLM config.
 - RTX 5070 had `--api-key sk-loc...5678` while LiteLLM sent `not-needed`.
-  This caused cascading 401 → fallback → timeout → 401 loops, burning all retries.
+  This caused cascading 401 → fallback → timeout → 401 loops.
- **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config.
+- **Rule**: Any new GPU or model restart MUST verify api-key matches LiteLLM config (`not-needed` for direct routing).
 ### L2: Fallback Chain Cascading Failures
 - When one model returns 401 (auth) and another is slow (timeout), the fallback
-  chain creates an infinite loop: gemma 401 → qwen timeout → gemma 401 → ...
+  chain creates an infinite loop.
 - **Rule**: If a model returns 401 (auth error), do NOT fall back to it again.
  Mark it as permanently failed for this request.
 ### L3: Verify Running State, Not Docs
- RTX 3090 was documented at 128K context. Actually running at 256K.
+- RTX 3090 was documented at 128K context. Running at 128K (verified 2026-07-18).
- Parallel count wrong (docs said 2, actual is 1 on RTX 3090).
+- Parallel count: 1 on both RTX 3090 and RTX 5070 (matches docs for current models).
 - **Rule**: Before making decisions, check `/proc/PID/cmdline` on GPU hosts.
 ### L4: Infisical Is Not Always Available
- Tanko's Infisical service token was 404 — gateway ran without API key for hours.
+- Keep a local `.env` fallback for `LITELLM_API_KEY`.
- **Rule**: Always keep a local `.env` fallback for `LITELLM_API_KEY`.
+- **Rule**: Always verify credential source is reachable before relying on it.
 - Contract hermes-config-template Rule 3 updated.
-### L5: Zulip Event Queue Can Silently Die
+### L5: GPU Monitor Response Size Can Cause Self-Heal Crash
- Mumuni's queue accumulated 41 errors/reconnects then stopped polling.
+- gpu-self-heal crashed with KeyboardInterrupt during json.loads() of 20MB response.
-  Gateway was running but ignoring all messages.
+- Root cause: router poll returns accumulated data → cache balloons.
- **Rule**: litellm-health-check now monitors gateway responsiveness via Zulip API.
+- **Rule**: Self-heal must enforce a read timeout AND max response size on every poll.
  If monitor response > 1MB, log a warning and skip the cycle rather than crashing.
 ### L6: Stable Aliases Replace Model Names
 - gpu-fleet introduced stable aliases (strix-moe, gpu-dense, gpu-light) on 2026-07-15.
 - Self-heal must use aliases for reporting and alerting, not model-specific names.
 - **Rule**: All alert messages and KG nodes use the stable alias as the GPU identifier.
@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
      "models": [
        { "id": "syslog-auto" },
        { "id": "strix-moe" },
        { "id": "gpu-dense" },
        { "id": "gpu-light" },
        { "id": "qwen3.6-27B-code" },
        { "id": "gemma-4-12b" }
      ]
@@ -6,11 +6,10 @@ description: >
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
  UPDATED 2026-07-16: Compression model is the stable alias `strix-moe` (NOT `ornith-1.0-35b`,
-  which LiteLLM does not serve). All 3 GPUs verified 256K (RTX 5070 bumped 131K→256K on Jul 15).
+  which LiteLLM does not serve). All 3 GPUs verified at 128K (reduced from 256K 2026-07-17 for stability).
  Added Rule 12 (Context-Issue Diagnostic) + Rule 13 (.env fallback enforcement) from the
  2026-07-16 Mumuni root-cause investigation (WAL #1300).
-  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context
+  UPDATED 2026-07-12: GPU workload redistributed. Compression → Strix Halo. RTX 3090 context verified at 128K. Infisical .env fallback required (Rule 3/13).
  verified at 256K. Infisical .env fallback required (Rule 3/13).
 ---
 ## Maintains
@@ -36,7 +35,7 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 | Mumuni | `mumuni` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
 | Abiba | `abiba-pi` | 192.168.68.24 | local | — |
 | Koby | `koby` | CT 111 (tdunna) | Zulip | — |
-| Koonimo | `koonimo` | CT 113 (baggy) | Zulip | — |
+| Koonimo | `koonimo` | CT 114 (baggy) | SSH root | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |
 > CT hostnames (tdunna, baggy) differ from agent identities (koby, koonimo).
@@ -101,8 +100,8 @@ model:
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Injected via infisical run -- wrapper
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
-  context_length: 262144          # For syslog-auto (all GPUs support 256K).
+  context_length: 131072          # For syslog-auto (all GPUs at 128K for stability).
-                                  # Set 131072 if using gemma-4-12b directly (12GB VRAM constraint).
+                                  # Set 65536 if using gemma-4-12b directly (tight VRAM).
 fallback_providers:
  provider: deepseek
@@ -133,7 +132,7 @@ compression:
  enabled: true
  model: strix-moe               # ⚠️ Must match auxiliary.compression.model. Stable alias (gpu-fleet § Stable Role-Based Aliases). NOT ornith-1.0-35b (LiteLLM does not serve that name).
  provider: harness
-  max_context_window: 262144      # MUST match actual GPU capacity. All 3 GPUs are 256K (Jul 15).
+  max_context_window: 131072      # MUST match actual GPU capacity. All 3 GPUs are 128K (Jul 17).
  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
  target_ratio: 0.30
  protect_last_n: 40
@@ -250,7 +249,7 @@ The following MUST be identical across ALL profiles:
 ### Rule 7: Auxiliary Model Consistency (UPDATED 2026-07-16)
 - Vision and web_extract use `gemma-4-12b` (RTX 5070 — 12GB, vision-optimized)
- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 256K ctx, compression-optimized)
+- Compression uses `strix-moe` (stable alias for Strix Halo — 64GB, 128K ctx, compression-optimized)
 - **`strix-moe` is the only valid compression model name** — LiteLLM does NOT serve `ornith-1.0-35b`
  (it serves `strix-moe`, `qwen3.6-35B-udq4`, `gpu-dense`, `gpu-light`, `syslog-auto`, `gemma-4-12b`, `qwen3.6-27B-code`). Old configs with `ornith-1.0-35b` cause 403/model-not-found on compression calls.
 - All auxiliary services MUST use identical routing:
@@ -258,31 +257,31 @@ The following MUST be identical across ALL profiles:
  - `api_key_env: LITELLM_API_KEY`
 - **Do NOT use `syslog-auto`** for auxiliary tasks — it routes unpredictably
 - **Compression on Strix Halo**: The strix-moe alias routes to Strix Halo
-  (64GB UMA, 256K context) — the designated compression GPU. This frees the
+  (64GB UMA, 128K context) — the designated compression GPU. This frees the
  RTX 5070 for vision and web search, and the RTX 3090 for heavy reasoning.
 - The `compression:` block's `model` MUST match `auxiliary: compression: model`
- The `compression: max_context_window: 262144` MUST match Strix Halo's actual capacity
+- The `compression: max_context_window: 131072` MUST match actual GPU capacity (128K)
 ### Rule 8: GPU Workload Distribution (UPDATED 2026-07-16)
- **RTX 3090 (24GB, 256K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
+- **RTX 3090 (24GB, 128K ctx, qwen3.6-27B-code)**: Heavy reasoning, code gen, long conversations
- **RTX 5070 (12GB, 256K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (bumped 131K→256K Jul 15; IQ4_NL+MTP, 88% VRAM)
+- **RTX 5070 (12GB, 128K ctx, gemma-4-12b)**: Vision, web search, quick tasks, web_extract (IQ4_NL+MTP, ~65% VRAM at 128K)
- **Strix Halo (64GB, 256K ctx, strix-moe)**: Context compression, summarization, long docs
+- **Strix Halo (64GB, 128K ctx, strix-moe)**: Context compression, summarization, long docs
 - Agent profiles MUST route auxiliary tasks to the correct GPU:
  - `auxiliary.vision.model: gemma-4-12b` (RTX 5070)
  - `auxiliary.web_extract.model: gemma-4-12b` (RTX 5070)
  - `auxiliary.compression.model: strix-moe` (Strix Halo)
 - Default model (`model.default`) and custom_provider remain `syslog-auto` for auto-routing
- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
+- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
+- Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin
- `max_context_window: 262144` MUST match the model's actual capacity
+- `max_context_window: 131072` MUST match the model's actual capacity (128K)
 - See `devops-hermes-compression` skill for full reference
-### Rule 9: Compression Threshold for 256K Models
+### Rule 9: Compression Threshold for 128K Models
- For 262K context window: `threshold: 0.65` (fires at ~170K tokens)
+- For 128K context window: `threshold: 0.65` (fires at ~85K tokens)
 - Do NOT use `threshold: 0.25` — this fires at 65K, causing premature context loss
- Do NOT use `threshold: 0.80` — this delays until 209K, risking the gateway hygiene layer
+- Do NOT use `threshold: 0.80` — this delays until ~105K, leaving only 23K margin
- `max_context_window: 262144` MUST match the model's actual capacity (Strix Halo = 256K)
+- `max_context_window: 131072` MUST match the model's actual capacity (all GPUs = 128K)
 - See `devops-hermes-compression` skill for full reference
 ### Rule 10: Default Model Must Be `syslog-auto` (All Agents)
@@ -312,7 +311,7 @@ When an agent shows "context issues" (premature compression, 401s, 504s, DeepSee
 verify ALL FOUR of these against the live config. They are the only root causes found in production:
 1. **max_context_window correct?** — BOTH `compression.max_context_window` AND `context.max_context_window`
-   MUST be `262144` (all GPUs are 256K). A value of `131072` causes premature compression at
+   MUST be `131072` (all GPUs are 128K). A value of `262144` causes instability near 100K and must NOT be used.
   ~83K instead of ~170K. Check: `grep -n max_context_window ~/.hermes/config.yaml`
 2. **base_url uses /v1 NOT /litellm/v1?** — `custom_providers[0].base_url`, `delegation.base_url`,
   and ALL `auxiliary.*.base_url` MUST be `http://192.168.68.116/v1` (Rule 5). nginx `/litellm/`
@@ -326,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
 One-line agent health check (run on the agent host):
 ```bash
-PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
+# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
 PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
 cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
 curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
 ```
@@ -352,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
 The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
 See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
 **⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
 the wrapper will inject an empty key and the gateway will silently get 401 errors
 on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
 is present but the vault takes precedence when the secret key exists (even if empty).
 **Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
 is empty or shorter than 20 chars, log a warning and either fail with a clear error
 message or fall back to the `.env` value before starting the gateway.
 **Verification (all agents):**
 ```bash
-GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
+# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
 GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
 K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
 curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models  # must be 200
 ```
@@ -0,0 +1,104 @@
 ---
 name: inference-optimization
 kind: responsibility
 description: >
  Optimizes the full Syslog inference stack — LiteLLM routing weights, GPU model
  assignments, agent context management, and prompt caching — to reduce response
  times to sub-15s average. All GPUs now at 128K context (stable ceiling).
 id: 067NC6KP02RG60S50M40E30928
 ---
 ### Goal
 Syslog inference response times reduced to sub-15s average by optimizing the full
 stack: LiteLLM routing weights, GPU model assignments, Hermes agent context
 management, and prompt caching — without sacrificing agent capability.
 ### Requires
 - `inference-metrics`: current SpendLogs from CT116 LiteLLM Postgres — avg
  request_duration_ms, prompt_tokens, completion_tokens, model_group breakdown,
  cache_hit rate over the last 3 hours
 - `agent-configs`: current config.yaml from each active Hermes agent (Mumuni
  .123, any others on .129/.122) including compression, model, context_window,
  prompt_caching, memory settings
 - `gpu-health`: health check response from all 3 GPU backends (ornith .15:8080,
  qwen .8:8080, gemma .110:8080)
 ### Maintains
 The optimized inference stack configuration — every change is applied and
 verified end-to-end. Postcondition: avg request_duration_ms ≤ 15000 for 90% of
 non-ornith traffic; ≤ 30000 for ornith-bound agentic calls.
 #### liteLLM-routing
 The syslog-auto routing weights, model-specific timeouts, RPM limits, and
 model_list entries on CT116 `/opt/inference-harness/litellm_config.yaml`.
 #### agent-compression
 Each Hermes agent's `~/.hermes/config.yaml` compression, context_window,
 prompt_caching, and model sections.
 #### prompt-caching
 LiteLLM cache configuration and llama.cpp `--cache-prompt` flag on GPU hosts.
 #### verification
 End-to-end latency measurements after changes applied — at least 3 test
 inference calls per model path measuring ttft (time-to-first-token) and total
 duration.
 ### Continuity
 - input-driven
 ### Strategies
 **Context is the root cause.** Every ~46K prompt token costs ~87s of ornith
 prefill time at 532 tok/s. Fix context first, routing second.
 - **Route by task**: ornith for multi-step reasoning only; qwen for code/standard
  queries; gemma for compression/auxiliary. Never send simple completion to a
  35B MoE.
 - **Compress aggressively**: threshold at 40% (not 65%) — a 256K window should
  compact at 102K, not 166K. Target 15% tail (not 30%).
 - **Cache everything repeated**: system prompts, skill docs, AGENTS.md — these
  never change between turns. Single-digit cache hit rate is unacceptable.
 - **Lower context ceiling**: 128K window is the stable ceiling for agent conversations.
  GPUs reduced from 256K to 128K (2026-07-17). For larger contexts, route to external providers.
 ### Shape
 - `self`: analyze metrics, compute optimal configs, apply changes, verify
 - `delegates`:
  - `apply-liteLLM`: update litellm_config.yaml and reload
  - `apply-agent-config`: update hermes config.yaml per agent
  - `verify-latency`: run test inference calls and measure response
 ### Execution
 ```prose
 -- Phase 1: Analyze current state (already complete)
 -- Phase 2: Apply LiteLLM routing optimization
 call apply-liteLLM-routing
  config_path: /opt/inference-harness/litellm_config.yaml
  host: 192.168.68.116
 -- Phase 3: Apply agent context compression optimization  
 call apply-agent-compression
  agent: mumuni
  host: 192.168.68.123
  config_path: /root/.hermes/config.yaml
 -- Phase 4: Enable llama.cpp prompt caching on GPU hosts
 call enable-prompt-caching
  hosts: [192.168.68.15, 192.168.68.8, 192.168.68.110]
 -- Phase 5: Verify end-to-end latency
 call verify-latency
  host: 192.168.68.116
  models: [syslog-auto, qwen3.6-27B-code, gemma-4-12b, ornith-1.0-35b]
 ```
@@ -0,0 +1,200 @@
 ---
 kind: responsibility
 name: infrastructure-maintenance
 description: >
  Weekly system-level maintenance for the Syslog inference fleet: OS package
  updates on the primary host, Docker image pulls for LiteLLM/SearXNG and other
  running containers, container restarts with health verification, post-update
  verification of every critical service (LiteLLM proxy, SearXNG, Zulip, Gitea,
  PM2 processes, Hermes gateways), and rollback on failure. Consolidates the
  raw shell scripts that previously did this piecemeal. This contract owns the
  HOST-LEVEL weekly maintenance loop on the primary host plus Docker image
  pulls ONLY for .116 and .7, while infrastructure-update owns the FULL-FLEET
  cluster-wide wave (apt across the full PVE cluster + CTs/VMs AND its Docker
  image Wave 3 across all stacks). Runs Sunday 2am ET. Owner:
  ops (firstmate secondmate). Blast radius: an unverified image pull can break
  LiteLLM (all agents lose inference) or SearXNG (search-stack down); a bad apt
  upgrade can leave the host in a half-upgraded state. Pre-update backup check
  and rollback are mandatory for this reason.
 agent: ops
 triggers:
  - weekly (Sunday 02:00 ET) via cron
  - on demand when ops/abiba triggers "infra maintenance"
 version: 1.0.0
 ---
 ## Maintains
 - maintenance-status: { phase: idle|preflight|apt|images|restarts|verify|rollback|done|failed, host, step, result, timestamp }
 - image-baseline: { service, current_tag, pulled_tag, digest, updated_at } — last known-good image per container
 - apt-state: { upgradable_before, upgradable_after, held_broken, kernel_reboot_required }
 - health-baseline: snapshot of critical-service health captured pre-update (used for regression check post-update)
 - rollback-snapshot: { backup_path, configs, image_digests, timestamp } — restore point created in preflight
 - maintenance-history: array of past runs with phase results and any escalations
 ## Scope
 Primary host is the maintenance host where apt updates apply. Docker image pulls
 span the two Docker ecosystems that run critical services. infrastructure-update
 runs the full-fleet cluster-wide wave (including its Wave 3 Docker pulls across
 all stacks/hosts); this contract runs a narrower host-level weekly pull limited
 to .116 and .7. Topology, CT IDs, and IPs are live-state fields — verify against
 `infrastructure-control.prose.md` (the source of truth) and the live system
 before mutating.
 | Host | IP | Role | Trust |
 |------|----|------|-------|
 | CT 116 (syslog-api) | 192.168.68.116 | LiteLLM proxy + Grafana + Prometheus (inference harness) | ⚠️ VERIFY-BEFORE-USE |
 | VM 109 (docker-vm) | 192.168.68.7 | SearXNG + Firecrawl + home stack (Docker host) | ⚠️ VERIFY-BEFORE-USE |
 | CT 117 (zulip) | 192.168.68.19 | Zulip (storepve bridge IP .19) | ⚠️ VERIFY-BEFORE-USE |
 | Gitea | https://git.sysloggh.net | Prose-contracts + agent configs source control | ⚠️ VERIFY-BEFORE-USE |
 | CT 100 (abiba/pi) | 192.168.68.24 | PM2 processes (pi agent harness) | ⚠️ VERIFY-BEFORE-USE |
 > "Primary host" for the apt phase is the host the ops agent runs maintenance
 > from. Confirm which host that is against infrastructure-control before
 > running; do not assume. If the ops agent is containerized/CT-based, apt runs
 > inside that CT.
 ## Requires
 - SSH/exec access to CT 116 (.116) and VM 109 (.7) for Docker operations
 - `apt`, `docker`, `docker compose` available on target hosts
 - LiteLLM master key available (Infisical vault, `LITELLM_API_KEY`) for health verification
 - `infrastructure-monitoring` run completed within the last 30 minutes — provides the pre-update health baseline used by the regression check
 - Writable backup directory `/tmp/infra-maintenance-backup-<date>/` on each mutated host
 - Proxmox snapshot of the primary host available (or confirmed not required) before apt phase
 ## Continuity
 - Self-driven: weekly cron `0 2 * * 0` (Sunday 02:00 ET)
 - Also wakes on: explicit "infra maintenance" trigger from ops/abiba
 - Depends on `infrastructure-monitoring` for the pre-update health baseline — do not run if the last monitoring run is stale (>30 min) or RED; abort and escalate instead
 ## Execution
 ### Phase 0 — Preflight (snapshot/backup check + health baseline)
 1. **Capture health baseline** — run the `infrastructure-monitoring` postcondition checks (LiteLLM, Zulip, Gitea, SearXNG, Proxmox API) and record results as `health-baseline`. If any critical service is already down, **abort**: maintenance must not run on a degraded fleet.
 2. **Backup check** — confirm a Proxmox snapshot of the primary host exists OR `/tmp/infra-maintenance-backup-<date>/` was created this run. Snapshot critical config files into the backup dir:
   - `/opt/inference-harness/docker-compose.yml`, `/opt/inference-harness/litellm_config.yaml` (CT 116)
   - `/opt/search-stack/searxng/docker-compose.yml`, `/opt/search-stack/firecrawl-source/docker-compose.yaml` (VM 109)
 3. **Record image baseline** — `docker inspect --format '{{.Image}} {{.Config.Image}}' <container>` for every running container on .116 and .7; store digests in `image-baseline` so rollback can restore them.
 4. **Disk check** — `df -h` on each mutated host; abort if free space <20% (apt upgrade + image pulls need headroom).
 ### Phase 1 — OS package updates (primary host)
 ```bash
 # On the primary host only (VERIFY host against infrastructure-control first)
 apt update
 apt upgrade -y
 ```
 - Capture `apt list --upgradable` before and after → store in `apt-state`.
 - If apt reports held/broken packages (`apt-get -s upgrade | grep -i broken`, or non-zero exit), **stop** — do not force. Record `held_broken` and go to rollback/escalate.
 - If `/var/run/reboot-required` exists after upgrade, flag `kernel_reboot_required: true` in `apt-state` but **do not reboot automatically** — that's a separate coordinated action (see infra-update Wave 4). Note it in the report.
 ### Phase 2 — Docker image pulls
 Pull latest stable tags for every running container. Do NOT pin to `:main`/`:nightly` — use stable tags where the compose file specifies them; otherwise `latest`.
 ```bash
 # CT 116 (.116) — inference harness
 cd /opt/inference-harness && docker compose pull
 # VM 109 (.7) — search + home stacks
 cd /opt/search-stack/searxng && docker compose pull
 cd /opt/search-stack/firecrawl-source && docker compose pull
 # any other running stacks on .7 (home stack, audiobookshelf) — pull per their compose files
 ```
 - LiteLLM and SearXNG are the two explicitly required pulls; "any other running containers" means every stack with a compose file on .116 and .7.
 - Record pulled tag + digest per service in `image-baseline`.
 ### Phase 3 — Container restarts with health verification
 Restart one stack at a time, verify health before moving to the next. Do not restart everything at once — a failure mid-wave must leave the rest running.
 ```bash
 # CT 116
 cd /opt/inference-harness && docker compose up -d
 # VM 109
 cd /opt/search-stack/searxng && docker compose up -d
 cd /opt/search-stack/firecrawl-source && docker compose up -d
 ```
 After each stack comes up, wait for health (max 120s):
 - `docker ps` shows the container `Up` (and `healthy` if a healthcheck is defined)
 - Service-specific probe passes (see Phase 4 probes)
 If a stack fails to come up within 120s, **stop the wave** and go to rollback for that stack only; do not proceed to the next.
 ### Phase 4 — Post-update service verification
 After ALL updates (apt + images + restarts), verify every critical service is back up and matches the pre-update baseline. This is the regression gate.
 | Service | Probe | Expect |
 |---------|-------|--------|
 | LiteLLM proxy | `curl -sf http://192.168.68.116/litellm/v1/models` | 200 OK, models returned |
 | LiteLLM MCP gateway | `curl -sf http://192.168.68.116:4000/mcp-rest/tools/list -H "Authorization: Bearer $MASTER_KEY"` | 90 tools (23 RA-H OS + 67 GitHub) |
 | SearXNG | `curl -sf http://192.168.68.7:8888` | 200 OK |
 | Zulip | `curl -sf https://chat.sysloggh.net/api/v1/server_settings` | 200 OK |
 | Gitea | `curl -sf https://git.sysloggh.net/api/v1/version` | 200 OK |
 | PM2 processes | `pm2 jlist` (CT 100) | all pi-agent processes `online` |
 | Hermes gateways | SSH to Mumuni CT 114, Tanko CT 112; `systemctl is-active hermes-gateway` | `active` for each |
 Regression check: every service that was GREEN in `health-baseline` must still be GREEN. A service that was already RED (and caused a preflight abort) is excluded — but Phase 0 should have aborted before we got here.
 ## Rollback Protocol
 If ANY service in Phase 4 fails to come back up (or regresses vs baseline):
 1. **Image rollback** — for the failing stack, restore the previous image:
   ```bash
   # Restore from recorded image-baseline digest
   docker compose down
   # Pin the service image to the recorded digest in compose, then recreate
   #   image: <name>@sha256:<previous_digest>
   docker compose pull && docker compose up -d
   ```
 2. **APT rollback** — restore the primary host from the Proxmox snapshot taken/confirmed in Phase 0. If no snapshot, `apt install <pkg>=<old_version>` per package using apt history (`/var/log/apt/history.log`).
 3. **Config rollback** — restore configs from `/tmp/infra-maintenance-backup-<date>/`.
 4. **Re-verify** — re-run the Phase 4 probes on the rolled-back service. If still failing, escalate (do not loop — circuit breaker below).
 5. **Escalate** — send a Zulip DM to abiba + mumuni with: failing service, phase, baseline vs current, rollback actions taken, backup path.
 ## Circuit Breaker
 - `max_retries: 2` per failing phase — after 2 rollback attempts on the same service, stop and escalate.
 - `window: 7200` seconds — no more than 2 retries within a 2-hour window.
 - `trip_action: escalate_to_fatal` — when tripped, escalate to fatal (abiba + mumuni + kwame) and pause; a human must clear before the next scheduled run.
 ## Report
 After completion (or on abort), emit a receipt (JSON) to `~/.hermes/runs/infrastructure-maintenance/` and send a Zulip DM summary:
 ```
 🛠 Infrastructure Maintenance — YYYY-MM-DD
 Phase: apt | images | restarts | verify | rollback
 Primary host: <host>
 APT: <N> packages upgraded, <M> held/broken, kernel_reboot_required=<bool>
 Images pulled: LiteLLM <tag>, SearXNG <tag>, <others>
 Services: all GREEN | <service> FAILED (rolled back)
 Baseline regression: none | <details>
 Backup: /tmp/infra-maintenance-backup-YYYYMMDD/
 Escalation: none | warning | critical | fatal
 ```
 ## Verification Postconditions
 - All critical services running after update (Phase 4 all GREEN)
 - No regressions from pre-update health baseline (Phase 0 baseline)
 - Docker containers on latest stable tags (`image-baseline.pulled_tag` recorded)
 - APT packages up to date with no held broken packages (`apt-state.held_broken == 0`)
 ## Related Contracts
 - `infrastructure-update.prose.md` — owns the full-fleet cluster-wide wave INCLUDING its Wave 3 Docker image updates across all stacks (SearXNG, Firecrawl, Inference Harness on .116, home stack, audiobookshelf); infrastructure-maintenance is a deliberately narrower host-level weekly pull scoped to .116 and .7.
 - `infrastructure-monitoring.prose.md` — provides the pre-update health baseline (depends_on).
 - `infrastructure-control.prose.md` — topology source of truth (CT IDs, IPs, hostnames).
 - `litellm-health.prose.md` — LiteLLM probe details.
 - `proxmox-monitor.prose.md` — Docker stats + monitoring stack health.
@@ -20,9 +20,20 @@ description: >
  Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
  Abiba's key is now a proper agent key (NOT the master key — stale note removed).
  UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
  standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
  infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
  injection → .env fallback → exec python. Systemd drop-ins are IMMUNE to hermes gateway
  install which overwrites the unit file ExecStart. Infisical CLI updated to 0.43.109 on
  all agents (was 0.38.0). Service token st.8e848433 shared across fleet (st.353699cd
  for tanko was deleted). .env fallback on every agent protects against token loss.
  Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
  — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
  (3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
  Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
  Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
-  on CT 116. Last verified: 2026-07-16.
+  on CT 116. Last verified: 2026-07-17.
 ---
 ## Parameters
@@ -74,88 +85,147 @@ description: >
   - Confirm key alias matches agent_name in LiteLLM key list
   - Verify agent gateway uses vault wrapper: `cat /proc/<pid>/cmdline` shows `infisical run`
-## Production Vault Access Process (canonical, 2026-07-16)
+## Production Vault Access Process (canonical, 2026-07-17)
-The non-fail approach to agentic vault access. Deployed on 4/5 agents (tanko pending —
+The non-fail approach to agentic vault access. Deployed on all 4 Hermes agents
-runs as user `jerome`, not systemd root, needs user-scope adaptation).
+(Mumuni, Tanko, Koby, Koonimo) as of 2026-07-17. Abiba (pi) uses a similar pattern
 through its agent wrapper.
 ### The canonical pattern
-1. **infisical CLI** installed on the host (`/usr/local/bin/infisical` or `/usr/bin/infisical`).
+1. **infisical CLI** installed on the host at `/usr/bin/infisical` (v0.43.109+, from
-2. **Service token** (Infisical Machine Identity, `st.…`) stored root-only at `/root/.infisical-token` (`chmod 600`).
+   artifacts-cli.infisical.com apt repo). Update procedure:
-   - Interim: the shared `abiba` service token (`st.8e848433…`) has READ+WRITE on the `agents` project.
+   ```bash
-   - Proper: one machine identity per agent (create in Infisical UI → Project Settings → Machine Identities).
+   curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | sudo -E bash
-3. **`infisical-gateway.sh` wrapper** at `/root/.hermes/infisical-gateway.sh` (`chmod 700`):
+   sudo apt-get update && sudo apt-get install -y infisical
   # Remove stale old binary if present
   rm -f /usr/local/bin/infisical /bin/infisical
   ```
   Wrappers use absolute path `/usr/bin/infisical run`. Never rely on PATH resolution.
 2. **Service token** (Infisical Machine Identity, `st.…`) stored at `~/.infisical-token`
   (`chmod 600`). Current: shared `st.8e848433…` (abiba, READ+WRITE on agents project).
   Tanko's `st.353699cd…` (tanko-agent) was deleted — reverted to shared token.
   Proper: one machine identity per agent (create in Infisical UI → Project Settings →
   Machine Identities).
 3. **`infisical-gateway.sh` wrapper** at `~/.hermes/infisical-gateway.sh` (`chmod 700`):
   ```bash
   #!/bin/bash
   export INFISICAL_API_URL="https://vault.sysloggh.net"
-   TOKEN=$(cat /root/.infisical-token)
+   TOKEN=$(cat $HOME/.infisical-token)
-   LOG=/root/.hermes/logs/gateway.log; mkdir -p /root/.hermes/logs
+   LOG=$HOME/.hermes/logs/gateway.log; mkdir -p $HOME/.hermes/logs
   while true; do
-     infisical run --token="$TOKEN" --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
+     echo "[$(date -Iseconds)] Starting gateway with Infisical injection..." >> $LOG
     /usr/bin/infisical run --token="$TOKEN" \
       --projectId=322fceab-39da-4854-a55a-568e76c0f13f \
       --env=prod --domain=https://vault.sysloggh.net -- bash -c '
-         . /root/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3] safety net only
+         . $HOME/.hermes/.env 2>/dev/null   # [FALLBACK Rule 3]
-         export LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"
+         export LITELLM_API_KEY="${<AGENT>_LITELLM_API_KEY}"
-         exec <HERMES_VENV>/bin/python -m hermes_cli.main gateway run
+         export ZULIP_API_KEY="${<AGENT>_ZULIP_API_KEY}"
         export ZULIP_SITE="https://chat.sysloggh.net"
         export ZULIP_EMAIL="<agent>-bot@chat.sysloggh.net"
         export SEARXNG_URL="http://192.168.68.7:8888"
         # ⚠️  HARDCODE the full venv path. NEVER use $VENV inside single quotes.
         exec /root/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main gateway run
       ' >> $LOG 2>&1
-     sleep 5   # restart on exit
+     EXIT_CODE=$?
     echo "[$(date -Iseconds)] Gateway exited with code $EXIT_CODE — restarting in 5s..." >> $LOG
     sleep 5
   done
   ```
-4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` (e.g. `KOBY_LITELLM_API_KEY`). Vault = source of truth.
+   **CRITICAL: VENV PATH.** The inner `bash -c '...'` uses single quotes. Shell
-5. **`.env` fallback** at `/root/.hermes/.env` (`chmod 600`) with the same key — safety net ONLY for vault outage (Rule 3/13). Must be kept in sync on rotation.
+   variables set in the outer wrapper are NOT expanded inside single quotes.
-6. **systemd service** `hermes-gateway.service` with `ExecStart=/root/.hermes/infisical-gateway.sh`. NO `litellm-key.conf` drop-in (those hardcode keys and rot).
+   `$VENV/bin/python` resolves to `/bin/python` (file not found). Always hardcode
-7. **NEVER hardcode** LiteLLM keys in systemd drop-ins, config.yaml, or /etc/environment. The wrapper injects live from vault.
+   the absolute path to the venv python binary.
 4. **Agent key in vault** as `<AGENT>_LITELLM_API_KEY` and `<AGENT>_ZULIP_API_KEY`.
   Vault = source of truth for ALL platform credentials.
 5. **`.env` fallback** at `~/.hermes/.env` (`chmod 600`) with agent-specific keys —
   safety net for vault outage or token revocation. Must be kept in sync on rotation.
   Example:
   ```bash
   MUMUNI_LITELLM_API_KEY=sk-OzuWsoX22Hmb3Ps3JY01gw
   MUMUNI_ZULIP_API_KEY=H8dY6V7aHmWNcfgNtJaDBPZ1dGWn0Ttt
   ```
 6. **systemd drop-in** at `~/.config/systemd/user/hermes-gateway.service.d/50-vault-wrapper.conf`:
   ```ini
   [Service]
   ExecStart=
   ExecStart=/root/.hermes/infisical-gateway.sh
   ```
   The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
   then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
   regeneration** by `hermes gateway install` — the drop-in always wins.
   **Why a drop-in instead of editing the unit file:** `hermes gateway install`
   (called during Hermes updates and some self-heal operations) regenerates the
   systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
   Editing the unit file directly is futile — it will be overwritten. The drop-in
   approach explicitly resets ExecStart and sets the wrapper regardless of what the
   main unit file says.
 7. **NEVER hardcode** API keys in systemd drop-ins, config.yaml, or /etc/environment.
   The wrapper injects live from vault at every start.
 ### Why this is non-fail
 - **No rot**: keys pulled live from vault at every gateway start. Rotation = one `infisical secrets set` + `systemctl restart`. No per-host file edits.
- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable.
+- **Survives vault outage**: the `.env` fallback (Rule 3) keeps the gateway running if Infisical is unreachable or the service token is revoked.
- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=on-failure` revive the gateway.
+- **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows the live key; `infisical secrets` shows the vault source.
+- **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
 - **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
 - **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
-### Migration status (2026-07-16)
+### Migration status (2026-07-17)
-| Agent | Host | Pattern | Vault key | Status |
+| Agent | Host | Pattern | Keys | Status |
-|-------|------|---------|-----------|--------|
+|-------|------|---------|------|--------|
-| abiba | .24 | `infisical run` (pi agent wrapper, service token) | ABIBA_LITELLM_API_KEY | ✅ vault-backed |
+| abiba | .24 | pi agent wrapper | ABIBA_LITELLM_API_KEY + ABIBA_ZULIP_API_KEY | ✅ vault-backed |
-| mumuni | .123 | infisical-gateway.sh + user-login machine identity | MUMUNI_LITELLM_API_KEY | ✅ vault-backed |
+| mumuni | .123 | systemd drop-in + while-true wrapper + st.8e848433 | MUMUNI_LITELLM_API_KEY + MUMUNI_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
-| koby | .129 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOBY_LITELLM_API_KEY | ✅ vault-backed, Zulip (tanko-bot@) + Telegram |
+| tanko | .122 | systemd drop-in + while-true wrapper + st.8e848433 (user jerome) | TANKO_LITELLM_API_KEY + TANKO_ZULIP_API_KEY | ✅ vault-backed + .env fallback |
-| koonimo | .114 | infisical-gateway.sh + service token (migrated 2026-07-16) | KOONIMO_LITELLM_API_KEY | ✅ vault-backed |
+| koby | .129 | systemd drop-in + while-true wrapper + st.8e848433 | KOBY_LITELLM_API_KEY, shares TANKO_ZULIP_API_KEY (tanko-bot) | ✅ vault-backed |
-> **Baggy = Koonimo (CT 113).** Deleted `BAGGY_LITELLM_API_KEY` from vault 2026-07-16. Only `KOONIMO_LITELLM_API_KEY` exists — one secret per agent.
+| koonimo | .114 | systemd drop-in + while-true wrapper + st.8e848433 | KOONIMO_LITELLM_API_KEY + KOONIMO_ZULIP_API_KEY | ✅ vault-backed |
 <<<<<<< HEAD
 | tanko | .122 | infisical-gateway.sh + service token (migrated 2026-07-16) | TANKO_LITELLM_API_KEY | ✅ vault-backed |
 =======
 | tanko | .122 | **hardcoded in config.yaml** (runs as user jerome, not systemd) | TANKO_LITELLM_API_KEY | ⚠️ TODO: migrate to user-scope wrapper |
 >>>>>>> origin/master
-### Tanko migration (pending)
+> Tanko runs as user `jerome` — wrapper/token at `~/.hermes/infisical-gateway.sh` and
 > `~/.infisical-token`. Linger enabled (`loginctl enable-linger jerome`) for boot startup.
-Tanko runs the gateway as user `jerome` (not root/systemd), with the key hardcoded in
+### Tanko migration (COMPLETED 2026-07-17)
 `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`, valid but not vault-sourced).
 Migration: create a user-scope systemd service (`~/.config/systemd/user/hermes-gateway.service`)
 with `infisical-gateway.sh` wrapper in jerome's home, token at `~/.infisical-token`, lingering
 enabled (`loginctl enable-linger jerome`) so the user service runs without a login session.
-### Koby migration lessons (2026-07-16)
+Tanko was the last agent migrated from hardcoded keys to vault wrapper.
 Previously: key hardcoded in `/home/jerome/.hermes/config.yaml` (`api_key: sk-CggiHWlamQy…`)
 and `zulip-env.conf` systemd drop-in. Now: user-scope systemd service with drop-in
 `50-vault-wrapper.conf`, `infisical-gateway.sh` wrapper with while-true loop, token at
 `~/.infisical-token`, `.env` fallback at `~/.hermes/.env`. Keys injected live from vault.
 ### Koby migration lessons (2026-07-16, updated 2026-07-17)
 Migrated Koby from hardcoded systemd drop-in → `infisical-gateway.sh` wrapper.
-**Two mistakes I made that broke the agent:**
+**Three mistakes made:**
 1. **Overwrote `/root/.hermes/.env`** without backing it up. The Zulip API key only existed
   in the running process memory — the old .env was minimal (just LiteLLM key). Zulip creds were
   inherited from the pre-migration gateway env, not stored in any file. Lost on restart.
 2. **Only injected `LITELLM_API_KEY`** in the wrapper — forgot Zulip + Telegram credentials.
   Agents need ALL their platform env vars. Missing vars cause silent adapter failures.
 3. (2026-07-17 fix) **VENV variable in single-quoted bash -c**: `exec "$VENV/bin/python"`
   inside single quotes resolved to `exec "/bin/python"` (file not found). Hardcoded full path.
-**How Koby actually connects (2026-07-16):**
+**How Koby actually connects:**
 - Zulip: shares **Tanko's bot** (`tanko-bot@chat.sysloggh.net`, `TANKO_ZULIP_API_KEY=5PeD6f3zo…`).
-  Koby doesn't have its own Zulip bot (koby-bot@ doesn't exist in the swarm config).
+- Telegram: token from `.env` fallback. Allowed users: 6679773481.
 - Telegram: token `828640…` recovered from `.env.bak-20260603` (18KB backup from June 2026).
  Allowed users: 6679773481. Home channel: 6679773481.
 - Both platforms now connect through the wrapper's env injection.
-**Golden rule for gateway restarts:** always `cat /proc/<pid>/environ` before killing the old
+**Golden rules for gateway restarts:**
-process — captures the live env set. Especially important when migrating gateways between
+1. Always `cat /proc/<pid>/environ` before killing the old process — captures the live env set.
-injection mechanisms.
+2. Hardcode venv python path in wrapper — never use variables inside single-quoted bash -c.
 3. Use systemd drop-ins (not unit file edits) to override ExecStart — survives Hermes updates.
 ### Fleet-wide standardization lessons (2026-07-17)
 After auditing all 4 agents, five systemic patterns caused repeated failures:
 1. **Three incompatible startup patterns** coexisted (systemd drop-in, direct python, orphaned wrapper)
 2. **Systemd unit files reverted** by `hermes gateway install` during updates
 3. **VENV variable scoping** broke wrappers on Koby and Mumuni (single-quote bash -c)
 4. **Service token expiry** — Tanko's `st.353699cd` was deleted from Infisical
 5. **No ZULIP_API_KEY** in env on Tanko — wrapper bypassed by systemd direct python
 All resolved by the canonical drop-in + while-true wrapper pattern documented above.
 ### Key rotation procedure (one vault operation with this standard)
@@ -165,69 +235,41 @@ injection mechanisms.
 4. Restart: `systemctl restart hermes-gateway`. The wrapper pulls the new key live.
 5. Verify: `curl -H "Authorization: Bearer sk-NEW" http://192.168.68.116/v1/models` → 200.
-## Machine Identity for Vault Writes (ADDED 2026-07-16, WAL #1300)
+## Machine Identity for Vault Writes (UPDATED 2026-07-17)
-**Problem:** The infisical CLI on agent hosts is logged in as a user session (jerome@sysloggh.com).
+**Current state:** Infisical CLI updated to v0.43.109 on all agents (from v0.38.0).
-In CLI v0.38.0, `infisical secrets set` / `infisical export` fail with "project id missing" / "workspace
+The v0.38.0 bug (user-session auth fails for `secrets set`/`export`) is resolved.
-key 404" — a known bug where user-session auth works for `run` but NOT for `secrets set`. The apt
+Service token `st.8e848433…` (abiba, READ+WRITE) can write to vault from CLI.
 repo only ships 0.38.0, so `apt upgrade` does not help.
-**Proper fix — Machine Identity (Infisical automation best practice):**
+**Proper fix — per-agent Machine Identities:**
-Create a machine identity with READ+WRITE scope on the `agents` project (project_id=
+Create machine identities in Infisical UI → Project Settings → Machine Identities
-`322fceab-39da-4854-a55a-568e76c0f13f`, env `prod`). Store client_id + client_secret securely.
+for each agent with READ-only scope on the `agents` project. Store client_id +
-Then vault writes work from any host:
+client_secret per agent. Then vault writes use the shared abiba identity, and
-```bash
+reads use per-agent identities. This eliminates the single shared token risk.
 # Get a machine-identity access token
 TOKEN=$(curl -fsSL -X POST https://vault.sysloggh.net/api/v1/auth/universal-auth/login \
  -H 'Content-Type: application/json' \
  -d '{"clientId":"<CLIENT_ID>","clientSecret":"<CLIENT_SECRET>"}' | jq -r .accessToken)
 # Write a secret via REST API v3
 curl -fsSL -X PATCH https://vault.sysloggh.net/api/v3/secrets/MUMUNI_LITELLM_API_KEY \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"environment":"prod","secretValue":"sk-<NEW_KEY>","workspaceId":"<WORKSPACE_ID>","type":"shared"}'
 # OR via CLI: infisical secrets set --token=$TOKEN --projectId=322fceab... --env=prod ...
 ```
 <<<<<<< HEAD
 **2026-07-16 UPDATE — vault SYNCED + CLEANED.** The abiba service token (`st.8e848433…`, READ+WRITE)
 can write to the vault. All session-13 rotated keys in vault and validate 200 against LiteLLM.
 4 stale secrets deprecated, 5 personal creds flagged for separate project. Tanko migrated from
 hardcoded keys to `infisical-gateway.sh` wrapper. Koonimo Zulip key restored.
-**Machine Identity:** `8ddb9438-74fc-4ab6-bd74-929e7c47b53b` exists in Infisical UI.
+**Service Token Inventory (2026-07-17):**
-Client secret needed to use universal auth for vault writes from automation.
+| Token ID | Name | Permissions | Used By | Status |
 |----------|------|-------------|---------|--------|
 | `st.8e848433…` | tanko-gateway | READ+WRITE | Mumuni, Tanko, Koby, Koonimo, Abiba | ✅ Active |
 | `st.353699cd…` | tanko-agent | READ-only | — | ❌ Deleted from Infisical |
-**Service Token Inventory (2026-07-16):**
+**Per-agent .env fallback inventory (2026-07-17):**
-| Token ID | Name | Permissions | Used By |
+| Agent | .env Keys |
-|----------|------|-------------|---------|
+|-------|-----------|
-| `st.8e848433…` | tanko-gateway | READ+WRITE | Abiba, Koonimo |
+| Mumuni | MUMUNI_LITELLM_API_KEY, MUMUNI_ZULIP_API_KEY |
-| `st.353699cd…` | tanko-agent | READ-only | Tanko |
+| Tanko | TANKO_LITELLM_API_KEY, TANKO_ZULIP_API_KEY |
-=======
+| Koby | (wrapper injects from vault — .env has Telegram token) |
-Creation requires the Infisical web UI (https://vault.sysloggh.net) under Project Settings →
+| Koonimo | KOONIMO_LITELLM_API_KEY, KOONIMO_ZULIP_API_KEY |
 Machine Identities, or an admin API call. **TODO: create `abiba-automation` machine identity
 and store its credentials in the vault itself (or a root-only file).**
 **Interim (working now):** the `.env` fallback (hermes-config-template Rule 3/13). The
 infisical-gateway.sh wrapper sources `~/.hermes/.env`, so its `<AGENT>_LITELLM_API_KEY`
 overrides a stale vault value.
 **2026-07-16 UPDATE — vault is now SYNCED.** The abiba service token (`st.8e848433…`, READ+WRITE)
 can write to the vault, so the session-13 rotated keys (mumuni `sk-OzuWsoX2…`, koby `sk-BqRRMboTI…`,
 koonimo `sk-OEK7z26n6E…`) are now in the vault as `MUMUNI_LITELLM_API_KEY` / `KOBY_LITELLM_API_KEY` /
 `KOONIMO_LITELLM_API_KEY` and validate 200 against LiteLLM. The vault is the source of truth again.
 Creating a dedicated `abiba-automation` machine identity (via UI) is still the proper long-term fix
 so the shared service token isn't reused across hosts — but it is no longer blocking.
 >>>>>>> origin/master
 ## Key Rotation Log
 | Date | Agent | Action | Notes |
 |------|-------|--------|-------|
-<<<<<<< HEAD
+| 2026-07-17 | fleet | standardize | All 4 agents standardized on systemd drop-in + while-true wrapper + infisical v0.43.109. Removed conflicting zulip-env.conf + litellm-key.conf drop-ins. Added .env fallbacks with ZULIP keys. WAL #1322. |
-| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault (Tt2bUL…). Updated wrapper to inject ZULIP_API_KEY + ZULIP_EMAIL. Restarted gateway → Zulip connected as koonimo-bot@ (bot_id=17). 3 platforms now. |
+| 2026-07-17 | tanko | fix-zulip | Added ZULIP_API_KEY to env (was missing — systemd bypassed vault). Updated wrapper from exec to while-true. Created .env fallback. Removed hardcoded zulip-env.conf drop-in. WAL #1321. |
-| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml key to infisical-gateway.sh wrapper + service token st.353699cd… (tanko-agent). Systemd user service updated, hardcoded key drop-ins removed. Verified LITELLM_API_KEY from vault, 3 platforms connected. |
+| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated. 5 personal creds flagged. |
-| 2026-07-16 | vault | cleanup | 4 stale secrets deprecated: KAGENZ0_LITELLM_API_KEY, HERMES_OPENROUTER_KEY, LITELLM_API_KEY (generic duplicate), ZULIP_API_KEY (generic duplicate). 5 personal creds flagged for separate project. |
+| 2026-07-16 | koonimo | add-zulip | Added KOONIMO_ZULIP_API_KEY to vault. Wrapper injects ZULIP_API_KEY + ZULIP_EMAIL. 3 platforms. |
-=======
+| 2026-07-16 | tanko | migrate | Migrated from hardcoded config.yaml to infisical-gateway.sh + st.353699cd. NOTE: st.353699cd later deleted — reverted to st.8e848433 on 2026-07-17. |
 >>>>>>> origin/master
 | 2026-07-16 | mumuni | rotate | Old key malformed (sk-_SWAl_Vu_, 47 chars, not LiteLLM format) → 401. Deleted old `mumuni` key (token 15cbca18…), generated fresh (alias `mumuni`, 7 models: syslog-auto, qwen3.6-27B-code, gemma-4-12b, strix-moe, gpu-dense, gpu-light, qwen3.6-35B-udq4). New key sk-OzuWsoX2… written to /root/.hermes/.env (Rule 3/13 fallback). Vault sync PENDING (needs machine identity). WAL #1300. |
 | 2026-07-16 | koby | rotate | Old key sk-6sbCNjz (401, stale in /etc/environment). Deleted old `koby` key, generated fresh (alias `koby`). New key sk-BqRRMboTI… in systemd drop-in `hermes-gateway.service.d/litellm-key.conf` + /etc/environment. Created `hermes-gateway.service` unit (was missing — gateway wasn't persistent) with `--replace`. Verified HTTP 200, Telegram connected. |
 | 2026-07-16 | baggy (koonimo) | rotate | Old key sk-krnw_zGB (401, hardcoded in systemd drop-in). Deleted old `baggy` key, generated fresh (alias `baggy`, metadata agent=koonimo). New key sk-OEK7z26n6E… in drop-in `hermes-gateway.service.d/litellm-key.conf`. CT113 IP changed .113→.114. Verified HTTP 200, Zulip connected. |
@@ -184,8 +184,8 @@ def check_agents():
            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
            continue
-        # Gateway process
+        # Gateway process (exclude the infisical bash wrapper that contains the same string)
-        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
+        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
        if not pid:
            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
            FAIL.append(f"gateway-down:{name}")
@@ -409,3 +409,68 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
 | Queue expiry handling | Crash | Auto re-register |
 | Busy worker deadlock | Router death | Worker SIGKILL + error DM |
 | PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |
 ---
 ## Incident Log — 2026-07-18 Fleet-Wide Audit
 ### Fleet State After Audit
 | Agent | Platform | Zulip State | Issues Found | Fix Applied |
 |-------|----------|-------------|--------------|-------------|
 | **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
 | **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
 | **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |
 ### Key Fixes Applied
 **1. Abiba — Credential Fallback (L4 Pattern)**
 - Root cause: `zulip.api_key` in config.yaml is `""` (expected from Infisical). Infisical vault `ABIBA_ZULIP_API_KEY` wasn't being injected into the process environment.
 - Fix: Added `.env` file fallback at `/root/.pi/agent/extensions/zulip/.env` with known-working key, sourced before the Infisical `exec`.
 - Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local `.env` fallback.
 **2. Abiba — Poll Timeout Handling**
 - Root cause: Zulip long-poll uses `AbortSignal.timeout(65000)`. Zulip's default `event_queue_longpoll_timeout_seconds` can exceed 65s. When the signal fires, an `AbortError` is thrown and caught by the circuit breaker as a failure.
 - Fix: Caught `AbortError` inside `poll()` and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default.
 - Reference: [Zulip Events System — long-poll timeout](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html)
 **3. Tanko — Gateway Restart**
 - Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (`infisical-gateway.sh`) restarts on crash but the gateway wasn't re-establishing Zulip on restart.
 - Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully.
 ### Fleet-Wide Zulip Health Metrics (as of 2026-07-18)
 | Metric | Value |
 |--------|-------|
 | Zulip server | ✅ HTTP 200 |
 | Agents connected | 3/3 (Abiba, Tanko, Mumuni) |
 | Abiba circuit breaker | CLOSED (0 failures) |
 | Abiba uptime | 2D (post-restart) |
 | Tanko gateway uptime | Ongoing |
 | Mumuni gateway uptime | Ongoing |
 | Watchdog status | ✅ Online (2D uptime) |
 ### Hermes Agent Zulip Plugin Improvements
 Based on the audit, improvements that should be ported to all Hermes Zulip adapters:
 1. **Circuit breaker pattern** — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff.
 2. **Credential fallback** — All Hermes agents use Infisical for credentials. Add `.env` local fallback per L4 pattern for `ZULIP_API_KEY`.
 3. **Queue re-registration** — Handle `BAD_EVENT_QUEUE_ID` with automatic re-registration instead of gateway restart.
 4. **Supervisor watchdog** — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks.
 5. **Streaming** — All agents have `streaming: true` in their zulip config. Verify `edit_message()` is implemented in each adapter.
 ### Abiba pi Zulip Extension v2 — Implemented Resilience Summary
 | Feature | Status | Notes |
 |---------|--------|-------|
 | Circuit breaker | ✅ | CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout |
 | Retry with jitter | ✅ | 2 attempts, 200ms base, 50-100% jitter |
 | Queue lifecycle | ✅ | 10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling |
 | Crash prevention | ✅ | uncaughtException + unhandledRejection recovery |
 | Worker timeout | ✅ | 5min busy timeout → SIGKILL + error DM |
 | Health endpoint | ✅ | :9200 with circuit breaker metrics |
 | Echo prevention | ✅ | Dynamic bot user resolution |
 | Poll timeout (AbortError) | ✅ v2.1 | Normal timeout returns [] instead of error |
 | Credential fallback | ✅ v2.1 | .env file before Infisical exec |
 | Provider auto-fix | ✅ | Detects reasoning_content models, switches to compatible |
Author	SHA1	Message	Date
jerome	5d2ecbace6	Merge branch 'master' into fm/infra-maint-contract PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details	2026-07-18 23:12:20 +00:00
root	99789a00a1	no-mistakes(document): Reframed infra-maintenance contract as deliberate partition PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details	2026-07-18 22:16:09 +00:00
root	a68879e904	no-mistakes(review): fix docker rollback command to pin digest then compose up	2026-07-18 22:07:15 +00:00
jerome	06d2bcbc9e	Merge pull request 'fix: fleet config issues from 2026-07-18 relay review' (#23 ) from fix/fleet-config-issues-20260718 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 0s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details Reviewed-on: #23	2026-07-18 22:06:48 +00:00
jerome	c4a8c45835	Merge pull request 'zulip-resilience: fleet-wide audit findings and fixes 2026-07-18' (#22 ) from feat/zulip-resilience-audit-20260718 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Reviewed-on: #22	2026-07-18 22:06:34 +00:00
root	1d027f71f6	feat(contracts): add infrastructure-maintenance contract New responsibility contract consolidating host-level system maintenance and Docker image lifecycle management, filling the gap left by infrastructure-update (which owns cluster-wide apt waves). Scope: - OS package updates on primary host with pre-update snapshot/backup check - Docker image pulls for LiteLLM, SearXNG, and other running containers - Container restarts with per-stack health verification - Post-update verification: LiteLLM, SearXNG, Zulip, Gitea, PM2, Hermes gateways - Rollback on failure (image/apt/config restore) with circuit breaker Owner: ops (firstmate secondmate). Trigger: weekly Sunday 2am ET. Escalation: warning/critical->abiba+mumuni, fatal->abiba+mumuni+kwame. circuit_breaker: max_retries 2, window 7200, trip_action escalate_to_fatal. depends_on: infrastructure-monitoring (pre-update health baseline). Registry: - Add infrastructure-maintenance to by_category.maintenance, by_domain.infrastructure, by_owner.ops (new), by_trigger.scheduled, by_sensitivity.high - Add 'ops' to owners list - Move infrastructure-update owner abiba -> ops (in contracts entry + by_owner index) Also adds '## Maintaining this file' section to AGENTS.md per fm-ensure-agents-md.	2026-07-18 22:02:45 +00:00
jerome	a74229ee74	Merge branch 'master' into fix/fleet-config-issues-20260718 PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details	2026-07-18 18:06:19 +00:00
root	aebc98ead6	fix: remove trailing whitespace from litellm-api-keys.prose.md PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 2s Details	2026-07-18 17:48:52 +00:00
root	17a77e6b3f	fix: fleet config issues from 2026-07-18 relay review PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details - hermes-agent-baseline: add gpu-dense and gpu-light to models list - hermes-config-template: fix pgrep traps (exclude infisical wrapper) + add vault empty-key guard documentation in Rule 13 - litellm-api-keys: fix pgrep pattern in auditable check - scripts/agent-health-check: fix pgrep to exclude infisical bash wrapper Addresses issues found during relay inbox resolution session: 1. pgrep -f 'hermes_cli.main gateway run' matches both the real python gateway and the infisical bash wrapper, causing false health readings 2. infisical vault stores empty key silently — no guard/monitoring 3. gpu-dense/gpu-light stable aliases missing from baseline config	2026-07-18 17:39:46 +00:00
root	23f3f378c5	zulip-resilience: add fleet-wide audit findings and fixes from 2026-07-18 PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 6s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details - Added Incident Log section documenting fleet-wide Zulip audit - Abiba: poll timeout AbortError fix (returns [] instead of error) - Abiba: credential fallback .env file for Infisical outages - Tanko: full gateway restart to recover Zulip connection - Fleet health metrics summary table - Hermes agent improvement recommendations (circuit breaker, credential fallback, queue re-registration, watchdog) - Abiba pi extension v2.1 resilience feature matrix	2026-07-18 16:33:12 +00:00
jerome	14d27a09b5	Merge pull request 'gpu-self-heal: refresh to current fleet baseline and topology' (#21 ) from feat/gpu-self-heal-refresh-20260718 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Reviewed-on: #21	2026-07-18 08:36:37 +00:00
root	bddbb22f03	gpu-self-heal: refresh to current fleet baseline and topology PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s Details - Synced model assignments to 2026-07-17 swaps (ThinkingCap, HauhauCS QAT, Genesis Hermes V3) - Added stable role-based aliases from gpu-fleet (gpu-dense, gpu-light, strix-moe) - Updated benchmark baselines to live values (74.9/169.6/62.9 tok/s) - Replaced router (port 9000) references with LiteLLM + direct routing - Replaced Prometheus exporter rule with sidecar health probe - Updated VRAM thresholds to match operational data (300/300/200 MB/h) - Added response size limit (1MB) to prevent OOM crashes - Added Lessons L5 (response size crash) and L6 (stable aliases) - Removed deprecated Rules 11-12 (router-specific distribution balance)	2026-07-18 08:07:03 +00:00
root	31ec70ae36	gpu-fleet: RTX 3090 swap to ThinkingCap-Qwen3.6-27B Q4_K_M - Model: Qwopus Q4_K_M (16GB, 63 tok/s) → ThinkingCap Q4_K_M (15.7GB, 68 tok/s) - RL-finetuned: 50% fewer thinking tokens, 0.85 MMLU-Pro (vs 0.83 base) - Self-spec MTP (n=4) REQUIRED for stability — segfaults without it - Added vision via mmproj (0.9GB) — new capability for this GPU - VRAM: 20.9/24.6GB (85%), tighter but stable - Outputs reasoning_content (hidden from Hermes agent)	2026-07-17 15:50:50 +00:00
root	5eb6d3bfbd	gpu-fleet: RTX 5070 swap to HauhauCS Gemma4-12B QAT Uncensored Balanced - Model: IQ4_NL (6.3GB, 191 tok/s) → Q4_K_M QAT (6.9GB, 87 tok/s) - MTP draft: Q8_0 (444MB) → tuned draft (242MB), saves 200MB VRAM - mmproj: F16 → BF16 (same size, matched to new model) - Benefits: 0/465 refusals, agent-optimized tuning, QAT quality - Trade: 54% slower generation (acceptable for gpu-light role) - Config: --parallel 1, --ctx-size 131072, single-slot full 128K - VRAM: 10.0/12.2GB (82%), healthy headroom	2026-07-17 15:24:13 +00:00
jerome	33cb88d571	Merge pull request 'feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo' (#20 ) from feat/gpu-128k-genesis-hermes-v3-20260717 into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Reviewed-on: #20	2026-07-17 11:21:32 +00:00
root	4a1f476623	fix: add missing description to inference-optimization frontmatter PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 0s Details	2026-07-17 11:11:27 +00:00
root	ba2c55c7e6	ci: re-trigger pipeline for PR #20 PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details	2026-07-17 11:10:29 +00:00
root	b9149bce47	feat: GPU context 256K→128K fleet-wide + Genesis Hermes V3 on Strix Halo PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details GPU Changes: - All 3 GPUs reduced from 256K (-c 262144) to 128K (-c 131072) for stability - Observed instability near 100K at 256K — 128K is the stable ceiling - VRAM improved: RTX 3090 ~70% (was 90%), RTX 5070 ~65% (was 88%) - Strix Halo swapped to LuffyTheFox/Genesis Hermes V3 APEX - Hermes agent fine-tune, tensor repair (3 SSM layers, 76% W1 improvement) - Uncensored (0/465 refusals), multimodal (mmproj F16) - Speed: 65 tok/s gen, 140 tok/s prompt - Alias strix-moe maintained Agent Updates: - Mumuni: max_context_window 262144→131072, already aligned on strix-moe/0.65 - Tanko: max_context_window 262144→131072 - Koonimo: max_context_window + context_length 262144→131072 - CT114 SSH access confirmed (was 'Zulip only') LiteLLM (CT116): - Updated backend model references qwen3.6-35B-udq4→strix-moe - Removed stale ornith-1.0-35b from model_cost - Fallback chains updated Contracts Updated: - gpu-fleet.prose.md: topology, VRAM, benchmarks, config lines, model assignments - gpu-self-heal.prose.md: Rule 9/10 context targets - hermes-config-template.prose.md: template values, Rules 7-9, compression thresholds - inference-optimization.prose.md: added to repo, 128K recommendation Compression: 0.65 fires at ~85K (~43K headroom before 128K ceiling) For >128K workloads: route to external providers (deepseek)	2026-07-17 10:59:45 +00:00
root	65c99dab50	contract: update litellm-api-keys to v2026-07-17 — fleet standardization - Document canonical systemd drop-in pattern (ExecStart= reset + wrapper) - Hardcode venv paths — never use variables in single-quoted bash -c - Update migration status: all 4 agents on while-true wrapper + st.8e848433 - Infisical CLI update procedure (0.38.0 → 0.43.109 via artifacts-cli) - Service token inventory + .env fallback inventory - Key rotation log: fleet standardize + tanko fix-zulip entries - Remove git merge conflict artifacts - Add fleet-wide standardization lessons section	2026-07-17 10:17:23 +00:00
jerome	20cbb96e2d	Merge pull request 'Vault cleanup + contract sync (WAL #1316 )' (#19 ) from fix/vault-cleanup-contract-sync into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details Reviewed-on: #19	2026-07-16 21:24:59 +00:00
jerome	24cd7f72ac	Merge branch 'master' into feat/zulip-v3-resilience	2026-07-16 21:20:24 +00:00