Merge branch 'master' into fm/infra-maint-contract

gpu-fleet.prose.md

@@ -9,9 +9,9 @@ description: >
   gpu-dense, gpu-light. These never change — only the underlying model does.
   Strix Halo: strix-moe → unsloth/Qwen3.6-35B-A3B-MTP (UD-Q4_K_M, 22GB).
   RTX 5070: gemma-4-12b Q4_K_M → IQ4_NL + MTP draft (122 tok/s, 2x faster).
-  UPDATED 2026-07-17: Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored,
+  UPDATED 2026-07-17: Context reduced fleet-wide from 256K to 128K for stability.
+  Strix Halo model swapped to Genesis Hermes V3 APEX (LuffyTheFox, 24GB, uncensored,
   Hermes agent fine-tune, tensor repair, multimodal with mmproj).
-  RTX 5070 swapped to HauhauCS Gemma4-12B QAT Uncensored Balanced (Q4_K_M, 87 tok/s, 0/465 refusals).
   Instability observed near 100K at 256K. 128K is the stable ceiling.
   For larger context needs → fall back to external providers (deepseek).
   VRAM headroom improved: RTX 3090 ~70%, RTX 5070 ~65%.
@@ -87,32 +87,20 @@ When a model is swapped on a GPU, ONLY the infrastructure layer changes — agen
 | Alias | GPU | Current Model | Will Route To |
 |-------|-----|---------------|---------------|
 | `strix-moe` | Strix Halo (.15) | qwen3.6-35B-udq4 | Whatever runs on Strix Halo |
-| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code (ThinkingCap) | Whatever runs on RTX 3090 |
+| `gpu-dense` | RTX 3090 (.8) | qwen3.6-27B-code | Whatever runs on RTX 3090 |
 | `gpu-light` | RTX 5070 (.110) | gemma-4-12b | Whatever runs on RTX 5070 |
 
 **Backward compatibility**: Old model-specific names (qwen3.6-27B-code, gemma-4-12b, qwen3.6-35B-udq4) still work
 but are deprecated for agent configs. Only the stable aliases survive model swaps.
 
-## Current Model Assignments (2026-07-17)
+## Current Model Assignments (2026-07-15)
 
 | Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
 |-------|-----|------|------|-----|----------|----------|-------------|--------|
-| qwen3.6-27B-code (ThinkingCap) | RTX 3090 | .8 (llm-gpu) | ~20.9/24.6GB (85%) | **128K** | turbo4 | 1 | default | ✅ 68 tok/s |
+| qwen3.6-27B-code (MTP) | RTX 3090 | .8 (llm-gpu) | ~17/24.6GB (70%) | **128K** | turbo4 | 2 | default | ✅ 63 tok/s |
-| gemma-4-12b (HauhauCS QAT) | RTX 5070 | .110 (ocu-llm) | ~10.0/12.2GB (82%) | 128K | q4_0 | 1 | 2048/1024 | ✅ 87 tok/s |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | ~7.8/12.2GB (65%) | 128K | q4_0 | 2 | 2048/1024 | ✅ healthy |
 | Genesis Hermes V3 APEX | Strix Halo Vulkan | .15 (amdpve) | ~10GB/64GB | 128K | q4_0 | 1 | 4096/1024 | ✅ 65 tok/s |
 
-> **RTX 5070 model swap (2026-07-17)**: Switched from `gemma-4-12b-it-IQ4_NL` (Unsloth, 191 tok/s)
-> to `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` (Q4_K_M QAT, 87 tok/s).
-> Trade: 54% slower generation for QAT quality, 0/465 refusals, and agent-optimized tuning.
-> MTP draft also swapped: Q8_0 (444MB) → tuned draft (242MB), saving 200MB VRAM.
-> Role unchanged: gpu-light (vision, web extract, light auxiliary tasks).
-
-> **RTX 3090 model swap (2026-07-17)**: Switched from `Qwopus3.6-27B-v2-MTP-Q4_K_M` (63 tok/s)
-> to `bottlecapai/ThinkingCap-Qwen3.6-27B` (Q4_K_M QAT, 68 tok/s).
-> RL-finetuned: 50% fewer thinking tokens, MMLU-Pro 0.85 vs 0.83 base.
-> Self-spec MTP REQUIRED (crashes without it on turboquant build).
-> Added vision via mmproj (0.9GB). VRAM 85%.
-
 ## Routing Configuration (LiteLLM — July 2026)
 
 ### syslog-auto Weighted Pool (Direct GPU — bypasses router)
@@ -130,16 +118,16 @@ Note: All syslog-auto entries route directly to GPUs with `api_key: not-needed`.
 | Model | RPM Cap | Notes |
 |-------|---------|-------|
 | strix-moe (Hermes V3) | 40 | Tight cap — prevents Strix overload |
-| qwen3.6-27B-code | 500 | ThinkingCap Q4_K_M + MTP self-spec + vision, 68 tok/s |
+| qwen3.6-27B-code | 500 | High cap — primary workhorse |
-| gemma-4-12b | 500 | HauhauCS QAT Uncensored Balanced + MTP, 87 tok/s |
+| gemma-4-12b | 500 | High cap — IQ4_NL+MTP, 122 tok/s |
 
 ### Stable Aliases (for agent configs — never change)
 
 | Alias | RPM Cap | Routes To | Purpose |
 |-------|---------|-----------|---------|
 | `strix-moe` | 40 | Strix Halo | Compression tasks (MoE models) |
-| `gpu-dense` | 500 | RTX 3090 (ThinkingCap) | Heavy reasoning, code gen, delegation |
+| `gpu-dense` | 500 | RTX 3090 | Heavy reasoning |
-| `gpu-light` | 500 | RTX 5070 (HauhauCS QAT) | Vision, web extract, light tasks |
+| `gpu-light` | 500 | RTX 5070 | Vision, web extract, light tasks |
 
 ### Fallback Chains
 - gemma → qwen
@@ -261,8 +249,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
 - **VRAM (2026-07-15)**: RTX 3090 at ~17/24.6GB (~70%) with **128K context** (reduced from 256K 2026-07-17). RTX 5070 at ~7.8/12.2GB (~65%) with 128K context + MTP. Strix Halo at ~7GB/64GB.
 - **RTX 3090 runs `--parallel 2`** with MTP draft (spec-type draft-mtp, spec-draft-n-max 2).
-- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 1 --flash-attn on --cont-batching --spec-type draft-mtp --spec-draft-n-max 4`. ThinkingCap Qwen3.6-27B Q4_K_M (15.7GB) + mmproj (0.9GB) + MTP self-spec. VRAM: ~85%. Service: `/home/llmuser/llama-wrapper.sh`. ⚠️ MTP REQUIRED for stability on this turboquant build — model segfaults without `--spec-type draft-mtp`. Outputs reasoning_content (hidden from agent, improves answer quality).
+- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2 --flash-attn on --cont-batching --spec-type draft-mtp`. Context reduced to 128K (2026-07-17, was 256K). VRAM: ~70%. Service: `/home/llmuser/llama-wrapper.sh`.
-- **RTX 5070 config (2026-07-17)**: HauhauCS Gemma4-12B QAT Uncensored Balanced (Q4_K_M) + tuned MTP draft (242MB) at 128K context, single slot. Gen speed: 87 tok/s (vs 191 IQ4_NL). VRAM: ~10.0/12.2GB (~82%). Service: `/home/llmuser/llama-wrapper.sh`. Model: `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`, MTP: `mtp-gemma-4-12B-it.gguf`, mmproj: `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`. Recommended sampling: temp 0.6, top_k 64, top_p 0.9, min_p 0.05, repeat_penalty 1.1.
+- **RTX 5070 config (2026-07-15)**: Switched to IQ4_NL + MTP draft (Q8_0) at 128K context. Gen speed: 122 tok/s. VRAM: ~7.8/12.2GB (~65%). Service: `/home/llmuser/llama-wrapper.sh`. Config: `--model gemma-4-12b-it-IQ4_NL.gguf --spec-draft-model gemma-4-12b-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 4 --ctx-size 131072`.
 - **LiteLLM timeout tuning (verified 2026-07-16 against `/opt/inference-harness/litellm_config.yaml` on CT 116)**: gemma-4-12b 120s, qwen3.6-27B-code 300s, qwen3.6-35B-udq4 300s, strix-moe 300s, syslog-auto routes all 300s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `strix-server.service` on port 8080, model: `LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V3-GGUF` (APEX quant), alias `strix-moe`, 128K context, flash-attn + q4 KV, multimodal (mmproj loaded). Hermes agent fine-tune, tensor repair (SSM layers fixed via SVD), uncensored (0/465 refusals).
 - **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
@@ -278,8 +266,8 @@ If no SSH access, send Zulip DM via abiba-bot with vault update instructions.
 
 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Context |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | ThinkingCap-Qwen3.6-27B Q4_K_M | **68** | — | — | **128K** |
+| RTX 3090 (.8) | qwen3.6-27B-code (MTP) | **63** | — | — | **128K** |
-| RTX 5070 (.110) | HauhauCS QAT Uncensored Balanced | **87** | — | — | **128K** |
+| RTX 5070 (.110) | gemma-4-12b (IQ4_NL+MTP) | **191** | — | — | **128K** |
 | Strix Halo (.15) | Genesis Hermes V3 APEX | **65** | 140 | — | **128K** |
 
 Benchmarks from 2026-07-17. Strix Halo swapped to Genesis Hermes V3 APEX (LuffyTheFox). RTX 5070 MTP provides 2.7x speedup over pre-upgrade 70 tok/s.

hermes-agent-baseline.prose.md

@@ -193,6 +193,8 @@ Key is injected via `infisical run --` wrapper at PM2 startup:
       "models": [
         { "id": "syslog-auto" },
         { "id": "strix-moe" },
+        { "id": "gpu-dense" },
+        { "id": "gpu-light" },
         { "id": "qwen3.6-27B-code" },
         { "id": "gemma-4-12b" }
       ]

hermes-config-template.prose.md

@@ -325,7 +325,8 @@ verify ALL FOUR of these against the live config. They are the only root causes
 
 One-line agent health check (run on the agent host):
 ```bash
-PID=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
+# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
+PID=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
 cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | sed 's/=.*/<set>/'
 curl -s -o /dev/null -w 'key_health: %{http_code}\n' -H "Authorization: Bearer $(cat /proc/$PID/environ | tr '\0' '\n' | grep ^LITELLM_API_KEY= | cut -d= -f2)" http://192.168.68.116/v1/models
 ```
@@ -351,9 +352,19 @@ directly (no infisical). Apply with `systemctl daemon-reload && systemctl restar
 The wrapper sources `~/.hermes/.env` then exports `LITELLM_API_KEY="$<AGENT>_LITELLM_API_KEY"`.
 See litellm-api-keys.prose.md § Machine Identity for Vault Writes for vault sync.
 
+**⚠️ Vault empty-key guard:** If the vault stores the secret as an empty string,
+the wrapper will inject an empty key and the gateway will silently get 401 errors
+on all LiteLLM requests (triggering silent DeepSeek fallback). The `.env` fallback
+is present but the vault takes precedence when the secret key exists (even if empty).
+
+**Fix:** The wrapper MUST validate the key length after injection. If LITELLM_API_KEY
+is empty or shorter than 20 chars, log a warning and either fail with a clear error
+message or fall back to the `.env` value before starting the gateway.
+
 **Verification (all agents):**
 ```bash
-GP=$(pgrep -f "python -m hermes_cli.main gateway run" | head -1)
+# Use grep -v infisical to avoid matching the bash wrapper that contains the same string
+GP=$(pgrep -f "python -m hermes_cli.main gateway run" | grep -v infisical | head -1)
 K=$(cat /proc/$GP/environ | tr '\0' '\n' | grep '^LITELLM_API_KEY=' | cut -d= -f2)
 curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $K" http://192.168.68.116/v1/models  # must be 200
 ```

litellm-api-keys.prose.md

@@ -8,18 +8,18 @@ description: >
   Ensures agents never use the master key directly. Rotation is event-driven,
   not calendar-driven — rotate only on compromise, personnel change, or
   periodic security hygiene (quarterly/annually).
-  
+
   UPDATED 2026-07-12: Keys are stored in Infisical vault (project=agents, env=production)
   BUT each agent host MUST keep a local .env fallback. Infisical service tokens can
   expire/404. The .env fallback prevents agents from running without keys.
   Tanko incident: token 404 → gateway had no LITELLM_API_KEY for hours.
-  
+
   UPDATED 2026-07-16: Vault is SYNCED (session-13 keys written to vault via abiba service
   token, all validate 200). Koby/Koonimo migrated from hardcoded drop-ins to the
   infisical-gateway.sh wrapper (live vault injection). 4/5 agents now vault-backed.
   Canonical process: see § Production Vault Access Process. Tanko (user jerome) pending.
   Abiba's key is now a proper agent key (NOT the master key — stale note removed).
-  
+
   UPDATED 2026-07-17: FLEET-WIDE STANDARDIZATION. All 4 agents (Mumuni, Tanko, Koby, Koonimo)
   standardized on a single pattern: systemd drop-in (ExecStart= reset + wrapper path) →
   infisical-gateway.sh while-true loop → /usr/bin/infisical run --token → bash -c key
@@ -30,7 +30,7 @@ description: >
   Critical lessons: (1) NEVER use shell variables inside single-quoted bash -c in wrappers
   — hardcode absolute paths. (2) Drop-ins override unit file ExecStart permanently.
   (3) Capture /proc/<pid>/environ before gateway restarts to preserve running env set.
-  
+
   Current key inventory and agent list: see gpu-fleet.prose.md § Agent Keys.
   Source of truth for LiteLLM config: /opt/inference-harness/litellm_config.yaml
   on CT 116. Last verified: 2026-07-17.
@@ -154,7 +154,7 @@ through its agent wrapper.
    The `ExecStart=` (empty reset) clears any ExecStart from the main unit file,
    then the second `ExecStart=` sets the wrapper. This drop-in **survives unit file
    regeneration** by `hermes gateway install` — the drop-in always wins.
-   
+
    **Why a drop-in instead of editing the unit file:** `hermes gateway install`
    (called during Hermes updates and some self-heal operations) regenerates the
    systemd unit file with `ExecStart=/path/to/python -m hermes_cli.main gateway run`.
@@ -171,7 +171,7 @@ through its agent wrapper.
 - **Survives gateway crash**: the wrapper's `while true` + systemd `Restart=always` revive the gateway. Two-layer defense.
 - **Survives Hermes updates**: systemd drop-in overrides unit file ExecStart — `hermes gateway install` cannot break the vault injection.
 - **Survives reboot**: systemd user service + `loginctl enable-linger` ensures gateway starts at boot without a login session.
-- **Auditable**: `cat /proc/$(pgrep hermes_cli)/environ` shows all injected keys; `infisical secrets` shows the vault source.
+- **Auditable**: `cat /proc/$(pgrep -f 'python.*hermes_cli.main.gateway.run' | grep -v infisical | head -1)/environ` shows all injected keys (note: pipe through grep -v infisical to avoid matching the bash wrapper); `infisical secrets` shows the vault source.
 
 ### Migration status (2026-07-17)
 

scripts/agent-health-check.py

@@ -184,8 +184,8 @@ def check_agents():
             print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
             continue
 
-        # Gateway process
+        # Gateway process (exclude the infisical bash wrapper that contains the same string)
-        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
+        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | grep -v infisical | head -1", user=user)
         if not pid:
             print(f"  ❌ {name}: GATEWAY NOT RUNNING")
             FAIL.append(f"gateway-down:{name}")

zulip-resilience-v3.prose.md

@@ -409,3 +409,68 @@ Backup v2 before starting: `cp index.js index.js.v2-backup-$(date +%Y%m%d-%H%M%S
 | Queue expiry handling | Crash | Auto re-register |
 | Busy worker deadlock | Router death | Worker SIGKILL + error DM |
 | PM2 restart exhaustion | Yes (max_restarts=10) | No (max_restarts=100 + watchdog) |
+
+---
+
+## Incident Log — 2026-07-18 Fleet-Wide Audit
+
+### Fleet State After Audit
+
+| Agent | Platform | Zulip State | Issues Found | Fix Applied |
+|-------|----------|-------------|--------------|-------------|
+| **Abiba** | pi (CT 100) | ✅ Connected | API key missing from Infisical injection; poll timeout noise | Added .env fallback; AbortError treated as empty poll (no retry); poll timeout 65s→90s |
+| **Tanko** | Hermes (CT 112) | ✅ Connected | Gateway disconnected since Jul 11; watchdog restart didn't re-establish Zulip | Full gateway restart (kill wrapper, let infisical-gateway.sh respawn) |
+| **Mumuni** | Hermes (CT 114) | ✅ Connected | No issues found | None needed |
+
+### Key Fixes Applied
+
+**1. Abiba — Credential Fallback (L4 Pattern)**
+- Root cause: `zulip.api_key` in config.yaml is `""` (expected from Infisical). Infisical vault `ABIBA_ZULIP_API_KEY` wasn't being injected into the process environment.
+- Fix: Added `.env` file fallback at `/root/.pi/agent/extensions/zulip/.env` with known-working key, sourced before the Infisical `exec`.
+- Lesson: Per L4 from gpu-self-heal, Infisical is not always available — always keep a local `.env` fallback.
+
+**2. Abiba — Poll Timeout Handling**
+- Root cause: Zulip long-poll uses `AbortSignal.timeout(65000)`. Zulip's default `event_queue_longpoll_timeout_seconds` can exceed 65s. When the signal fires, an `AbortError` is thrown and caught by the circuit breaker as a failure.
+- Fix: Caught `AbortError` inside `poll()` and return empty array (no events) instead of throwing. Extended timeout to 90s to match Zulip server default.
+- Reference: [Zulip Events System — long-poll timeout](https://zulip.readthedocs.io/en/11.6/subsystems/events-system.html)
+
+**3. Tanko — Gateway Restart**
+- Root cause: Gateway process was running but Zulip platform stayed in "disconnected" state since Jul 11, 2026. The wrapper script (`infisical-gateway.sh`) restarts on crash but the gateway wasn't re-establishing Zulip on restart.
+- Fix: Killed gateway PID to trigger wrapper restart. New gateway (PID 331991) established Zulip connection successfully.
+
+### Fleet-Wide Zulip Health Metrics (as of 2026-07-18)
+
+| Metric | Value |
+|--------|-------|
+| Zulip server | ✅ HTTP 200 |
+| Agents connected | 3/3 (Abiba, Tanko, Mumuni) |
+| Abiba circuit breaker | CLOSED (0 failures) |
+| Abiba uptime | 2D (post-restart) |
+| Tanko gateway uptime | Ongoing |
+| Mumuni gateway uptime | Ongoing |
+| Watchdog status | ✅ Online (2D uptime) |
+
+### Hermes Agent Zulip Plugin Improvements
+
+Based on the audit, improvements that should be ported to all Hermes Zulip adapters:
+
+1. **Circuit breaker pattern** — Already in Abiba's pi extension. Hermes adapters should add the same CLOSED→OPEN→HALF_OPEN state machine with exponential backoff.
+2. **Credential fallback** — All Hermes agents use Infisical for credentials. Add `.env` local fallback per L4 pattern for `ZULIP_API_KEY`.
+3. **Queue re-registration** — Handle `BAD_EVENT_QUEUE_ID` with automatic re-registration instead of gateway restart.
+4. **Supervisor watchdog** — Hermes uses PM2 which auto-restarts on crash, but has no health-check watchdog. Add lightweight external health checks.
+5. **Streaming** — All agents have `streaming: true` in their zulip config. Verify `edit_message()` is implemented in each adapter.
+
+### Abiba pi Zulip Extension v2 — Implemented Resilience Summary
+
+| Feature | Status | Notes |
+|---------|--------|-------|
+| Circuit breaker | ✅ | CLOSED→OPEN→HALF_OPEN; 50% failure threshold; 30s reset timeout |
+| Retry with jitter | ✅ | 2 attempts, 200ms base, 50-100% jitter |
+| Queue lifecycle | ✅ | 10min idle_queue_timeout; BAD_EVENT_QUEUE_ID handling |
+| Crash prevention | ✅ | uncaughtException + unhandledRejection recovery |
+| Worker timeout | ✅ | 5min busy timeout → SIGKILL + error DM |
+| Health endpoint | ✅ | :9200 with circuit breaker metrics |
+| Echo prevention | ✅ | Dynamic bot user resolution |
+| Poll timeout (AbortError) | ✅ v2.1 | Normal timeout returns [] instead of error |
+| Credential fallback | ✅ v2.1 | .env file before Infisical exec |
+| Provider auto-fix | ✅ | Detects reasoning_content models, switches to compatible |