gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes

- All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3) - .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache - .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV - .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm - LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s - Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma - Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)
fix(ci): change kind from reference to template in hermes-agent-baseline.prose.md
2026-07-08 17:25:42 +00:00 · 2026-07-07 20:57:48 -04:00 · 2026-07-07 07:18:45 +00:00 · 2026-07-06 22:56:48 +00:00 · 2026-07-06 18:54:12 -04:00 · 2026-07-06 18:21:30 -04:00
13 changed files with 900 additions and 96 deletions
@@ -21,7 +21,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -34,7 +34,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -64,7 +64,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -77,7 +77,7 @@ jobs:
    steps:
      - name: Checkout repository
        run: |
-          git clone --depth=50 "http://git.sysloggh.net/${{ gitea.repository }}.git" .
+          git clone --depth=50 "http://192.168.68.17:3000/${{ gitea.repository }}.git" .
          git fetch origin "${{ gitea.ref }}" --depth=50
          git checkout "${{ gitea.sha }}"

@@ -21,9 +21,11 @@ description: >
  - connected == true — Establishes and maintains Zulip event queue
  - reconnect_on_failure == true — Reconnects after queue expiry
  - recovers_from_network_loss == true — Handles temporary network drops
+  - stream_subscription_check == true — Bot is subscribed to primary streams (Gen 5+)
  
  *Message Flow*
  - dm_response_time_ms < 10000 — DMs get a response within 10 seconds
+  - dm_response_time_offline_verifiable == true — simulate_dm_latency() passes without live Zulip (Gen 6+)
  - stream_mention_detection == true — @mentions in streams are detected and routed
  - all_bots_detection == true — @all-bots mentions are detected
  - echo_loop_prevention == true — Never replies to its own messages
@@ -38,6 +40,14 @@ description: >
  - graceful_disconnect == true — shutdown doesn't crash
  - handles_malformed_messages == true — Bad JSON doesn't kill the poll loop
  - typing_indicators_work == true — Sends typing notifications
+  - known_issues_tracked == true — add_known_issue() persists problems across generations (Gen 6+)
+
+## Status
+
+**Active** — for Hermes agents only. This plugin is NOT retired. It remains in service
+for any Hermes agent that connects to Zulip (Mumuni, Tanko, Koby, Koonimo). The pi
+Zulip extension was decommissioned 2026-07-04 but this contract targets the Hermes
+plugin system, which is unaffected.

 ## Parameters

@@ -90,13 +100,28 @@ Each generation stores:

 ## Execution

-1. **Read state** — Check previous generations from knowledge graph
+1. **Read state** — Check previous generations from knowledge graph node `build-zulip-plugin:state`
 2. **Generate or improve** — Based on generation count:
   - Generation 1: Scaffold full plugin from scratch
   - Generation N: Read previous code, apply improvements
 3. **Validate** — Syntax check, structure check, dependency check
-4. **Log** — Save generation result to knowledge graph as [LEARN]
-5. **Report** — Output what was generated, what changed, what needs review
+4. **Run offline tests** — Always run `simulate_dm_latency()` and selftest() if no live Zulip
+5. **Log** — Save generation result to knowledge graph as [LEARN] node. Update the
+   `build-zulip-plugin:state` node with current generation_count, plugin_version,
+   and known_issues.
+6. **Report** — Output what was generated, what changed, what needs review
+
+## State Persistence (Gen 6+)
+
+All generation state is stored in a knowledge graph node titled `build-zulip-plugin:state`:
+- `generation_count` — incremented on each verified run
+- `plugin_version` — current version string
+- `last_generation` — ISO timestamp of most recent run
+- `known_issues` — array of unresolved issues carried forward
+- `last_postcondition_results` — pass/fail map from most recent verification
+
+Agents executing this contract MUST read this node on startup and update it after
+validation. If the node does not exist, create it with generation_count=1.

 ## Example Output

@@ -5,8 +5,9 @@ description: >
  Manages the GPU inference fleet across all hosts. Handles model deployment,
  registration, health checks, LiteLLM sync, agent key management, GPU
  saturation watchdog, Prometheus/Grafana monitoring, and self-healing.
-  Current as of 2026-06-30 post-migration: nginx routes /v1 → LiteLLM,
-  router runs behind LiteLLM on :9000 (internal-only).
+  Current as of 2026-07-08: context reduced to 128K on NVIDIA GPUs, parallel 2
+  on all GPUs, LiteLLM timeouts tuned (gemma 25→120s, qwen 40→90s), router fully
+  deprecated — nginx routes /v1 → LiteLLM directly.
 agent: abiba
 triggers:
  - on model add/remove
@@ -27,6 +28,7 @@ triggers:
 - benchmarks: { tok_per_sec: map, baseline: map, history: array } — Inference speed benchmarks tracked over time
 - grafana: { status: "running", dashboards: ["gpu-fleet"] } — Grafana on CT 116 (:3001)
 - prometheus: { status: "running", targets: 5 } — Scrapes GPU :9400 exporters + LiteLLM
+- port_conflict_detection: { status: "active" } — All 3 GPU wrappers detect ghost processes before binding

 ## Fleet Topology (Current — June 2026)

@@ -62,21 +64,21 @@ triggers:
 │ CT 8     │  │ CT 110      │  │ CT 15     │  │ pi (.24)     │
 │ RTX 3090 │  │ RTX 5070    │  │ Strix Halo│  │ GPU Monitor  │
 │ 24GB     │  │ 12GB        │  │ 64GB UMA  │  │ :9100        │
-│          │  │             │  │ llama-srv │  │ Watchdog     │
+│ 128K ctx │  │ 128K ctx    │  │ 256K ctx  │  │ Watchdog     │
 │ qwen3.6  │  │ gemma-4-12b │  │ ornith35B │  │ Prometheus   │
-│ 27B-code │  │ :8090       │  │ :8080     │  │ exporter     │
-│ :8090    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
+│ 27B-code │  │ :8080       │  │ :8080     │  │ exporter     │
+│ :8080    │  │ :9400 (exp) │  │ :9400(exp)│  │ :9401        │
 │ :9400    │  └─────────────┘  └───────────┘  └──────────────┘
 └──────────┘
 ```

-## Current Model Assignments
+## Current Model Assignments (2026-07-08)

-| Model | GPU | Host | VRAM | Status |
-|-------|-----|------|------|--------|
-| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 23.4/24GB | ✅ healthy |
-| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 11.2/12.2GB | ✅ healthy |
-| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 22.7/64GB | ✅ healthy |
+| Model | GPU | Host | VRAM | Ctx | KV Cache | Parallel | Batch/Ubatch | Status |
+|-------|-----|------|------|-----|----------|----------|-------------|--------|
+| qwen3.6-27B-code | RTX 3090 | .8 (llm-gpu) | 20.3/24GB (83%) | 128K | turbo4 | 2 | 512/512 | ✅ healthy |
+| gemma-4-12b | RTX 5070 | .110 (ocu-llm) | 9.4/12.2GB (77%) | 128K | q4_0 | 2 | 2048/512 | ✅ healthy |
+| ornith-1.0-35b | Strix Halo Vulkan | .15 (amdpve) | 24.4/64GB (35%) | 256K | q8_0 | 2 | 2048/512 | ✅ healthy |

 ## Operations

@@ -112,7 +114,7 @@ triggers:

 ### sync-keys
 1. List all agent keys in LiteLLM DB via `GET /key/list`
-2. Compare against expected agent list: [tanko, mumuni, abiba, koby, koonimo, kagenz0]
+2. Compare against expected agent list: [tanko, mumuni, abiba, tdunna, baggy, kagenz0]
 3. Generate missing keys via `POST /key/generate` with unlimited budget
 4. Update agent configs — `/etc/environment` LITELLM_API_KEY
 5. Send Zulip DM to agents that can't be reached via SSH
@@ -120,18 +122,30 @@ triggers:
 7. Document keys in knowledge graph

 ### list
-Show full fleet status: GPUs, models, VRAM, active requests, circuit breakers, keys
+Show full fleet status: GPUs, models, VRAM, context windows, parallel slots, active requests, circuit breakers, keys
+
+### health-check
+1. Check GPU hardware: nvidia-smi (.8, .110) + amdgpu sysfs (.15 via /sys/class/drm/card0/device/)
+2. Check llama-server processes: `ps aux | grep llama-server` on all 3 hosts
+3. Check LiteLLM: `curl http://192.168.68.116/health` (expect "I'm alive!")
+4. Check LiteLLM models: `curl -H "Authorization: Bearer $MASTER_KEY" http://192.168.68.116/v1/models`
+5. Check LiteLLM timeouts: `grep -n 'timeout:' /opt/inference-harness/litellm_config.yaml`
+   - gemma-4-12b: 120s, qwen3.6-27B: 90s, ornith-1.0-35b: 120s
+   - global request_timeout: 300s, nginx proxy_read_timeout: 600s
+6. Check AMD metrics: `curl http://192.168.68.15:9400/metrics` (Radeon 8060S, util%, VRAM, temp, power)
+7. Check port conflicts: verify only one llama-server on :8080 per host
+8. Verify agent keys: 9 keys in LiteLLM DB (`GET /key/list`)

 ## Agent Keys (LiteLLM DB — Current 2026-06-30)

-| Agent | Key | Host | Access |
-|-------|-----|------|--------|
-| Tanko | `sk-3ZsdWJbbNSo9zSnJN2OsJw` | .122 | SSH jerome |
-| Mumuni | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | .123 | SSH root |
-| Abiba | `sk-kxbPgbnV2Zkn6TKQbAEcEg` | .24 | local |
-| Koby | `sk-lDTsWy7H3T19UwUFEN6JSg` | ? | Zulip DM (regenerated 2026-07-01 — old `sk-eb3e6...` was router key, not LiteLLM) |
-| Koonimo | `sk-Prx_vRXYb2VEzDmhPCbNeg` | .114 (baggy) | no SSH |
-| Kagenz0 | `sk-Dh4CDkaHebMLEp8qqq20qA` | ? | no SSH |
+| Agent | CT | IP | Key | Access |
+|-------|-----|-----|-----|--------|
+| Tanko | 112 | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | SSH jerome |
+| Mumuni | 114 | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | SSH root |
+| Abiba | 100 | .65 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | local (pi agent) |
+| Tdunna | 111 | ? | `sk-6sbCNjz2T6lTVDBdlNHXsA` | Zulip DM |
+| Baggy | 113 | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | no SSH |
+| Kagenz0 | 105 | ? | `sk-Dh4CDkaHebMLEp8qqq20qA` | no SSH |

 **Key update procedure**: Update `/etc/environment` → `LITELLM_API_KEY=sk-...` → restart Hermes.
 If no SSH access, send Zulip DM via abiba-bot.
@@ -145,10 +159,10 @@ If no SSH access, send Zulip DM via abiba-bot.
 | `/opt/inference-harness/router/router.py` | CT 116 | Router source (builds via compose) |
 | `/etc/nginx/nginx.conf` | CT 116 (nginx container) | Routes /v1→LiteLLM, /dashboard/, /litellm/, /health |
 | `/opt/monitoring/prometheus.yml` | CT 116 | Prometheus scrape config (5 targets) |
-| `/root/scripts/gpu-monitor-server.py` | pi (.24) | GPU fleet monitor v2.1.0 (with benchmarks) |
-| `/root/scripts/gpu_benchmark.py` | pi (.24) | GPU inference benchmark module (tok/s tracking) |
-| `/root/scripts/gpu-saturation-watchdog.py` | pi (.24) | Auto-restart stuck llama-server |
-| `/root/dashboard/gpu-fleet.html` | pi (.24) | Live HTML dashboard |
+| `/root/scripts/gpu-monitor-server.py` | pi (.65) | GPU fleet monitor v2.1.0 (with benchmarks) |
+| `/root/scripts/gpu_benchmark.py` | pi (.65) | GPU inference benchmark module (tok/s tracking) |
+| `/root/scripts/gpu-saturation-watchdog.py` | pi (.65) | Auto-restart stuck llama-server |
+| `/root/dashboard/gpu-fleet.html` | pi (.65) | Live HTML dashboard |
 | `/etc/systemd/system/llama-server.service` | .8, .110 | llama-server daemons (Nvidia GPUs) |
 | `/etc/systemd/system/ornith-server.service` | .15 (amdpve) | llama-server daemon (Vulkan, Strix Halo). Note: `llama-server.service` and `llama-server@.service` are **masked** on .15 to prevent port 8080 collisions. |

@@ -167,10 +181,13 @@ If no SSH access, send Zulip DM via abiba-bot.
 - **Router startup race**: Compose router.py doesn't call load_roster(). Reload thread sleeps 30s first.
  Fix: trigger roster reload via SSH after restart, or rebuild image with startup load_roster().
 - **LiteLLM /metrics**: Requires auth. Prometheus uses `/health/liveliness` as workaround.
- **VRAM**: RTX 3090 at ~9.6/24GB (60% free), RTX 5070 at ~6.5/12GB (46% free). Healthy.
- **RTX 3090 optimization**: `--parallel 2 --batch-size 4096` — doubled concurrent throughput.
- **RTX 5070 optimization**: Removed draft model freed ~2GB VRAM for vision tasks. `--parallel 2`.
+- **VRAM (2026-07-08)**: RTX 3090 at 20.3/24GB (83%), RTX 5070 at 9.4/12.2GB (77%), Strix Halo at 24.4/64GB (35%). Context reduced from 256K→128K on NVIDIA GPUs freed ~3.3GB (.8) and ~1.5GB (.110).
+- **All GPUs at `--parallel 2` (2026-07-08)**: Fleet serves 6 concurrent requests (was 3). 2× throughput.
+- **RTX 3090 config**: `-c 131072 -ctk turbo4 -ctv turbo4 --parallel 2`. No explicit batch flags (512/512 default). Service: `/home/llmuser/llama-wrapper.sh`.
+- **RTX 5070 config**: `--ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 2048 --ubatch-size 512 --parallel 2`. Ubatch fixed 4096→512 (was inverted — ubatch > batch killed prompt throughput). Service: `/home/llmuser/llama-wrapper.sh`.
+- **LiteLLM timeout tuning (2026-07-08)**: gemma-4-12b 25→120s, qwen3.6-27B-code 40→90s, syslog-auto (qwen route) 40→90s. Nginx proxy_read_timeout: 600s. Global request_timeout: 300s. Config at `/opt/inference-harness/litellm_config.yaml`.
 - **Strix Halo GPU**: Vulkan is the working backend (ROCm/HIP path abandoned — HSA runtime blocked on Debian 13). Build at `/root/llama.cpp/build-vk/`, commit `4fc4ec5` (2026-07-01), ggml 0.15.3 shared-lib arch. Mesa RADV 25.0.7, KHR_coopmat fast path active. ~70 tok/s gen, 532 tok/s prompt. Service: `ornith-server.service` on port 8080, 256K context, flash-attn + q8 KV.
+- **Port conflict detection (2026-07-05)**: All 3 GPU wrappers now detect ghost processes squatting port 8080 before starting. `.8` and `.110` use inline pre-start check in `llama-wrapper.sh`; `.15` uses `/usr/local/bin/port-cleanup.sh` ExecStartPre. Replaces the blanket `pkill -9 -x llama-server` on .15 which would kill ALL llama-server instances regardless of port. Ghost detection was the root cause of .8 crash-looping for 27+ restarts (stale pid 25836 squatting 8080 after OOM kill).
 - **Strix Halo thermal safeguard (2026-07-02)**: `ornith-server.service` has `-n 8192` (hard generation cap per request). Without it, `--predict` defaults to -1 (infinity) — a runaway request from .123 (Mumuni) decoded 39,868 tokens over 24 min, pushing Tctl to 98°C (crit 89.8°C) and throttling 70→29 t/s. The cap bounds worst-case generation to ~5 min. Do NOT remove `-n` without a replacement ceiling. Sustained load hits ~84°C even at 92s; the APU is fanless/low-flow. Clients MUST also set `max_tokens`.
 - **Port 8080 firewall**: amdpve iptables restricts 8080 to 192.168.68.116 (LiteLLM/router host) only. All inbound connections are from .116 (LiteLLM proxied via nginx). Localhost curls hang (SYN dropped). Always test from .116.
 - **Router sidecar fallback**: `router.py` `check_gpu_health()` now probes GPU `/health` directly when sidecar at :8090 is absent. Sidecar JSON exporters not deployed on any GPU host — router relies on GPU-direct fallback.
@@ -183,8 +200,8 @@ If no SSH access, send Zulip DM via abiba-bot.

 | GPU | Model | Gen tok/s | Prompt tok/s | Baseline | Samples |
 |-----|-------|-----------|--------------|----------|---------|
-| RTX 3090 (.8) | gemma-4-12b | 75 | 305 | 74 | 6 |
-| RTX 5070 (.110) | qwen3.6-27B-code | 75 | 323 | 75 | 6 |
+| RTX 3090 (.8) | qwen3.6-27B-code | 75 | 305 | 74 | 6 |
+| RTX 5070 (.110) | gemma-4-12b | 75 | 323 | 75 | 6 |
 | Strix Halo (.15) | ornith-1.0-35b | 70 | 532 | 70 | 6 |

 Benchmarks run through LiteLLM proxy (192.168.68.116:4001) every 5 minutes.
@@ -192,3 +209,13 @@ Degradation alerts fire at 30% (warning) and 50% (critical) below baseline.
 History stored at `/root/data/toks-history.json` with 7-day rolling window.

 **Note (2026-07-01)**: Strix Halo prompt tok/s jumped 209→532 after Vulkan rebuild (cooperative-matrix fast path now active on GFX1151). Baseline may need re-calibration.
+
+## Agent Config Implications (2026-07-08)
+
+With NVIDIA GPUs at 128K context:
+- Agents using `syslog-auto` (50/50 qwen+ornith): keep `context_length: 262144` — ornith supports it, Litellm fallbacks handle qwen overflow
+- Agents using `qwen3.6-27B-code` directly: set `context_length: 131072` and `max_tokens: 4096` per thermal safety rule
+- Agents using `gemma-4-12b` directly (auxiliary tasks): set `context_length: 131072`
+- Compression threshold at 0.65: fires at ~170K for syslog-auto (262K ctx), ~85K for direct qwen/gemma (128K ctx)
+- All Hermes clients MUST set `max_tokens: 4096` — first line of defense before server-side `-n 8192` cap
+- Port 8080 is used on all 3 GPU hosts (not 8090 as previously documented)
@@ -0,0 +1,262 @@
+---
+kind: template
+name: hermes-agent-baseline
+version: 1.0.0
+description: >
+  Canonical known-good baseline for all Syslog Hermes agents. Captures the exact
+  configuration state, keys, workarounds, and audit procedure. When an agent's
+  configuration goes sideways, restore from this baseline. Last verified 2026-07-08. GPU context reduced to 128K on .8/.110, parallel 2 fleet-wide.
+author: Abiba (pi agent)
+---
+
+# Hermes Agent Baseline — Canonical Good State
+
+## Quick Restore
+
+```bash
+# Verify all agents against baseline in one command:
+for ct in 112 114 111 113; do
+    echo "CT $ct: $(pct-run $ct grep api_key: /root/.hermes/config.yaml | grep -c sk-) api_keys found"
+done
+```
+
+## Agent Map
+
+| Agent | CT | Node | IP | LiteLLM Key | LiteLLM Alias | Platform |
+|-------|-----|------|-----|-------------|---------------|----------|
+| Tanko | 112 | amdpve | .122 | `sk-CggiHWlamQyShxWC3Hx6uw` | `tanko` | Hermes |
+| Mumuni | 114 | minipve | .123 | `sk-VrqCNlwUgzoNGOpikJ7nwQ` | `mumuni` | Hermes |
+| Tdunna | 111 | amdpve | srv1079750 | `sk-Qvzi4uYQBhlSK_XstEhcyQ` | `tdunna` | **pi** |
+| Baggy | 113 | amdpve | ? | `sk-krnw_zGBwvvL5b7l2t-s-A` | `baggy` | Hermes |
+
+Access: `pct-run <CT_ID> <command>` — no IPs needed. GPU hosts (.8, .110, .15) use SSH.
+
+## Key Architecture
+
+```
+Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → Router (:9000) → GPU (llama-server)
+                                                     └── Key DB (Postgres)
+```
+
+- **Master key**: `sk-litellm-7f96080dd99b15c36bd4b333b58a6796` — ADMIN ONLY, never in agent configs
+- **Agent keys**: Each agent has a dedicated key in LiteLLM's database with alias matching the agent name
+- **Key source**: `/etc/environment` → `LITELLM_API_KEY=sk-...` (systemd service sources this)
+- **Override**: `/home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf` (if present, must match)
+
+## Config Pattern — Mandatory Fields
+
+### For Hermes Agents (Tanko, Mumuni, Baggy)
+
+Every agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
+
+### 1. Main Model
+```yaml
+model:
+  default: syslog-auto
+  provider: custom:harness       # or: harness
+  api_key_env: LITELLM_API_KEY
+  max_tokens: 4096
+```
+
+### 2. Custom Provider
+```yaml
+custom_providers:
+  - name: harness
+    model: syslog-auto
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_mode: chat_completions
+```
+
+### 3. Vision (CRITICAL — must have api_key directly!)
+```yaml
+auxiliary:
+  vision:
+    provider: harness
+    model: gemma-4-12b              # or syslog-auto
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    timeout: 60
+    download_timeout: 30
+```
+
+### 4. Compression (must have api_key!)
+```yaml
+  compression:
+    enabled: true
+    threshold: 0.65
+    target_ratio: 0.3
+    provider: harness
+    model: syslog-auto               # or gemma-4-12b
+    base_url: http://192.168.68.116/v1
+    api_key_env: LITELLM_API_KEY
+    api_key: <ACTUAL_KEY_FROM_/etc/environment>   # ← MANDATORY workaround
+    timeout: 120
+```
+
+## Known Bug: `api_key_env` Ignored by Auxiliary Client
+
+**Bug location**: `agent/auxiliary_client.py` → `_resolve_task_provider_model()` (line ~5478)
+
+**What happens**: The function reads `api_key` from auxiliary task configs but does NOT
+resolve `api_key_env`. If only `api_key_env` is set (no `api_key`), the key resolves
+to `None`, and the explicit_base_url branch in `resolve_provider_client` falls through
+to `"no-key-required"` → 401 from LiteLLM.
+
+**Impact**: Vision analysis, compression, and any other auxiliary task calling
+LiteLLM/harness will fail with:
+```
+401: LiteLLM Virtual Key expected. Received=no-k****ired, expected to start with 'sk-'
+```
+
+**Workaround**: Set `api_key` directly (copy the value from `/etc/environment`) alongside
+`api_key_env` in every auxiliary task config that uses the harness provider.
+
+**Permanent fix**: Patch `_resolve_task_provider_model()` to resolve `api_key_env` when
+`api_key` is empty:
+```python
+cfg_api_key = str(task_config.get("api_key", "")).strip() or None
+if not cfg_api_key:
+    key_env = str(task_config.get("api_key_env", "")).strip()
+    if key_env:
+        cfg_api_key = os.getenv(key_env, "").strip() or None
+```
+
+## Audit Procedure
+
+### Full Audit (all agents)
+```bash
+for ct in 112 114 111 113; do
+    echo "=== CT $ct ==="
+    pct-run $ct grep LITELLM_API_KEY /etc/environment
+    pct-run $ct grep "api_key: sk-" /root/.hermes/config.yaml | grep -v api_key_env
+    echo ""
+done
+```
+
+### Master Key Leak Check
+```bash
+# On every agent:
+pct-run <CT> grep -rl "sk-litellm-7f96080dd" /root/ /etc/ 2>/dev/null
+# Must return empty
+```
+
+### Verify Key Works
+```bash
+curl -s http://192.168.68.116:80/v1/models \
+  -H "Authorization: Bearer <AGENT_KEY>" | grep syslog-auto
+# Must return model list
+```
+
+### Verify Vision/Compression
+```bash
+# Check both api_key and api_key_env are present:
+pct-run <CT> grep -A8 "vision:" /root/.hermes/config.yaml | grep api_key
+# Must show both api_key: sk-... and api_key_env: LITELLM_API_KEY
+```
+
+### For pi Agents (Tdunna)
+
+Tdunna (CT111) runs pi 0.80.3 via PM2 with the Zulip extension (router-worker architecture).
+Config files: `~/.pi/agent/models.json`, `~/.pi/agent/settings.json`.
+
+**models.json** — Must only list models authorized for the agent's LiteLLM key:
+```json
+{
+  "providers": {
+    "syslog-harness": {
+      "baseUrl": "http://192.168.68.116/v1",
+      "api": "openai-completions",
+      "apiKey": "sk-...",
+      "models": [
+        { "id": "syslog-auto" },
+        { "id": "ornith-1.0-35b" },
+        { "id": "qwen3.6-27B-code" },
+        { "id": "gemma-4-12b" }
+      ]
+    }
+  }
+}
+```
+
+**settings.json** — Always use `syslog-auto` as default:
+```json
+{
+  "defaultProvider": "syslog-harness",
+  "defaultModel": "syslog-auto"
+}
+```
+
+**Validation**: Verify models match LiteLLM's authorized list:
+```bash
+curl -s http://192.168.68.116:4000/v1/models \
+  -H "Authorization: Bearer $(grep apiKey ~/.pi/agent/models.json | head -1 | cut -d'"' -f4)" \
+  | jq '.data[].id'
+```
+
+**Stuck worker detection**: In PM2 logs, `workers=[<id>:busy:N]` with growing N indicates
+a stuck worker (model error, no `agent_end` emitted). Fix: correct models.json, delete
+stale sessions from `~/.pi/agent/sessions/zulip/`, restart PM2.
+
+**Service**: `pm2 restart koby-zulip`, health at `:9201/health`.
+
+## Systemd Pattern
+
+For agents where the gateway runs as a user service:
+```ini
+# /home/jerome/.config/systemd/user/hermes-gateway.service.d/env.conf
+[Service]
+Environment="LITELLM_API_KEY=sk-..."    # must match /etc/environment
+```
+
+For agents where the gateway runs as root:
+```ini
+# /root/.config/systemd/user/hermes-gateway.service
+EnvironmentFile=/etc/environment        # sources LITELLM_API_KEY
+```
+
+**No drop-in that hardcodes the master key. Ever.**
+
+## Attachment Cache Locations
+
+| Type | Path |
+|------|------|
+| Images | `~/.hermes/cache/images/` |
+| Documents | `~/.hermes/cache/documents/` |
+| Audio | `~/.hermes/cache/audio/` |
+
+## Health Verification
+
+Run the consolidated health check:
+```bash
+python3 /root/scripts/agent-health-check.py
+```
+This validates all 4 LiteLLM keys, detects GPU port conflicts (ghost processes),
+verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
+and counts recent errors. Non-disruptive — never restarts anything.
+
+## GPU Port Conflict Detection
+
+All 3 GPU hosts have pre-start ghost detection in their launch wrappers:
+- `.8` and `.110`: inline check in `llama-wrapper.sh`
+- `.15`: `/usr/local/bin/port-cleanup.sh` (ExecStartPre, replaces blanket `pkill`)
+
+Detection pattern: `ss -tlnp` on port 8080 → compare pid against `systemctl MainPID`.
+If they differ → ghost detected → kill ghost → start fresh.
+
+## Related Contracts
+
+- `hermes-key-enforcement.prose.md` — key policy, rotation, detection query
+- `hermes-config-template.prose.md` — full configuration template
+- `gpu-fleet.prose.md` — GPU fleet and agent key table
+- `infrastructure-control.prose.md` — CT inventory with pct-run access
+- `litellm-health.prose.md` — LiteLLM stack health verification
+
+## Change Log
+
+| Date | Change |
+|------|--------|
+| 2026-07-08 | Tdunna: fixed model mismatch (qwen3.6-35B-A3B→syslog-auto), added pi-specific config section. Key updated to sk-Qvzi4uYQBhlSK_XstEhcyQ. Added Failure Mode #11 to zulip-adapter-lessons. |
+| 2026-07-06 | Port conflict detection added to all 3 GPU wrappers. Consolidated health check script deployed. Zulip streaming edit_message enabled for Tanko/Mumuni. |
+| 2026-07-05 | Baseline created. All 4 agents audited, master key removed, api_key workaround applied |
@@ -5,14 +5,15 @@ description: >
  Standard Hermes configuration template for Syslog Solution LLC agents.
  Enforces shared infrastructure setup (Firecrawl, SearXNG, local models,
  RA-H OS MCP) while keeping agent-specific API keys and model choices.
-  Updated 2026-06-30: new LiteLLM keys, nginx routing, router back online.
+  Updated 2026-07-08: GPU context reduced to 128K on NVIDIA (.8, .110),
+  parallel 2 on all GPUs, LiteLLM timeouts tuned, context_length guidance added.
 ---

 ## Maintains

 - template_version: "2.1.0"
 - last_applied: timestamp
- agents_configured: ["tanko", "mumuni", "abiba", "koby", "koonimo", "kagenz0"]
+- agents_configured: ["tanko", "mumuni", "abiba", "tdunna", "baggy", "kagenz0"]
 - agent_keys: map (see Agent Keys section)
 - infra_endpoints_verified: array

@@ -29,8 +30,8 @@ Sub-agent profiles inherit auth from the main config — no separate keys needed
 | Tanko | `tanko-*` | 192.168.68.122 | jerome@.122 | — |
 | Mumuni | `mumuni-jul2026` | 192.168.68.123 | root@.123 | 6 profiles ✱ |
 | Abiba | `abiba-*` | 192.168.68.24 | local | — |
-| Koby | `koby-*` | ? | Zulip | — |
-| Koonimo | `koonimo-*` | 192.168.68.114 | Zulip | — |
+| Tdunna | `tdunna-*` | ? | Zulip | — |
+| Baggy | `baggy-*` | ? | Zulip | — |
 | Kagenz0 | `kagenz0-*` | ? | Zulip | — |

 ✱ Mumuni sub-agents: syslog-code, syslog-devops, syslog-email, syslog-research,
@@ -92,7 +93,8 @@ model:
  base_url: http://192.168.68.116/v1
  api_key_env: LITELLM_API_KEY    # Set in /etc/environment AND ~/.hermes/.env
  max_tokens: 4096                # ⚠️ CRITICAL: Prevents unbounded generation
-  context_length: 262144          # Must match model's max context window
+  context_length: 262144          # For syslog-auto (ornith route supports 256K).
+                                  # Set 131072 if using qwen3.6-27B-code or gemma-4-12b directly.

 fallback_providers:
  provider: deepseek
@@ -123,8 +125,9 @@ compression:
  enabled: true
  model: gemma-4-12b              # ⚠️ Must match auxiliary.compression.model
  provider: harness
-  max_context_window: 262144      # Must match model's actual capacity
-  threshold: 0.65                 # Fires at ~170K for 262K window (not 0.25!)
+  max_context_window: 262144      # For syslog-auto (ornith supports 256K).
+                                  # Set 131072 if using qwen or gemma directly.
+  threshold: 0.65                 # Fires at ~170K for 262K window, ~85K for 128K
  target_ratio: 0.30
  protect_last_n: 40
  hygiene_hard_message_limit: 350
@@ -236,6 +239,28 @@ The following MUST be identical across ALL profiles:
 - `max_context_window: 262144` MUST match the model's actual capacity
 - See `devops-hermes-compression` skill for full reference

+### Rule 9: Default Model Must Be `syslog-auto` (All Agents)
+- **Hermes agents**: `model.default: syslog-auto`, `custom_providers[0].model: syslog-auto`
+- **pi agents**: `defaultModel: syslog-auto` in `settings.json`, first model in `models.json`
+- `syslog-auto` is the LiteLLM routing model — it load-balances between ornith-1.0-35b
+  and qwen3.6-27B-code, with gemma-4-12b as fallback. Using it protects against:
+  - Model name typos that cause 403 errors and silent worker failures
+  - Single GPU downtime (routing falls back automatically)
+  - Key/model authorization mismatches
+- **Exception**: Sub-agent profiles (Mumuni's 6 profiles) may specify explicit models
+  for specialized tasks, but MUST validate those models exist in the key's authorized list
+
+### Rule 10: Validate Model IDs Before Deployment (pi Agents)
+- After configuring a pi agent's `models.json`, verify every model ID:
+  ```bash
+  curl -s http://192.168.68.116:4000/v1/models \
+    -H "Authorization: Bearer <AGENT_KEY>" | jq '.data[].id'
+  ```
+- All model IDs in `models.json` MUST appear in the LiteLLM response
+- The agent's API key may have a SUBSET of the full model catalog — check per-key
+- A non-existent model ID causes 403 errors that silently break the pi RPC worker
+  (no `agent_end` emitted, worker stays "busy", Zulip messages pile up unprocessed)
+
 ## Execution

 1. **Check current config** — Read the target agent's config.yaml
@@ -74,12 +74,21 @@ model:
 Run on any Hermes host to detect violations:

 ```bash
-grep -rn 'api_key: sk-' /root/.hermes/ \
+# 1. Check config.yaml for hardcoded harness keys
+grep -rn 'api_key: sk-' /home/jerome/.hermes/ \
  --include='config.yaml' \
  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
+
+# 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
+grep -rn 'LITELLM_API_KEY' /home/jerome/.config/systemd/user/ 2>/dev/null
+grep -rn 'LITELLM_API_KEY=sk-litellm-7f96080d' /home/jerome/.config/systemd/ 2>/dev/null
+
+# 3. Verify running process env matches dedicated key
+cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
+  | tr '\0' '\n' | grep LITELLM_API_KEY
 ```

-If any output — violation. Fix immediately.
+If any output from step 2 — **critical violation** (master key leaked). Fix immediately.

 ## Rotation Procedure

@@ -102,7 +111,7 @@ curl -s -H "Authorization: Bearer sk-NEW_KEY" http://192.168.68.116/v1/models
 **Keys are permanent and use bare agent name aliases.**

 - **Duration**: `null` — keys never expire. This is enforced by `default_key_generate_params` in `litellm_config.yaml`.
- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `koby`, `koonimo`). No dates, no versions. The alias IS the identity.
+- **Alias convention**: bare agent name only (e.g., `tanko`, `mumuni`, `tdunna`, `baggy`). No dates, no versions. The alias IS the identity.
 - **Rotation triggers**: compromise, personnel departure, or quarterly security hygiene. NOT calendar-driven.
 - **Max budget**: $100 per key (config default).

@@ -117,16 +126,16 @@ litellm_settings:
      purpose: "agent-inference"
 ```

-## Verified Agents (2026-07-04 final)
+## Verified Agents (2026-07-05 update)

-| Agent | CT | Status | LiteLLM Alias | Key Type | Systemd Source | Last Verified |
-|-------|-----|--------|---------------|----------|----------------|---------------|
-| Tanko | 112 | ✅ Compliant | `tanko` | Dedicated | `/etc/environment` only | 23:00 EDT |
-| Mumuni | 114 | ✅ Compliant | `mumuni` | Dedicated | `/etc/environment` only | 23:00 EDT |
-| Koby | 111 | ✅ Compliant | `koby` | Dedicated | User service + drop-in | 23:00 EDT |
-| Koonimo | 113 | ✅ Compliant | `koonimo` | Dedicated | System service + drop-in | 23:00 EDT |
-| Abiba | 100 | ✅ N/A (pi native) | — | — | — | 23:00 EDT |
-| Kagenz0 | 105 | ❌ DOWN | — | — | — | 19:14 EDT |
+| Agent | CT | IP | LiteLLM Alias | Key | Status | Systemd Source | Last Verified |
+|-------|-----|-----|---------------|-----|--------|----------------|---------------|
+| Tanko | 112 | .122 | `tanko` | `sk-CggiHWlamQyShxWC3Hx6uw` | ✅ Fixed | User drop-in `env.conf` | 20:17 UTC Jul 5 |
+| Mumuni | 114 | .123 | `mumuni` | `sk-XY2aUfvy2BIs6kp1ZPh6VA` | ⚠️ Unverified | `/etc/environment` | 23:00 EDT Jul 4 |
+| Tdunna | 111 | ? | `tdunna` | `sk-6sbCNjz2T6lTVDBdlNHXsA` | ✅ Fixed | `/etc/environment` + drop-in | 23:30 UTC Jul 5 |
+| Baggy | 113 | ? | `baggy` | `sk-krnw_zGBwvvL5b7l2t-s-A` | ✅ Fixed | `/etc/environment` | 23:30 UTC Jul 5 |
+| Abiba | 100 | .65 | — | — | ✅ N/A (pi native) | — | 19:44 UTC Jul 5 |
+| Kagenz0 | 105 | ? | — | — | ❌ DOWN | — | 19:14 EDT Jul 4 |

 ### Systemd Service Pattern (2026-07-04 fix)

@@ -180,3 +189,37 @@ auth → validate → lint → ai-review → gate
 - **Status check**: `PR Pipeline — Authorize → Validate → Review → Merge` required before merge
 - **Runner**: `runner-ct110` (Gitea Actions v0.6.1) on CT 110
 - **Config**: `.gitea/workflows/pr-pipeline.yaml`
+
+## Known Bug: auxiliary_client ignores api_key_env (2026-07-05)
+
+**Bug**: `_resolve_task_provider_model()` in `agent/auxiliary_client.py` reads
+`api_key` from auxiliary task configs (vision, compression, etc.) but does NOT
+resolve `api_key_env`. The custom provider resolution path handles `api_key_env`,
+but auxiliary tasks take a different code path that ignores it.
+
+**Impact**: Vision analysis and compression calls fall through to the `"no-key-required"`
+placeholder, causing 401 errors on LiteLLM/harness (which require `sk-*` keys).
+
+**Workaround**: Set `api_key` directly alongside `api_key_env` in each auxiliary
+task config:
+
+```yaml
+auxiliary:
+  vision:
+    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
+    api_key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/v1
+    model: gemma-4-12b
+    provider: harness
+  compression:
+    api_key: sk-CggiHWlamQyShxWC3Hx6uw      # ← workaround
+    api_key_env: LITELLM_API_KEY
+    base_url: http://192.168.68.116/v1
+    model: gemma-4-12b
+    provider: harness
+```
+
+**Affected agents**: All Hermes agents with harness/LiteLLM provider and
+`api_key_env` in auxiliary configs (all 4 Hermes agents patched 2026-07-05).
+
+**Source location**: `agent/auxiliary_client.py` line 5478
@@ -68,7 +68,7 @@ description: >
 | CT 116 (syslog-api) | SSH root | SSH key | ✅ |
 | Tanko CT (.122) | SSH jerome | id_ed25519 | ✅ |
 | Mumuni CT (.123) | SSH root | id_ed25519 | ✅ |
-| Koonimo CT (.114) | SSH jerome | ❌ no key access |
+| Baggy CT (113) | SSH jerome | ❌ no key access |
 | Netbird (.17) | SSH root | SSH key | ✅ |
 | Gitea | API token | abiba-bot token | ✅ |
 | Zulip | Bot API key | abiba-bot@chat.sysloggh.net | ✅ |
@@ -85,7 +85,7 @@ description: >

 ### Reachability Matrix

-| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Koonimo (.114) |
+| From / To | PVE API | docker-vm (.7) | CT 116 | Tanko (.122) | Mumuni (.123) | Baggy (.114) |
 |-----------|---------|----------------|--------|-------------|---------------|----------------|
 | **Abiba** (.24) | ✅ :443 | ✅ SSH | ✅ SSH | ✅ SSH jerome | ✅ SSH root | ❌ SSH |
 | **Tanko** (.122) | ❌ | ❌ | ❌ via NetBird | ✅ | ❌ | ❌ |
@@ -585,7 +585,7 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | 110 | gitea | minipve | — | Git | ❌ |
 | 111 | tdunna | amdpve | — | ? | ❌ |
 | 112 | tanko | amdpve | .122 | Hermes agent | ✅ |
-| 113 | baggy/koonimo | amdpve | .114 | Hermes agent | ✅ |
+| 113 | baggy | amdpve | ? | Hermes agent | ✅ |
 | 114 | mumuni | minipve | .123 | Hermes agent | ✅ |
 | 115 | scottdenya | amdpve | — | ? | ❌ |
 | 116 | syslog-api | minipve | .116 | LiteLLM + Grafana | ❌ |
@@ -602,3 +602,59 @@ ssh root@192.168.68.110 "systemctl restart llama-server"
 | docker-vm (.7) | Audiobookshelf | `/opt/audiobookshelf/docker-compose.yml` |
 | CT 116 (.116) | Inference Harness | `/opt/inference-harness/docker-compose.yml` |
 | Netbird (.17) | Netbird | Docker run (not compose) |
+
+## CT Access (pct-run — no IPs needed)
+
+All CTs are accessible via `pct-run <CT_ID> <command>`. No IP addresses required.
+Source of truth: `/root/scripts/pct-run.sh` or `prose-contracts/scripts/pct-run.sh`.
+
+| CT | Name | Node | pct-run |
+|-----|------|------|---------|
+| 100 | abiba | amdpve | `pct-run 100` |
+| 105 | kagentz | amdpve | `pct-run 105` |
+| 111 | tdunna | amdpve | `pct-run 111` |
+| 112 | tanko | amdpve | `pct-run 112` |
+| 113 | baggy | amdpve | `pct-run 113` |
+| 115 | scottdenya | amdpve | `pct-run 115` |
+| 104 | authentik | minipve | `pct-run 104` |
+| 110 | gitea | minipve | `pct-run 110` |
+| 114 | mumuni | minipve | `pct-run 114` |
+| 116 | syslog-api | minipve | `pct-run 116` |
+| 106 | ra-h-os | storepve | `pct-run 106` |
+| 107 | proxmox-backup | storepve | `pct-run 107` |
+| 108 | media | storepve | `pct-run 108` |
+| 117 | zulip | storepve | `pct-run 117` |
+| 102 | adguard | acerpve | `pct-run 102` |
+
+GPU bare-metal hosts (.8 acerpve, .110 ocupve, .15 amdpve) are NOT CTs — use SSH directly:
+```bash
+ssh root@192.168.68.8   # RTX 3090
+ssh root@192.168.68.110 # RTX 5070
+ssh root@192.168.68.15  # Strix Halo
+```
+
+## Section 7: Agent Health Check (consolidated — 2026-07-05)
+
+Replaces 7 scattered Zulip health scripts with a single non-disruptive check
+running every 10 minutes via cron (`/root/scripts/agent-health-check.py`).
+
+The script is **read-only** — it never restarts, kills, or modifies anything.
+Disruptive cron-based gateways restarts (like Mumuni's zulip-watchdog.sh,
+which was kill+nohup outside systemd) are banned by policy.
+
+### Checks Performed
+
+| Check | Frequency | What It Detects |
+|-------|-----------|-----------------|
+| LiteLLM key validation | 10 min | All 4 agent keys authenticate and return models |
+| GPU port conflict | 10 min | Ghost processes squatting port 8080 (ss vs systemd MainPID) |
+| Gateway liveness | 10 min | Gateway process running, state file readable |
+| Zulip streaming | 10 min | `edit_message` present in adapter (streaming supported) |
+| Recent errors | 10 min | Error count in journald for last 10 min |
+
+### Disabled Scripts
+
+| Script | Why Disabled |
+|--------|-------------|
+| `zulip-watchdog.sh` (Mumuni) | kill+nohup bypassed systemd, 27 restarts, pattern mismatch |
+| `zulip-monitor.sh` (Abiba) | Replaced by agent-health-check.py + PM2 auto-restart |
@@ -3,7 +3,7 @@ kind: responsibility
 name: infrastructure-update
 description: >
  Autonomous system-wide update contract covering all 5 Proxmox nodes,
-  15+ containers/VMs, and 3 Docker ecosystems. Updates apt packages,
+  15+ containers/VMs, and 4 Docker ecosystems. Updates apt packages,
  Docker images, and container stacks in safe waves with health checks
  and automatic rollback on failure.
 agent: abiba
@@ -11,7 +11,7 @@ triggers:
  - on "infra update" command
  - weekly (Sunday 03:00 EDT) via cron
  - on security advisory relay from Mumuni
-version: 1.0.0
+version: 1.1.0
 ---

 ## Maintains
@@ -24,11 +24,11 @@ version: 1.0.0

 Before ANY update wave:
 1. ✅ All Proxmox nodes online (`GET /api2/json/nodes`)
-2. ✅ All critical CTs running (100, 109, 116, 117, 122, 123)
-3. ✅ Docker healthy on .7, .116
+2. ✅ All critical VMs/CTs running (VM 101 llm-gpu, VM 103 ocu-llm, VM 109 docker-vm, CT 116 syslog-api, CT 117 zulip, CT 106 ra-h-os)
+3. ✅ Docker healthy on VM 109 (.7), CT 116 (.116)
 4. ✅ LiteLLM health check passing
 5. ✅ Zulip server reachable
-6. ✅ GPU fleet healthy (all 3 GPUs)
+6. ✅ GPU fleet healthy (all 3 GPUs: RTX 3090, RTX 5070, RX 7600)
 7. ✅ Disk >20% free on all nodes
 8. 📋 Snapshot critical configs (LiteLLM, nginx, docker-compose files)

@@ -38,7 +38,7 @@ Before ANY update wave:
 |--------|------|---------|---------|
 | storepve (.6) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | minipve (.12) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
-| CT 109 (.7) | Docker host | `apt update && apt upgrade -y` | 5 min |
+| VM 109 (.7) | Docker host VM | `apt update && apt upgrade -y` | 5 min |
 | CT 106 (.65) | RA-H OS | `apt update && apt upgrade -y` | 3 min |
 | CT 117 (zulip, storepve) | Zulip | `apt update && apt upgrade -y` | 3 min |

@@ -55,16 +55,16 @@ Before ANY update wave:
 | acerpve (.9) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | ocupve (.5) | Proxmox node | `apt update && apt upgrade -y` | 5 min |
 | CT 100 (.24) | Abiba (pi) | `apt update && apt upgrade -y` | 3 min |
-| CT 116 (.116) | LiteLLM host | `apt update && apt upgrade -y` | 3 min |
+| CT 116 (.116) | syslog-api (LiteLLM host) | `apt update && apt upgrade -y` | 3 min |
 | CT 112 (tanko, amdpve) | Tanko | `apt update && apt upgrade -y` | 3 min |
 | CT 114 (mumuni, minipve) | Mumuni | `apt update && apt upgrade -y` | 3 min |
-| CT 101 (.8) | RTX 3090 GPU | `apt update && apt upgrade -y` | 3 min |
-| CT 103 (.110) | RTX 5070 GPU | `apt update && apt upgrade -y` | 3 min |
+| VM 101 (.8) | llm-gpu (RTX 3090) | `apt update && apt upgrade -y` | 3 min |
+| VM 103 (.110) | ocu-llm (RTX 5070) | `apt update && apt upgrade -y` | 3 min |

 **Verify after Wave 2:**
- All CTs running: check via Proxmox API
- LiteLLM healthy: `curl :4001/health/liveliness`
- GPU servers responding: check :8080 on .8, .110; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
+- All VMs/CTs running: check via Proxmox API
+- LiteLLM healthy: `curl localhost:4000/health/liveliness` (via CT 116)
+- GPU servers responding: check :8080 on VM 101, VM 103; check ornith via router (http://192.168.68.116/health/unified — .15:8080 is firewalled to .116 only)
 - Zulip agents connected: check Mumuni/Tanko gateway state
 - Abiba PM2 processes online: `pm2 status`

@@ -72,20 +72,20 @@ Before ANY update wave:

 | Host | Stack | Command |
 |------|-------|---------|
-| CT 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
-| CT 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
-| CT 109 (.7) | Home stack | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
-| CT 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
-| CT 116 (.116) | Inference Harness | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Firecrawl | `cd /opt/search-stack/firecrawl-source && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | SearXNG | `cd /opt/search-stack/searxng && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Home stack (Pulse, Stirling PDF, JDownloader 2) | `cd /opt/home_stack && docker compose pull && docker compose up -d` |
+| VM 109 (.7) | Audiobookshelf | `cd /opt/audiobookshelf && docker compose pull && docker compose up -d` |
+| CT 116 (.116) | Inference Harness (LiteLLM, Prometheus, Grafana) | `cd /opt/inference-harness && docker compose pull && docker compose up -d` |
 | CT 117 (zulip, storepve) | Zulip | `docker pull zulip/docker-zulip:latest && docker restart zulip-zulip-1` |

 **Verify after Wave 3:**
 - All containers healthy: `docker ps` on each host
- End-to-end inference test: `curl :4001/v1/chat/completions` with syslog-auto
+- End-to-end inference test: `curl localhost:4000/v1/chat/completions` (via CT 116) with syslog-auto
 - Zulip test: send test message to #agent-hub
- Dashboard loading: `curl :3001/`
- Firecrawl test: `curl :3002/health`
- SearXNG test: `curl storepve:8888`
+- Dashboard loading: `curl localhost:3001/` (via CT 116)
+- Firecrawl test: `curl :3002/`
+- SearXNG test: `curl :8888`

 ## Wave 4: Proxmox Kernel Reboot (if needed)

@@ -111,16 +111,20 @@ If ANY verification fails:

 Before Wave 1, snapshot these files:
 ```
-/opt/inference-harness/litellm_config.yaml        (.116)
-/opt/inference-harness/docker-compose.yml          (.116)
-/etc/nginx/nginx.conf                              (.116 container)
-/opt/search-stack/*/docker-compose.yaml            (.7)
-/root/.pi/agent/extensions/config.yaml             (.24)
-/etc/systemd/system/ornith-server.service          (.15)
-/etc/systemd/system/llama-server.service           (.8, .110)
+/opt/inference-harness/docker-compose.yml               (CT 116 .116)
+/opt/inference-harness/litellm_config.yaml              (CT 116 .116)
+/opt/monitoring/prometheus.yml                          (CT 116 .116)
+/etc/nginx/nginx.conf                                   (harness-nginx on CT 116)
+/opt/search-stack/firecrawl-source/docker-compose.yaml  (VM 109 .7)
+/opt/search-stack/searxng/docker-compose.yml            (VM 109 .7)
+/opt/home_stack/docker-compose.yml                      (VM 109 .7)
+/opt/audiobookshelf/docker-compose.yml                  (VM 109 .7)
+/root/.pi/agent/extensions/config.yaml                  (CT 100 .24)
+/etc/systemd/system/ornith-server.service               (amdpve .15)
+/etc/systemd/system/llama-server.service                (VM 101 .8, VM 103 .110)
 ```

-Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
+Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && rsync -av ...`

 ## Security-Specific Updates

@@ -134,8 +138,8 @@ Run: `mkdir -p /tmp/infra-update-backup-$(date +%Y%m%d) && scp ...`
 ## Success Criteria

 - [ ] All 5 PVE nodes updated, no reboot-loop
- [ ] All CTs running post-update
- [ ] All 22 Docker containers healthy
+- [ ] All VMs/CTs running post-update
+- [ ] All Docker containers healthy (VM 109 + CT 116 + CT 117)
 - [ ] LiteLLM inference passing (syslog-auto test)
 - [ ] Zulip server + all 3 agents connected
 - [ ] GPU fleet at full capacity (3/3)
@@ -148,7 +152,7 @@ After completion, send Zulip DM:
 ```
 📋 Infrastructure Update — YYYY-MM-DD

-Updated: 5 PVE nodes, 12 CTs, 22 containers
+Updated: 5 PVE nodes, 12 CTs/VMs, 30+ containers
 Security fixes: N CVEs patched
 Downtime: <service> <duration>
 Failures: none / <details>
@@ -1,7 +1,7 @@
 ---
 name: memory-audit-maintenance
 kind: responsibility
-description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
+description: Shared memory audit and maintenance contract for all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own isolated memory files — no cross-agent access, no shared state. Detects staleness, enforces writer registry, and rotates canary tokens.
 id: 067NC4KG01RG50R40M30E20918
 ---

@@ -15,13 +15,13 @@ Autonomously audit and reorganize an agent's native memory (MEMORY.md, USER.md,

 ### Scope

-This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Koby, Koonimo). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.
+This contract is the **Hermes Agent standard** for memory maintenance. It is shared across all Hermes agents (Mumuni, Tanko, Tdunna, Baggy). Each agent runs it against its own memory files only  no cross-agent access, no shared state, no shared ledger, no shared canary. The contract is the standard; each agent enforces it independently with fully isolated data.

 **Agent Roster:**
 - Mumuni
 - Tanko
- Koby
- Koonimo
+- Tdunna
+- Baggy

 **Isolation Principle:** Each agent has its own:
 - `MEMORY.md` and `USER.md`
@@ -343,4 +343,4 @@ return {

 ### Per-Agent Notes

-Each Hermes agent (Mumuni, Tanko, Koby, Koonimo) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
+Each Hermes agent (Mumuni, Tanko, Tdunna, Baggy) runs this contract against its own `~/.hermes/memories/` directory. The contract is identical across agents, but all data is fully isolated: separate ledgers, separate writer registries, separate canaries. If a new agent is added to the roster, it must be listed in `### Scope` above and given its own isolated memory directory.
@@ -0,0 +1,228 @@
+#!/usr/bin/env python3
+"""
+/root/scripts/agent-health-check.py — Consolidated Agent Health Verification
+
+Single non-disruptive health check replacing 7 scattered scripts.
+Verifies: LiteLLM keys, GPU port conflicts, agent Zulip streaming,
+gateway liveness, and gateway log health. NEVER restarts anything.
+
+Usage:
+  python3 /root/scripts/agent-health-check.py          # Full check
+  python3 /root/scripts/agent-health-check.py --json   # Machine-readable
+  python3 /root/scripts/agent-health-check.py --quiet  # Only output on failure
+
+Cron: */10 * * * * python3 /root/scripts/agent-health-check.py --quiet
+"""
+
+import subprocess, json, sys, os, time
+from datetime import datetime
+
+LITELLM = "http://192.168.68.116:80"
+
+AGENTS = {
+    "tanko":   {"ct": 112, "host": "192.168.68.122", "key": "sk-CggiHWlamQyShxWC3Hx6uw", "user": "jerome"},
+    "mumuni":  {"ct": 114, "host": "192.168.68.123", "key": "sk-VrqCNlwUgzoNGOpikJ7nwQ", "user": "root"},
+    "tdunna":  {"ct": 111, "host": None,              "key": "sk-6sbCNjz2T6lTVDBdlNHXsA", "user": None},
+    "baggy":   {"ct": 113, "host": None,              "key": "sk-krnw_zGBwvvL5b7l2t-s-A", "user": None},
+}
+
+GPU_HOSTS = {
+    "gpu-rtx3090 (.8)":     {"host": "192.168.68.8",   "port": 8080, "service": "llama-server"},
+    "gpu-rtx5070 (.110)":   {"host": "192.168.68.110", "port": 8080, "service": "llama-server"},
+    "gpu-strixhalo (.15)":  {"host": "192.168.68.15",  "port": 8080, "service": "ornith-server"},
+}
+
+FAIL = []
+
+def ssh(host, cmd, user="root"):
+    """Execute a command on a remote host, return stdout or None."""
+    try:
+        result = subprocess.run(
+            ["ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=8",
+             f"{user}@{host}", cmd],
+            capture_output=True, text=True, timeout=15
+        )
+        return result.stdout.strip() if result.returncode == 0 else None
+    except:
+        return None
+
+def http_get(url, headers=None, timeout=5):
+    """Return HTTP status code as string."""
+    try:
+        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout), "-o", "/dev/null", "-w", "%{http_code}"]
+        if headers:
+            for k, v in headers.items():
+                cmd.extend(["-H", f"{k}: {v}"])
+        cmd.append(url)
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
+        return result.stdout.strip() or "000"
+    except:
+        return "timeout"
+
+def http_json(url, headers=None, timeout=5):
+    """Return parsed JSON from URL, or None."""
+    try:
+        cmd = ["curl", "-sfk", "--connect-timeout", str(timeout)]
+        if headers:
+            for k, v in headers.items():
+                cmd.extend(["-H", f"{k}: {v}"])
+        cmd.append(url)
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout+3)
+        return json.loads(result.stdout) if result.returncode == 0 and result.stdout else None
+    except:
+        return None
+
+
+# ═══════════════════════════════════════════════════════════════════
+# CHECK 1: LiteLLM Key Validation
+# ═══════════════════════════════════════════════════════════════════
+
+def check_keys():
+    for name, agent in AGENTS.items():
+        data = http_json(f"{LITELLM}/v1/models",
+                         headers={"Authorization": f"Bearer {agent['key']}"})
+        if data and data.get("data"):
+            model = data["data"][0].get("id", "?")
+            print(f"  ✅ {name}: key valid → {model}")
+        else:
+            print(f"  ❌ {name}: KEY FAILURE — auth rejected or unreachable")
+            FAIL.append(f"key:{name}")
+
+
+# ═══════════════════════════════════════════════════════════════════
+# CHECK 2: GPU Port Conflict Detection
+# ═══════════════════════════════════════════════════════════════════
+
+def check_gpu_ports():
+    for label, gpu in GPU_HOSTS.items():
+        host = gpu["host"]
+        port = gpu["port"]
+        svc = gpu["service"]
+
+        svc_status = ssh(host, f"systemctl is-active {svc}")
+        port_owner = ssh(host, f"ss -tlnp 2>/dev/null | grep -Po ':{port}\\s+.*pid=\\K[0-9]+' | head -1")
+
+        if not svc_status:
+            print(f"  ❌ {label}: UNREACHABLE")
+            FAIL.append(f"gpu-unreachable:{host}")
+            continue
+
+        if not port_owner:
+            print(f"  ❌ {label}: PORT {port} NOT LISTENING (svc={svc_status})")
+            FAIL.append(f"gpu-no-port:{label}")
+        elif svc_status != "active":
+            svc_pid = ssh(host, f"systemctl show {svc} -p MainPID 2>/dev/null | cut -d= -f2")
+            if svc_pid and port_owner != svc_pid:
+                print(f"  ❌ {label}: GHOST PROCESS — port owned by pid {port_owner}, svc pid {svc_pid} (svc={svc_status})")
+                FAIL.append(f"gpu-ghost:{label}:{port_owner}")
+            else:
+                print(f"  ⚠️  {label}: svc={svc_status}, port owned by {port_owner}")
+        else:
+            # Verify health endpoint
+            health = ssh(host, f"curl -s --max-time 5 http://localhost:{port}/health")
+            if health and '"status":"ok"' in health:
+                print(f"  ✅ {label}: healthy (pid={port_owner})")
+            elif health and '"status":"no slot available"' in health:
+                print(f"  ✅ {label}: healthy (loading, pid={port_owner})")
+            elif health and '"error"' in health.lower():
+                print(f"  ⚠️  {label}: error response (pid={port_owner}): {health[:80]}")
+            else:
+                print(f"  ⚠️  {label}: unknown health (pid={port_owner}): {str(health)[:80]}")
+
+
+# ═══════════════════════════════════════════════════════════════════
+# CHECK 3: Agent Gateway Liveness + Streaming
+# ═══════════════════════════════════════════════════════════════════
+
+def check_agents():
+    for name, agent in AGENTS.items():
+        host = agent.get("host")
+        user = agent.get("user")
+        ct = agent["ct"]
+
+        if not host or not user:
+            print(f"  ⬜ {name} (CT {ct}): cannot SSH — skip liveness check")
+            continue
+
+        # Gateway process
+        pid = ssh(host, "pgrep -f 'hermes_cli.main gateway run' | head -1", user=user)
+        if not pid:
+            print(f"  ❌ {name}: GATEWAY NOT RUNNING")
+            FAIL.append(f"gateway-down:{name}")
+            continue
+
+        # Gateway state file
+        state = ssh(host, "cat ~/.hermes/gateway_state.json 2>/dev/null", user=user)
+        if state:
+            try:
+                st = json.loads(state)
+                gw_state = st.get("gateway_state", "?")
+                zulip = st.get("platforms", {}).get("zulip", {}).get("state", "?")
+            except:
+                gw_state, zulip = "corrupt", "?"
+        else:
+            gw_state, zulip = "no-state-file", "?"
+
+        # Zulip streaming: does adapter have edit_message?
+        adapter_paths = [
+            "~/.hermes/plugins/zulip-platform/adapter.py",
+            "~/.hermes/plugins/platforms/zulip/adapter.py",
+        ]
+        streaming = "no"
+        for p in adapter_paths:
+            has_edit = ssh(host, f"grep -c 'async def edit_message' {p} 2>/dev/null", user=user)
+            if has_edit and has_edit != "0":
+                streaming = "yes"
+                break
+
+        # Recent errors
+        recent_errors = ssh(host,
+            r"journalctl --user -u hermes-gateway --since '10 min ago' -o cat --no-pager 2>/dev/null "
+            r"| grep -ci 'error\|traceback\|exception\|401\|403\|500' || echo 0",
+            user=user)
+        recent_errors = (recent_errors or "0").strip().split("\n")[-1]  # take last line
+
+        print(f"  {'✅' if gw_state == 'running' and zulip == 'connected' else '⚠️'} "
+              f"{name}: gw={gw_state} zulip={zulip} streaming={streaming} "
+              f"errors_10m={recent_errors.strip() or '0'} pid={pid}")
+
+
+# ═══════════════════════════════════════════════════════════════════
+# MAIN
+# ═══════════════════════════════════════════════════════════════════
+
+def main():
+    quiet = "--quiet" in sys.argv
+    as_json = "--json" in sys.argv
+
+    if not quiet:
+        print(f"🏥 Agent Health Check — {datetime.now().strftime('%Y-%m-%d %H:%M UTC')}")
+        print()
+
+    print("🔑 LiteLLM Keys:")
+    check_keys()
+    print()
+
+    print("🎮 GPU Port Health:")
+    check_gpu_ports()
+    print()
+
+    print("🤖 Agent Gateways:")
+    check_agents()
+
+    if FAIL:
+        print(f"\n❌ {len(FAIL)} FAILURE(S): {' | '.join(FAIL)}")
+        if quiet:
+            # In quiet mode, only print failures as a single alert line
+            print(f"ALERT agent-health:{','.join(FAIL)}")
+    elif not quiet:
+        print("\n✅ All checks passed")
+
+    if as_json:
+        print(json.dumps({"timestamp": datetime.now().isoformat(),
+                          "failures": FAIL, "healthy": len(FAIL) == 0}))
+
+    sys.exit(1 if FAIL else 0)
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+# pct-run — Run commands inside Proxmox CTs by CT ID only. No IPs needed.
+# Usage: pct-run <CT_ID> [command...]
+# If no command given, opens a shell.
+#
+# Source of truth: Proxmox pct config on each node.
+# No hardcoded IPs. No prose contract drift.
+
+set -euo pipefail
+
+# ── CT ID → PVE Node mapping (maintained HERE, not in prose contracts) ──
+declare -A CT_NODES=(
+    # amdpve (192.168.68.15)
+    [100]=amdpve    # abiba
+    [105]=amdpve    # kagentz
+    [111]=amdpve    # tdunna
+    [112]=amdpve    # tanko
+    [113]=amdpve    # baggy
+    [115]=amdpve    # scottdenya
+    # minipve (192.168.68.12)
+    [104]=minipve   # authentik
+    [110]=minipve   # gitea
+    [114]=minipve   # mumuni
+    [116]=minipve   # syslog-api
+    # storepve (192.168.68.6)
+    [106]=storepve  # ra-h-os
+    [107]=storepve  # proxmox-backup
+    [108]=storepve  # media
+    [117]=storepve  # zulip
+    # acerpve (192.168.68.9)
+    [102]=acerpve   # adguard
+    # ocupve (192.168.68.5) — no CTs (bare metal GPU .110)
+)
+
+# Each node must be root-accessible via SSH hostname
+# (Ensure ~/.ssh/config or /etc/hosts has these resolvable)
+
+# ── PVE node → IP (needed only because SSH needs an address) ──
+declare -A NODE_IPS=(
+    [amdpve]=192.168.68.15
+    [minipve]=192.168.68.12
+    [storepve]=192.168.68.6
+    [acerpve]=192.168.68.9
+    [ocupve]=192.168.68.5
+)
+
+resolve_node() {
+    local ct_id="$1"
+    local node="${CT_NODES[$ct_id]:-}"
+    if [[ -z "$node" ]]; then
+        echo "ERROR: CT $ct_id not found in mapping" >&2
+        exit 1
+    fi
+    echo "$node"
+}
+
+node_ip() {
+    local node="$1"
+    local ip="${NODE_IPS[$node]:-}"
+    if [[ -z "$ip" ]]; then
+        echo "ERROR: Node $node not found in IP mapping" >&2
+        exit 1
+    fi
+    echo "$ip"
+}
+
+main() {
+    if [[ $# -lt 1 ]]; then
+        echo "Usage: pct-run <CT_ID> [command...]" >&2
+        echo "  pct-run 112 cat /etc/hostname" >&2
+        echo "  pct-run 114 systemctl status hermes-gateway" >&2
+        echo ""
+        echo "Known CTs:" >&2
+        for ct in $(echo "${!CT_NODES[@]}" | tr ' ' '\n' | sort -n); do
+            echo "  CT $ct → ${CT_NODES[$ct]}" >&2
+        done
+        exit 1
+    fi
+
+    local ct_id="$1"
+    shift
+    local node
+    node=$(resolve_node "$ct_id")
+    local ip
+    ip=$(node_ip "$node")
+
+    if [[ $# -eq 0 ]]; then
+        # Interactive shell
+        exec ssh -t "root@${ip}" "pct enter ${ct_id}"
+    else
+        # One-shot command
+        ssh "root@${ip}" "pct exec ${ct_id} -- $*"
+    fi
+}
+
+main "$@"
@@ -73,6 +73,25 @@ description: >
 - **Fix**: If edit fails, send the response as a new message instead
 - **Applies to**: pi extension (fixed), Hermes adapter (verify edit fallback exists)

+### 11. Model-ID Mismatch Causes Silent Worker Failure (pi-specific)
+- **Symptom**: User sends DM, sees placeholder, but never gets response. Worker stays
+  "busy" forever, accumulating pending replies (10+). Health endpoint shows green but
+  `messages_processed` stalls.
+- **Root cause**: Agent's model config (`models.json` or `settings.json`) references a
+  model ID that doesn't exist in LiteLLM's authorized model list. Example: `qwen3.6-35B-A3B`
+  configured but LiteLLM only exposes `ornith-1.0-35b` under that key. pi's session
+  workers emit 403 on first prompt, then never recover because the error doesn't trigger
+  `agent_end` — worker stays `busy` and all subsequent messages pile up in the steer queue.
+- **Detection**: Compare `~/.pi/agent/models.json` model IDs against `curl -H "Authorization: Bearer <KEY>" http://192.168.68.116/v1/models` output. A stuck worker shows
+  `workers=[<id>:busy:N]` with growing N in extension logs.
+- **Fix**: (1) Update `models.json` to only include models from the authorized list.
+  (2) Set `defaultModel` to `syslog-auto` (safe routing model). (3) Delete stale session
+  JSONL files from `~/.pi/agent/sessions/zulip/`. (4) Restart PM2 process.
+- **Prevention**: Use `syslog-auto` as default model for all agents — it handles model
+  routing and fallback automatically. Direct model IDs (`ornith-1.0-35b`, etc.) should
+  only be used when explicitly requested. Validate model IDs at agent setup time.
+- **Applies to**: pi extension (Tdunna CT111, fixed 2026-07-08), any agent using `syslog-harness` provider
+
 ## Deployment Checklist

 When deploying a new Zulip adapter, verify:
@@ -86,3 +105,4 @@ When deploying a new Zulip adapter, verify:
 - [ ] `@all-bots` detected via configurable user_id
 - [ ] Poll uses long-poll (not `dont_block=true` polling)
 - [ ] Stuck/idle detection accounts for quiet periods
+- [ ] Model IDs in config validated against `GET /v1/models` with actual API key
@@ -77,6 +77,24 @@ Log the condition as "pending restart" with the timestamp. If the condition pers
 Log as "unreachable" — don't treat as critical unless it persists for 3+ consecutive checks.

 ### Severity escalation
+
+## Streaming Support (2026-07-05)
+
+Zulip agents now support progressive message editing during agent generation.
+When a Hermes agent (Tanko, Mumuni) processes a message, the response is
+streamed in real-time via Zulip's `PATCH /api/v1/messages/{id}` API:
+
+- Adapter implements `edit_message()` using `_api_patch()` helper
+- Gateway stream consumer progressively edits the Zulip message
+- User sees real-time agent thinking instead of waiting for full response
+- Verified: Tanko (CT 112) and Mumuni (CT 114) both have streaming active
+
+### Verification
+```bash
+# Check if agent has streaming:
+grep -c "async def edit_message" ~/.hermes/plugins/*/zulip*/adapter.py
+# Must return 1 — streaming is active
+```
 - Single agent warning → log only
 - Single agent critical → relay message to user
 - Two or more agents critical → immediate relay + attempt auto-recovery
Author	SHA1	Message	Date
root	776df418ca	gpu-fleet: July 8 optimization sweep — parallel 2 fleet-wide, 128K ctx on NVIDIA, LiteLLM timeout fixes PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details - All GPUs: --parallel 1 → 2 (6 concurrent slots, was 3) - .8 RTX 3090: ctx 256K→128K, VRAM 96%→83%, turbo4 KV cache - .110 RTX 5070: ctx 256K→128K, ubatch 4096→512 (was inverted), VRAM 90%→77%, q4_0 KV - .15 Strix Halo: parallel 1→2, 256K ctx (41GB free), q8_0 KV, AMD metrics via /sys/class/drm - LiteLLM: gemma timeout 25→120s, qwen timeout 40→90s, syslog-auto (qwen) 40→90s - Agent configs: context_length 262144 for syslog-auto, 131072 for direct qwen/gemma - Updated health-check operation, agent config implications, benchmark table (fixed model↔GPU mapping)	2026-07-08 17:25:42 +00:00
jerome	f6e8791352	fix(ci): change kind from reference to template in hermes-agent-baseline.prose.md PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details The validate job was failing because 'reference' is not an allowed kind. Changed to 'template' since this document serves as a canonical baseline template for restoring agent configurations.	2026-07-07 20:57:48 -04:00
root	b1ef4cbe9b	docs: self-update contract — Gen 6 postconditions, status clarification, state persistence PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 7s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details - Added stream_subscription_check, dm_response_time_offline_verifiable, known_issues_tracked postconditions - Added 'Status' section clarifying active for Hermes agents only (not retired with pi Zulip) - Added 'State Persistence' section with KG node schema for build-zulip-plugin:state - Execution step 4: always run offline tests (simulate_dm_latency, selftest) - Execution step 5: update KG state node after each generation	2026-07-07 07:18:45 +00:00
kagentz-bot	41ef546d21	Merge pull request 'fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net for CI checkout' (#4 ) from fix/ci-checkout-ip into master PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 5s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 5s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details	2026-07-06 22:56:48 +00:00
Agent Zero	8f89603c31	fix: use IP 192.168.68.17:3000 instead of git.sysloggh.net hostname for CI checkout PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Failing after 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Has been skipped Details The Gitea Actions runner cannot resolve the internal hostname git.sysloggh.net during checkout. Using the direct IP address resolves the DNS resolution issue in CI pipeline steps.	2026-07-06 18:54:12 -04:00
Agent Zero	f1a031b682	fix: Firecrawl health endpoint is / not /health PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details	2026-07-06 18:21:30 -04:00
Agent Zero	f14b2fc04e	infra-update contract: v1.1.0 - fix CT→VM refs, remove deprecated stacks PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details	2026-07-06 18:05:16 -04:00
Agent Zero	09ffbf10aa	infra-update contract: v1.1.0 — fix CT→VM refs, add missing stacks, correct paths	2026-07-06 18:05:16 -04:00
root	5aa93117de	docs: capture all session changes — port conflict, health check, streaming PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details Updated contracts to reflect all infrastructure changes made 2026-07-05/06: gpu-fleet.prose.md: - Port conflict detection on all 3 GPU wrappers - Ghost process root cause documented (.8 stale pid 25836) infrastructure-control.prose.md: - Section 7: Agent Health Check (consolidated, non-disruptive) - Disabled scripts documented (zulip-watchdog, zulip-monitor) hermes-agent-baseline.prose.md: - Health verification section + GPU port conflict detection - Change log updated zulip-health.prose.md: - Streaming support section (edit_message + PATCH API)	2026-07-06 02:46:41 +00:00
root	79855ea9e1	feat: consolidated agent health check — key validation + GPU port conflict + streaming Single non-disruptive script replacing 7 scattered Zulip health checks. Checks every 10 min via cron, never restarts anything: - LiteLLM key validation for all 4 Hermes agents - GPU port conflict / ghost process detection - Agent gateway liveness + Zulip streaming status - Recent gateway error count Port conflict detection added to all 3 GPU wrappers: - .8 (qwen): wrapper detects ghost on port 8080 before starting - .110 (gemma): same pattern - .15 (ornith): port-cleanup.sh replaces blanket pkill -x llama-server	2026-07-06 02:23:54 +00:00
root	2ea9cac23a	docs: hermes-agent-baseline — canonical good-state reference PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details Single source of truth for all 4 Hermes agents. Captures: - Agent map (CT, IP, key, alias) - Key architecture and config patterns - Mandatory api_key workaround with explanation - Known bug: auxiliary_client ignores api_key_env - Audit procedure (full audit, leak check, key verification) - Systemd patterns - Attachment cache locations	2026-07-05 23:37:56 +00:00
root	35e864bdb7	audit: all 4 Hermes agents verified — no master key, api_key workaround applied PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Agent audit (2026-07-05): Tanko CT 112: sk-CggiHWlamQyShxWC3Hx6uw — vision+compression patched Mumuni CT 114: sk-VrqCNlwUgzoNGOpikJ7nwQ — vision patched Tdunna CT 111: sk-6sbCNjz2T6lTVDBdlNHXsA — vision+compression patched Baggy CT 113: sk-krnw_zGBwvvL5b7l2t-s-A — vision+compression patched All 4 agents: zero master key references, dedicated LiteLLM keys, api_key workaround applied to bypass auxiliary_client bug.	2026-07-05 23:36:00 +00:00
root	53d4290886	docs: document auxiliary_client api_key_env bug + workaround PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details _resolve_task_provider_model() in agent/auxiliary_client.py ignores api_key_env for auxiliary tasks (vision, compression). The custom provider path resolves it, but aux tasks take a different code path. Workaround: set api_key directly alongside api_key_env. Applied to Tanko (vision + compression).	2026-07-05 23:17:37 +00:00
root	c8dcd8b63d	fix: Tanko key regenerated, Mumuni key verified from /etc/environment PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 1s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 1s Details Tanko: old key sk-620a05e95a... was from docker-compose router config, not LiteLLM key DB. Deleted + regenerated via LiteLLM API: sk-CggiHWlamQyShxWC3Hx6uw (alias: tanko, budget: $100) Mumuni: verified via pct-run 114 — key is sk-VrqCNlwUgzoNGOpikJ7nwQ (was previously listed as sk-XY2aUfvy2... in contracts)	2026-07-05 20:18:05 +00:00
root	5c90eb8f6c	rename: Koonimo→Baggy, Koby→Tdunna — match hostnames PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Successful in 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Successful in 4s Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Successful in 0s Details Standardize agent names across all contracts and pct-run.sh. No functional changes — same CT IDs, same keys, same nodes.	2026-07-05 20:04:22 +00:00
root	b65116b847	feat: pct-run.sh — run commands in CTs by ID, no IPs needed PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 3s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details Adds /scripts/pct-run.sh that maps CT IDs to PVE nodes and runs pct exec directly. No hardcoded IPs — Proxmox is the source of truth. Updates infrastructure-control with CT access table using pct-run. Usage examples: pct-run 112 cat /etc/hostname # Tanko pct-run 114 grep LITELLM /etc/environment # Mumuni key check pct-run 116 docker ps # LiteLLM containers	2026-07-05 20:02:20 +00:00
root	0db95a4d1c	fix: correct IPs, CT IDs, and key status across all contracts PR Pipeline — Authorize → Validate → Review → Merge / auth (push) Successful in 3s Details PR Pipeline — Authorize → Validate → Review → Merge / validate (push) Failing after 2s Details PR Pipeline — Authorize → Validate → Review → Merge / lint (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / ai-review (push) Has been skipped Details PR Pipeline — Authorize → Validate → Review → Merge / gate (push) Has been skipped Details hermes-key-enforcement.prose.md: - Add IP column to verified agents table - Fix Tanko status: was using master key via systemd drop-in (not compliant) - Add detection query step 2: check systemd drop-ins for master key leaks - Add step 3: verify running process env against dedicated key - Mark Mumuni, Koby, Koonimo as Unverified (last check was surface-level only) gpu-fleet.prose.md: - Add CT ID column to agent table for cross-reference - Fix Abiba IP .24 → .65 (pi agent runs on RA-H OS host) - Update Abiba LiteLLM key to match actual models.json key - Set Koonimo IP to unknown (was .114 — incorrect per user) - Fix pi-specific paths .24 → .65 hermes-config-template.prose.md: - Set Koonimo IP to unknown infrastructure-control.prose.md: - Set Koonimo IP to unknown - Fix Koonimo CT reference	2026-07-05 19:58:14 +00:00