Merge master into fix/search-stack-multiprovider-20261003 (pick up the #150 lint allowlist fix so the PR's lint job can pass)

The synthetic example 'sk-synthetic-litellm-' in hermes-key-enforcement.prose.md:217
was flagged by the secret-assign rule as a credential-shaped assignment. The
existing allowlist entry only covered openai-key, so secret-assign still failed.

Fix: Change the rule from openai-key to * (matches any rule) and update the
reason to explain why (2026-10-03: secret-assign also matches the credential-shaped
assignment).

This restores prose-lint on master, which was RED and blocking every open PR's
lint job (including PR #149's lint job 2160).

Chose option (b) (allowlist) over option (a) (restructure) because:
- The existing entry already documented the synthetic example
- Changing the rule to * is the minimal, honest fix
- Restructuring the command would risk weakening the enforcement check

Correlation: corr=7c202734ec6c452a

hermes-agent-baseline.prose.md

@@ -70,10 +70,10 @@ Agent (systemd) → LITELLM_API_KEY → LiteLLM (:116/v1) → GPU (llama-server)
 
 ## Config Pattern — Mandatory Fields
 
-### For Hermes Agents (Mumuni, Koonimo)
+### For Hermes Agents (Mumuni, Koonimo, Tanko-hybrid)
 
 Every Hermes agent's `/root/.hermes/config.yaml` (or `/home/jerome/.hermes/config.yaml`) MUST have:
-(Tanko is excluded — migrated to DSH/DeepSeek Harness on 2026-08-27, no longer uses Hermes config.)
+(Tanko is hybrid — runs both DSH and Hermes since 2026-08-27, so its Hermes config is also checked.)
 
 ### 1. Main Model
 ```yaml
@@ -297,7 +297,7 @@ Run the consolidated health check:
 ```bash
 python3 /root/scripts/agent-health-check.py
 ```
-This validates tanko's live LiteLLM proxy at 192.168.68.116:4000 (valid master key; the other 3 CTs are DSH-only), detects GPU port conflicts (ghost processes),
+This validates each agent's live LiteLLM key against the gateway, including tanko, which runs HYBRID (DSH + Hermes) since 2026-08-27; detects GPU port conflicts (ghost processes),
 verifies gateway liveness, confirms Zulip streaming (`edit_message` present),
 and counts recent errors. Non-disruptive — never restarts anything.
 

hermes-key-enforcement.prose.md

@@ -186,30 +186,60 @@ Agent keys live in `.env` or `.env.vault` files with 600 permissions (koonimo's
 
 ## Detection Query
 
-Run on any Hermes host to detect violations:
+Run on any Hermes host to detect violations.
+
+**Timeout policy (2026-10-02):** The scan timeout is **15 seconds**, set from measured cost on the largest target (koby, 16 GB `.hermes` tree; full scan: **cold ≈ 5.7 s**, warm ≈ 0.44 s; bounded scan: warm ≈ 0.37 s, over SSH, measured 2026-10-02). The 15 s bound is justified by the COLD cost, not the warm cost — a 13× cold/warm spread means the warm figure alone would understate the real worst case by an order of magnitude. The SSH connection timeout is **10 seconds** (separate from the scan timeout). A scan timeout renders as `probe-failed: <agent> <ip> (timeout after 15s)` — **never** as "unreachable" or "may be down". An SSH connection failure (exit status 255) renders as `unreachable: <agent> <ip> (ssh connect failed)`. The original failure (2026-10-02 koby) was a slow/cold scan that exceeded whatever bound the prior run used and was rendered as a host-down verdict; the exact prior timeout was never reproduced, so this is the only proven fix: honest failure-kind rendering plus the bounded scan.
+
+**Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report.
 
 ```bash
-# 1. Check config.yaml for hardcoded harness keys
-grep -rn 'api_key: sk-' /root/.hermes/ \
-  --include='config.yaml' \
-  | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'
+# 1. Check config.yaml for hardcoded harness keys (bounded scan — excludes state-snapshots)
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'" \
+  2>/dev/null
+
+# Interpret exit status:
+#   0 = match found (violation)
+#   1 = no match (pass)
+#   124 = timeout (probe-failed, not unreachable)
+#   255 = ssh connect failed (unreachable)
+#   other = probe-failed (record the actual code)
 
 # 1b. Check for double-path bug: base_url ending with /responses
 # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
-grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  "grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml" 2>/dev/null
 # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
 
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
-grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null
-grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-…' /root/.config/systemd/ 2>/dev/null
+# NOTE: Both greps are inside ONE quoted remote command, separated by ; (not two separate ssh arguments)
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null; grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-' /root/.config/systemd/ 2>/dev/null" ; true
 
 # 3. Verify running process env matches dedicated key
-cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ \
-  | tr '\0' '\n' | grep LITELLM_API_KEY
+# NOTE: Single-quoted remote command so $(...) expands on the REMOTE host, not the runner
+timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  'cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ | tr "\0" "\n" | grep LITELLM_API_KEY' \
+  && echo "step3: PASS" || echo "step3: probe-failed (exit $?; see stderr above)"
 ```
 
 If any output from step 2 — **critical violation** (master key leaked). Fix immediately.
 
+### Negative control (probe-failed vs unreachable) — deterministic
+
+To prove the distinction between a scan timeout and a connection failure, run a command that CANNOT finish in time (sleep 5s) with a 1-second timeout:
+
+```bash
+# Negative control: 1-second timeout on koby — sleep 5s guarantees timeout
+timeout 1 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@192.168.68.129 "sleep 5"; echo "exit=$?"
+# Expected: exit=124 (timeout) → render as "probe-failed: koby 192.168.68.129 (timeout after 1s)"
+# NOT: "unreachable" or "may be down"
+
+# Run TWICE to prove determinism:
+# Run 1: timeout 1 ssh ... "sleep 5"; echo "exit=$?" → exit=124
+# Run 2: timeout 1 ssh ... "sleep 5"; echo "exit=$?" → exit=124
+```
+
 ## Rotation Procedure
 
 With this standard enforced, key rotation is one vault update:

scripts/search-stack-check.py

@@ -61,11 +61,23 @@ EXTRACT_URL = os.environ.get(
 # engines (images, videos, translate, currency, arxiv, npm, ...) are excluded on
 # purpose -- contributing nothing to a general query is correct for them.
 DEFAULT_EXPECTED_ENGINES = [
+    # Multi-engine expansion 2026-10-03.  The stack had fallen to Bing-only:
+    # brave and google cse are suspended upstream, duckduckgo CAPTCHAs both
+    # egresses and yandex flaps.  The seven below all returned real results
+    # from this network and are the engines a general query must draw on.
     "bing",
-    "brave",
-    "google cse",
     "yandex",
+    "yep",
+    "mwmbl",
+    "naver",
+    "seznam",
+    "yahoo",
+    # Best-effort canaries: intentionally left enabled so a recovery shows up
+    # as a contribution and a failure stays visible in unresponsive_engines.
+    # All three are blocked upstream today.
+    "brave",
     "duckduckgo",
+    "google cse",
 ]
 EXPECTED_ENGINES = [
     e.strip()

scripts/secret-allowlist.tsv

@@ -35,7 +35,7 @@ bearer-token	agent-zero-fix-summary.md	«vault: agents/production OPENROUTER_API
 # example is always an explicit exception, never a pattern-level exemption.
 *	hermes-key-enforcement.prose.md	sk-synthetic-external-example	Rule 15 illustration of a hardcoded external key that is tolerated; fabricated, never a live key.
 *	hermes-key-enforcement.prose.md	sk-synthetic-example-12345	Rule 15 illustration of a forbidden hardcoded key; fabricated, never a live key.
-openai-key	hermes-key-enforcement.prose.md	sk-synthetic-litellm-	Fabricated key name inside a `grep 'LITELLM_API_KEY=...'` example; not a live key.
+*	hermes-key-enforcement.prose.md	sk-synthetic-litellm-	Fabricated key name inside a `grep 'LITELLM_API_KEY=...'` example; not a live key. (2026-10-03: changed rule from openai-key to * because secret-assign also matches the credential-shaped assignment)
 secret-assign	hermes-key-enforcement.prose.md	sk-NEW_KEY	Placeholder standing for the rotated key in an `infisical secrets set` command; not a literal key.
 openrouter-key	agent-zero-openrouter-key.prose.md	sk-or-v1-synthetic	Synthetic key prefix in the contract's example response; the real key is read from the vault.
 openai-key	litellm-api-keys.prose.md	sk-synthetic-tanko-example	Fabricated key name in migration history prose; not a live key.

search-stack-visibility.prose.md

@@ -17,11 +17,16 @@ description: >
     * reports every silent-zero engine explicitly (enabled, not in
       unresponsive_engines, contributed no results).
 
-  Multi-engine state (2026-09-25): bing, google cse, brave and yandex
-  contribute on every query. duckduckgo is NOT working: the house egress IP
-  and the VPS fallback egress are both flagged by DuckDuckGo and it reports
-  CAPTCHA. It is left enabled as best-effort coverage so that a recovery shows
-  up as a contribution.
+  Multi-engine state (2026-10-03): the stack had fallen to Bing-only -- brave
+  and google cse are suspended upstream, duckduckgo CAPTCHAs both egresses and
+  yandex flaps. Every no-credential free general engine this build ships was
+  enabled and probed. Seven now contribute real results on a general query:
+  bing, yandex, yep, mwmbl, naver, seznam and yahoo. brave, duckduckgo and
+  google cse are left enabled as best-effort canaries so a recovery shows up as
+  a contribution and their failure stays visible in unresponsive_engines.
+  mojeek, startpage and dogpile are `inactive: true` in the build (proof-of-work
+  CAPTCHA), marginalia needs an API key, and qwant and fireball were tested and
+  dropped (CAPTCHA and access-denied).
 
   google cse is a third party's public search-engine id hardcoded in the
   SearXNG build. Quota and availability are outside our control.
@@ -29,7 +34,7 @@ description: >
   SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15,
   via scripts/contract-run.sh search-stack-visibility. Logs land in
   /var/log/contract-runs/. A failure also raises a firstmate inbox note.
-version: 1.1.0
+version: 1.2.0
 ---
 
 ## Purpose
@@ -54,11 +59,12 @@ firstmate inbox note through `bin/fm-inbox.sh`.
 
 ```
 $ bash scripts/contract-run.sh search-stack-visibility
-Expected engines, enabled (5): ['bing', 'brave', 'duckduckgo', 'google cse', 'yandex']
-queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex
-                                    unresponsive: duckduckgo=CAPTCHA
-         'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex
-EXTRACTION: 71016 chars of markdown returned
+Expected engines, enabled (10): ['bing', 'brave', 'duckduckgo', 'google cse',
+                                'mwmbl', 'naver', 'seznam', 'yahoo', 'yandex', 'yep']
+queries: 'proxmox backup server'  -> contributing: bing, mwmbl, naver, seznam, yahoo, yandex, yep
+                                     unresponsive: brave, duckduckgo, google cse
+         'python asyncio tutorial' -> contributing: bing, mwmbl, naver, seznam, yandex, yep
+EXTRACTION: 71532 chars of markdown returned
 VERDICT: PASS -- multiple engines contributing, extraction healthy
 ```