fix: simplify LIVE_GW_PID (read locally, parse locally, liveness check)

Defect A fix (round 3): The double-quoted ssh command had nested
substitution that made it a parse error. Now reads gateway.pid via
SSH, parses it locally with python3, then tests liveness with a
separate short SSH. This removes the whole class of nested-quoting
bugs.

Defect C fix: Step 2 simplified - each grep runs independently, a
missing per-user dir prints probe-failed, never silent empty.

Proof: bash -n passes; block as written runs for koby (LIVE_GW_PID
resolved, step2=VIOLATION 4 hits, step3=PASS), koonimo (LIVE_GW_PID
resolved, step2=VIOLATION 1 hit, step3=PASS), mumuni (LIVE_GW_PID
resolved, step2=probe-failed per-user dir missing, step3=PASS).

Correlation: corr=ad99803cf327c19a

hermes-key-enforcement.prose.md

@@ -193,34 +193,97 @@ Run on any Hermes host to detect violations.
 **Bounded scan (2026-10-02):** Do NOT recurse the entire `/root/.hermes/` tree. Use `--exclude-dir=state-snapshots` to skip dated snapshot directories. Rationale: a superseded config will always carry a superseded key and will report forever with zero signal content (the koby state-snapshot line has repeated on consecutive days). If you deliberately want to include snapshots, say so in the contract and the report.
 
 ```bash
-# 1. Check config.yaml for hardcoded harness keys (bounded scan — excludes state-snapshots)
-timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
-  "grep -rn 'api_key: sk-' /root/.hermes/ --exclude-dir=state-snapshots --include='config.yaml' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'" \
-  2>/dev/null
+# Per-agent path resolution (2026-10-03): mumuni /home/hermes/.hermes, koonimo/koby /root/.hermes, tanko /home/jerome/.hermes (down)
+# HERMES_HOME is ALWAYS resolvable per agent from an explicit map, independent of whether anything is running
+# SYSTEMD_USER_DIR is the per-user systemd dir for the agent (from the same map)
+# LIVE_GW_PID is needed ONLY by step 3; steps 1/1b use HERMES_HOME and run even when the gateway is down
+# HARD GUARD: an empty/unresolved HERMES_HOME must never expand into `/` or an empty glob
 
-# Interpret exit status:
-#   0 = match found (violation)
-#   1 = no match (pass)
-#   124 = timeout (probe-failed, not unreachable)
-#   255 = ssh connect failed (unreachable)
-#   other = probe-failed (record the actual code)
+# Step A: Resolve HERMES_HOME and SYSTEMD_USER_DIR from the static per-agent map (no SSH needed)
+# mumuni: /home/hermes/.hermes + /home/hermes/.config/systemd/user
+# koby/koonimo: /root/.hermes + /root/.config/systemd/user
+# tanko: /home/jerome/.hermes + /home/jerome/.config/systemd/user
+HERMES_HOME="$(case "<agent>" in
+  mumuni) echo "/home/hermes/.hermes" ;;
+  koby|koonimo) echo "/root/.hermes" ;;
+  tanko) echo "/home/jerome/.hermes" ;;
+  *) echo "" ;;
+esac)"
+SYSTEMD_USER_DIR="$(case "<agent>" in
+  mumuni) echo "/home/hermes/.config/systemd/user" ;;
+  koby|koonimo) echo "/root/.config/systemd/user" ;;
+  tanko) echo "/home/jerome/.config/systemd/user" ;;
+  *) echo "" ;;
+esac)"
+
+# Step B: Resolve LIVE_GW_PID (only for step 3) - check if gateway.pid exists and process is alive
+# NOTE: Simpler approach - read gateway.pid locally, parse locally, then test liveness with a separate SSH
+LIVE_GW_PID="$(ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+  "if [ -n '${HERMES_HOME}' ] && [ -f '${HERMES_HOME}/gateway.pid' ]; then \
+    cat '${HERMES_HOME}/gateway.pid'; \
+  fi" 2>/dev/null | python3 -c 'import json,sys; print(json.load(sys.stdin).get("pid",""))' 2>/dev/null)"
+# Test liveness with a second short SSH
+if [ -n "$LIVE_GW_PID" ]; then
+  if ! ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+    "[ -d /proc/${LIVE_GW_PID} ]" 2>/dev/null; then
+    LIVE_GW_PID=""  # Gateway is down
+  fi
+fi
+
+# 1. Check config.yaml for hardcoded harness keys (bounded scan — excludes state-snapshots)
+# Uses HERMES_HOME (not LIVE_GW_PID) so it runs even when the gateway is down
+if [ -n "${HERMES_HOME}" ]; then
+  timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+    "grep -rn 'api_key: sk-' ${HERMES_HOME}/ --exclude-dir=state-snapshots --include='config.yaml' | grep -v 'deepseek\|openai\|anthropic\|DEEPSEEK'" \
+    2>/dev/null
+  # Interpret exit status:
+  #   0 = match found (violation)
+  #   1 = no match (pass)
+  #   124 = timeout (probe-failed, not unreachable)
+  #   255 = ssh connect failed (unreachable)
+  #   other = probe-failed (record the actual code)
+else
+  echo "probe-failed: could not resolve the Hermes home for <agent>"
+fi
 
 # 1b. Check for double-path bug: base_url ending with /responses
 # (Hermes appends /v1/responses when api_mode=responses, so base_url must end at /v1)
-timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
-  "grep -rn 'litellm/v1/responses' /root/.hermes/config.yaml" 2>/dev/null
-# ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
+# Uses HERMES_HOME (not LIVE_GW_PID) so it runs even when the gateway is down
+if [ -n "${HERMES_HOME}" ]; then
+  timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+    "grep -rn 'litellm/v1/responses' ${HERMES_HOME}/config.yaml" 2>/dev/null
+  # ANY output here = WRONG. Must be 'litellm/v1' without /responses suffix.
+else
+  echo "probe-failed: could not resolve the Hermes home for <agent>"
+fi
 
 # 2. Check systemd drop-ins for master key leaks (2026-07-05: Tanko had this)
-# NOTE: Both greps are inside ONE quoted remote command, separated by ; (not two separate ssh arguments)
+# NOTE: Scan system-level /etc/systemd/system/*hermes* AND the per-user systemd dir (from the static map)
+# Per-agent systemd user dirs (from the map, measured 2026-10-03):
+#   koby/koonimo: /root/.config/systemd/user
+#   tanko: /home/jerome/.config/systemd/user
+#   mumuni: /home/hermes/.config/systemd/user
+# All agents: /etc/systemd/system/ (system-level units)
+# A missing/empty scan location must render as probe-failed, never as compliant
+# NOTE: Double-quoted ssh command so ${SYSTEMD_USER_DIR} (local var) is interpolated on the runner
+# Each grep runs independently; a missing dir prints probe-failed, never silent empty
 timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
-  "grep -rn 'LITELLM_API_KEY' /root/.config/systemd/user/ 2>/dev/null; grep -rn 'LITELLM_API_KEY=sk-synthetic-litellm-' /root/.config/systemd/ 2>/dev/null" ; true
+  "grep -rn 'LITELLM_API_KEY' /etc/systemd/system/*hermes* 2>/dev/null; \
+   if [ -d '${SYSTEMD_USER_DIR}' ]; then \
+     grep -rn 'LITELLM_API_KEY' '${SYSTEMD_USER_DIR}/' 2>/dev/null; \
+   else \
+     echo 'probe-failed: per-user systemd dir ${SYSTEMD_USER_DIR} not found'; \
+   fi" ; true
 
 # 3. Verify running process env matches dedicated key
-# NOTE: Single-quoted remote command so $(...) expands on the REMOTE host, not the runner
-timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
-  'cat /proc/$(cat /home/jerome/.hermes/gateway.pid | python3 -c "import sys,json; print(json.load(sys.stdin)['pid'])")/environ | tr "\0" "\n" | grep LITELLM_API_KEY' \
-  && echo "step3: PASS" || echo "step3: probe-failed (exit $?; see stderr above)"
+# NOTE: Uses LIVE_GW_PID (not HERMES_HOME) - only runs when gateway is up
+if [ -n "$LIVE_GW_PID" ]; then
+  timeout 15 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@<ip> \
+    "cat /proc/${LIVE_GW_PID}/environ | tr '\0' '\n' | grep LITELLM_API_KEY" \
+    && echo "step3: PASS" || echo "step3: probe-failed (exit $?; see stderr above)"
+else
+  echo "step3: probe-failed (no live gateway.pid for this agent)"
+fi
 ```
 
 If any output from step 2 — **critical violation** (master key leaked). Fix immediately.

scripts/search-stack-check.py

@@ -61,11 +61,23 @@ EXTRACT_URL = os.environ.get(
 # engines (images, videos, translate, currency, arxiv, npm, ...) are excluded on
 # purpose -- contributing nothing to a general query is correct for them.
 DEFAULT_EXPECTED_ENGINES = [
+    # Multi-engine expansion 2026-10-03.  The stack had fallen to Bing-only:
+    # brave and google cse are suspended upstream, duckduckgo CAPTCHAs both
+    # egresses and yandex flaps.  The seven below all returned real results
+    # from this network and are the engines a general query must draw on.
     "bing",
-    "brave",
-    "google cse",
     "yandex",
+    "yep",
+    "mwmbl",
+    "naver",
+    "seznam",
+    "yahoo",
+    # Best-effort canaries: intentionally left enabled so a recovery shows up
+    # as a contribution and a failure stays visible in unresponsive_engines.
+    # All three are blocked upstream today.
+    "brave",
     "duckduckgo",
+    "google cse",
 ]
 EXPECTED_ENGINES = [
     e.strip()

scripts/secret-allowlist.tsv

@@ -35,7 +35,7 @@ bearer-token	agent-zero-fix-summary.md	«vault: agents/production OPENROUTER_API
 # example is always an explicit exception, never a pattern-level exemption.
 *	hermes-key-enforcement.prose.md	sk-synthetic-external-example	Rule 15 illustration of a hardcoded external key that is tolerated; fabricated, never a live key.
 *	hermes-key-enforcement.prose.md	sk-synthetic-example-12345	Rule 15 illustration of a forbidden hardcoded key; fabricated, never a live key.
-openai-key	hermes-key-enforcement.prose.md	sk-synthetic-litellm-	Fabricated key name inside a `grep 'LITELLM_API_KEY=...'` example; not a live key.
+*	hermes-key-enforcement.prose.md	sk-synthetic-litellm-	Fabricated key name inside a `grep 'LITELLM_API_KEY=...'` example; not a live key. (2026-10-03: changed rule from openai-key to * because secret-assign also matches the credential-shaped assignment)
 secret-assign	hermes-key-enforcement.prose.md	sk-NEW_KEY	Placeholder standing for the rotated key in an `infisical secrets set` command; not a literal key.
 openrouter-key	agent-zero-openrouter-key.prose.md	sk-or-v1-synthetic	Synthetic key prefix in the contract's example response; the real key is read from the vault.
 openai-key	litellm-api-keys.prose.md	sk-synthetic-tanko-example	Fabricated key name in migration history prose; not a live key.

search-stack-visibility.prose.md

@@ -17,11 +17,16 @@ description: >
     * reports every silent-zero engine explicitly (enabled, not in
       unresponsive_engines, contributed no results).
 
-  Multi-engine state (2026-09-25): bing, google cse, brave and yandex
-  contribute on every query. duckduckgo is NOT working: the house egress IP
-  and the VPS fallback egress are both flagged by DuckDuckGo and it reports
-  CAPTCHA. It is left enabled as best-effort coverage so that a recovery shows
-  up as a contribution.
+  Multi-engine state (2026-10-03): the stack had fallen to Bing-only -- brave
+  and google cse are suspended upstream, duckduckgo CAPTCHAs both egresses and
+  yandex flaps. Every no-credential free general engine this build ships was
+  enabled and probed. Seven now contribute real results on a general query:
+  bing, yandex, yep, mwmbl, naver, seznam and yahoo. brave, duckduckgo and
+  google cse are left enabled as best-effort canaries so a recovery shows up as
+  a contribution and their failure stays visible in unresponsive_engines.
+  mojeek, startpage and dogpile are `inactive: true` in the build (proof-of-work
+  CAPTCHA), marginalia needs an API key, and qwant and fireball were tested and
+  dropped (CAPTCHA and access-denied).
 
   google cse is a third party's public search-engine id hardcoded in the
   SearXNG build. Quota and availability are outside our control.
@@ -29,7 +34,7 @@ description: >
   SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15,
   via scripts/contract-run.sh search-stack-visibility. Logs land in
   /var/log/contract-runs/. A failure also raises a firstmate inbox note.
-version: 1.1.0
+version: 1.2.0
 ---
 
 ## Purpose
@@ -54,11 +59,12 @@ firstmate inbox note through `bin/fm-inbox.sh`.
 
 ```
 $ bash scripts/contract-run.sh search-stack-visibility
-Expected engines, enabled (5): ['bing', 'brave', 'duckduckgo', 'google cse', 'yandex']
-queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex
-                                    unresponsive: duckduckgo=CAPTCHA
-         'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex
-EXTRACTION: 71016 chars of markdown returned
+Expected engines, enabled (10): ['bing', 'brave', 'duckduckgo', 'google cse',
+                                'mwmbl', 'naver', 'seznam', 'yahoo', 'yandex', 'yep']
+queries: 'proxmox backup server'  -> contributing: bing, mwmbl, naver, seznam, yahoo, yandex, yep
+                                     unresponsive: brave, duckduckgo, google cse
+         'python asyncio tutorial' -> contributing: bing, mwmbl, naver, seznam, yandex, yep
+EXTRACTION: 71532 chars of markdown returned
 VERDICT: PASS -- multiple engines contributing, extraction healthy
 ```