diff --git a/scripts/search-stack-check.py b/scripts/search-stack-check.py index c06d474..73b7bdd 100755 --- a/scripts/search-stack-check.py +++ b/scripts/search-stack-check.py @@ -174,7 +174,10 @@ def main() -> int: ) if silent: silent_zero_all[query] = silent - print(f" SILENT ZERO (enabled, no error, no results): {silent}") + print( + " SILENT ZERO (enabled, no error, no results -- reported, " + f"not fatal): {silent}" + ) print("=" * 72) print("Engine contribution across all queries:") @@ -187,6 +190,11 @@ def main() -> int: print("SILENT-ZERO ENGINES REPORTED (no error raised, no results returned):") for query, names in silent_zero_all.items(): print(f" {query!r}: {names}") + print(" NOTE: a silent zero is REPORTED, not counted as a failure. These") + print(" engines are expected to answer a general query, but contributing") + print(" nothing to one query can be legitimate (result de-duplication, or") + print(" an engine that only fires on certain query shapes). Only the") + print(f" <{MIN_ENGINES}-contributing-engine floor and the extraction leg fail the run.") print("-" * 72) print(f"EXTRACTION: scraping {EXTRACT_URL} via {FIRECRAWL_URL}/v1/scrape") diff --git a/search-stack-visibility.prose.md b/search-stack-visibility.prose.md index 65c95e7..3a54b45 100644 --- a/search-stack-visibility.prose.md +++ b/search-stack-visibility.prose.md @@ -17,14 +17,19 @@ description: > * reports every silent-zero engine explicitly (enabled, not in unresponsive_engines, contributed no results). - Multi-engine post-fix state (2026-09-25): bing, google cse, brave, yandex - contribute on every query; duckduckgo is best-effort via a VPS egress - network and is expected to regress when DuckDuckGo flags the datacenter IP. + Multi-engine state (2026-09-25): bing, google cse, brave and yandex + contribute on every query. duckduckgo is NOT working: the house egress IP + and the VPS fallback egress are both flagged by DuckDuckGo and it reports + CAPTCHA. It is left enabled as best-effort coverage so that a recovery shows + up as a contribution. + + google cse is a third party's public search-engine id hardcoded in the + SearXNG build. Quota and availability are outside our control. SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15, via scripts/contract-run.sh search-stack-visibility. Logs land in /var/log/contract-runs/. A failure also raises a firstmate inbox note. -version: 1.0.0 +version: 1.1.0 --- ## Purpose @@ -49,11 +54,10 @@ firstmate inbox note through `bin/fm-inbox.sh`. ``` $ bash scripts/contract-run.sh search-stack-visibility -Expected engines, enabled (4): ['bing', 'brave', 'google cse', 'yandex'] +Expected engines, enabled (5): ['bing', 'brave', 'duckduckgo', 'google cse', 'yandex'] queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex - unresponsive: (none) + unresponsive: duckduckgo=CAPTCHA 'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex - unresponsive: (none) EXTRACTION: 71016 chars of markdown returned VERDICT: PASS -- multiple engines contributing, extraction healthy ``` @@ -63,8 +67,34 @@ VERDICT: PASS -- multiple engines contributing, extraction healthy * A query whose results come from fewer than `SEARCH_CHECK_MIN_ENGINES` engines (default 2) fails and names the engines that did contribute. * An extraction request that errors or returns empty markdown fails. -* An enabled, expected engine that contributed nothing without reporting an - error is printed under `SILENT-ZERO ENGINES REPORTED`. + +## Silent zeros are reported, not fatal + +An enabled, expected engine that contributed nothing **without reporting an +error** is printed under `SILENT-ZERO ENGINES REPORTED`, and each occurrence is +annotated `reported, not fatal`. This is deliberate: + +* a general query can legitimately draw zero results from an engine that only + fires on certain query shapes, and results are de-duplicated across engines, + so a zero does not by itself prove the engine is broken; +* the run therefore fails only on the two conditions that do prove loss of + capability -- fewer than two contributing engines, and a broken extraction + leg. + +A run can consequently print `VERDICT: PASS` while still listing a silent +zero. That is the intended relationship: the zero is *visible*, not *fatal*. +An engine that fails with an error (for example DuckDuckGo returning CAPTCHA) +appears in `unresponsive_engines` instead. + +## Google coverage is third-party, not ours + +The free Google-derived results come from the SearXNG build's built-in +`google cse` engine. It uses **a third party's public search-engine id +hardcoded in the build** (`google_cse.py`, `CX = "partner-pub-8993..."`, +blackle.com), not a key or id we own. Its quota and availability are outside +our control and it can be rate-limited or withdrawn without notice. No engine +in this build accepts our own Google Custom Search key; using our own free key +would require a small wrapper service, which is deliberately **not** built. ## Configuration @@ -81,9 +111,14 @@ Environment overrides (see the script docstring for the full list): ## Known residual risk -DuckDuckGo is reached through a forward proxy on the VPS -(`10.10.10.1:3128`, WireGuard) because the house egress IP is CAPTCHA'd. The -VPS is a datacenter address and DuckDuckGo may flag it, so DuckDuckGo is -coverage, never a required engine. When it regresses it appears either in -`unresponsive_engines` (an explicit error) or under the silent-zero report; -neither fails the check, but both are visible. +DuckDuckGo is **not** working. The house egress IP is CAPTCHA'd by +DuckDuckGo, and a forward proxy on the VPS (`10.10.10.1:3128`, WireGuard) was +built as a second egress -- but DuckDuckGo has since flagged the VPS address +too (HTTP 202 with challenge markers), so DuckDuckGo now reports CAPTCHA on +both paths. It is left enabled as best-effort coverage: if DuckDuckGo +unflags either address it will show up as a contribution, and until then it is +visible in `unresponsive_engines` every run. It is never a required engine. + +The VPS forward proxy remains a real service (`/opt/fwd-proxy`, +`restart: unless-stopped`, healthy healthcheck, Docker enabled at boot) so the +second egress path is available for any engine that benefits from it in future.