root
6608d3162f
The visibility contract's expected-engine set was still the 2026-09-25 list (bing, brave, google cse, yandex, duckduckgo). Three of those five are blocked upstream today, so the check had gone quiet on the engines that DO carry the stack and noisy on ones that cannot. The live stack now runs ten engines enabled: seven that returned real results from this network (bing, yandex, yep, mwmbl, naver, seznam, yahoo) plus the three best-effort canaries kept for recovery visibility (brave, duckduckgo, google cse). The expected set is updated to match, so a silent zero is reported for every engine the stack actually runs. Live proof after the engine expansion (CT 100, 2026-10-03): 'proxmox backup server' -> 7 contributing engines 'python asyncio tutorial' -> 6 contributing engines VERDICT: PASS Before the change both queries contributed from bing alone, one engine above the two-engine floor. Verified broken cases still fail and name the cause: a single-engine floor exits 1 naming the sole contributor, a broken extraction exits 1, and an unreachable SearXNG exits 2. Contract text and version updated to the 2026-10-03 state.
6.5 KiB
kind, name, description, version
| kind | name | description | version |
|---|---|---|---|
| function | search-stack-visibility | Makes the shared search stack observable. Every agent reaches one SearXNG instance (http://192.168.68.7:8888) and one extraction service (Firecrawl, http://192.168.68.7:3002). Before this check the stack could degrade to a single engine, or an enabled engine could return nothing at all, without any error surfacing anywhere. This contract runs scripts/search-stack-check.py, which: * runs two fixed queries against SearXNG and FAILS when fewer than two engines contribute, printing the contributing engines and every unresponsive_engines entry; * checks extraction by scraping a known page through Firecrawl and FAILS when the returned markdown is empty or the request fails; * reports every silent-zero engine explicitly (enabled, not in unresponsive_engines, contributed no results). Multi-engine state (2026-10-03): the stack had fallen to Bing-only -- brave and google cse are suspended upstream, duckduckgo CAPTCHAs both egresses and yandex flaps. Every no-credential free general engine this build ships was enabled and probed. Seven now contribute real results on a general query: bing, yandex, yep, mwmbl, naver, seznam and yahoo. brave, duckduckgo and google cse are left enabled as best-effort canaries so a recovery shows up as a contribution and their failure stays visible in unresponsive_engines. mojeek, startpage and dogpile are `inactive: true` in the build (proof-of-work CAPTCHA), marginalia needs an API key, and qwant and fireball were tested and dropped (CAPTCHA and access-denied). google cse is a third party's public search-engine id hardcoded in the SearXNG build. Quota and availability are outside our control. SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15, via scripts/contract-run.sh search-stack-visibility. Logs land in /var/log/contract-runs/. A failure also raises a firstmate inbox note. | 1.2.0 |
Purpose
The fleet has exactly one search endpoint and one extraction endpoint. If either degrades, every agent silently loses capability at the same moment. The failure mode this contract exists to close is silent degradation: a query that still returns a page of results while all but one engine have stopped contributing, or an enabled engine that answers with zero results and raises no error.
Execution model
The contract is a host-scheduled check, not an agent workflow. It is driven by
scripts/contract-run.sh search-stack-visibility from
/etc/cron.d/contract-runner on CT 100. contract-run.sh resolves the
mapping to scripts/search-stack-check.py, runs it under a timeout, writes a
timestamped log to /var/log/contract-runs/, and on non-zero exit raises a
firstmate inbox note through bin/fm-inbox.sh.
What passing looks like
$ bash scripts/contract-run.sh search-stack-visibility
Expected engines, enabled (10): ['bing', 'brave', 'duckduckgo', 'google cse',
'mwmbl', 'naver', 'seznam', 'yahoo', 'yandex', 'yep']
queries: 'proxmox backup server' -> contributing: bing, mwmbl, naver, seznam, yahoo, yandex, yep
unresponsive: brave, duckduckgo, google cse
'python asyncio tutorial' -> contributing: bing, mwmbl, naver, seznam, yandex, yep
EXTRACTION: 71532 chars of markdown returned
VERDICT: PASS -- multiple engines contributing, extraction healthy
What failing looks like
- A query whose results come from fewer than
SEARCH_CHECK_MIN_ENGINESengines (default 2) fails and names the engines that did contribute. - An extraction request that errors or returns empty markdown fails.
Silent zeros are reported, not fatal
An enabled, expected engine that contributed nothing without reporting an
error is printed under SILENT-ZERO ENGINES REPORTED, and each occurrence is
annotated reported, not fatal. This is deliberate:
- a general query can legitimately draw zero results from an engine that only fires on certain query shapes, and results are de-duplicated across engines, so a zero does not by itself prove the engine is broken;
- the run therefore fails only on the two conditions that do prove loss of capability -- fewer than two contributing engines, and a broken extraction leg.
A run can consequently print VERDICT: PASS while still listing a silent
zero. That is the intended relationship: the zero is visible, not fatal.
An engine that fails with an error (for example DuckDuckGo returning CAPTCHA)
appears in unresponsive_engines instead.
Google coverage is third-party, not ours
The free Google-derived results come from the SearXNG build's built-in
google cse engine. It uses a third party's public search-engine id
hardcoded in the build (google_cse.py, CX = "partner-pub-8993...",
blackle.com), not a key or id we own. Its quota and availability are outside
our control and it can be rate-limited or withdrawn without notice. No engine
in this build accepts our own Google Custom Search key; using our own free key
would require a small wrapper service, which is deliberately not built.
Configuration
Environment overrides (see the script docstring for the full list):
| Variable | Default | Meaning |
|---|---|---|
SEARXNG_URL |
http://192.168.68.7:8888 |
SearXNG base URL |
FIRECRAWL_URL |
http://192.168.68.7:3002 |
Firecrawl base URL |
SEARCH_CHECK_QUERIES |
proxmox backup server,python asyncio tutorial |
fixed queries |
SEARCH_CHECK_MIN_ENGINES |
2 |
minimum contributing engines per query |
SEARCH_CHECK_ENGINES |
bing,brave,google cse,yandex,duckduckgo |
engines a silent zero is reported for |
SEARCH_CHECK_EXTRACT_URL |
Wikipedia Proxmox article | page used for the extraction leg |
Known residual risk
DuckDuckGo is not working. The house egress IP is CAPTCHA'd by
DuckDuckGo, and a forward proxy on the VPS (10.10.10.1:3128, WireGuard) was
built as a second egress -- but DuckDuckGo has since flagged the VPS address
too (HTTP 202 with challenge markers), so DuckDuckGo now reports CAPTCHA on
both paths. It is left enabled as best-effort coverage: if DuckDuckGo
unflags either address it will show up as a contribution, and until then it is
visible in unresponsive_engines every run. It is never a required engine.
The VPS forward proxy remains a real service (/opt/fwd-proxy,
restart: unless-stopped, healthy healthcheck, Docker enabled at boot) so the
second egress path is available for any engine that benefits from it in future.