root
8b2eba4f7a
Follow-up corrections after review: 1. DuckDuckGo is NOT fixed. The VPS fallback egress has since been flagged by DuckDuckGo too (HTTP 202 + challenge markers), so it reports CAPTCHA on both paths. The contract and script docstring now say so instead of claiming a fix that had already expired. It stays enabled as best-effort coverage so a recovery shows up as a contribution. 2. The relationship between silent zeros and the verdict is now explicit in both the script output and the contract: an enabled expected engine that contributes zero with no error is REPORTED, not fatal. Only the <SEARCH_CHECK_MIN_ENGINES> floor and the extraction leg fail the run. This is deliberate - de-duplication and query-shape make a zero non-probative. 3. Recorded that 'google cse' uses a THIRD PARTY's public search-engine id hardcoded in the SearXNG build, not a key we own; its quota and availability are outside our control, and our own free key would need a wrapper (not built). VPS forward proxy is now a real service: /opt/fwd-proxy docker compose with restart: unless-stopped, a healthy healthcheck, and Docker enabled at boot.
6.0 KiB
kind, name, description, version
| kind | name | description | version |
|---|---|---|---|
| function | search-stack-visibility | Makes the shared search stack observable. Every agent reaches one SearXNG instance (http://192.168.68.7:8888) and one extraction service (Firecrawl, http://192.168.68.7:3002). Before this check the stack could degrade to a single engine, or an enabled engine could return nothing at all, without any error surfacing anywhere. This contract runs scripts/search-stack-check.py, which: * runs two fixed queries against SearXNG and FAILS when fewer than two engines contribute, printing the contributing engines and every unresponsive_engines entry; * checks extraction by scraping a known page through Firecrawl and FAILS when the returned markdown is empty or the request fails; * reports every silent-zero engine explicitly (enabled, not in unresponsive_engines, contributed no results). Multi-engine state (2026-09-25): bing, google cse, brave and yandex contribute on every query. duckduckgo is NOT working: the house egress IP and the VPS fallback egress are both flagged by DuckDuckGo and it reports CAPTCHA. It is left enabled as best-effort coverage so that a recovery shows up as a contribution. google cse is a third party's public search-engine id hardcoded in the SearXNG build. Quota and availability are outside our control. SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15, via scripts/contract-run.sh search-stack-visibility. Logs land in /var/log/contract-runs/. A failure also raises a firstmate inbox note. | 1.1.0 |
Purpose
The fleet has exactly one search endpoint and one extraction endpoint. If either degrades, every agent silently loses capability at the same moment. The failure mode this contract exists to close is silent degradation: a query that still returns a page of results while all but one engine have stopped contributing, or an enabled engine that answers with zero results and raises no error.
Execution model
The contract is a host-scheduled check, not an agent workflow. It is driven by
scripts/contract-run.sh search-stack-visibility from
/etc/cron.d/contract-runner on CT 100. contract-run.sh resolves the
mapping to scripts/search-stack-check.py, runs it under a timeout, writes a
timestamped log to /var/log/contract-runs/, and on non-zero exit raises a
firstmate inbox note through bin/fm-inbox.sh.
What passing looks like
$ bash scripts/contract-run.sh search-stack-visibility
Expected engines, enabled (5): ['bing', 'brave', 'duckduckgo', 'google cse', 'yandex']
queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex
unresponsive: duckduckgo=CAPTCHA
'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex
EXTRACTION: 71016 chars of markdown returned
VERDICT: PASS -- multiple engines contributing, extraction healthy
What failing looks like
- A query whose results come from fewer than
SEARCH_CHECK_MIN_ENGINESengines (default 2) fails and names the engines that did contribute. - An extraction request that errors or returns empty markdown fails.
Silent zeros are reported, not fatal
An enabled, expected engine that contributed nothing without reporting an
error is printed under SILENT-ZERO ENGINES REPORTED, and each occurrence is
annotated reported, not fatal. This is deliberate:
- a general query can legitimately draw zero results from an engine that only fires on certain query shapes, and results are de-duplicated across engines, so a zero does not by itself prove the engine is broken;
- the run therefore fails only on the two conditions that do prove loss of capability -- fewer than two contributing engines, and a broken extraction leg.
A run can consequently print VERDICT: PASS while still listing a silent
zero. That is the intended relationship: the zero is visible, not fatal.
An engine that fails with an error (for example DuckDuckGo returning CAPTCHA)
appears in unresponsive_engines instead.
Google coverage is third-party, not ours
The free Google-derived results come from the SearXNG build's built-in
google cse engine. It uses a third party's public search-engine id
hardcoded in the build (google_cse.py, CX = "partner-pub-8993...",
blackle.com), not a key or id we own. Its quota and availability are outside
our control and it can be rate-limited or withdrawn without notice. No engine
in this build accepts our own Google Custom Search key; using our own free key
would require a small wrapper service, which is deliberately not built.
Configuration
Environment overrides (see the script docstring for the full list):
| Variable | Default | Meaning |
|---|---|---|
SEARXNG_URL |
http://192.168.68.7:8888 |
SearXNG base URL |
FIRECRAWL_URL |
http://192.168.68.7:3002 |
Firecrawl base URL |
SEARCH_CHECK_QUERIES |
proxmox backup server,python asyncio tutorial |
fixed queries |
SEARCH_CHECK_MIN_ENGINES |
2 |
minimum contributing engines per query |
SEARCH_CHECK_ENGINES |
bing,brave,google cse,yandex,duckduckgo |
engines a silent zero is reported for |
SEARCH_CHECK_EXTRACT_URL |
Wikipedia Proxmox article | page used for the extraction leg |
Known residual risk
DuckDuckGo is not working. The house egress IP is CAPTCHA'd by
DuckDuckGo, and a forward proxy on the VPS (10.10.10.1:3128, WireGuard) was
built as a second egress -- but DuckDuckGo has since flagged the VPS address
too (HTTP 202 with challenge markers), so DuckDuckGo now reports CAPTCHA on
both paths. It is left enabled as best-effort coverage: if DuckDuckGo
unflags either address it will show up as a contribution, and until then it is
visible in unresponsive_engines every run. It is never a required engine.
The VPS forward proxy remains a real service (/opt/fwd-proxy,
restart: unless-stopped, healthy healthcheck, Docker enabled at boot) so the
second egress path is available for any engine that benefits from it in future.