---
kind: function
name: search-stack-visibility
description: >
  Makes the shared search stack observable. Every agent reaches one SearXNG
  instance (http://192.168.68.7:8888) and one extraction service (Firecrawl,
  http://192.168.68.7:3002). Before this check the stack could degrade to a
  single engine, or an enabled engine could return nothing at all, without any
  error surfacing anywhere.

  This contract runs scripts/search-stack-check.py, which:
    * runs two fixed queries against SearXNG and FAILS when fewer than two
      engines contribute, printing the contributing engines and every
      unresponsive_engines entry;
    * checks extraction by scraping a known page through Firecrawl and FAILS
      when the returned markdown is empty or the request fails;
    * reports every silent-zero engine explicitly (enabled, not in
      unresponsive_engines, contributed no results).

  Multi-engine state (2026-09-25): bing, google cse, brave and yandex
  contribute on every query. duckduckgo is NOT working: the house egress IP
  and the VPS fallback egress are both flagged by DuckDuckGo and it reports
  CAPTCHA. It is left enabled as best-effort coverage so that a recovery shows
  up as a contribution.

  google cse is a third party's public search-engine id hardcoded in the
  SearXNG build. Quota and availability are outside our control.

  SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15,
  via scripts/contract-run.sh search-stack-visibility. Logs land in
  /var/log/contract-runs/. A failure also raises a firstmate inbox note.
version: 1.1.0
---

## Purpose

The fleet has exactly one search endpoint and one extraction endpoint. If
either degrades, every agent silently loses capability at the same moment.
The failure mode this contract exists to close is *silent* degradation: a
query that still returns a page of results while all but one engine have
stopped contributing, or an enabled engine that answers with zero results and
raises no error.

## Execution model

The contract is a host-scheduled check, not an agent workflow. It is driven by
`scripts/contract-run.sh search-stack-visibility` from
`/etc/cron.d/contract-runner` on CT 100. `contract-run.sh` resolves the
mapping to `scripts/search-stack-check.py`, runs it under a timeout, writes a
timestamped log to `/var/log/contract-runs/`, and on non-zero exit raises a
firstmate inbox note through `bin/fm-inbox.sh`.

## What passing looks like

```
$ bash scripts/contract-run.sh search-stack-visibility
Expected engines, enabled (5): ['bing', 'brave', 'duckduckgo', 'google cse', 'yandex']
queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex
                                    unresponsive: duckduckgo=CAPTCHA
         'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex
EXTRACTION: 71016 chars of markdown returned
VERDICT: PASS -- multiple engines contributing, extraction healthy
```

## What failing looks like

* A query whose results come from fewer than `SEARCH_CHECK_MIN_ENGINES`
  engines (default 2) fails and names the engines that did contribute.
* An extraction request that errors or returns empty markdown fails.

## Silent zeros are reported, not fatal

An enabled, expected engine that contributed nothing **without reporting an
error** is printed under `SILENT-ZERO ENGINES REPORTED`, and each occurrence is
annotated `reported, not fatal`.  This is deliberate:

* a general query can legitimately draw zero results from an engine that only
  fires on certain query shapes, and results are de-duplicated across engines,
  so a zero does not by itself prove the engine is broken;
* the run therefore fails only on the two conditions that do prove loss of
  capability -- fewer than two contributing engines, and a broken extraction
  leg.

A run can consequently print `VERDICT: PASS` while still listing a silent
zero.  That is the intended relationship: the zero is *visible*, not *fatal*.
An engine that fails with an error (for example DuckDuckGo returning CAPTCHA)
appears in `unresponsive_engines` instead.

## Google coverage is third-party, not ours

The free Google-derived results come from the SearXNG build's built-in
`google cse` engine.  It uses **a third party's public search-engine id
hardcoded in the build** (`google_cse.py`, `CX = "partner-pub-8993..."`,
blackle.com), not a key or id we own.  Its quota and availability are outside
our control and it can be rate-limited or withdrawn without notice.  No engine
in this build accepts our own Google Custom Search key; using our own free key
would require a small wrapper service, which is deliberately **not** built.

## Configuration

Environment overrides (see the script docstring for the full list):

| Variable | Default | Meaning |
| --- | --- | --- |
| `SEARXNG_URL` | `http://192.168.68.7:8888` | SearXNG base URL |
| `FIRECRAWL_URL` | `http://192.168.68.7:3002` | Firecrawl base URL |
| `SEARCH_CHECK_QUERIES` | `proxmox backup server,python asyncio tutorial` | fixed queries |
| `SEARCH_CHECK_MIN_ENGINES` | `2` | minimum contributing engines per query |
| `SEARCH_CHECK_ENGINES` | `bing,brave,google cse,yandex,duckduckgo` | engines a silent zero is reported for |
| `SEARCH_CHECK_EXTRACT_URL` | Wikipedia Proxmox article | page used for the extraction leg |

## Known residual risk

DuckDuckGo is **not** working.  The house egress IP is CAPTCHA'd by
DuckDuckGo, and a forward proxy on the VPS (`10.10.10.1:3128`, WireGuard) was
built as a second egress -- but DuckDuckGo has since flagged the VPS address
too (HTTP 202 with challenge markers), so DuckDuckGo now reports CAPTCHA on
both paths.  It is left enabled as best-effort coverage: if DuckDuckGo
unflags either address it will show up as a contribution, and until then it is
visible in `unresponsive_engines` every run.  It is never a required engine.

The VPS forward proxy remains a real service (`/opt/fwd-proxy`,
`restart: unless-stopped`, healthy healthcheck, Docker enabled at boot) so the
second egress path is available for any engine that benefits from it in future.