---
kind: function
name: search-stack-visibility
description: >
  Makes the shared search stack observable. Every agent reaches one SearXNG
  instance (http://192.168.68.7:8888) and one extraction service (Firecrawl,
  http://192.168.68.7:3002). Before this check the stack could degrade to a
  single engine, or an enabled engine could return nothing at all, without any
  error surfacing anywhere.

  This contract runs scripts/search-stack-check.py, which:
    * runs two fixed queries against SearXNG and FAILS when fewer than two
      engines contribute, printing the contributing engines and every
      unresponsive_engines entry;
    * checks extraction by scraping a known page through Firecrawl and FAILS
      when the returned markdown is empty or the request fails;
    * reports every silent-zero engine explicitly (enabled, not in
      unresponsive_engines, contributed no results).

  Multi-engine post-fix state (2026-09-25): bing, google cse, brave, yandex
  contribute on every query; duckduckgo is best-effort via a VPS egress
  network and is expected to regress when DuckDuckGo flags the datacenter IP.

  SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15,
  via scripts/contract-run.sh search-stack-visibility. Logs land in
  /var/log/contract-runs/. A failure also raises a firstmate inbox note.
version: 1.0.0
---

## Purpose

The fleet has exactly one search endpoint and one extraction endpoint. If
either degrades, every agent silently loses capability at the same moment.
The failure mode this contract exists to close is *silent* degradation: a
query that still returns a page of results while all but one engine have
stopped contributing, or an enabled engine that answers with zero results and
raises no error.

## Execution model

The contract is a host-scheduled check, not an agent workflow. It is driven by
`scripts/contract-run.sh search-stack-visibility` from
`/etc/cron.d/contract-runner` on CT 100. `contract-run.sh` resolves the
mapping to `scripts/search-stack-check.py`, runs it under a timeout, writes a
timestamped log to `/var/log/contract-runs/`, and on non-zero exit raises a
firstmate inbox note through `bin/fm-inbox.sh`.

## What passing looks like

```
$ bash scripts/contract-run.sh search-stack-visibility
Expected engines, enabled (4): ['bing', 'brave', 'google cse', 'yandex']
queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex
                                    unresponsive: (none)
         'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex
                                    unresponsive: (none)
EXTRACTION: 71016 chars of markdown returned
VERDICT: PASS -- multiple engines contributing, extraction healthy
```

## What failing looks like

* A query whose results come from fewer than `SEARCH_CHECK_MIN_ENGINES`
  engines (default 2) fails and names the engines that did contribute.
* An extraction request that errors or returns empty markdown fails.
* An enabled, expected engine that contributed nothing without reporting an
  error is printed under `SILENT-ZERO ENGINES REPORTED`.

## Configuration

Environment overrides (see the script docstring for the full list):

| Variable | Default | Meaning |
| --- | --- | --- |
| `SEARXNG_URL` | `http://192.168.68.7:8888` | SearXNG base URL |
| `FIRECRAWL_URL` | `http://192.168.68.7:3002` | Firecrawl base URL |
| `SEARCH_CHECK_QUERIES` | `proxmox backup server,python asyncio tutorial` | fixed queries |
| `SEARCH_CHECK_MIN_ENGINES` | `2` | minimum contributing engines per query |
| `SEARCH_CHECK_ENGINES` | `bing,brave,google cse,yandex,duckduckgo` | engines a silent zero is reported for |
| `SEARCH_CHECK_EXTRACT_URL` | Wikipedia Proxmox article | page used for the extraction leg |

## Known residual risk

DuckDuckGo is reached through a forward proxy on the VPS
(`10.10.10.1:3128`, WireGuard) because the house egress IP is CAPTCHA'd. The
VPS is a datacenter address and DuckDuckGo may flag it, so DuckDuckGo is
coverage, never a required engine. When it regresses it appears either in
`unresponsive_engines` (an explicit error) or under the silent-zero report;
neither fails the check, but both are visible.