root
8b2eba4f7a
PR Pipeline — Authorize → Validate → Review → Merge / auth (pull_request) Successful in 3s
PR Pipeline — Authorize → Validate → Review → Merge / validate (pull_request) Successful in 4s
PR Pipeline — Authorize → Validate → Review → Merge / lint (pull_request) Successful in 11s
PR Pipeline — Authorize → Validate → Review → Merge / ai-review (pull_request) Successful in 8s
PR Pipeline — Authorize → Validate → Review → Merge / gate (pull_request) Successful in 1s
Follow-up corrections after review: 1. DuckDuckGo is NOT fixed. The VPS fallback egress has since been flagged by DuckDuckGo too (HTTP 202 + challenge markers), so it reports CAPTCHA on both paths. The contract and script docstring now say so instead of claiming a fix that had already expired. It stays enabled as best-effort coverage so a recovery shows up as a contribution. 2. The relationship between silent zeros and the verdict is now explicit in both the script output and the contract: an enabled expected engine that contributes zero with no error is REPORTED, not fatal. Only the <SEARCH_CHECK_MIN_ENGINES> floor and the extraction leg fail the run. This is deliberate - de-duplication and query-shape make a zero non-probative. 3. Recorded that 'google cse' uses a THIRD PARTY's public search-engine id hardcoded in the SearXNG build, not a key we own; its quota and availability are outside our control, and our own free key would need a wrapper (not built). VPS forward proxy is now a real service: /opt/fwd-proxy docker compose with restart: unless-stopped, a healthy healthcheck, and Docker enabled at boot.
125 lines
6.0 KiB
Markdown
125 lines
6.0 KiB
Markdown
---
|
|
kind: function
|
|
name: search-stack-visibility
|
|
description: >
|
|
Makes the shared search stack observable. Every agent reaches one SearXNG
|
|
instance (http://192.168.68.7:8888) and one extraction service (Firecrawl,
|
|
http://192.168.68.7:3002). Before this check the stack could degrade to a
|
|
single engine, or an enabled engine could return nothing at all, without any
|
|
error surfacing anywhere.
|
|
|
|
This contract runs scripts/search-stack-check.py, which:
|
|
* runs two fixed queries against SearXNG and FAILS when fewer than two
|
|
engines contribute, printing the contributing engines and every
|
|
unresponsive_engines entry;
|
|
* checks extraction by scraping a known page through Firecrawl and FAILS
|
|
when the returned markdown is empty or the request fails;
|
|
* reports every silent-zero engine explicitly (enabled, not in
|
|
unresponsive_engines, contributed no results).
|
|
|
|
Multi-engine state (2026-09-25): bing, google cse, brave and yandex
|
|
contribute on every query. duckduckgo is NOT working: the house egress IP
|
|
and the VPS fallback egress are both flagged by DuckDuckGo and it reports
|
|
CAPTCHA. It is left enabled as best-effort coverage so that a recovery shows
|
|
up as a contribution.
|
|
|
|
google cse is a third party's public search-engine id hardcoded in the
|
|
SearXNG build. Quota and availability are outside our control.
|
|
|
|
SCHEDULED: /etc/cron.d/contract-runner on CT 100 (abiba), hourly at :15,
|
|
via scripts/contract-run.sh search-stack-visibility. Logs land in
|
|
/var/log/contract-runs/. A failure also raises a firstmate inbox note.
|
|
version: 1.1.0
|
|
---
|
|
|
|
## Purpose
|
|
|
|
The fleet has exactly one search endpoint and one extraction endpoint. If
|
|
either degrades, every agent silently loses capability at the same moment.
|
|
The failure mode this contract exists to close is *silent* degradation: a
|
|
query that still returns a page of results while all but one engine have
|
|
stopped contributing, or an enabled engine that answers with zero results and
|
|
raises no error.
|
|
|
|
## Execution model
|
|
|
|
The contract is a host-scheduled check, not an agent workflow. It is driven by
|
|
`scripts/contract-run.sh search-stack-visibility` from
|
|
`/etc/cron.d/contract-runner` on CT 100. `contract-run.sh` resolves the
|
|
mapping to `scripts/search-stack-check.py`, runs it under a timeout, writes a
|
|
timestamped log to `/var/log/contract-runs/`, and on non-zero exit raises a
|
|
firstmate inbox note through `bin/fm-inbox.sh`.
|
|
|
|
## What passing looks like
|
|
|
|
```
|
|
$ bash scripts/contract-run.sh search-stack-visibility
|
|
Expected engines, enabled (5): ['bing', 'brave', 'duckduckgo', 'google cse', 'yandex']
|
|
queries: 'proxmox backup server' -> contributing: bing, brave, google cse, yandex
|
|
unresponsive: duckduckgo=CAPTCHA
|
|
'python asyncio tutorial' -> contributing: bing, brave, google cse, yandex
|
|
EXTRACTION: 71016 chars of markdown returned
|
|
VERDICT: PASS -- multiple engines contributing, extraction healthy
|
|
```
|
|
|
|
## What failing looks like
|
|
|
|
* A query whose results come from fewer than `SEARCH_CHECK_MIN_ENGINES`
|
|
engines (default 2) fails and names the engines that did contribute.
|
|
* An extraction request that errors or returns empty markdown fails.
|
|
|
|
## Silent zeros are reported, not fatal
|
|
|
|
An enabled, expected engine that contributed nothing **without reporting an
|
|
error** is printed under `SILENT-ZERO ENGINES REPORTED`, and each occurrence is
|
|
annotated `reported, not fatal`. This is deliberate:
|
|
|
|
* a general query can legitimately draw zero results from an engine that only
|
|
fires on certain query shapes, and results are de-duplicated across engines,
|
|
so a zero does not by itself prove the engine is broken;
|
|
* the run therefore fails only on the two conditions that do prove loss of
|
|
capability -- fewer than two contributing engines, and a broken extraction
|
|
leg.
|
|
|
|
A run can consequently print `VERDICT: PASS` while still listing a silent
|
|
zero. That is the intended relationship: the zero is *visible*, not *fatal*.
|
|
An engine that fails with an error (for example DuckDuckGo returning CAPTCHA)
|
|
appears in `unresponsive_engines` instead.
|
|
|
|
## Google coverage is third-party, not ours
|
|
|
|
The free Google-derived results come from the SearXNG build's built-in
|
|
`google cse` engine. It uses **a third party's public search-engine id
|
|
hardcoded in the build** (`google_cse.py`, `CX = "partner-pub-8993..."`,
|
|
blackle.com), not a key or id we own. Its quota and availability are outside
|
|
our control and it can be rate-limited or withdrawn without notice. No engine
|
|
in this build accepts our own Google Custom Search key; using our own free key
|
|
would require a small wrapper service, which is deliberately **not** built.
|
|
|
|
## Configuration
|
|
|
|
Environment overrides (see the script docstring for the full list):
|
|
|
|
| Variable | Default | Meaning |
|
|
| --- | --- | --- |
|
|
| `SEARXNG_URL` | `http://192.168.68.7:8888` | SearXNG base URL |
|
|
| `FIRECRAWL_URL` | `http://192.168.68.7:3002` | Firecrawl base URL |
|
|
| `SEARCH_CHECK_QUERIES` | `proxmox backup server,python asyncio tutorial` | fixed queries |
|
|
| `SEARCH_CHECK_MIN_ENGINES` | `2` | minimum contributing engines per query |
|
|
| `SEARCH_CHECK_ENGINES` | `bing,brave,google cse,yandex,duckduckgo` | engines a silent zero is reported for |
|
|
| `SEARCH_CHECK_EXTRACT_URL` | Wikipedia Proxmox article | page used for the extraction leg |
|
|
|
|
## Known residual risk
|
|
|
|
DuckDuckGo is **not** working. The house egress IP is CAPTCHA'd by
|
|
DuckDuckGo, and a forward proxy on the VPS (`10.10.10.1:3128`, WireGuard) was
|
|
built as a second egress -- but DuckDuckGo has since flagged the VPS address
|
|
too (HTTP 202 with challenge markers), so DuckDuckGo now reports CAPTCHA on
|
|
both paths. It is left enabled as best-effort coverage: if DuckDuckGo
|
|
unflags either address it will show up as a contribution, and until then it is
|
|
visible in `unresponsive_engines` every run. It is never a required engine.
|
|
|
|
The VPS forward proxy remains a real service (`/opt/fwd-proxy`,
|
|
`restart: unless-stopped`, healthy healthcheck, Docker enabled at boot) so the
|
|
second egress path is available for any engine that benefits from it in future.
|