Extends search-stack-visibility with a ranking assertion: for the fixed query
set, no config demote_domains host may appear in the top 3, and the known
non-answers (bestbuy.com, merriam-webster.com) must not be returned at all.
Without this the layer could rot back to raw engine ordering unnoticed - the same
way the endpoint colours silently rotted before 2026-09-26. It reads the demote
list from the SAME config the layer uses, so the guard cannot drift from the
policy it is guarding.
Live: 'ok: no demoted host in the top 3; no banned non-answer returned';
visibility contract still PASSES end to end.
Raw multi-engine aggregation had no dedupe, no filtering and no reranking.
Measured 2026-09-26: 'best practices agent context management' returned
bestbuy.com and merriam-webster.com, plus 4 content farms, with medium.com twice;
'proxmox thin pool metadata exhaustion recovery' put four SEO blogs ABOVE the
real Proxmox forum threads. Identical queries also ranked DIFFERENTLY between
runs, which is why the fix is deterministic rather than trusting the engines.
scripts/search-agent-consume.py:
1. dedupe by normalised URL (tracking params and fragments stripped)
2. drop non-answers - shopping/dictionary hosts, navigational host roots,
search/shopping/cart/login paths and query keys
3. demote content farms and promote primary sources
4. STABLE sort (score desc, then original position) so runs are reproducible
5. extract page text for the top N via Firecrawl POST /v1/scrape under an
explicit character budget, so an agent gets usable material in ONE call
6. emit stable JSON with engine provenance and source_type
Policy is config, not code: config/search-ranking.yaml holds demote_domains,
prefer_domains, non_answer rules and the extraction budget, so it is reviewable
and changeable without touching the module. Content farms are DEMOTED rather
than dropped so a useful hit is not lost, it just cannot outrank a primary.
A '/products/' path rule was REMOVED after the before/after run caught it
dropping docs.digitalocean.com/products/inference/... - a legitimate docs page.
Shopping is caught by the host list instead, which has no such false positive.
Measured: 'best practices...' top 8 becomes anthropic, langchain, jetbrains,
blog.jetbrains, docs.langchain, reddit, cursor, reddit - no content farm.
'proxmox thin pool...' moves the forum threads from positions 5-9 to 1-4.
Extraction: 5 items, 12000 chars of 12000 budget, 0 failures, 5.28s; whole run
6.4s wall.
Contract: search-agent-consumption.prose.md, including the honest reachability
gap - the pi MCP search server's shape is not ours to change, so this layer is
NOT wired into it.
root