kyle
35e2356541
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
1.8 KiB
P3 §10 bake-off protocol
Query list: search_queries.txt (28, Kyle-approved 2026-09-23).
Targets: search_targets.txt (same order; open = no required domain).
Do not enable sources or start runs until this commit is on kyle/sift master.
SearXNG journal redaction (searx.service) is a prerequisite for search.searxng.enabled.
Configurations (never shown to the judge)
- SearXNG alone
- SearXNG + Brave LLM Context
- SearXNG + Brave LLM Context + Tavily
- Current Hermes Tavily
web_search(live tool, not sift)
Sift does not take source names from the request. Configs 1–3 are three orion
[search] settings (or three sequential enables), not query parameters.
Config 4 is a fleet-host Hermes call, same 28 queries, limit matching sift k.
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
Blind judging
The runner writes only labelled packs A B C D. Each pack is a
JSON list of 28 hit-lists (query order from search_queries.txt). A hit is
exactly {rank, url, title, snippet} — no scores, sources, warnings, latency,
credits, or Hermes description/position. Vendor tokens (searxng, brave,
tavily, hermes) outside url/snippet fail the shuffle. Raw dumps keep
the extra fields for unblinding.
The mapping A–D → 1–4 goes in a separate key file that the judge does
not read until scoring is done. Default path (gitignored):
tests/testdata/bakeoff-out/key.json
tests/testdata/bakeoff-out/{A,B,C,D}.json
Shuffle with tests/bakeoff/blind.py after the four raw dumps exist.
Judges: talos or Claude. Score each query: would you sift fetch the
correct URL from this list? Against search_targets.txt when it is not open.
After scores land, open key.json and report: winner (becomes default-on),
cost/query, p90 latency.