kyle
9a8dae3948
Kyle-approved fleet harvest. Sibling targets (domain/page or open). Runner labels A-D; mapping stays in a sealed key file until scoring.
1.6 KiB
P3 §10 bake-off protocol
Query list: search_queries.txt (28, Kyle-approved 2026-09-23).
Targets: search_targets.txt (same order; open = no required domain).
Do not enable sources or start runs until this commit is on kyle/sift master.
SearXNG journal redaction (searx.service) is a prerequisite for search.searxng.enabled.
Configurations (never shown to the judge)
- SearXNG alone
- SearXNG + Brave LLM Context
- SearXNG + Brave LLM Context + Tavily
- Current Hermes Tavily
web_search(live tool, not sift)
Sift does not take source names from the request. Configs 1–3 are three orion
[search] settings (or three sequential enables), not query parameters.
Config 4 is a fleet-host Hermes call, same 28 queries, limit matching sift k.
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
Blind judging
The runner writes only labelled packs A B C D. Each pack is the
full 28-query result set for one configuration, with no source names, no
warnings that name a vendor, and no config id.
The mapping A–D → 1–4 goes in a separate key file that the judge does
not read until scoring is done. Default path (gitignored):
tests/testdata/bakeoff-out/key.json
tests/testdata/bakeoff-out/{A,B,C,D}.json
Shuffle with tests/bakeoff/blind.py after the four raw dumps exist.
Judges: talos or Claude. Score each query: would you sift fetch the
correct URL from this list? Against search_targets.txt when it is not open.
After scores land, open key.json and report: winner (becomes default-on),
cost/query, p90 latency.