p3: harvest 28 bake-off queries with targets and blind protocol

Kyle-approved fleet harvest. Sibling targets (domain/page or open). Runner labels A-D; mapping stays in a sealed key file until scoring.
2026-09-23 14:09:42 -07:00
parent 8ca3adec9e
commit 9a8dae3948
6 changed files with 206 additions and 6 deletions
@@ -0,0 +1,42 @@
+# P3 §10 bake-off protocol
+
+Query list: `search_queries.txt` (28, Kyle-approved 2026-09-23).
+Targets: `search_targets.txt` (same order; `open` = no required domain).
+
+Do not enable sources or start runs until this commit is on `kyle/sift` master.
+SearXNG journal redaction (`searx.service`) is a prerequisite for `search.searxng.enabled`.
+
+## Configurations (never shown to the judge)
+
+1. SearXNG alone
+2. SearXNG + Brave LLM Context
+3. SearXNG + Brave LLM Context + Tavily
+4. Current Hermes Tavily `web_search` (live tool, not sift)
+
+Sift does not take source names from the request. Configs 1–3 are three orion
+`[search]` settings (or three sequential enables), not query parameters.
+Config 4 is a fleet-host Hermes call, same 28 queries, `limit` matching sift `k`.
+
+Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
+
+## Blind judging
+
+The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is the
+full 28-query result set for one configuration, with no source names, no
+warnings that name a vendor, and no config id.
+
+The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
+not read until scoring is done. Default path (gitignored):
+
+```
+tests/testdata/bakeoff-out/key.json
+tests/testdata/bakeoff-out/{A,B,C,D}.json
+```
+
+Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.
+
+Judges: **talos or Claude**. Score each query: would you `sift fetch` the
+correct URL from this list? Against `search_targets.txt` when it is not `open`.
+
+After scores land, open `key.json` and report: winner (becomes default-on),
+cost/query, p90 latency.