p3: actually blind bake-off packs

Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
This commit is contained in:
2026-09-23 14:13:26 -07:00
parent 9a8dae3948
commit 35e2356541
3 changed files with 236 additions and 21 deletions
+6 -3
View File
@@ -21,9 +21,12 @@ Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave c
## Blind judging
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is the
full 28-query result set for one configuration, with no source names, no
warnings that name a vendor, and no config id.
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is a
JSON list of 28 hit-lists (query order from `search_queries.txt`). A hit is
exactly `{rank, url, title, snippet}` — no scores, sources, warnings, latency,
credits, or Hermes `description`/`position`. Vendor tokens (`searxng`, `brave`,
`tavily`, `hermes`) outside `url`/`snippet` fail the shuffle. Raw dumps keep
the extra fields for unblinding.
The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
not read until scoring is done. Default path (gitignored):