p3: actually blind bake-off packs
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
This commit is contained in:
Vendored
+6
-3
@@ -21,9 +21,12 @@ Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave c
|
||||
|
||||
## Blind judging
|
||||
|
||||
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is the
|
||||
full 28-query result set for one configuration, with no source names, no
|
||||
warnings that name a vendor, and no config id.
|
||||
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is a
|
||||
JSON list of 28 hit-lists (query order from `search_queries.txt`). A hit is
|
||||
exactly `{rank, url, title, snippet}` — no scores, sources, warnings, latency,
|
||||
credits, or Hermes `description`/`position`. Vendor tokens (`searxng`, `brave`,
|
||||
`tavily`, `hermes`) outside `url`/`snippet` fail the shuffle. Raw dumps keep
|
||||
the extra fields for unblinding.
|
||||
|
||||
The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
|
||||
not read until scoring is done. Default path (gitignored):
|
||||
|
||||
Reference in New Issue
Block a user