kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

p3: actually blind bake-off packs

Browse Source 
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
This commit is contained in:
kyle
2026-09-23 14:13:26 -07:00
parent 9a8dae3948
commit 35e2356541
3 changed files with 236 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/P3.md
+2 -1
View File
@@ -281,7 +281,8 @@ For each query, run four configurations and record top-k URLs + latency + credit
3. SearXNG + Brave LLM Context + Tavily
4. Current Hermes Tavily `web_search` (the live tool, not sift)
Judged **blind**. The runner writes result packs labelled A–D with no config names.
Judged **blind**. The runner writes result packs labelled A–D; each pack is
only `[{rank, url, title, snippet}, …]` per query, no scores/sources/warnings.
The A–D → 1–4 mapping is a separate key file the judge does not read until scoring
is done. Protocol: `tests/testdata/BAKEOFF.md`. Shuffle helper: `tests/bakeoff/blind.py`.
Judges: talos or Claude. Score: would you `sift fetch` the correct URL from this list?