kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

p3: blind titles, cap snippets, equalise k; abort cached dumps

Browse Source 
Exempt title from the vendor-token check. Truncate snippets to 160
chars and clip all four packs to the same per-query k. run.py fails
cached replies with a rerun message instead of a config mismatch.
This commit is contained in:
kyle
2026-09-23 15:32:45 -07:00
parent 7558eca46d
commit eb4e46d036
3 changed files with 74 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/testdata/BAKEOFF.md
Vendored
+9 -6
View File
@@ -25,8 +25,10 @@ The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is a
JSON list of 28 hit-lists (query order from `search_queries.txt`). A hit is
exactly `{rank, url, title, snippet}` — no scores, sources, warnings, latency,
credits, or Hermes `description`/`position`. Vendor tokens (`searxng`, `brave`,
`tavily`, `hermes`) outside `url`/`snippet` fail the shuffle. Raw dumps keep
the extra fields for unblinding.
`tavily`, `hermes`) outside `url`/`title`/`snippet` fail the shuffle. Snippets
are capped at 160 characters. All four packs are clipped to the same per-query
`k` (min across configs, then `--k`) so hit-count and snippet length cannot
name the configuration. Raw dumps keep the extra fields for unblinding.
The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
not read until scoring is done. Default path (gitignored):
@@ -41,15 +43,16 @@ Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.
Collect a dump after orion is set to that configuration:
```
python3 tests/bakeoff/run.py --config searxng
python3 tests/bakeoff/run.py --config searxng_brave
python3 tests/bakeoff/run.py --config searxng_brave_tavily
python3 tests/bakeoff/run.py --config searxng_brave
python3 tests/bakeoff/run.py --config searxng
TAVILY_API_KEY=… python3 tests/bakeoff/run.py --config hermes_tavily
python3 tests/bakeoff/blind.py --raw-dir tests/testdata/bakeoff-raw
```
`run.py` refuses to stamp a sift dump if `sizes.sources_ok` does not match `--config`.
Titan does not enable sources or edit nixos.
All three sources on → collect config 3 first, then turn off tavily (config 2),
then brave_llm (config 1). `run.py` refuses a dump whose `sources_ok` does not
match `--config`, and refuses `cached: true` (rerun after the hour or change k).
Judges: **talos or Claude**. Score each query: would you `sift fetch` the
correct URL from this list? Against `search_targets.txt` when it is not `open`.