kyle
7558eca46d
run.py records one named config at a time. Sift dumps fail if sizes.sources_ok does not match --config. Hermes Tavily uses the live-tool request shape, not sift. Raw dumps stay gitignored.
59 lines
2.3 KiB
Markdown
59 lines
2.3 KiB
Markdown
# P3 §10 bake-off protocol
|
||
|
||
Query list: `search_queries.txt` (28, Kyle-approved 2026-09-23).
|
||
Targets: `search_targets.txt` (same order; `open` = no required domain).
|
||
|
||
Do not enable sources or start runs until this commit is on `kyle/sift` master.
|
||
SearXNG journal redaction (`searx.service`) is a prerequisite for `search.searxng.enabled`.
|
||
|
||
## Configurations (never shown to the judge)
|
||
|
||
1. SearXNG alone
|
||
2. SearXNG + Brave LLM Context
|
||
3. SearXNG + Brave LLM Context + Tavily
|
||
4. Current Hermes Tavily `web_search` (live tool, not sift)
|
||
|
||
Sift does not take source names from the request. Configs 1–3 are three orion
|
||
`[search]` settings (or three sequential enables), not query parameters.
|
||
Config 4 is a fleet-host Hermes call, same 28 queries, `limit` matching sift `k`.
|
||
|
||
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
|
||
|
||
## Blind judging
|
||
|
||
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is a
|
||
JSON list of 28 hit-lists (query order from `search_queries.txt`). A hit is
|
||
exactly `{rank, url, title, snippet}` — no scores, sources, warnings, latency,
|
||
credits, or Hermes `description`/`position`. Vendor tokens (`searxng`, `brave`,
|
||
`tavily`, `hermes`) outside `url`/`snippet` fail the shuffle. Raw dumps keep
|
||
the extra fields for unblinding.
|
||
|
||
The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
|
||
not read until scoring is done. Default path (gitignored):
|
||
|
||
```
|
||
tests/testdata/bakeoff-out/key.json
|
||
tests/testdata/bakeoff-out/{A,B,C,D}.json
|
||
```
|
||
|
||
Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.
|
||
|
||
Collect a dump after orion is set to that configuration:
|
||
|
||
```
|
||
python3 tests/bakeoff/run.py --config searxng
|
||
python3 tests/bakeoff/run.py --config searxng_brave
|
||
python3 tests/bakeoff/run.py --config searxng_brave_tavily
|
||
TAVILY_API_KEY=… python3 tests/bakeoff/run.py --config hermes_tavily
|
||
python3 tests/bakeoff/blind.py --raw-dir tests/testdata/bakeoff-raw
|
||
```
|
||
|
||
`run.py` refuses to stamp a sift dump if `sizes.sources_ok` does not match `--config`.
|
||
Titan does not enable sources or edit nixos.
|
||
|
||
Judges: **talos or Claude**. Score each query: would you `sift fetch` the
|
||
correct URL from this list? Against `search_targets.txt` when it is not `open`.
|
||
|
||
After scores land, open `key.json` and report: winner (becomes default-on),
|
||
cost/query, p90 latency.
|