# P3 §10 bake-off protocol

Query list: `search_queries.txt` (28, Kyle-approved 2026-09-23).
Targets: `search_targets.txt` (same order; `open` = no required domain).

Do not enable sources or start runs until this commit is on `kyle/sift` master.
SearXNG journal redaction (`searx.service`) is a prerequisite for `search.searxng.enabled`.

## Configurations (never shown to the judge)

1. SearXNG alone
2. SearXNG + Brave LLM Context
3. SearXNG + Brave LLM Context + Tavily
4. Current Hermes Tavily `web_search` (live tool, not sift)

Sift does not take source names from the request. Configs 1–3 are three orion
`[search]` settings (or three sequential enables), not query parameters.
Config 4 is a fleet-host Hermes call, same 28 queries, `limit` matching sift `k`.

Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.

## Blind judging

The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is a
JSON list of 28 hit-lists (query order from `search_queries.txt`). A hit is
exactly `{rank, url, title, snippet}` — no scores, sources, warnings, latency,
credits, or Hermes `description`/`position`. Vendor tokens (`searxng`, `brave`,
`tavily`, `hermes`) outside `url`/`snippet` fail the shuffle. Raw dumps keep
the extra fields for unblinding.

The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
not read until scoring is done. Default path (gitignored):

```
tests/testdata/bakeoff-out/key.json
tests/testdata/bakeoff-out/{A,B,C,D}.json
```

Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.

Collect a dump after orion is set to that configuration:

```
python3 tests/bakeoff/run.py --config searxng
python3 tests/bakeoff/run.py --config searxng_brave
python3 tests/bakeoff/run.py --config searxng_brave_tavily
TAVILY_API_KEY=… python3 tests/bakeoff/run.py --config hermes_tavily
python3 tests/bakeoff/blind.py --raw-dir tests/testdata/bakeoff-raw
```

`run.py` refuses to stamp a sift dump if `sizes.sources_ok` does not match `--config`.
Titan does not enable sources or edit nixos.

Judges: **talos or Claude**. Score each query: would you `sift fetch` the
correct URL from this list? Against `search_targets.txt` when it is not `open`.

After scores land, open `key.json` and report: winner (becomes default-on),
cost/query, p90 latency.