# P3 §10 bake-off protocol

Query list: `search_queries.txt` (28, Kyle-approved 2026-09-23).
Targets: `search_targets.txt` (same order; `open` = no required domain).

Do not enable sources or start runs until this commit is on `kyle/sift` master.
SearXNG journal redaction (`searx.service`) is a prerequisite for `search.searxng.enabled`.

## Configurations (never shown to the judge)

1. SearXNG alone
2. SearXNG + Brave LLM Context
3. SearXNG + Brave LLM Context + Tavily
4. Current Hermes Tavily `web_search` (live tool, not sift)

Sift does not take source names from the request. Configs 1–3 are three orion
`[search]` settings (or three sequential enables), not query parameters.
Config 4 is a fleet-host Hermes call, same 28 queries, `limit` matching sift `k`.

Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.

## Blind judging

The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is the
full 28-query result set for one configuration, with no source names, no
warnings that name a vendor, and no config id.

The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
not read until scoring is done. Default path (gitignored):

```
tests/testdata/bakeoff-out/key.json
tests/testdata/bakeoff-out/{A,B,C,D}.json
```

Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.

Judges: **talos or Claude**. Score each query: would you `sift fetch` the
correct URL from this list? Against `search_targets.txt` when it is not `open`.

After scores land, open `key.json` and report: winner (becomes default-on),
cost/query, p90 latency.