kyle
eb4e46d036
Exempt title from the vendor-token check. Truncate snippets to 160 chars and clip all four packs to the same per-query k. run.py fails cached replies with a rerun message instead of a config mismatch.
62 lines
2.6 KiB
Markdown
62 lines
2.6 KiB
Markdown
# P3 §10 bake-off protocol
|
||
|
||
Query list: `search_queries.txt` (28, Kyle-approved 2026-09-23).
|
||
Targets: `search_targets.txt` (same order; `open` = no required domain).
|
||
|
||
Do not enable sources or start runs until this commit is on `kyle/sift` master.
|
||
SearXNG journal redaction (`searx.service`) is a prerequisite for `search.searxng.enabled`.
|
||
|
||
## Configurations (never shown to the judge)
|
||
|
||
1. SearXNG alone
|
||
2. SearXNG + Brave LLM Context
|
||
3. SearXNG + Brave LLM Context + Tavily
|
||
4. Current Hermes Tavily `web_search` (live tool, not sift)
|
||
|
||
Sift does not take source names from the request. Configs 1–3 are three orion
|
||
`[search]` settings (or three sequential enables), not query parameters.
|
||
Config 4 is a fleet-host Hermes call, same 28 queries, `limit` matching sift `k`.
|
||
|
||
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
|
||
|
||
## Blind judging
|
||
|
||
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is a
|
||
JSON list of 28 hit-lists (query order from `search_queries.txt`). A hit is
|
||
exactly `{rank, url, title, snippet}` — no scores, sources, warnings, latency,
|
||
credits, or Hermes `description`/`position`. Vendor tokens (`searxng`, `brave`,
|
||
`tavily`, `hermes`) outside `url`/`title`/`snippet` fail the shuffle. Snippets
|
||
are capped at 160 characters. All four packs are clipped to the same per-query
|
||
`k` (min across configs, then `--k`) so hit-count and snippet length cannot
|
||
name the configuration. Raw dumps keep the extra fields for unblinding.
|
||
|
||
The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
|
||
not read until scoring is done. Default path (gitignored):
|
||
|
||
```
|
||
tests/testdata/bakeoff-out/key.json
|
||
tests/testdata/bakeoff-out/{A,B,C,D}.json
|
||
```
|
||
|
||
Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.
|
||
|
||
Collect a dump after orion is set to that configuration:
|
||
|
||
```
|
||
python3 tests/bakeoff/run.py --config searxng_brave_tavily
|
||
python3 tests/bakeoff/run.py --config searxng_brave
|
||
python3 tests/bakeoff/run.py --config searxng
|
||
TAVILY_API_KEY=… python3 tests/bakeoff/run.py --config hermes_tavily
|
||
python3 tests/bakeoff/blind.py --raw-dir tests/testdata/bakeoff-raw
|
||
```
|
||
|
||
All three sources on → collect config 3 first, then turn off tavily (config 2),
|
||
then brave_llm (config 1). `run.py` refuses a dump whose `sources_ok` does not
|
||
match `--config`, and refuses `cached: true` (rerun after the hour or change k).
|
||
|
||
Judges: **talos or Claude**. Score each query: would you `sift fetch` the
|
||
correct URL from this list? Against `search_targets.txt` when it is not `open`.
|
||
|
||
After scores land, open `key.json` and report: winner (becomes default-on),
|
||
cost/query, p90 latency.
|