kyle
9a8dae3948
Kyle-approved fleet harvest. Sibling targets (domain/page or open). Runner labels A-D; mapping stays in a sealed key file until scoring.
43 lines
1.6 KiB
Markdown
43 lines
1.6 KiB
Markdown
# P3 §10 bake-off protocol
|
||
|
||
Query list: `search_queries.txt` (28, Kyle-approved 2026-09-23).
|
||
Targets: `search_targets.txt` (same order; `open` = no required domain).
|
||
|
||
Do not enable sources or start runs until this commit is on `kyle/sift` master.
|
||
SearXNG journal redaction (`searx.service`) is a prerequisite for `search.searxng.enabled`.
|
||
|
||
## Configurations (never shown to the judge)
|
||
|
||
1. SearXNG alone
|
||
2. SearXNG + Brave LLM Context
|
||
3. SearXNG + Brave LLM Context + Tavily
|
||
4. Current Hermes Tavily `web_search` (live tool, not sift)
|
||
|
||
Sift does not take source names from the request. Configs 1–3 are three orion
|
||
`[search]` settings (or three sequential enables), not query parameters.
|
||
Config 4 is a fleet-host Hermes call, same 28 queries, `limit` matching sift `k`.
|
||
|
||
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
|
||
|
||
## Blind judging
|
||
|
||
The runner writes **only** labelled packs `A` `B` `C` `D`. Each pack is the
|
||
full 28-query result set for one configuration, with no source names, no
|
||
warnings that name a vendor, and no config id.
|
||
|
||
The mapping `A–D → 1–4` goes in a **separate key file** that the judge does
|
||
not read until scoring is done. Default path (gitignored):
|
||
|
||
```
|
||
tests/testdata/bakeoff-out/key.json
|
||
tests/testdata/bakeoff-out/{A,B,C,D}.json
|
||
```
|
||
|
||
Shuffle with `tests/bakeoff/blind.py` after the four raw dumps exist.
|
||
|
||
Judges: **talos or Claude**. Score each query: would you `sift fetch` the
|
||
correct URL from this list? Against `search_targets.txt` when it is not `open`.
|
||
|
||
After scores land, open `key.json` and report: winner (becomes default-on),
|
||
cost/query, p90 latency.
|