kyle
eb4e46d036
Exempt title from the vendor-token check. Truncate snippets to 160 chars and clip all four packs to the same per-query k. run.py fails cached replies with a rerun message instead of a config mismatch.
2.6 KiB
P3 §10 bake-off protocol
Query list: search_queries.txt (28, Kyle-approved 2026-09-23).
Targets: search_targets.txt (same order; open = no required domain).
Do not enable sources or start runs until this commit is on kyle/sift master.
SearXNG journal redaction (searx.service) is a prerequisite for search.searxng.enabled.
Configurations (never shown to the judge)
- SearXNG alone
- SearXNG + Brave LLM Context
- SearXNG + Brave LLM Context + Tavily
- Current Hermes Tavily
web_search(live tool, not sift)
Sift does not take source names from the request. Configs 1–3 are three orion
[search] settings (or three sequential enables), not query parameters.
Config 4 is a fleet-host Hermes call, same 28 queries, limit matching sift k.
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
Blind judging
The runner writes only labelled packs A B C D. Each pack is a
JSON list of 28 hit-lists (query order from search_queries.txt). A hit is
exactly {rank, url, title, snippet} — no scores, sources, warnings, latency,
credits, or Hermes description/position. Vendor tokens (searxng, brave,
tavily, hermes) outside url/title/snippet fail the shuffle. Snippets
are capped at 160 characters. All four packs are clipped to the same per-query
k (min across configs, then --k) so hit-count and snippet length cannot
name the configuration. Raw dumps keep the extra fields for unblinding.
The mapping A–D → 1–4 goes in a separate key file that the judge does
not read until scoring is done. Default path (gitignored):
tests/testdata/bakeoff-out/key.json
tests/testdata/bakeoff-out/{A,B,C,D}.json
Shuffle with tests/bakeoff/blind.py after the four raw dumps exist.
Collect a dump after orion is set to that configuration:
python3 tests/bakeoff/run.py --config searxng_brave_tavily
python3 tests/bakeoff/run.py --config searxng_brave
python3 tests/bakeoff/run.py --config searxng
TAVILY_API_KEY=… python3 tests/bakeoff/run.py --config hermes_tavily
python3 tests/bakeoff/blind.py --raw-dir tests/testdata/bakeoff-raw
All three sources on → collect config 3 first, then turn off tavily (config 2),
then brave_llm (config 1). run.py refuses a dump whose sources_ok does not
match --config, and refuses cached: true (rerun after the hour or change k).
Judges: talos or Claude. Score each query: would you sift fetch the
correct URL from this list? Against search_targets.txt when it is not open.
After scores land, open key.json and report: winner (becomes default-on),
cost/query, p90 latency.