kyle
7558eca46d
run.py records one named config at a time. Sift dumps fail if sizes.sources_ok does not match --config. Hermes Tavily uses the live-tool request shape, not sift. Raw dumps stay gitignored.
2.3 KiB
P3 §10 bake-off protocol
Query list: search_queries.txt (28, Kyle-approved 2026-09-23).
Targets: search_targets.txt (same order; open = no required domain).
Do not enable sources or start runs until this commit is on kyle/sift master.
SearXNG journal redaction (searx.service) is a prerequisite for search.searxng.enabled.
Configurations (never shown to the judge)
- SearXNG alone
- SearXNG + Brave LLM Context
- SearXNG + Brave LLM Context + Tavily
- Current Hermes Tavily
web_search(live tool, not sift)
Sift does not take source names from the request. Configs 1–3 are three orion
[search] settings (or three sequential enables), not query parameters.
Config 4 is a fleet-host Hermes call, same 28 queries, limit matching sift k.
Record per query: top-k URLs, titles, snippets, latency, Tavily credits, Brave calls.
Blind judging
The runner writes only labelled packs A B C D. Each pack is a
JSON list of 28 hit-lists (query order from search_queries.txt). A hit is
exactly {rank, url, title, snippet} — no scores, sources, warnings, latency,
credits, or Hermes description/position. Vendor tokens (searxng, brave,
tavily, hermes) outside url/snippet fail the shuffle. Raw dumps keep
the extra fields for unblinding.
The mapping A–D → 1–4 goes in a separate key file that the judge does
not read until scoring is done. Default path (gitignored):
tests/testdata/bakeoff-out/key.json
tests/testdata/bakeoff-out/{A,B,C,D}.json
Shuffle with tests/bakeoff/blind.py after the four raw dumps exist.
Collect a dump after orion is set to that configuration:
python3 tests/bakeoff/run.py --config searxng
python3 tests/bakeoff/run.py --config searxng_brave
python3 tests/bakeoff/run.py --config searxng_brave_tavily
TAVILY_API_KEY=… python3 tests/bakeoff/run.py --config hermes_tavily
python3 tests/bakeoff/blind.py --raw-dir tests/testdata/bakeoff-raw
run.py refuses to stamp a sift dump if sizes.sources_ok does not match --config.
Titan does not enable sources or edit nixos.
Judges: talos or Claude. Score each query: would you sift fetch the
correct URL from this list? Against search_targets.txt when it is not open.
After scores land, open key.json and report: winner (becomes default-on),
cost/query, p90 latency.