From 7f4f0d50bf81d64f945ba8f7e8199e79f2de5bbf Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Tue, 29 Sep 2026 15:03:14 -0700 Subject: [PATCH] docs: P3.1 search filters and multi-query merge Native /v1/search knobs (domains, time, topic, exact match, min_score) plus multi-query URL dedupe. Tavily passthrough; SearXNG approximations with warnings. No crawl/map/research. Claude reviews, titan implements. --- docs/P3.1.md | 234 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 234 insertions(+) create mode 100644 docs/P3.1.md diff --git a/docs/P3.1.md b/docs/P3.1.md new file mode 100644 index 0000000..40aa43a --- /dev/null +++ b/docs/P3.1.md @@ -0,0 +1,234 @@ +# Sift P3.1 — search filters and multi-query merge + +Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook +patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan. + +P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily +fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily +client hardcodes `search_depth: "basic"` and sends no filters +(`internal/search/tavily.go`). Native search accepts only `query`, `k`, +`rerank`, `digest`. + +This phase adds the cookbook's high-leverage search knobs without taking crawl, +map, research, or `include_answer`. + +## 1. Scope + +In: + +- Native filters on `POST /v1/search` and `sift search`. +- Passing those filters through to Tavily when that source is enabled. +- Honest SearXNG approximations where the engine can do it; warnings where it cannot. +- Multi-query search: several focused queries, one URL-deduped result list. +- Optional `min_score` cut after merge. +- `published_date` on a hit when a source supplies it (Tavily news). +- Skill recipe: search → score/domain filter → `sift batch`, do not fetch every hit. + +Out: + +- `/v2/search` stays the Firecrawl contract (`query`, `limit` only). Unknown + fields ignored. No filters on v2. +- Crawl, map, Tavily `/extract` `/crawl` `/map`, `include_answer`, + `include_raw_content`, `include_images` — P3 §3.3 still holds. +- HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases. +- Changing `internal/guard/` or `internal/auth/`. +- Fetching result URLs inside search. + +## 2. Native request (`POST /v1/search`) + +Existing fields unchanged. Additive: + +```json +{ + "query": "…", + "queries": ["…", "…"], + "k": 8, + "rerank": true, + "digest": false, + "include_domains": ["openai.com"], + "exclude_domains": ["pinterest.com"], + "include_domains_mode": "restrict", + "time_range": "month", + "start_date": "2026-09-01", + "end_date": "2026-09-29", + "topic": "news", + "exact_match": false, + "min_score": 0 +} +``` + +Validation (400 `invalid query` or 400 `invalid search`): + +- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both + set, or `queries` longer than 4, is 400. Each string still has P3's 1500-rune cap. +- `include_domains` / `exclude_domains`: hostnames only (no scheme, no path). + Max 20 include, 20 exclude. Empty list = omitted. +- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without + `include_domains` → 400. +- `time_range`: omitted or `day`/`week`/`month`/`year` (and Tavily aliases + `d`/`w`/`m`/`y`, normalised to the long form). +- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` → 400. +- `topic`: omitted/`general`/`news`/`finance`. +- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`. +- `digest: true` still warns `digest ignored` (P3). + +`/v2/search` does not grow. Hermes never sends these fields. + +## 3. Response + +`/v1/search` results gain optional `"published_date"` (RFC 1123 or ISO date +string as the vendor sent it; omit the key when unknown). Do not invent dates. + +`warnings` grows with per-source honesty, e.g. `searxng: exclude_domains ignored`, +`brave_llm: domain filters ignored`. + +`sizes` unchanged. `cached` unchanged. + +## 4. Tavily client + +Still `POST` the configured URL only. Still `include_answer: false`, +`include_raw_content: false`, `include_images: false`, `search_depth: "basic"`. +Still drop inbound `answer`. Still never call `/extract`, `/crawl`, `/map`. + +When the request has filters, add only the fields that are set: + +- `include_domains`, `exclude_domains`, `include_domains_mode` +- `time_range` or `start_date`/`end_date` +- `topic` when not `general` +- `exact_match` when true +- `include_published_date: true` when `topic` is `news` or a date filter is set + +Map `results[].published_date` onto `Hit`. Count `usage.credits` as today. + +A fixture test must still prove `include_answer` is false and no extract/crawl +path exists. + +## 5. SearXNG and Brave + +Do not pretend they support Tavily's full filter set. + +**SearXNG** (`GET {url}/search`, same client/dialer as P3): + +- One `include_domains` entry: prefix the query with `site: `. +- Several: `site:a OR site:b OR …` up to 3, then warn `searxng: extra include_domains dropped`. +- `exclude_domains`: ignored + warning. +- `include_domains_mode=prefer`: ignored (site: is always restrict) + warning. +- `time_range`: pass SearXNG's `time_range` query param (`day`/`week`/`month`/`year`) + if we already talk to a SearXNG that accepts it; if the live instance 400s in + tests against the fixture, keep the warning path instead of guessing. +- `topic=news`: prefix or `!news` only if the existing SearXNG config actually + has that engine; otherwise warning `searxng: topic ignored`. Default: warning. +- `exact_match`: wrap the query in ASCII double quotes. +- `start_date`/`end_date`: ignored + warning unless SearXNG is proven to accept them. + +**Brave LLM Context**: no documented domain/date filters. Ignore all of the new +fields and add one warning `brave_llm: filters ignored` when any filter was set. +Do not add a second Brave HTTP API. + +## 6. Multi-query merge + +`queries: [q1, q2, …]` (max 4): + +1. Run the existing fan-in **once per query** (same sources, same filters, + same per-source timeout). Parallel across queries is allowed; do not exceed + the existing per-source client. A Tavily call still counts toward the daily cap; + if the cap trips mid-batch, later queries skip Tavily with the same warning. +2. Concatenate hits in query order, then **canonical-URL dedupe** (P3 `Dedupe`: + first source wins attribution; later longer snippet replaces text). +3. RRF as today, treating each `(source, query)` list as a ranked list, **or** + RRF per query then merge by URL — pick one and test it. Simpler and enough: + concatenate per-query RRF results, then `Dedupe`, then truncate to `k`. +4. Apply `min_score` after merge (hits with `Score == 0` from sources that do + not score are kept unless `min_score > 0`, in which case score-less hits are + dropped — document that SearXNG/Brave may have 0). +5. Cache key must include the normalised query list and the filter canonical + form (see §7). A single-`query` request is the list of one. + +Do not fetch any result URL. + +## 7. Cache + +P3 key is `sha256(norm(query) | sources | k | hour_bucket | caller_class)`. + +Extend with `| filters` where `filters` is a stable encoding of the §2 fields +that affect retrieval (sorted domains, mode, time_range, dates, topic, +exact_match, min_score, the query list). Same query without filters must not +hit a filtered entry. + +Still: do not cache if any attempted source failed. Still no page bodies, no +Tavily answers. + +## 8. CLI + +``` +sift search "" [--k n] [--json] + [--include-domain host]... [--exclude-domain host]... + [--domains-mode restrict|prefer] + [--time-range day|week|month|year] + [--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD] + [--topic general|news|finance] + [--exact-match] + [--min-score 0.5] + [--queries-file path] +``` + +`--queries-file` is a JSON array of 1–4 strings. It is mutually exclusive with +the positional query. Repeatable `--include-domain` / `--exclude-domain`. + +Default text output: if `published_date` is present, print it as a fourth line +under the snippet. `--json` is the native envelope. + +## 9. Skill + +`skill/SKILL.md`: + +- Document the new flags with one example matching the cookbook Product News + Tracker: official domain search + `topic=news` + `time_range=month`. +- **Recipe:** after `sift search --json`, do not `sift fetch` every URL. Drop + hits below a score you care about (or take top N), then `sift batch submit`. +- `web_search` remains the fallback only when sift is unreachable. Hermes + `web_search` still hits `/v2/search` and does not get filters — say so. + +## 10. Tests (minimum) + +- Tavily outbound JSON: filters present when set; `include_answer` always false; + `search_depth` basic; no extract/crawl. +- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted. +- Validation table for §2. +- SearXNG: `site:` prefix for one domain; exclude_domains warning; exact_match quotes. +- Brave: any filter → `brave_llm: filters ignored`, request URL unchanged. +- Multi-query: two queries, overlapping URL, one hit after Dedupe; k cap applies. +- Cache: same query different `include_domains` → miss; same filters → hit. +- `/v2/search` golden unchanged (no new keys required). +- Daily cap still skips Tavily on the Nth call across queries in one request. +- Restricted caller: same sources; logs still have no query/host/URL. + +`make gate` (and contract if P3's contract harness is still in the tree). + +## 11. Code layout + +Touch, do not explode: + +- `internal/search` — `Filter` type; extend `Source.Search`; Tavily/SearXNG/Brave +- `internal/api/search.go` — native decode + validation +- `cmd/sift/searchcmd.go` — flags +- `internal/dcache` search key +- `skill/SKILL.md`, `README.md` + +No new dependency. No `internal/api/api.go` growth if a new file is cleaner +(`internal/api/search_filter.go`). Guard and auth stay closed. + +## 12. Rollout + +1. Claude reviews this doc (filters vs P3 guard/contract; SearXNG honesty). +2. titan implements on `git.wntrmute.dev:kyle/sift` after that review. Split + commits if it helps; one phase, not a P6-style task-file plan unless the + reviewer asks for one. +3. Claude reviews the code. +4. straylight bumps the orion pin. titan does not edit `~/src/nixos`. + +## 13. Explicit non-goals (do not sneak in) + +Map, crawl, HTML BM25 chunks, Research, `include_answer`, social-platform +crawlers, changing `/v2/search`. Those were ranked later in the cookbook +write-up on purpose.