sift/docs/P3.1.md

Sift P3.1 — search filters and multi-query merge

Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.

P3 shipped /v1/search + /v2/search with SearXNG / Brave LLM Context / Tavily fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily client hardcodes search_depth: "basic" and sends no filters (internal/search/tavily.go). Native search accepts only query, k, rerank, digest.

This phase adds the cookbook's high-leverage search knobs without taking crawl, map, research, or include_answer.

1. Scope

In:

• Native filters on POST /v1/search and sift search.
• Passing those filters through to Tavily when that source is enabled.
• Honest SearXNG approximations where the engine can do it; warnings where it cannot.
• Multi-query search: several focused queries, one URL-deduped result list.
• Optional min_score cut after merge.
• published_date on a hit when a source supplies it (Tavily news).
• Skill recipe: search → score/domain filter → sift batch, do not fetch every hit.

Out:

• /v2/search stays the Firecrawl contract (query, limit only). Unknown fields ignored. No filters on v2.
• Crawl, map, Tavily /extract /crawl /map, include_answer, include_raw_content, include_images — P3 §3.3 still holds.
• HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases.
• Changing internal/guard/ or internal/auth/.
• Fetching result URLs inside search.

2. Native request (POST /v1/search)

Existing fields unchanged. Additive:

{
  "query": "…",
  "queries": ["…", "…"],
  "k": 8,
  "rerank": true,
  "digest": false,
  "include_domains": ["openai.com"],
  "exclude_domains": ["pinterest.com"],
  "include_domains_mode": "restrict",
  "time_range": "month",
  "start_date": "2026-09-01",
  "end_date": "2026-09-29",
  "topic": "news",
  "exact_match": false,
  "min_score": 0
}

Validation (400 invalid query or 400 invalid search):

• Exactly one of query (non-empty) or queries (1–4 non-empty strings). Both set, or queries longer than 4, is 400. Each string still has P3's 1500-rune cap.
• include_domains / exclude_domains: hostnames only (no scheme, no path). Max 20 include, 20 exclude. Empty list = omitted.
• include_domains_mode: omitted/restrict/prefer. Set without include_domains → 400.
• time_range: omitted or day/week/month/year (and Tavily aliases d/w/m/y, normalised to the long form).
• start_date / end_date: YYYY-MM-DD or omitted. end before start → 400.
• topic: omitted/general/news/finance.
• min_score: omitted or 0 = off; otherwise 0 < x ≤ 1.
• digest: true still warns digest ignored (P3).

/v2/search does not grow. Hermes never sends these fields.

3. Response

/v1/search results gain optional "published_date" (RFC 1123 or ISO date string as the vendor sent it; omit the key when unknown). Do not invent dates.

warnings grows with per-source honesty, e.g. searxng: exclude_domains ignored, brave_llm: domain filters ignored.

sizes unchanged. cached unchanged.

4. Tavily client

Still POST the configured URL only. Still include_answer: false, include_raw_content: false, include_images: false, search_depth: "basic". Still drop inbound answer. Still never call /extract, /crawl, /map.

When the request has filters, add only the fields that are set:

• include_domains, exclude_domains, include_domains_mode
• time_range or start_date/end_date
• topic when not general
• exact_match when true
• include_published_date: true when topic is news or a date filter is set

Map results[].published_date onto Hit. Count usage.credits as today.

A fixture test must still prove include_answer is false and no extract/crawl path exists.

5. SearXNG and Brave

Do not pretend they support Tavily's full filter set.

SearXNG (GET {url}/search, same client/dialer as P3):

• One include_domains entry: prefix the query with site:<host> .
• Several: site:a OR site:b OR … up to 3, then warn searxng: extra include_domains dropped.
• exclude_domains: ignored + warning.
• include_domains_mode=prefer: ignored (site: is always restrict) + warning.
• time_range: pass SearXNG's time_range query param (day/week/month/year) if we already talk to a SearXNG that accepts it; if the live instance 400s in tests against the fixture, keep the warning path instead of guessing.
• topic=news: prefix or !news only if the existing SearXNG config actually has that engine; otherwise warning searxng: topic ignored. Default: warning.
• exact_match: wrap the query in ASCII double quotes.
• start_date/end_date: ignored + warning unless SearXNG is proven to accept them.

Brave LLM Context: no documented domain/date filters. Ignore all of the new fields and add one warning brave_llm: filters ignored when any filter was set. Do not add a second Brave HTTP API.

6. Multi-query merge

queries: [q1, q2, …] (max 4):

1. Run the existing fan-in once per query (same sources, same filters, same per-source timeout). Parallel across queries is allowed; do not exceed the existing per-source client. A Tavily call still counts toward the daily cap; if the cap trips mid-batch, later queries skip Tavily with the same warning.
2. Concatenate hits in query order, then canonical-URL dedupe (P3 Dedupe: first source wins attribution; later longer snippet replaces text).
3. RRF as today, treating each (source, query) list as a ranked list, or RRF per query then merge by URL — pick one and test it. Simpler and enough: concatenate per-query RRF results, then Dedupe, then truncate to k.
4. Apply min_score after merge (hits with Score == 0 from sources that do not score are kept unless min_score > 0, in which case score-less hits are dropped — document that SearXNG/Brave may have 0).
5. Cache key must include the normalised query list and the filter canonical form (see §7). A single-query request is the list of one.

Do not fetch any result URL.

7. Cache

P3 key is sha256(norm(query) | sources | k | hour_bucket | caller_class).

Extend with | filters where filters is a stable encoding of the §2 fields that affect retrieval (sorted domains, mode, time_range, dates, topic, exact_match, min_score, the query list). Same query without filters must not hit a filtered entry.

Still: do not cache if any attempted source failed. Still no page bodies, no Tavily answers.

8. CLI

sift search "<query>" [--k n] [--json]
            [--include-domain host]... [--exclude-domain host]...
            [--domains-mode restrict|prefer]
            [--time-range day|week|month|year]
            [--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD]
            [--topic general|news|finance]
            [--exact-match]
            [--min-score 0.5]
            [--queries-file path]

--queries-file is a JSON array of 1–4 strings. It is mutually exclusive with the positional query. Repeatable --include-domain / --exclude-domain.

Default text output: if published_date is present, print it as a fourth line under the snippet. --json is the native envelope.

9. Skill

skill/SKILL.md:

• Document the new flags with one example matching the cookbook Product News Tracker: official domain search + topic=news + time_range=month.
• Recipe: after sift search --json, do not sift fetch every URL. Drop hits below a score you care about (or take top N), then sift batch submit.
• web_search remains the fallback only when sift is unreachable. Hermes web_search still hits /v2/search and does not get filters — say so.

10. Tests (minimum)

• Tavily outbound JSON: filters present when set; include_answer always false; search_depth basic; no extract/crawl.
• Tavily inbound: answer dropped; published_date mapped; missing date omitted.
• Validation table for §2.
• SearXNG: site: prefix for one domain; exclude_domains warning; exact_match quotes.
• Brave: any filter → brave_llm: filters ignored, request URL unchanged.
• Multi-query: two queries, overlapping URL, one hit after Dedupe; k cap applies.
• Cache: same query different include_domains → miss; same filters → hit.
• /v2/search golden unchanged (no new keys required).
• Daily cap still skips Tavily on the Nth call across queries in one request.
• Restricted caller: same sources; logs still have no query/host/URL.

make gate (and contract if P3's contract harness is still in the tree).

11. Code layout

Touch, do not explode:

• internal/search — Filter type; extend Source.Search; Tavily/SearXNG/Brave
• internal/api/search.go — native decode + validation
• cmd/sift/searchcmd.go — flags
• internal/dcache search key
• skill/SKILL.md, README.md

No new dependency. No internal/api/api.go growth if a new file is cleaner (internal/api/search_filter.go). Guard and auth stay closed.

12. Rollout

1. Claude reviews this doc (filters vs P3 guard/contract; SearXNG honesty).
2. titan implements on git.wntrmute.dev:kyle/sift after that review. Split commits if it helps; one phase, not a P6-style task-file plan unless the reviewer asks for one.
3. Claude reviews the code.
4. straylight bumps the orion pin. titan does not edit ~/src/nixos.

13. Explicit non-goals (do not sneak in)

Map, crawl, HTML BM25 chunks, Research, include_answer, social-platform crawlers, changing /v2/search. Those were ranked later in the cookbook write-up on purpose.