sift/docs/P3.1.md

Sift P3.1 — search filters and multi-query merge

Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.

Amended 2026-09-29 after Claude's review of 7f4f0d5 (R1–R10). Not ready for titan until Claude re-approves this SHA.

P3 shipped /v1/search + /v2/search with SearXNG / Brave LLM Context / Tavily fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily client hardcodes search_depth: "basic" and sends no filters (internal/search/tavily.go). Native search accepts only query, k, rerank, digest.

This phase adds the cookbook's high-leverage search knobs without taking crawl, map, research, or include_answer.

1. Scope

In:

• Native filters on POST /v1/search and sift search.
• Passing those filters through to Tavily when that source is enabled.
• Honest SearXNG approximations where the engine can do it; notes (not source failure) where it cannot.
• Post-merge domain enforcement in sift for restrict-mode include and exclude, for every source.
• Multi-query search: several focused queries, one URL-deduped result list, fused with RRF across per-query lists.
• Optional min_score cut on Tavily's own score, per source, before merge.
• published_date on a hit when a source supplies it (Tavily news; SearXNG publishedDate when present).
• Skill recipe: search → score/domain filter → sift batch, do not fetch every hit.

Out:

• /v2/search stays the Firecrawl contract (query, limit only). Unknown fields ignored. No filters on v2. v2 builds its own output map and must not leak published_date.
• Crawl, map, Tavily /extract /crawl /map, include_answer, include_raw_content, include_images — P3 §3.3 still holds.
• HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases.
• Changing internal/guard/ or internal/auth/.
• Fetching result URLs inside search.

Guard and contract: filters only change query text, SearXNG query params, and the Tavily JSON body. Endpoints stay config-built. Nothing from the request reaches a dialer. No guard or auth change.

2. Native request (POST /v1/search)

Existing fields unchanged. Additive:

{
  "query": "…",
  "queries": ["…", "…"],
  "k": 8,
  "rerank": true,
  "digest": false,
  "include_domains": ["openai.com"],
  "exclude_domains": ["pinterest.com"],
  "include_domains_mode": "restrict",
  "time_range": "month",
  "start_date": "2026-09-01",
  "end_date": "2026-09-29",
  "topic": "news",
  "exact_match": false,
  "min_score": 0
}

time_range and start_date/end_date are mutually exclusive (Tavily 400s if both are sent). The example above shows the fields, not a legal combination.

2.1 Error strings (tests compile against these)

• Query problems → 400, "error": "invalid query", empty hint.
• Filter problems → 400, "error": "invalid search", "hint" names the field (include_domains, exclude_domains, include_domains_mode, time_range, start_date, end_date, topic, min_score, exact_match).

2.2 Validation

• Exactly one of query (non-empty) or queries (1–4 non-empty strings). Both set, or queries longer than 4, or any empty element → invalid query. Each string still has P3's 1500-rune cap (invalid query).
• exact_match: true when the query (or any element of queries) already contains " → invalid query (do not nest quotes for SearXNG).
• include_domains / exclude_domains: hostnames only. Max 20 include, 20 exclude after normalise+dedupe. Empty list = omitted. Bad value → invalid search hint include_domains or exclude_domains.
• Domain normalisation (apply before validation and before any site: rewrite): lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode xn-- labels allowed); no scheme, port, path, wildcard, space, quote, or OR; at most 253 characters. Strict validation also stops operator injection into the SearXNG query.
• include_domains_mode: omitted/restrict/prefer. Set without include_domains → invalid search hint include_domains_mode. Omitted with include_domains present means restrict.
• time_range: omitted or day/week/month/year (and Tavily aliases d/w/m/y, normalised to the long form). Else invalid search hint time_range.
• start_date / end_date: YYYY-MM-DD or omitted. end before start → invalid search hint end_date. Either date together with time_range → invalid search hint time_range.
• topic: omitted/general/news/finance. Else invalid search hint topic.
• min_score: omitted or 0 = off; otherwise 0 < x ≤ 1. Else invalid search hint min_score.
• digest: true still warns digest ignored (P3). That warning is API-level, not a source failure.

/v2/search does not grow. Hermes never sends these fields.

3. Response and source notes

/v1/search results gain optional "published_date" (RFC 1123 or ISO date string as the vendor sent it; omit the key when unknown). Do not invent dates.

Hit grows PublishedDate string (json:"published_date,omitempty"). When Dedupe or RRF replaces a hit's text with a longer snippet, keep the first non-empty published_date. Map SearXNG publishedDate when the engine supplies it.

sizes unchanged. cached unchanged.

3.1 SourceOut.Notes (R1 — do not reuse Warn)

Today a non-empty SourceOut.Warn means failure (engine.go Run loop): the source's hits are discarded, it lands in sources_failed, it is counted in search_errors, and caching is disabled. Filter honesty must not take that path.

type SourceOut struct {
    Hits    []Hit
    Warn    string    // failure only (timeout, HTTP error, cap skip, pin)
    Notes   []string  // honesty: filters not applied, recall loss
    Credits int
}

• Warn keeps today's meaning. Hits discarded; source failed; no cache.
• Notes are appended to the response warnings (deduplicated across queries). Hits are kept; the source is in sources_ok; the result may be cached.
• Test: hits plus a note → hits kept, searxng in sources_ok, result cached.

4. Tavily client

Still POST the configured URL only. Still include_answer: false, include_raw_content: false, include_images: false, search_depth: "basic". Still drop inbound answer. Still never call /extract, /crawl, /map.

Always send auto_parameters: false. Auto mode can choose advanced, which costs 2 credits and breaks "credit cost is fixed".

When the request has filters, add only the fields that are set:

• include_domains, exclude_domains
• include_domains_mode only when include_domains is set
• time_range or start_date/end_date, never both
• topic when not general (omitted topic is general)
• exact_match when true
• include_published_date: true when topic is news or a date filter is set

Leave filter_by_published_date unset. Do not guess whether date filters use publish date without it; document that it is unset.

Leave include_usage unset. usage.credits only comes back when requested; today's count is the code's fallback of 1, which is correct for basic.

Map results[].published_date onto Hit. Count usage.credits as today (fallback 1).

A fixture test must still prove include_answer is false, auto_parameters is false, search_depth is basic, and no extract/crawl path exists.

5. SearXNG and Brave

Do not pretend they support Tavily's full filter set. Vendor gaps are notes, never Warn. Domain restrict/exclude are enforced in sift after merge (§6.3); site: is a recall optimisation only.

SearXNG (GET {url}/search, same client/dialer as P3):

• One include_domains entry: prefix the query with site:<host> .
• Several: site:a OR site:b OR … up to 3, then note searxng: extra include_domains dropped.
• exclude_domains: no query rewrite (post-filter enforces). No note unless recall is otherwise lost.
• include_domains_mode=prefer: no site: prefix, no post-filter; note searxng: include_domains_mode=prefer not applied.
• time_range: send SearXNG query param time_range= (day/week/month/year). Never a bang in the query text.
• topic=news: send categories=news. Never !news.
• topic=finance: note searxng: topic ignored (no finance category).
• exact_match: wrap the query in ASCII double quotes (rejected earlier if the query already contains ").
• start_date/end_date: ignored + note searxng: start_date/end_date not applied.
• min_score > 0: note searxng: min_score not applied; keep hits.

Live instance: straylight probes orion's SearXNG once before titan starts and posts whether time_range= and categories=news return 200. If either 400s, this spec is amended to the note path for that param and titan must not send it. Tests are offline fixtures and cannot discover this.

Brave LLM Context: no documented domain/date/topic/exact filters. Do not add a second Brave HTTP API. Post-filter still enforces restrict include and exclude. Notes when the corresponding filter was set:

• brave_llm: time_range not applied
• brave_llm: topic not applied
• brave_llm: exact_match not applied
• brave_llm: include_domains_mode=prefer not applied
• brave_llm: min_score not applied

Request URL unchanged aside from q (and exact-match quotes if we ever wrap; we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes across queries.

6. Pipeline

Source.Search grows a Filter argument (zero value = today's behaviour).

6.1 Extra queries run sequentially per source (R5)

Sources still run in parallel within one query, as today. Queries run sequentially. Up to 4 queries means up to 4 Brave and 4 Tavily paid calls; four parallel Brave calls will hit the plan's rate limit, and a 429 marks brave failed (Warn) and disables caching.

A Tavily call still counts toward the daily cap. If the cap trips mid-batch, later queries skip Tavily with the same tavily: daily cap warning (skip path, not a failed attempted source). The queries then ran against different source sets. Do not cache that result (the key's sources would be wrong). Test this.

6.2 min_score (R2)

After merge, Hit.Score is the RRF value Σ 1/(60+rank). With 3 sources the maximum is about 0.05, so a post-merge --min-score 0.5 would drop everything. With rerank:false scores are vendor-native (Tavily 0–1, SearXNG unbounded, Brave 0). With Tavily no_rerank, Tavily scores mix with RRF.

Apply min_score per source, before the merge, to Tavily's own relevance score only. That is the cookbook's usage. Other sources get a note (searxng: min_score not applied, brave_llm: min_score not applied) and keep their hits. Do not drop score-less Brave/SearXNG hits.

6.3 Domain post-filter (R3)

After each query's merge, for every source, drop hits that violate:

• include_domains when mode is restrict (omitted mode = restrict)
• exclude_domains (always)

Match is exact on the result host: host == d or host has suffix "." + d (host lowercased, trailing dot stripped). Cost is negligible.

• prefer mode: no post-filter. Tavily honours include_domains_mode; other sources get the prefer note.
• site: is a recall optimisation only. Notes are for recall loss, not for "filter ignored" when the post-filter already enforced it.
• As written without this, Brave ignores the filter entirely and SearXNG engines that do not understand site: return off-domain hits.

6.4 Multi-query merge (R4)

Do not concatenate-then-truncate: if q1 returns k or more hits, q2–q4 contribute nothing.

For queries: [q1, q2, …] (max 4), or a single query treated as a list of one:

1. For each query, sequentially: existing fan-in (parallel sources, same filters, same per-source timeout) → Tavily min_score → today's mergeSearch (RRF / no-rerank path) → domain post-filter.
2. RRF across the per-query fused lists (each query's list is one ranked list). URLs several queries found rise.
3. Dedupe (P3: first source wins attribution; later longer snippet replaces text; keep first non-empty published_date).
4. Truncate to k.

Test: q1 returns k or more unique hits, and q2's single distinct top hit still appears.

Do not fetch any result URL.

7. Cache

P3 key is sha256(norm(query) | sources | k | hour_bucket | caller_class).

Replace the payload with a canonical JSON object, keys sorted, then sha256. It must hold:

• caller_class, hour bucket (2006-01-02T15 UTC), k, attempted sources (sorted) — exactly as P3 has them
• queries: normalised, ordered list (a single query is a one-element list)
• include_domains / exclude_domains: normalised, sorted, deduplicated
• include_domains_mode (restrict or prefer; omit the key when no include list)
• time_range in long form (d → day); dates as YYYY-MM-DD or omit
• topic, with omitted meaning general so the two share a key
• exact_match (bool)
• min_score in one fixed number format (e.g. '%g' is not stable — use strconv.FormatFloat(x, 'f', -1, 64) or a documented equivalent, same for 0 and omitted)

Same query without filters must not hit a filtered entry.

Still: do not cache if any attempted source failed (Warn). Still do not cache if the Tavily cap tripped partway through a multi-query run. Still no page bodies, no Tavily answers.

8. Logs

P3 §8 already bans hosts in the search log for every caller, and filter values are hosts or query content.

• Never log domain values, query text, dates, or topic values.
• Restricted callers keep exactly P3's fields: caller, q_len, sources, n, cache.
• Fleet callers may add n_queries and the filter kinds (filters=domains,time,topic), never values. rerank_ms stays as today.
• q_len for several queries: sum of their rune lengths.

Test: restricted and fleet requests with include_domains: ["secret.example"] and 3 queries → no log record contains the host or any query.

9. CLI

sift search "<query>" [--k n] [--json]
            [--include-domain host]... [--exclude-domain host]...
            [--domains-mode restrict|prefer]
            [--time-range day|week|month|year]
            [--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD]
            [--topic general|news|finance]
            [--exact-match]
            [--min-score 0.5]
            [--queries-file path]

--min-score 0.5 is Tavily's 0–1 scale (applied only to Tavily, §6.2).

--queries-file is a JSON array of 1–4 strings. It is mutually exclusive with the positional query. Repeatable --include-domain / --exclude-domain.

Default text output: if published_date is present, print it as a fourth line under the snippet. --json is the native envelope.

10. Skill

skill/SKILL.md:

• Document the new flags with one example matching the cookbook Product News Tracker: official domain search + topic=news + time_range=month.
• Recipe: after sift search --json, do not sift fetch every URL. Drop hits below a Tavily score you care about (or take top N), then sift batch submit.
• web_search remains the fallback only when sift is unreachable. Hermes web_search still hits /v2/search and does not get filters — say so.

11. Tests (minimum)

Owner-given invariants (these are where a free-form implementation drifts):

• R1 notes: hits plus a note → hits kept, source in sources_ok, result cached. A Warn still fails the source and disables cache.
• R2 min_score: Tavily hit at 0.4 dropped when min_score=0.5; SearXNG / Brave hits kept; notes present. Post-merge RRF scores are not compared to min_score.
• R3 domain post-filter: Brave (and SearXNG) hit whose host is not in restrict-include, or is in exclude, is dropped. prefer keeps it. Invalid domain with a space or OR → 400 invalid search.
• R4 multi-query RRF: q1 returns k unique hits; q2's distinct top hit still appears.
• R5 cap mid-batch: Tavily cap trips after query 1 → later queries skip Tavily; result is not cached.
• R6 cache key: same query different include_domains → miss; same filters → hit; omitted topic and topic=general share a key; d and day share a key.
• R9 logs: restricted and fleet with include_domains:["secret.example"] and 3 queries → no log record contains the host or any query.
• Tavily outbound JSON: filters present when set; include_answer always false; auto_parameters false; search_depth basic; no extract/crawl; include_domains_mode absent unless include list set; never both time_range and dates.
• Tavily inbound: answer dropped; published_date mapped; missing date omitted.
• Validation table for §2 (invalid query vs invalid search + hint).
• SearXNG: site: prefix for one domain; exact_match quotes; existing " → 400 invalid query; time_range / categories as query params not bangs.
• Brave: request URL unchanged (aside from q); notes as §5; post-filter still drops off-domain hits.
• /v2/search golden unchanged (no new keys required, no published_date).
• Restricted caller: same sources; logs still have no query/host/URL.

make gate (and contract if P3's contract harness is still in the tree).

12. Code layout

Touch, do not explode:

• internal/search — Filter type; SourceOut.Notes; extend Source.Search; Tavily/SearXNG/Brave; engine loop; domain post-filter; multi-query RRF
• internal/api/search.go — native decode + validation + log fields
• cmd/sift/searchcmd.go — flags
• internal/dcache search key
• skill/SKILL.md, README.md

No new dependency. No internal/api/api.go growth if a new file is cleaner (internal/api/search_filter.go). Guard and auth stay closed.

13. Rollout

1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed).
2. straylight probes orion SearXNG time_range= and categories=news once and posts the status codes in ~agent. If either 400s, amend §5 to the note path before titan starts.
3. titan implements on git.wntrmute.dev:kyle/sift after Claude's re-approval and the SearXNG probe. Split commits if it helps; one phase doc, not a P6-style task-file plan (invariants are in §11). titan does not edit internal/guard/, internal/auth/, or ~/src/nixos.
4. Claude reviews the code.
5. straylight bumps the orion pin.

14. Explicit non-goals (do not sneak in)

Map, crawl, HTML BM25 chunks, Research, include_answer, social-platform crawlers, changing /v2/search. Those were ranked later in the cookbook write-up on purpose.