sift/docs/P3.1.md

# Sift P3.1 — search filters and multi-query merge

Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.

Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10) and of `73ff7b6`
(E1–E3). Spec approved; titan implements this SHA.

P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
client hardcodes `search_depth: "basic"` and sends no filters
(`internal/search/tavily.go`). Native search accepts only `query`, `k`,
`rerank`, `digest`.

This phase adds the cookbook's high-leverage search knobs without taking crawl,
map, research, or `include_answer`.

## 1. Scope

In:

- Native filters on `POST /v1/search` and `sift search`.
- Passing those filters through to Tavily when that source is enabled.
- Honest SearXNG approximations where the engine can do it; **notes** (not
  source failure) where it cannot.
- Post-merge domain enforcement in sift for restrict-mode include and exclude,
  for every source.
- Multi-query search: several focused queries, one URL-deduped result list,
  fused with RRF across per-query lists.
- Optional `min_score` cut on **Tavily's own score**, per source, before merge.
- `published_date` on a hit when a source supplies it (Tavily news; SearXNG
  `publishedDate` when present).
- Skill recipe: search → score/domain filter → `sift batch`, do not fetch every hit.

Out:

- `/v2/search` stays the Firecrawl contract (`query`, `limit` only). Unknown
  fields ignored. No filters on v2. v2 builds its own output map and must not
  leak `published_date`.
- Crawl, map, Tavily `/extract` `/crawl` `/map`, `include_answer`,
  `include_raw_content`, `include_images` — P3 §3.3 still holds.
- HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases.
- Changing `internal/guard/` or `internal/auth/`.
- Fetching result URLs inside search.

Guard and contract: filters only change query text, SearXNG query params, and
the Tavily JSON body. Endpoints stay config-built. Nothing from the request
reaches a dialer. No guard or auth change.

## 2. Native request (`POST /v1/search`)

Existing fields unchanged. Additive:

```json
{
  "query": "…",
  "queries": ["…", "…"],
  "k": 8,
  "rerank": true,
  "digest": false,
  "include_domains": ["openai.com"],
  "exclude_domains": ["pinterest.com"],
  "include_domains_mode": "restrict",
  "time_range": "month",
  "start_date": "2026-09-01",
  "end_date": "2026-09-29",
  "topic": "news",
  "exact_match": false,
  "min_score": 0
}
```

`time_range` and `start_date`/`end_date` are mutually exclusive (Tavily 400s if
both are sent). The example above shows the fields, not a legal combination.

### 2.1 Error strings (tests compile against these)

- Query problems → 400, `"error": "invalid query"`, empty hint.
- Filter problems → 400, `"error": "invalid search"`, `"hint"` names the field
  (`include_domains`, `exclude_domains`, `include_domains_mode`, `time_range`,
  `start_date`, `end_date`, `topic`, `min_score`, `exact_match`).

### 2.2 Validation

- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
  set, or `queries` longer than 4, or any empty element → `invalid query`.
  Each string still has P3's 1500-rune cap (`invalid query`). ASCII `"` is
  allowed in queries (agents quote phrases themselves).
- `exact_match: true` requires at least one balanced quoted phrase in **each**
  query (a `"`…`"` pair with one or more runes between; no unpaired `"`).
  Otherwise `invalid search` hint `exact_match`. Do not wrap or rewrite the
  query. `exact_match: false`/omitted: quotes are ordinary query text.
- `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20
  exclude after normalise+dedupe. Empty list = omitted. Bad value →
  `invalid search` hint `include_domains` or `exclude_domains`.
- Domain normalisation (apply before validation and before any `site:` rewrite):
  lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode
  `xn--` labels allowed); no scheme, port, path, wildcard, or space; at most
  253 characters. Letters/digits/hyphens already make operator injection
  impossible (that needs a space). Do **not** substring-reject `OR` —
  `oracle.com` is valid. A space, e.g. `a.com OR b.com`, → 400.
- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
  `include_domains` → `invalid search` hint `include_domains_mode`.
  Omitted with `include_domains` present means `restrict`.
- `time_range`: omitted or `day`/`week`/`month`/`year` (and Tavily aliases
  `d`/`w`/`m`/`y`, normalised to the long form). Else `invalid search` hint
  `time_range`.
- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` →
  `invalid search` hint `end_date`. Either date together with `time_range` →
  `invalid search` hint `time_range`.
- `topic`: omitted/`general`/`news`/`finance`. Else `invalid search` hint `topic`.
- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`. Else `invalid search`
  hint `min_score`.
- `digest: true` still warns `digest ignored` (P3). That warning is API-level,
  not a source failure.

`/v2/search` does not grow. Hermes never sends these fields.

## 3. Response and source notes

`/v1/search` results gain optional `"published_date"` (RFC 1123 or ISO date
string as the vendor sent it; omit the key when unknown). Do not invent dates.

`Hit` grows `PublishedDate string` (`json:"published_date,omitempty"`). When
Dedupe or RRF replaces a hit's text with a longer snippet, keep the first
non-empty `published_date`. Map SearXNG `publishedDate` when the engine
supplies it.

`sizes` unchanged. `cached` unchanged.

### 3.1 `SourceOut.Notes` (R1 — do not reuse `Warn`)

Today a non-empty `SourceOut.Warn` means failure (`engine.go` Run loop): the
source's hits are discarded, it lands in `sources_failed`, it is counted in
`search_errors`, and caching is disabled. Filter honesty must not take that path.

```
type SourceOut struct {
    Hits    []Hit
    Warn    string    // failure only (timeout, HTTP error, pin)
    Notes   []string  // honesty: filters not applied, recall loss
    Credits int
}
```

- `Warn` keeps today's meaning. Hits discarded; source failed; no cache.
- `Notes` are appended to the response `warnings` (deduplicated across queries).
  Hits are kept; the source is in `sources_ok`; the result may be cached.
- Test: hits plus a note → hits kept, searxng in `sources_ok`, result cached.

## 4. Tavily client

Still `POST` the configured URL only. Still `include_answer: false`,
`include_raw_content: false`, `include_images: false`, `search_depth: "basic"`.
Still drop inbound `answer`. Still never call `/extract`, `/crawl`, `/map`.

Always send `auto_parameters: false`. Auto mode can choose `advanced`, which
costs 2 credits and breaks "credit cost is fixed".

When the request has filters, add only the fields that are set:

- `include_domains`, `exclude_domains`
- `include_domains_mode` **only when** `include_domains` is set
- `time_range` **or** `start_date`/`end_date`, never both
- `topic` when not `general` (omitted `topic` is `general`)
- `exact_match: true` when the flag is set (query text unchanged, including
  the agent's quotes)
- `include_published_date: true` when `topic` is `news` or a date filter is set

Leave `filter_by_published_date` unset. Tavily's `time_range` and dates
filter on publish date **or last updated date**; `filter_by_published_date`
also drops undated results. Unset means undated and recently updated pages
can pass the date filter.

Leave `include_usage` unset. `usage.credits` only comes back when requested;
today's count is the code's fallback of 1, which is correct for `basic`.

Map `results[].published_date` onto `Hit`. Count `usage.credits` as today
(fallback 1).

A fixture test must still prove `include_answer` is false, `auto_parameters` is
false, `search_depth` is `basic`, and no extract/crawl path exists.

## 5. SearXNG and Brave

Do not pretend they support Tavily's full filter set. Vendor gaps are **notes**,
never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3);
`site:` is a recall optimisation only.

**SearXNG** (`GET {url}/search`, same client/dialer as P3):

- One `include_domains` entry: prefix the query with `site:<host> `.
- Several: `site:a OR site:b OR …` up to 3, then note
  `searxng: extra include_domains dropped`.
- `exclude_domains`: no query rewrite (post-filter enforces). No note unless
  recall is otherwise lost.
- `include_domains_mode=prefer`: no `site:` prefix, no post-filter; note
  `searxng: include_domains_mode=prefer not applied`.
- `time_range`: send SearXNG query param `time_range=` (`day`/`week`/`month`/`year`).
  Never a bang in the query text.
- `topic=news`: send `categories=news`. Never `!news`.
- `topic=finance`: note `searxng: topic ignored` (no finance category).
- `exact_match`: send the query unchanged. SearXNG engines already treat
  quotes as phrases. Do not wrap again.
- `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`.
- `min_score > 0`: note `searxng: min_score not applied`; keep hits.

Live instance (straylight, 2026-09-29, not independently verified here):
`time_range=month` and `categories=news` both returned 200 with normal result
lists. Send both params. Tests are offline fixtures.

**Brave LLM Context**: no documented domain/date/topic/exact filters. Do not
add a second Brave HTTP API. Post-filter still enforces restrict include and
exclude. Notes when the corresponding filter was set:

- `brave_llm: time_range not applied`
- `brave_llm: topic not applied`
- `brave_llm: exact_match not applied`
- `brave_llm: include_domains_mode=prefer not applied`
- `brave_llm: min_score not applied`

Request URL unchanged aside from `q` (do not wrap for `exact_match`; Brave
cannot apply it). Deduplicate identical notes across queries.

## 6. Pipeline

`Source.Search` grows a `Filter` argument (zero value = today's behaviour).

### 6.1 Extra queries run sequentially per source (R5)

Sources still run in parallel **within** one query, as today. Queries run
**sequentially**. Up to 4 queries means up to 4 Brave and 4 Tavily paid calls;
four parallel Brave calls will hit the plan's rate limit, and a 429 marks
brave failed (`Warn`) and disables caching.

A Tavily call still counts toward the daily cap. If the cap trips mid-batch,
later queries skip Tavily with the same `tavily: daily cap` warning (skip path,
not a failed attempted source). The queries then ran against different source
sets. **Do not cache** that result (the key's `sources` would be wrong). Test
this.

### 6.2 `min_score` (R2)

After merge, `Hit.Score` is the RRF value `Σ 1/(60+rank)`. With 3 sources the
maximum is about 0.05, so a post-merge `--min-score 0.5` would drop everything.
With `rerank:false` scores are vendor-native (Tavily 0–1, SearXNG unbounded,
Brave 0). With Tavily `no_rerank`, Tavily scores mix with RRF.

Apply `min_score` **per source, before the merge, to Tavily's own relevance
score only**. That is the cookbook's usage. Other sources get a note
(`searxng: min_score not applied`, `brave_llm: min_score not applied`) and keep
their hits. Do not drop score-less Brave/SearXNG hits.

### 6.3 Domain post-filter (R3)

After each query's merge, for **every** source, drop hits that violate:

- `include_domains` when mode is `restrict` (omitted mode = restrict)
- `exclude_domains` (always)

Match is exact on the result host: `host == d` or `host` has suffix `"." + d`
(host lowercased, trailing dot stripped). Cost is negligible.

- `prefer` mode: **no** post-filter. Tavily honours `include_domains_mode`;
  other sources get the prefer note.
- `site:` is a recall optimisation only. Notes are for recall loss, not for
  "filter ignored" when the post-filter already enforced it.
- As written without this, Brave ignores the filter entirely and SearXNG
  engines that do not understand `site:` return off-domain hits.

### 6.4 Multi-query merge (R4)

Do not concatenate-then-truncate: if q1 returns `k` or more hits, q2–q4
contribute nothing.

For `queries: [q1, q2, …]` (max 4), or a single `query` treated as a list of one:

1. For each query, sequentially: existing fan-in (parallel sources, same
   filters, same per-source timeout) → Tavily `min_score` → today's
   `mergeSearch` (RRF / no-rerank path) → domain post-filter.
2. **RRF across the per-query fused lists** (each query's list is one ranked
   list). URLs several queries found rise.
3. Dedupe (P3: first source wins attribution; later longer snippet replaces
   text; keep first non-empty `published_date`).
4. Truncate to `k`.

Test: q1 returns `k` or more unique hits, and q2's single distinct top hit
still appears.

Do not fetch any result URL.

## 7. Cache

P3 key is `sha256(norm(query) | sources | k | hour_bucket | caller_class)`.

Replace the payload with a canonical JSON object, keys sorted, then sha256.
It must hold:

- `caller_class`, hour bucket (`2006-01-02T15` UTC), `k`, attempted `sources`
  (sorted) — exactly as P3 has them
- `queries`: normalised, ordered list (a single `query` is a one-element list)
- `include_domains` / `exclude_domains`: normalised, sorted, deduplicated
- `include_domains_mode` (`restrict` or `prefer`; omit the key when no include
  list)
- `time_range` in long form (`d` → `day`); dates as `YYYY-MM-DD` or omit
- `topic`, with omitted meaning `general` so the two share a key
- `exact_match` (bool)
- `min_score` in one fixed number format (e.g. `'%g'` is not stable — use
  `strconv.FormatFloat(x, 'f', -1, 64)` or a documented equivalent, same for
  0 and omitted)

Same query without filters must not hit a filtered entry.

Still: do not cache if any attempted source failed (`Warn`). Still do not cache
if the Tavily cap tripped partway through a multi-query run. Still no page
bodies, no Tavily answers.

## 8. Logs

P3 §8 already bans hosts in the search log for every caller, and filter values
are hosts or query content.

- Never log domain values, query text, dates, or topic values.
- Restricted callers keep **exactly** P3's fields: `caller`, `q_len`, `sources`,
  `n`, `cache`.
- Fleet callers may add `n_queries` and the filter *kinds*
  (`filters=domains,time,topic`), never values. `rerank_ms` stays as today.
- `q_len` for several queries: sum of their rune lengths.

Test: restricted and fleet requests with `include_domains: ["secret.example"]`
and 3 queries → no log record contains the host or any query.

## 9. CLI

```
sift search "<query>" [--k n] [--json]
            [--include-domain host]... [--exclude-domain host]...
            [--domains-mode restrict|prefer]
            [--time-range day|week|month|year]
            [--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD]
            [--topic general|news|finance]
            [--exact-match]
            [--min-score 0.5]
            [--queries-file path]
```

`--min-score 0.5` is Tavily's 0–1 scale (applied only to Tavily, §6.2).

`--queries-file` is a JSON array of 1–4 strings. It is mutually exclusive with
the positional query. Repeatable `--include-domain` / `--exclude-domain`.

Default text output: if `published_date` is present, print it as a fourth line
under the snippet. `--json` is the native envelope.

## 10. Skill

`skill/SKILL.md`:

- Document the new flags with one example matching the cookbook Product News
  Tracker: official domain search + `topic=news` + `time_range=month`.
- **Recipe:** after `sift search --json`, do not `sift fetch` every URL. Drop
  hits below a Tavily score you care about (or take top N), then `sift batch submit`.
- `web_search` remains the fallback only when sift is unreachable. Hermes
  `web_search` still hits `/v2/search` and does not get filters — say so.

## 11. Tests (minimum)

Owner-given invariants (these are where a free-form implementation drifts):

- **R1 notes:** hits plus a note → hits kept, source in `sources_ok`, result
  cached. A `Warn` still fails the source and disables cache.
- **R2 min_score:** Tavily hit at 0.4 dropped when `min_score=0.5`; SearXNG /
  Brave hits kept; notes present. Post-merge RRF scores are not compared to
  `min_score`.
- **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in
  restrict-include, or is in exclude, is dropped. `prefer` keeps it.
  Invalid domain with a space, e.g. `a.com OR b.com`, → 400 `invalid search`.
  `oracle.com` is valid.
- **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still
  appears.
- **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip
  Tavily; result is not cached.
- **R6 cache key:** same query different `include_domains` → miss; same
  filters → hit; omitted `topic` and `topic=general` share a key; `d` and
  `day` share a key.
- **R9 logs:** restricted and fleet with `include_domains:["secret.example"]`
  and 3 queries → no log record contains the host or any query.
- Tavily outbound JSON: filters present when set; `include_answer` always
  false; `auto_parameters` false; `search_depth` basic; no extract/crawl;
  `include_domains_mode` absent unless include list set; never both
  `time_range` and dates; `exact_match: true` with query containing `"phrase"`
  → flag true and query unchanged (still quoted); `exact_match: true` with no
  quoted phrase → 400 `invalid search` hint `exact_match`.
- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
- Validation table for §2 (`invalid query` vs `invalid search` + hint).
- SearXNG: `site:` prefix for one domain; `exact_match` does not wrap `q`
  (query already quoted by the agent); `time_range` / `categories` as query
  params not bangs.
- Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still
  drops off-domain hits.
- `/v2/search` golden unchanged (no new keys required, no `published_date`).
- Restricted caller: same sources; logs still have no query/host/URL.

`make gate` (and contract if P3's contract harness is still in the tree).

## 12. Code layout

Touch, do not explode:

- `internal/search` — `Filter` type; `SourceOut.Notes`; extend `Source.Search`;
  Tavily/SearXNG/Brave; engine loop; domain post-filter; multi-query RRF
- `internal/api/search.go` — native decode + validation + log fields
- `cmd/sift/searchcmd.go` — flags
- `internal/dcache` search key
- `skill/SKILL.md`, `README.md`

No new dependency. No `internal/api/api.go` growth if a new file is cleaner
(`internal/api/search_filter.go`). Guard and auth stay closed.

## 13. Rollout

1. Claude approved `73ff7b6` once E1–E3 are in (this SHA). No further spec round.
2. SearXNG probe already returned 200 for `time_range=` and `categories=news`.
3. titan implements on `git.wntrmute.dev:kyle/sift` from this SHA. Split commits
   if it helps; one phase doc (invariants in §11 — titan writes those tests;
   Claude checks them in the code review). titan does not edit
   `internal/guard/`, `internal/auth/`, or `~/src/nixos`.
4. Claude reviews the code.
5. straylight bumps the orion pin.

## 14. Explicit non-goals (do not sneak in)

Map, crawl, HTML BM25 chunks, Research, `include_answer`, social-platform
crawlers, changing `/v2/search`. Those were ranked later in the cookbook
write-up on purpose.