sift/docs/P3.1.md

# Sift P3.1 — search filters and multi-query merge

Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.

P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
client hardcodes `search_depth: "basic"` and sends no filters
(`internal/search/tavily.go`). Native search accepts only `query`, `k`,
`rerank`, `digest`.

This phase adds the cookbook's high-leverage search knobs without taking crawl,
map, research, or `include_answer`.

## 1. Scope

In:

- Native filters on `POST /v1/search` and `sift search`.
- Passing those filters through to Tavily when that source is enabled.
- Honest SearXNG approximations where the engine can do it; warnings where it cannot.
- Multi-query search: several focused queries, one URL-deduped result list.
- Optional `min_score` cut after merge.
- `published_date` on a hit when a source supplies it (Tavily news).
- Skill recipe: search → score/domain filter → `sift batch`, do not fetch every hit.

Out:

- `/v2/search` stays the Firecrawl contract (`query`, `limit` only). Unknown
  fields ignored. No filters on v2.
- Crawl, map, Tavily `/extract` `/crawl` `/map`, `include_answer`,
  `include_raw_content`, `include_images` — P3 §3.3 still holds.
- HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases.
- Changing `internal/guard/` or `internal/auth/`.
- Fetching result URLs inside search.

## 2. Native request (`POST /v1/search`)

Existing fields unchanged. Additive:

```json
{
  "query": "…",
  "queries": ["…", "…"],
  "k": 8,
  "rerank": true,
  "digest": false,
  "include_domains": ["openai.com"],
  "exclude_domains": ["pinterest.com"],
  "include_domains_mode": "restrict",
  "time_range": "month",
  "start_date": "2026-09-01",
  "end_date": "2026-09-29",
  "topic": "news",
  "exact_match": false,
  "min_score": 0
}
```

Validation (400 `invalid query` or 400 `invalid search`):

- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
  set, or `queries` longer than 4, is 400. Each string still has P3's 1500-rune cap.
- `include_domains` / `exclude_domains`: hostnames only (no scheme, no path).
  Max 20 include, 20 exclude. Empty list = omitted.
- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
  `include_domains` → 400.
- `time_range`: omitted or `day`/`week`/`month`/`year` (and Tavily aliases
  `d`/`w`/`m`/`y`, normalised to the long form).
- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` → 400.
- `topic`: omitted/`general`/`news`/`finance`.
- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`.
- `digest: true` still warns `digest ignored` (P3).

`/v2/search` does not grow. Hermes never sends these fields.

## 3. Response

`/v1/search` results gain optional `"published_date"` (RFC 1123 or ISO date
string as the vendor sent it; omit the key when unknown). Do not invent dates.

`warnings` grows with per-source honesty, e.g. `searxng: exclude_domains ignored`,
`brave_llm: domain filters ignored`.

`sizes` unchanged. `cached` unchanged.

## 4. Tavily client

Still `POST` the configured URL only. Still `include_answer: false`,
`include_raw_content: false`, `include_images: false`, `search_depth: "basic"`.
Still drop inbound `answer`. Still never call `/extract`, `/crawl`, `/map`.

When the request has filters, add only the fields that are set:

- `include_domains`, `exclude_domains`, `include_domains_mode`
- `time_range` or `start_date`/`end_date`
- `topic` when not `general`
- `exact_match` when true
- `include_published_date: true` when `topic` is `news` or a date filter is set

Map `results[].published_date` onto `Hit`. Count `usage.credits` as today.

A fixture test must still prove `include_answer` is false and no extract/crawl
path exists.

## 5. SearXNG and Brave

Do not pretend they support Tavily's full filter set.

**SearXNG** (`GET {url}/search`, same client/dialer as P3):

- One `include_domains` entry: prefix the query with `site:<host> `.
- Several: `site:a OR site:b OR …` up to 3, then warn `searxng: extra include_domains dropped`.
- `exclude_domains`: ignored + warning.
- `include_domains_mode=prefer`: ignored (site: is always restrict) + warning.
- `time_range`: pass SearXNG's `time_range` query param (`day`/`week`/`month`/`year`)
  if we already talk to a SearXNG that accepts it; if the live instance 400s in
  tests against the fixture, keep the warning path instead of guessing.
- `topic=news`: prefix or `!news` only if the existing SearXNG config actually
  has that engine; otherwise warning `searxng: topic ignored`. Default: warning.
- `exact_match`: wrap the query in ASCII double quotes.
- `start_date`/`end_date`: ignored + warning unless SearXNG is proven to accept them.

**Brave LLM Context**: no documented domain/date filters. Ignore all of the new
fields and add one warning `brave_llm: filters ignored` when any filter was set.
Do not add a second Brave HTTP API.

## 6. Multi-query merge

`queries: [q1, q2, …]` (max 4):

1. Run the existing fan-in **once per query** (same sources, same filters,
   same per-source timeout). Parallel across queries is allowed; do not exceed
   the existing per-source client. A Tavily call still counts toward the daily cap;
   if the cap trips mid-batch, later queries skip Tavily with the same warning.
2. Concatenate hits in query order, then **canonical-URL dedupe** (P3 `Dedupe`:
   first source wins attribution; later longer snippet replaces text).
3. RRF as today, treating each `(source, query)` list as a ranked list, **or**
   RRF per query then merge by URL — pick one and test it. Simpler and enough:
   concatenate per-query RRF results, then `Dedupe`, then truncate to `k`.
4. Apply `min_score` after merge (hits with `Score == 0` from sources that do
   not score are kept unless `min_score > 0`, in which case score-less hits are
   dropped — document that SearXNG/Brave may have 0).
5. Cache key must include the normalised query list and the filter canonical
   form (see §7). A single-`query` request is the list of one.

Do not fetch any result URL.

## 7. Cache

P3 key is `sha256(norm(query) | sources | k | hour_bucket | caller_class)`.

Extend with `| filters` where `filters` is a stable encoding of the §2 fields
that affect retrieval (sorted domains, mode, time_range, dates, topic,
exact_match, min_score, the query list). Same query without filters must not
hit a filtered entry.

Still: do not cache if any attempted source failed. Still no page bodies, no
Tavily answers.

## 8. CLI

```
sift search "<query>" [--k n] [--json]
            [--include-domain host]... [--exclude-domain host]...
            [--domains-mode restrict|prefer]
            [--time-range day|week|month|year]
            [--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD]
            [--topic general|news|finance]
            [--exact-match]
            [--min-score 0.5]
            [--queries-file path]
```

`--queries-file` is a JSON array of 1–4 strings. It is mutually exclusive with
the positional query. Repeatable `--include-domain` / `--exclude-domain`.

Default text output: if `published_date` is present, print it as a fourth line
under the snippet. `--json` is the native envelope.

## 9. Skill

`skill/SKILL.md`:

- Document the new flags with one example matching the cookbook Product News
  Tracker: official domain search + `topic=news` + `time_range=month`.
- **Recipe:** after `sift search --json`, do not `sift fetch` every URL. Drop
  hits below a score you care about (or take top N), then `sift batch submit`.
- `web_search` remains the fallback only when sift is unreachable. Hermes
  `web_search` still hits `/v2/search` and does not get filters — say so.

## 10. Tests (minimum)

- Tavily outbound JSON: filters present when set; `include_answer` always false;
  `search_depth` basic; no extract/crawl.
- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
- Validation table for §2.
- SearXNG: `site:` prefix for one domain; exclude_domains warning; exact_match quotes.
- Brave: any filter → `brave_llm: filters ignored`, request URL unchanged.
- Multi-query: two queries, overlapping URL, one hit after Dedupe; k cap applies.
- Cache: same query different `include_domains` → miss; same filters → hit.
- `/v2/search` golden unchanged (no new keys required).
- Daily cap still skips Tavily on the Nth call across queries in one request.
- Restricted caller: same sources; logs still have no query/host/URL.

`make gate` (and contract if P3's contract harness is still in the tree).

## 11. Code layout

Touch, do not explode:

- `internal/search` — `Filter` type; extend `Source.Search`; Tavily/SearXNG/Brave
- `internal/api/search.go` — native decode + validation
- `cmd/sift/searchcmd.go` — flags
- `internal/dcache` search key
- `skill/SKILL.md`, `README.md`

No new dependency. No `internal/api/api.go` growth if a new file is cleaner
(`internal/api/search_filter.go`). Guard and auth stay closed.

## 12. Rollout

1. Claude reviews this doc (filters vs P3 guard/contract; SearXNG honesty).
2. titan implements on `git.wntrmute.dev:kyle/sift` after that review. Split
   commits if it helps; one phase, not a P6-style task-file plan unless the
   reviewer asks for one.
3. Claude reviews the code.
4. straylight bumps the orion pin. titan does not edit `~/src/nixos`.

## 13. Explicit non-goals (do not sneak in)

Map, crawl, HTML BM25 chunks, Research, `include_answer`, social-platform
crawlers, changing `/v2/search`. Those were ranked later in the cookbook
write-up on purpose.