docs: P3.1 search filters and multi-query merge
Native /v1/search knobs (domains, time, topic, exact match, min_score) plus multi-query URL dedupe. Tavily passthrough; SearXNG approximations with warnings. No crawl/map/research. Claude reviews, titan implements.
This commit is contained in:
+234
@@ -0,0 +1,234 @@
|
|||||||
|
# Sift P3.1 — search filters and multi-query merge
|
||||||
|
|
||||||
|
Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
|
||||||
|
patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.
|
||||||
|
|
||||||
|
P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
|
||||||
|
fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
|
||||||
|
client hardcodes `search_depth: "basic"` and sends no filters
|
||||||
|
(`internal/search/tavily.go`). Native search accepts only `query`, `k`,
|
||||||
|
`rerank`, `digest`.
|
||||||
|
|
||||||
|
This phase adds the cookbook's high-leverage search knobs without taking crawl,
|
||||||
|
map, research, or `include_answer`.
|
||||||
|
|
||||||
|
## 1. Scope
|
||||||
|
|
||||||
|
In:
|
||||||
|
|
||||||
|
- Native filters on `POST /v1/search` and `sift search`.
|
||||||
|
- Passing those filters through to Tavily when that source is enabled.
|
||||||
|
- Honest SearXNG approximations where the engine can do it; warnings where it cannot.
|
||||||
|
- Multi-query search: several focused queries, one URL-deduped result list.
|
||||||
|
- Optional `min_score` cut after merge.
|
||||||
|
- `published_date` on a hit when a source supplies it (Tavily news).
|
||||||
|
- Skill recipe: search → score/domain filter → `sift batch`, do not fetch every hit.
|
||||||
|
|
||||||
|
Out:
|
||||||
|
|
||||||
|
- `/v2/search` stays the Firecrawl contract (`query`, `limit` only). Unknown
|
||||||
|
fields ignored. No filters on v2.
|
||||||
|
- Crawl, map, Tavily `/extract` `/crawl` `/map`, `include_answer`,
|
||||||
|
`include_raw_content`, `include_images` — P3 §3.3 still holds.
|
||||||
|
- HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases.
|
||||||
|
- Changing `internal/guard/` or `internal/auth/`.
|
||||||
|
- Fetching result URLs inside search.
|
||||||
|
|
||||||
|
## 2. Native request (`POST /v1/search`)
|
||||||
|
|
||||||
|
Existing fields unchanged. Additive:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"query": "…",
|
||||||
|
"queries": ["…", "…"],
|
||||||
|
"k": 8,
|
||||||
|
"rerank": true,
|
||||||
|
"digest": false,
|
||||||
|
"include_domains": ["openai.com"],
|
||||||
|
"exclude_domains": ["pinterest.com"],
|
||||||
|
"include_domains_mode": "restrict",
|
||||||
|
"time_range": "month",
|
||||||
|
"start_date": "2026-09-01",
|
||||||
|
"end_date": "2026-09-29",
|
||||||
|
"topic": "news",
|
||||||
|
"exact_match": false,
|
||||||
|
"min_score": 0
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Validation (400 `invalid query` or 400 `invalid search`):
|
||||||
|
|
||||||
|
- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
|
||||||
|
set, or `queries` longer than 4, is 400. Each string still has P3's 1500-rune cap.
|
||||||
|
- `include_domains` / `exclude_domains`: hostnames only (no scheme, no path).
|
||||||
|
Max 20 include, 20 exclude. Empty list = omitted.
|
||||||
|
- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
|
||||||
|
`include_domains` → 400.
|
||||||
|
- `time_range`: omitted or `day`/`week`/`month`/`year` (and Tavily aliases
|
||||||
|
`d`/`w`/`m`/`y`, normalised to the long form).
|
||||||
|
- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` → 400.
|
||||||
|
- `topic`: omitted/`general`/`news`/`finance`.
|
||||||
|
- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`.
|
||||||
|
- `digest: true` still warns `digest ignored` (P3).
|
||||||
|
|
||||||
|
`/v2/search` does not grow. Hermes never sends these fields.
|
||||||
|
|
||||||
|
## 3. Response
|
||||||
|
|
||||||
|
`/v1/search` results gain optional `"published_date"` (RFC 1123 or ISO date
|
||||||
|
string as the vendor sent it; omit the key when unknown). Do not invent dates.
|
||||||
|
|
||||||
|
`warnings` grows with per-source honesty, e.g. `searxng: exclude_domains ignored`,
|
||||||
|
`brave_llm: domain filters ignored`.
|
||||||
|
|
||||||
|
`sizes` unchanged. `cached` unchanged.
|
||||||
|
|
||||||
|
## 4. Tavily client
|
||||||
|
|
||||||
|
Still `POST` the configured URL only. Still `include_answer: false`,
|
||||||
|
`include_raw_content: false`, `include_images: false`, `search_depth: "basic"`.
|
||||||
|
Still drop inbound `answer`. Still never call `/extract`, `/crawl`, `/map`.
|
||||||
|
|
||||||
|
When the request has filters, add only the fields that are set:
|
||||||
|
|
||||||
|
- `include_domains`, `exclude_domains`, `include_domains_mode`
|
||||||
|
- `time_range` or `start_date`/`end_date`
|
||||||
|
- `topic` when not `general`
|
||||||
|
- `exact_match` when true
|
||||||
|
- `include_published_date: true` when `topic` is `news` or a date filter is set
|
||||||
|
|
||||||
|
Map `results[].published_date` onto `Hit`. Count `usage.credits` as today.
|
||||||
|
|
||||||
|
A fixture test must still prove `include_answer` is false and no extract/crawl
|
||||||
|
path exists.
|
||||||
|
|
||||||
|
## 5. SearXNG and Brave
|
||||||
|
|
||||||
|
Do not pretend they support Tavily's full filter set.
|
||||||
|
|
||||||
|
**SearXNG** (`GET {url}/search`, same client/dialer as P3):
|
||||||
|
|
||||||
|
- One `include_domains` entry: prefix the query with `site:<host> `.
|
||||||
|
- Several: `site:a OR site:b OR …` up to 3, then warn `searxng: extra include_domains dropped`.
|
||||||
|
- `exclude_domains`: ignored + warning.
|
||||||
|
- `include_domains_mode=prefer`: ignored (site: is always restrict) + warning.
|
||||||
|
- `time_range`: pass SearXNG's `time_range` query param (`day`/`week`/`month`/`year`)
|
||||||
|
if we already talk to a SearXNG that accepts it; if the live instance 400s in
|
||||||
|
tests against the fixture, keep the warning path instead of guessing.
|
||||||
|
- `topic=news`: prefix or `!news` only if the existing SearXNG config actually
|
||||||
|
has that engine; otherwise warning `searxng: topic ignored`. Default: warning.
|
||||||
|
- `exact_match`: wrap the query in ASCII double quotes.
|
||||||
|
- `start_date`/`end_date`: ignored + warning unless SearXNG is proven to accept them.
|
||||||
|
|
||||||
|
**Brave LLM Context**: no documented domain/date filters. Ignore all of the new
|
||||||
|
fields and add one warning `brave_llm: filters ignored` when any filter was set.
|
||||||
|
Do not add a second Brave HTTP API.
|
||||||
|
|
||||||
|
## 6. Multi-query merge
|
||||||
|
|
||||||
|
`queries: [q1, q2, …]` (max 4):
|
||||||
|
|
||||||
|
1. Run the existing fan-in **once per query** (same sources, same filters,
|
||||||
|
same per-source timeout). Parallel across queries is allowed; do not exceed
|
||||||
|
the existing per-source client. A Tavily call still counts toward the daily cap;
|
||||||
|
if the cap trips mid-batch, later queries skip Tavily with the same warning.
|
||||||
|
2. Concatenate hits in query order, then **canonical-URL dedupe** (P3 `Dedupe`:
|
||||||
|
first source wins attribution; later longer snippet replaces text).
|
||||||
|
3. RRF as today, treating each `(source, query)` list as a ranked list, **or**
|
||||||
|
RRF per query then merge by URL — pick one and test it. Simpler and enough:
|
||||||
|
concatenate per-query RRF results, then `Dedupe`, then truncate to `k`.
|
||||||
|
4. Apply `min_score` after merge (hits with `Score == 0` from sources that do
|
||||||
|
not score are kept unless `min_score > 0`, in which case score-less hits are
|
||||||
|
dropped — document that SearXNG/Brave may have 0).
|
||||||
|
5. Cache key must include the normalised query list and the filter canonical
|
||||||
|
form (see §7). A single-`query` request is the list of one.
|
||||||
|
|
||||||
|
Do not fetch any result URL.
|
||||||
|
|
||||||
|
## 7. Cache
|
||||||
|
|
||||||
|
P3 key is `sha256(norm(query) | sources | k | hour_bucket | caller_class)`.
|
||||||
|
|
||||||
|
Extend with `| filters` where `filters` is a stable encoding of the §2 fields
|
||||||
|
that affect retrieval (sorted domains, mode, time_range, dates, topic,
|
||||||
|
exact_match, min_score, the query list). Same query without filters must not
|
||||||
|
hit a filtered entry.
|
||||||
|
|
||||||
|
Still: do not cache if any attempted source failed. Still no page bodies, no
|
||||||
|
Tavily answers.
|
||||||
|
|
||||||
|
## 8. CLI
|
||||||
|
|
||||||
|
```
|
||||||
|
sift search "<query>" [--k n] [--json]
|
||||||
|
[--include-domain host]... [--exclude-domain host]...
|
||||||
|
[--domains-mode restrict|prefer]
|
||||||
|
[--time-range day|week|month|year]
|
||||||
|
[--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD]
|
||||||
|
[--topic general|news|finance]
|
||||||
|
[--exact-match]
|
||||||
|
[--min-score 0.5]
|
||||||
|
[--queries-file path]
|
||||||
|
```
|
||||||
|
|
||||||
|
`--queries-file` is a JSON array of 1–4 strings. It is mutually exclusive with
|
||||||
|
the positional query. Repeatable `--include-domain` / `--exclude-domain`.
|
||||||
|
|
||||||
|
Default text output: if `published_date` is present, print it as a fourth line
|
||||||
|
under the snippet. `--json` is the native envelope.
|
||||||
|
|
||||||
|
## 9. Skill
|
||||||
|
|
||||||
|
`skill/SKILL.md`:
|
||||||
|
|
||||||
|
- Document the new flags with one example matching the cookbook Product News
|
||||||
|
Tracker: official domain search + `topic=news` + `time_range=month`.
|
||||||
|
- **Recipe:** after `sift search --json`, do not `sift fetch` every URL. Drop
|
||||||
|
hits below a score you care about (or take top N), then `sift batch submit`.
|
||||||
|
- `web_search` remains the fallback only when sift is unreachable. Hermes
|
||||||
|
`web_search` still hits `/v2/search` and does not get filters — say so.
|
||||||
|
|
||||||
|
## 10. Tests (minimum)
|
||||||
|
|
||||||
|
- Tavily outbound JSON: filters present when set; `include_answer` always false;
|
||||||
|
`search_depth` basic; no extract/crawl.
|
||||||
|
- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
|
||||||
|
- Validation table for §2.
|
||||||
|
- SearXNG: `site:` prefix for one domain; exclude_domains warning; exact_match quotes.
|
||||||
|
- Brave: any filter → `brave_llm: filters ignored`, request URL unchanged.
|
||||||
|
- Multi-query: two queries, overlapping URL, one hit after Dedupe; k cap applies.
|
||||||
|
- Cache: same query different `include_domains` → miss; same filters → hit.
|
||||||
|
- `/v2/search` golden unchanged (no new keys required).
|
||||||
|
- Daily cap still skips Tavily on the Nth call across queries in one request.
|
||||||
|
- Restricted caller: same sources; logs still have no query/host/URL.
|
||||||
|
|
||||||
|
`make gate` (and contract if P3's contract harness is still in the tree).
|
||||||
|
|
||||||
|
## 11. Code layout
|
||||||
|
|
||||||
|
Touch, do not explode:
|
||||||
|
|
||||||
|
- `internal/search` — `Filter` type; extend `Source.Search`; Tavily/SearXNG/Brave
|
||||||
|
- `internal/api/search.go` — native decode + validation
|
||||||
|
- `cmd/sift/searchcmd.go` — flags
|
||||||
|
- `internal/dcache` search key
|
||||||
|
- `skill/SKILL.md`, `README.md`
|
||||||
|
|
||||||
|
No new dependency. No `internal/api/api.go` growth if a new file is cleaner
|
||||||
|
(`internal/api/search_filter.go`). Guard and auth stay closed.
|
||||||
|
|
||||||
|
## 12. Rollout
|
||||||
|
|
||||||
|
1. Claude reviews this doc (filters vs P3 guard/contract; SearXNG honesty).
|
||||||
|
2. titan implements on `git.wntrmute.dev:kyle/sift` after that review. Split
|
||||||
|
commits if it helps; one phase, not a P6-style task-file plan unless the
|
||||||
|
reviewer asks for one.
|
||||||
|
3. Claude reviews the code.
|
||||||
|
4. straylight bumps the orion pin. titan does not edit `~/src/nixos`.
|
||||||
|
|
||||||
|
## 13. Explicit non-goals (do not sneak in)
|
||||||
|
|
||||||
|
Map, crawl, HTML BM25 chunks, Research, `include_answer`, social-platform
|
||||||
|
crawlers, changing `/v2/search`. Those were ranked later in the cookbook
|
||||||
|
write-up on purpose.
|
||||||
Reference in New Issue
Block a user