From 73ff7b6bb28687f1e5e3700fe9f735a4ef75f1b0 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Tue, 29 Sep 2026 15:09:17 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20P3.1=20fold=20Claude=20R1=E2=80=93R10?= =?UTF-8?q?=20before=20titan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Notes vs Warn, Tavily-only min_score, post-merge domain filter, per-query RRF, sequential extra queries, canonical cache key, Tavily body constraints, SearXNG param vs bang, log/error strings. --- docs/P3.1.md | 362 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 278 insertions(+), 84 deletions(-) diff --git a/docs/P3.1.md b/docs/P3.1.md index 40aa43a..b3a3c7d 100644 --- a/docs/P3.1.md +++ b/docs/P3.1.md @@ -3,6 +3,9 @@ Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan. +Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10). Not ready for +titan until Claude re-approves this SHA. + P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily client hardcodes `search_depth: "basic"` and sends no filters @@ -18,22 +21,32 @@ In: - Native filters on `POST /v1/search` and `sift search`. - Passing those filters through to Tavily when that source is enabled. -- Honest SearXNG approximations where the engine can do it; warnings where it cannot. -- Multi-query search: several focused queries, one URL-deduped result list. -- Optional `min_score` cut after merge. -- `published_date` on a hit when a source supplies it (Tavily news). +- Honest SearXNG approximations where the engine can do it; **notes** (not + source failure) where it cannot. +- Post-merge domain enforcement in sift for restrict-mode include and exclude, + for every source. +- Multi-query search: several focused queries, one URL-deduped result list, + fused with RRF across per-query lists. +- Optional `min_score` cut on **Tavily's own score**, per source, before merge. +- `published_date` on a hit when a source supplies it (Tavily news; SearXNG + `publishedDate` when present). - Skill recipe: search → score/domain filter → `sift batch`, do not fetch every hit. Out: - `/v2/search` stays the Firecrawl contract (`query`, `limit` only). Unknown - fields ignored. No filters on v2. + fields ignored. No filters on v2. v2 builds its own output map and must not + leak `published_date`. - Crawl, map, Tavily `/extract` `/crawl` `/map`, `include_answer`, `include_raw_content`, `include_images` — P3 §3.3 still holds. - HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases. - Changing `internal/guard/` or `internal/auth/`. - Fetching result URLs inside search. +Guard and contract: filters only change query text, SearXNG query params, and +the Tavily JSON body. Endpoints stay config-built. Nothing from the request +reaches a dialer. No guard or auth change. + ## 2. Native request (`POST /v1/search`) Existing fields unchanged. Additive: @@ -57,92 +70,217 @@ Existing fields unchanged. Additive: } ``` -Validation (400 `invalid query` or 400 `invalid search`): +`time_range` and `start_date`/`end_date` are mutually exclusive (Tavily 400s if +both are sent). The example above shows the fields, not a legal combination. + +### 2.1 Error strings (tests compile against these) + +- Query problems → 400, `"error": "invalid query"`, empty hint. +- Filter problems → 400, `"error": "invalid search"`, `"hint"` names the field + (`include_domains`, `exclude_domains`, `include_domains_mode`, `time_range`, + `start_date`, `end_date`, `topic`, `min_score`, `exact_match`). + +### 2.2 Validation - Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both - set, or `queries` longer than 4, is 400. Each string still has P3's 1500-rune cap. -- `include_domains` / `exclude_domains`: hostnames only (no scheme, no path). - Max 20 include, 20 exclude. Empty list = omitted. + set, or `queries` longer than 4, or any empty element → `invalid query`. + Each string still has P3's 1500-rune cap (`invalid query`). +- `exact_match: true` when the query (or any element of `queries`) already + contains `"` → `invalid query` (do not nest quotes for SearXNG). +- `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20 + exclude after normalise+dedupe. Empty list = omitted. Bad value → + `invalid search` hint `include_domains` or `exclude_domains`. +- Domain normalisation (apply before validation and before any `site:` rewrite): + lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode + `xn--` labels allowed); no scheme, port, path, wildcard, space, quote, or + `OR`; at most 253 characters. Strict validation also stops operator injection + into the SearXNG query. - `include_domains_mode`: omitted/`restrict`/`prefer`. Set without - `include_domains` → 400. + `include_domains` → `invalid search` hint `include_domains_mode`. + Omitted with `include_domains` present means `restrict`. - `time_range`: omitted or `day`/`week`/`month`/`year` (and Tavily aliases - `d`/`w`/`m`/`y`, normalised to the long form). -- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` → 400. -- `topic`: omitted/`general`/`news`/`finance`. -- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`. -- `digest: true` still warns `digest ignored` (P3). + `d`/`w`/`m`/`y`, normalised to the long form). Else `invalid search` hint + `time_range`. +- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` → + `invalid search` hint `end_date`. Either date together with `time_range` → + `invalid search` hint `time_range`. +- `topic`: omitted/`general`/`news`/`finance`. Else `invalid search` hint `topic`. +- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`. Else `invalid search` + hint `min_score`. +- `digest: true` still warns `digest ignored` (P3). That warning is API-level, + not a source failure. `/v2/search` does not grow. Hermes never sends these fields. -## 3. Response +## 3. Response and source notes `/v1/search` results gain optional `"published_date"` (RFC 1123 or ISO date string as the vendor sent it; omit the key when unknown). Do not invent dates. -`warnings` grows with per-source honesty, e.g. `searxng: exclude_domains ignored`, -`brave_llm: domain filters ignored`. +`Hit` grows `PublishedDate string` (`json:"published_date,omitempty"`). When +Dedupe or RRF replaces a hit's text with a longer snippet, keep the first +non-empty `published_date`. Map SearXNG `publishedDate` when the engine +supplies it. `sizes` unchanged. `cached` unchanged. +### 3.1 `SourceOut.Notes` (R1 — do not reuse `Warn`) + +Today a non-empty `SourceOut.Warn` means failure (`engine.go` Run loop): the +source's hits are discarded, it lands in `sources_failed`, it is counted in +`search_errors`, and caching is disabled. Filter honesty must not take that path. + +``` +type SourceOut struct { + Hits []Hit + Warn string // failure only (timeout, HTTP error, cap skip, pin) + Notes []string // honesty: filters not applied, recall loss + Credits int +} +``` + +- `Warn` keeps today's meaning. Hits discarded; source failed; no cache. +- `Notes` are appended to the response `warnings` (deduplicated across queries). + Hits are kept; the source is in `sources_ok`; the result may be cached. +- Test: hits plus a note → hits kept, searxng in `sources_ok`, result cached. + ## 4. Tavily client Still `POST` the configured URL only. Still `include_answer: false`, `include_raw_content: false`, `include_images: false`, `search_depth: "basic"`. Still drop inbound `answer`. Still never call `/extract`, `/crawl`, `/map`. +Always send `auto_parameters: false`. Auto mode can choose `advanced`, which +costs 2 credits and breaks "credit cost is fixed". + When the request has filters, add only the fields that are set: -- `include_domains`, `exclude_domains`, `include_domains_mode` -- `time_range` or `start_date`/`end_date` -- `topic` when not `general` +- `include_domains`, `exclude_domains` +- `include_domains_mode` **only when** `include_domains` is set +- `time_range` **or** `start_date`/`end_date`, never both +- `topic` when not `general` (omitted `topic` is `general`) - `exact_match` when true - `include_published_date: true` when `topic` is `news` or a date filter is set -Map `results[].published_date` onto `Hit`. Count `usage.credits` as today. +Leave `filter_by_published_date` unset. Do not guess whether date filters use +publish date without it; document that it is unset. -A fixture test must still prove `include_answer` is false and no extract/crawl -path exists. +Leave `include_usage` unset. `usage.credits` only comes back when requested; +today's count is the code's fallback of 1, which is correct for `basic`. + +Map `results[].published_date` onto `Hit`. Count `usage.credits` as today +(fallback 1). + +A fixture test must still prove `include_answer` is false, `auto_parameters` is +false, `search_depth` is `basic`, and no extract/crawl path exists. ## 5. SearXNG and Brave -Do not pretend they support Tavily's full filter set. +Do not pretend they support Tavily's full filter set. Vendor gaps are **notes**, +never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3); +`site:` is a recall optimisation only. **SearXNG** (`GET {url}/search`, same client/dialer as P3): - One `include_domains` entry: prefix the query with `site: `. -- Several: `site:a OR site:b OR …` up to 3, then warn `searxng: extra include_domains dropped`. -- `exclude_domains`: ignored + warning. -- `include_domains_mode=prefer`: ignored (site: is always restrict) + warning. -- `time_range`: pass SearXNG's `time_range` query param (`day`/`week`/`month`/`year`) - if we already talk to a SearXNG that accepts it; if the live instance 400s in - tests against the fixture, keep the warning path instead of guessing. -- `topic=news`: prefix or `!news` only if the existing SearXNG config actually - has that engine; otherwise warning `searxng: topic ignored`. Default: warning. -- `exact_match`: wrap the query in ASCII double quotes. -- `start_date`/`end_date`: ignored + warning unless SearXNG is proven to accept them. +- Several: `site:a OR site:b OR …` up to 3, then note + `searxng: extra include_domains dropped`. +- `exclude_domains`: no query rewrite (post-filter enforces). No note unless + recall is otherwise lost. +- `include_domains_mode=prefer`: no `site:` prefix, no post-filter; note + `searxng: include_domains_mode=prefer not applied`. +- `time_range`: send SearXNG query param `time_range=` (`day`/`week`/`month`/`year`). + Never a bang in the query text. +- `topic=news`: send `categories=news`. Never `!news`. +- `topic=finance`: note `searxng: topic ignored` (no finance category). +- `exact_match`: wrap the query in ASCII double quotes (rejected earlier if + the query already contains `"`). +- `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`. +- `min_score > 0`: note `searxng: min_score not applied`; keep hits. -**Brave LLM Context**: no documented domain/date filters. Ignore all of the new -fields and add one warning `brave_llm: filters ignored` when any filter was set. -Do not add a second Brave HTTP API. +Live instance: straylight probes orion's SearXNG once **before titan starts** +and posts whether `time_range=` and `categories=news` return 200. If either +400s, this spec is amended to the note path for that param and titan must not +send it. Tests are offline fixtures and cannot discover this. -## 6. Multi-query merge +**Brave LLM Context**: no documented domain/date/topic/exact filters. Do not +add a second Brave HTTP API. Post-filter still enforces restrict include and +exclude. Notes when the corresponding filter was set: -`queries: [q1, q2, …]` (max 4): +- `brave_llm: time_range not applied` +- `brave_llm: topic not applied` +- `brave_llm: exact_match not applied` +- `brave_llm: include_domains_mode=prefer not applied` +- `brave_llm: min_score not applied` -1. Run the existing fan-in **once per query** (same sources, same filters, - same per-source timeout). Parallel across queries is allowed; do not exceed - the existing per-source client. A Tavily call still counts toward the daily cap; - if the cap trips mid-batch, later queries skip Tavily with the same warning. -2. Concatenate hits in query order, then **canonical-URL dedupe** (P3 `Dedupe`: - first source wins attribution; later longer snippet replaces text). -3. RRF as today, treating each `(source, query)` list as a ranked list, **or** - RRF per query then merge by URL — pick one and test it. Simpler and enough: - concatenate per-query RRF results, then `Dedupe`, then truncate to `k`. -4. Apply `min_score` after merge (hits with `Score == 0` from sources that do - not score are kept unless `min_score > 0`, in which case score-less hits are - dropped — document that SearXNG/Brave may have 0). -5. Cache key must include the normalised query list and the filter canonical - form (see §7). A single-`query` request is the list of one. +Request URL unchanged aside from `q` (and exact-match quotes if we ever wrap; +we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes +across queries. + +## 6. Pipeline + +`Source.Search` grows a `Filter` argument (zero value = today's behaviour). + +### 6.1 Extra queries run sequentially per source (R5) + +Sources still run in parallel **within** one query, as today. Queries run +**sequentially**. Up to 4 queries means up to 4 Brave and 4 Tavily paid calls; +four parallel Brave calls will hit the plan's rate limit, and a 429 marks +brave failed (`Warn`) and disables caching. + +A Tavily call still counts toward the daily cap. If the cap trips mid-batch, +later queries skip Tavily with the same `tavily: daily cap` warning (skip path, +not a failed attempted source). The queries then ran against different source +sets. **Do not cache** that result (the key's `sources` would be wrong). Test +this. + +### 6.2 `min_score` (R2) + +After merge, `Hit.Score` is the RRF value `Σ 1/(60+rank)`. With 3 sources the +maximum is about 0.05, so a post-merge `--min-score 0.5` would drop everything. +With `rerank:false` scores are vendor-native (Tavily 0–1, SearXNG unbounded, +Brave 0). With Tavily `no_rerank`, Tavily scores mix with RRF. + +Apply `min_score` **per source, before the merge, to Tavily's own relevance +score only**. That is the cookbook's usage. Other sources get a note +(`searxng: min_score not applied`, `brave_llm: min_score not applied`) and keep +their hits. Do not drop score-less Brave/SearXNG hits. + +### 6.3 Domain post-filter (R3) + +After each query's merge, for **every** source, drop hits that violate: + +- `include_domains` when mode is `restrict` (omitted mode = restrict) +- `exclude_domains` (always) + +Match is exact on the result host: `host == d` or `host` has suffix `"." + d` +(host lowercased, trailing dot stripped). Cost is negligible. + +- `prefer` mode: **no** post-filter. Tavily honours `include_domains_mode`; + other sources get the prefer note. +- `site:` is a recall optimisation only. Notes are for recall loss, not for + "filter ignored" when the post-filter already enforced it. +- As written without this, Brave ignores the filter entirely and SearXNG + engines that do not understand `site:` return off-domain hits. + +### 6.4 Multi-query merge (R4) + +Do not concatenate-then-truncate: if q1 returns `k` or more hits, q2–q4 +contribute nothing. + +For `queries: [q1, q2, …]` (max 4), or a single `query` treated as a list of one: + +1. For each query, sequentially: existing fan-in (parallel sources, same + filters, same per-source timeout) → Tavily `min_score` → today's + `mergeSearch` (RRF / no-rerank path) → domain post-filter. +2. **RRF across the per-query fused lists** (each query's list is one ranked + list). URLs several queries found rise. +3. Dedupe (P3: first source wins attribution; later longer snippet replaces + text; keep first non-empty `published_date`). +4. Truncate to `k`. + +Test: q1 returns `k` or more unique hits, and q2's single distinct top hit +still appears. Do not fetch any result URL. @@ -150,15 +288,44 @@ Do not fetch any result URL. P3 key is `sha256(norm(query) | sources | k | hour_bucket | caller_class)`. -Extend with `| filters` where `filters` is a stable encoding of the §2 fields -that affect retrieval (sorted domains, mode, time_range, dates, topic, -exact_match, min_score, the query list). Same query without filters must not -hit a filtered entry. +Replace the payload with a canonical JSON object, keys sorted, then sha256. +It must hold: -Still: do not cache if any attempted source failed. Still no page bodies, no -Tavily answers. +- `caller_class`, hour bucket (`2006-01-02T15` UTC), `k`, attempted `sources` + (sorted) — exactly as P3 has them +- `queries`: normalised, ordered list (a single `query` is a one-element list) +- `include_domains` / `exclude_domains`: normalised, sorted, deduplicated +- `include_domains_mode` (`restrict` or `prefer`; omit the key when no include + list) +- `time_range` in long form (`d` → `day`); dates as `YYYY-MM-DD` or omit +- `topic`, with omitted meaning `general` so the two share a key +- `exact_match` (bool) +- `min_score` in one fixed number format (e.g. `'%g'` is not stable — use + `strconv.FormatFloat(x, 'f', -1, 64)` or a documented equivalent, same for + 0 and omitted) -## 8. CLI +Same query without filters must not hit a filtered entry. + +Still: do not cache if any attempted source failed (`Warn`). Still do not cache +if the Tavily cap tripped partway through a multi-query run. Still no page +bodies, no Tavily answers. + +## 8. Logs + +P3 §8 already bans hosts in the search log for every caller, and filter values +are hosts or query content. + +- Never log domain values, query text, dates, or topic values. +- Restricted callers keep **exactly** P3's fields: `caller`, `q_len`, `sources`, + `n`, `cache`. +- Fleet callers may add `n_queries` and the filter *kinds* + (`filters=domains,time,topic`), never values. `rerank_ms` stays as today. +- `q_len` for several queries: sum of their rune lengths. + +Test: restricted and fleet requests with `include_domains: ["secret.example"]` +and 3 queries → no log record contains the host or any query. + +## 9. CLI ``` sift search "" [--k n] [--json] @@ -172,45 +339,68 @@ sift search "" [--k n] [--json] [--queries-file path] ``` +`--min-score 0.5` is Tavily's 0–1 scale (applied only to Tavily, §6.2). + `--queries-file` is a JSON array of 1–4 strings. It is mutually exclusive with the positional query. Repeatable `--include-domain` / `--exclude-domain`. Default text output: if `published_date` is present, print it as a fourth line under the snippet. `--json` is the native envelope. -## 9. Skill +## 10. Skill `skill/SKILL.md`: - Document the new flags with one example matching the cookbook Product News Tracker: official domain search + `topic=news` + `time_range=month`. - **Recipe:** after `sift search --json`, do not `sift fetch` every URL. Drop - hits below a score you care about (or take top N), then `sift batch submit`. + hits below a Tavily score you care about (or take top N), then `sift batch submit`. - `web_search` remains the fallback only when sift is unreachable. Hermes `web_search` still hits `/v2/search` and does not get filters — say so. -## 10. Tests (minimum) +## 11. Tests (minimum) -- Tavily outbound JSON: filters present when set; `include_answer` always false; - `search_depth` basic; no extract/crawl. +Owner-given invariants (these are where a free-form implementation drifts): + +- **R1 notes:** hits plus a note → hits kept, source in `sources_ok`, result + cached. A `Warn` still fails the source and disables cache. +- **R2 min_score:** Tavily hit at 0.4 dropped when `min_score=0.5`; SearXNG / + Brave hits kept; notes present. Post-merge RRF scores are not compared to + `min_score`. +- **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in + restrict-include, or is in exclude, is dropped. `prefer` keeps it. + Invalid domain with a space or `OR` → 400 `invalid search`. +- **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still + appears. +- **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip + Tavily; result is not cached. +- **R6 cache key:** same query different `include_domains` → miss; same + filters → hit; omitted `topic` and `topic=general` share a key; `d` and + `day` share a key. +- **R9 logs:** restricted and fleet with `include_domains:["secret.example"]` + and 3 queries → no log record contains the host or any query. +- Tavily outbound JSON: filters present when set; `include_answer` always + false; `auto_parameters` false; `search_depth` basic; no extract/crawl; + `include_domains_mode` absent unless include list set; never both + `time_range` and dates. - Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted. -- Validation table for §2. -- SearXNG: `site:` prefix for one domain; exclude_domains warning; exact_match quotes. -- Brave: any filter → `brave_llm: filters ignored`, request URL unchanged. -- Multi-query: two queries, overlapping URL, one hit after Dedupe; k cap applies. -- Cache: same query different `include_domains` → miss; same filters → hit. -- `/v2/search` golden unchanged (no new keys required). -- Daily cap still skips Tavily on the Nth call across queries in one request. +- Validation table for §2 (`invalid query` vs `invalid search` + hint). +- SearXNG: `site:` prefix for one domain; exact_match quotes; existing `"` → 400 + `invalid query`; `time_range` / `categories` as query params not bangs. +- Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still + drops off-domain hits. +- `/v2/search` golden unchanged (no new keys required, no `published_date`). - Restricted caller: same sources; logs still have no query/host/URL. `make gate` (and contract if P3's contract harness is still in the tree). -## 11. Code layout +## 12. Code layout Touch, do not explode: -- `internal/search` — `Filter` type; extend `Source.Search`; Tavily/SearXNG/Brave -- `internal/api/search.go` — native decode + validation +- `internal/search` — `Filter` type; `SourceOut.Notes`; extend `Source.Search`; + Tavily/SearXNG/Brave; engine loop; domain post-filter; multi-query RRF +- `internal/api/search.go` — native decode + validation + log fields - `cmd/sift/searchcmd.go` — flags - `internal/dcache` search key - `skill/SKILL.md`, `README.md` @@ -218,16 +408,20 @@ Touch, do not explode: No new dependency. No `internal/api/api.go` growth if a new file is cleaner (`internal/api/search_filter.go`). Guard and auth stay closed. -## 12. Rollout +## 13. Rollout -1. Claude reviews this doc (filters vs P3 guard/contract; SearXNG honesty). -2. titan implements on `git.wntrmute.dev:kyle/sift` after that review. Split - commits if it helps; one phase, not a P6-style task-file plan unless the - reviewer asks for one. -3. Claude reviews the code. -4. straylight bumps the orion pin. titan does not edit `~/src/nixos`. +1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed). +2. straylight probes orion SearXNG `time_range=` and `categories=news` once and + posts the status codes in ~agent. If either 400s, amend §5 to the note path + before titan starts. +3. titan implements on `git.wntrmute.dev:kyle/sift` after Claude's re-approval + **and** the SearXNG probe. Split commits if it helps; one phase doc, not a + P6-style task-file plan (invariants are in §11). titan does not edit + `internal/guard/`, `internal/auth/`, or `~/src/nixos`. +4. Claude reviews the code. +5. straylight bumps the orion pin. -## 13. Explicit non-goals (do not sneak in) +## 14. Explicit non-goals (do not sneak in) Map, crawl, HTML BM25 chunks, Research, `include_answer`, social-platform crawlers, changing `/v2/search`. Those were ranked later in the cookbook