docs: P3.1 fold Claude R1–R10 before titan

Notes vs Warn, Tavily-only min_score, post-merge domain filter,
per-query RRF, sequential extra queries, canonical cache key,
Tavily body constraints, SearXNG param vs bang, log/error strings.

docs/P3.1.md

@@ -3,6 +3,9 @@
 Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
 patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.
 
+Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10). Not ready for
+titan until Claude re-approves this SHA.
+
 P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
 fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
 client hardcodes `search_depth: "basic"` and sends no filters
@@ -18,22 +21,32 @@ In:
 
 - Native filters on `POST /v1/search` and `sift search`.
 - Passing those filters through to Tavily when that source is enabled.
-- Honest SearXNG approximations where the engine can do it; warnings where it cannot.
-- Multi-query search: several focused queries, one URL-deduped result list.
-- Optional `min_score` cut after merge.
-- `published_date` on a hit when a source supplies it (Tavily news).
+- Honest SearXNG approximations where the engine can do it; **notes** (not
+  source failure) where it cannot.
+- Post-merge domain enforcement in sift for restrict-mode include and exclude,
+  for every source.
+- Multi-query search: several focused queries, one URL-deduped result list,
+  fused with RRF across per-query lists.
+- Optional `min_score` cut on **Tavily's own score**, per source, before merge.
+- `published_date` on a hit when a source supplies it (Tavily news; SearXNG
+  `publishedDate` when present).
 - Skill recipe: search → score/domain filter → `sift batch`, do not fetch every hit.
 
 Out:
 
 - `/v2/search` stays the Firecrawl contract (`query`, `limit` only). Unknown
-  fields ignored. No filters on v2.
+  fields ignored. No filters on v2. v2 builds its own output map and must not
+  leak `published_date`.
 - Crawl, map, Tavily `/extract` `/crawl` `/map`, `include_answer`,
   `include_raw_content`, `include_images` — P3 §3.3 still holds.
 - HTML lexical chunks, same-host map, Research/SSE/JSON-schema — later phases.
 - Changing `internal/guard/` or `internal/auth/`.
 - Fetching result URLs inside search.
 
+Guard and contract: filters only change query text, SearXNG query params, and
+the Tavily JSON body. Endpoints stay config-built. Nothing from the request
+reaches a dialer. No guard or auth change.
+
 ## 2. Native request (`POST /v1/search`)
 
 Existing fields unchanged. Additive:
@@ -57,92 +70,217 @@ Existing fields unchanged. Additive:
 }
 ```
 
-Validation (400 `invalid query` or 400 `invalid search`):
+`time_range` and `start_date`/`end_date` are mutually exclusive (Tavily 400s if
+both are sent). The example above shows the fields, not a legal combination.
+
+### 2.1 Error strings (tests compile against these)
+
+- Query problems → 400, `"error": "invalid query"`, empty hint.
+- Filter problems → 400, `"error": "invalid search"`, `"hint"` names the field
+  (`include_domains`, `exclude_domains`, `include_domains_mode`, `time_range`,
+  `start_date`, `end_date`, `topic`, `min_score`, `exact_match`).
+
+### 2.2 Validation
 
 - Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
-  set, or `queries` longer than 4, is 400. Each string still has P3's 1500-rune cap.
-- `include_domains` / `exclude_domains`: hostnames only (no scheme, no path).
-  Max 20 include, 20 exclude. Empty list = omitted.
+  set, or `queries` longer than 4, or any empty element → `invalid query`.
+  Each string still has P3's 1500-rune cap (`invalid query`).
+- `exact_match: true` when the query (or any element of `queries`) already
+  contains `"` → `invalid query` (do not nest quotes for SearXNG).
+- `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20
+  exclude after normalise+dedupe. Empty list = omitted. Bad value →
+  `invalid search` hint `include_domains` or `exclude_domains`.
+- Domain normalisation (apply before validation and before any `site:` rewrite):
+  lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode
+  `xn--` labels allowed); no scheme, port, path, wildcard, space, quote, or
+  `OR`; at most 253 characters. Strict validation also stops operator injection
+  into the SearXNG query.
 - `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
-  `include_domains` → 400.
+  `include_domains` → `invalid search` hint `include_domains_mode`.
+  Omitted with `include_domains` present means `restrict`.
 - `time_range`: omitted or `day`/`week`/`month`/`year` (and Tavily aliases
-  `d`/`w`/`m`/`y`, normalised to the long form).
-- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` → 400.
-- `topic`: omitted/`general`/`news`/`finance`.
-- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`.
-- `digest: true` still warns `digest ignored` (P3).
+  `d`/`w`/`m`/`y`, normalised to the long form). Else `invalid search` hint
+  `time_range`.
+- `start_date` / `end_date`: `YYYY-MM-DD` or omitted. `end` before `start` →
+  `invalid search` hint `end_date`. Either date together with `time_range` →
+  `invalid search` hint `time_range`.
+- `topic`: omitted/`general`/`news`/`finance`. Else `invalid search` hint `topic`.
+- `min_score`: omitted or `0` = off; otherwise `0 < x ≤ 1`. Else `invalid search`
+  hint `min_score`.
+- `digest: true` still warns `digest ignored` (P3). That warning is API-level,
+  not a source failure.
 
 `/v2/search` does not grow. Hermes never sends these fields.
 
-## 3. Response
+## 3. Response and source notes
 
 `/v1/search` results gain optional `"published_date"` (RFC 1123 or ISO date
 string as the vendor sent it; omit the key when unknown). Do not invent dates.
 
-`warnings` grows with per-source honesty, e.g. `searxng: exclude_domains ignored`,
-`brave_llm: domain filters ignored`.
+`Hit` grows `PublishedDate string` (`json:"published_date,omitempty"`). When
+Dedupe or RRF replaces a hit's text with a longer snippet, keep the first
+non-empty `published_date`. Map SearXNG `publishedDate` when the engine
+supplies it.
 
 `sizes` unchanged. `cached` unchanged.
 
+### 3.1 `SourceOut.Notes` (R1 — do not reuse `Warn`)
+
+Today a non-empty `SourceOut.Warn` means failure (`engine.go` Run loop): the
+source's hits are discarded, it lands in `sources_failed`, it is counted in
+`search_errors`, and caching is disabled. Filter honesty must not take that path.
+
+```
+type SourceOut struct {
+    Hits    []Hit
+    Warn    string    // failure only (timeout, HTTP error, cap skip, pin)
+    Notes   []string  // honesty: filters not applied, recall loss
+    Credits int
+}
+```
+
+- `Warn` keeps today's meaning. Hits discarded; source failed; no cache.
+- `Notes` are appended to the response `warnings` (deduplicated across queries).
+  Hits are kept; the source is in `sources_ok`; the result may be cached.
+- Test: hits plus a note → hits kept, searxng in `sources_ok`, result cached.
+
 ## 4. Tavily client
 
 Still `POST` the configured URL only. Still `include_answer: false`,
 `include_raw_content: false`, `include_images: false`, `search_depth: "basic"`.
 Still drop inbound `answer`. Still never call `/extract`, `/crawl`, `/map`.
 
+Always send `auto_parameters: false`. Auto mode can choose `advanced`, which
+costs 2 credits and breaks "credit cost is fixed".
+
 When the request has filters, add only the fields that are set:
 
-- `include_domains`, `exclude_domains`, `include_domains_mode`
-- `time_range` or `start_date`/`end_date`
-- `topic` when not `general`
+- `include_domains`, `exclude_domains`
+- `include_domains_mode` **only when** `include_domains` is set
+- `time_range` **or** `start_date`/`end_date`, never both
+- `topic` when not `general` (omitted `topic` is `general`)
 - `exact_match` when true
 - `include_published_date: true` when `topic` is `news` or a date filter is set
 
-Map `results[].published_date` onto `Hit`. Count `usage.credits` as today.
+Leave `filter_by_published_date` unset. Do not guess whether date filters use
+publish date without it; document that it is unset.
 
-A fixture test must still prove `include_answer` is false and no extract/crawl
-path exists.
+Leave `include_usage` unset. `usage.credits` only comes back when requested;
+today's count is the code's fallback of 1, which is correct for `basic`.
+
+Map `results[].published_date` onto `Hit`. Count `usage.credits` as today
+(fallback 1).
+
+A fixture test must still prove `include_answer` is false, `auto_parameters` is
+false, `search_depth` is `basic`, and no extract/crawl path exists.
 
 ## 5. SearXNG and Brave
 
-Do not pretend they support Tavily's full filter set.
+Do not pretend they support Tavily's full filter set. Vendor gaps are **notes**,
+never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3);
+`site:` is a recall optimisation only.
 
 **SearXNG** (`GET {url}/search`, same client/dialer as P3):
 
 - One `include_domains` entry: prefix the query with `site:<host> `.
-- Several: `site:a OR site:b OR …` up to 3, then warn `searxng: extra include_domains dropped`.
-- `exclude_domains`: ignored + warning.
-- `include_domains_mode=prefer`: ignored (site: is always restrict) + warning.
-- `time_range`: pass SearXNG's `time_range` query param (`day`/`week`/`month`/`year`)
-  if we already talk to a SearXNG that accepts it; if the live instance 400s in
-  tests against the fixture, keep the warning path instead of guessing.
-- `topic=news`: prefix or `!news` only if the existing SearXNG config actually
-  has that engine; otherwise warning `searxng: topic ignored`. Default: warning.
-- `exact_match`: wrap the query in ASCII double quotes.
-- `start_date`/`end_date`: ignored + warning unless SearXNG is proven to accept them.
+- Several: `site:a OR site:b OR …` up to 3, then note
+  `searxng: extra include_domains dropped`.
+- `exclude_domains`: no query rewrite (post-filter enforces). No note unless
+  recall is otherwise lost.
+- `include_domains_mode=prefer`: no `site:` prefix, no post-filter; note
+  `searxng: include_domains_mode=prefer not applied`.
+- `time_range`: send SearXNG query param `time_range=` (`day`/`week`/`month`/`year`).
+  Never a bang in the query text.
+- `topic=news`: send `categories=news`. Never `!news`.
+- `topic=finance`: note `searxng: topic ignored` (no finance category).
+- `exact_match`: wrap the query in ASCII double quotes (rejected earlier if
+  the query already contains `"`).
+- `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`.
+- `min_score > 0`: note `searxng: min_score not applied`; keep hits.
 
-**Brave LLM Context**: no documented domain/date filters. Ignore all of the new
-fields and add one warning `brave_llm: filters ignored` when any filter was set.
-Do not add a second Brave HTTP API.
+Live instance: straylight probes orion's SearXNG once **before titan starts**
+and posts whether `time_range=` and `categories=news` return 200. If either
+400s, this spec is amended to the note path for that param and titan must not
+send it. Tests are offline fixtures and cannot discover this.
 
-## 6. Multi-query merge
+**Brave LLM Context**: no documented domain/date/topic/exact filters. Do not
+add a second Brave HTTP API. Post-filter still enforces restrict include and
+exclude. Notes when the corresponding filter was set:
 
-`queries: [q1, q2, …]` (max 4):
+- `brave_llm: time_range not applied`
+- `brave_llm: topic not applied`
+- `brave_llm: exact_match not applied`
+- `brave_llm: include_domains_mode=prefer not applied`
+- `brave_llm: min_score not applied`
 
-1. Run the existing fan-in **once per query** (same sources, same filters,
-   same per-source timeout). Parallel across queries is allowed; do not exceed
-   the existing per-source client. A Tavily call still counts toward the daily cap;
-   if the cap trips mid-batch, later queries skip Tavily with the same warning.
-2. Concatenate hits in query order, then **canonical-URL dedupe** (P3 `Dedupe`:
-   first source wins attribution; later longer snippet replaces text).
-3. RRF as today, treating each `(source, query)` list as a ranked list, **or**
-   RRF per query then merge by URL — pick one and test it. Simpler and enough:
-   concatenate per-query RRF results, then `Dedupe`, then truncate to `k`.
-4. Apply `min_score` after merge (hits with `Score == 0` from sources that do
-   not score are kept unless `min_score > 0`, in which case score-less hits are
-   dropped — document that SearXNG/Brave may have 0).
-5. Cache key must include the normalised query list and the filter canonical
-   form (see §7). A single-`query` request is the list of one.
+Request URL unchanged aside from `q` (and exact-match quotes if we ever wrap;
+we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes
+across queries.
+
+## 6. Pipeline
+
+`Source.Search` grows a `Filter` argument (zero value = today's behaviour).
+
+### 6.1 Extra queries run sequentially per source (R5)
+
+Sources still run in parallel **within** one query, as today. Queries run
+**sequentially**. Up to 4 queries means up to 4 Brave and 4 Tavily paid calls;
+four parallel Brave calls will hit the plan's rate limit, and a 429 marks
+brave failed (`Warn`) and disables caching.
+
+A Tavily call still counts toward the daily cap. If the cap trips mid-batch,
+later queries skip Tavily with the same `tavily: daily cap` warning (skip path,
+not a failed attempted source). The queries then ran against different source
+sets. **Do not cache** that result (the key's `sources` would be wrong). Test
+this.
+
+### 6.2 `min_score` (R2)
+
+After merge, `Hit.Score` is the RRF value `Σ 1/(60+rank)`. With 3 sources the
+maximum is about 0.05, so a post-merge `--min-score 0.5` would drop everything.
+With `rerank:false` scores are vendor-native (Tavily 0–1, SearXNG unbounded,
+Brave 0). With Tavily `no_rerank`, Tavily scores mix with RRF.
+
+Apply `min_score` **per source, before the merge, to Tavily's own relevance
+score only**. That is the cookbook's usage. Other sources get a note
+(`searxng: min_score not applied`, `brave_llm: min_score not applied`) and keep
+their hits. Do not drop score-less Brave/SearXNG hits.
+
+### 6.3 Domain post-filter (R3)
+
+After each query's merge, for **every** source, drop hits that violate:
+
+- `include_domains` when mode is `restrict` (omitted mode = restrict)
+- `exclude_domains` (always)
+
+Match is exact on the result host: `host == d` or `host` has suffix `"." + d`
+(host lowercased, trailing dot stripped). Cost is negligible.
+
+- `prefer` mode: **no** post-filter. Tavily honours `include_domains_mode`;
+  other sources get the prefer note.
+- `site:` is a recall optimisation only. Notes are for recall loss, not for
+  "filter ignored" when the post-filter already enforced it.
+- As written without this, Brave ignores the filter entirely and SearXNG
+  engines that do not understand `site:` return off-domain hits.
+
+### 6.4 Multi-query merge (R4)
+
+Do not concatenate-then-truncate: if q1 returns `k` or more hits, q2–q4
+contribute nothing.
+
+For `queries: [q1, q2, …]` (max 4), or a single `query` treated as a list of one:
+
+1. For each query, sequentially: existing fan-in (parallel sources, same
+   filters, same per-source timeout) → Tavily `min_score` → today's
+   `mergeSearch` (RRF / no-rerank path) → domain post-filter.
+2. **RRF across the per-query fused lists** (each query's list is one ranked
+   list). URLs several queries found rise.
+3. Dedupe (P3: first source wins attribution; later longer snippet replaces
+   text; keep first non-empty `published_date`).
+4. Truncate to `k`.
+
+Test: q1 returns `k` or more unique hits, and q2's single distinct top hit
+still appears.
 
 Do not fetch any result URL.
 
@@ -150,15 +288,44 @@ Do not fetch any result URL.
 
 P3 key is `sha256(norm(query) | sources | k | hour_bucket | caller_class)`.
 
-Extend with `| filters` where `filters` is a stable encoding of the §2 fields
-that affect retrieval (sorted domains, mode, time_range, dates, topic,
-exact_match, min_score, the query list). Same query without filters must not
-hit a filtered entry.
+Replace the payload with a canonical JSON object, keys sorted, then sha256.
+It must hold:
 
-Still: do not cache if any attempted source failed. Still no page bodies, no
-Tavily answers.
+- `caller_class`, hour bucket (`2006-01-02T15` UTC), `k`, attempted `sources`
+  (sorted) — exactly as P3 has them
+- `queries`: normalised, ordered list (a single `query` is a one-element list)
+- `include_domains` / `exclude_domains`: normalised, sorted, deduplicated
+- `include_domains_mode` (`restrict` or `prefer`; omit the key when no include
+  list)
+- `time_range` in long form (`d` → `day`); dates as `YYYY-MM-DD` or omit
+- `topic`, with omitted meaning `general` so the two share a key
+- `exact_match` (bool)
+- `min_score` in one fixed number format (e.g. `'%g'` is not stable — use
+  `strconv.FormatFloat(x, 'f', -1, 64)` or a documented equivalent, same for
+  0 and omitted)
 
-## 8. CLI
+Same query without filters must not hit a filtered entry.
+
+Still: do not cache if any attempted source failed (`Warn`). Still do not cache
+if the Tavily cap tripped partway through a multi-query run. Still no page
+bodies, no Tavily answers.
+
+## 8. Logs
+
+P3 §8 already bans hosts in the search log for every caller, and filter values
+are hosts or query content.
+
+- Never log domain values, query text, dates, or topic values.
+- Restricted callers keep **exactly** P3's fields: `caller`, `q_len`, `sources`,
+  `n`, `cache`.
+- Fleet callers may add `n_queries` and the filter *kinds*
+  (`filters=domains,time,topic`), never values. `rerank_ms` stays as today.
+- `q_len` for several queries: sum of their rune lengths.
+
+Test: restricted and fleet requests with `include_domains: ["secret.example"]`
+and 3 queries → no log record contains the host or any query.
+
+## 9. CLI
 
 ```
 sift search "<query>" [--k n] [--json]
@@ -172,45 +339,68 @@ sift search "<query>" [--k n] [--json]
             [--queries-file path]
 ```
 
+`--min-score 0.5` is Tavily's 0–1 scale (applied only to Tavily, §6.2).
+
 `--queries-file` is a JSON array of 1–4 strings. It is mutually exclusive with
 the positional query. Repeatable `--include-domain` / `--exclude-domain`.
 
 Default text output: if `published_date` is present, print it as a fourth line
 under the snippet. `--json` is the native envelope.
 
-## 9. Skill
+## 10. Skill
 
 `skill/SKILL.md`:
 
 - Document the new flags with one example matching the cookbook Product News
   Tracker: official domain search + `topic=news` + `time_range=month`.
 - **Recipe:** after `sift search --json`, do not `sift fetch` every URL. Drop
-  hits below a score you care about (or take top N), then `sift batch submit`.
+  hits below a Tavily score you care about (or take top N), then `sift batch submit`.
 - `web_search` remains the fallback only when sift is unreachable. Hermes
   `web_search` still hits `/v2/search` and does not get filters — say so.
 
-## 10. Tests (minimum)
+## 11. Tests (minimum)
 
-- Tavily outbound JSON: filters present when set; `include_answer` always false;
-  `search_depth` basic; no extract/crawl.
+Owner-given invariants (these are where a free-form implementation drifts):
+
+- **R1 notes:** hits plus a note → hits kept, source in `sources_ok`, result
+  cached. A `Warn` still fails the source and disables cache.
+- **R2 min_score:** Tavily hit at 0.4 dropped when `min_score=0.5`; SearXNG /
+  Brave hits kept; notes present. Post-merge RRF scores are not compared to
+  `min_score`.
+- **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in
+  restrict-include, or is in exclude, is dropped. `prefer` keeps it.
+  Invalid domain with a space or `OR` → 400 `invalid search`.
+- **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still
+  appears.
+- **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip
+  Tavily; result is not cached.
+- **R6 cache key:** same query different `include_domains` → miss; same
+  filters → hit; omitted `topic` and `topic=general` share a key; `d` and
+  `day` share a key.
+- **R9 logs:** restricted and fleet with `include_domains:["secret.example"]`
+  and 3 queries → no log record contains the host or any query.
+- Tavily outbound JSON: filters present when set; `include_answer` always
+  false; `auto_parameters` false; `search_depth` basic; no extract/crawl;
+  `include_domains_mode` absent unless include list set; never both
+  `time_range` and dates.
 - Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
-- Validation table for §2.
-- SearXNG: `site:` prefix for one domain; exclude_domains warning; exact_match quotes.
-- Brave: any filter → `brave_llm: filters ignored`, request URL unchanged.
-- Multi-query: two queries, overlapping URL, one hit after Dedupe; k cap applies.
-- Cache: same query different `include_domains` → miss; same filters → hit.
-- `/v2/search` golden unchanged (no new keys required).
-- Daily cap still skips Tavily on the Nth call across queries in one request.
+- Validation table for §2 (`invalid query` vs `invalid search` + hint).
+- SearXNG: `site:` prefix for one domain; exact_match quotes; existing `"` → 400
+  `invalid query`; `time_range` / `categories` as query params not bangs.
+- Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still
+  drops off-domain hits.
+- `/v2/search` golden unchanged (no new keys required, no `published_date`).
 - Restricted caller: same sources; logs still have no query/host/URL.
 
 `make gate` (and contract if P3's contract harness is still in the tree).
 
-## 11. Code layout
+## 12. Code layout
 
 Touch, do not explode:
 
-- `internal/search` — `Filter` type; extend `Source.Search`; Tavily/SearXNG/Brave
-- `internal/api/search.go` — native decode + validation
+- `internal/search` — `Filter` type; `SourceOut.Notes`; extend `Source.Search`;
+  Tavily/SearXNG/Brave; engine loop; domain post-filter; multi-query RRF
+- `internal/api/search.go` — native decode + validation + log fields
 - `cmd/sift/searchcmd.go` — flags
 - `internal/dcache` search key
 - `skill/SKILL.md`, `README.md`
@@ -218,16 +408,20 @@ Touch, do not explode:
 No new dependency. No `internal/api/api.go` growth if a new file is cleaner
 (`internal/api/search_filter.go`). Guard and auth stay closed.
 
-## 12. Rollout
+## 13. Rollout
 
-1. Claude reviews this doc (filters vs P3 guard/contract; SearXNG honesty).
-2. titan implements on `git.wntrmute.dev:kyle/sift` after that review. Split
-   commits if it helps; one phase, not a P6-style task-file plan unless the
-   reviewer asks for one.
-3. Claude reviews the code.
-4. straylight bumps the orion pin. titan does not edit `~/src/nixos`.
+1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed).
+2. straylight probes orion SearXNG `time_range=` and `categories=news` once and
+   posts the status codes in ~agent. If either 400s, amend §5 to the note path
+   before titan starts.
+3. titan implements on `git.wntrmute.dev:kyle/sift` after Claude's re-approval
+   **and** the SearXNG probe. Split commits if it helps; one phase doc, not a
+   P6-style task-file plan (invariants are in §11). titan does not edit
+   `internal/guard/`, `internal/auth/`, or `~/src/nixos`.
+4. Claude reviews the code.
+5. straylight bumps the orion pin.
 
-## 13. Explicit non-goals (do not sneak in)
+## 14. Explicit non-goals (do not sneak in)
 
 Map, crawl, HTML BM25 chunks, Research, `include_answer`, social-platform
 crawlers, changing `/v2/search`. Those were ranked later in the cookbook