docs: P3.1 E1–E3 exact_match quotes, domain OR, Warn comment

Agents quote phrases; exact_match requires a balanced quoted phrase
and does not wrap. Domain rule no longer substring-rejects OR.
Cap skip is not Warn. Titan may implement this SHA.
This commit is contained in:
2026-09-29 15:11:28 -07:00
parent 73ff7b6bb2
commit 5c6a4a40ec
+39 -32
View File
@@ -3,8 +3,8 @@
Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan. patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.
Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10). Not ready for Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10) and of `73ff7b6`
titan until Claude re-approves this SHA. (E1–E3). Spec approved; titan implements this SHA.
P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
@@ -84,17 +84,21 @@ both are sent). The example above shows the fields, not a legal combination.
- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both - Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
set, or `queries` longer than 4, or any empty element → `invalid query`. set, or `queries` longer than 4, or any empty element → `invalid query`.
Each string still has P3's 1500-rune cap (`invalid query`). Each string still has P3's 1500-rune cap (`invalid query`). ASCII `"` is
- `exact_match: true` when the query (or any element of `queries`) already allowed in queries (agents quote phrases themselves).
contains `"` → `invalid query` (do not nest quotes for SearXNG). - `exact_match: true` requires at least one balanced quoted phrase in **each**
query (a `"`…`"` pair with one or more runes between; no unpaired `"`).
Otherwise `invalid search` hint `exact_match`. Do not wrap or rewrite the
query. `exact_match: false`/omitted: quotes are ordinary query text.
- `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20 - `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20
exclude after normalise+dedupe. Empty list = omitted. Bad value → exclude after normalise+dedupe. Empty list = omitted. Bad value →
`invalid search` hint `include_domains` or `exclude_domains`. `invalid search` hint `include_domains` or `exclude_domains`.
- Domain normalisation (apply before validation and before any `site:` rewrite): - Domain normalisation (apply before validation and before any `site:` rewrite):
lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode
`xn--` labels allowed); no scheme, port, path, wildcard, space, quote, or `xn--` labels allowed); no scheme, port, path, wildcard, or space; at most
`OR`; at most 253 characters. Strict validation also stops operator injection 253 characters. Letters/digits/hyphens already make operator injection
into the SearXNG query. impossible (that needs a space). Do **not** substring-reject `OR` —
`oracle.com` is valid. A space, e.g. `a.com OR b.com`, → 400.
- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without - `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
`include_domains` → `invalid search` hint `include_domains_mode`. `include_domains` → `invalid search` hint `include_domains_mode`.
Omitted with `include_domains` present means `restrict`. Omitted with `include_domains` present means `restrict`.
@@ -133,7 +137,7 @@ source's hits are discarded, it lands in `sources_failed`, it is counted in
``` ```
type SourceOut struct { type SourceOut struct {
Hits []Hit Hits []Hit
Warn string // failure only (timeout, HTTP error, cap skip, pin) Warn string // failure only (timeout, HTTP error, pin)
Notes []string // honesty: filters not applied, recall loss Notes []string // honesty: filters not applied, recall loss
Credits int Credits int
} }
@@ -159,11 +163,14 @@ When the request has filters, add only the fields that are set:
- `include_domains_mode` **only when** `include_domains` is set - `include_domains_mode` **only when** `include_domains` is set
- `time_range` **or** `start_date`/`end_date`, never both - `time_range` **or** `start_date`/`end_date`, never both
- `topic` when not `general` (omitted `topic` is `general`) - `topic` when not `general` (omitted `topic` is `general`)
- `exact_match` when true - `exact_match: true` when the flag is set (query text unchanged, including
the agent's quotes)
- `include_published_date: true` when `topic` is `news` or a date filter is set - `include_published_date: true` when `topic` is `news` or a date filter is set
Leave `filter_by_published_date` unset. Do not guess whether date filters use Leave `filter_by_published_date` unset. Tavily's `time_range` and dates
publish date without it; document that it is unset. filter on publish date **or last updated date**; `filter_by_published_date`
also drops undated results. Unset means undated and recently updated pages
can pass the date filter.
Leave `include_usage` unset. `usage.credits` only comes back when requested; Leave `include_usage` unset. `usage.credits` only comes back when requested;
today's count is the code's fallback of 1, which is correct for `basic`. today's count is the code's fallback of 1, which is correct for `basic`.
@@ -193,15 +200,14 @@ never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3);
Never a bang in the query text. Never a bang in the query text.
- `topic=news`: send `categories=news`. Never `!news`. - `topic=news`: send `categories=news`. Never `!news`.
- `topic=finance`: note `searxng: topic ignored` (no finance category). - `topic=finance`: note `searxng: topic ignored` (no finance category).
- `exact_match`: wrap the query in ASCII double quotes (rejected earlier if - `exact_match`: send the query unchanged. SearXNG engines already treat
the query already contains `"`). quotes as phrases. Do not wrap again.
- `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`. - `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`.
- `min_score > 0`: note `searxng: min_score not applied`; keep hits. - `min_score > 0`: note `searxng: min_score not applied`; keep hits.
Live instance: straylight probes orion's SearXNG once **before titan starts** Live instance (straylight, 2026-09-29, not independently verified here):
and posts whether `time_range=` and `categories=news` return 200. If either `time_range=month` and `categories=news` both returned 200 with normal result
400s, this spec is amended to the note path for that param and titan must not lists. Send both params. Tests are offline fixtures.
send it. Tests are offline fixtures and cannot discover this.
**Brave LLM Context**: no documented domain/date/topic/exact filters. Do not **Brave LLM Context**: no documented domain/date/topic/exact filters. Do not
add a second Brave HTTP API. Post-filter still enforces restrict include and add a second Brave HTTP API. Post-filter still enforces restrict include and
@@ -213,9 +219,8 @@ exclude. Notes when the corresponding filter was set:
- `brave_llm: include_domains_mode=prefer not applied` - `brave_llm: include_domains_mode=prefer not applied`
- `brave_llm: min_score not applied` - `brave_llm: min_score not applied`
Request URL unchanged aside from `q` (and exact-match quotes if we ever wrap; Request URL unchanged aside from `q` (do not wrap for `exact_match`; Brave
we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes cannot apply it). Deduplicate identical notes across queries.
across queries.
## 6. Pipeline ## 6. Pipeline
@@ -369,7 +374,8 @@ Owner-given invariants (these are where a free-form implementation drifts):
`min_score`. `min_score`.
- **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in - **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in
restrict-include, or is in exclude, is dropped. `prefer` keeps it. restrict-include, or is in exclude, is dropped. `prefer` keeps it.
Invalid domain with a space or `OR` → 400 `invalid search`. Invalid domain with a space, e.g. `a.com OR b.com`, → 400 `invalid search`.
`oracle.com` is valid.
- **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still - **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still
appears. appears.
- **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip - **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip
@@ -382,11 +388,14 @@ Owner-given invariants (these are where a free-form implementation drifts):
- Tavily outbound JSON: filters present when set; `include_answer` always - Tavily outbound JSON: filters present when set; `include_answer` always
false; `auto_parameters` false; `search_depth` basic; no extract/crawl; false; `auto_parameters` false; `search_depth` basic; no extract/crawl;
`include_domains_mode` absent unless include list set; never both `include_domains_mode` absent unless include list set; never both
`time_range` and dates. `time_range` and dates; `exact_match: true` with query containing `"phrase"`
→ flag true and query unchanged (still quoted); `exact_match: true` with no
quoted phrase → 400 `invalid search` hint `exact_match`.
- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted. - Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
- Validation table for §2 (`invalid query` vs `invalid search` + hint). - Validation table for §2 (`invalid query` vs `invalid search` + hint).
- SearXNG: `site:` prefix for one domain; exact_match quotes; existing `"` → 400 - SearXNG: `site:` prefix for one domain; `exact_match` does not wrap `q`
`invalid query`; `time_range` / `categories` as query params not bangs. (query already quoted by the agent); `time_range` / `categories` as query
params not bangs.
- Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still - Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still
drops off-domain hits. drops off-domain hits.
- `/v2/search` golden unchanged (no new keys required, no `published_date`). - `/v2/search` golden unchanged (no new keys required, no `published_date`).
@@ -410,13 +419,11 @@ No new dependency. No `internal/api/api.go` growth if a new file is cleaner
## 13. Rollout ## 13. Rollout
1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed). 1. Claude approved `73ff7b6` once E1–E3 are in (this SHA). No further spec round.
2. straylight probes orion SearXNG `time_range=` and `categories=news` once and 2. SearXNG probe already returned 200 for `time_range=` and `categories=news`.
posts the status codes in ~agent. If either 400s, amend §5 to the note path 3. titan implements on `git.wntrmute.dev:kyle/sift` from this SHA. Split commits
before titan starts. if it helps; one phase doc (invariants in §11 — titan writes those tests;
3. titan implements on `git.wntrmute.dev:kyle/sift` after Claude's re-approval Claude checks them in the code review). titan does not edit
**and** the SearXNG probe. Split commits if it helps; one phase doc, not a
P6-style task-file plan (invariants are in §11). titan does not edit
`internal/guard/`, `internal/auth/`, or `~/src/nixos`. `internal/guard/`, `internal/auth/`, or `~/src/nixos`.
4. Claude reviews the code. 4. Claude reviews the code.
5. straylight bumps the orion pin. 5. straylight bumps the orion pin.