docs: P3.1 E1–E3 exact_match quotes, domain OR, Warn comment

Agents quote phrases; exact_match requires a balanced quoted phrase
and does not wrap. Domain rule no longer substring-rejects OR.
Cap skip is not Warn. Titan may implement this SHA.
This commit is contained in:
2026-09-29 15:11:28 -07:00
parent 73ff7b6bb2
commit 5c6a4a40ec
+39 -32
View File
@@ -3,8 +3,8 @@
Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.
Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10). Not ready for
titan until Claude re-approves this SHA.
Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10) and of `73ff7b6`
(E1–E3). Spec approved; titan implements this SHA.
P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
@@ -84,17 +84,21 @@ both are sent). The example above shows the fields, not a legal combination.
- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
set, or `queries` longer than 4, or any empty element → `invalid query`.
Each string still has P3's 1500-rune cap (`invalid query`).
- `exact_match: true` when the query (or any element of `queries`) already
contains `"` → `invalid query` (do not nest quotes for SearXNG).
Each string still has P3's 1500-rune cap (`invalid query`). ASCII `"` is
allowed in queries (agents quote phrases themselves).
- `exact_match: true` requires at least one balanced quoted phrase in **each**
query (a `"`…`"` pair with one or more runes between; no unpaired `"`).
Otherwise `invalid search` hint `exact_match`. Do not wrap or rewrite the
query. `exact_match: false`/omitted: quotes are ordinary query text.
- `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20
exclude after normalise+dedupe. Empty list = omitted. Bad value →
`invalid search` hint `include_domains` or `exclude_domains`.
- Domain normalisation (apply before validation and before any `site:` rewrite):
lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode
`xn--` labels allowed); no scheme, port, path, wildcard, space, quote, or
`OR`; at most 253 characters. Strict validation also stops operator injection
into the SearXNG query.
`xn--` labels allowed); no scheme, port, path, wildcard, or space; at most
253 characters. Letters/digits/hyphens already make operator injection
impossible (that needs a space). Do **not** substring-reject `OR` —
`oracle.com` is valid. A space, e.g. `a.com OR b.com`, → 400.
- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
`include_domains` → `invalid search` hint `include_domains_mode`.
Omitted with `include_domains` present means `restrict`.
@@ -133,7 +137,7 @@ source's hits are discarded, it lands in `sources_failed`, it is counted in
```
type SourceOut struct {
Hits []Hit
Warn string // failure only (timeout, HTTP error, cap skip, pin)
Warn string // failure only (timeout, HTTP error, pin)
Notes []string // honesty: filters not applied, recall loss
Credits int
}
@@ -159,11 +163,14 @@ When the request has filters, add only the fields that are set:
- `include_domains_mode` **only when** `include_domains` is set
- `time_range` **or** `start_date`/`end_date`, never both
- `topic` when not `general` (omitted `topic` is `general`)
- `exact_match` when true
- `exact_match: true` when the flag is set (query text unchanged, including
the agent's quotes)
- `include_published_date: true` when `topic` is `news` or a date filter is set
Leave `filter_by_published_date` unset. Do not guess whether date filters use
publish date without it; document that it is unset.
Leave `filter_by_published_date` unset. Tavily's `time_range` and dates
filter on publish date **or last updated date**; `filter_by_published_date`
also drops undated results. Unset means undated and recently updated pages
can pass the date filter.
Leave `include_usage` unset. `usage.credits` only comes back when requested;
today's count is the code's fallback of 1, which is correct for `basic`.
@@ -193,15 +200,14 @@ never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3);
Never a bang in the query text.
- `topic=news`: send `categories=news`. Never `!news`.
- `topic=finance`: note `searxng: topic ignored` (no finance category).
- `exact_match`: wrap the query in ASCII double quotes (rejected earlier if
the query already contains `"`).
- `exact_match`: send the query unchanged. SearXNG engines already treat
quotes as phrases. Do not wrap again.
- `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`.
- `min_score > 0`: note `searxng: min_score not applied`; keep hits.
Live instance: straylight probes orion's SearXNG once **before titan starts**
and posts whether `time_range=` and `categories=news` return 200. If either
400s, this spec is amended to the note path for that param and titan must not
send it. Tests are offline fixtures and cannot discover this.
Live instance (straylight, 2026-09-29, not independently verified here):
`time_range=month` and `categories=news` both returned 200 with normal result
lists. Send both params. Tests are offline fixtures.
**Brave LLM Context**: no documented domain/date/topic/exact filters. Do not
add a second Brave HTTP API. Post-filter still enforces restrict include and
@@ -213,9 +219,8 @@ exclude. Notes when the corresponding filter was set:
- `brave_llm: include_domains_mode=prefer not applied`
- `brave_llm: min_score not applied`
Request URL unchanged aside from `q` (and exact-match quotes if we ever wrap;
we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes
across queries.
Request URL unchanged aside from `q` (do not wrap for `exact_match`; Brave
cannot apply it). Deduplicate identical notes across queries.
## 6. Pipeline
@@ -369,7 +374,8 @@ Owner-given invariants (these are where a free-form implementation drifts):
`min_score`.
- **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in
restrict-include, or is in exclude, is dropped. `prefer` keeps it.
Invalid domain with a space or `OR` → 400 `invalid search`.
Invalid domain with a space, e.g. `a.com OR b.com`, → 400 `invalid search`.
`oracle.com` is valid.
- **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still
appears.
- **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip
@@ -382,11 +388,14 @@ Owner-given invariants (these are where a free-form implementation drifts):
- Tavily outbound JSON: filters present when set; `include_answer` always
false; `auto_parameters` false; `search_depth` basic; no extract/crawl;
`include_domains_mode` absent unless include list set; never both
`time_range` and dates.
`time_range` and dates; `exact_match: true` with query containing `"phrase"`
→ flag true and query unchanged (still quoted); `exact_match: true` with no
quoted phrase → 400 `invalid search` hint `exact_match`.
- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
- Validation table for §2 (`invalid query` vs `invalid search` + hint).
- SearXNG: `site:` prefix for one domain; exact_match quotes; existing `"` → 400
`invalid query`; `time_range` / `categories` as query params not bangs.
- SearXNG: `site:` prefix for one domain; `exact_match` does not wrap `q`
(query already quoted by the agent); `time_range` / `categories` as query
params not bangs.
- Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still
drops off-domain hits.
- `/v2/search` golden unchanged (no new keys required, no `published_date`).
@@ -410,13 +419,11 @@ No new dependency. No `internal/api/api.go` growth if a new file is cleaner
## 13. Rollout
1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed).
2. straylight probes orion SearXNG `time_range=` and `categories=news` once and
posts the status codes in ~agent. If either 400s, amend §5 to the note path
before titan starts.
3. titan implements on `git.wntrmute.dev:kyle/sift` after Claude's re-approval
**and** the SearXNG probe. Split commits if it helps; one phase doc, not a
P6-style task-file plan (invariants are in §11). titan does not edit
1. Claude approved `73ff7b6` once E1–E3 are in (this SHA). No further spec round.
2. SearXNG probe already returned 200 for `time_range=` and `categories=news`.
3. titan implements on `git.wntrmute.dev:kyle/sift` from this SHA. Split commits
if it helps; one phase doc (invariants in §11 — titan writes those tests;
Claude checks them in the code review). titan does not edit
`internal/guard/`, `internal/auth/`, or `~/src/nixos`.
4. Claude reviews the code.
5. straylight bumps the orion pin.