docs: P3.1 E1–E3 exact_match quotes, domain OR, Warn comment
Agents quote phrases; exact_match requires a balanced quoted phrase and does not wrap. Domain rule no longer substring-rejects OR. Cap skip is not Warn. Titan may implement this SHA.
This commit is contained in:
+39
-32
@@ -3,8 +3,8 @@
|
||||
Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook
|
||||
patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan.
|
||||
|
||||
Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10). Not ready for
|
||||
titan until Claude re-approves this SHA.
|
||||
Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10) and of `73ff7b6`
|
||||
(E1–E3). Spec approved; titan implements this SHA.
|
||||
|
||||
P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily
|
||||
fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily
|
||||
@@ -84,17 +84,21 @@ both are sent). The example above shows the fields, not a legal combination.
|
||||
|
||||
- Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both
|
||||
set, or `queries` longer than 4, or any empty element → `invalid query`.
|
||||
Each string still has P3's 1500-rune cap (`invalid query`).
|
||||
- `exact_match: true` when the query (or any element of `queries`) already
|
||||
contains `"` → `invalid query` (do not nest quotes for SearXNG).
|
||||
Each string still has P3's 1500-rune cap (`invalid query`). ASCII `"` is
|
||||
allowed in queries (agents quote phrases themselves).
|
||||
- `exact_match: true` requires at least one balanced quoted phrase in **each**
|
||||
query (a `"`…`"` pair with one or more runes between; no unpaired `"`).
|
||||
Otherwise `invalid search` hint `exact_match`. Do not wrap or rewrite the
|
||||
query. `exact_match: false`/omitted: quotes are ordinary query text.
|
||||
- `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20
|
||||
exclude after normalise+dedupe. Empty list = omitted. Bad value →
|
||||
`invalid search` hint `include_domains` or `exclude_domains`.
|
||||
- Domain normalisation (apply before validation and before any `site:` rewrite):
|
||||
lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode
|
||||
`xn--` labels allowed); no scheme, port, path, wildcard, space, quote, or
|
||||
`OR`; at most 253 characters. Strict validation also stops operator injection
|
||||
into the SearXNG query.
|
||||
`xn--` labels allowed); no scheme, port, path, wildcard, or space; at most
|
||||
253 characters. Letters/digits/hyphens already make operator injection
|
||||
impossible (that needs a space). Do **not** substring-reject `OR` —
|
||||
`oracle.com` is valid. A space, e.g. `a.com OR b.com`, → 400.
|
||||
- `include_domains_mode`: omitted/`restrict`/`prefer`. Set without
|
||||
`include_domains` → `invalid search` hint `include_domains_mode`.
|
||||
Omitted with `include_domains` present means `restrict`.
|
||||
@@ -133,7 +137,7 @@ source's hits are discarded, it lands in `sources_failed`, it is counted in
|
||||
```
|
||||
type SourceOut struct {
|
||||
Hits []Hit
|
||||
Warn string // failure only (timeout, HTTP error, cap skip, pin)
|
||||
Warn string // failure only (timeout, HTTP error, pin)
|
||||
Notes []string // honesty: filters not applied, recall loss
|
||||
Credits int
|
||||
}
|
||||
@@ -159,11 +163,14 @@ When the request has filters, add only the fields that are set:
|
||||
- `include_domains_mode` **only when** `include_domains` is set
|
||||
- `time_range` **or** `start_date`/`end_date`, never both
|
||||
- `topic` when not `general` (omitted `topic` is `general`)
|
||||
- `exact_match` when true
|
||||
- `exact_match: true` when the flag is set (query text unchanged, including
|
||||
the agent's quotes)
|
||||
- `include_published_date: true` when `topic` is `news` or a date filter is set
|
||||
|
||||
Leave `filter_by_published_date` unset. Do not guess whether date filters use
|
||||
publish date without it; document that it is unset.
|
||||
Leave `filter_by_published_date` unset. Tavily's `time_range` and dates
|
||||
filter on publish date **or last updated date**; `filter_by_published_date`
|
||||
also drops undated results. Unset means undated and recently updated pages
|
||||
can pass the date filter.
|
||||
|
||||
Leave `include_usage` unset. `usage.credits` only comes back when requested;
|
||||
today's count is the code's fallback of 1, which is correct for `basic`.
|
||||
@@ -193,15 +200,14 @@ never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3);
|
||||
Never a bang in the query text.
|
||||
- `topic=news`: send `categories=news`. Never `!news`.
|
||||
- `topic=finance`: note `searxng: topic ignored` (no finance category).
|
||||
- `exact_match`: wrap the query in ASCII double quotes (rejected earlier if
|
||||
the query already contains `"`).
|
||||
- `exact_match`: send the query unchanged. SearXNG engines already treat
|
||||
quotes as phrases. Do not wrap again.
|
||||
- `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`.
|
||||
- `min_score > 0`: note `searxng: min_score not applied`; keep hits.
|
||||
|
||||
Live instance: straylight probes orion's SearXNG once **before titan starts**
|
||||
and posts whether `time_range=` and `categories=news` return 200. If either
|
||||
400s, this spec is amended to the note path for that param and titan must not
|
||||
send it. Tests are offline fixtures and cannot discover this.
|
||||
Live instance (straylight, 2026-09-29, not independently verified here):
|
||||
`time_range=month` and `categories=news` both returned 200 with normal result
|
||||
lists. Send both params. Tests are offline fixtures.
|
||||
|
||||
**Brave LLM Context**: no documented domain/date/topic/exact filters. Do not
|
||||
add a second Brave HTTP API. Post-filter still enforces restrict include and
|
||||
@@ -213,9 +219,8 @@ exclude. Notes when the corresponding filter was set:
|
||||
- `brave_llm: include_domains_mode=prefer not applied`
|
||||
- `brave_llm: min_score not applied`
|
||||
|
||||
Request URL unchanged aside from `q` (and exact-match quotes if we ever wrap;
|
||||
we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes
|
||||
across queries.
|
||||
Request URL unchanged aside from `q` (do not wrap for `exact_match`; Brave
|
||||
cannot apply it). Deduplicate identical notes across queries.
|
||||
|
||||
## 6. Pipeline
|
||||
|
||||
@@ -369,7 +374,8 @@ Owner-given invariants (these are where a free-form implementation drifts):
|
||||
`min_score`.
|
||||
- **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in
|
||||
restrict-include, or is in exclude, is dropped. `prefer` keeps it.
|
||||
Invalid domain with a space or `OR` → 400 `invalid search`.
|
||||
Invalid domain with a space, e.g. `a.com OR b.com`, → 400 `invalid search`.
|
||||
`oracle.com` is valid.
|
||||
- **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still
|
||||
appears.
|
||||
- **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip
|
||||
@@ -382,11 +388,14 @@ Owner-given invariants (these are where a free-form implementation drifts):
|
||||
- Tavily outbound JSON: filters present when set; `include_answer` always
|
||||
false; `auto_parameters` false; `search_depth` basic; no extract/crawl;
|
||||
`include_domains_mode` absent unless include list set; never both
|
||||
`time_range` and dates.
|
||||
`time_range` and dates; `exact_match: true` with query containing `"phrase"`
|
||||
→ flag true and query unchanged (still quoted); `exact_match: true` with no
|
||||
quoted phrase → 400 `invalid search` hint `exact_match`.
|
||||
- Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted.
|
||||
- Validation table for §2 (`invalid query` vs `invalid search` + hint).
|
||||
- SearXNG: `site:` prefix for one domain; exact_match quotes; existing `"` → 400
|
||||
`invalid query`; `time_range` / `categories` as query params not bangs.
|
||||
- SearXNG: `site:` prefix for one domain; `exact_match` does not wrap `q`
|
||||
(query already quoted by the agent); `time_range` / `categories` as query
|
||||
params not bangs.
|
||||
- Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still
|
||||
drops off-domain hits.
|
||||
- `/v2/search` golden unchanged (no new keys required, no `published_date`).
|
||||
@@ -410,13 +419,11 @@ No new dependency. No `internal/api/api.go` growth if a new file is cleaner
|
||||
|
||||
## 13. Rollout
|
||||
|
||||
1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed).
|
||||
2. straylight probes orion SearXNG `time_range=` and `categories=news` once and
|
||||
posts the status codes in ~agent. If either 400s, amend §5 to the note path
|
||||
before titan starts.
|
||||
3. titan implements on `git.wntrmute.dev:kyle/sift` after Claude's re-approval
|
||||
**and** the SearXNG probe. Split commits if it helps; one phase doc, not a
|
||||
P6-style task-file plan (invariants are in §11). titan does not edit
|
||||
1. Claude approved `73ff7b6` once E1–E3 are in (this SHA). No further spec round.
|
||||
2. SearXNG probe already returned 200 for `time_range=` and `categories=news`.
|
||||
3. titan implements on `git.wntrmute.dev:kyle/sift` from this SHA. Split commits
|
||||
if it helps; one phase doc (invariants in §11 — titan writes those tests;
|
||||
Claude checks them in the code review). titan does not edit
|
||||
`internal/guard/`, `internal/auth/`, or `~/src/nixos`.
|
||||
4. Claude reviews the code.
|
||||
5. straylight bumps the orion pin.
|
||||
|
||||
Reference in New Issue
Block a user