From 5c6a4a40ec144e763714b93fb2810058282bfcd3 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Tue, 29 Sep 2026 15:11:28 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20P3.1=20E1=E2=80=93E3=20exact=5Fmatch=20?= =?UTF-8?q?quotes,=20domain=20OR,=20Warn=20comment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Agents quote phrases; exact_match requires a balanced quoted phrase and does not wrap. Domain rule no longer substring-rejects OR. Cap skip is not Warn. Titan may implement this SHA. --- docs/P3.1.md | 71 +++++++++++++++++++++++++++++----------------------- 1 file changed, 39 insertions(+), 32 deletions(-) diff --git a/docs/P3.1.md b/docs/P3.1.md index b3a3c7d..f051457 100644 --- a/docs/P3.1.md +++ b/docs/P3.1.md @@ -3,8 +3,8 @@ Drafted 2026-09-29 by talos from Kyle's ask to steal useful Tavily-cookbook patterns for sift (thread in Mattermost). Reviewer: Claude. Implementer: titan. -Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10). Not ready for -titan until Claude re-approves this SHA. +Amended 2026-09-29 after Claude's review of `7f4f0d5` (R1–R10) and of `73ff7b6` +(E1–E3). Spec approved; titan implements this SHA. P3 shipped `/v1/search` + `/v2/search` with SearXNG / Brave LLM Context / Tavily fan-in, canonical-URL dedupe, RRF, cache, and a daily Tavily cap. The Tavily @@ -84,17 +84,21 @@ both are sent). The example above shows the fields, not a legal combination. - Exactly one of `query` (non-empty) or `queries` (1–4 non-empty strings). Both set, or `queries` longer than 4, or any empty element → `invalid query`. - Each string still has P3's 1500-rune cap (`invalid query`). -- `exact_match: true` when the query (or any element of `queries`) already - contains `"` → `invalid query` (do not nest quotes for SearXNG). + Each string still has P3's 1500-rune cap (`invalid query`). ASCII `"` is + allowed in queries (agents quote phrases themselves). +- `exact_match: true` requires at least one balanced quoted phrase in **each** + query (a `"`…`"` pair with one or more runes between; no unpaired `"`). + Otherwise `invalid search` hint `exact_match`. Do not wrap or rewrite the + query. `exact_match: false`/omitted: quotes are ordinary query text. - `include_domains` / `exclude_domains`: hostnames only. Max 20 include, 20 exclude after normalise+dedupe. Empty list = omitted. Bad value → `invalid search` hint `include_domains` or `exclude_domains`. - Domain normalisation (apply before validation and before any `site:` rewrite): lowercase; strip a trailing dot; ASCII letters/digits/hyphens (punycode - `xn--` labels allowed); no scheme, port, path, wildcard, space, quote, or - `OR`; at most 253 characters. Strict validation also stops operator injection - into the SearXNG query. + `xn--` labels allowed); no scheme, port, path, wildcard, or space; at most + 253 characters. Letters/digits/hyphens already make operator injection + impossible (that needs a space). Do **not** substring-reject `OR` — + `oracle.com` is valid. A space, e.g. `a.com OR b.com`, → 400. - `include_domains_mode`: omitted/`restrict`/`prefer`. Set without `include_domains` → `invalid search` hint `include_domains_mode`. Omitted with `include_domains` present means `restrict`. @@ -133,7 +137,7 @@ source's hits are discarded, it lands in `sources_failed`, it is counted in ``` type SourceOut struct { Hits []Hit - Warn string // failure only (timeout, HTTP error, cap skip, pin) + Warn string // failure only (timeout, HTTP error, pin) Notes []string // honesty: filters not applied, recall loss Credits int } @@ -159,11 +163,14 @@ When the request has filters, add only the fields that are set: - `include_domains_mode` **only when** `include_domains` is set - `time_range` **or** `start_date`/`end_date`, never both - `topic` when not `general` (omitted `topic` is `general`) -- `exact_match` when true +- `exact_match: true` when the flag is set (query text unchanged, including + the agent's quotes) - `include_published_date: true` when `topic` is `news` or a date filter is set -Leave `filter_by_published_date` unset. Do not guess whether date filters use -publish date without it; document that it is unset. +Leave `filter_by_published_date` unset. Tavily's `time_range` and dates +filter on publish date **or last updated date**; `filter_by_published_date` +also drops undated results. Unset means undated and recently updated pages +can pass the date filter. Leave `include_usage` unset. `usage.credits` only comes back when requested; today's count is the code's fallback of 1, which is correct for `basic`. @@ -193,15 +200,14 @@ never `Warn`. Domain restrict/exclude are enforced in sift after merge (§6.3); Never a bang in the query text. - `topic=news`: send `categories=news`. Never `!news`. - `topic=finance`: note `searxng: topic ignored` (no finance category). -- `exact_match`: wrap the query in ASCII double quotes (rejected earlier if - the query already contains `"`). +- `exact_match`: send the query unchanged. SearXNG engines already treat + quotes as phrases. Do not wrap again. - `start_date`/`end_date`: ignored + note `searxng: start_date/end_date not applied`. - `min_score > 0`: note `searxng: min_score not applied`; keep hits. -Live instance: straylight probes orion's SearXNG once **before titan starts** -and posts whether `time_range=` and `categories=news` return 200. If either -400s, this spec is amended to the note path for that param and titan must not -send it. Tests are offline fixtures and cannot discover this. +Live instance (straylight, 2026-09-29, not independently verified here): +`time_range=month` and `categories=news` both returned 200 with normal result +lists. Send both params. Tests are offline fixtures. **Brave LLM Context**: no documented domain/date/topic/exact filters. Do not add a second Brave HTTP API. Post-filter still enforces restrict include and @@ -213,9 +219,8 @@ exclude. Notes when the corresponding filter was set: - `brave_llm: include_domains_mode=prefer not applied` - `brave_llm: min_score not applied` -Request URL unchanged aside from `q` (and exact-match quotes if we ever wrap; -we do not — exact_match is Tavily + SearXNG only). Deduplicate identical notes -across queries. +Request URL unchanged aside from `q` (do not wrap for `exact_match`; Brave +cannot apply it). Deduplicate identical notes across queries. ## 6. Pipeline @@ -369,7 +374,8 @@ Owner-given invariants (these are where a free-form implementation drifts): `min_score`. - **R3 domain post-filter:** Brave (and SearXNG) hit whose host is not in restrict-include, or is in exclude, is dropped. `prefer` keeps it. - Invalid domain with a space or `OR` → 400 `invalid search`. + Invalid domain with a space, e.g. `a.com OR b.com`, → 400 `invalid search`. + `oracle.com` is valid. - **R4 multi-query RRF:** q1 returns k unique hits; q2's distinct top hit still appears. - **R5 cap mid-batch:** Tavily cap trips after query 1 → later queries skip @@ -382,11 +388,14 @@ Owner-given invariants (these are where a free-form implementation drifts): - Tavily outbound JSON: filters present when set; `include_answer` always false; `auto_parameters` false; `search_depth` basic; no extract/crawl; `include_domains_mode` absent unless include list set; never both - `time_range` and dates. + `time_range` and dates; `exact_match: true` with query containing `"phrase"` + → flag true and query unchanged (still quoted); `exact_match: true` with no + quoted phrase → 400 `invalid search` hint `exact_match`. - Tavily inbound: `answer` dropped; `published_date` mapped; missing date omitted. - Validation table for §2 (`invalid query` vs `invalid search` + hint). -- SearXNG: `site:` prefix for one domain; exact_match quotes; existing `"` → 400 - `invalid query`; `time_range` / `categories` as query params not bangs. +- SearXNG: `site:` prefix for one domain; `exact_match` does not wrap `q` + (query already quoted by the agent); `time_range` / `categories` as query + params not bangs. - Brave: request URL unchanged (aside from `q`); notes as §5; post-filter still drops off-domain hits. - `/v2/search` golden unchanged (no new keys required, no `published_date`). @@ -410,13 +419,11 @@ No new dependency. No `internal/api/api.go` growth if a new file is cleaner ## 13. Rollout -1. Claude re-reviews this amended doc (R1–R10 in, guard/contract still closed). -2. straylight probes orion SearXNG `time_range=` and `categories=news` once and - posts the status codes in ~agent. If either 400s, amend §5 to the note path - before titan starts. -3. titan implements on `git.wntrmute.dev:kyle/sift` after Claude's re-approval - **and** the SearXNG probe. Split commits if it helps; one phase doc, not a - P6-style task-file plan (invariants are in §11). titan does not edit +1. Claude approved `73ff7b6` once E1–E3 are in (this SHA). No further spec round. +2. SearXNG probe already returned 200 for `time_range=` and `categories=news`. +3. titan implements on `git.wntrmute.dev:kyle/sift` from this SHA. Split commits + if it helps; one phase doc (invariants in §11 — titan writes those tests; + Claude checks them in the code review). titan does not edit `internal/guard/`, `internal/auth/`, or `~/src/nixos`. 4. Claude reviews the code. 5. straylight bumps the orion pin.