From 2d376d3ad1ad3f9490a4778858e0941997431565 Mon Sep 17 00:00:00 2001 From: Kyle Isom Date: Wed, 23 Sep 2026 11:43:03 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20P3=20spec=20=E2=80=94=20search=20API,?= =?UTF-8?q?=20sources,=20cache,=20bake-off?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit No code. Default-on sources left to the bake-off. Restricted-caller paid-engine question left open for Kyle. --- docs/P3.md | 347 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 347 insertions(+) create mode 100644 docs/P3.md diff --git a/docs/P3.md b/docs/P3.md new file mode 100644 index 0000000..1afb370 --- /dev/null +++ b/docs/P3.md @@ -0,0 +1,347 @@ +# Sift P3 — search + +Drafted 2026-09-23 by titan. Reviewer: Claude (doc, guard boundary, contract tests). +Implementer: titan. Orion pin bump: straylight, after Claude approves. +Builds on P2.5 (`85113f9`). No P3 code until this doc is reviewed. + +## 1. Scope + +In: `POST /v1/search` (native, PLAN §4.2), `POST /v2/search` (Firecrawl-compatible, what +Hermes `web_search` actually sends), CLI `sift search`, per-source SearXNG / Brave LLM +Context / Tavily fan-in, canonical-URL dedupe, honcho-embed rerank, search-result cache, +per-source timeouts with partial results, cost counters and a daily cap on paid vendors, +contract tests for Hermes 0.19 and 0.21. + +Out: P4 URL/fetch cache; crawl/map; Tavily extract/crawl/answer; fetching result URLs +inside search (agents still `sift fetch` through the guard); P2.5 is already shipped. + +**Default-on sources are not chosen in this doc.** Each source is a config switch. The +bake-off in §10 decides which combination ships enabled on orion. + +## 2. Contract facts (verified 2026-09-23 on titan's hermes-agent tree) + +Both fleet Hermes versions speak Firecrawl through **`firecrawl-py==4.17.0`** (P1). Search +goes through the same client. + +**What Hermes 0.21 actually sends** (`plugins/web/firecrawl/provider.py`): + +- Keyless/raw client: `POST {api_url}/v2/search` with body exactly + `{"query": "", "limit": }`. Timeout 60 s. Bearer when `FIRECRAWL_API_KEY` is set. +- SDK path: `Firecrawl(api_url=…, api_key=…).search(query=query, limit=limit)`. +- Hermes tool default `limit` is 5 (`tools/web_tools.py`); cap elsewhere in the plugin + layer is 20 (`SEARCH_LIMIT_CAP`). + +Decode **leniently**: ignore unknown fields (`scrapeOptions`, `sources`, `tbs`, …). Search +never scrapes result URLs even if the SDK asks. + +**What Hermes reads** (`_extract_web_search_results` in the same file), in order: + +1. `data` if it is a list of objects +2. `data.web`, then `data.results`, then top-level `web` / `results` +3. SDK attribute `.web` + +PLAN §4.1: respond with `data.web`. Each hit needs `url`, `title`, `description`. Extra +keys are allowed (Hermes keeps them). Empty list is a successful search, not an error. + +**Response Hermes then wraps** (`plugins/web/_common.py`): + +```json +{"success": true, "data": {"web": [{"url": "…", "title": "…", "description": "…", "position": 1}]}} +``` + +`position` is added by some providers; Hermes does not require it. Sift sets it on `/v2/search` +(1-based after rerank) so the two Hermes versions see the same shape. + +**Errors:** same table as P1. Non-2xx uses `{"success": false, "error": "…"}`. Never 502 +(SDK retries only on 502). A search with zero sources configured, or every source failed +with no cached hit, is **422** `no search results` (not 500). Auth 401 unchanged. + +**Native** (`POST /v1/search`), PLAN §4.2: + +```json +{"query": "…", "question": "optional", "k": 8, "rerank": true, "digest": false} +``` + +```json +{"results": [{"url": "…", "title": "…", "snippet": "…", "score": 0.0, "source": "searxng|brave_llm|tavily"}], + "warnings": [], + "sizes": {"results": 8, "sources_ok": ["searxng"], "sources_failed": []}, + "timings": {"search_ms": 0, "rerank_ms": 0}, + "cached": false} +``` + +`question` is reserved for a later digest-of-results path; in P3 it is accepted and +ignored (with a warning if `digest: true`). `digest` defaults **false** and stays off. + +## 3. Sources + +Config, not the request. Each source has `enabled` and a fixed endpoint. Missing key or +`enabled = false` → source is skipped, no warning. + +```toml +[search] +k = 8 +timeout = "8s" # per source +rerank = true +digest = false +cache_ttl = "1h" + +[search.searxng] +enabled = false # bake-off decides +url = "http://127.0.0.1:8888" + +[search.brave_llm] +enabled = false +url = "https://api.search.brave.com/res/v1/llm/context" + +[search.tavily] +enabled = false +url = "https://api.tavily.com/search" +daily_credit_cap = 500 +``` + +Upstream URLs are only those strings. A request cannot change them. + +### 3.1 SearXNG (orion loopback) + +`GET {url}/search?q=&format=json&pageno=1`, `Accept: application/json`, 8 s. +Map `results[].url|title|content` → snippet. Sort by SearXNG `score` descending before +the shared fan-in. This is the Hermes SearXNG provider's request (`plugins/web/searxng/provider.py`). + +### 3.2 Brave LLM Context + +PLAN §2, verified earlier: `GET /res/v1/llm/context?q=…` with `BRAVE_API_KEY` from +`/var/lib/sift/secret.env` (0640 root:sift). **Do not read SearXNG's secret.env.** +Header `X-Subscription-Token`. Map `sources[]` / `grounding` URLs + snippets; drop the +model-written `grounding` answer text if present — we want sources, not Brave's summary. + +This is **not** Brave Web Search (`/res/v1/web/search`). SearXNG already runs `braveapi` +as an engine; doubling that is wasted spend. + +### 3.3 Tavily (Kyle, #inference-infrastructure) + +- Key only in `/var/lib/sift/secret.env` as `TAVILY_API_KEY`. Never in chat, logs, metrics + labels, or the Nix store. +- `POST {url}` JSON: `query`, `max_results` (min(k, 20)), `include_answer: false`, + `include_raw_content: false`, `include_images: false`. `Authorization: Bearer`. +- **Never request or return `answer`.** If Tavily still sends one, drop it before the + fan-in. Do not call `/extract`, `/crawl`, or `/map`. +- Map `results[].url|title|content` + `score`. Count `usage.credits` toward the daily cap. + +Nix activation already creates `secret.env` with a Brave placeholder; add a Tavily +placeholder line the same way (empty, on-box fill). + +## 4. Pipeline + +1. Auth (same bearer as fetch). Reject empty query / query > 1500 chars (Tavily's published + cap; apply to all sources) with 400 `invalid query`. +2. Cache lookup (§7). Hit → return, no upstream, `cached: true`. +3. Fan-in: one goroutine per enabled source, each with `search.timeout`. Collect whatever + arrived; a timeout or non-2xx is a `warnings` entry (`searxng: timeout`, `tavily: 429`) + and that source contributes nothing. +4. Dedupe on **canonical URL**: scheme+host lowercased, default ports stripped, fragment + dropped, trailing slash dropped on the path, `www.` left as-is (different sites). First + snippet wins; `source` becomes the first source that produced it; later sources only + affect rerank input if they brought a longer snippet (keep the longer one). +5. Rerank (§5) unless `rerank: false` or Dixie cannot take the call. +6. Truncate to `k` (native) or `limit` (`/v2/search`, default 5, max 20). +7. Store in the search cache. Return. + +Search **never GETs a result URL**. Agents `sift fetch` those through the P1 guard. + +## 5. Rerank and Dixie + +Embed the query and each remaining snippet with `honcho-embed` (same engine, same +`dixie.max_inflight` FIFO as digest). Cosine similarity → `score`. Restore nothing to +source order except on fallback. + +**Dixie saturation must not fail a search:** if the embed permit cannot be had within +`dixie.queue_wait`, or embed errors, or the queue is full → keep source-fan-in order +(SearXNG, then Brave, then Tavily, then original rank inside a source), warning +`rerank fell back: `. Never call chat/digest for search in P3. + +## 6. Guard boundary (SSRF) + +The P1 `Guard` dialer **blocks loopback**, which is where SearXNG lives. Search clients +must not use `Guard.Client()` / `fetch.Fetcher`. + +Each source owns a dedicated `http.Client` built at process start from its **config URL +only**: + +1. Parse the config URL: `http`/`https` only, no userinfo. SearXNG may be loopback; + Brave and Tavily must be public hostnames (`api.search.brave.com`, `api.tavily.com`). +2. Resolve that hostname once (and refresh on a long interval, e.g. 10 m). The dialer + `Control` hook allows **only those resolved IPs, only that port**. Anything else, + including a rebinding answer, is refused. +3. No cookies, no proxy env, no redirects to a different host (max 1 hop, same host). + Redirect to a different host → source failure, not a follow. +4. Request path/query are code-built (`/search?q=&format=json`, `/res/v1/llm/context?q=`, + `POST /search`). Query text is query-escaped; it never becomes a URL. + +That is not an SSRF hole: the agent cannot name the target. A test proves a request body +cannot point search at `http://127.0.0.1:11434` or `http://rift:8000`. A second test +proves the SearXNG client can dial the configured loopback URL and **cannot** dial a +different loopback port. + +`sift fetch` of a search result still goes through the guard as today. + +## 7. Search-result cache + +SQLite, same `/var/lib/sift/cache.db` as P2 digest cache, **new table** `search` (not +the digest/embed tables). + +- Key: `sha256(norm(query) | sources | k | hour_bucket)`. + `norm` = trim, collapse whitespace, lowercase (same as digest questions). + `sources` = sorted list of enabled source names that were actually attempted (not + the ones skipped for missing keys). + `hour_bucket` = UTC `YYYY-MM-DDTHH` of the request. +- TTL: `search.cache_ttl` (default 1 h). Hour-bucket already bounds freshness; TTL is + the eviction/serve window if a process spans the hour mark. +- Value: URLs, titles, snippets, scores, source names. **No page bodies, no Tavily + answers, no Honcho context.** +- Failure to read/write → warning, serve uncached. `/metrics`: `search_cache_hit`, + `search_cache_miss`. + +The URL/fetch cache stays in P4. + +## 8. Cost, metrics, logs + +`/metrics` (JSON, additive to P1/P2): + +- `search_queries{caller,source}` counts +- `search_credits{caller,source}` (Tavily `usage.credits`; Brave 1 per call if 2xx; + SearXNG 0) +- `search_errors{source,kind}` +- `search_cache_hit` / `search_cache_miss` +- `search_rerank_fallback` +- `search_daily_credits{source}` and `search_capped{source}` + +**Daily cap** (paid vendors, currently Tavily): calendar UTC day, in-process counter +plus a `search_cap` row in SQLite so a restart does not reset it. Hitting the cap +skips that source for the rest of the day (`warnings: tavily: daily cap`) and does +not fail the search. Brave LLM Context gets a cap too if the key is billed; default +off until the bake-off shows a rate. + +Logs: one INFO line `search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…`. +Never log the query text beyond `q_len`, never keys, never snippets. `permit_restricted` +callers: `host` is not a search field; do not log result hosts either. + +## 9. Restricted callers — open question for Kyle + +Should `permit_restricted` callers' queries go to third-party engines (Brave, Tavily) +at all? SearXNG is on-box; Brave and Tavily see the query in vendor logs. + +Do **not** decide this in implementation. Until Kyle answers, P3 code: + +- still serves search for those callers +- still redacts hosts in fetch logs (P2.5) +- does **not** special-case their search queries +- ships a config knob `search.permit_restricted_paid = false` **commented, unused**, + so flipping it later is one line + +List this as unresolved in the review ping. + +## 10. Bake-off (acceptance) + +A fixed set of **~30 real queries** harvested from recent fleet `web_search` use +(gateway/`agent.log` on straylight, talos, titan — the `web_search` / Tavily / +SearXNG lines). Deduped, no secrets, committed as `tests/testdata/search_queries.txt`. +Do not invent the set. + +For each query, run four configurations and record top-k URLs + latency + credits: + +1. SearXNG alone +2. SearXNG + Brave LLM Context +3. SearXNG + Brave LLM Context + Tavily +4. Current Hermes Tavily `web_search` (the live tool, not sift) + +Judged **blind** on whether the result set would get the agent to the right source +(would you `sift fetch` the correct URL from this list?). Scoring notes live next to +the query file; the judge does not see the configuration name. + +Report in this thread: + +- which configuration wins (this becomes default-on) +- cost per query per configuration (Tavily credits, Brave calls) +- p90 latency per configuration + +Plus the mechanical gates: + +1. `make test lint contract` green (new tests in §12). +2. Contract: firecrawl-py 4.17.0 `.search()` and a raw `POST /v2/search` both parse + through Hermes `_extract_web_search_results`. +3. Guard: search cannot be aimed at an arbitrary URL; SearXNG client cannot dial a + non-configured loopback port; `sift fetch` of a result is still guarded. +4. Tavily fixture: request body has `include_answer: false` and no extract/crawl; + response mapping drops `answer` if present. +5. Partial results: one source 500 + two 200 → 200 with `warnings`. +6. Dixie busy → rerank fallback, still 200. +7. Cache hit/miss metrics on a repeated query inside the hour bucket. + +## 11. Code layout + +``` +internal/search Source interface; SearXNG, BraveLLM, Tavily clients; fan-in; dedupe +internal/search/dial fixed-upstream http.Client (config URL → allowed IP:port) +internal/rerank honcho-embed cosine; fallback +internal/dcache add table search (§7) +internal/api POST /v1/search, POST /v2/search +cmd/sift search subcommand +tests/contract sdk_test.py grows a .search() case; raw JSON case +``` + +Secrets: `secret.env` loaded at serve start into the process env (or a small map). +Do not print the file. Tests use httptest upstreams, never live keys. + +## 12. Tests + +- Canonical URL table (fragment, case, default port, trailing slash). +- Fan-in: 3 fake sources, one timeout, one 500, one 200 → partial + warning. +- Tavily client: assert outbound JSON (`include_answer` false, no extract); drop + `answer` from inbound. +- Brave/SearXNG response mapping fixtures (from PLAN's live shapes + Hermes providers). +- Dialer: configured loopback allowed; other loopback port refused; request-supplied + URL ignored. +- Rerank fallback when embed gate is full. +- Cache key: same norm/query/sources/k/hour → hit; next hour → miss. +- Daily cap: Nth Tavily call skipped. +- `/v2/search` golden: `success` + `data.web`. +- Contract harness: `Firecrawl(...).search("contract", limit=3)` against sifttest + (search fixtures on the local HTTP server, `-test-allow-prefix` already exists). + Also `POST /v2/search` raw JSON → run the result through a copied + `_extract_web_search_results` (or import Hermes if the tree is present; otherwise + a 20-line replica of that function in the harness, with a comment pointing at the + Hermes path and a test that the replica still matches). + +## 13. CLI and skill + +``` +sift search "" [--k N] [--json] +``` + +Client config unchanged (`~/.config/sift/client.toml`). Default output: one line per +hit, `title\nurl\nsnippet`, blank line between. `--json` = native envelope. Non-zero +exits match fetch (5 = sift unreachable). + +`skill/SKILL.md`: `sift search` becomes the first step for web search; `web_search` +is the fallback when sift is unreachable. Still `sift fetch` for URLs. + +## 14. Rollout + +1. This doc reviewed by Claude (guard boundary + contract). +2. titan implements on `git.wntrmute.dev:kyle/sift`, evidence in this thread. +3. Claude reviews the code. +4. straylight bumps the orion pin in `~/src/nixos` committed master (titan does not + edit that repo). +5. Bake-off against live orion; Kyle picks default-on sources; second pin bump if + config changes. +6. Hermes `web.search_backend: firecrawl` on one host (talos first, same as P1) only + after the bake-off. + +## 15. Open questions for Kyle + +1. §9 — do `permit_restricted` queries go to Brave/Tavily? +2. Tavily `daily_credit_cap` default (500 is a guess). +3. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer; + confirm we should not also call Brave Web Search. +)