docs: P3 spec — search API, sources, cache, bake-off

No code. Default-on sources left to the bake-off. Restricted-caller
paid-engine question left open for Kyle.

docs/P3.md

@@ -0,0 +1,347 @@
+# Sift P3 — search
+
+Drafted 2026-09-23 by titan. Reviewer: Claude (doc, guard boundary, contract tests).
+Implementer: titan. Orion pin bump: straylight, after Claude approves.
+Builds on P2.5 (`85113f9`). No P3 code until this doc is reviewed.
+
+## 1. Scope
+
+In: `POST /v1/search` (native, PLAN §4.2), `POST /v2/search` (Firecrawl-compatible, what
+Hermes `web_search` actually sends), CLI `sift search`, per-source SearXNG / Brave LLM
+Context / Tavily fan-in, canonical-URL dedupe, honcho-embed rerank, search-result cache,
+per-source timeouts with partial results, cost counters and a daily cap on paid vendors,
+contract tests for Hermes 0.19 and 0.21.
+
+Out: P4 URL/fetch cache; crawl/map; Tavily extract/crawl/answer; fetching result URLs
+inside search (agents still `sift fetch` through the guard); P2.5 is already shipped.
+
+**Default-on sources are not chosen in this doc.** Each source is a config switch. The
+bake-off in §10 decides which combination ships enabled on orion.
+
+## 2. Contract facts (verified 2026-09-23 on titan's hermes-agent tree)
+
+Both fleet Hermes versions speak Firecrawl through **`firecrawl-py==4.17.0`** (P1). Search
+goes through the same client.
+
+**What Hermes 0.21 actually sends** (`plugins/web/firecrawl/provider.py`):
+
+- Keyless/raw client: `POST {api_url}/v2/search` with body exactly
+  `{"query": "<str>", "limit": <int>}`. Timeout 60 s. Bearer when `FIRECRAWL_API_KEY` is set.
+- SDK path: `Firecrawl(api_url=…, api_key=…).search(query=query, limit=limit)`.
+- Hermes tool default `limit` is 5 (`tools/web_tools.py`); cap elsewhere in the plugin
+  layer is 20 (`SEARCH_LIMIT_CAP`).
+
+Decode **leniently**: ignore unknown fields (`scrapeOptions`, `sources`, `tbs`, …). Search
+never scrapes result URLs even if the SDK asks.
+
+**What Hermes reads** (`_extract_web_search_results` in the same file), in order:
+
+1. `data` if it is a list of objects
+2. `data.web`, then `data.results`, then top-level `web` / `results`
+3. SDK attribute `.web`
+
+PLAN §4.1: respond with `data.web`. Each hit needs `url`, `title`, `description`. Extra
+keys are allowed (Hermes keeps them). Empty list is a successful search, not an error.
+
+**Response Hermes then wraps** (`plugins/web/_common.py`):
+
+```json
+{"success": true, "data": {"web": [{"url": "…", "title": "…", "description": "…", "position": 1}]}}
+```
+
+`position` is added by some providers; Hermes does not require it. Sift sets it on `/v2/search`
+(1-based after rerank) so the two Hermes versions see the same shape.
+
+**Errors:** same table as P1. Non-2xx uses `{"success": false, "error": "…"}`. Never 502
+(SDK retries only on 502). A search with zero sources configured, or every source failed
+with no cached hit, is **422** `no search results` (not 500). Auth 401 unchanged.
+
+**Native** (`POST /v1/search`), PLAN §4.2:
+
+```json
+{"query": "…", "question": "optional", "k": 8, "rerank": true, "digest": false}
+```
+
+```json
+{"results": [{"url": "…", "title": "…", "snippet": "…", "score": 0.0, "source": "searxng|brave_llm|tavily"}],
+ "warnings": [],
+ "sizes": {"results": 8, "sources_ok": ["searxng"], "sources_failed": []},
+ "timings": {"search_ms": 0, "rerank_ms": 0},
+ "cached": false}
+```
+
+`question` is reserved for a later digest-of-results path; in P3 it is accepted and
+ignored (with a warning if `digest: true`). `digest` defaults **false** and stays off.
+
+## 3. Sources
+
+Config, not the request. Each source has `enabled` and a fixed endpoint. Missing key or
+`enabled = false` → source is skipped, no warning.
+
+```toml
+[search]
+k = 8
+timeout = "8s"          # per source
+rerank = true
+digest = false
+cache_ttl = "1h"
+
+[search.searxng]
+enabled = false         # bake-off decides
+url = "http://127.0.0.1:8888"
+
+[search.brave_llm]
+enabled = false
+url = "https://api.search.brave.com/res/v1/llm/context"
+
+[search.tavily]
+enabled = false
+url = "https://api.tavily.com/search"
+daily_credit_cap = 500
+```
+
+Upstream URLs are only those strings. A request cannot change them.
+
+### 3.1 SearXNG (orion loopback)
+
+`GET {url}/search?q=<query>&format=json&pageno=1`, `Accept: application/json`, 8 s.
+Map `results[].url|title|content` → snippet. Sort by SearXNG `score` descending before
+the shared fan-in. This is the Hermes SearXNG provider's request (`plugins/web/searxng/provider.py`).
+
+### 3.2 Brave LLM Context
+
+PLAN §2, verified earlier: `GET /res/v1/llm/context?q=…` with `BRAVE_API_KEY` from
+`/var/lib/sift/secret.env` (0640 root:sift). **Do not read SearXNG's secret.env.**
+Header `X-Subscription-Token`. Map `sources[]` / `grounding` URLs + snippets; drop the
+model-written `grounding` answer text if present — we want sources, not Brave's summary.
+
+This is **not** Brave Web Search (`/res/v1/web/search`). SearXNG already runs `braveapi`
+as an engine; doubling that is wasted spend.
+
+### 3.3 Tavily (Kyle, #inference-infrastructure)
+
+- Key only in `/var/lib/sift/secret.env` as `TAVILY_API_KEY`. Never in chat, logs, metrics
+  labels, or the Nix store.
+- `POST {url}` JSON: `query`, `max_results` (min(k, 20)), `include_answer: false`,
+  `include_raw_content: false`, `include_images: false`. `Authorization: Bearer`.
+- **Never request or return `answer`.** If Tavily still sends one, drop it before the
+  fan-in. Do not call `/extract`, `/crawl`, or `/map`.
+- Map `results[].url|title|content` + `score`. Count `usage.credits` toward the daily cap.
+
+Nix activation already creates `secret.env` with a Brave placeholder; add a Tavily
+placeholder line the same way (empty, on-box fill).
+
+## 4. Pipeline
+
+1. Auth (same bearer as fetch). Reject empty query / query > 1500 chars (Tavily's published
+   cap; apply to all sources) with 400 `invalid query`.
+2. Cache lookup (§7). Hit → return, no upstream, `cached: true`.
+3. Fan-in: one goroutine per enabled source, each with `search.timeout`. Collect whatever
+   arrived; a timeout or non-2xx is a `warnings` entry (`searxng: timeout`, `tavily: 429`)
+   and that source contributes nothing.
+4. Dedupe on **canonical URL**: scheme+host lowercased, default ports stripped, fragment
+   dropped, trailing slash dropped on the path, `www.` left as-is (different sites). First
+   snippet wins; `source` becomes the first source that produced it; later sources only
+   affect rerank input if they brought a longer snippet (keep the longer one).
+5. Rerank (§5) unless `rerank: false` or Dixie cannot take the call.
+6. Truncate to `k` (native) or `limit` (`/v2/search`, default 5, max 20).
+7. Store in the search cache. Return.
+
+Search **never GETs a result URL**. Agents `sift fetch` those through the P1 guard.
+
+## 5. Rerank and Dixie
+
+Embed the query and each remaining snippet with `honcho-embed` (same engine, same
+`dixie.max_inflight` FIFO as digest). Cosine similarity → `score`. Restore nothing to
+source order except on fallback.
+
+**Dixie saturation must not fail a search:** if the embed permit cannot be had within
+`dixie.queue_wait`, or embed errors, or the queue is full → keep source-fan-in order
+(SearXNG, then Brave, then Tavily, then original rank inside a source), warning
+`rerank fell back: <reason>`. Never call chat/digest for search in P3.
+
+## 6. Guard boundary (SSRF)
+
+The P1 `Guard` dialer **blocks loopback**, which is where SearXNG lives. Search clients
+must not use `Guard.Client()` / `fetch.Fetcher`.
+
+Each source owns a dedicated `http.Client` built at process start from its **config URL
+only**:
+
+1. Parse the config URL: `http`/`https` only, no userinfo. SearXNG may be loopback;
+   Brave and Tavily must be public hostnames (`api.search.brave.com`, `api.tavily.com`).
+2. Resolve that hostname once (and refresh on a long interval, e.g. 10 m). The dialer
+   `Control` hook allows **only those resolved IPs, only that port**. Anything else,
+   including a rebinding answer, is refused.
+3. No cookies, no proxy env, no redirects to a different host (max 1 hop, same host).
+   Redirect to a different host → source failure, not a follow.
+4. Request path/query are code-built (`/search?q=&format=json`, `/res/v1/llm/context?q=`,
+   `POST /search`). Query text is query-escaped; it never becomes a URL.
+
+That is not an SSRF hole: the agent cannot name the target. A test proves a request body
+cannot point search at `http://127.0.0.1:11434` or `http://rift:8000`. A second test
+proves the SearXNG client can dial the configured loopback URL and **cannot** dial a
+different loopback port.
+
+`sift fetch` of a search result still goes through the guard as today.
+
+## 7. Search-result cache
+
+SQLite, same `/var/lib/sift/cache.db` as P2 digest cache, **new table** `search` (not
+the digest/embed tables).
+
+- Key: `sha256(norm(query) | sources | k | hour_bucket)`.
+  `norm` = trim, collapse whitespace, lowercase (same as digest questions).
+  `sources` = sorted list of enabled source names that were actually attempted (not
+  the ones skipped for missing keys).
+  `hour_bucket` = UTC `YYYY-MM-DDTHH` of the request.
+- TTL: `search.cache_ttl` (default 1 h). Hour-bucket already bounds freshness; TTL is
+  the eviction/serve window if a process spans the hour mark.
+- Value: URLs, titles, snippets, scores, source names. **No page bodies, no Tavily
+  answers, no Honcho context.**
+- Failure to read/write → warning, serve uncached. `/metrics`: `search_cache_hit`,
+  `search_cache_miss`.
+
+The URL/fetch cache stays in P4.
+
+## 8. Cost, metrics, logs
+
+`/metrics` (JSON, additive to P1/P2):
+
+- `search_queries{caller,source}` counts
+- `search_credits{caller,source}` (Tavily `usage.credits`; Brave 1 per call if 2xx;
+  SearXNG 0)
+- `search_errors{source,kind}`
+- `search_cache_hit` / `search_cache_miss`
+- `search_rerank_fallback`
+- `search_daily_credits{source}` and `search_capped{source}`
+
+**Daily cap** (paid vendors, currently Tavily): calendar UTC day, in-process counter
+plus a `search_cap` row in SQLite so a restart does not reset it. Hitting the cap
+skips that source for the rest of the day (`warnings: tavily: daily cap`) and does
+not fail the search. Brave LLM Context gets a cap too if the key is billed; default
+off until the bake-off shows a rate.
+
+Logs: one INFO line `search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…`.
+Never log the query text beyond `q_len`, never keys, never snippets. `permit_restricted`
+callers: `host` is not a search field; do not log result hosts either.
+
+## 9. Restricted callers — open question for Kyle
+
+Should `permit_restricted` callers' queries go to third-party engines (Brave, Tavily)
+at all? SearXNG is on-box; Brave and Tavily see the query in vendor logs.
+
+Do **not** decide this in implementation. Until Kyle answers, P3 code:
+
+- still serves search for those callers
+- still redacts hosts in fetch logs (P2.5)
+- does **not** special-case their search queries
+- ships a config knob `search.permit_restricted_paid = false` **commented, unused**,
+  so flipping it later is one line
+
+List this as unresolved in the review ping.
+
+## 10. Bake-off (acceptance)
+
+A fixed set of **~30 real queries** harvested from recent fleet `web_search` use
+(gateway/`agent.log` on straylight, talos, titan — the `web_search` / Tavily /
+SearXNG lines). Deduped, no secrets, committed as `tests/testdata/search_queries.txt`.
+Do not invent the set.
+
+For each query, run four configurations and record top-k URLs + latency + credits:
+
+1. SearXNG alone
+2. SearXNG + Brave LLM Context
+3. SearXNG + Brave LLM Context + Tavily
+4. Current Hermes Tavily `web_search` (the live tool, not sift)
+
+Judged **blind** on whether the result set would get the agent to the right source
+(would you `sift fetch` the correct URL from this list?). Scoring notes live next to
+the query file; the judge does not see the configuration name.
+
+Report in this thread:
+
+- which configuration wins (this becomes default-on)
+- cost per query per configuration (Tavily credits, Brave calls)
+- p90 latency per configuration
+
+Plus the mechanical gates:
+
+1. `make test lint contract` green (new tests in §12).
+2. Contract: firecrawl-py 4.17.0 `.search()` and a raw `POST /v2/search` both parse
+   through Hermes `_extract_web_search_results`.
+3. Guard: search cannot be aimed at an arbitrary URL; SearXNG client cannot dial a
+   non-configured loopback port; `sift fetch` of a result is still guarded.
+4. Tavily fixture: request body has `include_answer: false` and no extract/crawl;
+   response mapping drops `answer` if present.
+5. Partial results: one source 500 + two 200 → 200 with `warnings`.
+6. Dixie busy → rerank fallback, still 200.
+7. Cache hit/miss metrics on a repeated query inside the hour bucket.
+
+## 11. Code layout
+
+```
+internal/search     Source interface; SearXNG, BraveLLM, Tavily clients; fan-in; dedupe
+internal/search/dial  fixed-upstream http.Client (config URL → allowed IP:port)
+internal/rerank     honcho-embed cosine; fallback
+internal/dcache     add table search (§7)
+internal/api        POST /v1/search, POST /v2/search
+cmd/sift            search subcommand
+tests/contract      sdk_test.py grows a .search() case; raw JSON case
+```
+
+Secrets: `secret.env` loaded at serve start into the process env (or a small map).
+Do not print the file. Tests use httptest upstreams, never live keys.
+
+## 12. Tests
+
+- Canonical URL table (fragment, case, default port, trailing slash).
+- Fan-in: 3 fake sources, one timeout, one 500, one 200 → partial + warning.
+- Tavily client: assert outbound JSON (`include_answer` false, no extract); drop
+  `answer` from inbound.
+- Brave/SearXNG response mapping fixtures (from PLAN's live shapes + Hermes providers).
+- Dialer: configured loopback allowed; other loopback port refused; request-supplied
+  URL ignored.
+- Rerank fallback when embed gate is full.
+- Cache key: same norm/query/sources/k/hour → hit; next hour → miss.
+- Daily cap: Nth Tavily call skipped.
+- `/v2/search` golden: `success` + `data.web`.
+- Contract harness: `Firecrawl(...).search("contract", limit=3)` against sifttest
+  (search fixtures on the local HTTP server, `-test-allow-prefix` already exists).
+  Also `POST /v2/search` raw JSON → run the result through a copied
+  `_extract_web_search_results` (or import Hermes if the tree is present; otherwise
+  a 20-line replica of that function in the harness, with a comment pointing at the
+  Hermes path and a test that the replica still matches).
+
+## 13. CLI and skill
+
+```
+sift search "<query>" [--k N] [--json]
+```
+
+Client config unchanged (`~/.config/sift/client.toml`). Default output: one line per
+hit, `title\nurl\nsnippet`, blank line between. `--json` = native envelope. Non-zero
+exits match fetch (5 = sift unreachable).
+
+`skill/SKILL.md`: `sift search` becomes the first step for web search; `web_search`
+is the fallback when sift is unreachable. Still `sift fetch` for URLs.
+
+## 14. Rollout
+
+1. This doc reviewed by Claude (guard boundary + contract).
+2. titan implements on `git.wntrmute.dev:kyle/sift`, evidence in this thread.
+3. Claude reviews the code.
+4. straylight bumps the orion pin in `~/src/nixos` committed master (titan does not
+   edit that repo).
+5. Bake-off against live orion; Kyle picks default-on sources; second pin bump if
+   config changes.
+6. Hermes `web.search_backend: firecrawl` on one host (talos first, same as P1) only
+   after the bake-off.
+
+## 15. Open questions for Kyle
+
+1. §9 — do `permit_restricted` queries go to Brave/Tavily?
+2. Tavily `daily_credit_cap` default (500 is a guess).
+3. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer;
+   confirm we should not also call Brave Web Search.
+)