sift/docs/P3.md

# Sift P3 — search

Drafted 2026-09-23 by titan. Reviewer: Claude (doc, guard boundary, contract tests).
Implementer: titan. Orion pin bump: straylight, after Claude approves.
Builds on P2.5 (`85113f9`). No P3 code until this doc is reviewed.

## 1. Scope

In: `POST /v1/search` (native, PLAN §4.2), `POST /v2/search` (Firecrawl-compatible, what
Hermes `web_search` actually sends), CLI `sift search`, per-source SearXNG / Brave LLM
Context / Tavily fan-in, canonical-URL dedupe, honcho-embed rerank, search-result cache,
per-source timeouts with partial results, cost counters and a daily cap on paid vendors,
contract tests for Hermes 0.19 and 0.21.

Out: P4 URL/fetch cache; crawl/map; Tavily extract/crawl/answer; fetching result URLs
inside search (agents still `sift fetch` through the guard); P2.5 is already shipped.

**Default-on sources are not chosen in this doc.** Each source is a config switch. The
bake-off in §10 decides which combination ships enabled on orion.

## 2. Contract facts (verified 2026-09-23 on titan's hermes-agent tree)

Both fleet Hermes versions speak Firecrawl through **`firecrawl-py==4.17.0`** (P1). Search
goes through the same client.

**What Hermes 0.21 actually sends** (`plugins/web/firecrawl/provider.py`):

- Keyless/raw client: `POST {api_url}/v2/search` with body exactly
  `{"query": "<str>", "limit": <int>}`. Timeout 60 s. Bearer when `FIRECRAWL_API_KEY` is set.
- SDK path: `Firecrawl(api_url=…, api_key=…).search(query=query, limit=limit)`.
- Hermes tool default `limit` is 5 (`tools/web_tools.py`); cap elsewhere in the plugin
  layer is 20 (`SEARCH_LIMIT_CAP`).

Decode **leniently**: ignore unknown fields (`scrapeOptions`, `sources`, `tbs`, …). Search
never scrapes result URLs even if the SDK asks.

**What Hermes reads** (`_extract_web_search_results` in the same file), in order:

1. `data` if it is a list of objects
2. `data.web`, then `data.results`, then top-level `web` / `results`
3. SDK attribute `.web`

PLAN §4.1: respond with `data.web`. Each hit needs `url`, `title`, `description`. Extra
keys are allowed (Hermes keeps them). Empty list is a successful search, not an error.

**Response Hermes then wraps** (`plugins/web/_common.py`):

```json
{"success": true, "data": {"web": [{"url": "…", "title": "…", "description": "…", "position": 1}]}}
```

`position` is added by some providers; Hermes does not require it. Sift sets it on `/v2/search`
(1-based after rerank) so the two Hermes versions see the same shape.

**Errors:** same table as P1. Non-2xx uses `{"success": false, "error": "…"}`. Never 502
(SDK retries only on 502). A search with zero sources configured, or every source failed
with no cached hit, is **422** `no search results` (not 500). Auth 401 unchanged.

**Native** (`POST /v1/search`), PLAN §4.2:

```json
{"query": "…", "question": "optional", "k": 8, "rerank": true, "digest": false}
```

```json
{"results": [{"url": "…", "title": "…", "snippet": "…", "score": 0.0, "source": "searxng|brave_llm|tavily"}],
 "warnings": [],
 "sizes": {"results": 8, "sources_ok": ["searxng"], "sources_failed": []},
 "timings": {"search_ms": 0, "rerank_ms": 0},
 "cached": false}
```

`question` is reserved for a later digest-of-results path; in P3 it is accepted and
ignored (with a warning if `digest: true`). `digest` defaults **false** and stays off.

## 3. Sources

Config, not the request. Each source has `enabled` and a fixed endpoint. Missing key or
`enabled = false` → source is skipped, no warning.

```toml
[search]
k = 8
timeout = "8s"          # per source
rerank = true
digest = false
cache_ttl = "1h"

[search.searxng]
enabled = false         # bake-off decides
url = "http://127.0.0.1:8888"

[search.brave_llm]
enabled = false
url = "https://api.search.brave.com/res/v1/llm/context"

[search.tavily]
enabled = false
url = "https://api.tavily.com/search"
daily_credit_cap = 500
```

Upstream URLs are only those strings. A request cannot change them.

### 3.1 SearXNG (orion loopback)

`GET {url}/search?q=<query>&format=json&pageno=1`, `Accept: application/json`, 8 s.
Map `results[].url|title|content` → snippet. Sort by SearXNG `score` descending before
the shared fan-in. This is the Hermes SearXNG provider's request (`plugins/web/searxng/provider.py`).

### 3.2 Brave LLM Context

PLAN §2, verified earlier: `GET /res/v1/llm/context?q=…` with `BRAVE_API_KEY` from
`/var/lib/sift/secret.env` (0640 root:sift). **Do not read SearXNG's secret.env.**
Header `X-Subscription-Token`. Map `sources[]` / `grounding` URLs + snippets; drop the
model-written `grounding` answer text if present — we want sources, not Brave's summary.

This is **not** Brave Web Search (`/res/v1/web/search`). SearXNG already runs `braveapi`
as an engine; doubling that is wasted spend.

### 3.3 Tavily (Kyle, #inference-infrastructure)

- Key only in `/var/lib/sift/secret.env` as `TAVILY_API_KEY`. Never in chat, logs, metrics
  labels, or the Nix store.
- `POST {url}` JSON: `query`, `max_results` (min(k, 20)), `include_answer: false`,
  `include_raw_content: false`, `include_images: false`. `Authorization: Bearer`.
- **Never request or return `answer`.** If Tavily still sends one, drop it before the
  fan-in. Do not call `/extract`, `/crawl`, or `/map`.
- Map `results[].url|title|content` + `score`. Count `usage.credits` toward the daily cap.

Nix activation already creates `secret.env` with a Brave placeholder; add a Tavily
placeholder line the same way (empty, on-box fill).

## 4. Pipeline

1. Auth (same bearer as fetch). Reject empty query / query > 1500 chars (Tavily's published
   cap; apply to all sources) with 400 `invalid query`.
2. Cache lookup (§7). Hit → return, no upstream, `cached: true`.
3. Fan-in: one goroutine per enabled source, each with `search.timeout`. Collect whatever
   arrived; a timeout or non-2xx is a `warnings` entry (`searxng: timeout`, `tavily: 429`)
   and that source contributes nothing.
4. Dedupe on **canonical URL**: scheme+host lowercased, default ports stripped, fragment
   dropped, trailing slash dropped on the path, `www.` left as-is (different sites). First
   snippet wins; `source` becomes the first source that produced it; later sources only
   affect rerank input if they brought a longer snippet (keep the longer one).
5. Rerank (§5) unless `rerank: false` or Dixie cannot take the call.
6. Truncate to `k` (native) or `limit` (`/v2/search`, default 5, max 20).
7. Store in the search cache. Return.

Search **never GETs a result URL**. Agents `sift fetch` those through the P1 guard.

## 5. Rerank and Dixie

Embed the query and each remaining snippet with `honcho-embed` (same engine, same
`dixie.max_inflight` FIFO as digest). Cosine similarity → `score`. Restore nothing to
source order except on fallback.

**Dixie saturation must not fail a search:** if the embed permit cannot be had within
`dixie.queue_wait`, or embed errors, or the queue is full → keep source-fan-in order
(SearXNG, then Brave, then Tavily, then original rank inside a source), warning
`rerank fell back: <reason>`. Never call chat/digest for search in P3.

## 6. Guard boundary (SSRF)

The P1 `Guard` dialer **blocks loopback**, which is where SearXNG lives. Search clients
must not use `Guard.Client()` / `fetch.Fetcher`.

Each source owns a dedicated `http.Client` built at process start from its **config URL
only**:

1. Parse the config URL: `http`/`https` only, no userinfo. SearXNG may be loopback;
   Brave and Tavily must be public hostnames (`api.search.brave.com`, `api.tavily.com`).
2. Resolve that hostname once (and refresh on a long interval, e.g. 10 m). The dialer
   `Control` hook allows **only those resolved IPs, only that port**. Anything else,
   including a rebinding answer, is refused.
3. No cookies, no proxy env, no redirects to a different host (max 1 hop, same host).
   Redirect to a different host → source failure, not a follow.
4. Request path/query are code-built (`/search?q=&format=json`, `/res/v1/llm/context?q=`,
   `POST /search`). Query text is query-escaped; it never becomes a URL.

That is not an SSRF hole: the agent cannot name the target. A test proves a request body
cannot point search at `http://127.0.0.1:11434` or `http://rift:8000`. A second test
proves the SearXNG client can dial the configured loopback URL and **cannot** dial a
different loopback port.

`sift fetch` of a search result still goes through the guard as today.

## 7. Search-result cache

SQLite, same `/var/lib/sift/cache.db` as P2 digest cache, **new table** `search` (not
the digest/embed tables).

- Key: `sha256(norm(query) | sources | k | hour_bucket)`.
  `norm` = trim, collapse whitespace, lowercase (same as digest questions).
  `sources` = sorted list of enabled source names that were actually attempted (not
  the ones skipped for missing keys).
  `hour_bucket` = UTC `YYYY-MM-DDTHH` of the request.
- TTL: `search.cache_ttl` (default 1 h). Hour-bucket already bounds freshness; TTL is
  the eviction/serve window if a process spans the hour mark.
- Value: URLs, titles, snippets, scores, source names. **No page bodies, no Tavily
  answers, no Honcho context.**
- Failure to read/write → warning, serve uncached. `/metrics`: `search_cache_hit`,
  `search_cache_miss`.

The URL/fetch cache stays in P4.

## 8. Cost, metrics, logs

`/metrics` (JSON, additive to P1/P2):

- `search_queries{caller,source}` counts
- `search_credits{caller,source}` (Tavily `usage.credits`; Brave 1 per call if 2xx;
  SearXNG 0)
- `search_errors{source,kind}`
- `search_cache_hit` / `search_cache_miss`
- `search_rerank_fallback`
- `search_daily_credits{source}` and `search_capped{source}`

**Daily cap** (paid vendors, currently Tavily): calendar UTC day, in-process counter
plus a `search_cap` row in SQLite so a restart does not reset it. Hitting the cap
skips that source for the rest of the day (`warnings: tavily: daily cap`) and does
not fail the search. Brave LLM Context gets a cap too if the key is billed; default
off until the bake-off shows a rate.

Logs: one INFO line `search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…`.
Never log the query text beyond `q_len`, never keys, never snippets. `permit_restricted`
callers: `host` is not a search field; do not log result hosts either.

## 9. Restricted callers — open question for Kyle

Should `permit_restricted` callers' queries go to third-party engines (Brave, Tavily)
at all? SearXNG is on-box; Brave and Tavily see the query in vendor logs.

Do **not** decide this in implementation. Until Kyle answers, P3 code:

- still serves search for those callers
- still redacts hosts in fetch logs (P2.5)
- does **not** special-case their search queries
- ships a config knob `search.permit_restricted_paid = false` **commented, unused**,
  so flipping it later is one line

List this as unresolved in the review ping.

## 10. Bake-off (acceptance)

A fixed set of **~30 real queries** harvested from recent fleet `web_search` use
(gateway/`agent.log` on straylight, talos, titan — the `web_search` / Tavily /
SearXNG lines). Deduped, no secrets, committed as `tests/testdata/search_queries.txt`.
Do not invent the set.

For each query, run four configurations and record top-k URLs + latency + credits:

1. SearXNG alone
2. SearXNG + Brave LLM Context
3. SearXNG + Brave LLM Context + Tavily
4. Current Hermes Tavily `web_search` (the live tool, not sift)

Judged **blind** on whether the result set would get the agent to the right source
(would you `sift fetch` the correct URL from this list?). Scoring notes live next to
the query file; the judge does not see the configuration name.

Report in this thread:

- which configuration wins (this becomes default-on)
- cost per query per configuration (Tavily credits, Brave calls)
- p90 latency per configuration

Plus the mechanical gates:

1. `make test lint contract` green (new tests in §12).
2. Contract: firecrawl-py 4.17.0 `.search()` and a raw `POST /v2/search` both parse
   through Hermes `_extract_web_search_results`.
3. Guard: search cannot be aimed at an arbitrary URL; SearXNG client cannot dial a
   non-configured loopback port; `sift fetch` of a result is still guarded.
4. Tavily fixture: request body has `include_answer: false` and no extract/crawl;
   response mapping drops `answer` if present.
5. Partial results: one source 500 + two 200 → 200 with `warnings`.
6. Dixie busy → rerank fallback, still 200.
7. Cache hit/miss metrics on a repeated query inside the hour bucket.

## 11. Code layout

```
internal/search     Source interface; SearXNG, BraveLLM, Tavily clients; fan-in; dedupe
internal/search/dial  fixed-upstream http.Client (config URL → allowed IP:port)
internal/rerank     honcho-embed cosine; fallback
internal/dcache     add table search (§7)
internal/api        POST /v1/search, POST /v2/search
cmd/sift            search subcommand
tests/contract      sdk_test.py grows a .search() case; raw JSON case
```

Secrets: `secret.env` loaded at serve start into the process env (or a small map).
Do not print the file. Tests use httptest upstreams, never live keys.

## 12. Tests

- Canonical URL table (fragment, case, default port, trailing slash).
- Fan-in: 3 fake sources, one timeout, one 500, one 200 → partial + warning.
- Tavily client: assert outbound JSON (`include_answer` false, no extract); drop
  `answer` from inbound.
- Brave/SearXNG response mapping fixtures (from PLAN's live shapes + Hermes providers).
- Dialer: configured loopback allowed; other loopback port refused; request-supplied
  URL ignored.
- Rerank fallback when embed gate is full.
- Cache key: same norm/query/sources/k/hour → hit; next hour → miss.
- Daily cap: Nth Tavily call skipped.
- `/v2/search` golden: `success` + `data.web`.
- Contract harness: `Firecrawl(...).search("contract", limit=3)` against sifttest
  (search fixtures on the local HTTP server, `-test-allow-prefix` already exists).
  Also `POST /v2/search` raw JSON → run the result through a copied
  `_extract_web_search_results` (or import Hermes if the tree is present; otherwise
  a 20-line replica of that function in the harness, with a comment pointing at the
  Hermes path and a test that the replica still matches).

## 13. CLI and skill

```
sift search "<query>" [--k N] [--json]
```

Client config unchanged (`~/.config/sift/client.toml`). Default output: one line per
hit, `title\nurl\nsnippet`, blank line between. `--json` = native envelope. Non-zero
exits match fetch (5 = sift unreachable).

`skill/SKILL.md`: `sift search` becomes the first step for web search; `web_search`
is the fallback when sift is unreachable. Still `sift fetch` for URLs.

## 14. Rollout

1. This doc reviewed by Claude (guard boundary + contract).
2. titan implements on `git.wntrmute.dev:kyle/sift`, evidence in this thread.
3. Claude reviews the code.
4. straylight bumps the orion pin in `~/src/nixos` committed master (titan does not
   edit that repo).
5. Bake-off against live orion; Kyle picks default-on sources; second pin bump if
   config changes.
6. Hermes `web.search_backend: firecrawl` on one host (talos first, same as P1) only
   after the bake-off.

## 15. Open questions for Kyle

1. §9 — do `permit_restricted` queries go to Brave/Tavily?
2. Tavily `daily_credit_cap` default (500 is a guess).
3. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer;
   confirm we should not also call Brave Web Search.
)