sift/docs/P3.md

Sift P3 — search

Drafted 2026-09-23 by titan. Reviewer: Claude (doc, guard boundary, contract tests). Implementer: titan. Orion pin bump: straylight, after Claude approves. Builds on P2.5 (85113f9). No P3 code until this doc is reviewed.

1. Scope

In: POST /v1/search (native, PLAN §4.2), POST /v2/search (Firecrawl-compatible, what Hermes web_search actually sends), CLI sift search, per-source SearXNG / Brave LLM Context / Tavily fan-in, canonical-URL dedupe, honcho-embed rerank, search-result cache, per-source timeouts with partial results, cost counters and a daily cap on paid vendors, contract tests for Hermes 0.19 and 0.21.

Out: P4 URL/fetch cache; crawl/map; Tavily extract/crawl/answer; fetching result URLs inside search (agents still sift fetch through the guard); P2.5 is already shipped.

Default-on sources are not chosen in this doc. Each source is a config switch. The bake-off in §10 decides which combination ships enabled on orion.

2. Contract facts (verified 2026-09-23 on titan's hermes-agent tree)

Both fleet Hermes versions speak Firecrawl through firecrawl-py==4.17.0 (P1). Search goes through the same client.

What Hermes 0.21 actually sends (plugins/web/firecrawl/provider.py):

• Keyless/raw client: POST {api_url}/v2/search with body exactly {"query": "<str>", "limit": <int>}. Timeout 60 s. Bearer when FIRECRAWL_API_KEY is set.
• SDK path: Firecrawl(api_url=…, api_key=…).search(query=query, limit=limit).
• Hermes tool default limit is 5 (tools/web_tools.py); cap elsewhere in the plugin layer is 20 (SEARCH_LIMIT_CAP).

Decode leniently: ignore unknown fields (scrapeOptions, sources, tbs, …). Search never scrapes result URLs even if the SDK asks.

What Hermes reads (_extract_web_search_results in the same file), in order:

1. data if it is a list of objects
2. data.web, then data.results, then top-level web / results
3. SDK attribute .web

PLAN §4.1: respond with data.web. Each hit needs url, title, description. Extra keys are allowed (Hermes keeps them). Empty list is a successful search, not an error.

Response Hermes then wraps (plugins/web/_common.py):

{"success": true, "data": {"web": [{"url": "…", "title": "…", "description": "…", "position": 1}]}}

position is added by some providers; Hermes does not require it. Sift sets it on /v2/search (1-based after rerank) so the two Hermes versions see the same shape.

Errors: same table as P1. Non-2xx uses {"success": false, "error": "…"}. Never 502 (SDK retries only on 502). A search with zero sources configured, or every source failed with no cached hit, is 422 no search results (not 500). Auth 401 unchanged.

Native (POST /v1/search), PLAN §4.2:

{"query": "…", "question": "optional", "k": 8, "rerank": true, "digest": false}

{"results": [{"url": "…", "title": "…", "snippet": "…", "score": 0.0, "source": "searxng|brave_llm|tavily"}],
 "warnings": [],
 "sizes": {"results": 8, "sources_ok": ["searxng"], "sources_failed": []},
 "timings": {"search_ms": 0, "rerank_ms": 0},
 "cached": false}

question is reserved for a later digest-of-results path; in P3 it is accepted and ignored (with a warning if digest: true). digest defaults false and stays off.

3. Sources

Config, not the request. Each source has enabled and a fixed endpoint. Missing key or enabled = false → source is skipped, no warning.

[search]
k = 8
timeout = "8s"          # per source
rerank = true
digest = false
cache_ttl = "1h"

[search.searxng]
enabled = false         # bake-off decides
url = "http://127.0.0.1:8888"

[search.brave_llm]
enabled = false
url = "https://api.search.brave.com/res/v1/llm/context"

[search.tavily]
enabled = false
url = "https://api.tavily.com/search"
daily_credit_cap = 500

Upstream URLs are only those strings. A request cannot change them.

3.1 SearXNG (orion loopback)

GET {url}/search?q=<query>&format=json&pageno=1, Accept: application/json, 8 s. Map results[].url|title|content → snippet. Sort by SearXNG score descending before the shared fan-in. This is the Hermes SearXNG provider's request (plugins/web/searxng/provider.py).

3.2 Brave LLM Context

PLAN §2, verified earlier: GET /res/v1/llm/context?q=… with BRAVE_API_KEY from /var/lib/sift/secret.env (0640 root:sift). Do not read SearXNG's secret.env. Header X-Subscription-Token. Map sources[] / grounding URLs + snippets; drop the model-written grounding answer text if present — we want sources, not Brave's summary.

This is not Brave Web Search (/res/v1/web/search). SearXNG already runs braveapi as an engine; doubling that is wasted spend.

3.3 Tavily (Kyle, #inference-infrastructure)

• Key only in /var/lib/sift/secret.env as TAVILY_API_KEY. Never in chat, logs, metrics labels, or the Nix store.
• POST {url} JSON: query, max_results (min(k, 20)), include_answer: false, include_raw_content: false, include_images: false. Authorization: Bearer.
• Never request or return answer. If Tavily still sends one, drop it before the fan-in. Do not call /extract, /crawl, or /map.
• Map results[].url|title|content + score. Count usage.credits toward the daily cap.

Nix activation already creates secret.env with a Brave placeholder; add a Tavily placeholder line the same way (empty, on-box fill).

4. Pipeline

1. Auth (same bearer as fetch). Reject empty query / query > 1500 chars (Tavily's published cap; apply to all sources) with 400 invalid query.
2. Cache lookup (§7). Hit → return, no upstream, cached: true.
3. Fan-in: one goroutine per enabled source, each with search.timeout. Collect whatever arrived; a timeout or non-2xx is a warnings entry (searxng: timeout, tavily: 429) and that source contributes nothing.
4. Dedupe on canonical URL: scheme+host lowercased, default ports stripped, fragment dropped, trailing slash dropped on the path, www. left as-is (different sites). First snippet wins; source becomes the first source that produced it; later sources only affect rerank input if they brought a longer snippet (keep the longer one).
5. Rerank (§5) unless rerank: false or Dixie cannot take the call.
6. Truncate to k (native) or limit (/v2/search, default 5, max 20).
7. Store in the search cache. Return.

Search never GETs a result URL. Agents sift fetch those through the P1 guard.

5. Rerank and Dixie

Embed the query and each remaining snippet with honcho-embed (same engine, same dixie.max_inflight FIFO as digest). Cosine similarity → score. Restore nothing to source order except on fallback.

Dixie saturation must not fail a search: if the embed permit cannot be had within dixie.queue_wait, or embed errors, or the queue is full → keep source-fan-in order (SearXNG, then Brave, then Tavily, then original rank inside a source), warning rerank fell back: <reason>. Never call chat/digest for search in P3.

6. Guard boundary (SSRF)

The P1 Guard dialer blocks loopback, which is where SearXNG lives. Search clients must not use Guard.Client() / fetch.Fetcher.

Each source owns a dedicated http.Client built at process start from its config URL only:

1. Parse the config URL: http/https only, no userinfo. SearXNG may be loopback; Brave and Tavily must be public hostnames (api.search.brave.com, api.tavily.com).
2. Resolve that hostname once (and refresh on a long interval, e.g. 10 m). The dialer Control hook allows only those resolved IPs, only that port. Anything else, including a rebinding answer, is refused.
3. No cookies, no proxy env, no redirects to a different host (max 1 hop, same host). Redirect to a different host → source failure, not a follow.
4. Request path/query are code-built (/search?q=&format=json, /res/v1/llm/context?q=, POST /search). Query text is query-escaped; it never becomes a URL.

That is not an SSRF hole: the agent cannot name the target. A test proves a request body cannot point search at http://127.0.0.1:11434 or http://rift:8000. A second test proves the SearXNG client can dial the configured loopback URL and cannot dial a different loopback port.

sift fetch of a search result still goes through the guard as today.

7. Search-result cache

SQLite, same /var/lib/sift/cache.db as P2 digest cache, new table search (not the digest/embed tables).

• Key: sha256(norm(query) | sources | k | hour_bucket). norm = trim, collapse whitespace, lowercase (same as digest questions). sources = sorted list of enabled source names that were actually attempted (not the ones skipped for missing keys). hour_bucket = UTC YYYY-MM-DDTHH of the request.
• TTL: search.cache_ttl (default 1 h). Hour-bucket already bounds freshness; TTL is the eviction/serve window if a process spans the hour mark.
• Value: URLs, titles, snippets, scores, source names. No page bodies, no Tavily answers, no Honcho context.
• Failure to read/write → warning, serve uncached. /metrics: search_cache_hit, search_cache_miss.

The URL/fetch cache stays in P4.

8. Cost, metrics, logs

/metrics (JSON, additive to P1/P2):

• search_queries{caller,source} counts
• search_credits{caller,source} (Tavily usage.credits; Brave 1 per call if 2xx; SearXNG 0)
• search_errors{source,kind}
• search_cache_hit / search_cache_miss
• search_rerank_fallback
• search_daily_credits{source} and search_capped{source}

Daily cap (paid vendors, currently Tavily): calendar UTC day, in-process counter plus a search_cap row in SQLite so a restart does not reset it. Hitting the cap skips that source for the rest of the day (warnings: tavily: daily cap) and does not fail the search. Brave LLM Context gets a cap too if the key is billed; default off until the bake-off shows a rate.

Logs: one INFO line search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…. Never log the query text beyond q_len, never keys, never snippets. permit_restricted callers: host is not a search field; do not log result hosts either.

9. Restricted callers — open question for Kyle

Should permit_restricted callers' queries go to third-party engines (Brave, Tavily) at all? SearXNG is on-box; Brave and Tavily see the query in vendor logs.

Do not decide this in implementation. Until Kyle answers, P3 code:

• still serves search for those callers
• still redacts hosts in fetch logs (P2.5)
• does not special-case their search queries
• ships a config knob search.permit_restricted_paid = false commented, unused, so flipping it later is one line

List this as unresolved in the review ping.

10. Bake-off (acceptance)

A fixed set of ~30 real queries harvested from recent fleet web_search use (gateway/agent.log on straylight, talos, titan — the web_search / Tavily / SearXNG lines). Deduped, no secrets, committed as tests/testdata/search_queries.txt. Do not invent the set.

For each query, run four configurations and record top-k URLs + latency + credits:

1. SearXNG alone
2. SearXNG + Brave LLM Context
3. SearXNG + Brave LLM Context + Tavily
4. Current Hermes Tavily web_search (the live tool, not sift)

Judged blind on whether the result set would get the agent to the right source (would you sift fetch the correct URL from this list?). Scoring notes live next to the query file; the judge does not see the configuration name.

Report in this thread:

• which configuration wins (this becomes default-on)
• cost per query per configuration (Tavily credits, Brave calls)
• p90 latency per configuration

Plus the mechanical gates:

1. make test lint contract green (new tests in §12).
2. Contract: firecrawl-py 4.17.0 .search() and a raw POST /v2/search both parse through Hermes _extract_web_search_results.
3. Guard: search cannot be aimed at an arbitrary URL; SearXNG client cannot dial a non-configured loopback port; sift fetch of a result is still guarded.
4. Tavily fixture: request body has include_answer: false and no extract/crawl; response mapping drops answer if present.
5. Partial results: one source 500 + two 200 → 200 with warnings.
6. Dixie busy → rerank fallback, still 200.
7. Cache hit/miss metrics on a repeated query inside the hour bucket.

11. Code layout

internal/search     Source interface; SearXNG, BraveLLM, Tavily clients; fan-in; dedupe
internal/search/dial  fixed-upstream http.Client (config URL → allowed IP:port)
internal/rerank     honcho-embed cosine; fallback
internal/dcache     add table search (§7)
internal/api        POST /v1/search, POST /v2/search
cmd/sift            search subcommand
tests/contract      sdk_test.py grows a .search() case; raw JSON case

Secrets: secret.env loaded at serve start into the process env (or a small map). Do not print the file. Tests use httptest upstreams, never live keys.

12. Tests

• Canonical URL table (fragment, case, default port, trailing slash).
• Fan-in: 3 fake sources, one timeout, one 500, one 200 → partial + warning.
• Tavily client: assert outbound JSON (include_answer false, no extract); drop answer from inbound.
• Brave/SearXNG response mapping fixtures (from PLAN's live shapes + Hermes providers).
• Dialer: configured loopback allowed; other loopback port refused; request-supplied URL ignored.
• Rerank fallback when embed gate is full.
• Cache key: same norm/query/sources/k/hour → hit; next hour → miss.
• Daily cap: Nth Tavily call skipped.
• /v2/search golden: success + data.web.
• Contract harness: Firecrawl(...).search("contract", limit=3) against sifttest (search fixtures on the local HTTP server, -test-allow-prefix already exists). Also POST /v2/search raw JSON → run the result through a copied _extract_web_search_results (or import Hermes if the tree is present; otherwise a 20-line replica of that function in the harness, with a comment pointing at the Hermes path and a test that the replica still matches).

13. CLI and skill

sift search "<query>" [--k N] [--json]

Client config unchanged (~/.config/sift/client.toml). Default output: one line per hit, title\nurl\nsnippet, blank line between. --json = native envelope. Non-zero exits match fetch (5 = sift unreachable).

skill/SKILL.md: sift search becomes the first step for web search; web_search is the fallback when sift is unreachable. Still sift fetch for URLs.

14. Rollout

1. This doc reviewed by Claude (guard boundary + contract).
2. titan implements on git.wntrmute.dev:kyle/sift, evidence in this thread.
3. Claude reviews the code.
4. straylight bumps the orion pin in ~/src/nixos committed master (titan does not edit that repo).
5. Bake-off against live orion; Kyle picks default-on sources; second pin bump if config changes.
6. Hermes web.search_backend: firecrawl on one host (talos first, same as P1) only after the bake-off.

15. Open questions for Kyle

1. §9 — do permit_restricted queries go to Brave/Tavily?
2. Tavily daily_credit_cap default (500 is a guess).
3. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer; confirm we should not also call Brave Web Search. )