sift/docs/P3.md

Sift P3 — search

Drafted 2026-09-23 by titan. Reviewer: Claude (doc, guard boundary, contract tests). Implementer: titan. Orion pin bump: straylight, after Claude approves P2.6 then this P3.

1. Scope

In: POST /v1/search (native, PLAN §4.2), POST /v2/search (Firecrawl-compatible, what Hermes web_search actually sends), CLI sift search, per-source SearXNG / Brave LLM Context / Tavily fan-in, canonical-URL dedupe, honcho-embed rerank, search-result cache, per-source timeouts with partial results, cost counters and a daily cap on paid vendors, contract tests for Hermes 0.19 and 0.21.

Out: P4 URL/fetch cache; crawl/map; Tavily extract/crawl/answer; fetching result URLs inside search (agents still sift fetch through the guard); P2.5 is already shipped.

Default-on sources are not chosen in this doc. Each source is a config switch. The bake-off in §10 decides which combination ships enabled on orion.

2. Contract facts (verified 2026-09-23 on titan's hermes-agent tree)

Both fleet Hermes versions speak Firecrawl through firecrawl-py==4.17.0 (P1). Search goes through the same client.

What Hermes 0.21 actually sends (plugins/web/firecrawl/provider.py):

• Keyless/raw client: POST {api_url}/v2/search with body exactly {"query": "<str>", "limit": <int>}. Timeout 60 s. Bearer when FIRECRAWL_API_KEY is set.
• SDK path: Firecrawl(api_url=…, api_key=…).search(query=query, limit=limit).
• Hermes tool default limit is 5 (tools/web_tools.py); cap elsewhere in the plugin layer is 20 (SEARCH_LIMIT_CAP).

Decode leniently: ignore unknown fields (scrapeOptions, sources, tbs, …). Search never scrapes result URLs even if the SDK asks.

What Hermes reads (_extract_web_search_results in the same file), in order:

1. data if it is a list of objects
2. data.web, then data.results, then top-level web / results
3. SDK attribute .web

PLAN §4.1: respond with data.web. Each hit needs url, title, description. Extra keys are allowed (Hermes keeps them). Empty list is a successful search, not an error.

Response Hermes then wraps (plugins/web/_common.py):

{"success": true, "data": {"web": [{"url": "…", "title": "…", "description": "…", "position": 1}]}}

position is added by some providers; Hermes does not require it. Sift sets it on /v2/search (1-based after rerank) so the two Hermes versions see the same shape.

Errors: same table as P1. Non-2xx uses {"success": false, "error": "…"}. Never 502 (SDK retries only on 502). A search with zero sources configured, or every source failed with no cached hit, is 422 no search results (not 500). Auth 401 unchanged.

Native (POST /v1/search), PLAN §4.2:

{"query": "…", "question": "optional", "k": 8, "rerank": true, "digest": false}

{"results": [{"url": "…", "title": "…", "snippet": "…", "score": 0.0, "source": "searxng|brave_llm|tavily"}],
 "warnings": [],
 "sizes": {"results": 8, "sources_ok": ["searxng"], "sources_failed": []},
 "timings": {"search_ms": 0, "rerank_ms": 0},
 "cached": false}

question is reserved for a later digest-of-results path; in P3 it is accepted and ignored (with a warning if digest: true). digest defaults false and stays off.

3. Sources

Config, not the request. Each source has enabled and a fixed endpoint. enabled = false → source is skipped, no warning. A source that is enabled but missing its key is not skipped silently: log once at startup, and /healthz lists it as degraded (search.<name>: missing key). Requests still run the other sources and add warnings: <name>: missing key.

[search]
k = 8
timeout = "8s"          # per source
rerank = true
digest = false
cache_ttl = "1h"

[search.searxng]
enabled = false         # bake-off decides
url = "http://127.0.0.1:8888"

[search.brave_llm]
enabled = false
url = "https://api.search.brave.com/res/v1/llm/context"

[search.tavily]
enabled = false
url = "https://api.tavily.com/search"
daily_credit_cap = 500

Upstream URLs are only those strings. A request cannot change them.

3.1 SearXNG (orion loopback)

GET {url}/search?q=<query>&format=json&pageno=1, Accept: application/json, 8 s. Map results[].url|title|content → snippet. Sort by SearXNG score descending before the shared fan-in. This is the Hermes SearXNG provider's request (plugins/web/searxng/provider.py).

3.2 Brave LLM Context

PLAN §2, verified earlier: GET /res/v1/llm/context?q=… with BRAVE_API_KEY from /var/lib/sift/secret.env (0640 root:sift). Do not read SearXNG's secret.env. Header X-Subscription-Token. Map result URLs and snippets from the live JSON (sources and any URL-bearing fields). Do not assume a model-written grounding answer exists — confirm against a live response before dropping or keeping any field that is not a URL/title/snippet.

This is not Brave Web Search (/res/v1/web/search). SearXNG already runs braveapi as an engine; doubling that is wasted spend.

3.3 Tavily (Kyle, #inference-infrastructure)

• Key only in /var/lib/sift/secret.env as TAVILY_API_KEY. Never in chat, logs, metrics labels, or the Nix store.
• POST {url} JSON: query, max_results (min(k, 20)), include_answer: false, include_raw_content: false, include_images: false, search_depth: "basic" (explicit, so credit cost is fixed). Authorization: Bearer.
• Never request or return answer. If Tavily still sends one, drop it before the fan-in. Do not call /extract, /crawl, or /map.
• Map results[].url|title|content + score. Count usage.credits toward the daily cap.

Nix activation already creates secret.env with a Brave placeholder; add a Tavily placeholder line the same way (empty, on-box fill).

4. Pipeline

1. Auth (same bearer as fetch). Reject empty query / query > 1500 chars (Tavily's published cap; apply to all sources) with 400 invalid query.
2. Cache lookup (§7). Hit → return, no upstream, cached: true.
3. Fan-in: one goroutine per enabled source, each with search.timeout. Collect whatever arrived; a timeout or non-2xx is a warnings entry (searxng: timeout, tavily: 429) and that source contributes nothing.
4. Dedupe on canonical URL: scheme+host lowercased, default ports stripped, fragment dropped, trailing slash dropped on the path, www. left as-is (different sites). First snippet wins; source becomes the first source that produced it; later sources only affect rerank input if they brought a longer snippet (keep the longer one).
5. Rerank (§5) unless rerank: false or Dixie cannot take the call.
6. Truncate to k (native) or limit (/v2/search, default 5, max 20).
7. Store in the search cache only if every attempted source succeeded and rerank did not fall back. A timeout or fallback must not pin a partial result for the rest of the hour (skip the write, or use a short TTL such as 2 m — skip is the default). Return.

Search never GETs a result URL. Agents sift fetch those through the P1 guard.

5. Rerank and Dixie

Embed the query and each remaining snippet with honcho-embed (same engine, same dixie.max_inflight FIFO as digest). Cosine similarity → score. Restore nothing to source order except on fallback.

Dixie saturation must not fail a search: if the embed permit cannot be had within dixie.queue_wait, or embed errors, or the queue is full → keep source-fan-in order (SearXNG, then Brave, then Tavily, then original rank inside a source), warning rerank fell back: <reason>. Never call chat/digest for search in P3.

6. Guard boundary (SSRF)

The P1 Guard dialer blocks loopback, which is where SearXNG lives. Search clients must not use Guard.Client() / fetch.Fetcher.

Each source owns a dedicated http.Client built at process start from its config URL only:

1. Parse the config URL: http/https only, no userinfo. SearXNG may be loopback; Brave and Tavily must be public hostnames (api.search.brave.com, api.tavily.com).
2. Resolve that hostname once (and refresh on a long interval, e.g. 10 m). A source marked public must resolve only to public addresses — run the guard's public-IP check (CheckAddr) on every record. A poisoned api.tavily.com → 127.0.0.1 is refused. The dialer Control hook allows only those resolved IPs, only that port. Anything else, including a rebinding answer, is refused.
3. TLS against system roots, no override, no InsecureSkipVerify. No cookies, no proxy env, no redirects to a different host (max 1 hop, same host). Redirect to a different host → source failure, not a follow.
4. Request path/query are code-built (/search?q=&format=json, /res/v1/llm/context?q=, POST /search). Query text is query-escaped; it never becomes a URL.

That is not an SSRF hole: the agent cannot name the target. A test proves a request body cannot point search at http://127.0.0.1:11434 or http://rift:8000. A second test proves the SearXNG client can dial the configured loopback URL and cannot dial a different loopback port.

sift fetch of a search result still goes through the guard as today.

7. Search-result cache

SQLite, same /var/lib/sift/cache.db as P2 digest cache, new table search (not the digest/embed tables).

• Key: sha256(norm(query) | sources | k | hour_bucket | caller_class). norm = trim, collapse whitespace, lowercase (same as digest questions). sources = sorted list of enabled source names that were actually attempted (not the ones skipped for missing keys). hour_bucket = UTC YYYY-MM-DDTHH of the request. caller_class = restricted if the caller is in permit_restricted, else fleet. This blocks a fleet caller from observing cached: true on a query only the restricted caller issued this hour. Restricted callers read and write their own partition only.
• Do not cache a result if any source failed or rerank fell back (§4 step 7).
• TTL: search.cache_ttl (default 1 h). Hour-bucket already bounds freshness; TTL is the eviction/serve window if a process spans the hour mark.
• Value: URLs, titles, snippets, scores, source names. No page bodies, no Tavily answers, no Honcho context.
• Failure to read/write → warning, serve uncached. /metrics: search_cache_hit, search_cache_miss.

The URL/fetch cache stays in P4.

Digest/question-embedding keys are partitioned the same way in P2.6.

8. Cost, metrics, logs

/metrics (JSON, additive to P1/P2):

• search_queries{caller,source} counts
• search_credits{caller,source} (Tavily usage.credits; Brave 1 per call if 2xx; SearXNG 0)
• search_errors{source,kind}
• search_cache_hit / search_cache_miss
• search_rerank_fallback
• search_daily_credits{source} and search_capped{source}

Daily cap (paid vendors, currently Tavily): calendar UTC day, in-process counter plus a search_cap row in SQLite so a restart does not reset it. Hitting the cap skips that source for the rest of the day (warnings: tavily: daily cap) and does not fail the search. Brave LLM Context gets a cap too if the key is billed; default off until the bake-off shows a rate.

Logs: one INFO line search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…. Never log the query text beyond q_len, never keys, never snippets, never hosts or URLs. For permit_restricted callers, also drop upstream error text; vendor warnings that name no target (tavily: 429) are fine.

9. Restricted callers (Kyle, 2026-09-23)

"Restricted" means no other agent may see this work, not limited in what it may do. permit_restricted callers search exactly like the fleet: same sources, including Brave LLM Context and Tavily. There is no fail-closed 403 and no search.permit_restricted_paid knob.

Confidentiality:

• Logs: no query text, no hosts or URLs, and no upstream error text. Log caller, q_len, source names, n, and cache hit or miss only. Vendor warnings that name no target (tavily: 429) are fine.
• Caches: every search-cache key includes caller_class (fleet | restricted). A fleet caller can never hit, or detect via cached: true, a restricted entry. Restricted callers still read and write their own partition. P2.6 does the same for digest and question-embedding keys in dcache.
• Query harvest (§10): still exclude guardian-star logs and restricted callers. Kyle reviews the list before commit.

10. Bake-off (acceptance)

A fixed set of 28 real queries harvested from recent fleet web_search use (gateway/agent.log on straylight, talos, titan). Exclude the guardian-star profile and anything from permit_restricted callers. Deduped, no secrets. Kyle approved the list; it is tests/testdata/search_queries.txt. Do not invent the set.

Sibling tests/testdata/search_targets.txt: for each query, the page or domain a good result set must contain, or open when there isn't a clear one.

For each query, run four configurations and record top-k URLs + latency + credits:

1. SearXNG alone
2. SearXNG + Brave LLM Context
3. SearXNG + Brave LLM Context + Tavily
4. Current Hermes Tavily web_search (the live tool, not sift)

Judged blind. The runner writes result packs labelled A–D with no config names. The A–D → 1–4 mapping is a separate key file the judge does not read until scoring is done. Protocol: tests/testdata/BAKEOFF.md. Shuffle helper: tests/bakeoff/blind.py. Judges: talos or Claude. Score: would you sift fetch the correct URL from this list?

Report in this thread:

• which configuration wins (this becomes default-on)
• cost per query per configuration (Tavily credits, Brave calls)
• p90 latency per configuration

Plus the mechanical gates:

1. make test lint contract green (new tests in §12).
2. Contract: firecrawl-py 4.17.0 .search() and a raw POST /v2/search both parse through Hermes _extract_web_search_results.
3. Guard: search cannot be aimed at an arbitrary URL; SearXNG client cannot dial a non-configured loopback port; sift fetch of a result is still guarded.
4. Tavily fixture: request body has include_answer: false and no extract/crawl; response mapping drops answer if present.
5. Partial results: one source 500 + two 200 → 200 with warnings.
6. Dixie busy → rerank fallback, still 200.
7. Cache hit/miss metrics on a repeated query inside the hour bucket.
8. Restricted vs fleet: same query, different caller_class in the cache key; fleet cannot observe cached: true on a restricted write. Logs for restricted callers contain no query, host, URL, or upstream error text.

11. Code layout

internal/search     Source interface; SearXNG, BraveLLM, Tavily clients; fan-in; dedupe
internal/search/dial  fixed-upstream http.Client (config URL → allowed IP:port)
internal/rerank     honcho-embed cosine; fallback
internal/dcache     add table search (§7)
internal/api        POST /v1/search, POST /v2/search
cmd/sift            search subcommand
tests/contract      sdk_test.py grows a .search() case; raw JSON case

Secrets: secret.env loaded at serve start into the process env (or a small map). Do not print the file. Tests use httptest upstreams, never live keys.

12. Tests

• Canonical URL table (fragment, case, default port, trailing slash).
• Fan-in: 3 fake sources, one timeout, one 500, one 200 → partial + warning.
• Tavily client: assert outbound JSON (include_answer false, search_depth basic, no extract); drop answer from inbound.
• Public-source dialer: poisoned private A record refused; TLS uses system roots.
• Brave/SearXNG response mapping fixtures (from PLAN's live shapes + Hermes providers).
• Dialer: configured loopback allowed; other loopback port refused; request-supplied URL ignored.
• Rerank fallback when embed gate is full.
• Cache key: same norm/query/sources/k/hour/caller_class → hit; other class → miss; next hour → miss.
• Daily cap: Nth Tavily call skipped.
• Restricted caller: same sources as fleet; logs have no query/host/URL; cache partitioned.
• /v2/search golden: success + data.web.
• Contract harness: Firecrawl(...).search("contract", limit=3) against sifttest (search fixtures on the local HTTP server, -test-allow-prefix already exists). Also POST /v2/search raw JSON → run the result through a copied _extract_web_search_results (or import Hermes if the tree is present; otherwise a 20-line replica of that function in the harness, with a comment pointing at the Hermes path and a test that the replica still matches).

13. CLI and skill

sift search "<query>" [--k N] [--json]

Client config unchanged (~/.config/sift/client.toml). Default output: one line per hit, title\nurl\nsnippet, blank line between. --json = native envelope. Non-zero exits match fetch (5 = sift unreachable).

skill/SKILL.md: sift search becomes the first step for web search; web_search is the fallback when sift is unreachable. Restricted callers use the same path (same sources). Still sift fetch for URLs.

14. Rollout

1. This doc reviewed by Claude (guard boundary + contract).
2. titan implements on git.wntrmute.dev:kyle/sift, evidence in this thread.
3. Claude reviews the code.
4. straylight bumps the orion pin in ~/src/nixos committed master (titan does not edit that repo).
5. Bake-off against live orion; Kyle picks default-on sources; second pin bump if config changes.
6. Hermes web.search_backend: firecrawl on one host (talos first, same as P1) only after the bake-off.

15. Open questions for Kyle

1. Tavily daily_credit_cap default (500 is a guess).
2. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer; confirm we should not also call Brave Web Search.