kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

p3: keep original URLs on RRF/dedupe; rerank=false is not a merge

Browse Source 
CanonicalURL is the dedupe key only. Hits keep the first source's
original URL, including a trailing slash. Dedupe has the same rule.
This commit is contained in:
kyle
2026-09-23 16:04:45 -07:00
parent b6e1cdf825
commit 1c57845ebd
4 changed files with 36 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/P3.md
+10 -6
View File
@@ -146,10 +146,10 @@ placeholder line the same way (empty, on-box fill).
3. Fan-in: one goroutine per enabled source, each with `search.timeout`. Collect whatever
arrived; a timeout or non-2xx is a `warnings` entry (`searxng: timeout`, `tavily: 429`)
and that source contributes nothing.
4. Dedupe on **canonical URL**: scheme+host lowercased, default ports stripped, fragment
dropped, trailing slash dropped on the path, `www.` left as-is (different sites). First
snippet wins; `source` becomes the first source that produced it; later sources only
affect rerank input if they brought a longer snippet (keep the longer one).
4. Dedupe on **canonical URL as a key only** (scheme+host lowercased, default ports
stripped, fragment dropped, trailing slash dropped on the path; `www.` left as-is).
The hit returned to the agent keeps the **first source's original URL**. First
source wins attribution; a later longer snippet replaces the text only.
5. Merge: reciprocal-rank fusion across source lists unless `rerank: false`.
A source with `no_rerank = true` keeps its own order as a prefix (first-source
wins on URL). Cosine/honcho-embed is not used for search.
@@ -165,8 +165,12 @@ Fan-in uses reciprocal-rank fusion (`1/(60+rank)`) over per-source lists. Search
does not call `honcho-embed`; Dixie saturation cannot fail a search.
`[search.tavily] no_rerank = true` keeps Tavily's own order at the front of the
result list; remaining sources still RRF. `rerank = false` skips RRF and keeps
source-fan-in order (SearXNG, then Brave, then Tavily).
result list; remaining sources still RRF.
**`rerank = false` is not a merge.** It concatenates in source order (SearXNG,
then Brave, then Tavily) and dedupes. Do not enable two sources with
`rerank = false` expecting RRF or Tavily-first. That only happens with
`rerank = true` (and `no_rerank` on Tavily if you want Tavily's order).
## 6. Guard boundary (SSRF)