p2.6: partition dcache by caller class; pin restricted workspace
Digest, chunk-embed, and question-embed keys include fleet|restricted so a fleet caller cannot observe a restricted cache hit. Digest.Run takes the class from permit_restricted. A permit_restricted caller whose keys.toml workspace is not guardian-star is refused at Search (no Honcho request) and flagged at serve start and keys reload. docs: P3 restricted callers search like the fleet; cache/logs isolate.
This commit is contained in:
+31
-32
@@ -209,9 +209,8 @@ the digest/embed tables).
|
||||
`hour_bucket` = UTC `YYYY-MM-DDTHH` of the request.
|
||||
`caller_class` = `restricted` if the caller is in `permit_restricted`, else `fleet`.
|
||||
This blocks a fleet caller from observing `cached: true` on a query only the
|
||||
restricted caller issued this hour. Restricted callers currently fail closed
|
||||
(§9) and never read or write this cache; the class is still in the key so a
|
||||
later option (b)/(c) cannot reopen the side channel.
|
||||
restricted caller issued this hour. Restricted callers read and write their own
|
||||
partition only.
|
||||
- Do not cache a result if any source failed or rerank fell back (§4 step 7).
|
||||
- TTL: `search.cache_ttl` (default 1 h). Hour-bucket already bounds freshness; TTL is
|
||||
the eviction/serve window if a process spans the hour mark.
|
||||
@@ -222,9 +221,7 @@ the digest/embed tables).
|
||||
|
||||
The URL/fetch cache stays in P4.
|
||||
|
||||
**Follow-up (not P3):** the P2 digest cache is content-addressed by page hash +
|
||||
question, so a weaker same-query side channel exists there too. Note it; do not
|
||||
fix it in P3.
|
||||
Digest/question-embedding keys are partitioned the same way in P2.6.
|
||||
|
||||
## 8. Cost, metrics, logs
|
||||
|
||||
@@ -245,26 +242,28 @@ not fail the search. Brave LLM Context gets a cap too if the key is billed; defa
|
||||
off until the bake-off shows a rate.
|
||||
|
||||
Logs: one INFO line `search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…`.
|
||||
Never log the query text beyond `q_len`, never keys, never snippets. `permit_restricted`
|
||||
callers: `host` is not a search field; do not log result hosts either.
|
||||
Never log the query text beyond `q_len`, never keys, never snippets, never hosts or
|
||||
URLs. For `permit_restricted` callers, also drop upstream error text; vendor warnings
|
||||
that name no target (`tavily: 429`) are fine.
|
||||
|
||||
## 9. Restricted callers — fail closed until Kyle picks
|
||||
## 9. Restricted callers (Kyle, 2026-09-23)
|
||||
|
||||
SearXNG is **not** private: on orion it forwards every query to bing, `braveapi`
|
||||
(Kyle's Brave key), wikipedia, github and arxiv. Loopback only hides the hop, not
|
||||
the vendor.
|
||||
**"Restricted" means no other agent may see this work, not limited in what it may do.**
|
||||
`permit_restricted` callers search **exactly like the fleet**: same sources, including
|
||||
Brave LLM Context and Tavily. There is no fail-closed 403 and no
|
||||
`search.permit_restricted_paid` knob.
|
||||
|
||||
Kyle (2026-09-23) confirmed these searches are sensitive. Until he picks one of:
|
||||
Confidentiality:
|
||||
|
||||
- **(a)** restricted callers get no search from sift at all
|
||||
- **(b)** a fixed, reduced SearXNG engine set for them
|
||||
- **(c)** the same sources as the fleet
|
||||
|
||||
…the code **fails closed**: `403` `search not permitted` for `permit_restricted`
|
||||
callers on `/v1/search` and `/v2/search`. No upstream call, no cache read or write,
|
||||
no query text in logs. Skill: do not fall through to Hermes `web_search`.
|
||||
|
||||
Do not implement (b) or (c) on a guess.
|
||||
- **Logs:** no query text, no hosts or URLs, and no upstream error text. Log `caller`,
|
||||
`q_len`, source names, `n`, and cache hit or miss only. Vendor warnings that name no
|
||||
target (`tavily: 429`) are fine.
|
||||
- **Caches:** every search-cache key includes `caller_class` (`fleet` | `restricted`).
|
||||
A fleet caller can never hit, or detect via `cached: true`, a restricted entry.
|
||||
Restricted callers still read and write their own partition. P2.6 does the same for
|
||||
digest and question-embedding keys in `dcache`.
|
||||
- **Query harvest (§10):** still exclude guardian-star logs and restricted callers.
|
||||
Kyle reviews the list before commit.
|
||||
|
||||
## 10. Bake-off (acceptance)
|
||||
|
||||
@@ -303,8 +302,9 @@ Plus the mechanical gates:
|
||||
5. Partial results: one source 500 + two 200 → 200 with `warnings`.
|
||||
6. Dixie busy → rerank fallback, still 200.
|
||||
7. Cache hit/miss metrics on a repeated query inside the hour bucket.
|
||||
8. `permit_restricted` caller: `403 search not permitted`, zero outbound (httptest
|
||||
spies); a fleet caller with the same query still hits the fakes.
|
||||
8. Restricted vs fleet: same query, different `caller_class` in the cache key;
|
||||
fleet cannot observe `cached: true` on a restricted write. Logs for restricted
|
||||
callers contain no query, host, URL, or upstream error text.
|
||||
|
||||
## 11. Code layout
|
||||
|
||||
@@ -332,9 +332,10 @@ Do not print the file. Tests use httptest upstreams, never live keys.
|
||||
- Dialer: configured loopback allowed; other loopback port refused; request-supplied
|
||||
URL ignored.
|
||||
- Rerank fallback when embed gate is full.
|
||||
- Cache key: same norm/query/sources/k/hour → hit; next hour → miss.
|
||||
- Cache key: same norm/query/sources/k/hour/caller_class → hit; other class → miss;
|
||||
next hour → miss.
|
||||
- Daily cap: Nth Tavily call skipped.
|
||||
- Restricted caller: 403, zero upstream requests, no cache access.
|
||||
- Restricted caller: same sources as fleet; logs have no query/host/URL; cache partitioned.
|
||||
- `/v2/search` golden: `success` + `data.web`.
|
||||
- Contract harness: `Firecrawl(...).search("contract", limit=3)` against sifttest
|
||||
(search fixtures on the local HTTP server, `-test-allow-prefix` already exists).
|
||||
@@ -354,8 +355,8 @@ hit, `title\nurl\nsnippet`, blank line between. `--json` = native envelope. Non-
|
||||
exits match fetch (5 = sift unreachable).
|
||||
|
||||
`skill/SKILL.md`: `sift search` becomes the first step for web search; `web_search`
|
||||
is the fallback when sift is unreachable — **except** for `permit_restricted`
|
||||
callers (403 from sift; do not fall through to Tavily). Still `sift fetch` for URLs.
|
||||
is the fallback when sift is unreachable. Restricted callers use the same path
|
||||
(same sources). Still `sift fetch` for URLs.
|
||||
|
||||
## 14. Rollout
|
||||
|
||||
@@ -371,8 +372,6 @@ callers (403 from sift; do not fall through to Tavily). Still `sift fetch` for U
|
||||
|
||||
## 15. Open questions for Kyle
|
||||
|
||||
1. §9 — restricted search: **(a)** none, **(b)** reduced SearXNG engines, **(c)** same
|
||||
as fleet. Code fails closed at 403 until this is picked.
|
||||
2. Tavily `daily_credit_cap` default (500 is a guess).
|
||||
3. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer;
|
||||
1. Tavily `daily_credit_cap` default (500 is a guess).
|
||||
2. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer;
|
||||
confirm we should not also call Brave Web Search.
|
||||
|
||||
Reference in New Issue
Block a user