p2.6: partition dcache by caller class; pin restricted workspace

Digest, chunk-embed, and question-embed keys include fleet|restricted so
a fleet caller cannot observe a restricted cache hit. Digest.Run takes
the class from permit_restricted.

A permit_restricted caller whose keys.toml workspace is not
guardian-star is refused at Search (no Honcho request) and flagged at
serve start and keys reload.

docs: P3 restricted callers search like the fleet; cache/logs isolate.
This commit is contained in:
2026-09-23 11:51:39 -07:00
parent 7be4ed1d43
commit 3687636ed7
12 changed files with 202 additions and 67 deletions
+14 -8
View File
@@ -78,13 +78,15 @@ Out: `search` (P3), URL→content fetch cache (P4), knowledge graph. (Digest + c
Content-addressed, SQLite in `/var/lib/sift/cache.db` (`modernc.org/sqlite`), so there is no
staleness problem: a changed page has a different hash.
1. **Digest cache** — key `sha256(content_hash | normalised_question | model | prompt_version)`
→ digest text + `chunks_used`. Normalise the question: trim, collapse whitespace,
lowercase. Hit → no Dixie call at all (`digest.cached: true`). Bump `prompt_version` whenever
the prompt or chunking changes.
2. **Chunk-embedding cache** — key `(content_hash, chunk_index, embed_model)` → vector
1. **Digest cache** — key `sha256(content_hash | normalised_question | model | prompt_version | caller_class)`
→ digest text + `chunks_used`. `caller_class` is `fleet` or `restricted` (P2.6): a fleet
caller must never hit a restricted entry (`cached: true` would leak that the query was
asked). Normalise the question: trim, collapse whitespace, lowercase. Hit → no Dixie
call at all (`digest.cached: true`). Bump `prompt_version` whenever the prompt or
chunking changes.
2. **Chunk-embedding cache** — key `(content_hash, chunk_index, embed_model, caller_class)` → vector
(float32 blob). A second question on the same long page skips re-embedding. Also cache
the question embedding by normalised question (short TTL not needed; it's content-addressed).
the question embedding by normalised question **and caller class**.
Rules: **Honcho context is never cached** (per-workspace, changes constantly) — fetch it
fresh and attach to a cached digest; nothing in the cache is workspace-specific. Size-bound
@@ -93,7 +95,8 @@ failures are warnings, never errors. `/metrics`: hit/miss per cache, evictions,
The URL → cleaned-content **fetch cache stays in P4** (it needs TTL/staleness rules).
Tests: hit/miss on identical vs changed content; normalisation; prompt_version bump
invalidates; context not stored; eviction under size cap; corrupt DB → warn and bypass.
invalidates; caller_class partitions fleet vs restricted; context not stored; eviction
under size cap; corrupt DB → warn and bypass.
## 3. Context (`internal/context`, Honcho retrieval only)
@@ -104,7 +107,10 @@ invalidates; context not stored; eviction under size cap; corrupt DB → warn an
(`context.workspaces`); anything else → no context + WARN. **Restricted workspace
`guardian-star`**: allowed only when the authenticated caller is in
`context.permit_restricted` *and* that caller's mapped workspace is `guardian-star`.
A fleet caller remapped to it is still refused (tested). Workspace is never taken
A fleet caller remapped to it is still refused (tested). **P2.6 reverse:** a
`permit_restricted` caller whose mapped workspace is not `guardian-star` is
refused (ERROR log, no Honcho request, never fall back to `hermes-fleet`).
Checked at Search, at serve start, and on keys reload. Workspace is never taken
from the request.
- Query: the question if given, else title + top 3 headings.
- Return trimmed items: `{peer, content (≤300 chars), created_at, session}`; label clearly
+31 -32
View File
@@ -209,9 +209,8 @@ the digest/embed tables).
`hour_bucket` = UTC `YYYY-MM-DDTHH` of the request.
`caller_class` = `restricted` if the caller is in `permit_restricted`, else `fleet`.
This blocks a fleet caller from observing `cached: true` on a query only the
restricted caller issued this hour. Restricted callers currently fail closed
(§9) and never read or write this cache; the class is still in the key so a
later option (b)/(c) cannot reopen the side channel.
restricted caller issued this hour. Restricted callers read and write their own
partition only.
- Do not cache a result if any source failed or rerank fell back (§4 step 7).
- TTL: `search.cache_ttl` (default 1 h). Hour-bucket already bounds freshness; TTL is
the eviction/serve window if a process spans the hour mark.
@@ -222,9 +221,7 @@ the digest/embed tables).
The URL/fetch cache stays in P4.
**Follow-up (not P3):** the P2 digest cache is content-addressed by page hash +
question, so a weaker same-query side channel exists there too. Note it; do not
fix it in P3.
Digest/question-embedding keys are partitioned the same way in P2.6.
## 8. Cost, metrics, logs
@@ -245,26 +242,28 @@ not fail the search. Brave LLM Context gets a cap too if the key is billed; defa
off until the bake-off shows a rate.
Logs: one INFO line `search caller=… q_len=… sources=… n=… cache=hit|miss rerank_ms=…`.
Never log the query text beyond `q_len`, never keys, never snippets. `permit_restricted`
callers: `host` is not a search field; do not log result hosts either.
Never log the query text beyond `q_len`, never keys, never snippets, never hosts or
URLs. For `permit_restricted` callers, also drop upstream error text; vendor warnings
that name no target (`tavily: 429`) are fine.
## 9. Restricted callers — fail closed until Kyle picks
## 9. Restricted callers (Kyle, 2026-09-23)
SearXNG is **not** private: on orion it forwards every query to bing, `braveapi`
(Kyle's Brave key), wikipedia, github and arxiv. Loopback only hides the hop, not
the vendor.
**"Restricted" means no other agent may see this work, not limited in what it may do.**
`permit_restricted` callers search **exactly like the fleet**: same sources, including
Brave LLM Context and Tavily. There is no fail-closed 403 and no
`search.permit_restricted_paid` knob.
Kyle (2026-09-23) confirmed these searches are sensitive. Until he picks one of:
Confidentiality:
- **(a)** restricted callers get no search from sift at all
- **(b)** a fixed, reduced SearXNG engine set for them
- **(c)** the same sources as the fleet
…the code **fails closed**: `403` `search not permitted` for `permit_restricted`
callers on `/v1/search` and `/v2/search`. No upstream call, no cache read or write,
no query text in logs. Skill: do not fall through to Hermes `web_search`.
Do not implement (b) or (c) on a guess.
- **Logs:** no query text, no hosts or URLs, and no upstream error text. Log `caller`,
`q_len`, source names, `n`, and cache hit or miss only. Vendor warnings that name no
target (`tavily: 429`) are fine.
- **Caches:** every search-cache key includes `caller_class` (`fleet` | `restricted`).
A fleet caller can never hit, or detect via `cached: true`, a restricted entry.
Restricted callers still read and write their own partition. P2.6 does the same for
digest and question-embedding keys in `dcache`.
- **Query harvest (§10):** still exclude guardian-star logs and restricted callers.
Kyle reviews the list before commit.
## 10. Bake-off (acceptance)
@@ -303,8 +302,9 @@ Plus the mechanical gates:
5. Partial results: one source 500 + two 200 → 200 with `warnings`.
6. Dixie busy → rerank fallback, still 200.
7. Cache hit/miss metrics on a repeated query inside the hour bucket.
8. `permit_restricted` caller: `403 search not permitted`, zero outbound (httptest
spies); a fleet caller with the same query still hits the fakes.
8. Restricted vs fleet: same query, different `caller_class` in the cache key;
fleet cannot observe `cached: true` on a restricted write. Logs for restricted
callers contain no query, host, URL, or upstream error text.
## 11. Code layout
@@ -332,9 +332,10 @@ Do not print the file. Tests use httptest upstreams, never live keys.
- Dialer: configured loopback allowed; other loopback port refused; request-supplied
URL ignored.
- Rerank fallback when embed gate is full.
- Cache key: same norm/query/sources/k/hour → hit; next hour → miss.
- Cache key: same norm/query/sources/k/hour/caller_class → hit; other class → miss;
next hour → miss.
- Daily cap: Nth Tavily call skipped.
- Restricted caller: 403, zero upstream requests, no cache access.
- Restricted caller: same sources as fleet; logs have no query/host/URL; cache partitioned.
- `/v2/search` golden: `success` + `data.web`.
- Contract harness: `Firecrawl(...).search("contract", limit=3)` against sifttest
(search fixtures on the local HTTP server, `-test-allow-prefix` already exists).
@@ -354,8 +355,8 @@ hit, `title\nurl\nsnippet`, blank line between. `--json` = native envelope. Non-
exits match fetch (5 = sift unreachable).
`skill/SKILL.md`: `sift search` becomes the first step for web search; `web_search`
is the fallback when sift is unreachable — **except** for `permit_restricted`
callers (403 from sift; do not fall through to Tavily). Still `sift fetch` for URLs.
is the fallback when sift is unreachable. Restricted callers use the same path
(same sources). Still `sift fetch` for URLs.
## 14. Rollout
@@ -371,8 +372,6 @@ callers (403 from sift; do not fall through to Tavily). Still `sift fetch` for U
## 15. Open questions for Kyle
1. §9 — restricted search: **(a)** none, **(b)** reduced SearXNG engines, **(c)** same
as fleet. Code fails closed at 403 until this is picked.
2. Tavily `daily_credit_cap` default (500 is a guess).
3. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer;
1. Tavily `daily_credit_cap` default (500 is a guess).
2. Brave LLM Context vs leaving Brave entirely inside SearXNG: bake-off should answer;
confirm we should not also call Brave Web Search.
+4 -3
View File
@@ -235,9 +235,10 @@ data.
- Read: `search` endpoints only (fast, no LLM). Never dialectic from sift (30–50 s of 9B
generation, competes with the deriver).
- Write: **none in v1.** v2 option (Kyle to decide): a one-line provenance message
("<agent> read <url> for <question>") into a dedicated session of the caller's workspace,
so dialectic can later answer "what have we already read about X". Page content never.
- Write: **none in v1.** If P4 provenance writes are approved: write only to the
caller's mapped workspace, under the same invariant as reads — a
`permit_restricted` caller never writes to `hermes-fleet`, and a fleet caller
never writes to `guardian-star`. Page content never.
## 7. Auth, secrets, config