diff --git a/docs/P2.md b/docs/P2.md index c5bc6a7..bdb7d1d 100644 --- a/docs/P2.md +++ b/docs/P2.md @@ -13,8 +13,12 @@ requests** — the deriver stalled for ~20 min until llama-server was restarted. never add to that failure mode: - **Bounded in-flight to Dixie**: at most `dixie.max_inflight` (default **2**) digest/embed - calls from sift at once; a request that can't get a permit within 5 s **falls back to - `clean`** (with a warning) instead of queueing on Dixie. + calls from sift at once. Waiting requests sit in a **bounded FIFO queue inside sift** + (`dixie.queue_depth` default **8**, `dixie.queue_wait` default **20 s**); a request that + can't get a permit within the wait, or arrives to a full queue, **falls back to `clean`** + (with a warning). Sift's queue is visible and cancellable; Dixie's own queue never is — + never let requests queue *on Dixie*. (20 s wait + ~15 s digest fits Hermes's 60 s + `web_extract` budget.) - **Streaming requests** (`stream: true`) for chat completions, and cancel the context on timeout, so closing the connection actually stops generation (verify this with a test against a real llama-server: start a long digest, cancel at 2 s, confirm the slot frees — @@ -69,6 +73,28 @@ Out: `search` (P3), cache (P4), knowledge graph. 6. Fallbacks (always 200 with `warnings`): permit timeout, Dixie error, 45 s timeout, empty output → `clean` truncated to the output cap, `digest.fallback: ""`. +## 2a. Caches (pulled forward from P4, Kyle 2026-09-22) + +Content-addressed, SQLite in `/var/lib/sift/cache.db` (`modernc.org/sqlite`), so there is no +staleness problem: a changed page has a different hash. + +1. **Digest cache** — key `sha256(content_hash | normalised_question | model | prompt_version)` + → digest text + `chunks_used`. Normalise the question: trim, collapse whitespace, + lowercase. Hit → no Dixie call at all (`digest.cached: true`). Bump `prompt_version` whenever + the prompt or chunking changes. +2. **Chunk-embedding cache** — key `(content_hash, chunk_index, embed_model)` → vector + (float32 blob). A second question on the same long page skips re-embedding. Also cache + the question embedding by normalised question (short TTL not needed; it's content-addressed). + +Rules: **Honcho context is never cached** (per-workspace, changes constantly) — fetch it +fresh and attach to a cached digest; nothing in the cache is workspace-specific. Size-bound +the DB (default 500 MB, LRU eviction by last-hit time; evict in a background tick). Cache +failures are warnings, never errors. `/metrics`: hit/miss per cache, evictions, DB size. +The URL → cleaned-content **fetch cache stays in P4** (it needs TTL/staleness rules). + +Tests: hit/miss on identical vs changed content; normalisation; prompt_version bump +invalidates; context not stored; eviction under size cap; corrupt DB → warn and bypass. + ## 3. Context (`internal/context`, Honcho retrieval only) - `POST {honcho}/v3/workspaces/{ws}/search` `{"query": q, "limit": 5}`; response is a JSON @@ -126,8 +152,9 @@ digest_ms fallback`. Never log page content, questions longer than 80 chars, or - Fake Dixie (`httptest`, streaming SSE) for: prompt framing (document inside tags, system prompt present), `enable_thinking:false` + `max_tokens` sent, fallback on timeout / error / - empty, permit cap (3 concurrent requests with `max_inflight=2` → one falls back), context - cancel propagates (fake server observes the disconnect). + empty, permit cap + queue (with `max_inflight=2, queue_depth=1`: 2 run, 1 waits, the 4th + falls back immediately; a waiter past `queue_wait` falls back), context cancel propagates + (fake server observes the disconnect). - Chunk selection: document order restored, token budget respected, no-question path. - Context: allowlist, `guardian-star` refused even if configured, 3 s timeout → warning. - Recover: Wayback JSON parsing (available/unavailable), `id_` rewrite, fake-success fixtures