Owner review of p5b (tasks 01-03 by Ornith):
- background.go: `seen` was never cleared, so a document whose background embedding failed
(Dixie hiccup) or was abandoned could never be queued again: every later question on it stayed
BM25 for the life of the process. Dedupe now covers only queued and in-progress documents.
- background.go: acquireGate gave up after one retry and dropped the document; the task said to
keep retrying once a second. It now waits (background work has no deadline).
- retrieval.go/background.go: a vector the server left out is not cached as an empty vector.
- bm25.go: Scores on a nil *BM25 dereferenced it; returns nil.
Regression tests in review_test.go; both fail on the old code.
End-to-end against Dixie (local serve): Karpathy's thesis, 95 pages / 51 chunks: bm25 in 19 s,
hybrid (51 embedded) after the background; RP2040 datasheet, 642 pages / 284 chunks (a 422
before P5b): bm25 in 13.5 s, all 284 embedded within 90 s, hybrid answer correct with its quote.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cached chunk embeddings change what a long document answers with: BM25 first,
embedding uncached chunks (or the whole document) in the background so the next
question on the same doc scores hybrid. selectText now returns its mode and the
number of chunks that had a vector at answer-build time; the digest cache stores
only hybrid answers, never a provisional bm25 one. The background worker batches
through a Dixie gate and WaitBackground polls for true idle.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
kyleandClaude Opus 5.5
