P5b review: background embedder re-queues a failed document and waits for a busy gate; no empty vectors cached

Owner review of p5b (tasks 01-03 by Ornith):
- background.go: `seen` was never cleared, so a document whose background embedding failed
  (Dixie hiccup) or was abandoned could never be queued again: every later question on it stayed
  BM25 for the life of the process. Dedupe now covers only queued and in-progress documents.
- background.go: acquireGate gave up after one retry and dropped the document; the task said to
  keep retrying once a second. It now waits (background work has no deadline).
- retrieval.go/background.go: a vector the server left out is not cached as an empty vector.
- bm25.go: Scores on a nil *BM25 dereferenced it; returns nil.
Regression tests in review_test.go; both fail on the old code.

End-to-end against Dixie (local serve): Karpathy's thesis, 95 pages / 51 chunks: bm25 in 19 s,
hybrid (51 embedded) after the background; RP2040 datasheet, 642 pages / 284 chunks (a 422
before P5b): bm25 in 13.5 s, all 284 embedded within 90 s, hybrid answer correct with its quote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-26 19:02:51 -07:00
co-authored by Claude Opus 5.5
parent 8cdb3e8537
commit 64d08a358d
4 changed files with 118 additions and 19 deletions
+4 -1
View File
@@ -54,7 +54,10 @@ func NewBM25(docs []string) *BM25 {
// Scores returns one score per document, in order, each >= 0 and never NaN or Inf. An empty
// query, an empty index or a document without any query term scores 0.
func (b *BM25) Scores(query string) []float64 {
if b == nil || b.nDocs == 0 {
if b == nil {
return nil
}
if b.nDocs == 0 {
return make([]float64, len(b.toks))
}
out := make([]float64, len(b.toks))