P5b review: background embedder re-queues a failed document and waits for a busy gate; no empty vectors cached
Owner review of p5b (tasks 01-03 by Ornith): - background.go: `seen` was never cleared, so a document whose background embedding failed (Dixie hiccup) or was abandoned could never be queued again: every later question on it stayed BM25 for the life of the process. Dedupe now covers only queued and in-progress documents. - background.go: acquireGate gave up after one retry and dropped the document; the task said to keep retrying once a second. It now waits (background work has no deadline). - retrieval.go/background.go: a vector the server left out is not cached as an empty vector. - bm25.go: Scores on a nil *BM25 dereferenced it; returns nil. Regression tests in review_test.go; both fail on the old code. End-to-end against Dixie (local serve): Karpathy's thesis, 95 pages / 51 chunks: bm25 in 19 s, hybrid (51 embedded) after the background; RP2040 datasheet, 642 pages / 284 chunks (a 422 before P5b): bm25 in 13.5 s, all 284 embedded within 90 s, hybrid answer correct with its quote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -54,7 +54,10 @@ func NewBM25(docs []string) *BM25 {
|
||||
// Scores returns one score per document, in order, each >= 0 and never NaN or Inf. An empty
|
||||
// query, an empty index or a document without any query term scores 0.
|
||||
func (b *BM25) Scores(query string) []float64 {
|
||||
if b == nil || b.nDocs == 0 {
|
||||
if b == nil {
|
||||
return nil
|
||||
}
|
||||
if b.nDocs == 0 {
|
||||
return make([]float64, len(b.toks))
|
||||
}
|
||||
out := make([]float64, len(b.toks))
|
||||
|
||||
Reference in New Issue
Block a user