P5b review: background embedder re-queues a failed document and waits for a busy gate; no empty vectors cached

Owner review of p5b (tasks 01-03 by Ornith):
- background.go: `seen` was never cleared, so a document whose background embedding failed
  (Dixie hiccup) or was abandoned could never be queued again: every later question on it stayed
  BM25 for the life of the process. Dedupe now covers only queued and in-progress documents.
- background.go: acquireGate gave up after one retry and dropped the document; the task said to
  keep retrying once a second. It now waits (background work has no deadline).
- retrieval.go/background.go: a vector the server left out is not cached as an empty vector.
- bm25.go: Scores on a nil *BM25 dereferenced it; returns nil.
Regression tests in review_test.go; both fail on the old code.

End-to-end against Dixie (local serve): Karpathy's thesis, 95 pages / 51 chunks: bm25 in 19 s,
hybrid (51 embedded) after the background; RP2040 datasheet, 642 pages / 284 chunks (a 422
before P5b): bm25 in 13.5 s, all 284 embedded within 90 s, hybrid answer correct with its quote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

internal/digest/background.go

@@ -85,7 +85,10 @@ func (w *bgWorker) run() {
 
 		w.process(job)
 
+		// Forget the document whether it finished or not: dedupe covers only queued and
+		// in-progress work, so a later question re-queues a document whose embedding failed.
 		w.mu.Lock()
+		delete(w.seen, job.contentHash+"|"+job.callerClass)
 		w.inProg = false
 		w.mu.Unlock()
 	}
@@ -121,12 +124,7 @@ func (w *bgWorker) process(job embedJob) {
 		for k := range batch {
 			batch[k] = job.chunks[idx[b+k]]
 		}
-		if !w.acquireGate() {
-			if e.Log != nil {
-				e.Log.Warn("background embed gate")
-			}
-			return
-		}
+		w.acquireGate()
 		ectx, cancel := context.WithTimeout(context.Background(), e.timeout())
 		vecs, err := e.embed(ectx, batch)
 		cancel()
@@ -138,22 +136,22 @@ func (w *bgWorker) process(job embedJob) {
 			return
 		}
 		for k := range batch {
-			e.Cache.PutEmbed(key(idx[b+k]), vecs[k])
+			if k < len(vecs) && len(vecs[k]) > 0 {
+				e.Cache.PutEmbed(key(idx[b+k]), vecs[k])
+			}
 		}
 	}
 }
 
-// acquireGate takes the Dixie gate, retrying once per second on a refusal so interactive digests
-// get turns. It returns false only if the gate cannot be taken after the first retry, so the
-// document is not stuck behind a permanently busy gate.
-func (w *bgWorker) acquireGate() bool {
-	ok, _ := w.engine.Gate.Acquire(context.Background())
-	if ok {
-		return true
+// acquireGate takes the Dixie gate, sleeping a second after each refusal (queue full or wait
+// expired) so interactive digests get their turns. Background work has no deadline: it waits.
+func (w *bgWorker) acquireGate() {
+	for {
+		if ok, _ := w.engine.Gate.Acquire(context.Background()); ok {
+			return
+		}
+		time.Sleep(time.Second)
 	}
-	time.Sleep(1 * time.Second)
-	ok, _ = w.engine.Gate.Acquire(context.Background())
-	return ok
 }
 
 // WaitBackground returns when the background worker is idle: the queue empty and nothing embedding.