P5b and P5c plans: long-document retrieval (BM25 first, background embedding) and /v1/extract + sift extract; given tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plans/p5b/01-splitpages-bm25.md

@@ -0,0 +1,63 @@
+# P5b task 01: page-aware chunks and BM25
+
+**Branch:** `p5b` (`git switch -c p5b master` if it does not exist, else `git switch p5b`; `git status --short` must be empty, otherwise stop)
+**Commit subject:** `digest: SplitPages keeps every chunk on its page; BM25 scorer`
+
+## Goal
+
+Two pure functions for task 02. `SplitPages` is `Split` for text with `--- page N ---`
+markers (what `pdfdoc.Full` produces): a chunk that starts inside a page begins with
+`--- page N (cont.) ---`, so every chunk names its page. `BM25` ranks chunks lexically.
+
+## Files
+
+- Copy: `internal/digest/splitpages_test.go`, `internal/digest/bm25_test.go`
+- Create: `internal/digest/splitpages.go`, `internal/digest/bm25.go`
+- Modify: `docs/implementer-log.md` only
+
+## Interfaces
+
+```go
+package digest
+
+// SplitPages splits text into chunks of at most maxTok tokens (EstTokens) on line boundaries,
+// like Split. A marker line matching ^--- page (\d+) ---$ sets the current page; a chunk that
+// starts after the first marker but not at a marker line begins with
+// "--- page N (cont.) ---\n\n" for the current page N. Text with no marker lines gives exactly
+// Split's result.
+func SplitPages(text string, maxTok int) []string
+
+type BM25 struct{ /* unexported */ }
+
+// NewBM25 indexes docs (k1 = 1.2, b = 0.75). Terms are the same tokens as terms() — lower-case
+// runs of letters and digits, at least 2 characters — but counted, not as a set.
+func NewBM25(docs []string) *BM25
+
+// Scores returns one score per document, in order, each >= 0 and never NaN or Inf. An empty
+// query, an empty index or a document without any query term scores 0.
+func (b *BM25) Scores(query string) []float64
+```
+
+## Rules
+
+1. `SplitPages`: the continuation marker counts toward the chunk's budget (the test allows a
+   marker line of slack). Never split inside a line. Do not change `Split` (a P5a given test
+   pins it).
+2. BM25 IDF: `ln(1 + (N − df + 0.5) / (df + 0.5))`, which is always positive. Average document
+   length over all documents; an all-empty corpus scores 0 everywhere.
+
+## Steps
+
+- [ ] **1.** Branch; copy the two given tests. **2. See them fail** (undefined names).
+- [ ] **3.** `splitpages.go`, **4.** `bm25.go`.
+- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/digest/` → `ok`. **6.** `make gate`.
+- [ ] **7.** Row `p5b/01-splitpages-bm25`; commit.
+
+```sh
+git add internal/digest docs/implementer-log.md
+git commit -F .state/commit-msg.txt
+```
+
+## Done when
+
+- The given tests pass, every existing digest test passes, `make gate` ok, given files byte-identical.