sift/docs/plans/p5b/01-splitpages-bm25.md

P5b task 01: page-aware chunks and BM25

Branch: p5b (git switch -c p5b master if it does not exist, else git switch p5b; git status --short must be empty, otherwise stop) Commit subject: digest: SplitPages keeps every chunk on its page; BM25 scorer

Goal

Two pure functions for task 02. SplitPages is Split for text with --- page N --- markers (what pdfdoc.Full produces): a chunk that starts inside a page begins with --- page N (cont.) ---, so every chunk names its page. BM25 ranks chunks lexically.

Files

• Copy: internal/digest/splitpages_test.go, internal/digest/bm25_test.go
• Create: internal/digest/splitpages.go, internal/digest/bm25.go
• Modify: docs/implementer-log.md only

Interfaces

package digest

// SplitPages splits text into chunks of at most maxTok tokens (EstTokens) on line boundaries,
// like Split. A marker line matching ^--- page (\d+) ---$ sets the current page; a chunk that
// starts after the first marker but not at a marker line begins with
// "--- page N (cont.) ---\n\n" for the current page N. Text with no marker lines gives exactly
// Split's result.
func SplitPages(text string, maxTok int) []string

type BM25 struct{ /* unexported */ }

// NewBM25 indexes docs (k1 = 1.2, b = 0.75). Terms are the same tokens as terms() — lower-case
// runs of letters and digits, at least 2 characters — but counted, not as a set.
func NewBM25(docs []string) *BM25

// Scores returns one score per document, in order, each >= 0 and never NaN or Inf. An empty
// query, an empty index or a document without any query term scores 0.
func (b *BM25) Scores(query string) []float64

Rules

1. SplitPages: the continuation marker counts toward the chunk's budget (the test allows a marker line of slack). Never split inside a line. Do not change Split (a P5a given test pins it).
2. BM25 IDF: ln(1 + (N − df + 0.5) / (df + 0.5)), which is always positive. Average document length over all documents; an all-empty corpus scores 0 everywhere.

Steps

• 1. Branch; copy the two given tests. 2. See them fail (undefined names).
• 3. splitpages.go, 4. bm25.go.
• 5. gofmt -w; go test -race -count=3 ./internal/digest/ → ok. 6. make gate.
• 7. Row p5b/01-splitpages-bm25; commit.

git add internal/digest docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The given tests pass, every existing digest test passes, make gate ok, given files byte-identical.