# P5b task 01: page-aware chunks and BM25

**Branch:** `p5b` (`git switch -c p5b master` if it does not exist, else `git switch p5b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `digest: SplitPages keeps every chunk on its page; BM25 scorer`

## Goal

Two pure functions for task 02. `SplitPages` is `Split` for text with `--- page N ---`
markers (what `pdfdoc.Full` produces): a chunk that starts inside a page begins with
`--- page N (cont.) ---`, so every chunk names its page. `BM25` ranks chunks lexically.

## Files

- Copy: `internal/digest/splitpages_test.go`, `internal/digest/bm25_test.go`
- Create: `internal/digest/splitpages.go`, `internal/digest/bm25.go`
- Modify: `docs/implementer-log.md` only

## Interfaces

```go
package digest

// SplitPages splits text into chunks of at most maxTok tokens (EstTokens) on line boundaries,
// like Split. A marker line matching ^--- page (\d+) ---$ sets the current page; a chunk that
// starts after the first marker but not at a marker line begins with
// "--- page N (cont.) ---\n\n" for the current page N. Text with no marker lines gives exactly
// Split's result.
func SplitPages(text string, maxTok int) []string

type BM25 struct{ /* unexported */ }

// NewBM25 indexes docs (k1 = 1.2, b = 0.75). Terms are the same tokens as terms() — lower-case
// runs of letters and digits, at least 2 characters — but counted, not as a set.
func NewBM25(docs []string) *BM25

// Scores returns one score per document, in order, each >= 0 and never NaN or Inf. An empty
// query, an empty index or a document without any query term scores 0.
func (b *BM25) Scores(query string) []float64
```

## Rules

1. `SplitPages`: the continuation marker counts toward the chunk's budget (the test allows a
   marker line of slack). Never split inside a line. Do not change `Split` (a P5a given test
   pins it).
2. BM25 IDF: `ln(1 + (N − df + 0.5) / (df + 0.5))`, which is always positive. Average document
   length over all documents; an all-empty corpus scores 0 everywhere.

## Steps

- [ ] **1.** Branch; copy the two given tests. **2. See them fail** (undefined names).
- [ ] **3.** `splitpages.go`, **4.** `bm25.go`.
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/digest/` → `ok`. **6.** `make gate`.
- [ ] **7.** Row `p5b/01-splitpages-bm25`; commit.

```sh
git add internal/digest docs/implementer-log.md
git commit -F .state/commit-msg.txt
```

## Done when

- The given tests pass, every existing digest test passes, `make gate` ok, given files byte-identical.