64 lines
2.5 KiB
Markdown
64 lines
2.5 KiB
Markdown
# P5b task 01: page-aware chunks and BM25
|
||
|
||
**Branch:** `p5b` (`git switch -c p5b master` if it does not exist, else `git switch p5b`; `git status --short` must be empty, otherwise stop)
|
||
**Commit subject:** `digest: SplitPages keeps every chunk on its page; BM25 scorer`
|
||
|
||
## Goal
|
||
|
||
Two pure functions for task 02. `SplitPages` is `Split` for text with `--- page N ---`
|
||
markers (what `pdfdoc.Full` produces): a chunk that starts inside a page begins with
|
||
`--- page N (cont.) ---`, so every chunk names its page. `BM25` ranks chunks lexically.
|
||
|
||
## Files
|
||
|
||
- Copy: `internal/digest/splitpages_test.go`, `internal/digest/bm25_test.go`
|
||
- Create: `internal/digest/splitpages.go`, `internal/digest/bm25.go`
|
||
- Modify: `docs/implementer-log.md` only
|
||
|
||
## Interfaces
|
||
|
||
```go
|
||
package digest
|
||
|
||
// SplitPages splits text into chunks of at most maxTok tokens (EstTokens) on line boundaries,
|
||
// like Split. A marker line matching ^--- page (\d+) ---$ sets the current page; a chunk that
|
||
// starts after the first marker but not at a marker line begins with
|
||
// "--- page N (cont.) ---\n\n" for the current page N. Text with no marker lines gives exactly
|
||
// Split's result.
|
||
func SplitPages(text string, maxTok int) []string
|
||
|
||
type BM25 struct{ /* unexported */ }
|
||
|
||
// NewBM25 indexes docs (k1 = 1.2, b = 0.75). Terms are the same tokens as terms() — lower-case
|
||
// runs of letters and digits, at least 2 characters — but counted, not as a set.
|
||
func NewBM25(docs []string) *BM25
|
||
|
||
// Scores returns one score per document, in order, each >= 0 and never NaN or Inf. An empty
|
||
// query, an empty index or a document without any query term scores 0.
|
||
func (b *BM25) Scores(query string) []float64
|
||
```
|
||
|
||
## Rules
|
||
|
||
1. `SplitPages`: the continuation marker counts toward the chunk's budget (the test allows a
|
||
marker line of slack). Never split inside a line. Do not change `Split` (a P5a given test
|
||
pins it).
|
||
2. BM25 IDF: `ln(1 + (N − df + 0.5) / (df + 0.5))`, which is always positive. Average document
|
||
length over all documents; an all-empty corpus scores 0 everywhere.
|
||
|
||
## Steps
|
||
|
||
- [ ] **1.** Branch; copy the two given tests. **2. See them fail** (undefined names).
|
||
- [ ] **3.** `splitpages.go`, **4.** `bm25.go`.
|
||
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/digest/` → `ok`. **6.** `make gate`.
|
||
- [ ] **7.** Row `p5b/01-splitpages-bm25`; commit.
|
||
|
||
```sh
|
||
git add internal/digest docs/implementer-log.md
|
||
git commit -F .state/commit-msg.txt
|
||
```
|
||
|
||
## Done when
|
||
|
||
- The given tests pass, every existing digest test passes, `make gate` ok, given files byte-identical.
|