sift/docs/plans/p5b

P5b implementation plan: questions on long documents

For the implementing model: do not work from this file. The owner gives you one task file at a time. This file is the index for the owner and the reviewer.

Goal: P5a refuses question on a PDF longer than 200 digest chunks (~240k tokens) with "use grep or pages". After P5b such a question is answered: at once from BM25 over page-aware chunks, while the document's chunks are embedded in the background; once they are, later questions use today's hybrid score. Two facts found while planning:

• The embedding cache (dcache GetEmbed/PutEmbed, EmbedKey) exists but nothing uses it: every question re-embeds every chunk.
• Split is line-based, so a chunk that starts mid-page carries no --- page N --- marker and the model cannot cite its page.

Task	Package	Given
01-splitpages-bm25	internal/digest (pure functions)	digest/splitpages_test.go, digest/bm25_test.go
02-retrieval	internal/digest	digest/retrieval_test.go
03-api	internal/api, README.md, skill/SKILL.md	api/pdf_retrieval_test.go; replaces api/pdf_test.go (P5a: the 200-chunk default becomes 5000)

How this plan was made: acceptance tests first, no reference implementation; the given tests compiled against a panic-only skeleton on master 1311cf3 and failed for the intended reasons.

Changes during the run

(none yet)