sift/docs/plans/p5a/05-pdfdoc.md

# P5a task 05: render an extracted document for an agent

**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `pdfdoc: page markers, the map for long documents, page ranges, grep`

## Goal

An agent's context is small and a thesis is 60k–150k tokens. This task is the pure function that
decides what an agent sees: the whole text with page markers when it fits; otherwise a **map**
(title, outline with page numbers, the first pages, and how to ask for more); a **page range**;
or **grep** results with page numbers. No I/O, no PDFs: it works on a `*pdftext.Doc`.

## Files

- Copy: `internal/pdfdoc/pdfdoc_test.go`
- Create: `internal/pdfdoc/pdfdoc.go` (`Render`, `Full`, `Marker`, the map, page ranges),
  `internal/pdfdoc/grep.go`
- Modify: `docs/implementer-log.md`

## Interfaces

```go
package pdfdoc

type Options struct {
	Pages  string // "" = no range; "a-b", "a" or "a-" (1-based, inclusive)
	Grep   string // "" = no grep; an RE2 pattern
	OutCap int    // characters (runes) the content may use
}

type Rendered struct {
	Content     string
	Map         bool   // Content is the map of a document longer than OutCap
	First, Last int    // pages whose text is in Content; 0, 0 for a grep answer
	GrepHits    int    // matches found (more may exist than are shown)
	Note        string // "" or how the answer was cut and how to continue
}

var (
	ErrBadPages = errors.New("invalid pages")
	ErrBadGrep  = errors.New("invalid grep")
)

func Marker(page int) string                                  // "--- page N ---"
func Full(doc *pdftext.Doc) string                            // every extracted page with its marker
func Render(doc *pdftext.Doc, opt Options) (Rendered, error)
```

## Rules

1. **Full.** For each extracted page p: `Marker(p) + "\n\n" + strings.TrimSpace(text)`, joined
   with `"\n\n"`. The given test shows the exact string.
2. **No Pages, no Grep.**
   - If the rune count of `Full(doc)` is at most `OutCap`: `Content = Full(doc)`, `First = 1`,
     `Last` = pages extracted.
   - Otherwise `Map = true` and the content is, in this order:
     ```
     # <Title, or "Untitled PDF">

     <Pages> pages · ~<T> tokens[ · pages 1-<E> extracted]

     ## Outline                        (only when the outline is not empty)
     - <title> (p. <page>)             (one line per item, indented two spaces per level below 1)

     This document is longer than one answer: ask for pages="a-b" or grep="pattern".

     <leading pages>
     ```
     - `T` is the rune count of `Full(doc)` divided by 4.
     - The `pages 1-E extracted` part appears only when fewer pages were extracted than
       `Pages`. The separator is " · " (U+00B7).
     - Leading pages are whole pages rendered as in `Full`, from page 1, as many as fit in
       `OutCap/4` runes. At least page 1 is shown, cut to fit if it alone is larger.
     - The whole content is at most `OutCap` runes; if the outline is too long, cut the outline,
       not the header or the instruction line.
     - `First = 1`, `Last` = the last whole page shown.
3. **Pages.**
   - Parse after trimming spaces: `a-b`, `a`, `a-` (a to the last extracted page). Anything else
     → `ErrBadPages`.
   - `a < 1`, `b < a`, or `a` past the last extracted page → an error wrapping `ErrBadPages`
     whose text is `pages out of range (1-E)`. `b` past the end is clamped to `E`.
   - At most **100 pages** and at most `OutCap` runes: whole pages from `a`. A single page larger
     than `OutCap` is cut to fit.
   - When the range was cut, `Note` says so and names the rest as `pages="k-b"`: for example
     `cut at page 13; ask for pages="14-40"`.
4. **Grep.**
   - The pattern must be at most 200 bytes and compile with `regexp` (RE2), else an error
     wrapping `ErrBadGrep`. Prefix `(?i)` unless the pattern contains an upper-case letter
     ("smart case").
   - Search every line of every page in the range (all extracted pages when `Pages` is empty).
   - The first line of the content is `grep "<pattern>": N matches` (`1 match` for one; `no matches`
     for none). Add `, showing the first 100` when N is over 100.
   - Then, per match, up to the first 100: the previous line of the same page as
     `p. P- <line>`, the matching line as `p. P: <line>`, the next line as `p. P- <line>`,
     then a `--` line. Lines are trimmed of surrounding spaces.
   - `GrepHits = N`, `First = Last = 0`. The content is at most `OutCap` runes (drop whole
     match blocks at the end and say so in `Note`). No match is not an error.
5. Every length here is in **runes**, never bytes: the cap protects an agent's context, and CJK
   text is three bytes a rune.

## Steps

- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined names).
- [ ] **3.** `pdfdoc.go`, **4.** `grep.go`.
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/pdfdoc/` → `ok`.
- [ ] **6.** `make gate`. **7.** Row `p5a/05-pdfdoc`; commit.

```sh
git add internal/pdfdoc docs/implementer-log.md
git commit -F .state/commit-msg.txt
```

## Done when

- The given test passes; `make gate` ok; given file byte-identical.