sift/docs/plans/p5a/05-pdfdoc.md

P5a task 05: render an extracted document for an agent

Branch: p5a (git switch p5a; git status --short must be empty, otherwise stop) Commit subject: pdfdoc: page markers, the map for long documents, page ranges, grep

Goal

An agent's context is small and a thesis is 60k–150k tokens. This task is the pure function that decides what an agent sees: the whole text with page markers when it fits; otherwise a map (title, outline with page numbers, the first pages, and how to ask for more); a page range; or grep results with page numbers. No I/O, no PDFs: it works on a *pdftext.Doc.

Files

• Copy: internal/pdfdoc/pdfdoc_test.go
• Create: internal/pdfdoc/pdfdoc.go (Render, Full, Marker, the map, page ranges), internal/pdfdoc/grep.go
• Modify: docs/implementer-log.md

Interfaces

package pdfdoc

type Options struct {
	Pages  string // "" = no range; "a-b", "a" or "a-" (1-based, inclusive)
	Grep   string // "" = no grep; an RE2 pattern
	OutCap int    // characters (runes) the content may use
}

type Rendered struct {
	Content     string
	Map         bool   // Content is the map of a document longer than OutCap
	First, Last int    // pages whose text is in Content; 0, 0 for a grep answer
	GrepHits    int    // matches found (more may exist than are shown)
	Note        string // "" or how the answer was cut and how to continue
}

var (
	ErrBadPages = errors.New("invalid pages")
	ErrBadGrep  = errors.New("invalid grep")
)

func Marker(page int) string                                  // "--- page N ---"
func Full(doc *pdftext.Doc) string                            // every extracted page with its marker
func Render(doc *pdftext.Doc, opt Options) (Rendered, error)

Rules

1. Full. For each extracted page p: Marker(p) + "\n\n" + strings.TrimSpace(text), joined with "\n\n". The given test shows the exact string.
2. No Pages, no Grep.
   • If the rune count of Full(doc) is at most OutCap: Content = Full(doc), First = 1, Last = pages extracted.
   • Otherwise Map = true and the content is, in this order:

     # <Title, or "Untitled PDF">
     
     <Pages> pages · ~<T> tokens[ · pages 1-<E> extracted]
     
     ## Outline                        (only when the outline is not empty)
     - <title> (p. <page>)             (one line per item, indented two spaces per level below 1)
     
     This document is longer than one answer: ask for pages="a-b" or grep="pattern".
     
     <leading pages>
     

     • T is the rune count of Full(doc) divided by 4.
     • The pages 1-E extracted part appears only when fewer pages were extracted than Pages. The separator is " · " (U+00B7).
     • Leading pages are whole pages rendered as in Full, from page 1, as many as fit in OutCap/4 runes. At least page 1 is shown, cut to fit if it alone is larger.
     • The whole content is at most OutCap runes; if the outline is too long, cut the outline, not the header or the instruction line.
     • First = 1, Last = the last whole page shown.
3. Pages.
   • Parse after trimming spaces: a-b, a, a- (a to the last extracted page). Anything else → ErrBadPages.
   • a < 1, b < a, or a past the last extracted page → an error wrapping ErrBadPages whose text is pages out of range (1-E). b past the end is clamped to E.
   • At most 100 pages and at most OutCap runes: whole pages from a. A single page larger than OutCap is cut to fit.
   • When the range was cut, Note says so and names the rest as pages="k-b": for example cut at page 13; ask for pages="14-40".
4. Grep.
   • The pattern must be at most 200 bytes and compile with regexp (RE2), else an error wrapping ErrBadGrep. Prefix (?i) unless the pattern contains an upper-case letter ("smart case").
   • Search every line of every page in the range (all extracted pages when Pages is empty).
   • The first line of the content is grep "<pattern>": N matches (1 match for one; no matches for none). Add , showing the first 100 when N is over 100.
   • Then, per match, up to the first 100: the previous line of the same page as p. P- <line>, the matching line as p. P: <line>, the next line as p. P- <line>, then a -- line. Lines are trimmed of surrounding spaces.
   • GrepHits = N, First = Last = 0. The content is at most OutCap runes (drop whole match blocks at the end and say so in Note). No match is not an error.
5. Every length here is in runes, never bytes: the cap protects an agent's context, and CJK text is three bytes a rune.

Steps

• 1. git switch p5a; copy the given test. 2. See it fail (undefined names).
• 3. pdfdoc.go, 4. grep.go.
• 5. gofmt -w; go test -race -count=3 ./internal/pdfdoc/ → ok.
• 6. make gate. 7. Row p5a/05-pdfdoc; commit.

git add internal/pdfdoc docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The given test passes; make gate ok; given file byte-identical.