P5a plan: PDF documents — seven Ornith tasks with acceptance tests first
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,115 @@
|
||||
# P5a task 05: render an extracted document for an agent
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `pdfdoc: page markers, the map for long documents, page ranges, grep`
|
||||
|
||||
## Goal
|
||||
|
||||
An agent's context is small and a thesis is 60k–150k tokens. This task is the pure function that
|
||||
decides what an agent sees: the whole text with page markers when it fits; otherwise a **map**
|
||||
(title, outline with page numbers, the first pages, and how to ask for more); a **page range**;
|
||||
or **grep** results with page numbers. No I/O, no PDFs: it works on a `*pdftext.Doc`.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/pdfdoc/pdfdoc_test.go`
|
||||
- Create: `internal/pdfdoc/pdfdoc.go` (`Render`, `Full`, `Marker`, the map, page ranges),
|
||||
`internal/pdfdoc/grep.go`
|
||||
- Modify: `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package pdfdoc
|
||||
|
||||
type Options struct {
|
||||
Pages string // "" = no range; "a-b", "a" or "a-" (1-based, inclusive)
|
||||
Grep string // "" = no grep; an RE2 pattern
|
||||
OutCap int // characters (runes) the content may use
|
||||
}
|
||||
|
||||
type Rendered struct {
|
||||
Content string
|
||||
Map bool // Content is the map of a document longer than OutCap
|
||||
First, Last int // pages whose text is in Content; 0, 0 for a grep answer
|
||||
GrepHits int // matches found (more may exist than are shown)
|
||||
Note string // "" or how the answer was cut and how to continue
|
||||
}
|
||||
|
||||
var (
|
||||
ErrBadPages = errors.New("invalid pages")
|
||||
ErrBadGrep = errors.New("invalid grep")
|
||||
)
|
||||
|
||||
func Marker(page int) string // "--- page N ---"
|
||||
func Full(doc *pdftext.Doc) string // every extracted page with its marker
|
||||
func Render(doc *pdftext.Doc, opt Options) (Rendered, error)
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Full.** For each extracted page p: `Marker(p) + "\n\n" + strings.TrimSpace(text)`, joined
|
||||
with `"\n\n"`. The given test shows the exact string.
|
||||
2. **No Pages, no Grep.**
|
||||
- If the rune count of `Full(doc)` is at most `OutCap`: `Content = Full(doc)`, `First = 1`,
|
||||
`Last` = pages extracted.
|
||||
- Otherwise `Map = true` and the content is, in this order:
|
||||
```
|
||||
# <Title, or "Untitled PDF">
|
||||
|
||||
<Pages> pages · ~<T> tokens[ · pages 1-<E> extracted]
|
||||
|
||||
## Outline (only when the outline is not empty)
|
||||
- <title> (p. <page>) (one line per item, indented two spaces per level below 1)
|
||||
|
||||
This document is longer than one answer: ask for pages="a-b" or grep="pattern".
|
||||
|
||||
<leading pages>
|
||||
```
|
||||
- `T` is the rune count of `Full(doc)` divided by 4.
|
||||
- The `pages 1-E extracted` part appears only when fewer pages were extracted than
|
||||
`Pages`. The separator is " · " (U+00B7).
|
||||
- Leading pages are whole pages rendered as in `Full`, from page 1, as many as fit in
|
||||
`OutCap/4` runes. At least page 1 is shown, cut to fit if it alone is larger.
|
||||
- The whole content is at most `OutCap` runes; if the outline is too long, cut the outline,
|
||||
not the header or the instruction line.
|
||||
- `First = 1`, `Last` = the last whole page shown.
|
||||
3. **Pages.**
|
||||
- Parse after trimming spaces: `a-b`, `a`, `a-` (a to the last extracted page). Anything else
|
||||
→ `ErrBadPages`.
|
||||
- `a < 1`, `b < a`, or `a` past the last extracted page → an error wrapping `ErrBadPages`
|
||||
whose text is `pages out of range (1-E)`. `b` past the end is clamped to `E`.
|
||||
- At most **100 pages** and at most `OutCap` runes: whole pages from `a`. A single page larger
|
||||
than `OutCap` is cut to fit.
|
||||
- When the range was cut, `Note` says so and names the rest as `pages="k-b"`: for example
|
||||
`cut at page 13; ask for pages="14-40"`.
|
||||
4. **Grep.**
|
||||
- The pattern must be at most 200 bytes and compile with `regexp` (RE2), else an error
|
||||
wrapping `ErrBadGrep`. Prefix `(?i)` unless the pattern contains an upper-case letter
|
||||
("smart case").
|
||||
- Search every line of every page in the range (all extracted pages when `Pages` is empty).
|
||||
- The first line of the content is `grep "<pattern>": N matches` (`1 match` for one; `no matches`
|
||||
for none). Add `, showing the first 100` when N is over 100.
|
||||
- Then, per match, up to the first 100: the previous line of the same page as
|
||||
`p. P- <line>`, the matching line as `p. P: <line>`, the next line as `p. P- <line>`,
|
||||
then a `--` line. Lines are trimmed of surrounding spaces.
|
||||
- `GrepHits = N`, `First = Last = 0`. The content is at most `OutCap` runes (drop whole
|
||||
match blocks at the end and say so in `Note`). No match is not an error.
|
||||
5. Every length here is in **runes**, never bytes: the cap protects an agent's context, and CJK
|
||||
text is three bytes a rune.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined names).
|
||||
- [ ] **3.** `pdfdoc.go`, **4.** `grep.go`.
|
||||
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/pdfdoc/` → `ok`.
|
||||
- [ ] **6.** `make gate`. **7.** Row `p5a/05-pdfdoc`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/pdfdoc docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given test passes; `make gate` ok; given file byte-identical.
|
||||
Reference in New Issue
Block a user