sift/docs/plans/p5a/02-pdftext.md

# P5a task 02: `pdftext` — extract a PDF in a child process

**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `pdftext: extract PDF text in a re-exec'd child (sift pdftext) with a deadline`

## Goal

Parsing a PDF from the internet is the riskiest thing sift will do. The parser is pure Go
(`github.com/go-pdfkit/extract` + `github.com/go-pdfkit/reader`, already in `vendor/`), but a
goroutine cannot be killed, so it runs in a **child process**: the sift binary re-executed as
`sift pdftext`, reading the PDF on stdin and writing JSON Lines on stdout. The parent kills it at
a deadline. This task builds both halves; nothing calls them yet (task 06 does).

## Files

- Copy: `internal/pdftest/pdftest.go` (test support that builds PDFs; never edit it),
  `tests/testdata/pdf/encrypted-user.pdf`, `tests/testdata/pdf/encrypted-owner.pdf`,
  `tests/testdata/pdf/README.md`, `internal/pdftext/pdftext_test.go`
- Create: `internal/pdftext/pdftext.go` (types, errors, `Extract`, `ChildMain`),
  `internal/pdftext/outline.go` (the outline walk), `internal/pdftext/runner.go` (`Runner`)
- Modify: `cmd/sift/main.go` (one `case "pdftext":` in the command switch), `docs/implementer-log.md`
- Delete: `tools.go` (it only pinned the two modules until this package imports them)

## Interfaces

```go
package pdftext

type OutlineItem struct {
	Title string `json:"title"`
	Page  int    `json:"page"`  // 1-based
	Level int    `json:"level"` // 1 = top level
}

type Meta struct {
	Pages   int           `json:"pages"`   // pages in the document (not only those extracted)
	Title   string        `json:"title"`   // /Info /Title, "" when absent
	Outline []OutlineItem `json:"outline"` // never nil: [] when the document has none
}

// Doc is an extracted document: Text[i] is page i+1's text, for the pages extracted
// (len(Text) ≤ Pages). Failed lists pages whose extraction failed (their Text is "").
type Doc struct {
	Meta
	Text   []string
	Failed []int
}

var (
	ErrEncrypted   = errors.New("encrypted PDF (password required)")
	ErrUnreadable  = errors.New("unreadable PDF")
	ErrNoText      = errors.New("no text layer (scanned PDF?)")
	ErrTimeout     = errors.New("pdf extraction timed out")
	ErrTooMuchText = errors.New("pdf text over the output limit")
)

// Extract writes one {"meta":{…}} line, then one {"page":n,"text":"…"} line per page from 1 up
// to min(Pages, maxPages); a page that fails yields {"page":n,"error":"…"} and extraction goes
// on. It returns ErrEncrypted when the document needs a user password, ErrUnreadable when it
// cannot be opened or has no pages, and never panics.
func Extract(pdf []byte, maxPages int, w io.Writer) error

// ChildMain is `sift pdftext`: flags (-max-pages N, default 2000), the PDF on stdin (read with a
// 200 MiB cap), Extract to stdout. Exit codes: 0 ok, 3 ErrEncrypted, 4 ErrUnreadable, 1 anything
// else.
func ChildMain(args []string, stdin io.Reader, stdout io.Writer) int

type Runner struct {
	Exe      string        // "" = /proc/self/exe
	Args     []string      // nil = []string{"pdftext"}; -max-pages is appended
	Env      []string      // the child's whole environment; nil = none
	Deadline time.Duration // 0 = 60 s
	MaxText  int64         // bytes of child output; 0 = 64 MiB
	MaxPages int           // 0 = 2000
}

// Run executes the child, writes pdf to its stdin, and parses its output into a Doc.
func (r *Runner) Run(ctx context.Context, pdf []byte) (*Doc, error)
```

## Rules

1. **Extract.**
   - Open with `reader.Open(pdf)`. `reader.ErrWrongPassword` (from `errors.Is`) → `ErrEncrypted`.
     Any other open error → `ErrUnreadable`. `PageCount() == 0` → `ErrUnreadable`.
   - Each page: `extract.Text(doc, n)` with `n` counting from 1, inside a `recover()`; an error
     or a panic becomes that page's `"error"` line.
   - Title: the trailer's `/Info` dictionary, key `Title`, as a text string (rule 3).
   - The meta line comes first and carries the full page count even when `maxPages` is smaller.
2. **Outline** (`outline.go`). From the trailer's `/Root` (the catalog) follow `/Outlines` →
   `/First`, then each item's `/Next`; an item's children start at its `/First` (level + 1).
   - An item's page comes from `/Dest` (an array whose first element is a page reference), or
     from `/A` when it is a dictionary with `/S /GoTo` and a `/D` array. Map the reference to a
     1-based page number by comparing it with `doc.PageRef(i)` for `i` = 1…`PageCount()`.
   - Skip an item whose destination is a name or cannot be resolved; keep walking.
   - Stop after 10,000 items, and never visit an object twice (a malformed outline can loop).
3. **Text strings** (`/Title` in Info and in outline items): a `reader.String` is raw bytes.
   Bytes starting `FE FF` are UTF-16BE (decode the rest with `unicode/utf16`); otherwise treat
   each byte as a Latin-1 code point (close enough to PDFDocEncoding for titles). Trim spaces.
4. **ChildMain** runs with a soft memory limit: call `debug.SetMemoryLimit(768 << 20)` first.
   It must not read any environment variable and must not open the network.
5. **Runner.Run.**
   - `exec.CommandContext` with a context that ends at `Deadline` (or `ctx`, whichever is sooner);
     `cmd.Env = r.Env` exactly (nil means an empty environment — the parent's never leaks in).
   - Write the PDF to stdin from a goroutine, then close stdin.
   - Read stdout through `io.LimitReader(…, MaxText+1)`; more than `MaxText` → kill the child and
     return `ErrTooMuchText`. Keep at most 4 KiB of stderr, for the error message only.
   - Always `Wait` for the child, so it is reaped before `Run` returns.
   - Deadline passed → `ErrTimeout` (even if the kill produced some other error).
   - Exit 3 → `ErrEncrypted`, exit 4 → `ErrUnreadable`, any other non-zero exit or unparseable
     output → `ErrUnreadable` wrapped with the first line of stderr.
   - Parse the lines into a `Doc`. If every extracted page's text is blank after
     `strings.TrimSpace` (or no page was extracted), return `ErrNoText`.
6. **`cmd/sift/main.go`:** `case "pdftext": os.Exit(pdftext.ChildMain(os.Args[1:], os.Stdin, os.Stdout))`.
   It is not listed in the help text: users never run it.

## Facts

- The tests run the **test binary itself** as the child: `TestMain` becomes the child when
  `SIFT_PDFTEXT_CHILD=1` is in its environment. The `SIFT_PDFTEXT_SLEEP` and
  `SIFT_PDFTEXT_PIDFILE` variables exist only in that test file; production code never reads
  them.
- `extract.Text(doc, 1)` returns `""` with a nil error for an image-only page.
- Reader API you need: `reader.Open`, `doc.PageCount()`, `doc.PageRef(i) (Ref, bool)`,
  `doc.Trailer() Dict`, `doc.Resolve(o) (Object, error)`, `Dict.Get(Name) Object`, and the types
  `reader.Dict`, `reader.Array`, `reader.Ref`, `reader.String`, `reader.Name`. Read their
  definitions under `vendor/github.com/go-pdfkit/reader/` (`object.go`, `document.go`) — that is
  inside this repository.

## Steps

- [ ] **1.** `git switch p5a`; copy the given files; `git rm tools.go`.
- [ ] **2. See them fail:** `go test -count=1 ./internal/pdftext/` (undefined names).
- [ ] **3.** `pdftext.go`, **4.** `outline.go`, **5.** `runner.go`, **6.** `cmd/sift/main.go`.
- [ ] **7.** `gofmt -w`; `go test -race -count=3 ./internal/pdftext/` → `ok`.
- [ ] **8.** `make build && printf 'nope' | bin/sift pdftext; echo "exit $?"` → `exit 4`.
- [ ] **9.** `make gate`. **10.** Row `p5a/02-pdftext`; commit.

```sh
git add internal/pdftext internal/pdftest tests/testdata/pdf cmd/sift docs/implementer-log.md   # tools.go is already staged by git rm
git commit -F .state/commit-msg.txt
```

## Done when

- The given tests pass under `-race -count=3`, including the deadline test (the child is gone
  after `Run` returns); `make gate` ok; given files byte-identical; `go.mod` untouched.

## Stop and report if

- Removing `tools.go` changes `go.mod`, `go.sum` or `vendor/modules.txt` (it must not: the
  package now imports both modules), or a vendored package is missing.