sift/docs/plans/p5a/02-pdftext.md

P5a task 02: pdftext — extract a PDF in a child process

Branch: p5a (git switch p5a; git status --short must be empty, otherwise stop) Commit subject: pdftext: extract PDF text in a re-exec'd child (sift pdftext) with a deadline

Goal

Parsing a PDF from the internet is the riskiest thing sift will do. The parser is pure Go (github.com/go-pdfkit/extract + github.com/go-pdfkit/reader, already in vendor/), but a goroutine cannot be killed, so it runs in a child process: the sift binary re-executed as sift pdftext, reading the PDF on stdin and writing JSON Lines on stdout. The parent kills it at a deadline. This task builds both halves; nothing calls them yet (task 06 does).

Files

• Copy: internal/pdftest/pdftest.go (test support that builds PDFs; never edit it), tests/testdata/pdf/encrypted-user.pdf, tests/testdata/pdf/encrypted-owner.pdf, tests/testdata/pdf/README.md, internal/pdftext/pdftext_test.go
• Create: internal/pdftext/pdftext.go (types, errors, Extract, ChildMain), internal/pdftext/outline.go (the outline walk), internal/pdftext/runner.go (Runner)
• Modify: cmd/sift/main.go (one case "pdftext": in the command switch), docs/implementer-log.md
• Delete: tools.go (it only pinned the two modules until this package imports them)

Interfaces

package pdftext

type OutlineItem struct {
	Title string `json:"title"`
	Page  int    `json:"page"`  // 1-based
	Level int    `json:"level"` // 1 = top level
}

type Meta struct {
	Pages   int           `json:"pages"`   // pages in the document (not only those extracted)
	Title   string        `json:"title"`   // /Info /Title, "" when absent
	Outline []OutlineItem `json:"outline"` // never nil: [] when the document has none
}

// Doc is an extracted document: Text[i] is page i+1's text, for the pages extracted
// (len(Text) ≤ Pages). Failed lists pages whose extraction failed (their Text is "").
type Doc struct {
	Meta
	Text   []string
	Failed []int
}

var (
	ErrEncrypted   = errors.New("encrypted PDF (password required)")
	ErrUnreadable  = errors.New("unreadable PDF")
	ErrNoText      = errors.New("no text layer (scanned PDF?)")
	ErrTimeout     = errors.New("pdf extraction timed out")
	ErrTooMuchText = errors.New("pdf text over the output limit")
)

// Extract writes one {"meta":{…}} line, then one {"page":n,"text":"…"} line per page from 1 up
// to min(Pages, maxPages); a page that fails yields {"page":n,"error":"…"} and extraction goes
// on. It returns ErrEncrypted when the document needs a user password, ErrUnreadable when it
// cannot be opened or has no pages, and never panics.
func Extract(pdf []byte, maxPages int, w io.Writer) error

// ChildMain is `sift pdftext`: flags (-max-pages N, default 2000), the PDF on stdin (read with a
// 200 MiB cap), Extract to stdout. Exit codes: 0 ok, 3 ErrEncrypted, 4 ErrUnreadable, 1 anything
// else.
func ChildMain(args []string, stdin io.Reader, stdout io.Writer) int

type Runner struct {
	Exe      string        // "" = /proc/self/exe
	Args     []string      // nil = []string{"pdftext"}; -max-pages is appended
	Env      []string      // the child's whole environment; nil = none
	Deadline time.Duration // 0 = 60 s
	MaxText  int64         // bytes of child output; 0 = 64 MiB
	MaxPages int           // 0 = 2000
}

// Run executes the child, writes pdf to its stdin, and parses its output into a Doc.
func (r *Runner) Run(ctx context.Context, pdf []byte) (*Doc, error)

Rules

1. Extract.
   • Open with reader.Open(pdf). reader.ErrWrongPassword (from errors.Is) → ErrEncrypted. Any other open error → ErrUnreadable. PageCount() == 0 → ErrUnreadable.
   • Each page: extract.Text(doc, n) with n counting from 1, inside a recover(); an error or a panic becomes that page's "error" line.
   • Title: the trailer's /Info dictionary, key Title, as a text string (rule 3).
   • The meta line comes first and carries the full page count even when maxPages is smaller.
2. Outline (outline.go). From the trailer's /Root (the catalog) follow /Outlines → /First, then each item's /Next; an item's children start at its /First (level + 1).
   • An item's page comes from /Dest (an array whose first element is a page reference), or from /A when it is a dictionary with /S /GoTo and a /D array. Map the reference to a 1-based page number by comparing it with doc.PageRef(i) for i = 1…PageCount().
   • Skip an item whose destination is a name or cannot be resolved; keep walking.
   • Stop after 10,000 items, and never visit an object twice (a malformed outline can loop).
3. Text strings (/Title in Info and in outline items): a reader.String is raw bytes. Bytes starting FE FF are UTF-16BE (decode the rest with unicode/utf16); otherwise treat each byte as a Latin-1 code point (close enough to PDFDocEncoding for titles). Trim spaces.
4. ChildMain runs with a soft memory limit: call debug.SetMemoryLimit(768 << 20) first. It must not read any environment variable and must not open the network.
5. Runner.Run.
   • exec.CommandContext with a context that ends at Deadline (or ctx, whichever is sooner); cmd.Env = r.Env exactly (nil means an empty environment — the parent's never leaks in).
   • Write the PDF to stdin from a goroutine, then close stdin.
   • Read stdout through io.LimitReader(…, MaxText+1); more than MaxText → kill the child and return ErrTooMuchText. Keep at most 4 KiB of stderr, for the error message only.
   • Always Wait for the child, so it is reaped before Run returns.
   • Deadline passed → ErrTimeout (even if the kill produced some other error).
   • Exit 3 → ErrEncrypted, exit 4 → ErrUnreadable, any other non-zero exit or unparseable output → ErrUnreadable wrapped with the first line of stderr.
   • Parse the lines into a Doc. If every extracted page's text is blank after strings.TrimSpace (or no page was extracted), return ErrNoText.
6. cmd/sift/main.go: case "pdftext": os.Exit(pdftext.ChildMain(os.Args[1:], os.Stdin, os.Stdout)). It is not listed in the help text: users never run it.

Facts

• The tests run the test binary itself as the child: TestMain becomes the child when SIFT_PDFTEXT_CHILD=1 is in its environment. The SIFT_PDFTEXT_SLEEP and SIFT_PDFTEXT_PIDFILE variables exist only in that test file; production code never reads them.
• extract.Text(doc, 1) returns "" with a nil error for an image-only page.
• Reader API you need: reader.Open, doc.PageCount(), doc.PageRef(i) (Ref, bool), doc.Trailer() Dict, doc.Resolve(o) (Object, error), Dict.Get(Name) Object, and the types reader.Dict, reader.Array, reader.Ref, reader.String, reader.Name. Read their definitions under vendor/github.com/go-pdfkit/reader/ (object.go, document.go) — that is inside this repository.

Steps

• 1. git switch p5a; copy the given files; git rm tools.go.
• 2. See them fail: go test -count=1 ./internal/pdftext/ (undefined names).
• 3. pdftext.go, 4. outline.go, 5. runner.go, 6. cmd/sift/main.go.
• 7. gofmt -w; go test -race -count=3 ./internal/pdftext/ → ok.
• 8. make build && printf 'nope' | bin/sift pdftext; echo "exit $?" → exit 4.
• 9. make gate. 10. Row p5a/02-pdftext; commit.

git add internal/pdftext internal/pdftest tests/testdata/pdf cmd/sift docs/implementer-log.md   # tools.go is already staged by git rm
git commit -F .state/commit-msg.txt

Done when

• The given tests pass under -race -count=3, including the deadline test (the child is gone after Run returns); make gate ok; given files byte-identical; go.mod untouched.

Stop and report if

• Removing tools.go changes go.mod, go.sum or vendor/modules.txt (it must not: the package now imports both modules), or a vendored package is missing.