sift/docs/plans/p5a

P5a implementation plan: PDF documents

For the implementing model: do not work from this file. The owner gives you one task file at a time. This file is the index for the owner and the reviewer.

Goal: docs/P5a.md. sift fetches, extracts and serves PDFs, so agents stop running pdftotext in the shell. Engine: github.com/go-pdfkit/extract v0.3.0 (bake-off, P5a §3), vendored by the owner before the run.

Task	Package(s)	Given
01-kind-fetch	internal/clean, internal/fetch	clean/kind_test.go, fetch/pdf_test.go
02-pdftext	internal/pdftext, cmd/sift (one dispatch line)	pdftext/pdftext_test.go, internal/pdftest/pdftest.go, tests/testdata/pdf/*
03-dcache	internal/dcache	dcache/pdf_test.go
04-pdfpool	internal/pdfpool	pdfpool/pdfpool_test.go
05-pdfdoc	internal/pdfdoc	pdfdoc/pdfdoc_test.go
06-api	internal/api	api/pdf_test.go
07-digest-cli-docs	internal/digest, internal/metrics, cmd/sift, internal/api (metric calls), skill/, nix/module.nix, README.md	digest/pages_test.go, metrics/pdf_test.go

Order matters: 05 imports 02's types; 06 uses 01–05; 07 wires metrics into 06's file.

How this plan was made: acceptance tests first, no reference implementation. The given tests compiled against a panic-only skeleton of every new name, on this repository at the plan commit, and failed there for the intended reasons; every earlier test stayed green. The PDFs the tests use are built by internal/pdftest (a given file) at test time. The only binary fixtures are two small qpdf-encrypted copies of the three-page test document (tests/testdata/pdf/README.md).

Owner changes made before the run (plan commit):

• go.mod, go.sum, vendor/: go-pdfkit/extract v0.3.0, go-pdfkit/reader v0.6.0 (and its pdffont), pinned through tools.go until task 02 imports them. flake.nix uses vendorHash = null, so the nix build reads vendor/.
• internal/api/api_test.go: TestUnsupportedType now uses application/zip; a PDF is no longer unsupported.
• gofmt -w on internal/clean/clean.go, internal/guard/guard_test.go, internal/guard/iana.go (whitespace only), so the new make gate target passes on master.
• AGENTS.md, tools/run-plan.sh, docs/implementer-log.md, make gate.

Attribution: tools/run-plan.sh gives the model the exact trailer Co-Authored-By: <model> <<model>@llama.cpp.invalid> and refuses a commit without it.

Where it runs: a separate checkout (~/src/sift-impl on straylight), branch p5a. Nothing reaches orion until the owner's review, the merge, and straylight's pin bump.

Changes during the run

• 2026-09-25, task 01: owner test fault. TestFetchIf took its mutex with defer mu.Unlock() and then made a third request, whose server handler needs that mutex: a deadlock that surfaced as a 20 s upstream timeout. Ornith spent 55 minutes debugging it correctly (its notes had the handler blocked on mu.Lock at the test line) and ended on a refused /tmp read, with the implementation done but debug prints in pdf.go and a zz_debug_test.go. Owner fixed the test (copy the slice under the lock, release before the next request), removed the debug scaffolding, and resumed the session to finish and commit.