# P5a implementation plan: PDF documents

> **For the implementing model:** do not work from this file. The owner gives you one task file
> at a time. This file is the index for the owner and the reviewer.

**Goal:** `docs/P5a.md`. sift fetches, extracts and serves PDFs, so agents stop running
`pdftotext` in the shell. Engine: `github.com/go-pdfkit/extract` v0.3.0 (bake-off, P5a §3),
vendored by the owner before the run.

| Task | Package(s) | Given |
|---|---|---|
| 01-kind-fetch | `internal/clean`, `internal/fetch` | `clean/kind_test.go`, `fetch/pdf_test.go` |
| 02-pdftext | `internal/pdftext`, `cmd/sift` (one dispatch line) | `pdftext/pdftext_test.go`, `internal/pdftest/pdftest.go`, `tests/testdata/pdf/*` |
| 03-dcache | `internal/dcache` | `dcache/pdf_test.go` |
| 04-pdfpool | `internal/pdfpool` | `pdfpool/pdfpool_test.go` |
| 05-pdfdoc | `internal/pdfdoc` | `pdfdoc/pdfdoc_test.go` |
| 06-api-pdf | `internal/api` (+ one field in `internal/fetch`) | `api/pdf_test.go` |
| 07-api-cache-pages | `internal/api` | (same file; its last 3 tests) |
| 08-digest-cli-docs | `internal/digest`, `internal/metrics`, `cmd/sift`, `internal/api` (metric calls), `skill/`, `nix/module.nix`, `README.md` | `digest/pages_test.go`, `metrics/pdf_test.go` |

**Order matters:** 05 imports 02's types; 06 and 07 use 01–05; 08 wires metrics into 06/07's file.

**How this plan was made:** acceptance tests first, no reference implementation. The given tests
compiled against a panic-only skeleton of every new name, on this repository at the plan
commit, and failed there for the intended reasons; every earlier test stayed green. The PDFs the
tests use are built by `internal/pdftest` (a given file) at test time. The only binary fixtures
are two small qpdf-encrypted copies of the three-page test document (`tests/testdata/pdf/README.md`).

**Owner changes made before the run** (plan commit):
- `go.mod`, `go.sum`, `vendor/`: `go-pdfkit/extract` v0.3.0, `go-pdfkit/reader` v0.6.0 (and its
  `pdffont`), pinned through `tools.go` until task 02 imports them. `flake.nix` uses
  `vendorHash = null`, so the nix build reads `vendor/`.
- `internal/api/api_test.go`: `TestUnsupportedType` now uses `application/zip`; a PDF is no
  longer unsupported.
- `gofmt -w` on `internal/clean/clean.go`, `internal/guard/guard_test.go`,
  `internal/guard/iana.go` (whitespace only), so the new `make gate` target passes on master.
- `AGENTS.md`, `tools/run-plan.sh`, `docs/implementer-log.md`, `make gate`.

**Attribution:** `tools/run-plan.sh` gives the model the exact trailer
`Co-Authored-By: <model> <<model>@llama.cpp.invalid>` and refuses a commit without it.

**Where it runs:** a separate checkout (`~/src/sift-impl` on straylight), branch `p5a`. Nothing
reaches orion until the owner's review, the merge, and straylight's pin bump.

## Changes during the run

- 2026-09-25, task 01: **owner test fault.** `TestFetchIf` took its mutex with `defer mu.Unlock()`
  and then made a third request, whose server handler needs that mutex: a deadlock that surfaced as
  a 20 s upstream timeout. Ornith spent 55 minutes debugging it correctly (its notes had the
  handler blocked on `mu.Lock` at the test line) and ended on a refused `/tmp` read, with the
  implementation done but debug prints in `pdf.go` and a `zz_debug_test.go`. Owner fixed the test
  (copy the slice under the lock, release before the next request), removed the debug scaffolding,
  and resumed the session to finish and commit.
- 2026-09-26, task 06 (the original, API integration in one task): **owner sizing fault.** The
  session looped for 3 h 13 min (731 KB log, repeated context compaction, false beliefs about
  the tree: "the given test is committed", "the child is /bin/true") without a compiling
  `pdf.go`. It is the same failure as crossbar v1/06: one task spanning detection, pool,
  errors, rendering, cache, pages, grep, digest and scrape. Killed by pid. Split into
  **06-api-pdf** (detection, extraction under the pool, errors, full text/map, scrape; 7 of the
  10 given tests) and **07-api-cache-pages** (cache and revalidation, pages, grep, the question
  limit; the other 3). The old 07 is now 08. Both task texts state the exact API of tasks 01–05,
  so the model need not re-read five packages. The working tree was reset: the draft
  `pdf.go`, the `api.go` hooks and a `fetch.Error.RetryAfter` field that was outside the task's
  files (task 06 now allows exactly that field).
- 2026-09-26, the given test's gate note: task 06 ends with `make gate` red on exactly the three
  task-07 tests (the pattern crossbar v2.2 used); the gate runs green after 07.
- 2026-09-26, tasks 06–08 after the split: 52, 19 and 14 min, each first-gate. `run-plan`: all
  tasks done at 04:37.
- **Owner review** (details in `docs/implementer-log.md`): end-to-end on real PDFs (arXiv, NIST,
  RP2040 datasheet, a PhD thesis) against dixie's 9B found one real defect the given tests could
  not see — **no outline for any LaTeX or Acrobat PDF**: named destinations were skipped (my task
  02 said to skip them) and reader's `Dict.Get` answers `reader.Null{}`, not nil, for a missing
  key, which hid the `/A` action. Fixed with a name-tree lookup; `internal/pdftest` gained
  `NamedDests` (owner change to a given file; the copy here is the new reference). Also fixed: a
  race on the default pool, revalidation outside the per-host limit (my spec), the runner waiting
  for the deadline on over-limit output, `Accept` lacking `application/pdf` (my spec).