P5a plan: PDF documents — seven Ornith tasks with acceptance tests first

docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables,
04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI,
skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the
intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted
fixtures.

Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via
tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for
the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard;
TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and
writeTimeout 120→240 s for the PDF path's worst case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-25 20:47:38 -07:00
co-authored by Claude Opus 5.5
parent 3f2b78d4aa
commit cfde35397a
115 changed files with 26422 additions and 43 deletions
+17 -9
View File
@@ -61,9 +61,7 @@ Notes:
**Known extract defect:** the Russian UDHR (Acrobat Distiller 5, 2002) comes out with words glued
together and digits dropped (recall 0.03; gopdf and pdfium read it cleanly).
- Keep it as a regression fixture: `tests/testdata/pdf/udhr-ru.pdf`, expected to fail today and
marked as such.
- File the upstream issue.
- File the upstream issue. No fixture is committed: the document is not ours to redistribute.
- Pin `extract v0.3.0` / `reader v0.6.0` in `go.mod` and vendor them, as for every other dependency.
## 4. Detection and fetch
@@ -115,8 +113,9 @@ Kyle, 2026-09-25: "one or two workers seems tenable".
- **Pool:** 2 extraction slots and 4 waiting places. A request that waits more than 20 s, or
finds the queue full, gets `503 {"error":"pdf workers busy"}` with `Retry-After: 10`.
- **Single-flight:** keyed by (caller class, final URL). Concurrent requests for the same
document share one download and one extraction.
- **Single-flight:** keyed by (caller class, content hash). Concurrent requests for the same
bytes share one extraction. Downloads may still happen twice: a URL is not known to be a PDF
until it has been fetched.
- **Per caller:** one PDF job in flight. A second concurrent one gets the existing
`429 busy` path.
- **Cache first:** a request answered from the cache (§7) never touches the pool.
@@ -136,9 +135,11 @@ Both count toward `dcache` `maxBytes` and its eviction, whole documents at a tim
## 8. What callers get back
Text is assembled with page markers, so both agents and digests can cite pages:
Text is assembled with a marker before **every** page, so both agents and digests can cite pages:
```
--- page 1 ---
<page 1 text>
--- page 2 ---
@@ -158,7 +159,7 @@ Text is assembled with page markers, so both agents and digests can cite pages:
- `pages` and `grep` together means grep within those pages.
- The JSON envelope gains:
```
"pdf": {"pages": N, "title": "…", "outline": [...], "returned": [a, b], "map": bool, "grep_hits": n}
"pdf": {"pages", "title", "outline", "extracted", "first", "last", "map", "grep_hits", "note"}
```
**`/v1/fetch`, mode `digest` (`question`):**
@@ -231,8 +232,7 @@ All run offline against `tests/testdata/pdf/` fixtures served by an httptest ser
13. **Digest:** `question` on the 3-page fixture returns a digest containing `(p. ` for a fact
known to be on one page (the model is stubbed, as in the existing digest tests; the check is
that the page markers reach the prompt and the prompt line is present).
14. **The `udhr-ru` regression fixture** is present and marked as a known failure (skip with a
reason naming the upstream issue).
14. (Dropped: no third-party fixture is committed; the extract defect is tracked upstream.)
## 12. Rollout
@@ -242,3 +242,11 @@ All run offline against `tests/testdata/pdf/` fixtures served by an httptest ser
3. The skill update ships with the pin bump to talos, straylight and titan.
4. Watch `sift_pdf_*` and tirith for a week. The success measure is that the
`cp … pdftotext …` shell pattern disappears from agents' commands.
## 13. Changes after the plan was written (2026-09-25)
- The implementation plan is `docs/plans/p5a/` (seven tasks, Ornith implementing, acceptance tests
first). Where this doc and a task file differ, the task file and its given tests win.
- `requestDeadline` 90 s → 210 s and `writeTimeout` 120 s → 240 s, so the PDF path's worst case
(120 s download + 20 s queue + 60 s extraction) fits; other paths keep their own timeouts.
- The map line reads `pages 1-E extracted` (ASCII hyphen) when the child stopped at its page cap.