P5a plan: PDF documents — seven Ornith tasks with acceptance tests first
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+17
-9
@@ -61,9 +61,7 @@ Notes:
|
||||
|
||||
**Known extract defect:** the Russian UDHR (Acrobat Distiller 5, 2002) comes out with words glued
|
||||
together and digits dropped (recall 0.03; gopdf and pdfium read it cleanly).
|
||||
- Keep it as a regression fixture: `tests/testdata/pdf/udhr-ru.pdf`, expected to fail today and
|
||||
marked as such.
|
||||
- File the upstream issue.
|
||||
- File the upstream issue. No fixture is committed: the document is not ours to redistribute.
|
||||
- Pin `extract v0.3.0` / `reader v0.6.0` in `go.mod` and vendor them, as for every other dependency.
|
||||
|
||||
## 4. Detection and fetch
|
||||
@@ -115,8 +113,9 @@ Kyle, 2026-09-25: "one or two workers seems tenable".
|
||||
|
||||
- **Pool:** 2 extraction slots and 4 waiting places. A request that waits more than 20 s, or
|
||||
finds the queue full, gets `503 {"error":"pdf workers busy"}` with `Retry-After: 10`.
|
||||
- **Single-flight:** keyed by (caller class, final URL). Concurrent requests for the same
|
||||
document share one download and one extraction.
|
||||
- **Single-flight:** keyed by (caller class, content hash). Concurrent requests for the same
|
||||
bytes share one extraction. Downloads may still happen twice: a URL is not known to be a PDF
|
||||
until it has been fetched.
|
||||
- **Per caller:** one PDF job in flight. A second concurrent one gets the existing
|
||||
`429 busy` path.
|
||||
- **Cache first:** a request answered from the cache (§7) never touches the pool.
|
||||
@@ -136,9 +135,11 @@ Both count toward `dcache` `maxBytes` and its eviction, whole documents at a tim
|
||||
|
||||
## 8. What callers get back
|
||||
|
||||
Text is assembled with page markers, so both agents and digests can cite pages:
|
||||
Text is assembled with a marker before **every** page, so both agents and digests can cite pages:
|
||||
|
||||
```
|
||||
--- page 1 ---
|
||||
|
||||
<page 1 text>
|
||||
|
||||
--- page 2 ---
|
||||
@@ -158,7 +159,7 @@ Text is assembled with page markers, so both agents and digests can cite pages:
|
||||
- `pages` and `grep` together means grep within those pages.
|
||||
- The JSON envelope gains:
|
||||
```
|
||||
"pdf": {"pages": N, "title": "…", "outline": [...], "returned": [a, b], "map": bool, "grep_hits": n}
|
||||
"pdf": {"pages", "title", "outline", "extracted", "first", "last", "map", "grep_hits", "note"}
|
||||
```
|
||||
|
||||
**`/v1/fetch`, mode `digest` (`question`):**
|
||||
@@ -231,8 +232,7 @@ All run offline against `tests/testdata/pdf/` fixtures served by an httptest ser
|
||||
13. **Digest:** `question` on the 3-page fixture returns a digest containing `(p. ` for a fact
|
||||
known to be on one page (the model is stubbed, as in the existing digest tests; the check is
|
||||
that the page markers reach the prompt and the prompt line is present).
|
||||
14. **The `udhr-ru` regression fixture** is present and marked as a known failure (skip with a
|
||||
reason naming the upstream issue).
|
||||
14. (Dropped: no third-party fixture is committed; the extract defect is tracked upstream.)
|
||||
|
||||
## 12. Rollout
|
||||
|
||||
@@ -242,3 +242,11 @@ All run offline against `tests/testdata/pdf/` fixtures served by an httptest ser
|
||||
3. The skill update ships with the pin bump to talos, straylight and titan.
|
||||
4. Watch `sift_pdf_*` and tirith for a week. The success measure is that the
|
||||
`cp … pdftotext …` shell pattern disappears from agents' commands.
|
||||
|
||||
## 13. Changes after the plan was written (2026-09-25)
|
||||
|
||||
- The implementation plan is `docs/plans/p5a/` (seven tasks, Ornith implementing, acceptance tests
|
||||
first). Where this doc and a task file differ, the task file and its given tests win.
|
||||
- `requestDeadline` 90 s → 210 s and `writeTimeout` 120 s → 240 s, so the PDF path's worst case
|
||||
(120 s download + 20 s queue + 60 s extraction) fits; other paths keep their own timeouts.
|
||||
- The map line reads `pages 1-E extracted` (ASCII hyphen) when the child stopped at its page cap.
|
||||
|
||||
Reference in New Issue
Block a user