P5a plan: PDF documents — seven Ornith tasks with acceptance tests first
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+91
@@ -0,0 +1,91 @@
|
||||
# extract
|
||||
|
||||
[](https://github.com/go-pdfkit/extract/actions/workflows/ci.yml)
|
||||
[](https://pkg.go.dev/github.com/go-pdfkit/extract)
|
||||
[](LICENSE)
|
||||
[](#how-it-is-checked)
|
||||
|
||||
Reads a PDF page back: the text on it, with where each piece of it sits, and
|
||||
the images it places.
|
||||
|
||||
```go
|
||||
text, err := extract.Text(doc, 1) // the page as words, lines put back together
|
||||
runs, err := extract.Runs(doc, 1) // every piece, with its place and its size
|
||||
imgs, err := extract.Images(doc, 1) // every picture, and where it lands
|
||||
```
|
||||
|
||||
Pure Go, no C. It builds for `GOOS=js/wasm` and every architecture the fleet
|
||||
targets.
|
||||
|
||||
## What a page actually holds
|
||||
|
||||
Not text. A page holds instructions for drawing glyphs, and the text is what
|
||||
those glyphs were meant to say — which the document says only if it was asked
|
||||
to. Where it does not, this says so rather than guessing: a run whose codes
|
||||
could not be worked out comes back marked `Unreadable`, so a caller can tell
|
||||
*this page says nothing* from *this page could not be read*.
|
||||
|
||||
Three things are tried, in order:
|
||||
|
||||
1. the font's `/ToUnicode` map, which is the document's own word on what its
|
||||
text says;
|
||||
2. the name its encoding gives the code — but only a name the document chose
|
||||
itself, since a **symbolic** font read through an assumed encoding gives
|
||||
the wrong letter with nothing to say so;
|
||||
3. what the **embedded font program** calls the code, read with
|
||||
[`go-opentype`](https://github.com/go-opentype/opentype) — which is the
|
||||
honest answer for a symbolic font, and the reason a page of mathematics
|
||||
comes back as mathematics.
|
||||
|
||||
Text drawn in the mode that puts no ink on the page — how a scanner writes
|
||||
what it read underneath the picture it read it from — is returned and marked
|
||||
`Invisible`.
|
||||
|
||||
## Images
|
||||
|
||||
A picture comes back as the file holds it, with the box it covers on the page.
|
||||
A JPEG comes out as a JPEG: re-encoding it would lose something for nothing.
|
||||
Turning the other kinds into pixels means reading their colour space, which is
|
||||
[`render`](https://github.com/go-pdfkit/render)'s work rather than this
|
||||
package's.
|
||||
|
||||
## How it is checked
|
||||
|
||||
100% of statements, including every branch that handles a page saying
|
||||
something it should not.
|
||||
|
||||
Then on the corpus — 118 833 real PDFs, 121 946 pages:
|
||||
|
||||
```
|
||||
pages read 121946
|
||||
with text 94865 (77.8%)
|
||||
saying nothing 27081, of which 4812 did draw glyphs
|
||||
characters 33976414 (17272226 of them letters)
|
||||
runs 10797045
|
||||
unreadable 203607 (1.89%)
|
||||
invisible 29917
|
||||
images 300685
|
||||
panics 0
|
||||
```
|
||||
|
||||
The 22% of pages that say nothing are figures — this corpus is arXiv's, and
|
||||
most of it is plots. Of the pages that *did* draw glyphs and still said
|
||||
nothing, nearly all are Type 3 fonts with no `/ToUnicode`: dvips bitmap fonts
|
||||
whose glyphs are called `a1` and `s32`, which no reader can turn back into
|
||||
letters.
|
||||
|
||||
## The rest of the family
|
||||
|
||||
[**reader**](https://github.com/go-pdfkit/reader) reads and writes the format ·
|
||||
[**ops**](https://github.com/go-pdfkit/ops) is the verbs and the `pdfops` command ·
|
||||
[**render**](https://github.com/go-pdfkit/render) turns a page into pixels ·
|
||||
[**pdffont**](https://github.com/go-pdfkit/pdffont) is what a document says about a font ·
|
||||
[**extract**](https://github.com/go-pdfkit/extract) reads a page back ·
|
||||
[**coedit**](https://github.com/go-pdfkit/coedit) is a PDF several people edit at once ·
|
||||
[**app**](https://github.com/go-pdfkit/app) is the browser workbench.
|
||||
|
||||
Documentation for all of it: <https://go-pdfkit.github.io/docs/>
|
||||
|
||||
## Licence
|
||||
|
||||
BSD-3-Clause.
|
||||
Reference in New Issue
Block a user