P5a plan: PDF documents — seven Ornith tasks with acceptance tests first
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+144
@@ -0,0 +1,144 @@
|
||||
# reader
|
||||
|
||||
[](https://github.com/go-pdfkit/reader/actions/workflows/ci.yml)
|
||||
[](https://pkg.go.dev/github.com/go-pdfkit/reader)
|
||||
[](https://goreportcard.com/report/github.com/go-pdfkit/reader)
|
||||
[](LICENSE)
|
||||
[](#testing)
|
||||
|
||||
A pure-Go, **zero-C** PDF **reader** — the parsing half of
|
||||
[go-pdfkit](https://github.com/go-pdfkit). Where
|
||||
[`pdfkit`](https://github.com/go-pdfkit/pdfkit) authors new PDF 1.7, this
|
||||
module takes existing PDF bytes apart: lexer, object model, cross-reference
|
||||
tables and streams, stream filters, the standard security handler, the page
|
||||
tree, and content streams. It also writes the same object graph back out,
|
||||
which is what every operation on an existing file needs.
|
||||
Nothing outside the Go standard library is required, so it builds for
|
||||
`GOOS=js/wasm` and every 64-bit architecture the fleet targets.
|
||||
|
||||
## Status
|
||||
|
||||
Landing in waves. This module currently provides the **object layer** and
|
||||
the **document structure**:
|
||||
|
||||
- **Lexer** — the full PDF object syntax: integers and reals in the shapes
|
||||
producers actually write, literal strings with their escapes, hex strings,
|
||||
names with `#xx` escapes, arrays, dictionaries, streams, and keywords.
|
||||
- **Objects** — `Null`, `Bool`, `Integer`, `Real`, `String`, `Name`, `Array`,
|
||||
`Dict`, `Stream`, and indirect `Ref`, with the `N G R` / `N G obj` lookahead
|
||||
resolved the way the specification requires.
|
||||
- **Filters** — `FlateDecode`, `LZWDecode` (with PDF's `EarlyChange`, which the
|
||||
standard library's LZW does not implement), `ASCIIHexDecode`,
|
||||
`ASCII85Decode`, `RunLengthDecode`, `Crypt`, plus the PNG and TIFF
|
||||
predictors. Image filters (`DCTDecode`, `JPXDecode`, `CCITTFaxDecode`,
|
||||
`JBIG2Decode`) are reported rather than applied, so an image consumer gets
|
||||
the encoded bytes.
|
||||
- **Filters that fail** — a chain that cannot be run to the end is not the end
|
||||
of the stream. `DecodeRecovering` returns what the filters before the failure
|
||||
produced — the prefix a damaged Flate stream did inflate, not its compressed
|
||||
bytes — and says so through `Decoded.Recovered` and `Decoded.Cause`. Bytes no
|
||||
filter decoded arrive in `Decoded.Undecoded`, never in `Decoded.Data`, so a
|
||||
compressed stream cannot be painted as samples or tokenised as content by a
|
||||
caller that forgot to check a flag.
|
||||
`Decode` is the strict reading and refuses such a stream outright, so a
|
||||
caller that must not act on damaged data says which it wants by which one it
|
||||
calls. Page content takes the lenient reading, because a page whose content
|
||||
will not decode is still a page; `PageContentDecoded` reports whether any
|
||||
salvaging happened. Cross-reference streams take the strict one: a table that
|
||||
has to be guessed at is worse than no table, and the repair below is the
|
||||
answer to one.
|
||||
- **Cross-references** — classic tables, cross-reference **streams**, **object
|
||||
streams**, `/Prev` chains, and the `/XRefStm` of a hybrid file, newest
|
||||
definition winning.
|
||||
- **Repair** — a file whose tables are missing, truncated or simply wrong is
|
||||
rebuilt by scanning it for object headers, trailers and, failing those, for
|
||||
a catalogue; a file that kept its pages but lost its catalogue gets one.
|
||||
This is not an exceptional path: it is what makes a reader usable. The
|
||||
rebuild establishes the file key before it reads object streams, so a file
|
||||
whose catalogue lives in an encrypted one can be rebuilt at all; it takes
|
||||
`/Encrypt` from a cross-reference stream's dictionary when the file has no
|
||||
trailer keyword to name it in. Where two object streams claim the same
|
||||
object, the later one in the file wins, the same rule the header scan
|
||||
applies to objects written directly — so the same damaged file reads the
|
||||
same way twice. When a rebuild fails, the error says what the rebuild found
|
||||
and not merely that the tables could not be read.
|
||||
- **Documents** — `Open`, object resolution with cycle and recursion guards,
|
||||
the trailer, the catalogue, and the page tree with the four attributes a
|
||||
page inherits from its ancestors.
|
||||
- **Encryption** — the standard security handler in every revision, both ways:
|
||||
reading RC4 at 40 and 128 bits, AESV2, and the AES-256 of `/R 5` and `/R 6`,
|
||||
with crypt filters, `/Identity` and `/EncryptMetadata`; and writing AES-256
|
||||
or AES-128, with permissions, a user password and an owner password.
|
||||
A password is tried as the user password and as the owner password, and
|
||||
`Open` uses the empty one, so a file protected only against editing opens
|
||||
with no password at all.
|
||||
and steps over rubbish rather than losing the operations around it, with
|
||||
inline images read whole. Where an inline image ends is the one genuinely
|
||||
ambiguous thing in a content stream, since its data may spell EI itself;
|
||||
the length is computed from the image's own geometry where it can be, and
|
||||
otherwise every candidate EI is tried until the data before one actually
|
||||
decodes.
|
||||
- **Writing** — the exact inverse of the parser: objects rendered in PDF
|
||||
syntax with stable dictionary ordering and strings in whichever of the two
|
||||
forms is shorter, and a `Writer` that copies whole object graphs out of one
|
||||
or more documents, renumbering as it goes, then lays down a cross-reference
|
||||
table and a trailer. `NewPackedWriter` instead puts what it can into
|
||||
compressed **object streams** and ends in a **cross-reference stream**,
|
||||
which is what makes a file small.
|
||||
Measured against a corpus of **118 863 real PDFs** — Matplotlib, cairo,
|
||||
pdfTeX, Ghostscript, Adobe, R, Apache FOP, PDF 1.3 through 1.7 — `Open`
|
||||
succeeds on **118 833** of them and finds 138 337 pages, in 8 seconds, with no
|
||||
panics. Of the thirty it refuses, twenty-seven are PNG files with a `.pdf`
|
||||
extension, one is PostScript, and two are PDFs truncated past recovery.
|
||||
|
||||
The one encrypted PDF the corpus holds — `/V 4 /R 4`, written by macOS
|
||||
Quartz — decrypts to readable metadata and a valid content stream, which is
|
||||
the only genuinely independent check of the key derivation there is here;
|
||||
the other revisions are round-tripped against a test encryptor written from
|
||||
the same algorithms.
|
||||
|
||||
The content-stream tokeniser reads **1 536 769 753 operations** across those
|
||||
138 337 pages in a minute and a half, with no panics, no page failing to
|
||||
decode, and no operator outside the seventy the format defines — beyond four
|
||||
`arc` and six `nan` written by a producer that was simply wrong.
|
||||
|
||||
Rewriting is checked the same way: every one of the **118 833** files that
|
||||
open is copied object by object into a new file, which is then re-read and
|
||||
compared on page count, media boxes and the bytes of every content stream.
|
||||
**All 118 833 match**, 35.5 GB in and 35.2 GB out, in twenty-eight seconds —
|
||||
and **all 118 833 match again** when the packed writer is used instead, which
|
||||
brings the output down to **94.6%** of the input.
|
||||
|
||||
Next wave: the operations built on all of this.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
go get github.com/go-pdfkit/reader
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
go test -covermode=set ./...
|
||||
```
|
||||
|
||||
CI gates on **exact 100% statement coverage**, `go vet`, and a cross-compile
|
||||
across `linux/{amd64,arm64,riscv64,loong64,ppc64le,s390x}`, `js/wasm`,
|
||||
`darwin/arm64` and `windows/amd64`.
|
||||
|
||||
## The rest of the family
|
||||
|
||||
[**reader**](https://github.com/go-pdfkit/reader) reads and writes the format ·
|
||||
[**ops**](https://github.com/go-pdfkit/ops) is the verbs and the `pdfops` command ·
|
||||
[**render**](https://github.com/go-pdfkit/render) turns a page into pixels ·
|
||||
[**pdffont**](https://github.com/go-pdfkit/pdffont) is what a document says about a font ·
|
||||
[**extract**](https://github.com/go-pdfkit/extract) reads a page back ·
|
||||
[**coedit**](https://github.com/go-pdfkit/coedit) is a PDF several people edit at once ·
|
||||
[**app**](https://github.com/go-pdfkit/app) is the browser workbench.
|
||||
|
||||
Documentation for all of it: <https://go-pdfkit.github.io/docs/>
|
||||
|
||||
## License
|
||||
|
||||
BSD-3-Clause — see [LICENSE](LICENSE). Copyright the go-pdfkit/reader authors.
|
||||
Reference in New Issue
Block a user