P5a plan: PDF documents — seven Ornith tasks with acceptance tests first
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+1
@@ -0,0 +1 @@
|
||||
coverage.out
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
BSD 3-Clause License
|
||||
|
||||
Copyright (c) 2026 the go-pdfkit/render authors
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
1. Redistributions of source code must retain the above copyright notice,
|
||||
this list of conditions and the following disclaimer.
|
||||
|
||||
2. Redistributions in binary form must reproduce the above copyright notice,
|
||||
this list of conditions and the following disclaimer in the documentation
|
||||
and/or other materials provided with the distribution.
|
||||
|
||||
3. Neither the name of the copyright holder nor the names of its contributors
|
||||
may be used to endorse or promote products derived from this software
|
||||
without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
|
||||
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
||||
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
|
||||
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
||||
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
|
||||
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
||||
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||
POSSIBILITY OF SUCH DAMAGE.
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
# extract
|
||||
|
||||
[](https://github.com/go-pdfkit/extract/actions/workflows/ci.yml)
|
||||
[](https://pkg.go.dev/github.com/go-pdfkit/extract)
|
||||
[](LICENSE)
|
||||
[](#how-it-is-checked)
|
||||
|
||||
Reads a PDF page back: the text on it, with where each piece of it sits, and
|
||||
the images it places.
|
||||
|
||||
```go
|
||||
text, err := extract.Text(doc, 1) // the page as words, lines put back together
|
||||
runs, err := extract.Runs(doc, 1) // every piece, with its place and its size
|
||||
imgs, err := extract.Images(doc, 1) // every picture, and where it lands
|
||||
```
|
||||
|
||||
Pure Go, no C. It builds for `GOOS=js/wasm` and every architecture the fleet
|
||||
targets.
|
||||
|
||||
## What a page actually holds
|
||||
|
||||
Not text. A page holds instructions for drawing glyphs, and the text is what
|
||||
those glyphs were meant to say — which the document says only if it was asked
|
||||
to. Where it does not, this says so rather than guessing: a run whose codes
|
||||
could not be worked out comes back marked `Unreadable`, so a caller can tell
|
||||
*this page says nothing* from *this page could not be read*.
|
||||
|
||||
Three things are tried, in order:
|
||||
|
||||
1. the font's `/ToUnicode` map, which is the document's own word on what its
|
||||
text says;
|
||||
2. the name its encoding gives the code — but only a name the document chose
|
||||
itself, since a **symbolic** font read through an assumed encoding gives
|
||||
the wrong letter with nothing to say so;
|
||||
3. what the **embedded font program** calls the code, read with
|
||||
[`go-opentype`](https://github.com/go-opentype/opentype) — which is the
|
||||
honest answer for a symbolic font, and the reason a page of mathematics
|
||||
comes back as mathematics.
|
||||
|
||||
Text drawn in the mode that puts no ink on the page — how a scanner writes
|
||||
what it read underneath the picture it read it from — is returned and marked
|
||||
`Invisible`.
|
||||
|
||||
## Images
|
||||
|
||||
A picture comes back as the file holds it, with the box it covers on the page.
|
||||
A JPEG comes out as a JPEG: re-encoding it would lose something for nothing.
|
||||
Turning the other kinds into pixels means reading their colour space, which is
|
||||
[`render`](https://github.com/go-pdfkit/render)'s work rather than this
|
||||
package's.
|
||||
|
||||
## How it is checked
|
||||
|
||||
100% of statements, including every branch that handles a page saying
|
||||
something it should not.
|
||||
|
||||
Then on the corpus — 118 833 real PDFs, 121 946 pages:
|
||||
|
||||
```
|
||||
pages read 121946
|
||||
with text 94865 (77.8%)
|
||||
saying nothing 27081, of which 4812 did draw glyphs
|
||||
characters 33976414 (17272226 of them letters)
|
||||
runs 10797045
|
||||
unreadable 203607 (1.89%)
|
||||
invisible 29917
|
||||
images 300685
|
||||
panics 0
|
||||
```
|
||||
|
||||
The 22% of pages that say nothing are figures — this corpus is arXiv's, and
|
||||
most of it is plots. Of the pages that *did* draw glyphs and still said
|
||||
nothing, nearly all are Type 3 fonts with no `/ToUnicode`: dvips bitmap fonts
|
||||
whose glyphs are called `a1` and `s32`, which no reader can turn back into
|
||||
letters.
|
||||
|
||||
## The rest of the family
|
||||
|
||||
[**reader**](https://github.com/go-pdfkit/reader) reads and writes the format ·
|
||||
[**ops**](https://github.com/go-pdfkit/ops) is the verbs and the `pdfops` command ·
|
||||
[**render**](https://github.com/go-pdfkit/render) turns a page into pixels ·
|
||||
[**pdffont**](https://github.com/go-pdfkit/pdffont) is what a document says about a font ·
|
||||
[**extract**](https://github.com/go-pdfkit/extract) reads a page back ·
|
||||
[**coedit**](https://github.com/go-pdfkit/coedit) is a PDF several people edit at once ·
|
||||
[**app**](https://github.com/go-pdfkit/app) is the browser workbench.
|
||||
|
||||
Documentation for all of it: <https://go-pdfkit.github.io/docs/>
|
||||
|
||||
## Licence
|
||||
|
||||
BSD-3-Clause.
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
// Package extract reads a PDF page back: the text on it, with where each piece
|
||||
// of it sits, and the images it places.
|
||||
//
|
||||
// A page does not hold text. It holds instructions for drawing glyphs, and the
|
||||
// text is what those glyphs were meant to say — which the document says only
|
||||
// if it was asked to. Where it does not, this says so rather than guessing:
|
||||
// a run whose characters could not be worked out comes back marked, so that a
|
||||
// caller can tell "this page says nothing" from "this page could not be read".
|
||||
package extract
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/go-pdfkit/pdffont"
|
||||
"github.com/go-pdfkit/reader"
|
||||
)
|
||||
|
||||
// A Run is a piece of text drawn in one go: one font, one size, one place.
|
||||
type Run struct {
|
||||
// Text is what the run says.
|
||||
Text string
|
||||
// X and Y are where it starts, in the page's own coordinates — up from
|
||||
// the bottom left corner, in points.
|
||||
X, Y float64
|
||||
// Width is how far the pen moved drawing it, and Size how tall the text
|
||||
// is, both in points.
|
||||
Width, Size float64
|
||||
// Font is the name the page's resources give the font.
|
||||
Font string
|
||||
// Space is how wide a space would be here, which is what says whether a
|
||||
// gap between two runs is a word break or only kerning.
|
||||
Space float64
|
||||
// Invisible reports text drawn in the mode that puts no ink on the page.
|
||||
// It is how a scanner puts what it read underneath the picture it read it
|
||||
// from, so it is text worth having — and worth knowing about.
|
||||
Invisible bool
|
||||
// Unreadable reports a run some of whose codes said nothing about which
|
||||
// characters they were: the font carried no map and named its glyphs
|
||||
// after numbers. What Text holds is then only the part that could be read.
|
||||
Unreadable bool
|
||||
}
|
||||
|
||||
// Text is everything the page says, in reading order, with the lines put back
|
||||
// together.
|
||||
func Text(d *reader.Document, page int) (string, error) {
|
||||
runs, err := Runs(d, page)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return Assemble(runs), nil
|
||||
}
|
||||
|
||||
// Runs is every piece of text on the page, in the order it was drawn.
|
||||
func Runs(d *reader.Document, page int) ([]Run, error) {
|
||||
e, err := walk(d, page, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return e.runs, nil
|
||||
}
|
||||
|
||||
// walk reads a page's content stream once, keeping what it finds.
|
||||
//
|
||||
// A page whose content decoded only part of the way is read as far as it went:
|
||||
// 263 streams in 212 of the 1 633 real forms in the corpus cannot be decoded
|
||||
// cleanly, and half a page of text is worth more to somebody searching than
|
||||
// none of it. A page that decoded no bytes at all is a different answer and
|
||||
// stays an error — a conversion that silently produces an empty document from
|
||||
// an unreadable page is worse than one that says it could not read it, and
|
||||
// go-pdfkit/latex relies on being told.
|
||||
//
|
||||
// wantPictures says whether the caller is going to look at the pictures. A
|
||||
// page's images have nothing to do with its text, and undoing their filters is
|
||||
// the most expensive thing on the page: one arXiv figure holds 378 MB of image
|
||||
// once decompressed, so asking that page what it *said* used to cost 1 094 MB
|
||||
// and 1.3 seconds spent inflating pictures the answer throws away. Where they
|
||||
// land is still noted either way, since that costs a matrix multiply; only the
|
||||
// bytes are left alone.
|
||||
func walk(d *reader.Document, page int, wantPictures bool) (*extractor, error) {
|
||||
dec, err := d.PageContentDecoded(page)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(dec.Data) == 0 && dec.Cause != nil {
|
||||
return nil, dec.Cause
|
||||
}
|
||||
content := dec.Data
|
||||
p, _ := d.Page(page)
|
||||
resources, _ := d.GetDict(p, "Resources")
|
||||
e := &extractor{doc: d, fonts: map[int]*pdffont.Font{}, wantPictures: wantPictures}
|
||||
e.run(content, resources, initialState(d, p), 0)
|
||||
return e, nil
|
||||
}
|
||||
|
||||
// Assemble puts runs back together as lines of text: everything on one
|
||||
// baseline in the order it sits across the page, with a space wherever the
|
||||
// gap between two pieces is wide enough to be one.
|
||||
func Assemble(runs []Run) string {
|
||||
if len(runs) == 0 {
|
||||
return ""
|
||||
}
|
||||
lines := groupIntoLines(runs)
|
||||
var out strings.Builder
|
||||
for i, line := range lines {
|
||||
if i > 0 {
|
||||
out.WriteByte('\n')
|
||||
}
|
||||
out.WriteString(joinLine(line))
|
||||
}
|
||||
return out.String()
|
||||
}
|
||||
|
||||
// groupIntoLines gathers runs that sit on the same baseline, in the order they
|
||||
// appear down the page.
|
||||
func groupIntoLines(runs []Run) [][]Run {
|
||||
sorted := append([]Run{}, runs...)
|
||||
sort.SliceStable(sorted, func(i, j int) bool {
|
||||
if !sameLine(sorted[i], sorted[j]) {
|
||||
return sorted[i].Y > sorted[j].Y
|
||||
}
|
||||
return sorted[i].X < sorted[j].X
|
||||
})
|
||||
var lines [][]Run
|
||||
for _, r := range sorted {
|
||||
if n := len(lines); n > 0 && sameLine(lines[n-1][0], r) {
|
||||
lines[n-1] = append(lines[n-1], r)
|
||||
continue
|
||||
}
|
||||
lines = append(lines, []Run{r})
|
||||
}
|
||||
return lines
|
||||
}
|
||||
|
||||
// sameLine reports whether two runs sit on the same baseline. A line is
|
||||
// allowed to wobble by a fraction of its own height, which is what a
|
||||
// superscript or a slightly different font does.
|
||||
func sameLine(a, b Run) bool {
|
||||
tol := a.Size * 0.3
|
||||
if tol <= 0 {
|
||||
tol = 1
|
||||
}
|
||||
d := a.Y - b.Y
|
||||
return d < tol && d > -tol
|
||||
}
|
||||
|
||||
// joinLine writes one line out, putting a space wherever the gap between two
|
||||
// pieces is wide enough to be one.
|
||||
func joinLine(line []Run) string {
|
||||
var out strings.Builder
|
||||
for i, r := range line {
|
||||
if i > 0 {
|
||||
prev := line[i-1]
|
||||
gap := r.X - (prev.X + prev.Width)
|
||||
if wantsSpace(prev, r, gap) {
|
||||
out.WriteByte(' ')
|
||||
}
|
||||
}
|
||||
out.WriteString(r.Text)
|
||||
}
|
||||
return strings.TrimRight(out.String(), " ")
|
||||
}
|
||||
|
||||
// wantsSpace decides whether a gap between two pieces of a line is a word
|
||||
// break. A space of its own is never doubled, and a gap of about a third of a
|
||||
// space is where kerning ends and words begin.
|
||||
func wantsSpace(prev, next Run, gap float64) bool {
|
||||
if strings.HasSuffix(prev.Text, " ") || strings.HasPrefix(next.Text, " ") {
|
||||
return false
|
||||
}
|
||||
space := prev.Space
|
||||
if space <= 0 {
|
||||
space = prev.Size * 0.25
|
||||
}
|
||||
return gap > space*0.3
|
||||
}
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"github.com/go-pdfkit/reader"
|
||||
)
|
||||
|
||||
// An Image is a picture the page places, and where it puts it.
|
||||
//
|
||||
// What comes back is the picture as the file holds it — the bytes and the
|
||||
// dictionary describing them — rather than pixels. A JPEG comes out as a JPEG,
|
||||
// which is what anyone pulling pictures out of a document wants: re-encoding
|
||||
// it would lose something for nothing. Turning the other kinds into pixels
|
||||
// means reading their colour space, which is the renderer's work rather than
|
||||
// this package's.
|
||||
type Image struct {
|
||||
// Name is what the page's resources call it, or empty for one written
|
||||
// into the content stream.
|
||||
Name string
|
||||
// Width and Height are the picture's own, in samples.
|
||||
Width, Height int
|
||||
// X and Y are where its bottom left corner lands on the page, and
|
||||
// DrawnWidth and DrawnHeight how large it is drawn, in points. A picture
|
||||
// turned or skewed by the page still reports the box it covers.
|
||||
X, Y float64
|
||||
DrawnWidth, DrawnHeight float64
|
||||
// Filter is the last filter still on the data: DCTDecode for a JPEG that
|
||||
// comes out as one, JPXDecode for a JPEG 2000, and empty for data this
|
||||
// has unfiltered into plain samples.
|
||||
Filter reader.Name
|
||||
// Data is the picture's bytes, with every filter this understands undone.
|
||||
Data []byte
|
||||
// Dict is the image's own dictionary, which says how to read the data:
|
||||
// its colour space, how many bits a sample takes, and whether it is a
|
||||
// mask.
|
||||
Dict reader.Dict
|
||||
// Inline reports a picture written into the content stream rather than
|
||||
// held as a resource of its own.
|
||||
Inline bool
|
||||
}
|
||||
|
||||
// Images is every picture the page places, in the order it places them. The
|
||||
// same picture placed twice comes back twice, since where it lands is part of
|
||||
// what is being asked for.
|
||||
func Images(d *reader.Document, page int) ([]Image, error) {
|
||||
e, err := walk(d, page, true)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return e.images, nil
|
||||
}
|
||||
|
||||
// doXObject handles the Do operator: a form is walked into, and a picture is
|
||||
// noted where it lands.
|
||||
func (e *extractor) doXObject(g *state, operands []reader.Object, resources reader.Dict, depth int) {
|
||||
if len(operands) == 0 {
|
||||
return
|
||||
}
|
||||
name, ok := reader.ToName(operands[0])
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
xobjects, ok := e.doc.GetDict(resources, "XObject")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
stream, ok := reader.ToStream(resolve(e.doc, xobjects.Get(name)))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
switch sub, _ := reader.ToName(resolve(e.doc, stream.Dict.Get("Subtype"))); sub {
|
||||
case "Form":
|
||||
e.doForm(g, stream, resources, depth)
|
||||
case "Image":
|
||||
e.noteImage(g, string(name), stream.Dict, stream.Raw, false)
|
||||
}
|
||||
}
|
||||
|
||||
// doForm walks into a form, which is a piece of a page held apart and drawn
|
||||
// where the page says.
|
||||
func (e *extractor) doForm(g *state, stream *reader.Stream, resources reader.Dict, depth int) {
|
||||
data, img, err := e.doc.DecodeStream(stream)
|
||||
if err != nil || img != "" {
|
||||
return
|
||||
}
|
||||
inner := *g
|
||||
if m := floatArray(e.doc, stream.Dict.Get("Matrix")); len(m) >= 6 {
|
||||
inner.ctm = matrix{m[0], m[1], m[2], m[3], m[4], m[5]}.mul(g.ctm)
|
||||
}
|
||||
own, ok := e.doc.GetDict(stream.Dict, "Resources")
|
||||
if !ok {
|
||||
own = resources
|
||||
}
|
||||
e.run(data, own, inner, depth+1)
|
||||
}
|
||||
|
||||
// inlineImage notes a picture written into the content stream.
|
||||
func (e *extractor) inlineImage(g *state, im *reader.InlineImage) {
|
||||
if im == nil {
|
||||
return
|
||||
}
|
||||
e.noteImage(g, "", im.Expanded(), im.Raw, true)
|
||||
}
|
||||
|
||||
// noteImage works out where a picture lands and records it. A picture occupies
|
||||
// the unit square of the transform in force, so its corners are where that
|
||||
// square's are.
|
||||
func (e *extractor) noteImage(g *state, name string, dict reader.Dict, raw []byte, inline bool) {
|
||||
minX, minY, maxX, maxY := unitSquareBox(g.ctm)
|
||||
img := Image{
|
||||
Name: name,
|
||||
Width: int(intOf(resolve(e.doc, dict.Get("Width")), 0)),
|
||||
Height: int(intOf(resolve(e.doc, dict.Get("Height")), 0)),
|
||||
X: minX,
|
||||
Y: minY,
|
||||
DrawnWidth: maxX - minX,
|
||||
DrawnHeight: maxY - minY,
|
||||
Dict: dict,
|
||||
Inline: inline,
|
||||
}
|
||||
// A caller after the text is not after the pixels, and undoing a
|
||||
// picture's filters to hand it nothing is the most expensive thing a page
|
||||
// can be made to do.
|
||||
if e.wantPictures {
|
||||
data, filter, err := reader.Decode(dict, raw, e.doc.Resolver())
|
||||
if err == nil {
|
||||
img.Data, img.Filter = data, filter
|
||||
} else {
|
||||
img.Data = raw
|
||||
}
|
||||
}
|
||||
e.images = append(e.images, img)
|
||||
}
|
||||
|
||||
// unitSquareBox is the box the unit square covers once the transform has had
|
||||
// its way with it.
|
||||
func unitSquareBox(m matrix) (minX, minY, maxX, maxY float64) {
|
||||
first := true
|
||||
for _, c := range [][2]float64{{0, 0}, {1, 0}, {1, 1}, {0, 1}} {
|
||||
x, y := m.apply(c[0], c[1])
|
||||
if first {
|
||||
minX, minY, maxX, maxY = x, y, x, y
|
||||
first = false
|
||||
continue
|
||||
}
|
||||
minX, maxX = smaller(minX, x), larger(maxX, x)
|
||||
minY, maxY = smaller(minY, y), larger(maxY, y)
|
||||
}
|
||||
return minX, minY, maxX, maxY
|
||||
}
|
||||
|
||||
func smaller(a, b float64) float64 {
|
||||
if a < b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func larger(a, b float64) float64 {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// intOf reads a whole number, or gives a default.
|
||||
func intOf(o reader.Object, def int64) int64 {
|
||||
if v, ok := reader.ToInt(o); ok {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
// floatArray reads an array of numbers, or nothing.
|
||||
func floatArray(d *reader.Document, o reader.Object) []float64 {
|
||||
arr, ok := reader.ToArray(resolve(d, o))
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
out := make([]float64, 0, len(arr))
|
||||
for _, e := range arr {
|
||||
v, ok := reader.ToFloat(resolve(d, e))
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
return out
|
||||
}
|
||||
+175
@@ -0,0 +1,175 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"github.com/go-opentype/opentype"
|
||||
"github.com/go-pdfkit/pdffont"
|
||||
"github.com/go-pdfkit/reader"
|
||||
)
|
||||
|
||||
// attachProgram gives a font a last resort for the codes the document says
|
||||
// nothing about: what its own embedded program calls them.
|
||||
//
|
||||
// A font that says it is symbolic is addressed through its own character map,
|
||||
// and an assumed encoding is a bad guess at it — a mathematical font puts a
|
||||
// capital gamma where the standard encoding puts an inverted exclamation mark.
|
||||
// The program knows, and this is how it is asked.
|
||||
//
|
||||
// There are two ways to ask, and a program answers at most one of them. A
|
||||
// PostScript program names its glyphs and carries an encoding from code to
|
||||
// name, so the name is the answer. A TrueType program usually does neither —
|
||||
// a subset of one names nothing worth reading and has no code-to-name
|
||||
// encoding at all — but it does carry character maps, and walking through one
|
||||
// to the glyph and back out of another says which character the glyph is for.
|
||||
func attachProgram(f *pdffont.Font) {
|
||||
if f.Kind() == pdffont.Composite {
|
||||
return
|
||||
}
|
||||
key, data, ok := f.Program()
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
program, err := readProgram(key, data)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
maps := chooseCharacterMaps(program)
|
||||
// Only a simple font reaches here, and its codes are bytes.
|
||||
f.SetFallback(func(code int) (string, bool) {
|
||||
if r, ok := runeByName(program, code); ok {
|
||||
return string(r), true
|
||||
}
|
||||
if r, ok := runeByCharacterMap(program, maps, code); ok {
|
||||
return string(r), true
|
||||
}
|
||||
return "", false
|
||||
})
|
||||
}
|
||||
|
||||
// runeByName asks the program's own encoding what it calls a code, and reads
|
||||
// the character out of that name. Only a PostScript program — a Type 1 one, or
|
||||
// the CFF outlines of an OpenType font — carries such an encoding.
|
||||
func runeByName(program *opentype.Font, code int) (rune, bool) {
|
||||
gid, ok := program.GlyphIndexByCode(byte(code))
|
||||
if !ok {
|
||||
return 0, false
|
||||
}
|
||||
// A glyph the program does not name comes back as no name at all,
|
||||
// which names no character either.
|
||||
name, _ := program.GlyphName(gid)
|
||||
return pdffont.RuneOfGlyphName(name)
|
||||
}
|
||||
|
||||
// characterMaps says which of a program's cmap subtables are worth addressing,
|
||||
// by index, or -1 for one the program does not carry.
|
||||
type characterMaps struct {
|
||||
// symbol is the Microsoft Symbol subtable, platform 3 encoding 0: a font's
|
||||
// own codes, conventionally written at 0xF000 + code.
|
||||
symbol int
|
||||
// mac is the Macintosh Roman subtable, platform 1 encoding 0, indexed by
|
||||
// single bytes.
|
||||
mac int
|
||||
// unicode is a Unicode subtable — Microsoft Unicode, platform 3 encoding 1
|
||||
// or 10, or anything on platform 0 — indexed by codepoint. This is the one
|
||||
// that is inverted, because it is the only one whose codes are characters.
|
||||
unicode int
|
||||
}
|
||||
|
||||
// chooseCharacterMaps picks out the subtables a font is addressed through and
|
||||
// the one that says which character a glyph is for.
|
||||
//
|
||||
// A font can carry several subtables of a kind; the first of each is taken,
|
||||
// which is what a font that repeats one means by repeating it.
|
||||
func chooseCharacterMaps(program *opentype.Font) characterMaps {
|
||||
m := characterMaps{symbol: -1, mac: -1, unicode: -1}
|
||||
for i := range program.NumCharacterMaps() {
|
||||
platform, encoding, _, _ := program.CharacterMap(i)
|
||||
switch {
|
||||
case platform == 3 && encoding == 0:
|
||||
if m.symbol < 0 {
|
||||
m.symbol = i
|
||||
}
|
||||
case platform == 1 && encoding == 0:
|
||||
if m.mac < 0 {
|
||||
m.mac = i
|
||||
}
|
||||
case platform == 0, platform == 3 && (encoding == 1 || encoding == 10):
|
||||
if m.unicode < 0 {
|
||||
m.unicode = i
|
||||
}
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// runeByCharacterMap walks a code through the font's own character map to a
|
||||
// glyph, and back out of its Unicode character map to the character that glyph
|
||||
// stands for.
|
||||
//
|
||||
// The way in is the one poppler uses for a font the document gave no encoding:
|
||||
// the Microsoft Symbol subtable if there is one, else the Macintosh Roman one,
|
||||
// addressed by the raw code and then, failing that, by 0xF000 + code, which is
|
||||
// where such subtables are conventionally written.
|
||||
//
|
||||
// The way out is the Unicode subtable, inverted. Without one there is no way
|
||||
// out: a glyph on its own says nothing about which character it is, and a
|
||||
// guess would be exactly the wrong letter the caller's guard exists to refuse.
|
||||
func runeByCharacterMap(program *opentype.Font, m characterMaps, code int) (rune, bool) {
|
||||
if m.unicode < 0 {
|
||||
return 0, false
|
||||
}
|
||||
in := m.symbol
|
||||
if in < 0 {
|
||||
in = m.mac
|
||||
}
|
||||
if in < 0 {
|
||||
return 0, false
|
||||
}
|
||||
gid, ok := program.GlyphIndexInMap(in, rune(code))
|
||||
if !ok {
|
||||
gid, ok = program.GlyphIndexInMap(in, rune(0xF000|code))
|
||||
}
|
||||
if !ok {
|
||||
return 0, false
|
||||
}
|
||||
r, ok := program.RuneOfGlyphInMap(m.unicode, gid)
|
||||
if !ok || !readableRune(r) {
|
||||
return 0, false
|
||||
}
|
||||
return r, true
|
||||
}
|
||||
|
||||
// readableRune reports whether a character recovered this way is worth
|
||||
// reporting as text.
|
||||
//
|
||||
// A private-use codepoint is not: it means whatever the font decided it means
|
||||
// and nothing outside the font can read it, so a page full of them searches no
|
||||
// better than a page of nothing and looks, wrongly, like it was read. Control
|
||||
// characters are refused for the same reason — a page does not say them.
|
||||
func readableRune(r rune) bool {
|
||||
switch {
|
||||
case r < 0x20, r >= 0x7F && r <= 0x9F:
|
||||
return false
|
||||
case r >= 0xE000 && r <= 0xF8FF:
|
||||
return false
|
||||
case r >= 0xF0000:
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// readProgram decodes an embedded font program. Which key it arrived under
|
||||
// says what it is: FontFile2 is TrueType, FontFile a PostScript Type 1
|
||||
// program, and FontFile3 either a bare CFF one or a whole OpenType font — the
|
||||
// key alone does not settle that one, so both are tried.
|
||||
func readProgram(key reader.Name, data []byte) (*opentype.Font, error) {
|
||||
switch key {
|
||||
case "FontFile":
|
||||
return opentype.ParseType1(data)
|
||||
case "FontFile3":
|
||||
if f, err := opentype.Parse(data); err == nil {
|
||||
return f, nil
|
||||
}
|
||||
return opentype.ParseCFF(data)
|
||||
}
|
||||
return opentype.Parse(data)
|
||||
}
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"extends": ["github>go-pdfkit/.github"]
|
||||
}
|
||||
+273
@@ -0,0 +1,273 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/go-pdfkit/pdffont"
|
||||
"github.com/go-pdfkit/reader"
|
||||
)
|
||||
|
||||
// maxFormDepth is how deeply one form may draw another before a page is taken
|
||||
// to be drawing itself.
|
||||
const maxFormDepth = 12
|
||||
|
||||
// An extractor walks a content stream keeping just enough state to say where
|
||||
// each piece of text was drawn.
|
||||
type extractor struct {
|
||||
doc *reader.Document
|
||||
runs []Run
|
||||
fonts map[int]*pdffont.Font
|
||||
// tm is where the next glyph goes and tlm where the line began; both
|
||||
// belong to the page rather than to the graphics state.
|
||||
tm, tlm matrix
|
||||
images []Image
|
||||
// wantPictures says whether anybody is going to read Image.Data. When
|
||||
// nobody is, the filters are left undone.
|
||||
wantPictures bool
|
||||
}
|
||||
|
||||
// run walks a content stream.
|
||||
func (e *extractor) run(content []byte, resources reader.Dict, g state, depth int) {
|
||||
if depth > maxFormDepth {
|
||||
return
|
||||
}
|
||||
// The operations are taken one at a time rather than all at once. The
|
||||
// walk only ever goes forwards, so nothing needs the list; keeping it
|
||||
// costs a struct and a slice header for every operator on the page, and
|
||||
// pages are not always small. One arXiv figure holds a single page of
|
||||
// 84.8 MB of content stream and 4 579 973 operations: holding them cost
|
||||
// 2 166 MB where reading them one by one costs 935 MB.
|
||||
scan := reader.NewContentScanner(content)
|
||||
var stack []state
|
||||
for {
|
||||
op, more := scan.Next()
|
||||
if !more {
|
||||
break
|
||||
}
|
||||
n := numbers(op.Operands)
|
||||
switch op.Operator {
|
||||
case "q":
|
||||
stack = append(stack, g)
|
||||
case "Q":
|
||||
if len(stack) > 0 {
|
||||
g = stack[len(stack)-1]
|
||||
stack = stack[:len(stack)-1]
|
||||
}
|
||||
case "cm":
|
||||
if len(n) >= 6 {
|
||||
g.ctm = matrix{n[0], n[1], n[2], n[3], n[4], n[5]}.mul(g.ctm)
|
||||
}
|
||||
case "BT":
|
||||
e.tm, e.tlm = identity, identity
|
||||
case "ET":
|
||||
e.tm, e.tlm = identity, identity
|
||||
case "Tf":
|
||||
e.setFont(&g, op.Operands, n, resources)
|
||||
case "Tc":
|
||||
if len(n) >= 1 {
|
||||
g.text.charSpace = n[0]
|
||||
}
|
||||
case "Tw":
|
||||
if len(n) >= 1 {
|
||||
g.text.wordSpace = n[0]
|
||||
}
|
||||
case "Tz":
|
||||
if len(n) >= 1 {
|
||||
g.text.scale = n[0] / 100
|
||||
}
|
||||
case "TL":
|
||||
if len(n) >= 1 {
|
||||
g.text.leading = n[0]
|
||||
}
|
||||
case "Ts":
|
||||
if len(n) >= 1 {
|
||||
g.text.rise = n[0]
|
||||
}
|
||||
case "Tr":
|
||||
if len(n) >= 1 {
|
||||
g.text.mode = int(n[0])
|
||||
}
|
||||
case "Td":
|
||||
if len(n) >= 2 {
|
||||
e.tlm = matrix{1, 0, 0, 1, n[0], n[1]}.mul(e.tlm)
|
||||
e.tm = e.tlm
|
||||
}
|
||||
case "TD":
|
||||
if len(n) >= 2 {
|
||||
g.text.leading = -n[1]
|
||||
e.tlm = matrix{1, 0, 0, 1, n[0], n[1]}.mul(e.tlm)
|
||||
e.tm = e.tlm
|
||||
}
|
||||
case "Tm":
|
||||
if len(n) >= 6 {
|
||||
e.tlm = matrix{n[0], n[1], n[2], n[3], n[4], n[5]}
|
||||
e.tm = e.tlm
|
||||
}
|
||||
case "T*":
|
||||
e.nextLine(&g)
|
||||
case "Tj":
|
||||
e.showOperands(&g, op.Operands)
|
||||
case "'":
|
||||
e.nextLine(&g)
|
||||
e.showOperands(&g, op.Operands)
|
||||
case "\"":
|
||||
if len(n) >= 2 {
|
||||
g.text.wordSpace, g.text.charSpace = n[0], n[1]
|
||||
}
|
||||
e.nextLine(&g)
|
||||
e.showOperands(&g, op.Operands)
|
||||
case "TJ":
|
||||
e.showArray(&g, op.Operands)
|
||||
case "Do":
|
||||
e.doXObject(&g, op.Operands, resources, depth)
|
||||
case "BI":
|
||||
e.inlineImage(&g, op.Image)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// setFont reads the font a Tf operator names.
|
||||
func (e *extractor) setFont(g *state, operands []reader.Object, n []float64, resources reader.Dict) {
|
||||
if len(operands) < 2 {
|
||||
return
|
||||
}
|
||||
name, ok := reader.ToName(operands[0])
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if len(n) >= 1 {
|
||||
g.text.size = n[len(n)-1]
|
||||
}
|
||||
g.text.fontName = string(name)
|
||||
g.text.font = e.font(name, resources)
|
||||
}
|
||||
|
||||
// font reads a font out of the page's resources, remembering the ones already
|
||||
// read so that a page setting the same font a thousand times reads it once.
|
||||
func (e *extractor) font(name reader.Name, resources reader.Dict) *pdffont.Font {
|
||||
fonts, ok := e.doc.GetDict(resources, "Font")
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
if ref, ok := fonts.Get(name).(reader.Ref); ok {
|
||||
if f, seen := e.fonts[ref.Num]; seen {
|
||||
return f
|
||||
}
|
||||
dict, ok := e.doc.GetDict(fonts, name)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
f := pdffont.Read(e.doc, dict)
|
||||
attachProgram(f)
|
||||
e.fonts[ref.Num] = f
|
||||
return f
|
||||
}
|
||||
dict, ok := e.doc.GetDict(fonts, name)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
f := pdffont.Read(e.doc, dict)
|
||||
attachProgram(f)
|
||||
return f
|
||||
}
|
||||
|
||||
// nextLine moves down by the leading, which is what T* and the quote
|
||||
// operators do.
|
||||
func (e *extractor) nextLine(g *state) {
|
||||
e.tlm = matrix{1, 0, 0, 1, 0, -g.text.leading}.mul(e.tlm)
|
||||
e.tm = e.tlm
|
||||
}
|
||||
|
||||
// showOperands shows the string a Tj-like operator carries.
|
||||
func (e *extractor) showOperands(g *state, operands []reader.Object) {
|
||||
for _, o := range operands {
|
||||
if s, ok := reader.ToString(o); ok {
|
||||
e.show(g, s)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// showArray shows the pieces a TJ operator carries, moving the pen by the
|
||||
// numbers between them.
|
||||
func (e *extractor) showArray(g *state, operands []reader.Object) {
|
||||
for _, o := range operands {
|
||||
arr, ok := reader.ToArray(o)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for _, item := range arr {
|
||||
if s, ok := reader.ToString(item); ok {
|
||||
e.show(g, s)
|
||||
continue
|
||||
}
|
||||
if v, ok := reader.ToFloat(item); ok {
|
||||
e.advance(g, -v/1000*g.text.size*g.text.scale)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// show reads one string as text and records where it was drawn.
|
||||
func (e *extractor) show(g *state, s []byte) {
|
||||
f := g.text.font
|
||||
if f == nil || len(s) == 0 {
|
||||
return
|
||||
}
|
||||
start := e.tm.mul(g.ctm)
|
||||
x, y := start.apply(0, g.text.rise)
|
||||
var text strings.Builder
|
||||
unreadable := false
|
||||
width := 0.0
|
||||
for _, code := range f.Codes(s) {
|
||||
if t, ok := f.Text(code); ok {
|
||||
text.WriteString(pdffont.TrimText(t))
|
||||
} else {
|
||||
unreadable = true
|
||||
}
|
||||
adv := (f.Width(code)*g.text.size + g.text.charSpace) * g.text.scale
|
||||
if f.Kind() != pdffont.Composite && code == ' ' {
|
||||
adv += g.text.wordSpace * g.text.scale
|
||||
}
|
||||
width += adv
|
||||
}
|
||||
e.advance(g, width)
|
||||
if text.Len() == 0 && !unreadable {
|
||||
return
|
||||
}
|
||||
// How far a length in text space reaches on the page: the text matrix and
|
||||
// the page's transform together, which is the same matrix the run's own
|
||||
// position came through.
|
||||
//
|
||||
// Only the page's transform used to be counted here, and the text matrix
|
||||
// was left out. A document is free to put the whole of its scale there —
|
||||
// "/F1 1 Tf" and then a text matrix of ten is how TeX writes every PDF it
|
||||
// has ever produced — and such a document reported its text as one point
|
||||
// tall with letters half a point wide, while saying quite correctly where
|
||||
// on the page each run began.
|
||||
//
|
||||
// Those are the two numbers a word break is decided by. A gap measured in
|
||||
// points was being weighed against a space width ten times too small, so
|
||||
// every ordinary kern between two letters looked like a space: "Original
|
||||
// Domain" came back as "Or ig inal D omain". Across four thousand arXiv
|
||||
// figures, 57% of the words a reader got back were one or two letters
|
||||
// long. Nothing failed, and nothing was slow; the text was simply no
|
||||
// longer text.
|
||||
scale := start.scale()
|
||||
e.runs = append(e.runs, Run{
|
||||
Text: text.String(),
|
||||
X: x,
|
||||
Y: y,
|
||||
Width: width * scale,
|
||||
Size: g.text.size * scale,
|
||||
Font: g.text.fontName,
|
||||
Space: f.Width(' ') * g.text.size * g.text.scale * scale,
|
||||
Invisible: g.text.mode == 3,
|
||||
Unreadable: unreadable,
|
||||
})
|
||||
}
|
||||
|
||||
// advance moves the pen along the line by a distance in text space.
|
||||
func (e *extractor) advance(g *state, by float64) {
|
||||
e.tm = matrix{1, 0, 0, 1, by, 0}.mul(e.tm)
|
||||
}
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
package extract
|
||||
|
||||
import (
|
||||
"github.com/go-pdfkit/pdffont"
|
||||
"github.com/go-pdfkit/reader"
|
||||
)
|
||||
|
||||
// A matrix is the six numbers a PDF transform is written as. Extraction needs
|
||||
// only to multiply them and to move a point, so it carries its own rather than
|
||||
// depending on a drawing library it has no other use for.
|
||||
type matrix struct{ a, b, c, d, e, f float64 }
|
||||
|
||||
// identity is the transform that changes nothing.
|
||||
var identity = matrix{1, 0, 0, 1, 0, 0}
|
||||
|
||||
// mul is the transform that applies m first and then n, which is the order a
|
||||
// PDF composes them in.
|
||||
func (m matrix) mul(n matrix) matrix {
|
||||
return matrix{
|
||||
a: m.a*n.a + m.b*n.c,
|
||||
b: m.a*n.b + m.b*n.d,
|
||||
c: m.c*n.a + m.d*n.c,
|
||||
d: m.c*n.b + m.d*n.d,
|
||||
e: m.e*n.a + m.f*n.c + n.e,
|
||||
f: m.e*n.b + m.f*n.d + n.f,
|
||||
}
|
||||
}
|
||||
|
||||
// apply moves a point.
|
||||
func (m matrix) apply(x, y float64) (float64, float64) {
|
||||
return m.a*x + m.c*y + m.e, m.b*x + m.d*y + m.f
|
||||
}
|
||||
|
||||
// scale is how much the transform stretches a length, taken as the geometric
|
||||
// mean of its two axes so that a rotation counts as no stretch at all.
|
||||
func (m matrix) scale() float64 {
|
||||
x := m.a*m.a + m.b*m.b
|
||||
y := m.c*m.c + m.d*m.d
|
||||
return sqrt(sqrt(x * y))
|
||||
}
|
||||
|
||||
// sqrt without pulling in the whole of math for one use.
|
||||
func sqrt(v float64) float64 {
|
||||
if v <= 0 {
|
||||
return 0
|
||||
}
|
||||
x := v
|
||||
for i := 0; i < 24; i++ {
|
||||
x = (x + v/x) / 2
|
||||
}
|
||||
return x
|
||||
}
|
||||
|
||||
// A textState is everything the showing operators read besides the string.
|
||||
type textState struct {
|
||||
font *pdffont.Font
|
||||
fontName string
|
||||
size float64
|
||||
charSpace float64
|
||||
wordSpace float64
|
||||
scale float64 // horizontal scaling, as a fraction
|
||||
leading float64
|
||||
rise float64
|
||||
mode int
|
||||
}
|
||||
|
||||
// A state is the graphics state, of which extraction needs only the transform
|
||||
// and the text parameters.
|
||||
type state struct {
|
||||
ctm matrix
|
||||
text textState
|
||||
}
|
||||
|
||||
// initialState is the transform a page starts in, which for extraction is the
|
||||
// one that undoes the page's own box: text comes back in the page's
|
||||
// coordinates, counting up from the bottom left of what is visible.
|
||||
func initialState(d *reader.Document, page reader.Dict) state {
|
||||
box := pageBox(d, page)
|
||||
return state{
|
||||
ctm: matrix{1, 0, 0, 1, -box[0], -box[1]},
|
||||
text: textState{scale: 1},
|
||||
}
|
||||
}
|
||||
|
||||
// pageBox is the area of the page that is visible.
|
||||
func pageBox(d *reader.Document, page reader.Dict) [4]float64 {
|
||||
for _, key := range []reader.Name{"CropBox", "MediaBox"} {
|
||||
if b, ok := rectangle(d, page.Get(key)); ok {
|
||||
return b
|
||||
}
|
||||
}
|
||||
return [4]float64{0, 0, 612, 792}
|
||||
}
|
||||
|
||||
// rectangle reads a PDF rectangle, put the right way round.
|
||||
func rectangle(d *reader.Document, o reader.Object) ([4]float64, bool) {
|
||||
var out [4]float64
|
||||
arr, ok := reader.ToArray(resolve(d, o))
|
||||
if !ok || len(arr) < 4 {
|
||||
return out, false
|
||||
}
|
||||
for i := 0; i < 4; i++ {
|
||||
v, ok := reader.ToFloat(resolve(d, arr[i]))
|
||||
if !ok {
|
||||
return out, false
|
||||
}
|
||||
out[i] = v
|
||||
}
|
||||
if out[0] > out[2] {
|
||||
out[0], out[2] = out[2], out[0]
|
||||
}
|
||||
if out[1] > out[3] {
|
||||
out[1], out[3] = out[3], out[1]
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
|
||||
// resolve follows an indirect reference. A document that opened cannot fail to
|
||||
// resolve one.
|
||||
func resolve(d *reader.Document, o reader.Object) reader.Object {
|
||||
out, _ := d.Resolve(o)
|
||||
return out
|
||||
}
|
||||
|
||||
// numbers reads the operands that are numbers.
|
||||
func numbers(operands []reader.Object) []float64 {
|
||||
out := make([]float64, 0, len(operands))
|
||||
for _, o := range operands {
|
||||
if v, ok := reader.ToFloat(o); ok {
|
||||
out = append(out, v)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
Reference in New Issue
Block a user