docs: P5a — PDF documents (pure-Go extract in a re-exec'd child, per-page cache, map, pages/grep, 2-worker pool)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+244
@@ -0,0 +1,244 @@
|
||||
# Sift P5a — PDF documents
|
||||
|
||||
Drafted 2026-09-25 by Claude (Kyle: "support longer / large docs … for ingesting research papers /
|
||||
thesis", thread `nhf5k9oiu3brzj6ejjbdo4o9te`). Reviewer: Claude. Implementer: Kyle's call.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
In: PDFs through `/v1/fetch`, `/v2/scrape` and `sift fetch`. Text extraction in pure Go, in a
|
||||
re-exec'd child of the sift binary. Per-page cache. A "map" answer for documents larger than the
|
||||
output cap. `--pages` and `--grep` over the cached text. Page markers so digests can cite pages.
|
||||
Bounded worker pool with single-flight.
|
||||
|
||||
Out:
|
||||
- **P5b:** retrieval tuned for very long documents (page-aware chunks, background embedding,
|
||||
BM25 while embeddings build).
|
||||
- **P5c:** `sift extract <file>` for PDFs already on a host.
|
||||
- OCR.
|
||||
- Any other binary type (docx, epub).
|
||||
|
||||
The guard, auth and restricted-caller rules do not change: a PDF is fetched exactly like a page.
|
||||
|
||||
## 2. Why agents need this (facts, 2026-09-25)
|
||||
|
||||
`cleanFetched`/`doFetch` answer every non-html/json/text type with
|
||||
`415 unsupported content type; hint: browser`. The skill says the same ("PDF in v1 → Browser tool").
|
||||
An agent that hits a PDF therefore downloads it itself and assembles
|
||||
`cp … ; file … ; pdfinfo … ; pdftotext -layout … | head -c 20000` in the shell. That is exactly the
|
||||
command shape tirith flags and the approval gate then has to judge.
|
||||
|
||||
## 3. Engine: `github.com/go-pdfkit/extract` (pinned), chosen by bake-off
|
||||
|
||||
Bake-off harness: `~/src/pdfbench` on straylight. Data: `~/data/pdfbench`.
|
||||
|
||||
Setup:
|
||||
- Run on orion. Each (engine, file) pair got its own `systemd-run --user` unit with sift's
|
||||
hardening: `MemoryDenyWriteExecute`, `MemoryMax=512M`, the same syscall filter,
|
||||
`RestrictNamespaces`, `PrivateNetwork`.
|
||||
- Corpus: 18 public PDFs plus 6 synthetic ones (scanned, AES-256/RC4 owner-only, user-password,
|
||||
2,100 pages, truncated, broken xref). No PDF came from a restricted caller.
|
||||
- Text scored against `pdftotext -layout`.
|
||||
- 200 byte-level mutants for robustness.
|
||||
|
||||
| engine | recall / precision / word order | median ms | median/max MB | fuzz: panics or hangs | salvage |
|
||||
|---|---|---|---|---|---|
|
||||
| **go-pdfkit/extract** | **0.937 / 0.943 / 0.915** | 128 | 16 / 66 | **0 / 0** (195 of 200 ok) | **0.79** |
|
||||
| Detective-XH/gopdf | 0.898 / 0.867 / 0.827 | 122 | 12 / 59 | 0 / 0 (68 of 200 ok) | 0.29 |
|
||||
| go-pdfium (wasm) | 0.941 / 0.945 / 0.915 **only with MDWE off** | 475 + 2.4 s compile | ~300 / OOM at 512 | 0 / 0 | 0.74 |
|
||||
| ledongthuc/pdf | 0.47 / 0.60 / 0.43 (glues words) | 31 | 8 / 47 | — | — |
|
||||
|
||||
Notes:
|
||||
- go-pdfium in compiled mode panics under `MemoryDenyWriteExecute` (all 24 files). In
|
||||
interpreter mode it takes 7.5 s per PDF and runs out of memory twice.
|
||||
- Whole documents with extract (no page cap), same sandbox:
|
||||
|
||||
| document | pages | time | memory |
|
||||
|---|---|---|---|
|
||||
| Karpathy thesis (14.7 MB) | 95 | 0.5 s | 39 MB |
|
||||
| Japanese white paper | 307 | 1.2 s | 83 MB |
|
||||
| RP2040 datasheet | 642 | 2.0 s | 34 MB |
|
||||
| synthetic | 2,100 | 10 s | 59 MB |
|
||||
|
||||
**Known extract defect:** the Russian UDHR (Acrobat Distiller 5, 2002) comes out with words glued
|
||||
together and digits dropped (recall 0.03; gopdf and pdfium read it cleanly).
|
||||
- Keep it as a regression fixture: `tests/testdata/pdf/udhr-ru.pdf`, expected to fail today and
|
||||
marked as such.
|
||||
- File the upstream issue.
|
||||
- Pin `extract v0.3.0` / `reader v0.6.0` in `go.mod` and vendor them, as for every other dependency.
|
||||
|
||||
## 4. Detection and fetch
|
||||
|
||||
- **Kind `pdf`:**
|
||||
- `Content-Type` of `application/pdf` or `application/x-pdf`; or
|
||||
- `clean.Sniff` finds `%PDF-` within the first 1024 bytes. The spec allows leading junk, and
|
||||
many servers send `application/octet-stream`.
|
||||
- **Accept header:** add `application/pdf;q=0.8` to the fetch `Accept`.
|
||||
- **Size and time limits.** For a response that is a PDF (by header, or by sniffing the first
|
||||
bytes of the stream), the body limit is **100 MB** instead of `DefaultMaxBody` (5 MB), and the
|
||||
fetch timeout is **120 s**. Other kinds keep today's limits. A body over the limit is
|
||||
`413 too large (>100 MB)`.
|
||||
|
||||
## 5. Extraction in a child process: `sift pdftext`
|
||||
|
||||
A hidden subcommand of the same binary, so there is no poppler, no new package, and no change to
|
||||
the unit's `MemoryDenyWriteExecute`.
|
||||
|
||||
- **Parent:**
|
||||
- `exec.CommandContext(ctx, "/proc/self/exe", "pdftext", "-max-pages", N)`;
|
||||
- empty environment; the PDF bytes on stdin, then close stdin;
|
||||
- reads stdout with a cap of 64 MiB of text, and stderr with a cap of 4 KiB;
|
||||
- deadline **60 s**, after which the child is killed and the result is
|
||||
`422 pdf extraction timed out`.
|
||||
- **Child:**
|
||||
- sets `debug.SetMemoryLimit(768 MiB)` and `RLIMIT_AS` 2 GiB before reading anything;
|
||||
- `recover()` around each page;
|
||||
- writes JSON Lines:
|
||||
```json
|
||||
{"meta":{"pages":212,"title":"…","outline":[{"title":"2 Background","page":14,"level":1}]}}
|
||||
{"page":1,"text":"…"}
|
||||
{"page":2,"text":"…"}
|
||||
```
|
||||
- `outline` comes from `/Outlines` through `go-pdfkit/reader` (title, destination page,
|
||||
depth). It is optional and may be `[]`. A page that fails yields
|
||||
`{"page":n,"error":"…"}` and extraction continues.
|
||||
- **Exit codes:** 0 = ok; 3 = encrypted and needs a user password; 4 = unreadable (not a PDF, or
|
||||
no page tree); anything else is `422 unreadable PDF`.
|
||||
- **Page limit:** `-max-pages` 2000. Past it, the map says `pages 1–2000 of N extracted`.
|
||||
- **Empty text:** if every page's text is blank after trimming, the answer is
|
||||
`422 no text layer (scanned PDF?)` with hint `ocr`.
|
||||
- **Logging:** the child never logs text. The parent logs pages, bytes, ms and the outcome, and
|
||||
for restricted callers only the outcome (P2.4 rules).
|
||||
|
||||
## 6. Worker pool, single-flight, fairness
|
||||
|
||||
Kyle, 2026-09-25: "one or two workers seems tenable".
|
||||
|
||||
- **Pool:** 2 extraction slots and 4 waiting places. A request that waits more than 20 s, or
|
||||
finds the queue full, gets `503 {"error":"pdf workers busy"}` with `Retry-After: 10`.
|
||||
- **Single-flight:** keyed by (caller class, final URL). Concurrent requests for the same
|
||||
document share one download and one extraction.
|
||||
- **Per caller:** one PDF job in flight. A second concurrent one gets the existing
|
||||
`429 busy` path.
|
||||
- **Cache first:** a request answered from the cache (§7) never touches the pool.
|
||||
|
||||
## 7. Cache (dcache)
|
||||
|
||||
Two tables, partitioned by caller class like the digest cache:
|
||||
|
||||
- `pdf_doc(url_key, caller_class, content_hash, etag, last_modified, meta_json, fetched_at)`
|
||||
- TTL 24 h. After that, a conditional GET (`If-None-Match` / `If-Modified-Since`); on 304,
|
||||
refresh `fetched_at`.
|
||||
- `pdf_page(content_hash, caller_class, page, text)`
|
||||
|
||||
Both count toward `dcache` `maxBytes` and its eviction, whole documents at a time (oldest
|
||||
`fetched_at` first). The digest and embedding caches already key on the content hash, so
|
||||
`--question` on a cached PDF reuses its embeddings.
|
||||
|
||||
## 8. What callers get back
|
||||
|
||||
Text is assembled with page markers, so both agents and digests can cite pages:
|
||||
|
||||
```
|
||||
<page 1 text>
|
||||
|
||||
--- page 2 ---
|
||||
|
||||
<page 2 text>
|
||||
```
|
||||
|
||||
**`/v1/fetch`, mode `clean`** (new optional request fields `pages`, `grep`):
|
||||
|
||||
| request | content |
|
||||
|---|---|
|
||||
| whole text ≤ `OutCap` | the whole text with page markers |
|
||||
| whole text > `OutCap` | **the map**: `# <title>`, `pages N · ~T tokens`, the outline as `- 2 Background (p. 14)` lines, a line saying `use pages="a-b" or grep=…`, then as many leading pages as fit in `OutCap/4` |
|
||||
| `pages: "41-60"` | those pages with markers; 1-based and inclusive; `a-b`, `a`, or `a-`; capped at 100 pages or `OutCap`, whichever is smaller, with a note when cut; out of range is `400 pages out of range (1–N)` |
|
||||
| `grep: "<RE2>"` | up to 100 matches as `p. N: <line>` with one line of context each side; Go RE2 (linear time); pattern at most 200 bytes; case-insensitive unless the pattern has upper case |
|
||||
|
||||
- `pages` and `grep` together means grep within those pages.
|
||||
- The JSON envelope gains:
|
||||
```
|
||||
"pdf": {"pages": N, "title": "…", "outline": [...], "returned": [a, b], "map": bool, "grep_hits": n}
|
||||
```
|
||||
|
||||
**`/v1/fetch`, mode `digest` (`question`):**
|
||||
- The existing engine runs over the text with page markers.
|
||||
- The system prompt gains one line: "When the document has `--- page N ---` markers, cite
|
||||
pages as (p. N)."
|
||||
- If `Split` yields more than 200 chunks (about 240k tokens), P5a answers
|
||||
`422 document too long for question; use grep or pages` and P5b removes that limit.
|
||||
|
||||
**`/v2/scrape` (Hermes `web_extract`):** `markdown` is what clean mode would return (the whole
|
||||
text or the map). `metadata` gains `contentType: application/pdf` and `numPages`. There is no new
|
||||
request field: Hermes cannot send one.
|
||||
|
||||
**Errors**, all under the existing error shape:
|
||||
- `413` too large;
|
||||
- `422` no text layer / encrypted / unreadable / timed out / too long for question;
|
||||
- `503` workers busy.
|
||||
|
||||
## 9. CLI and skill
|
||||
|
||||
- `sift fetch <url> --pages 41-60`
|
||||
- `sift fetch <url> --grep 'attention head'`
|
||||
- The fetch help text lists both.
|
||||
- `skill/SKILL.md`:
|
||||
- remove the 415/PDF row;
|
||||
- add "PDFs work; a long one answers with a map first — then use `--pages` / `--grep`, or
|
||||
`--question` for a digest that cites pages".
|
||||
|
||||
## 10. Unit and metrics
|
||||
|
||||
- `nix/module.nix`: `MemoryMax = "1G"` (was 512M; orion has 13 GB available). Nothing else in
|
||||
the hardening changes: the child inherits it.
|
||||
- Metrics:
|
||||
- `sift_pdf_extract_total{outcome}`
|
||||
- `sift_pdf_extract_seconds` (histogram)
|
||||
- `sift_pdf_pages_total`
|
||||
- `sift_pdf_pool_wait_seconds`
|
||||
- `sift_pdf_cache_hits_total`
|
||||
|
||||
## 11. Acceptance (contract) tests
|
||||
|
||||
All run offline against `tests/testdata/pdf/` fixtures served by an httptest server:
|
||||
- small public-domain PDFs only;
|
||||
- the synthetic scanned, encrypted, truncated and 2,100-page files are regenerated by a script,
|
||||
not committed.
|
||||
|
||||
1. `application/pdf` and `application/octet-stream` + `%PDF-` both extract; `text/html` never
|
||||
takes the PDF path.
|
||||
2. A 3-page text PDF returns all three pages with `--- page 2 ---` / `--- page 3 ---` markers
|
||||
and the envelope `pdf.pages == 3`.
|
||||
3. A document over `OutCap` returns `pdf.map == true`, the outline lines with page numbers, and
|
||||
no more than `OutCap` characters.
|
||||
4. `pages: "2-3"` returns exactly pages 2–3. `"0-1"`, `"5-4"` and `"999"` are 400.
|
||||
5. `grep` finds a phrase known to be on page 2 as `p. 2: …`. A 201-byte pattern is 400. The
|
||||
pathological regex `(a+)+$` on 1 MB of `a` finishes in under 1 s (RE2).
|
||||
6. Scanned fixture → `422 no text layer (scanned PDF?)`, hint `ocr`. User-password fixture →
|
||||
`422` encrypted. Truncated fixture → 200 with whatever extract salvages, or 422; never 500.
|
||||
7. **The child is killed on deadline:** a test hook makes `pdftext` sleep. With a 1 s deadline
|
||||
the parent answers `422 … timed out` within 2 s and no child process remains.
|
||||
8. **Pool:** with 2 slots, 4 waiting places and a slow child, the seventh concurrent request
|
||||
gets 503 with `Retry-After`. Two concurrent requests for the same URL cause **one**
|
||||
upstream GET and one child.
|
||||
9. **Cache:** a second `pages` request for a fetched document makes **no** upstream GET and
|
||||
starts no child. After the TTL, the conditional GET returns 304 and still starts no child.
|
||||
10. **Restricted caller:** the log lines for a restricted caller's PDF fetch carry no host,
|
||||
title or text; its pages land in its own cache partition, invisible to a fleet caller.
|
||||
11. **Body limit:** 100 MB + 1 byte → 413 for a PDF; an HTML body over 5 MB → 413 (unchanged).
|
||||
12. **`/v2/scrape`** with the Hermes 0.21 body `{"url","formats":["markdown"]}` on a PDF returns
|
||||
`success: true`, markdown with page markers, and `metadata.numPages`.
|
||||
13. **Digest:** `question` on the 3-page fixture returns a digest containing `(p. ` for a fact
|
||||
known to be on one page (the model is stubbed, as in the existing digest tests; the check is
|
||||
that the page markers reach the prompt and the prompt line is present).
|
||||
14. **The `udhr-ru` regression fixture** is present and marked as a known failure (skip with a
|
||||
reason naming the upstream issue).
|
||||
|
||||
## 12. Rollout
|
||||
|
||||
1. Implementer builds P5a on a branch; the reviewer (Claude) checks §11, the child-process code,
|
||||
and every new limit.
|
||||
2. Orion pin bump (straylight's lane, as before) and `MemoryMax=1G`.
|
||||
3. The skill update ships with the pin bump to talos, straylight and titan.
|
||||
4. Watch `sift_pdf_*` and tirith for a week. The success measure is that the
|
||||
`cp … pdftotext …` shell pattern disappears from agents' commands.
|
||||
Reference in New Issue
Block a user