sift/docs/P5a.md

Sift P5a — PDF documents

Drafted 2026-09-25 by Claude (Kyle: "support longer / large docs … for ingesting research papers / thesis", thread nhf5k9oiu3brzj6ejjbdo4o9te). Reviewer: Claude. Implementer: Kyle's call.

1. Scope

In: PDFs through /v1/fetch, /v2/scrape and sift fetch. Text extraction in pure Go, in a re-exec'd child of the sift binary. Per-page cache. A "map" answer for documents larger than the output cap. --pages and --grep over the cached text. Page markers so digests can cite pages. Bounded worker pool with single-flight.

Out:

• P5b: retrieval tuned for very long documents (page-aware chunks, background embedding, BM25 while embeddings build).
• P5c: sift extract <file> for PDFs already on a host.
• OCR.
• Any other binary type (docx, epub).

The guard, auth and restricted-caller rules do not change: a PDF is fetched exactly like a page.

2. Why agents need this (facts, 2026-09-25)

cleanFetched/doFetch answer every non-html/json/text type with 415 unsupported content type; hint: browser. The skill says the same ("PDF in v1 → Browser tool"). An agent that hits a PDF therefore downloads it itself and assembles cp … ; file … ; pdfinfo … ; pdftotext -layout … | head -c 20000 in the shell. That is exactly the command shape tirith flags and the approval gate then has to judge.

3. Engine: github.com/go-pdfkit/extract (pinned), chosen by bake-off

Bake-off harness: ~/src/pdfbench on straylight. Data: ~/data/pdfbench.

Setup:

• Run on orion. Each (engine, file) pair got its own systemd-run --user unit with sift's hardening: MemoryDenyWriteExecute, MemoryMax=512M, the same syscall filter, RestrictNamespaces, PrivateNetwork.
• Corpus: 18 public PDFs plus 6 synthetic ones (scanned, AES-256/RC4 owner-only, user-password, 2,100 pages, truncated, broken xref). No PDF came from a restricted caller.
• Text scored against pdftotext -layout.
• 200 byte-level mutants for robustness.

engine	recall / precision / word order	median ms	median/max MB	fuzz: panics or hangs	salvage
go-pdfkit/extract	0.937 / 0.943 / 0.915	128	16 / 66	0 / 0 (195 of 200 ok)	0.79
Detective-XH/gopdf	0.898 / 0.867 / 0.827	122	12 / 59	0 / 0 (68 of 200 ok)	0.29
go-pdfium (wasm)	0.941 / 0.945 / 0.915 only with MDWE off	475 + 2.4 s compile	~300 / OOM at 512	0 / 0	0.74
ledongthuc/pdf	0.47 / 0.60 / 0.43 (glues words)	31	8 / 47	—	—

Notes:

• go-pdfium in compiled mode panics under MemoryDenyWriteExecute (all 24 files). In interpreter mode it takes 7.5 s per PDF and runs out of memory twice.
• Whole documents with extract (no page cap), same sandbox:

  document	pages	time	memory
  Karpathy thesis (14.7 MB)	95	0.5 s	39 MB
  Japanese white paper	307	1.2 s	83 MB
  RP2040 datasheet	642	2.0 s	34 MB
  synthetic	2,100	10 s	59 MB

Known extract defect: the Russian UDHR (Acrobat Distiller 5, 2002) comes out with words glued together and digits dropped (recall 0.03; gopdf and pdfium read it cleanly).

• Keep it as a regression fixture: tests/testdata/pdf/udhr-ru.pdf, expected to fail today and marked as such.
• File the upstream issue.
• Pin extract v0.3.0 / reader v0.6.0 in go.mod and vendor them, as for every other dependency.

4. Detection and fetch

• Kind pdf:
  • Content-Type of application/pdf or application/x-pdf; or
  • clean.Sniff finds %PDF- within the first 1024 bytes. The spec allows leading junk, and many servers send application/octet-stream.
• Accept header: add application/pdf;q=0.8 to the fetch Accept.
• Size and time limits. For a response that is a PDF (by header, or by sniffing the first bytes of the stream), the body limit is 100 MB instead of DefaultMaxBody (5 MB), and the fetch timeout is 120 s. Other kinds keep today's limits. A body over the limit is 413 too large (>100 MB).

5. Extraction in a child process: sift pdftext

A hidden subcommand of the same binary, so there is no poppler, no new package, and no change to the unit's MemoryDenyWriteExecute.

• Parent:
  • exec.CommandContext(ctx, "/proc/self/exe", "pdftext", "-max-pages", N);
  • empty environment; the PDF bytes on stdin, then close stdin;
  • reads stdout with a cap of 64 MiB of text, and stderr with a cap of 4 KiB;
  • deadline 60 s, after which the child is killed and the result is 422 pdf extraction timed out.
• Child:
  • sets debug.SetMemoryLimit(768 MiB) and RLIMIT_AS 2 GiB before reading anything;
  • recover() around each page;
  • writes JSON Lines:

    {"meta":{"pages":212,"title":"…","outline":[{"title":"2 Background","page":14,"level":1}]}}
    {"page":1,"text":"…"}
    {"page":2,"text":"…"}
    

  • outline comes from /Outlines through go-pdfkit/reader (title, destination page, depth). It is optional and may be []. A page that fails yields {"page":n,"error":"…"} and extraction continues.
• Exit codes: 0 = ok; 3 = encrypted and needs a user password; 4 = unreadable (not a PDF, or no page tree); anything else is 422 unreadable PDF.
• Page limit: -max-pages 2000. Past it, the map says pages 1–2000 of N extracted.
• Empty text: if every page's text is blank after trimming, the answer is 422 no text layer (scanned PDF?) with hint ocr.
• Logging: the child never logs text. The parent logs pages, bytes, ms and the outcome, and for restricted callers only the outcome (P2.4 rules).

6. Worker pool, single-flight, fairness

Kyle, 2026-09-25: "one or two workers seems tenable".

• Pool: 2 extraction slots and 4 waiting places. A request that waits more than 20 s, or finds the queue full, gets 503 {"error":"pdf workers busy"} with Retry-After: 10.
• Single-flight: keyed by (caller class, final URL). Concurrent requests for the same document share one download and one extraction.
• Per caller: one PDF job in flight. A second concurrent one gets the existing 429 busy path.
• Cache first: a request answered from the cache (§7) never touches the pool.

7. Cache (dcache)

Two tables, partitioned by caller class like the digest cache:

• pdf_doc(url_key, caller_class, content_hash, etag, last_modified, meta_json, fetched_at)
  • TTL 24 h. After that, a conditional GET (If-None-Match / If-Modified-Since); on 304, refresh fetched_at.
• pdf_page(content_hash, caller_class, page, text)

Both count toward dcache maxBytes and its eviction, whole documents at a time (oldest fetched_at first). The digest and embedding caches already key on the content hash, so --question on a cached PDF reuses its embeddings.

8. What callers get back

Text is assembled with page markers, so both agents and digests can cite pages:

<page 1 text>

--- page 2 ---

<page 2 text>

/v1/fetch, mode clean (new optional request fields pages, grep):

request	content
whole text ≤ OutCap	the whole text with page markers
whole text > OutCap	the map: # <title>, pages N · ~T tokens, the outline as - 2 Background (p. 14) lines, a line saying use pages="a-b" or grep=…, then as many leading pages as fit in OutCap/4
pages: "41-60"	those pages with markers; 1-based and inclusive; a-b, a, or a-; capped at 100 pages or OutCap, whichever is smaller, with a note when cut; out of range is 400 pages out of range (1–N)
grep: "<RE2>"	up to 100 matches as p. N: <line> with one line of context each side; Go RE2 (linear time); pattern at most 200 bytes; case-insensitive unless the pattern has upper case

• pages and grep together means grep within those pages.
• The JSON envelope gains:

  "pdf": {"pages": N, "title": "…", "outline": [...], "returned": [a, b], "map": bool, "grep_hits": n}
  

/v1/fetch, mode digest (question):

• The existing engine runs over the text with page markers.
• The system prompt gains one line: "When the document has --- page N --- markers, cite pages as (p. N)."
• If Split yields more than 200 chunks (about 240k tokens), P5a answers 422 document too long for question; use grep or pages and P5b removes that limit.

/v2/scrape (Hermes web_extract): markdown is what clean mode would return (the whole text or the map). metadata gains contentType: application/pdf and numPages. There is no new request field: Hermes cannot send one.

Errors, all under the existing error shape:

• 413 too large;
• 422 no text layer / encrypted / unreadable / timed out / too long for question;
• 503 workers busy.

9. CLI and skill

• sift fetch <url> --pages 41-60
• sift fetch <url> --grep 'attention head'
• The fetch help text lists both.
• skill/SKILL.md:
  • remove the 415/PDF row;
  • add "PDFs work; a long one answers with a map first — then use --pages / --grep, or --question for a digest that cites pages".

10. Unit and metrics

• nix/module.nix: MemoryMax = "1G" (was 512M; orion has 13 GB available). Nothing else in the hardening changes: the child inherits it.
• Metrics:
  • sift_pdf_extract_total{outcome}
  • sift_pdf_extract_seconds (histogram)
  • sift_pdf_pages_total
  • sift_pdf_pool_wait_seconds
  • sift_pdf_cache_hits_total

11. Acceptance (contract) tests

All run offline against tests/testdata/pdf/ fixtures served by an httptest server:

• small public-domain PDFs only;
• the synthetic scanned, encrypted, truncated and 2,100-page files are regenerated by a script, not committed.

1. application/pdf and application/octet-stream + %PDF- both extract; text/html never takes the PDF path.
2. A 3-page text PDF returns all three pages with --- page 2 --- / --- page 3 --- markers and the envelope pdf.pages == 3.
3. A document over OutCap returns pdf.map == true, the outline lines with page numbers, and no more than OutCap characters.
4. pages: "2-3" returns exactly pages 2–3. "0-1", "5-4" and "999" are 400.
5. grep finds a phrase known to be on page 2 as p. 2: …. A 201-byte pattern is 400. The pathological regex (a+)+$ on 1 MB of a finishes in under 1 s (RE2).
6. Scanned fixture → 422 no text layer (scanned PDF?), hint ocr. User-password fixture → 422 encrypted. Truncated fixture → 200 with whatever extract salvages, or 422; never 500.
7. The child is killed on deadline: a test hook makes pdftext sleep. With a 1 s deadline the parent answers 422 … timed out within 2 s and no child process remains.
8. Pool: with 2 slots, 4 waiting places and a slow child, the seventh concurrent request gets 503 with Retry-After. Two concurrent requests for the same URL cause one upstream GET and one child.
9. Cache: a second pages request for a fetched document makes no upstream GET and starts no child. After the TTL, the conditional GET returns 304 and still starts no child.
10. Restricted caller: the log lines for a restricted caller's PDF fetch carry no host, title or text; its pages land in its own cache partition, invisible to a fleet caller.
11. Body limit: 100 MB + 1 byte → 413 for a PDF; an HTML body over 5 MB → 413 (unchanged).
12. /v2/scrape with the Hermes 0.21 body {"url","formats":["markdown"]} on a PDF returns success: true, markdown with page markers, and metadata.numPages.
13. Digest: question on the 3-page fixture returns a digest containing (p. for a fact known to be on one page (the model is stubbed, as in the existing digest tests; the check is that the page markers reach the prompt and the prompt line is present).
14. The udhr-ru regression fixture is present and marked as a known failure (skip with a reason naming the upstream issue).

12. Rollout

1. Implementer builds P5a on a branch; the reviewer (Claude) checks §11, the child-process code, and every new limit.
2. Orion pin bump (straylight's lane, as before) and MemoryMax=1G.
3. The skill update ships with the pin bump to talos, straylight and titan.
4. Watch sift_pdf_* and tirith for a week. The success measure is that the cp … pdftotext … shell pattern disappears from agents' commands.