sift/docs/P5a.md

# Sift P5a — PDF documents

Drafted 2026-09-25 by Claude (Kyle: "support longer / large docs … for ingesting research papers /
thesis", thread `nhf5k9oiu3brzj6ejjbdo4o9te`). Reviewer: Claude. Implementer: Kyle's call.

## 1. Scope

In: PDFs through `/v1/fetch`, `/v2/scrape` and `sift fetch`. Text extraction in pure Go, in a
re-exec'd child of the sift binary. Per-page cache. A "map" answer for documents larger than the
output cap. `--pages` and `--grep` over the cached text. Page markers so digests can cite pages.
Bounded worker pool with single-flight.

Out:
- **P5b:** retrieval tuned for very long documents (page-aware chunks, background embedding,
  BM25 while embeddings build).
- **P5c:** `sift extract <file>` for PDFs already on a host.
- OCR.
- Any other binary type (docx, epub).

The guard, auth and restricted-caller rules do not change: a PDF is fetched exactly like a page.

## 2. Why agents need this (facts, 2026-09-25)

`cleanFetched`/`doFetch` answer every non-html/json/text type with
`415 unsupported content type; hint: browser`. The skill says the same ("PDF in v1 → Browser tool").
An agent that hits a PDF therefore downloads it itself and assembles
`cp … ; file … ; pdfinfo … ; pdftotext -layout … | head -c 20000` in the shell. That is exactly the
command shape tirith flags and the approval gate then has to judge.

## 3. Engine: `github.com/go-pdfkit/extract` (pinned), chosen by bake-off

Bake-off harness: `~/src/pdfbench` on straylight. Data: `~/data/pdfbench`.

Setup:
- Run on orion. Each (engine, file) pair got its own `systemd-run --user` unit with sift's
  hardening: `MemoryDenyWriteExecute`, `MemoryMax=512M`, the same syscall filter,
  `RestrictNamespaces`, `PrivateNetwork`.
- Corpus: 18 public PDFs plus 6 synthetic ones (scanned, AES-256/RC4 owner-only, user-password,
  2,100 pages, truncated, broken xref). No PDF came from a restricted caller.
- Text scored against `pdftotext -layout`.
- 200 byte-level mutants for robustness.

| engine | recall / precision / word order | median ms | median/max MB | fuzz: panics or hangs | salvage |
|---|---|---|---|---|---|
| **go-pdfkit/extract** | **0.937 / 0.943 / 0.915** | 128 | 16 / 66 | **0 / 0** (195 of 200 ok) | **0.79** |
| Detective-XH/gopdf | 0.898 / 0.867 / 0.827 | 122 | 12 / 59 | 0 / 0 (68 of 200 ok) | 0.29 |
| go-pdfium (wasm) | 0.941 / 0.945 / 0.915 **only with MDWE off** | 475 + 2.4 s compile | ~300 / OOM at 512 | 0 / 0 | 0.74 |
| ledongthuc/pdf | 0.47 / 0.60 / 0.43 (glues words) | 31 | 8 / 47 | — | — |

Notes:
- go-pdfium in compiled mode panics under `MemoryDenyWriteExecute` (all 24 files). In
  interpreter mode it takes 7.5 s per PDF and runs out of memory twice.
- Whole documents with extract (no page cap), same sandbox:

  | document | pages | time | memory |
  |---|---|---|---|
  | Karpathy thesis (14.7 MB) | 95 | 0.5 s | 39 MB |
  | Japanese white paper | 307 | 1.2 s | 83 MB |
  | RP2040 datasheet | 642 | 2.0 s | 34 MB |
  | synthetic | 2,100 | 10 s | 59 MB |

**Known extract defect:** the Russian UDHR (Acrobat Distiller 5, 2002) comes out with words glued
together and digits dropped (recall 0.03; gopdf and pdfium read it cleanly).
- File the upstream issue. No fixture is committed: the document is not ours to redistribute.
- Pin `extract v0.3.0` / `reader v0.6.0` in `go.mod` and vendor them, as for every other dependency.

## 4. Detection and fetch

- **Kind `pdf`:**
  - `Content-Type` of `application/pdf` or `application/x-pdf`; or
  - `clean.Sniff` finds `%PDF-` within the first 1024 bytes. The spec allows leading junk, and
    many servers send `application/octet-stream`.
- **Accept header:** add `application/pdf;q=0.8` to the fetch `Accept`.
- **Size and time limits.** For a response that is a PDF (by header, or by sniffing the first
  bytes of the stream), the body limit is **100 MB** instead of `DefaultMaxBody` (5 MB), and the
  fetch timeout is **120 s**. Other kinds keep today's limits. A body over the limit is
  `413 too large (>100 MB)`.

## 5. Extraction in a child process: `sift pdftext`

A hidden subcommand of the same binary, so there is no poppler, no new package, and no change to
the unit's `MemoryDenyWriteExecute`.

- **Parent:**
  - `exec.CommandContext(ctx, "/proc/self/exe", "pdftext", "-max-pages", N)`;
  - empty environment; the PDF bytes on stdin, then close stdin;
  - reads stdout with a cap of 64 MiB of text, and stderr with a cap of 4 KiB;
  - deadline **60 s**, after which the child is killed and the result is
    `422 pdf extraction timed out`.
- **Child:**
  - sets `debug.SetMemoryLimit(768 MiB)` and `RLIMIT_AS` 2 GiB before reading anything;
  - `recover()` around each page;
  - writes JSON Lines:
    ```json
    {"meta":{"pages":212,"title":"…","outline":[{"title":"2 Background","page":14,"level":1}]}}
    {"page":1,"text":"…"}
    {"page":2,"text":"…"}
    ```
  - `outline` comes from `/Outlines` through `go-pdfkit/reader` (title, destination page,
    depth). It is optional and may be `[]`. A page that fails yields
    `{"page":n,"error":"…"}` and extraction continues.
- **Exit codes:** 0 = ok; 3 = encrypted and needs a user password; 4 = unreadable (not a PDF, or
  no page tree); anything else is `422 unreadable PDF`.
- **Page limit:** `-max-pages` 2000. Past it, the map says `pages 1–2000 of N extracted`.
- **Empty text:** if every page's text is blank after trimming, the answer is
  `422 no text layer (scanned PDF?)` with hint `ocr`.
- **Logging:** the child never logs text. The parent logs pages, bytes, ms and the outcome, and
  for restricted callers only the outcome (P2.4 rules).

## 6. Worker pool, single-flight, fairness

Kyle, 2026-09-25: "one or two workers seems tenable".

- **Pool:** 2 extraction slots and 4 waiting places. A request that waits more than 20 s, or
  finds the queue full, gets `503 {"error":"pdf workers busy"}` with `Retry-After: 10`.
- **Single-flight:** keyed by (caller class, content hash). Concurrent requests for the same
  bytes share one extraction. Downloads may still happen twice: a URL is not known to be a PDF
  until it has been fetched.
- **Per caller:** one PDF job in flight. A second concurrent one gets the existing
  `429 busy` path.
- **Cache first:** a request answered from the cache (§7) never touches the pool.

## 7. Cache (dcache)

Two tables, partitioned by caller class like the digest cache:

- `pdf_doc(url_key, caller_class, content_hash, etag, last_modified, meta_json, fetched_at)`
  - TTL 24 h. After that, a conditional GET (`If-None-Match` / `If-Modified-Since`); on 304,
    refresh `fetched_at`.
- `pdf_page(content_hash, caller_class, page, text)`

Both count toward `dcache` `maxBytes` and its eviction, whole documents at a time (oldest
`fetched_at` first). The digest and embedding caches already key on the content hash, so
`--question` on a cached PDF reuses its embeddings.

## 8. What callers get back

Text is assembled with a marker before **every** page, so both agents and digests can cite pages:

```
--- page 1 ---

<page 1 text>

--- page 2 ---

<page 2 text>
```

**`/v1/fetch`, mode `clean`** (new optional request fields `pages`, `grep`):

| request | content |
|---|---|
| whole text ≤ `OutCap` | the whole text with page markers |
| whole text > `OutCap` | **the map**: `# <title>`, `pages N · ~T tokens`, the outline as `- 2 Background (p. 14)` lines, a line saying `use pages="a-b" or grep=…`, then as many leading pages as fit in `OutCap/4` |
| `pages: "41-60"` | those pages with markers; 1-based and inclusive; `a-b`, `a`, or `a-`; capped at 100 pages or `OutCap`, whichever is smaller, with a note when cut; out of range is `400 pages out of range (1–N)` |
| `grep: "<RE2>"` | up to 100 matches as `p. N: <line>` with one line of context each side; Go RE2 (linear time); pattern at most 200 bytes; case-insensitive unless the pattern has upper case |

- `pages` and `grep` together means grep within those pages.
- The JSON envelope gains:
  ```
  "pdf": {"pages", "title", "outline", "extracted", "first", "last", "map", "grep_hits", "note"}
  ```

**`/v1/fetch`, mode `digest` (`question`):**
- The existing engine runs over the text with page markers.
- The system prompt gains one line: "When the document has `--- page N ---` markers, cite
  pages as (p. N)."
- If `Split` yields more than 200 chunks (about 240k tokens), P5a answers
  `422 document too long for question; use grep or pages` and P5b removes that limit.

**`/v2/scrape` (Hermes `web_extract`):** `markdown` is what clean mode would return (the whole
text or the map). `metadata` gains `contentType: application/pdf` and `numPages`. There is no new
request field: Hermes cannot send one.

**Errors**, all under the existing error shape:
- `413` too large;
- `422` no text layer / encrypted / unreadable / timed out / too long for question;
- `503` workers busy.

## 9. CLI and skill

- `sift fetch <url> --pages 41-60`
- `sift fetch <url> --grep 'attention head'`
- The fetch help text lists both.
- `skill/SKILL.md`:
  - remove the 415/PDF row;
  - add "PDFs work; a long one answers with a map first — then use `--pages` / `--grep`, or
    `--question` for a digest that cites pages".

## 10. Unit and metrics

- `nix/module.nix`: `MemoryMax = "1G"` (was 512M; orion has 13 GB available). Nothing else in
  the hardening changes: the child inherits it.
- Metrics:
  - `sift_pdf_extract_total{outcome}`
  - `sift_pdf_extract_seconds` (histogram)
  - `sift_pdf_pages_total`
  - `sift_pdf_pool_wait_seconds`
  - `sift_pdf_cache_hits_total`

## 11. Acceptance (contract) tests

All run offline against `tests/testdata/pdf/` fixtures served by an httptest server:
- small public-domain PDFs only;
- the synthetic scanned, encrypted, truncated and 2,100-page files are regenerated by a script,
  not committed.

1. `application/pdf` and `application/octet-stream` + `%PDF-` both extract; `text/html` never
   takes the PDF path.
2. A 3-page text PDF returns all three pages with `--- page 2 ---` / `--- page 3 ---` markers
   and the envelope `pdf.pages == 3`.
3. A document over `OutCap` returns `pdf.map == true`, the outline lines with page numbers, and
   no more than `OutCap` characters.
4. `pages: "2-3"` returns exactly pages 2–3. `"0-1"`, `"5-4"` and `"999"` are 400.
5. `grep` finds a phrase known to be on page 2 as `p. 2: …`. A 201-byte pattern is 400. The
   pathological regex `(a+)+$` on 1 MB of `a` finishes in under 1 s (RE2).
6. Scanned fixture → `422 no text layer (scanned PDF?)`, hint `ocr`. User-password fixture →
   `422` encrypted. Truncated fixture → 200 with whatever extract salvages, or 422; never 500.
7. **The child is killed on deadline:** a test hook makes `pdftext` sleep. With a 1 s deadline
   the parent answers `422 … timed out` within 2 s and no child process remains.
8. **Pool:** with 2 slots, 4 waiting places and a slow child, the seventh concurrent request
   gets 503 with `Retry-After`. Two concurrent requests for the same URL cause **one**
   upstream GET and one child.
9. **Cache:** a second `pages` request for a fetched document makes **no** upstream GET and
   starts no child. After the TTL, the conditional GET returns 304 and still starts no child.
10. **Restricted caller:** the log lines for a restricted caller's PDF fetch carry no host,
    title or text; its pages land in its own cache partition, invisible to a fleet caller.
11. **Body limit:** 100 MB + 1 byte → 413 for a PDF; an HTML body over 5 MB → 413 (unchanged).
12. **`/v2/scrape`** with the Hermes 0.21 body `{"url","formats":["markdown"]}` on a PDF returns
    `success: true`, markdown with page markers, and `metadata.numPages`.
13. **Digest:** `question` on the 3-page fixture returns a digest containing `(p. ` for a fact
    known to be on one page (the model is stubbed, as in the existing digest tests; the check is
    that the page markers reach the prompt and the prompt line is present).
14. (Dropped: no third-party fixture is committed; the extract defect is tracked upstream.)

## 12. Rollout

1. Implementer builds P5a on a branch; the reviewer (Claude) checks §11, the child-process code,
   and every new limit.
2. Orion pin bump (straylight's lane, as before) and `MemoryMax=1G`.
3. The skill update ships with the pin bump to talos, straylight and titan.
4. Watch `sift_pdf_*` and tirith for a week. The success measure is that the
   `cp … pdftotext …` shell pattern disappears from agents' commands.

## 13. Changes after the plan was written (2026-09-25)

- The implementation plan is `docs/plans/p5a/` (seven tasks, Ornith implementing, acceptance tests
  first). Where this doc and a task file differ, the task file and its given tests win.
- `requestDeadline` 90 s → 210 s and `writeTimeout` 120 s → 240 s, so the PDF path's worst case
  (120 s download + 20 s queue + 60 s extraction) fits; other paths keep their own timeouts.
- The map line reads `pages 1-E extracted` (ASCII hyphen) when the child stopped at its page cap.