P5a plan: PDF documents — seven Ornith tasks with acceptance tests first
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+17
-9
@@ -61,9 +61,7 @@ Notes:
|
||||
|
||||
**Known extract defect:** the Russian UDHR (Acrobat Distiller 5, 2002) comes out with words glued
|
||||
together and digits dropped (recall 0.03; gopdf and pdfium read it cleanly).
|
||||
- Keep it as a regression fixture: `tests/testdata/pdf/udhr-ru.pdf`, expected to fail today and
|
||||
marked as such.
|
||||
- File the upstream issue.
|
||||
- File the upstream issue. No fixture is committed: the document is not ours to redistribute.
|
||||
- Pin `extract v0.3.0` / `reader v0.6.0` in `go.mod` and vendor them, as for every other dependency.
|
||||
|
||||
## 4. Detection and fetch
|
||||
@@ -115,8 +113,9 @@ Kyle, 2026-09-25: "one or two workers seems tenable".
|
||||
|
||||
- **Pool:** 2 extraction slots and 4 waiting places. A request that waits more than 20 s, or
|
||||
finds the queue full, gets `503 {"error":"pdf workers busy"}` with `Retry-After: 10`.
|
||||
- **Single-flight:** keyed by (caller class, final URL). Concurrent requests for the same
|
||||
document share one download and one extraction.
|
||||
- **Single-flight:** keyed by (caller class, content hash). Concurrent requests for the same
|
||||
bytes share one extraction. Downloads may still happen twice: a URL is not known to be a PDF
|
||||
until it has been fetched.
|
||||
- **Per caller:** one PDF job in flight. A second concurrent one gets the existing
|
||||
`429 busy` path.
|
||||
- **Cache first:** a request answered from the cache (§7) never touches the pool.
|
||||
@@ -136,9 +135,11 @@ Both count toward `dcache` `maxBytes` and its eviction, whole documents at a tim
|
||||
|
||||
## 8. What callers get back
|
||||
|
||||
Text is assembled with page markers, so both agents and digests can cite pages:
|
||||
Text is assembled with a marker before **every** page, so both agents and digests can cite pages:
|
||||
|
||||
```
|
||||
--- page 1 ---
|
||||
|
||||
<page 1 text>
|
||||
|
||||
--- page 2 ---
|
||||
@@ -158,7 +159,7 @@ Text is assembled with page markers, so both agents and digests can cite pages:
|
||||
- `pages` and `grep` together means grep within those pages.
|
||||
- The JSON envelope gains:
|
||||
```
|
||||
"pdf": {"pages": N, "title": "…", "outline": [...], "returned": [a, b], "map": bool, "grep_hits": n}
|
||||
"pdf": {"pages", "title", "outline", "extracted", "first", "last", "map", "grep_hits", "note"}
|
||||
```
|
||||
|
||||
**`/v1/fetch`, mode `digest` (`question`):**
|
||||
@@ -231,8 +232,7 @@ All run offline against `tests/testdata/pdf/` fixtures served by an httptest ser
|
||||
13. **Digest:** `question` on the 3-page fixture returns a digest containing `(p. ` for a fact
|
||||
known to be on one page (the model is stubbed, as in the existing digest tests; the check is
|
||||
that the page markers reach the prompt and the prompt line is present).
|
||||
14. **The `udhr-ru` regression fixture** is present and marked as a known failure (skip with a
|
||||
reason naming the upstream issue).
|
||||
14. (Dropped: no third-party fixture is committed; the extract defect is tracked upstream.)
|
||||
|
||||
## 12. Rollout
|
||||
|
||||
@@ -242,3 +242,11 @@ All run offline against `tests/testdata/pdf/` fixtures served by an httptest ser
|
||||
3. The skill update ships with the pin bump to talos, straylight and titan.
|
||||
4. Watch `sift_pdf_*` and tirith for a week. The success measure is that the
|
||||
`cp … pdftotext …` shell pattern disappears from agents' commands.
|
||||
|
||||
## 13. Changes after the plan was written (2026-09-25)
|
||||
|
||||
- The implementation plan is `docs/plans/p5a/` (seven tasks, Ornith implementing, acceptance tests
|
||||
first). Where this doc and a task file differ, the task file and its given tests win.
|
||||
- `requestDeadline` 90 s → 210 s and `writeTimeout` 120 s → 240 s, so the PDF path's worst case
|
||||
(120 s download + 20 s queue + 60 s extraction) fits; other paths keep their own timeouts.
|
||||
- The map line reads `pages 1-E extracted` (ASCII hyphen) when the child stopped at its page cap.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Implementer log
|
||||
|
||||
One row per task session that ends in a commit. Newest last.
|
||||
|
||||
| Task | Date | Status | Sessions | Gate | Deviations | Summary |
|
||||
|---|---|---|---|---|---|---|
|
||||
@@ -0,0 +1,100 @@
|
||||
# P5a task 01: recognise PDFs; give them a larger body cap, a longer deadline, and revalidation
|
||||
|
||||
**Branch:** `p5a` (`git switch -c p5a master` if it does not exist, else `git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `clean, fetch: recognise PDFs; 100 MB / 120 s for PDFs; FetchIf revalidates`
|
||||
|
||||
## Goal
|
||||
|
||||
Today every response that is not HTML, JSON or text is `415 unsupported content type`. After this
|
||||
task `clean.Kind` names PDFs, and the fetcher lets a PDF be as large as 100 MB and take 120 s
|
||||
while a web page keeps 5 MB and 20 s. `FetchIf` sends `If-None-Match` / `If-Modified-Since` so a
|
||||
cached document can be revalidated with a 304. Nothing uses these yet (task 06 does).
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/clean/kind_test.go`, `internal/fetch/pdf_test.go`
|
||||
- Create: `internal/fetch/pdf.go` (the PDF rule and `FetchIf` live here; keep `fetch.go`'s
|
||||
changes to what the new file needs)
|
||||
- Modify: `internal/clean/clean.go`, `internal/fetch/fetch.go`, `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package clean
|
||||
|
||||
// Kind classifies a fetched body: "html", "json", "text", "pdf" or "other".
|
||||
func Kind(contentType string, body []byte) string
|
||||
|
||||
package fetch
|
||||
|
||||
const (
|
||||
DefaultPDFMaxBody = 100 << 20
|
||||
DefaultPDFTimeout = 120 * time.Second
|
||||
)
|
||||
|
||||
// in Fetcher:
|
||||
PDFMaxBody int64 // 0 = DefaultPDFMaxBody
|
||||
PDFTimeout time.Duration // 0 = DefaultPDFTimeout
|
||||
|
||||
// FetchIf is Fetch with validators: a non-empty etag is sent as If-None-Match and a non-empty
|
||||
// lastModified as If-Modified-Since. A 304 answer returns a Resource with Status 304, an empty
|
||||
// Body and a nil error. With both empty it is exactly Fetch.
|
||||
func (f *Fetcher) FetchIf(ctx context.Context, rawURL, etag, lastModified string) (*Resource, error)
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **`Kind`** (the media type is lower-cased and its parameters dropped, as `MediaKind` does):
|
||||
- `application/pdf` or `application/x-pdf` → `"pdf"`, whatever the body.
|
||||
- `text/html`, `application/xhtml+xml` → `"html"`; `application/json` or `*+json` → `"json"`;
|
||||
`text/plain`, `text/markdown`, `text/csv` → `"text"` (as today).
|
||||
- `application/octet-stream` or `binary/octet-stream` → `"pdf"` if `%PDF-` occurs within the
|
||||
first 1024 bytes of the body, else `"other"`.
|
||||
- empty Content-Type → `"pdf"` if `%PDF-` occurs within the first 1024 bytes, else today's
|
||||
`Sniff` (html / json / text).
|
||||
- anything else → `"other"`.
|
||||
`MediaKind` also returns `"pdf"` for the two PDF types; its other answers do not change.
|
||||
2. **Fetch limits.** A response is a PDF when its Content-Type is a PDF type, or when it is empty
|
||||
or octet-stream and the body starts (within 1024 bytes) with `%PDF-`. For a PDF the body cap
|
||||
is `PDFMaxBody` and the deadline is `PDFTimeout`, both counted from the start of the request;
|
||||
for anything else they stay `MaxBody` and `Timeout`. Over the cap is the existing
|
||||
`413 too large (>N MB decompressed)` with N from the cap that applied.
|
||||
3. **How to get the deadline right.** `Fetch` today makes one context with `Timeout`, and the
|
||||
shared `http.Client` has `Timeout` set too — both would cut a PDF off at 20 s. Make the
|
||||
client's `Timeout` at least the PDF deadline (or unset it and rely on contexts), give the
|
||||
request a context with the **PDF** deadline, and cancel it at the ordinary deadline unless the
|
||||
response has been recognised as a PDF by then (`time.AfterFunc` + a flag you set once the
|
||||
headers, or the first bytes of an octet-stream body, say PDF). Read the body through
|
||||
`io.LimitReader` with the cap that applies (+1 to detect overflow), as today.
|
||||
4. `FetchIf` goes through the same guard, client, limits and error mapping as `Fetch`; only the
|
||||
two request headers differ. Do not copy `Fetch`: factor the shared part.
|
||||
|
||||
## Facts
|
||||
|
||||
- The tests reach an `httptest` server on 127.0.0.1 through a `guard.Guard` with
|
||||
`AllowPrefixes: 127.0.0.0/8`; the guard itself is not yours to change.
|
||||
- `internal/fetch` has no other tests; `internal/api` tests exercise `Fetch` through the API and
|
||||
must stay green.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** Branch as above; copy the two given tests.
|
||||
- [ ] **2. See them fail:** `go test -count=1 ./internal/clean/ ./internal/fetch/` → `Kind`,
|
||||
`FetchIf`, `DefaultPDFMaxBody` undefined.
|
||||
- [ ] **3.** `clean.go`. **4.** `fetch/pdf.go` and the smallest changes to `fetch.go`.
|
||||
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/clean/ ./internal/fetch/ ./internal/api/` → `ok`.
|
||||
- [ ] **6.** `make gate`. **7.** Row `p5a/01-kind-fetch`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/clean internal/fetch docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given tests and every earlier test pass under `-race -count=3`; `make gate` prints
|
||||
`gate: ok`; given files byte-identical.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A given test needs a change to `internal/guard`.
|
||||
@@ -0,0 +1,152 @@
|
||||
# P5a task 02: `pdftext` — extract a PDF in a child process
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `pdftext: extract PDF text in a re-exec'd child (sift pdftext) with a deadline`
|
||||
|
||||
## Goal
|
||||
|
||||
Parsing a PDF from the internet is the riskiest thing sift will do. The parser is pure Go
|
||||
(`github.com/go-pdfkit/extract` + `github.com/go-pdfkit/reader`, already in `vendor/`), but a
|
||||
goroutine cannot be killed, so it runs in a **child process**: the sift binary re-executed as
|
||||
`sift pdftext`, reading the PDF on stdin and writing JSON Lines on stdout. The parent kills it at
|
||||
a deadline. This task builds both halves; nothing calls them yet (task 06 does).
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/pdftest/pdftest.go` (test support that builds PDFs; never edit it),
|
||||
`tests/testdata/pdf/encrypted-user.pdf`, `tests/testdata/pdf/encrypted-owner.pdf`,
|
||||
`tests/testdata/pdf/README.md`, `internal/pdftext/pdftext_test.go`
|
||||
- Create: `internal/pdftext/pdftext.go` (types, errors, `Extract`, `ChildMain`),
|
||||
`internal/pdftext/outline.go` (the outline walk), `internal/pdftext/runner.go` (`Runner`)
|
||||
- Modify: `cmd/sift/main.go` (one `case "pdftext":` in the command switch), `docs/implementer-log.md`
|
||||
- Delete: `tools.go` (it only pinned the two modules until this package imports them)
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package pdftext
|
||||
|
||||
type OutlineItem struct {
|
||||
Title string `json:"title"`
|
||||
Page int `json:"page"` // 1-based
|
||||
Level int `json:"level"` // 1 = top level
|
||||
}
|
||||
|
||||
type Meta struct {
|
||||
Pages int `json:"pages"` // pages in the document (not only those extracted)
|
||||
Title string `json:"title"` // /Info /Title, "" when absent
|
||||
Outline []OutlineItem `json:"outline"` // never nil: [] when the document has none
|
||||
}
|
||||
|
||||
// Doc is an extracted document: Text[i] is page i+1's text, for the pages extracted
|
||||
// (len(Text) ≤ Pages). Failed lists pages whose extraction failed (their Text is "").
|
||||
type Doc struct {
|
||||
Meta
|
||||
Text []string
|
||||
Failed []int
|
||||
}
|
||||
|
||||
var (
|
||||
ErrEncrypted = errors.New("encrypted PDF (password required)")
|
||||
ErrUnreadable = errors.New("unreadable PDF")
|
||||
ErrNoText = errors.New("no text layer (scanned PDF?)")
|
||||
ErrTimeout = errors.New("pdf extraction timed out")
|
||||
ErrTooMuchText = errors.New("pdf text over the output limit")
|
||||
)
|
||||
|
||||
// Extract writes one {"meta":{…}} line, then one {"page":n,"text":"…"} line per page from 1 up
|
||||
// to min(Pages, maxPages); a page that fails yields {"page":n,"error":"…"} and extraction goes
|
||||
// on. It returns ErrEncrypted when the document needs a user password, ErrUnreadable when it
|
||||
// cannot be opened or has no pages, and never panics.
|
||||
func Extract(pdf []byte, maxPages int, w io.Writer) error
|
||||
|
||||
// ChildMain is `sift pdftext`: flags (-max-pages N, default 2000), the PDF on stdin (read with a
|
||||
// 200 MiB cap), Extract to stdout. Exit codes: 0 ok, 3 ErrEncrypted, 4 ErrUnreadable, 1 anything
|
||||
// else.
|
||||
func ChildMain(args []string, stdin io.Reader, stdout io.Writer) int
|
||||
|
||||
type Runner struct {
|
||||
Exe string // "" = /proc/self/exe
|
||||
Args []string // nil = []string{"pdftext"}; -max-pages is appended
|
||||
Env []string // the child's whole environment; nil = none
|
||||
Deadline time.Duration // 0 = 60 s
|
||||
MaxText int64 // bytes of child output; 0 = 64 MiB
|
||||
MaxPages int // 0 = 2000
|
||||
}
|
||||
|
||||
// Run executes the child, writes pdf to its stdin, and parses its output into a Doc.
|
||||
func (r *Runner) Run(ctx context.Context, pdf []byte) (*Doc, error)
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Extract.**
|
||||
- Open with `reader.Open(pdf)`. `reader.ErrWrongPassword` (from `errors.Is`) → `ErrEncrypted`.
|
||||
Any other open error → `ErrUnreadable`. `PageCount() == 0` → `ErrUnreadable`.
|
||||
- Each page: `extract.Text(doc, n)` with `n` counting from 1, inside a `recover()`; an error
|
||||
or a panic becomes that page's `"error"` line.
|
||||
- Title: the trailer's `/Info` dictionary, key `Title`, as a text string (rule 3).
|
||||
- The meta line comes first and carries the full page count even when `maxPages` is smaller.
|
||||
2. **Outline** (`outline.go`). From the trailer's `/Root` (the catalog) follow `/Outlines` →
|
||||
`/First`, then each item's `/Next`; an item's children start at its `/First` (level + 1).
|
||||
- An item's page comes from `/Dest` (an array whose first element is a page reference), or
|
||||
from `/A` when it is a dictionary with `/S /GoTo` and a `/D` array. Map the reference to a
|
||||
1-based page number by comparing it with `doc.PageRef(i)` for `i` = 1…`PageCount()`.
|
||||
- Skip an item whose destination is a name or cannot be resolved; keep walking.
|
||||
- Stop after 10,000 items, and never visit an object twice (a malformed outline can loop).
|
||||
3. **Text strings** (`/Title` in Info and in outline items): a `reader.String` is raw bytes.
|
||||
Bytes starting `FE FF` are UTF-16BE (decode the rest with `unicode/utf16`); otherwise treat
|
||||
each byte as a Latin-1 code point (close enough to PDFDocEncoding for titles). Trim spaces.
|
||||
4. **ChildMain** runs with a soft memory limit: call `debug.SetMemoryLimit(768 << 20)` first.
|
||||
It must not read any environment variable and must not open the network.
|
||||
5. **Runner.Run.**
|
||||
- `exec.CommandContext` with a context that ends at `Deadline` (or `ctx`, whichever is sooner);
|
||||
`cmd.Env = r.Env` exactly (nil means an empty environment — the parent's never leaks in).
|
||||
- Write the PDF to stdin from a goroutine, then close stdin.
|
||||
- Read stdout through `io.LimitReader(…, MaxText+1)`; more than `MaxText` → kill the child and
|
||||
return `ErrTooMuchText`. Keep at most 4 KiB of stderr, for the error message only.
|
||||
- Always `Wait` for the child, so it is reaped before `Run` returns.
|
||||
- Deadline passed → `ErrTimeout` (even if the kill produced some other error).
|
||||
- Exit 3 → `ErrEncrypted`, exit 4 → `ErrUnreadable`, any other non-zero exit or unparseable
|
||||
output → `ErrUnreadable` wrapped with the first line of stderr.
|
||||
- Parse the lines into a `Doc`. If every extracted page's text is blank after
|
||||
`strings.TrimSpace` (or no page was extracted), return `ErrNoText`.
|
||||
6. **`cmd/sift/main.go`:** `case "pdftext": os.Exit(pdftext.ChildMain(os.Args[1:], os.Stdin, os.Stdout))`.
|
||||
It is not listed in the help text: users never run it.
|
||||
|
||||
## Facts
|
||||
|
||||
- The tests run the **test binary itself** as the child: `TestMain` becomes the child when
|
||||
`SIFT_PDFTEXT_CHILD=1` is in its environment. The `SIFT_PDFTEXT_SLEEP` and
|
||||
`SIFT_PDFTEXT_PIDFILE` variables exist only in that test file; production code never reads
|
||||
them.
|
||||
- `extract.Text(doc, 1)` returns `""` with a nil error for an image-only page.
|
||||
- Reader API you need: `reader.Open`, `doc.PageCount()`, `doc.PageRef(i) (Ref, bool)`,
|
||||
`doc.Trailer() Dict`, `doc.Resolve(o) (Object, error)`, `Dict.Get(Name) Object`, and the types
|
||||
`reader.Dict`, `reader.Array`, `reader.Ref`, `reader.String`, `reader.Name`. Read their
|
||||
definitions under `vendor/github.com/go-pdfkit/reader/` (`object.go`, `document.go`) — that is
|
||||
inside this repository.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the given files; `git rm tools.go`.
|
||||
- [ ] **2. See them fail:** `go test -count=1 ./internal/pdftext/` (undefined names).
|
||||
- [ ] **3.** `pdftext.go`, **4.** `outline.go`, **5.** `runner.go`, **6.** `cmd/sift/main.go`.
|
||||
- [ ] **7.** `gofmt -w`; `go test -race -count=3 ./internal/pdftext/` → `ok`.
|
||||
- [ ] **8.** `make build && printf 'nope' | bin/sift pdftext; echo "exit $?"` → `exit 4`.
|
||||
- [ ] **9.** `make gate`. **10.** Row `p5a/02-pdftext`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/pdftext internal/pdftest tests/testdata/pdf cmd/sift docs/implementer-log.md # tools.go is already staged by git rm
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given tests pass under `-race -count=3`, including the deadline test (the child is gone
|
||||
after `Run` returns); `make gate` ok; given files byte-identical; `go.mod` untouched.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- Removing `tools.go` changes `go.mod`, `go.sum` or `vendor/modules.txt` (it must not: the
|
||||
package now imports both modules), or a vendored package is missing.
|
||||
@@ -0,0 +1,82 @@
|
||||
# P5a task 03: a document cache for PDFs
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `dcache: PDF document cache (per URL and caller class, whole-document eviction)`
|
||||
|
||||
## Goal
|
||||
|
||||
Extracting a thesis takes a second; downloading it can take a minute. Once a document is
|
||||
extracted, later requests for other pages, a grep, or a question must not fetch or extract it
|
||||
again. This task adds two tables to the existing SQLite cache and four functions. Freshness (the
|
||||
24 h TTL and revalidation) is decided by the caller in task 06, not here.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/dcache/pdf_test.go`
|
||||
- Create: `internal/dcache/pdf.go`
|
||||
- Modify: `internal/dcache/dcache.go` (the two `CREATE TABLE`s, `Size`, `evict`), `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package dcache
|
||||
|
||||
// PDFEntry is one cached PDF: how to revalidate it, its metadata (the pdftext.Meta JSON, opaque
|
||||
// here) and the text of each extracted page (Pages[i] is page i+1).
|
||||
type PDFEntry struct {
|
||||
ContentHash string
|
||||
ETag string
|
||||
LastModified string
|
||||
FetchedAt time.Time
|
||||
Meta []byte
|
||||
Pages []string
|
||||
}
|
||||
|
||||
// PDFKey is the cache key for a document: a hash of the URL and the caller class, like the
|
||||
// other keys here (the URL never appears in the clear).
|
||||
func PDFKey(url, callerClass string) string
|
||||
|
||||
func (d *DB) PutPDF(key string, e PDFEntry) // replaces any entry under key, pages included
|
||||
func (d *DB) GetPDF(key string) (PDFEntry, bool)
|
||||
func (d *DB) TouchPDF(key string, fetchedAt time.Time) // after a 304: only FetchedAt moves; a miss is a no-op
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. Tables:
|
||||
- `pdf_doc(k TEXT PRIMARY KEY, content_hash, etag, last_modified, fetched_at INTEGER, meta BLOB, last_hit INTEGER, nbytes INTEGER)`
|
||||
- `pdf_page(k TEXT, page INTEGER, text TEXT, PRIMARY KEY (k, page))`
|
||||
`nbytes` is the document's size (meta plus every page's text) and counts toward `Size()`.
|
||||
2. `PutPDF` writes the document and its pages in **one transaction**, deleting the old pages
|
||||
first, then calls `evict`. `GetPDF` updates `last_hit`, and returns the pages in page order;
|
||||
an empty page keeps its place.
|
||||
3. `evict` treats a PDF document like any other entry, by `last_hit`, and removes it **whole**:
|
||||
its `pdf_doc` row and all its `pdf_page` rows together. A cache never holds half a document.
|
||||
4. Hit and miss counters: add `PDFH`, `PDFM` beside the others (the metrics task reads them later).
|
||||
5. A nil `*DB` is a no-op everywhere, as for the existing methods; a SQL error is logged at
|
||||
`Warn` (never with the key's text or page text) and behaves like a miss.
|
||||
|
||||
## Facts
|
||||
|
||||
- `DigestKey`, `EmbedKey` and `SearchKey` show how keys are hashed (`sha256` of the parts joined
|
||||
with a separator). Follow the same pattern.
|
||||
- The existing `evict` picks the oldest `last_hit` across the three existing tables; extend that
|
||||
union with `pdf_doc`.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined names).
|
||||
- [ ] **3.** `dcache.go` schema; `pdf.go`; `Size` and `evict`.
|
||||
- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/dcache/` → `ok` (the existing dcache
|
||||
tests too).
|
||||
- [ ] **5.** `make gate`. **6.** Row `p5a/03-dcache`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/dcache docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given and existing dcache tests pass under `-race -count=3`; `make gate` ok; given file
|
||||
byte-identical.
|
||||
@@ -0,0 +1,81 @@
|
||||
# P5a task 04: the extraction pool
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `pdfpool: a few extraction slots, a short queue, single-flight, one job per caller`
|
||||
|
||||
## Goal
|
||||
|
||||
A PDF extraction uses a child process and up to a few hundred MB, and sift shares 1 GB with it.
|
||||
Kyle, 2026-09-25: "I don't think we're seeing a ton of PDF requests simultaneously, so having one
|
||||
or two workers seems tenable." This task is the small in-process scheduler that enforces that:
|
||||
two slots, four waiting places, identical requests sharing one job, and no single caller
|
||||
holding more than one job at a time. Generic: it knows nothing about PDFs.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/pdfpool/pdfpool_test.go`
|
||||
- Create: `internal/pdfpool/pdfpool.go`
|
||||
- Modify: `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package pdfpool
|
||||
|
||||
var (
|
||||
ErrBusy = errors.New("pdf workers busy") // queue full, or waited longer than Wait
|
||||
ErrCallerBusy = errors.New("busy") // this caller already has a job in flight
|
||||
)
|
||||
|
||||
// New returns a pool with `slots` jobs running at once, at most `queue` waiting, and a waiter
|
||||
// refused after `wait`.
|
||||
func New(slots, queue int, wait time.Duration) *Pool
|
||||
|
||||
// Do runs fn under the pool and returns its result.
|
||||
func (p *Pool) Do(ctx context.Context, key, caller string, fn func(context.Context) (any, error)) (any, error)
|
||||
|
||||
func (p *Pool) Running() int // jobs holding a slot
|
||||
func (p *Pool) Queued() int // jobs waiting for a slot
|
||||
func (p *Pool) Waiting(key string) int // callers sharing key's job, not counting the one that started it
|
||||
```
|
||||
|
||||
## Rules, in the order Do applies them
|
||||
|
||||
1. **Join.** If a job for `key` is already running or queued, wait for its result (count it in
|
||||
`Waiting(key)`) and return it. Joining takes no slot, no queue place, and is allowed even if
|
||||
the caller has another job in flight. `fn` may be nil when joining.
|
||||
2. **Caller.** If `caller` already has a job in flight (running or queued) for another key,
|
||||
return `ErrCallerBusy` at once.
|
||||
3. **Slot.** If fewer than `slots` jobs are running, start at once. Otherwise, if `queue` jobs
|
||||
are already waiting, return `ErrBusy` at once. Otherwise wait, first come first served, until
|
||||
a slot frees (start), `wait` passes (`ErrBusy`), or `ctx` ends (`ctx.Err()`). A waiter that
|
||||
leaves gives up its place.
|
||||
4. **The job's context is not the caller's.** `fn` runs with a context that is **not** cancelled
|
||||
when the caller that started it goes away (`context.WithoutCancel(ctx)`): other callers may
|
||||
be sharing it. The work has its own deadline (the child's). A caller whose `ctx` ends while
|
||||
waiting for a result gets `ctx.Err()`; the job goes on for the others.
|
||||
5. **Results are not cached.** When the job ends, the key is forgotten: the next `Do` for it runs
|
||||
`fn` again. An error from `fn` reaches every sharer. A **panic** in `fn` is recovered,
|
||||
returned as an error to every sharer, and frees the slot and the caller.
|
||||
6. Everything is guarded by one mutex; no goroutine leaks when a waiter gives up.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined names).
|
||||
- [ ] **3.** `pdfpool.go`.
|
||||
- [ ] **4.** `gofmt -w`; `go test -race -count=5 ./internal/pdfpool/` → `ok`.
|
||||
- [ ] **5.** `make gate`. **6.** Row `p5a/04-pdfpool`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/pdfpool docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given test passes under `-race -count=5`; `make gate` ok; given file byte-identical.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A given test's timing looks impossible to meet without sleeping in production code (that is
|
||||
the owner's bug, per `AGENTS.md`).
|
||||
@@ -0,0 +1,115 @@
|
||||
# P5a task 05: render an extracted document for an agent
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `pdfdoc: page markers, the map for long documents, page ranges, grep`
|
||||
|
||||
## Goal
|
||||
|
||||
An agent's context is small and a thesis is 60k–150k tokens. This task is the pure function that
|
||||
decides what an agent sees: the whole text with page markers when it fits; otherwise a **map**
|
||||
(title, outline with page numbers, the first pages, and how to ask for more); a **page range**;
|
||||
or **grep** results with page numbers. No I/O, no PDFs: it works on a `*pdftext.Doc`.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/pdfdoc/pdfdoc_test.go`
|
||||
- Create: `internal/pdfdoc/pdfdoc.go` (`Render`, `Full`, `Marker`, the map, page ranges),
|
||||
`internal/pdfdoc/grep.go`
|
||||
- Modify: `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package pdfdoc
|
||||
|
||||
type Options struct {
|
||||
Pages string // "" = no range; "a-b", "a" or "a-" (1-based, inclusive)
|
||||
Grep string // "" = no grep; an RE2 pattern
|
||||
OutCap int // characters (runes) the content may use
|
||||
}
|
||||
|
||||
type Rendered struct {
|
||||
Content string
|
||||
Map bool // Content is the map of a document longer than OutCap
|
||||
First, Last int // pages whose text is in Content; 0, 0 for a grep answer
|
||||
GrepHits int // matches found (more may exist than are shown)
|
||||
Note string // "" or how the answer was cut and how to continue
|
||||
}
|
||||
|
||||
var (
|
||||
ErrBadPages = errors.New("invalid pages")
|
||||
ErrBadGrep = errors.New("invalid grep")
|
||||
)
|
||||
|
||||
func Marker(page int) string // "--- page N ---"
|
||||
func Full(doc *pdftext.Doc) string // every extracted page with its marker
|
||||
func Render(doc *pdftext.Doc, opt Options) (Rendered, error)
|
||||
```
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Full.** For each extracted page p: `Marker(p) + "\n\n" + strings.TrimSpace(text)`, joined
|
||||
with `"\n\n"`. The given test shows the exact string.
|
||||
2. **No Pages, no Grep.**
|
||||
- If the rune count of `Full(doc)` is at most `OutCap`: `Content = Full(doc)`, `First = 1`,
|
||||
`Last` = pages extracted.
|
||||
- Otherwise `Map = true` and the content is, in this order:
|
||||
```
|
||||
# <Title, or "Untitled PDF">
|
||||
|
||||
<Pages> pages · ~<T> tokens[ · pages 1-<E> extracted]
|
||||
|
||||
## Outline (only when the outline is not empty)
|
||||
- <title> (p. <page>) (one line per item, indented two spaces per level below 1)
|
||||
|
||||
This document is longer than one answer: ask for pages="a-b" or grep="pattern".
|
||||
|
||||
<leading pages>
|
||||
```
|
||||
- `T` is the rune count of `Full(doc)` divided by 4.
|
||||
- The `pages 1-E extracted` part appears only when fewer pages were extracted than
|
||||
`Pages`. The separator is " · " (U+00B7).
|
||||
- Leading pages are whole pages rendered as in `Full`, from page 1, as many as fit in
|
||||
`OutCap/4` runes. At least page 1 is shown, cut to fit if it alone is larger.
|
||||
- The whole content is at most `OutCap` runes; if the outline is too long, cut the outline,
|
||||
not the header or the instruction line.
|
||||
- `First = 1`, `Last` = the last whole page shown.
|
||||
3. **Pages.**
|
||||
- Parse after trimming spaces: `a-b`, `a`, `a-` (a to the last extracted page). Anything else
|
||||
→ `ErrBadPages`.
|
||||
- `a < 1`, `b < a`, or `a` past the last extracted page → an error wrapping `ErrBadPages`
|
||||
whose text is `pages out of range (1-E)`. `b` past the end is clamped to `E`.
|
||||
- At most **100 pages** and at most `OutCap` runes: whole pages from `a`. A single page larger
|
||||
than `OutCap` is cut to fit.
|
||||
- When the range was cut, `Note` says so and names the rest as `pages="k-b"`: for example
|
||||
`cut at page 13; ask for pages="14-40"`.
|
||||
4. **Grep.**
|
||||
- The pattern must be at most 200 bytes and compile with `regexp` (RE2), else an error
|
||||
wrapping `ErrBadGrep`. Prefix `(?i)` unless the pattern contains an upper-case letter
|
||||
("smart case").
|
||||
- Search every line of every page in the range (all extracted pages when `Pages` is empty).
|
||||
- The first line of the content is `grep "<pattern>": N matches` (`1 match` for one; `no matches`
|
||||
for none). Add `, showing the first 100` when N is over 100.
|
||||
- Then, per match, up to the first 100: the previous line of the same page as
|
||||
`p. P- <line>`, the matching line as `p. P: <line>`, the next line as `p. P- <line>`,
|
||||
then a `--` line. Lines are trimmed of surrounding spaces.
|
||||
- `GrepHits = N`, `First = Last = 0`. The content is at most `OutCap` runes (drop whole
|
||||
match blocks at the end and say so in `Note`). No match is not an error.
|
||||
5. Every length here is in **runes**, never bytes: the cap protects an agent's context, and CJK
|
||||
text is three bytes a rune.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined names).
|
||||
- [ ] **3.** `pdfdoc.go`, **4.** `grep.go`.
|
||||
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/pdfdoc/` → `ok`.
|
||||
- [ ] **6.** `make gate`. **7.** Row `p5a/05-pdfdoc`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/pdfdoc docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given test passes; `make gate` ok; given file byte-identical.
|
||||
@@ -0,0 +1,150 @@
|
||||
# P5a task 06: PDFs through `/v1/fetch` and `/v2/scrape`
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `api: serve PDFs — child extraction via the pool, document cache, map, pages, grep`
|
||||
|
||||
## Goal
|
||||
|
||||
Wire tasks 01–05 into the API. A PDF URL fetched through `/v1/fetch` or `/v2/scrape` is:
|
||||
1. extracted once in the child, under the pool;
|
||||
2. cached per page;
|
||||
3. answered with the whole text, the map, a page range, or grep results.
|
||||
|
||||
Later requests for the same document are served from the cache, revalidated after 24 h with a
|
||||
conditional GET. `question` digests the document's text, page markers included, unless the
|
||||
document is too long.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/api/pdf_test.go`
|
||||
- Create: `internal/api/pdf.go` — everything PDF lives here
|
||||
- Modify: `internal/api/api.go` — **only** the hooks listed in rule 7. `api.go` is already 840
|
||||
lines; do not move code into it.
|
||||
- Modify: `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```go
|
||||
package api
|
||||
|
||||
const (
|
||||
DefaultPDFTTL = 24 * time.Hour
|
||||
DefaultPDFMaxQuestionChunks = 200
|
||||
)
|
||||
|
||||
// in Server:
|
||||
PDFRunner *pdftext.Runner // nil = &pdftext.Runner{} (the child is /proc/self/exe pdftext)
|
||||
PDFPool *pdfpool.Pool // nil = pdfpool.New(2, 4, 20*time.Second)
|
||||
PDFTTL time.Duration // 0 = DefaultPDFTTL
|
||||
PDFMaxQuestionChunks int // 0 = DefaultPDFMaxQuestionChunks
|
||||
|
||||
// in v1Req:
|
||||
Pages string `json:"pages"`
|
||||
Grep string `json:"grep"`
|
||||
```
|
||||
|
||||
Defaults are applied where the field is read: a zero `Server` from `New` must work. Build the
|
||||
default pool once, with a `sync.Once` — never one per request.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Cache first.**
|
||||
- Before fetching, look up `dcache.PDFKey(rawURL, s.callerClass(c))` in `s.cache()` (which
|
||||
may be nil: then there is no cache).
|
||||
- **Fresh** (now − `FetchedAt` < TTL): build the `*pdftext.Doc` from the entry (unmarshal
|
||||
`Meta`; `Text` = `Pages`). Answer with no network call and no child. The source is marked
|
||||
`"cached": true`.
|
||||
- **Stale:** `s.Fetch.FetchIf(ctx, rawURL, e.ETag, e.LastModified)`.
|
||||
- 304 → `TouchPDF(key, now)` and answer from the entry, with no child.
|
||||
- 200 → go on as for a new document.
|
||||
- **Not cached:** the usual `s.Fetch.Fetch`, under the existing `acquire`/`release`.
|
||||
2. **A PDF** is `clean.Kind(res.ContentType, res.Body) == "pdf"`. Every other kind goes through
|
||||
today's path unchanged. Use `clean.Kind` in place of the `MediaKind`/`Sniff` pair in both
|
||||
`doFetch` and `cleanFetched`. `"other"` stays `415`.
|
||||
3. **Extraction.**
|
||||
- Call `s.PDFPool.Do(ctx, class+"|"+res.SHA256(), c.Name, fn)`, where `fn` runs
|
||||
`s.PDFRunner.Run(jobCtx, res.Body)`.
|
||||
- The key is the **content hash**, so identical bytes are extracted once however many callers
|
||||
or URLs ask. The key includes the caller class, so a restricted caller never shares a job
|
||||
with the fleet.
|
||||
- Then `PutPDF` under the URL key with: the content hash; the response's `ETag` and
|
||||
`Last-Modified`; `FetchedAt` = now; `Meta` = the JSON of `doc.Meta`; `Pages` = `doc.Text`.
|
||||
4. **Errors**, all through the existing `writeErr` shape (`{"success":false,"error":…,"hint":…}`):
|
||||
|
||||
| cause | status | error | hint |
|
||||
|---|---|---|---|
|
||||
| `pdftext.ErrNoText` | 422 | `no text layer (scanned PDF?)` | `ocr` |
|
||||
| `pdftext.ErrEncrypted` | 422 | `encrypted PDF (password required)` | |
|
||||
| `pdftext.ErrUnreadable` | 422 | `unreadable PDF` | |
|
||||
| `pdftext.ErrTimeout` | 422 | `pdf extraction timed out` | |
|
||||
| `pdftext.ErrTooMuchText` | 422 | `pdf text over the output limit` | |
|
||||
| `pdfpool.ErrBusy` | 503 | `pdf workers busy`, **header `Retry-After: 10`** | |
|
||||
| `pdfpool.ErrCallerBusy` | 429 | `busy` | |
|
||||
| `pdfdoc.ErrBadPages` / `ErrBadGrep` | 400 | the error's text | |
|
||||
| `pages` or `grep` on a non-PDF | 400 | `pages and grep apply to PDFs only` | |
|
||||
| `question` on a document over `PDFMaxQuestionChunks` chunks | 422 | `document too long for question (N chunks); use grep or pages` | |
|
||||
|
||||
A PDF path never answers 500 for anything the document did.
|
||||
5. **`/v1/fetch`.**
|
||||
- **Mode `clean`:** `content` is `pdfdoc.Render(doc, {Pages, Grep, OutCap: s.OutCap}).Content`.
|
||||
- The envelope gains:
|
||||
```
|
||||
"pdf": {"pages", "title", "outline", "extracted", "first", "last", "map", "grep_hits", "note"}
|
||||
```
|
||||
`extracted` is `len(doc.Text)`; `outline` is `doc.Outline`; the rest come from `Rendered`.
|
||||
- **Mode `digest`:** the text is `pdfdoc.Full(doc)`. First count
|
||||
`len(digest.Split(full, digest.ChunkTokens))`; if it is over `PDFMaxQuestionChunks`, answer
|
||||
the 422 above **before** any model call. Otherwise run the digest as for any page, with
|
||||
`full` as the content.
|
||||
- **Mode `raw`** on a PDF: 400 `raw mode does not apply to PDFs`.
|
||||
- `sizes.raw_tokens` is the rune count of `Full(doc)` / 4 for a PDF (there is no body on a
|
||||
cache hit).
|
||||
6. **`/v2/scrape`.**
|
||||
- `markdown` is `Render(doc, {OutCap: s.OutCap}).Content`; `html` is `""`.
|
||||
- `metadata.title` is `doc.Title`; `metadata.numPages` is `doc.Pages`; `metadata.contentType`
|
||||
is the response's (`application/pdf` on a cache hit).
|
||||
7. **`api.go` hooks (the only edits allowed there):**
|
||||
- the four `Server` fields and two `v1Req` fields;
|
||||
- the `Kind` switch in `doFetch`/`cleanFetched`, now returning the doc;
|
||||
- one call into `pdf.go` in `v1fetch` and in `scrape`;
|
||||
- `Retry-After` in `writeFetchErr` for a 503.
|
||||
|
||||
`doFetch` and `cleanFetched` may gain a result, `doc *pdftext.Doc` (nil unless a PDF).
|
||||
8. **Logging:**
|
||||
- Keep today's `fetch`/`scrape` log lines and add `pdf_pages` (a count) to them.
|
||||
- Never log the title, the outline or any text.
|
||||
- A restricted caller's lines go through the existing helpers (host blanked); the given test
|
||||
reads every log record.
|
||||
|
||||
## Facts
|
||||
|
||||
- The tests set `s.Digest = &digest.Engine{Cache: db}` only to give `s.cache()` a database: no
|
||||
model is configured.
|
||||
- The child in the tests is the test binary (`TestMain` in the given file).
|
||||
- `setupAPI`, `postJSON` and `recHandler` come from `api_test.go` in the same package.
|
||||
- `requestDeadline` (210 s) and `writeTimeout` (240 s) were raised by the owner for this path:
|
||||
download ≤ 120 s + queue ≤ 20 s + extraction ≤ 60 s.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined fields).
|
||||
- [ ] **3.** `pdf.go` (cache, extraction, errors, render), then the `api.go` hooks.
|
||||
- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/api/` → `ok`: the new tests and
|
||||
every existing api test (search, scrape, restricted logging, write timeout).
|
||||
- [ ] **5.** `make gate`. **6.** Row `p5a/06-api`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/api docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- `go test -race -count=3 ./internal/api/` is ok; `make gate` ok; given file byte-identical;
|
||||
`api.go` grew by the hooks only (the owner will diff it).
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A rule here contradicts an existing api test.
|
||||
- The pool's default wait (20 s) makes a test take longer than the test allows. The given tests
|
||||
set their own pool where timing matters.
|
||||
@@ -0,0 +1,108 @@
|
||||
# P5a task 07: page citations, metrics, CLI flags, skill, unit, README
|
||||
|
||||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `P5a: digests cite pages; sift_pdf_* metrics; sift fetch --pages/--grep; skill, unit and README`
|
||||
|
||||
## Goal
|
||||
|
||||
Finish P5a:
|
||||
- digests of PDFs cite `(p. N)`;
|
||||
- the PDF path is visible in `/metrics`;
|
||||
- `sift fetch` can ask for pages and grep;
|
||||
- the agents' skill tells them PDFs work;
|
||||
- the unit gets the memory the child needs;
|
||||
- the README describes all of it.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `internal/digest/pages_test.go`, `internal/metrics/pdf_test.go`, `cmd/sift/fetchpdf_test.go`
|
||||
- Modify:
|
||||
- `internal/digest/digest.go` (prompt line, `PromptVersion`)
|
||||
- `internal/metrics/metrics.go`
|
||||
- `internal/api/pdf.go` (the metric calls)
|
||||
- `cmd/sift/fetchcmd.go`
|
||||
- `skill/SKILL.md`
|
||||
- `nix/module.nix` (**only** `MemoryMax`)
|
||||
- `README.md`
|
||||
- `docs/implementer-log.md`
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Digest.**
|
||||
- Add one sentence to `systemPrompt`: `When the document has "--- page N ---" markers, cite
|
||||
the page of each fact as (p. N).`
|
||||
- Set `PromptVersion = "3"`. The digest cache keys on it, so digests made with the old
|
||||
prompt are not served.
|
||||
- `Split` already keeps markers whole; the given test pins that.
|
||||
2. **Metrics** (`metrics.go`):
|
||||
- `func (s *Set) ObservePDF(outcome string, d time.Duration, pages int)`. It feeds:
|
||||
- the counter `sift_pdf_extract_total{outcome}`;
|
||||
- the histogram `sift_pdf_extract_seconds`;
|
||||
- the counter `sift_pdf_pages_total`.
|
||||
|
||||
`outcome` must be one of `ok`, `no_text`, `encrypted`, `unreadable`, `timeout`,
|
||||
`too_large`, `busy`; anything else is counted as `other`. The label never carries raw
|
||||
text: the given test sends `blocked: evil.example`.
|
||||
- `func (s *Set) ObservePDFWait(d time.Duration)` feeds the histogram
|
||||
`sift_pdf_pool_wait_seconds`.
|
||||
- `func (s *Set) PDFCacheHit()` feeds the counter `sift_pdf_cache_hits_total`.
|
||||
- Register them on the Set's own registry, like `sift_requests_total`.
|
||||
- In `internal/api/pdf.go`:
|
||||
- call `ObservePDF` once per extraction attempt, with the outcome and the pages extracted;
|
||||
- call `ObservePDFWait` with the time spent waiting in `PDFPool.Do` before the job
|
||||
started (measure around `Do`: total minus the job's own time);
|
||||
- call `PDFCacheHit` for every answer served from the cache, fresh or after a 304.
|
||||
3. **CLI** (`fetchcmd.go`).
|
||||
- Parsing becomes a struct, and today's tuple function stays as a wrapper so the existing
|
||||
tests keep compiling:
|
||||
```go
|
||||
type fetchOpts struct {
|
||||
URL, Mode, JQ, Question, Pages, Grep string
|
||||
NoContext, Recover, JSON bool
|
||||
}
|
||||
func parseFetchOpts(args []string) (fetchOpts, error) // the parser; knows --pages and --grep (both take a value)
|
||||
func parseFetchArgs(args []string) (…the same 8 results as today…) // calls parseFetchOpts
|
||||
func fetchBody(o fetchOpts) map[string]any // the /v1/fetch request body
|
||||
```
|
||||
- `fetchBody` builds today's body: url, mode (defaulting to digest when there is a question,
|
||||
else clean), jq, question, context (question set and not `--no-context`), recover. It adds
|
||||
`"pages"` and `"grep"` **only when set**.
|
||||
- The command uses `parseFetchOpts` and `fetchBody`.
|
||||
- Add `[--pages a-b] [--grep RE]` to `fetchUsage`.
|
||||
- The given `cmd/sift/fetchpdf_test.go` and the existing flag tests must pass.
|
||||
4. **Skill** (`skill/SKILL.md`):
|
||||
- Remove the `415 unsupported type` row's mention of PDF.
|
||||
- Add a short "PDFs" section:
|
||||
- PDFs work;
|
||||
- a long one answers with a map (outline with page numbers, first pages);
|
||||
- then use `--pages 41-60` or `--grep 'pattern'`, or `--question` for a digest that cites
|
||||
pages;
|
||||
- scanned PDFs return `no text layer`;
|
||||
- never download a PDF and run `pdftotext` yourself.
|
||||
5. **Unit** (`nix/module.nix`): `MemoryMax = "1G";` (was `"512M"`). Nothing else in the file
|
||||
changes.
|
||||
6. **README:**
|
||||
- a "PDF documents" section covering: detection, limits (100 MB, 120 s download, 60 s
|
||||
extraction, 2,000 pages), the child process, the pool (2 slots, 4 waiting, 20 s), the
|
||||
cache (24 h, conditional GET), the answers (full / map / pages / grep / digest with
|
||||
citations), the errors table from task 06, and the metrics;
|
||||
- note that the engine is `go-pdfkit/extract` v0.3.0, chosen by the bake-off in `docs/P5a.md` §3.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1.** `git switch p5a`; copy the three given tests. **2. See them fail.**
|
||||
- [ ] **3.** digest, **4.** metrics and the calls in `pdf.go`, **5.** CLI, **6.** skill, unit,
|
||||
README.
|
||||
- [ ] **7.** `gofmt -w`; `make gate`.
|
||||
- [ ] **8.** `make build && bin/sift fetch 2>&1 | grep -c -- '--pages'` prints `1`.
|
||||
- [ ] **9.** Row `p5a/07-digest-cli-docs`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/digest internal/metrics internal/api/pdf.go cmd/sift skill nix/module.nix README.md docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- All tests pass; `make gate` ok; given files byte-identical; the `nix/module.nix` diff is the
|
||||
one line.
|
||||
@@ -0,0 +1,46 @@
|
||||
# P5a implementation plan: PDF documents
|
||||
|
||||
> **For the implementing model:** do not work from this file. The owner gives you one task file
|
||||
> at a time. This file is the index for the owner and the reviewer.
|
||||
|
||||
**Goal:** `docs/P5a.md`. sift fetches, extracts and serves PDFs, so agents stop running
|
||||
`pdftotext` in the shell. Engine: `github.com/go-pdfkit/extract` v0.3.0 (bake-off, P5a §3),
|
||||
vendored by the owner before the run.
|
||||
|
||||
| Task | Package(s) | Given |
|
||||
|---|---|---|
|
||||
| 01-kind-fetch | `internal/clean`, `internal/fetch` | `clean/kind_test.go`, `fetch/pdf_test.go` |
|
||||
| 02-pdftext | `internal/pdftext`, `cmd/sift` (one dispatch line) | `pdftext/pdftext_test.go`, `internal/pdftest/pdftest.go`, `tests/testdata/pdf/*` |
|
||||
| 03-dcache | `internal/dcache` | `dcache/pdf_test.go` |
|
||||
| 04-pdfpool | `internal/pdfpool` | `pdfpool/pdfpool_test.go` |
|
||||
| 05-pdfdoc | `internal/pdfdoc` | `pdfdoc/pdfdoc_test.go` |
|
||||
| 06-api | `internal/api` | `api/pdf_test.go` |
|
||||
| 07-digest-cli-docs | `internal/digest`, `internal/metrics`, `cmd/sift`, `internal/api` (metric calls), `skill/`, `nix/module.nix`, `README.md` | `digest/pages_test.go`, `metrics/pdf_test.go` |
|
||||
|
||||
**Order matters:** 05 imports 02's types; 06 uses 01–05; 07 wires metrics into 06's file.
|
||||
|
||||
**How this plan was made:** acceptance tests first, no reference implementation. The given tests
|
||||
compiled against a panic-only skeleton of every new name, on this repository at the plan
|
||||
commit, and failed there for the intended reasons; every earlier test stayed green. The PDFs the
|
||||
tests use are built by `internal/pdftest` (a given file) at test time. The only binary fixtures
|
||||
are two small qpdf-encrypted copies of the three-page test document (`tests/testdata/pdf/README.md`).
|
||||
|
||||
**Owner changes made before the run** (plan commit):
|
||||
- `go.mod`, `go.sum`, `vendor/`: `go-pdfkit/extract` v0.3.0, `go-pdfkit/reader` v0.6.0 (and its
|
||||
`pdffont`), pinned through `tools.go` until task 02 imports them. `flake.nix` uses
|
||||
`vendorHash = null`, so the nix build reads `vendor/`.
|
||||
- `internal/api/api_test.go`: `TestUnsupportedType` now uses `application/zip`; a PDF is no
|
||||
longer unsupported.
|
||||
- `gofmt -w` on `internal/clean/clean.go`, `internal/guard/guard_test.go`,
|
||||
`internal/guard/iana.go` (whitespace only), so the new `make gate` target passes on master.
|
||||
- `AGENTS.md`, `tools/run-plan.sh`, `docs/implementer-log.md`, `make gate`.
|
||||
|
||||
**Attribution:** `tools/run-plan.sh` gives the model the exact trailer
|
||||
`Co-Authored-By: <model> <<model>@llama.cpp.invalid>` and refuses a commit without it.
|
||||
|
||||
**Where it runs:** a separate checkout (`~/src/sift-impl` on straylight), branch `p5a`. Nothing
|
||||
reaches orion until the owner's review, the merge, and straylight's pin bump.
|
||||
|
||||
## Changes during the run
|
||||
|
||||
(none yet)
|
||||
@@ -0,0 +1,44 @@
|
||||
package main
|
||||
|
||||
// P5a task 07: `sift fetch --pages` and `--grep`.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestParseFetchOptsPDF(t *testing.T) {
|
||||
o, err := parseFetchOpts([]string{"https://example.org/t.pdf", "--pages", "41-60", "--grep", "attention head"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if o.URL != "https://example.org/t.pdf" || o.Pages != "41-60" || o.Grep != "attention head" {
|
||||
t.Errorf("got %+v", o)
|
||||
}
|
||||
o, err = parseFetchOpts([]string{"--grep", "x", "--question", "why", "https://example.org/t.pdf"})
|
||||
if err != nil || o.URL != "https://example.org/t.pdf" || o.Grep != "x" || o.Question != "why" {
|
||||
t.Errorf("flags before the URL: %+v %v", o, err)
|
||||
}
|
||||
if !strings.Contains(fetchUsage, "--pages") || !strings.Contains(fetchUsage, "--grep") {
|
||||
t.Errorf("usage lacks the new flags: %s", fetchUsage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchBodyPDFFields(t *testing.T) {
|
||||
b, _ := json.Marshal(fetchBody(fetchOpts{URL: "https://example.org/a.pdf"}))
|
||||
if strings.Contains(string(b), `"pages"`) || strings.Contains(string(b), `"grep"`) {
|
||||
t.Errorf("unset pages/grep must not be sent: %s", b)
|
||||
}
|
||||
b, _ = json.Marshal(fetchBody(fetchOpts{URL: "https://example.org/a.pdf", Pages: "2-3", Grep: "fox"}))
|
||||
var m map[string]any
|
||||
_ = json.Unmarshal(b, &m)
|
||||
if m["pages"] != "2-3" || m["grep"] != "fox" || m["url"] != "https://example.org/a.pdf" || m["mode"] != "clean" {
|
||||
t.Errorf("body = %s", b)
|
||||
}
|
||||
b, _ = json.Marshal(fetchBody(fetchOpts{URL: "u", Question: "q"}))
|
||||
_ = json.Unmarshal(b, &m)
|
||||
if m["mode"] != "digest" || m["question"] != "q" || m["context"] != true {
|
||||
t.Errorf("question body = %s (mode must default to digest, context on)", b)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,402 @@
|
||||
package api
|
||||
|
||||
// P5a task 06: PDFs through /v1/fetch and /v2/scrape — extraction in the child, the pool, the
|
||||
// document cache, the map, pages, grep, and the errors.
|
||||
//
|
||||
// The child is the test binary itself (TestMain below), as in internal/pdftext's tests.
|
||||
// SIFT_PDFTEXT_RUNLOG makes every child start append a line to a file, so a test can count
|
||||
// extractions; SIFT_PDFTEXT_SLEEP slows a child down. Both hooks live only in this file.
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/auth"
|
||||
"git.wntrmute.dev/kyle/sift/internal/dcache"
|
||||
"git.wntrmute.dev/kyle/sift/internal/digest"
|
||||
"git.wntrmute.dev/kyle/sift/internal/fetch"
|
||||
"git.wntrmute.dev/kyle/sift/internal/guard"
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdfpool"
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdftest"
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdftext"
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("SIFT_PDFTEXT_CHILD") == "1" {
|
||||
if p := os.Getenv("SIFT_PDFTEXT_RUNLOG"); p != "" {
|
||||
f, err := os.OpenFile(p, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o600)
|
||||
if err == nil {
|
||||
_, _ = f.WriteString("run\n")
|
||||
_ = f.Close()
|
||||
}
|
||||
}
|
||||
if d, err := time.ParseDuration(os.Getenv("SIFT_PDFTEXT_SLEEP")); err == nil {
|
||||
time.Sleep(d)
|
||||
}
|
||||
os.Exit(pdftext.ChildMain(os.Args[1:], os.Stdin, os.Stdout))
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func threePDF() []byte {
|
||||
return pdftest.Build(pdftest.Doc{
|
||||
Title: "Three Pages",
|
||||
Pages: []string{"Chapter one begins here.", "The quick brown fox is on page two.", "Last page."},
|
||||
Outline: []pdftest.Outline{{Title: "1 Intro", Page: 1, Level: 1}, {Title: "2 Fox", Page: 2, Level: 1}},
|
||||
})
|
||||
}
|
||||
|
||||
func longPDF(n int) []byte {
|
||||
var pages []string
|
||||
for p := 1; p <= n; p++ {
|
||||
pages = append(pages, pdftest.Lorem(p, 40))
|
||||
}
|
||||
return pdftest.Build(pdftest.Doc{Title: "Long Thesis", Pages: pages, Outline: []pdftest.Outline{{Title: "1 Introduction", Page: 1, Level: 1}, {Title: "2 Method", Page: n / 2, Level: 1}}})
|
||||
}
|
||||
|
||||
// origin serves one PDF and counts GETs. It honours If-None-Match against its current ETag.
|
||||
type origin struct {
|
||||
mu sync.Mutex
|
||||
ct string
|
||||
body []byte
|
||||
etag string
|
||||
gets atomic.Int32
|
||||
srv *httptest.Server
|
||||
}
|
||||
|
||||
func newOrigin(t *testing.T, ct string, body []byte) *origin {
|
||||
o := &origin{ct: ct, body: body, etag: `"v1"`}
|
||||
o.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
o.gets.Add(1)
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
if r.Header.Get("If-None-Match") == o.etag {
|
||||
w.WriteHeader(http.StatusNotModified)
|
||||
return
|
||||
}
|
||||
w.Header().Set("ETag", o.etag)
|
||||
if o.ct != "" {
|
||||
w.Header().Set("Content-Type", o.ct)
|
||||
}
|
||||
_, _ = w.Write(o.body)
|
||||
}))
|
||||
t.Cleanup(o.srv.Close)
|
||||
return o
|
||||
}
|
||||
|
||||
func (o *origin) replace(body []byte, etag string) {
|
||||
o.mu.Lock()
|
||||
o.body, o.etag = body, etag
|
||||
o.mu.Unlock()
|
||||
}
|
||||
|
||||
type pdfRig struct {
|
||||
s *Server
|
||||
fleet string // bearer key of caller "test" (hermes-fleet)
|
||||
other string // bearer key of caller "other" (hermes-fleet)
|
||||
gs string // bearer key of restricted caller "straylight-gs"
|
||||
runlog string
|
||||
logs *recHandler
|
||||
}
|
||||
|
||||
func newPDFRig(t *testing.T, childEnv ...string) *pdfRig {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
keys := filepath.Join(dir, "keys.toml")
|
||||
key := func(name, ws string) string {
|
||||
out := filepath.Join(dir, name)
|
||||
if err := auth.Add(keys, name, ws, out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, err := os.ReadFile(out)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
r := &pdfRig{fleet: key("test", "hermes-fleet"), other: key("other", "hermes-fleet"), gs: key("straylight-gs", "guardian-star"), runlog: filepath.Join(dir, "runs"), logs: &recHandler{}}
|
||||
store, err := auth.Load(keys)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
g := &guard.Guard{AllowPrefixes: []netip.Prefix{netip.MustParsePrefix("127.0.0.0/8")}}
|
||||
s := New("test", g, &fetch.Fetcher{Guard: g, UA: "sift-test"}, store, slog.New(r.logs))
|
||||
s.PermitRestricted = []string{"straylight-gs"}
|
||||
db, err := dcache.Open(filepath.Join(dir, "cache.db"), 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
s.Digest = &digest.Engine{Cache: db} // the cache lives on the digest engine; no model is configured
|
||||
s.PDFRunner = &pdftext.Runner{Exe: os.Args[0], Env: append([]string{"SIFT_PDFTEXT_CHILD=1", "SIFT_PDFTEXT_RUNLOG=" + r.runlog}, childEnv...), Deadline: 20 * time.Second}
|
||||
r.s = s
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *pdfRig) runs() int {
|
||||
b, _ := os.ReadFile(r.runlog)
|
||||
return strings.Count(string(b), "run\n")
|
||||
}
|
||||
|
||||
func (r *pdfRig) fetch(t *testing.T, key string, body map[string]any) (int, map[string]any, http.Header) {
|
||||
t.Helper()
|
||||
rec := postJSON(t, r.s, "/v1/fetch", key, body)
|
||||
var out map[string]any
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &out)
|
||||
return rec.Code, out, rec.Header()
|
||||
}
|
||||
|
||||
func pdfInfo(t *testing.T, out map[string]any) map[string]any {
|
||||
t.Helper()
|
||||
p, ok := out["pdf"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("envelope has no pdf object: %v", out)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func TestPDFCleanFetch(t *testing.T) {
|
||||
for _, ct := range []string{"application/pdf", "application/octet-stream", ""} {
|
||||
r := newPDFRig(t)
|
||||
o := newOrigin(t, ct, threePDF())
|
||||
code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL})
|
||||
if code != 200 {
|
||||
t.Fatalf("%q: %d %v", ct, code, out)
|
||||
}
|
||||
content, _ := out["content"].(string)
|
||||
if !strings.Contains(content, "--- page 2 ---") || !strings.Contains(content, "quick brown fox") || out["mode"] != "clean" {
|
||||
t.Errorf("%q: content %q mode %v", ct, content, out["mode"])
|
||||
}
|
||||
p := pdfInfo(t, out)
|
||||
if p["pages"] != 3.0 || p["title"] != "Three Pages" || p["map"] != false || p["first"] != 1.0 || p["last"] != 3.0 {
|
||||
t.Errorf("%q: pdf = %v", ct, p)
|
||||
}
|
||||
if ol, _ := p["outline"].([]any); len(ol) != 2 {
|
||||
t.Errorf("%q: outline = %v", ct, p["outline"])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFMap(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
r.s.OutCap = 20000
|
||||
o := newOrigin(t, "application/pdf", longPDF(40)) // ~40 × 2,700 characters
|
||||
code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL})
|
||||
if code != 200 {
|
||||
t.Fatalf("%d %v", code, out)
|
||||
}
|
||||
content, _ := out["content"].(string)
|
||||
p := pdfInfo(t, out)
|
||||
if p["map"] != true || !strings.Contains(content, "## Outline") || !strings.Contains(content, "- 2 Method (p. 20)") || len([]rune(content)) > 20000 {
|
||||
t.Errorf("map: pdf %v, %d chars:\n%.400s", p, len([]rune(content)), content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFPagesAndGrep(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
o := newOrigin(t, "application/pdf", threePDF())
|
||||
code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL, "pages": "2-3"})
|
||||
content, _ := out["content"].(string)
|
||||
if code != 200 || strings.Contains(content, "--- page 1 ---") || !strings.Contains(content, "--- page 3 ---") {
|
||||
t.Errorf("pages 2-3: %d %q", code, content)
|
||||
}
|
||||
if p := pdfInfo(t, out); p["first"] != 2.0 || p["last"] != 3.0 {
|
||||
t.Errorf("pages 2-3: pdf %v", p)
|
||||
}
|
||||
for _, bad := range []map[string]any{{"pages": "0-1"}, {"pages": "9"}, {"grep": strings.Repeat("a", 201)}, {"grep": "(unclosed"}} {
|
||||
bad["url"] = o.srv.URL
|
||||
if code, out, _ := r.fetch(t, r.fleet, bad); code != 400 || out["success"] != false {
|
||||
t.Errorf("%v: %d %v, want 400", bad, code, out)
|
||||
}
|
||||
}
|
||||
code, out, _ = r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL, "grep": "quick brown"})
|
||||
content, _ = out["content"].(string)
|
||||
if code != 200 || !strings.Contains(content, "p. 2: The quick brown fox is on page two.") || pdfInfo(t, out)["grep_hits"] != 1.0 {
|
||||
t.Errorf("grep: %d %q %v", code, content, out["pdf"])
|
||||
}
|
||||
// pages and grep on a non-PDF are refused rather than ignored.
|
||||
html := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = io.WriteString(w, "<html><body><p>hi</p></body></html>")
|
||||
}))
|
||||
defer html.Close()
|
||||
if code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": html.URL, "pages": "1"}); code != 400 {
|
||||
t.Errorf("pages on html: %d %v, want 400", code, out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFCache(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
o := newOrigin(t, "application/pdf", threePDF())
|
||||
get := func(extra map[string]any) string {
|
||||
body := map[string]any{"url": o.srv.URL}
|
||||
for k, v := range extra {
|
||||
body[k] = v
|
||||
}
|
||||
code, out, _ := r.fetch(t, r.fleet, body)
|
||||
if code != 200 {
|
||||
t.Fatalf("%v: %d %v", extra, code, out)
|
||||
}
|
||||
c, _ := out["content"].(string)
|
||||
return c
|
||||
}
|
||||
get(nil)
|
||||
if o.gets.Load() != 1 || r.runs() != 1 {
|
||||
t.Fatalf("first fetch: %d GETs, %d extractions; want 1 and 1", o.gets.Load(), r.runs())
|
||||
}
|
||||
if c := get(map[string]any{"pages": "2"}); !strings.Contains(c, "quick brown fox") {
|
||||
t.Errorf("cached pages=2: %q", c)
|
||||
}
|
||||
if o.gets.Load() != 1 || r.runs() != 1 {
|
||||
t.Errorf("fresh cache hit: %d GETs, %d extractions; want still 1 and 1", o.gets.Load(), r.runs())
|
||||
}
|
||||
// Stale: a conditional GET, answered 304, re-uses the text.
|
||||
r.s.PDFTTL = time.Nanosecond
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
get(map[string]any{"grep": "Chapter"})
|
||||
if o.gets.Load() != 2 || r.runs() != 1 {
|
||||
t.Errorf("after 304: %d GETs, %d extractions; want 2 and 1", o.gets.Load(), r.runs())
|
||||
}
|
||||
// The document changed: a full GET and a new extraction.
|
||||
o.replace(pdftest.Build(pdftest.Doc{Pages: []string{"Brand new text."}}), `"v2"`)
|
||||
if c := get(nil); !strings.Contains(c, "Brand new text.") {
|
||||
t.Errorf("changed document: %q", c)
|
||||
}
|
||||
if o.gets.Load() != 3 || r.runs() != 2 {
|
||||
t.Errorf("after change: %d GETs, %d extractions; want 3 and 2", o.gets.Load(), r.runs())
|
||||
}
|
||||
if DefaultPDFTTL != 24*time.Hour {
|
||||
t.Errorf("DefaultPDFTTL = %v, want 24h", DefaultPDFTTL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFErrors(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
enc, err := os.ReadFile(filepath.Join("..", "..", "tests", "testdata", "pdf", "encrypted-user.pdf"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
body []byte
|
||||
want string
|
||||
hint string
|
||||
}{
|
||||
{"scanned", pdftest.Build(pdftest.Doc{Pages: []string{"", ""}, Scanned: []int{1, 2}}), "no text layer", "ocr"},
|
||||
{"encrypted", enc, "encrypted", ""},
|
||||
{"garbage", []byte("%PDF-1.4\nthis is not really a pdf"), "unreadable", ""},
|
||||
} {
|
||||
o := newOrigin(t, "application/pdf", tc.body)
|
||||
code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL})
|
||||
msg, _ := out["error"].(string)
|
||||
if code != 422 || !strings.Contains(msg, tc.want) || (tc.hint != "" && out["hint"] != tc.hint) {
|
||||
t.Errorf("%s: %d %v; want 422 %q hint %q", tc.name, code, out, tc.want, tc.hint)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFTimeout(t *testing.T) {
|
||||
r := newPDFRig(t, "SIFT_PDFTEXT_SLEEP=10s")
|
||||
r.s.PDFRunner.Deadline = 300 * time.Millisecond
|
||||
o := newOrigin(t, "application/pdf", threePDF())
|
||||
code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL})
|
||||
if msg, _ := out["error"].(string); code != 422 || !strings.Contains(msg, "timed out") {
|
||||
t.Errorf("slow child: %d %v; want 422 … timed out", code, out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFBusy(t *testing.T) {
|
||||
r := newPDFRig(t, "SIFT_PDFTEXT_SLEEP=1500ms")
|
||||
r.s.PDFPool = pdfpool.New(1, 0, 100*time.Millisecond)
|
||||
a := newOrigin(t, "application/pdf", threePDF())
|
||||
b := newOrigin(t, "application/pdf", longPDF(2))
|
||||
done := make(chan int, 1)
|
||||
go func() {
|
||||
code, _, _ := r.fetch(t, r.fleet, map[string]any{"url": a.srv.URL})
|
||||
done <- code
|
||||
}()
|
||||
time.Sleep(400 * time.Millisecond) // the first extraction holds the only slot
|
||||
code, out, hdr := r.fetch(t, r.other, map[string]any{"url": b.srv.URL})
|
||||
if code != 503 || hdr.Get("Retry-After") != "10" || !strings.Contains(fmt.Sprint(out["error"]), "busy") {
|
||||
t.Errorf("second caller with the pool full: %d %v Retry-After %q; want 503, busy, 10", code, out, hdr.Get("Retry-After"))
|
||||
}
|
||||
code, out, _ = r.fetch(t, r.fleet, map[string]any{"url": b.srv.URL})
|
||||
if code != 429 {
|
||||
t.Errorf("same caller, second document while its first is running: %d %v; want 429", code, out)
|
||||
}
|
||||
if c := <-done; c != 200 {
|
||||
t.Errorf("first fetch: %d", c)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFRestrictedCaller(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
o := newOrigin(t, "application/pdf", threePDF())
|
||||
if code, out, _ := r.fetch(t, r.gs, map[string]any{"url": o.srv.URL}); code != 200 {
|
||||
t.Fatalf("restricted fetch: %d %v", code, out)
|
||||
}
|
||||
r.logs.mu.Lock()
|
||||
for _, rec := range r.logs.recs {
|
||||
var b strings.Builder
|
||||
b.WriteString(rec.Message)
|
||||
rec.Attrs(func(a slog.Attr) bool { fmt.Fprintf(&b, " %s=%v", a.Key, a.Value); return true })
|
||||
line := b.String()
|
||||
if strings.Contains(line, "127.0.0.1") || strings.Contains(line, "Three Pages") || strings.Contains(line, "quick brown") {
|
||||
t.Errorf("a restricted caller's PDF fetch logged %q", line)
|
||||
}
|
||||
}
|
||||
r.logs.mu.Unlock()
|
||||
// The fleet does not see the restricted caller's cached text: it extracts its own copy.
|
||||
if code, _, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL}); code != 200 || r.runs() != 2 {
|
||||
t.Errorf("fleet after restricted: %d, %d extractions; want 200 and 2 (separate cache partitions)", code, r.runs())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFScrape(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
o := newOrigin(t, "application/pdf", threePDF())
|
||||
rec := postJSON(t, r.s, "/v2/scrape", r.fleet, map[string]any{"url": o.srv.URL, "formats": []string{"markdown"}})
|
||||
var out struct {
|
||||
Success bool `json:"success"`
|
||||
Data struct {
|
||||
Markdown string `json:"markdown"`
|
||||
Metadata map[string]any `json:"metadata"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil || rec.Code != 200 || !out.Success {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(out.Data.Markdown, "--- page 2 ---") || out.Data.Metadata["numPages"] != 3.0 || out.Data.Metadata["title"] != "Three Pages" {
|
||||
t.Errorf("scrape: markdown %q metadata %v", out.Data.Markdown, out.Data.Metadata)
|
||||
}
|
||||
if ct, _ := out.Data.Metadata["contentType"].(string); !strings.HasPrefix(ct, "application/pdf") {
|
||||
t.Errorf("contentType = %q", ct)
|
||||
}
|
||||
}
|
||||
|
||||
// A question on a document longer than PDFMaxQuestionChunks is refused with a pointer to grep
|
||||
// and pages, before any model call.
|
||||
func TestPDFQuestionTooLong(t *testing.T) {
|
||||
r := newPDFRig(t)
|
||||
r.s.PDFMaxQuestionChunks = 2
|
||||
o := newOrigin(t, "application/pdf", longPDF(12))
|
||||
code, out, _ := r.fetch(t, r.fleet, map[string]any{"url": o.srv.URL, "question": "what is the method?"})
|
||||
if msg, _ := out["error"].(string); code != 422 || !strings.Contains(msg, "too long") || !strings.Contains(msg, "grep") {
|
||||
t.Errorf("long question: %d %v; want 422 … too long … grep", code, out)
|
||||
}
|
||||
if DefaultPDFMaxQuestionChunks != 200 {
|
||||
t.Errorf("DefaultPDFMaxQuestionChunks = %d, want 200", DefaultPDFMaxQuestionChunks)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package clean_test
|
||||
|
||||
// P5a task 01: Kind decides which cleaner a fetched body gets, now including "pdf".
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/clean"
|
||||
)
|
||||
|
||||
func TestKind(t *testing.T) {
|
||||
pdf := []byte("%PDF-1.7\n%\xe2\xe3\xcf\xd3\n1 0 obj << >> endobj\n")
|
||||
junkThenPDF := append([]byte(strings.Repeat(" ", 900)), pdf...) // the spec allows leading junk
|
||||
lateMagic := append([]byte(strings.Repeat("x", 2000)), pdf...) // past the first 1024 bytes
|
||||
binary := []byte{0x50, 0x4b, 0x03, 0x04, 0x00, 0x00} // a zip
|
||||
for _, tc := range []struct {
|
||||
name, ct string
|
||||
body []byte
|
||||
want string
|
||||
}{
|
||||
{"pdf by type", "application/pdf", pdf, "pdf"},
|
||||
{"pdf by type with params", "Application/PDF; qs=0.9", pdf, "pdf"},
|
||||
{"x-pdf", "application/x-pdf", pdf, "pdf"},
|
||||
{"pdf type, body not checked", "application/pdf", []byte("oops"), "pdf"},
|
||||
{"octet-stream sniffed as pdf", "application/octet-stream", pdf, "pdf"},
|
||||
{"binary/octet-stream sniffed as pdf", "binary/octet-stream", pdf, "pdf"},
|
||||
{"no type sniffed as pdf", "", pdf, "pdf"},
|
||||
{"leading junk within 1024 bytes", "application/octet-stream", junkThenPDF, "pdf"},
|
||||
{"magic too late", "application/octet-stream", lateMagic, "other"},
|
||||
{"octet-stream that is not a pdf", "application/octet-stream", binary, "other"},
|
||||
{"html stays html even if it mentions %PDF-", "text/html; charset=utf-8", []byte("<html>%PDF-1.4</html>"), "html"},
|
||||
{"json", "application/json", []byte(`{"a":1}`), "json"},
|
||||
{"problem+json", "application/problem+json", []byte(`{}`), "json"},
|
||||
{"text", "text/plain", []byte("hello"), "text"},
|
||||
{"no type, html", "", []byte("<!doctype html><html></html>"), "html"},
|
||||
{"no type, json", "", []byte(`[1,2]`), "json"},
|
||||
{"no type, text", "", []byte("just words"), "text"},
|
||||
{"zip", "application/zip", binary, "other"},
|
||||
{"image", "image/png", []byte{0x89, 'P', 'N', 'G'}, "other"},
|
||||
} {
|
||||
if got := clean.Kind(tc.ct, tc.body); got != tc.want {
|
||||
t.Errorf("%s: Kind(%q, …) = %q, want %q", tc.name, tc.ct, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MediaKind keeps answering for callers that have no body: the pdf types are "pdf" there too.
|
||||
func TestMediaKindKnowsPDF(t *testing.T) {
|
||||
for ct, want := range map[string]string{"application/pdf": "pdf", "application/x-pdf": "pdf", "text/html": "html", "": "sniff"} {
|
||||
if got := clean.MediaKind(ct); got != want {
|
||||
t.Errorf("MediaKind(%q) = %q, want %q", ct, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package dcache_test
|
||||
|
||||
// P5a task 03: the PDF document cache. One entry per (URL, caller class): how to revalidate it,
|
||||
// its metadata (opaque JSON here) and the text of every extracted page.
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/dcache"
|
||||
)
|
||||
|
||||
func openDB(t *testing.T, maxBytes int64) *dcache.DB {
|
||||
t.Helper()
|
||||
db, err := dcache.Open(filepath.Join(t.TempDir(), "c.db"), maxBytes, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
return db
|
||||
}
|
||||
|
||||
func entry(pages ...string) dcache.PDFEntry {
|
||||
return dcache.PDFEntry{
|
||||
ContentHash: "sha256:abc",
|
||||
ETag: `"v1"`,
|
||||
LastModified: "Fri, 25 Sep 2026 12:00:00 GMT",
|
||||
FetchedAt: time.Date(2026, 9, 25, 20, 0, 0, 0, time.UTC),
|
||||
Meta: []byte(`{"pages":3,"title":"T","outline":[]}`),
|
||||
Pages: pages,
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFRoundTrip(t *testing.T) {
|
||||
db := openDB(t, 0)
|
||||
k := dcache.PDFKey("https://example.org/a.pdf", dcache.ClassFleet)
|
||||
if _, ok := db.GetPDF(k); ok {
|
||||
t.Fatal("hit on an empty cache")
|
||||
}
|
||||
want := entry("page one", "page two", "", "page four") // an empty page keeps its place
|
||||
db.PutPDF(k, want)
|
||||
got, ok := db.GetPDF(k)
|
||||
if !ok {
|
||||
t.Fatal("miss after put")
|
||||
}
|
||||
if got.ContentHash != want.ContentHash || got.ETag != want.ETag || got.LastModified != want.LastModified ||
|
||||
!got.FetchedAt.Equal(want.FetchedAt) || string(got.Meta) != string(want.Meta) {
|
||||
t.Errorf("got %+v, want %+v", got, want)
|
||||
}
|
||||
if len(got.Pages) != 4 || got.Pages[0] != "page one" || got.Pages[2] != "" || got.Pages[3] != "page four" {
|
||||
t.Errorf("pages = %q", got.Pages)
|
||||
}
|
||||
// A second put replaces the whole entry, pages included.
|
||||
db.PutPDF(k, entry("only"))
|
||||
got, _ = db.GetPDF(k)
|
||||
if len(got.Pages) != 1 || got.Pages[0] != "only" {
|
||||
t.Errorf("after replace, pages = %q; want [only]", got.Pages)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFKeyPartitions(t *testing.T) {
|
||||
db := openDB(t, 0)
|
||||
u := "https://example.org/a.pdf"
|
||||
fleet, restricted := dcache.PDFKey(u, dcache.ClassFleet), dcache.PDFKey(u, dcache.ClassRestricted)
|
||||
if fleet == restricted {
|
||||
t.Fatal("the same key for both caller classes")
|
||||
}
|
||||
db.PutPDF(restricted, entry("secret page"))
|
||||
if _, ok := db.GetPDF(fleet); ok {
|
||||
t.Error("a restricted caller's document is visible to the fleet")
|
||||
}
|
||||
if dcache.PDFKey(u, dcache.ClassFleet) != fleet || dcache.PDFKey("https://example.org/b.pdf", dcache.ClassFleet) == fleet {
|
||||
t.Error("PDFKey must be deterministic and differ per URL")
|
||||
}
|
||||
if strings.Contains(fleet, "example.org") {
|
||||
t.Error("the key must not carry the URL in the clear (it is hashed like the other keys)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFTouch(t *testing.T) {
|
||||
db := openDB(t, 0)
|
||||
k := dcache.PDFKey("https://example.org/a.pdf", dcache.ClassFleet)
|
||||
db.PutPDF(k, entry("x"))
|
||||
later := time.Date(2026, 9, 26, 20, 0, 0, 0, time.UTC)
|
||||
db.TouchPDF(k, later) // after a 304: the text is unchanged, only its freshness moves
|
||||
got, _ := db.GetPDF(k)
|
||||
if !got.FetchedAt.Equal(later) || len(got.Pages) != 1 {
|
||||
t.Errorf("after touch: %+v", got)
|
||||
}
|
||||
db.TouchPDF(dcache.PDFKey("https://nope", dcache.ClassFleet), later) // a miss is a no-op
|
||||
}
|
||||
|
||||
// Documents count toward maxBytes and are evicted whole, oldest use first.
|
||||
func TestPDFEviction(t *testing.T) {
|
||||
db := openDB(t, 300<<10) // 300 KiB
|
||||
big := strings.Repeat("z", 50<<10)
|
||||
var keys []string
|
||||
for i := 0; i < 10; i++ { // 10 × 2 pages × 50 KiB = 1 MiB of text
|
||||
k := dcache.PDFKey("https://example.org/"+string(rune('a'+i))+".pdf", dcache.ClassFleet)
|
||||
keys = append(keys, k)
|
||||
db.PutPDF(k, entry(big, big))
|
||||
}
|
||||
if db.Size() > 300<<10 {
|
||||
t.Errorf("size %d after puts, want ≤ maxBytes %d", db.Size(), 300<<10)
|
||||
}
|
||||
if _, ok := db.GetPDF(keys[9]); !ok {
|
||||
t.Error("the newest document was evicted")
|
||||
}
|
||||
if _, ok := db.GetPDF(keys[0]); ok {
|
||||
t.Error("the oldest document survived; eviction did not reach PDF entries")
|
||||
}
|
||||
for _, k := range keys {
|
||||
if e, ok := db.GetPDF(k); ok && len(e.Pages) != 2 {
|
||||
t.Errorf("a partly evicted document: %d pages", len(e.Pages))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPDFNilDB(t *testing.T) {
|
||||
var db *dcache.DB
|
||||
db.PutPDF("k", entry("x"))
|
||||
db.TouchPDF("k", time.Now())
|
||||
if _, ok := db.GetPDF("k"); ok {
|
||||
t.Error("nil DB returned a hit")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package digest
|
||||
|
||||
// P5a task 07: a digest of a PDF cites pages. The page markers come from internal/pdfdoc; the
|
||||
// prompt tells the model to use them, and the prompt version moves so cached digests made with
|
||||
// the old prompt are not served.
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPromptCitesPages(t *testing.T) {
|
||||
if !strings.Contains(systemPrompt, "--- page N ---") || !strings.Contains(systemPrompt, "(p. N)") {
|
||||
t.Errorf("system prompt must tell the model to cite pages from --- page N --- markers:\n%s", systemPrompt)
|
||||
}
|
||||
if PromptVersion == "2" {
|
||||
t.Error("PromptVersion must change with the prompt (was \"2\")")
|
||||
}
|
||||
}
|
||||
|
||||
// Page markers survive chunking whole: none is cut across two chunks. (This holds for today's
|
||||
// Split, whose paragraph boundaries fall around each marker; the test keeps it that way.)
|
||||
func TestSplitKeepsPageMarkers(t *testing.T) {
|
||||
var b strings.Builder
|
||||
for p := 1; p <= 40; p++ {
|
||||
b.WriteString("--- page " + strconv.Itoa(p) + " ---\n\n")
|
||||
b.WriteString(strings.Repeat("word ", 600))
|
||||
b.WriteString("\n\n")
|
||||
}
|
||||
marker := regexp.MustCompile(`--- page \d+ ---`)
|
||||
n := 0
|
||||
for _, c := range Split(b.String(), ChunkTokens) {
|
||||
n += len(marker.FindAllString(c, -1))
|
||||
}
|
||||
if n != 40 {
|
||||
t.Errorf("%d whole page markers across the chunks, want 40", n)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
package fetch_test
|
||||
|
||||
// P5a task 01: a PDF response gets a larger body cap and a longer deadline than a web page, and
|
||||
// FetchIf revalidates a cached document with a conditional GET.
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/fetch"
|
||||
"git.wntrmute.dev/kyle/sift/internal/guard"
|
||||
)
|
||||
|
||||
func loopbackFetcher() *fetch.Fetcher {
|
||||
g := &guard.Guard{AllowPrefixes: []netip.Prefix{netip.MustParsePrefix("127.0.0.0/8")}}
|
||||
return &fetch.Fetcher{Guard: g, UA: "sift-test"}
|
||||
}
|
||||
|
||||
func body(prefix string, n int) []byte {
|
||||
b := bytes.Repeat([]byte("x"), n)
|
||||
copy(b, prefix)
|
||||
return b
|
||||
}
|
||||
|
||||
func serve(t *testing.T, ct string, b []byte) string {
|
||||
t.Helper()
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if ct != "" {
|
||||
w.Header().Set("Content-Type", ct)
|
||||
}
|
||||
_, _ = w.Write(b)
|
||||
}))
|
||||
t.Cleanup(ts.Close)
|
||||
return ts.URL
|
||||
}
|
||||
|
||||
func status(err error) int {
|
||||
var fe *fetch.Error
|
||||
if errors.As(err, &fe) {
|
||||
return fe.Status
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func TestPDFBodyCap(t *testing.T) {
|
||||
six := 6 << 20 // over DefaultMaxBody (5 MiB), under DefaultPDFMaxBody (100 MiB)
|
||||
for _, tc := range []struct {
|
||||
name, ct string
|
||||
b []byte
|
||||
want int // 0 = ok
|
||||
}{
|
||||
{"pdf by type", "application/pdf", body("%PDF-1.7\n", six), 0},
|
||||
{"pdf by sniffing octet-stream", "application/octet-stream", body("%PDF-1.7\n", six), 0},
|
||||
{"pdf with no type", "", body("%PDF-1.7\n", six), 0},
|
||||
{"html keeps the 5 MiB cap", "text/html", body("<html>", six), 413},
|
||||
{"octet-stream that is not a pdf keeps the cap", "application/octet-stream", body("PK\x03\x04", six), 413},
|
||||
} {
|
||||
f := loopbackFetcher()
|
||||
res, err := f.Fetch(context.Background(), serve(t, tc.ct, tc.b))
|
||||
if tc.want == 0 {
|
||||
if err != nil || res == nil || len(res.Body) != six {
|
||||
t.Errorf("%s: err %v, body %d bytes; want the whole %d", tc.name, err, lenBody(res), six)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if status(err) != tc.want {
|
||||
t.Errorf("%s: err %v (status %d), want %d", tc.name, err, status(err), tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func lenBody(r *fetch.Resource) int {
|
||||
if r == nil {
|
||||
return -1
|
||||
}
|
||||
return len(r.Body)
|
||||
}
|
||||
|
||||
func TestPDFMaxBodyIsConfigurable(t *testing.T) {
|
||||
f := loopbackFetcher()
|
||||
f.PDFMaxBody = 1 << 20
|
||||
_, err := f.Fetch(context.Background(), serve(t, "application/pdf", body("%PDF-1.7\n", 2<<20)))
|
||||
if status(err) != 413 || !strings.Contains(err.Error(), "too large") {
|
||||
t.Errorf("2 MiB pdf over a 1 MiB PDFMaxBody: %v, want 413 too large", err)
|
||||
}
|
||||
if fetch.DefaultPDFMaxBody != 100<<20 || fetch.DefaultPDFTimeout != 120*time.Second {
|
||||
t.Errorf("defaults = %d, %v; want 100 MiB, 120s", fetch.DefaultPDFMaxBody, fetch.DefaultPDFTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
// slow sends the headers, then the body in pieces over about 600 ms.
|
||||
func slow(t *testing.T, ct string, first string) string {
|
||||
t.Helper()
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", ct)
|
||||
w.WriteHeader(200)
|
||||
fl, _ := w.(http.Flusher)
|
||||
_, _ = w.Write([]byte(first))
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
for i := 0; i < 6; i++ {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
_, _ = w.Write(bytes.Repeat([]byte("y"), 1024))
|
||||
if fl != nil {
|
||||
fl.Flush()
|
||||
}
|
||||
}
|
||||
}))
|
||||
t.Cleanup(ts.Close)
|
||||
return ts.URL
|
||||
}
|
||||
|
||||
// The web-page deadline (Timeout) still applies to HTML; a PDF gets PDFTimeout.
|
||||
func TestPDFDeadline(t *testing.T) {
|
||||
f := loopbackFetcher()
|
||||
f.Timeout = 250 * time.Millisecond
|
||||
f.PDFTimeout = 3 * time.Second
|
||||
if _, err := f.Fetch(context.Background(), slow(t, "application/pdf", "%PDF-1.7\n")); err != nil {
|
||||
t.Errorf("slow pdf under PDFTimeout: %v", err)
|
||||
}
|
||||
if _, err := f.Fetch(context.Background(), slow(t, "application/octet-stream", "%PDF-1.7\n")); err != nil {
|
||||
t.Errorf("slow sniffed pdf under PDFTimeout: %v", err)
|
||||
}
|
||||
start := time.Now()
|
||||
if _, err := f.Fetch(context.Background(), slow(t, "text/html", "<html>")); err == nil {
|
||||
t.Errorf("slow html past Timeout succeeded; want an error")
|
||||
}
|
||||
if d := time.Since(start); d > 2*time.Second {
|
||||
t.Errorf("slow html took %v to fail; the 250 ms deadline was not applied", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchIf(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var got []string
|
||||
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
got = append(got, r.Header.Get("If-None-Match")+"|"+r.Header.Get("If-Modified-Since"))
|
||||
mu.Unlock()
|
||||
if r.Header.Get("If-None-Match") == `"v1"` {
|
||||
w.WriteHeader(http.StatusNotModified)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/pdf")
|
||||
w.Header().Set("ETag", `"v2"`)
|
||||
w.Header().Set("Last-Modified", "Fri, 25 Sep 2026 12:00:00 GMT")
|
||||
_, _ = w.Write([]byte("%PDF-1.7\nnew"))
|
||||
}))
|
||||
defer ts.Close()
|
||||
f := loopbackFetcher()
|
||||
|
||||
res, err := f.FetchIf(context.Background(), ts.URL, `"v1"`, "")
|
||||
if err != nil || res == nil || res.Status != http.StatusNotModified || len(res.Body) != 0 {
|
||||
t.Fatalf("matching ETag: res %+v err %v; want status 304, empty body, nil error", res, err)
|
||||
}
|
||||
res, err = f.FetchIf(context.Background(), ts.URL, `"old"`, "Thu, 24 Sep 2026 12:00:00 GMT")
|
||||
if err != nil || res.Status != 200 || string(res.Body) != "%PDF-1.7\nnew" || res.Header.Get("ETag") != `"v2"` {
|
||||
t.Fatalf("changed document: res %+v err %v; want 200 with the new body and ETag", res, err)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(got) != 2 || got[0] != `"v1"|` || got[1] != `"old"|Thu, 24 Sep 2026 12:00:00 GMT` {
|
||||
t.Errorf("validators sent: %q", got)
|
||||
}
|
||||
// With no validators FetchIf is a plain Fetch.
|
||||
if _, err := f.FetchIf(context.Background(), ts.URL, "", ""); err != nil {
|
||||
t.Errorf("FetchIf with no validators: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package metrics
|
||||
|
||||
// P5a task 07: the sift_pdf_* series.
|
||||
|
||||
import (
|
||||
"io"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type zeroSrc struct{}
|
||||
|
||||
func (zeroSrc) Snapshot() Snapshot { return Snapshot{} }
|
||||
|
||||
func TestPDFSeries(t *testing.T) {
|
||||
s := New(zeroSrc{})
|
||||
s.ObservePDF("ok", 1500*time.Millisecond, 212)
|
||||
s.ObservePDF("ok", 200*time.Millisecond, 3)
|
||||
s.ObservePDF("no_text", 50*time.Millisecond, 0)
|
||||
s.ObservePDF("blocked: evil.example", time.Millisecond, 0) // raw text never becomes a label
|
||||
s.ObservePDFWait(2 * time.Second)
|
||||
s.PDFCacheHit()
|
||||
s.PDFCacheHit()
|
||||
|
||||
req := httptest.NewRequest("GET", "/metrics", nil)
|
||||
req.RemoteAddr, req.Host = "127.0.0.1:9", "127.0.0.1:8890"
|
||||
rec := httptest.NewRecorder()
|
||||
s.Handler().ServeHTTP(rec, req)
|
||||
body, _ := io.ReadAll(rec.Body)
|
||||
out := string(body)
|
||||
for _, want := range []string{
|
||||
`sift_pdf_extract_total{outcome="ok"} 2`,
|
||||
`sift_pdf_extract_total{outcome="no_text"} 1`,
|
||||
`sift_pdf_extract_total{outcome="other"} 1`,
|
||||
`sift_pdf_pages_total 215`,
|
||||
`sift_pdf_cache_hits_total 2`,
|
||||
`sift_pdf_extract_seconds_count 4`,
|
||||
`sift_pdf_pool_wait_seconds_count 1`,
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("metrics lack %q", want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "evil.example") {
|
||||
t.Error("a raw outcome string leaked into a label")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
package pdfdoc_test
|
||||
|
||||
// P5a task 05: rendering an extracted document for an agent — the whole text with page markers,
|
||||
// the map for a long document, a page range, and grep.
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdfdoc"
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdftext"
|
||||
)
|
||||
|
||||
func small() *pdftext.Doc {
|
||||
return &pdftext.Doc{
|
||||
Meta: pdftext.Meta{Pages: 3, Title: "Three Pages", Outline: []pdftext.OutlineItem{{Title: "1 Intro", Page: 1, Level: 1}, {Title: "2 Fox", Page: 2, Level: 1}, {Title: "2.1 Den", Page: 2, Level: 2}}},
|
||||
Text: []string{"Chapter one begins here.\nSecond line.\n", " The quick brown fox is on page two.\nIt jumps.", "Last page."},
|
||||
}
|
||||
}
|
||||
|
||||
// big returns n pages of about 1,000 characters each, every line naming its page.
|
||||
func big(n int) *pdftext.Doc {
|
||||
d := &pdftext.Doc{Meta: pdftext.Meta{Pages: n, Title: "Long Thesis", Outline: []pdftext.OutlineItem{{Title: "1 Introduction", Page: 1, Level: 1}, {Title: "2 Method", Page: 12, Level: 1}, {Title: "2.3 Setup", Page: 15, Level: 2}}}}
|
||||
for p := 1; p <= n; p++ {
|
||||
var lines []string
|
||||
for l := 1; l <= 12; l++ {
|
||||
lines = append(lines, fmt.Sprintf("page %d line %d lorem ipsum dolor sit amet consectetur adipiscing elit sed do eiusmod", p, l))
|
||||
}
|
||||
if p == 37 {
|
||||
lines[4] = "the Needle is here on page 37"
|
||||
}
|
||||
d.Text = append(d.Text, strings.Join(lines, "\n"))
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func TestFull(t *testing.T) {
|
||||
want := "--- page 1 ---\n\nChapter one begins here.\nSecond line.\n\n--- page 2 ---\n\nThe quick brown fox is on page two.\nIt jumps.\n\n--- page 3 ---\n\nLast page."
|
||||
if got := pdfdoc.Full(small()); got != want {
|
||||
t.Errorf("Full =\n%q\nwant\n%q", got, want)
|
||||
}
|
||||
if pdfdoc.Marker(12) != "--- page 12 ---" {
|
||||
t.Errorf("Marker(12) = %q", pdfdoc.Marker(12))
|
||||
}
|
||||
}
|
||||
|
||||
func TestWholeDocumentWhenItFits(t *testing.T) {
|
||||
r, err := pdfdoc.Render(small(), pdfdoc.Options{OutCap: 10000})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.Map || r.First != 1 || r.Last != 3 || r.Content != pdfdoc.Full(small()) {
|
||||
t.Errorf("got %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapForALongDocument(t *testing.T) {
|
||||
d := big(60) // ~60k characters
|
||||
r, err := pdfdoc.Render(d, pdfdoc.Options{OutCap: 20000})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !r.Map || r.First != 1 || r.Last < 1 {
|
||||
t.Fatalf("got Map=%v First=%d Last=%d, want a map from page 1", r.Map, r.First, r.Last)
|
||||
}
|
||||
c := r.Content
|
||||
for _, want := range []string{
|
||||
"# Long Thesis\n",
|
||||
"60 pages · ~",
|
||||
"## Outline\n",
|
||||
"- 1 Introduction (p. 1)\n",
|
||||
"- 2 Method (p. 12)\n",
|
||||
" - 2.3 Setup (p. 15)\n", // level 2 is indented two spaces
|
||||
`pages="a-b"`,
|
||||
`grep="pattern"`,
|
||||
"--- page 1 ---",
|
||||
} {
|
||||
if !strings.Contains(c, want) {
|
||||
t.Errorf("map lacks %q", want)
|
||||
}
|
||||
}
|
||||
if utf8.RuneCountInString(c) > 20000 {
|
||||
t.Errorf("map is %d characters, over OutCap 20000", utf8.RuneCountInString(c))
|
||||
}
|
||||
// Leading pages take at most a quarter of the cap, and only whole pages.
|
||||
if strings.Contains(c, fmt.Sprintf("--- page %d ---", r.Last+1)) {
|
||||
t.Errorf("content has page %d past Last %d", r.Last+1, r.Last)
|
||||
}
|
||||
if !strings.Contains(c, fmt.Sprintf("page %d line 12", r.Last)) {
|
||||
t.Errorf("the last page shown (%d) is cut; pages in a map are whole", r.Last)
|
||||
}
|
||||
idx := strings.Index(c, "--- page 1 ---")
|
||||
if pagesPart := c[idx:]; utf8.RuneCountInString(pagesPart) > 20000/4 {
|
||||
t.Errorf("leading pages take %d characters, over OutCap/4", utf8.RuneCountInString(pagesPart))
|
||||
}
|
||||
}
|
||||
|
||||
func TestMapNotesPartialExtraction(t *testing.T) {
|
||||
d := big(30)
|
||||
d.Pages = 2500 // the child stopped at its page cap
|
||||
r, _ := pdfdoc.Render(d, pdfdoc.Options{OutCap: 5000})
|
||||
if !strings.Contains(r.Content, "2500 pages") || !strings.Contains(r.Content, "pages 1-30 extracted") {
|
||||
t.Errorf("map must say how much was extracted:\n%s", r.Content[:min(600, len(r.Content))])
|
||||
}
|
||||
}
|
||||
|
||||
func TestUntitledAndNoOutline(t *testing.T) {
|
||||
d := big(40)
|
||||
d.Title, d.Outline = "", nil
|
||||
r, _ := pdfdoc.Render(d, pdfdoc.Options{OutCap: 5000})
|
||||
if !strings.HasPrefix(r.Content, "# Untitled PDF\n") || strings.Contains(r.Content, "## Outline") {
|
||||
t.Errorf("untitled, outline-less map:\n%s", r.Content[:min(400, len(r.Content))])
|
||||
}
|
||||
}
|
||||
|
||||
func TestPages(t *testing.T) {
|
||||
d := big(60)
|
||||
for _, tc := range []struct {
|
||||
spec string
|
||||
first, last int
|
||||
}{
|
||||
{"2-3", 2, 3},
|
||||
{"7", 7, 7},
|
||||
{" 58- ", 58, 60},
|
||||
{"60-60", 60, 60},
|
||||
} {
|
||||
r, err := pdfdoc.Render(d, pdfdoc.Options{Pages: tc.spec, OutCap: 200000})
|
||||
if err != nil {
|
||||
t.Errorf("%q: %v", tc.spec, err)
|
||||
continue
|
||||
}
|
||||
if r.Map || r.First != tc.first || r.Last != tc.last {
|
||||
t.Errorf("%q: First %d Last %d Map %v, want %d-%d", tc.spec, r.First, r.Last, r.Map, tc.first, tc.last)
|
||||
}
|
||||
if !strings.HasPrefix(r.Content, pdfdoc.Marker(tc.first)+"\n\n") || strings.Contains(r.Content, pdfdoc.Marker(tc.last+1)) || (tc.first > 1 && strings.Contains(r.Content, pdfdoc.Marker(tc.first-1)+"\n")) {
|
||||
t.Errorf("%q: wrong pages in content: %.120q…", tc.spec, r.Content)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{"0-1", "5-4", "61", "999", "a-b", "1-2-3", "", "-5"} {
|
||||
if bad == "" {
|
||||
continue // empty means "no range", tested above
|
||||
}
|
||||
_, err := pdfdoc.Render(d, pdfdoc.Options{Pages: bad, OutCap: 200000})
|
||||
if !errors.Is(err, pdfdoc.ErrBadPages) {
|
||||
t.Errorf("%q: %v, want ErrBadPages", bad, err)
|
||||
}
|
||||
}
|
||||
_, err := pdfdoc.Render(d, pdfdoc.Options{Pages: "61", OutCap: 200000})
|
||||
if err == nil || !strings.Contains(err.Error(), "pages out of range (1-60)") {
|
||||
t.Errorf("out-of-range message: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A range is at most 100 pages and at most OutCap characters, cut at a whole page with a note
|
||||
// that says how to continue.
|
||||
func TestPagesAreCapped(t *testing.T) {
|
||||
d := big(300)
|
||||
r, err := pdfdoc.Render(d, pdfdoc.Options{Pages: "1-300", OutCap: 1 << 30})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.Last != 100 || !strings.Contains(r.Note, `pages="101-300"`) {
|
||||
t.Errorf("1-300: Last %d note %q; want 100 and a pointer to 101-300", r.Last, r.Note)
|
||||
}
|
||||
r, _ = pdfdoc.Render(d, pdfdoc.Options{Pages: "10-40", OutCap: 5000})
|
||||
if utf8.RuneCountInString(r.Content) > 5000 || r.First != 10 || r.Last < 10 || r.Last >= 40 {
|
||||
t.Errorf("10-40 under a 5000 cap: First %d Last %d, %d chars", r.First, r.Last, utf8.RuneCountInString(r.Content))
|
||||
}
|
||||
if !strings.Contains(r.Note, fmt.Sprintf(`pages="%d-40"`, r.Last+1)) {
|
||||
t.Errorf("note %q must point at the rest", r.Note)
|
||||
}
|
||||
// One page bigger than the cap is truncated rather than dropped.
|
||||
huge := &pdftext.Doc{Meta: pdftext.Meta{Pages: 1}, Text: []string{strings.Repeat("w ", 10000)}}
|
||||
r, _ = pdfdoc.Render(huge, pdfdoc.Options{Pages: "1", OutCap: 1000})
|
||||
if r.First != 1 || r.Last != 1 || utf8.RuneCountInString(r.Content) > 1000 || r.Note == "" {
|
||||
t.Errorf("oversized single page: %+v (%d chars)", r, utf8.RuneCountInString(r.Content))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGrep(t *testing.T) {
|
||||
d := big(60)
|
||||
r, err := pdfdoc.Render(d, pdfdoc.Options{Grep: "needle", OutCap: 200000})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.GrepHits != 1 || r.First != 0 || r.Map {
|
||||
t.Errorf("got %+v", r)
|
||||
}
|
||||
for _, want := range []string{
|
||||
`grep "needle": 1 match`,
|
||||
"p. 37- page 37 line 4",
|
||||
"p. 37: the Needle is here on page 37",
|
||||
"p. 37- page 37 line 6",
|
||||
} {
|
||||
if !strings.Contains(r.Content, want) {
|
||||
t.Errorf("grep output lacks %q:\n%s", want, r.Content)
|
||||
}
|
||||
}
|
||||
// Smart case: an upper-case letter makes the pattern case-sensitive.
|
||||
if r, _ := pdfdoc.Render(d, pdfdoc.Options{Grep: "NEEDLE", OutCap: 200000}); r.GrepHits != 0 {
|
||||
t.Errorf("NEEDLE matched %d times, want 0 (case-sensitive)", r.GrepHits)
|
||||
}
|
||||
if r, _ := pdfdoc.Render(d, pdfdoc.Options{Grep: "Needle", OutCap: 200000}); r.GrepHits != 1 {
|
||||
t.Errorf("Needle matched %d times, want 1", r.GrepHits)
|
||||
}
|
||||
// No match is an answer, not an error.
|
||||
r, err = pdfdoc.Render(d, pdfdoc.Options{Grep: "zebra", OutCap: 200000})
|
||||
if err != nil || r.GrepHits != 0 || !strings.Contains(r.Content, `grep "zebra": no matches`) {
|
||||
t.Errorf("no match: %+v %v", r, err)
|
||||
}
|
||||
// Within a page range.
|
||||
r, _ = pdfdoc.Render(d, pdfdoc.Options{Grep: "line 3 ", Pages: "5-6", OutCap: 200000})
|
||||
if r.GrepHits != 2 || strings.Contains(r.Content, "p. 7") {
|
||||
t.Errorf("grep in 5-6: %d hits\n%s", r.GrepHits, r.Content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGrepLimits(t *testing.T) {
|
||||
d := big(200)
|
||||
r, err := pdfdoc.Render(d, pdfdoc.Options{Grep: "lorem", OutCap: 1 << 30}) // 2,399 matches: every line but the needle
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r.GrepHits != 2399 || strings.Count(r.Content, ": page ") != 100 || !strings.Contains(r.Content, "showing the first 100") {
|
||||
t.Errorf("hits %d, shown %d", r.GrepHits, strings.Count(r.Content, ": page "))
|
||||
}
|
||||
if _, err := pdfdoc.Render(d, pdfdoc.Options{Grep: strings.Repeat("a", 201), OutCap: 1000}); !errors.Is(err, pdfdoc.ErrBadGrep) {
|
||||
t.Errorf("201-byte pattern: %v, want ErrBadGrep", err)
|
||||
}
|
||||
if _, err := pdfdoc.Render(d, pdfdoc.Options{Grep: "(unclosed", OutCap: 1000}); !errors.Is(err, pdfdoc.ErrBadGrep) {
|
||||
t.Errorf("bad regexp: %v, want ErrBadGrep", err)
|
||||
}
|
||||
// RE2 is linear: the classic catastrophic pattern over a 1 MB page finishes quickly.
|
||||
evil := &pdftext.Doc{Meta: pdftext.Meta{Pages: 1}, Text: []string{strings.Repeat("a", 1<<20) + "!"}}
|
||||
start := time.Now()
|
||||
_, _ = pdfdoc.Render(evil, pdfdoc.Options{Grep: "(a+)+$", OutCap: 1000})
|
||||
if d := time.Since(start); d > time.Second {
|
||||
t.Errorf("(a+)+$ took %v", d)
|
||||
}
|
||||
// Output stays under the cap.
|
||||
r, _ = pdfdoc.Render(d, pdfdoc.Options{Grep: "lorem", OutCap: 3000})
|
||||
if utf8.RuneCountInString(r.Content) > 3000 {
|
||||
t.Errorf("grep output %d characters over a 3000 cap", utf8.RuneCountInString(r.Content))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
package pdfpool_test
|
||||
|
||||
// P5a task 04: the extraction pool. A few slots, a short queue, single-flight per key, and at
|
||||
// most one job in flight per caller.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdfpool"
|
||||
)
|
||||
|
||||
func waitUntil(t *testing.T, cond func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for !cond() {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("condition not reached within 3 s")
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSingleFlight(t *testing.T) {
|
||||
p := pdfpool.New(2, 4, time.Second)
|
||||
var calls atomic.Int32
|
||||
release := make(chan struct{})
|
||||
fn := func(ctx context.Context) (any, error) {
|
||||
calls.Add(1)
|
||||
<-release
|
||||
return "text", nil
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
results := make(chan any, 5)
|
||||
for i := 0; i < 5; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
v, err := p.Do(context.Background(), "doc-1", "caller-"+string(rune('a'+i)), fn)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
results <- v
|
||||
}(i)
|
||||
}
|
||||
waitUntil(t, func() bool { return calls.Load() == 1 && p.Waiting("doc-1") == 4 })
|
||||
close(release)
|
||||
wg.Wait()
|
||||
close(results)
|
||||
for v := range results {
|
||||
if v != "text" {
|
||||
t.Errorf("a sharer got %v", v)
|
||||
}
|
||||
}
|
||||
if n := calls.Load(); n != 1 {
|
||||
t.Errorf("fn ran %d times for one key, want 1", n)
|
||||
}
|
||||
// Nothing is cached: the next call runs fn again.
|
||||
_, _ = p.Do(context.Background(), "doc-1", "x", func(ctx context.Context) (any, error) { calls.Add(1); return nil, nil })
|
||||
if n := calls.Load(); n != 2 {
|
||||
t.Errorf("after the flight ended, fn ran %d times in total, want 2", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Two slots and four waiting places: the seventh distinct job is refused at once.
|
||||
func TestSlotsAndQueue(t *testing.T) {
|
||||
p := pdfpool.New(2, 4, 5*time.Second)
|
||||
release := make(chan struct{})
|
||||
var running atomic.Int32
|
||||
block := func(ctx context.Context) (any, error) {
|
||||
running.Add(1)
|
||||
defer running.Add(-1)
|
||||
<-release
|
||||
return nil, nil
|
||||
}
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 6; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
if _, err := p.Do(context.Background(), "doc-"+string(rune('a'+i)), "caller-"+string(rune('a'+i)), block); err != nil {
|
||||
t.Errorf("job %d: %v", i, err)
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
waitUntil(t, func() bool { return running.Load() == 2 && p.Queued() == 4 })
|
||||
start := time.Now()
|
||||
_, err := p.Do(context.Background(), "doc-z", "caller-z", block)
|
||||
if !errors.Is(err, pdfpool.ErrBusy) {
|
||||
t.Errorf("seventh job: %v, want ErrBusy", err)
|
||||
}
|
||||
if d := time.Since(start); d > 200*time.Millisecond {
|
||||
t.Errorf("a full queue refused after %v; it must refuse at once", d)
|
||||
}
|
||||
if running.Load() > 2 {
|
||||
t.Errorf("%d jobs ran at once, want at most 2", running.Load())
|
||||
}
|
||||
close(release)
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// A queued job that waits longer than Wait is refused with ErrBusy and gives up its place.
|
||||
func TestQueueWait(t *testing.T) {
|
||||
p := pdfpool.New(1, 1, 150*time.Millisecond)
|
||||
release := make(chan struct{})
|
||||
defer close(release)
|
||||
go func() {
|
||||
_, _ = p.Do(context.Background(), "a", "ca", func(ctx context.Context) (any, error) { <-release; return nil, nil })
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Running() == 1 })
|
||||
start := time.Now()
|
||||
_, err := p.Do(context.Background(), "b", "cb", func(ctx context.Context) (any, error) { return nil, nil })
|
||||
if !errors.Is(err, pdfpool.ErrBusy) {
|
||||
t.Fatalf("waiter: %v, want ErrBusy", err)
|
||||
}
|
||||
if d := time.Since(start); d < 100*time.Millisecond || d > time.Second {
|
||||
t.Errorf("waiter refused after %v, want about 150 ms", d)
|
||||
}
|
||||
if p.Queued() != 0 {
|
||||
t.Errorf("queued = %d after the waiter gave up, want 0", p.Queued())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCallerBusy(t *testing.T) {
|
||||
p := pdfpool.New(2, 4, time.Second)
|
||||
release := make(chan struct{})
|
||||
go func() {
|
||||
_, _ = p.Do(context.Background(), "doc-1", "straylight", func(ctx context.Context) (any, error) { <-release; return nil, nil })
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Running() == 1 })
|
||||
if _, err := p.Do(context.Background(), "doc-2", "straylight", func(ctx context.Context) (any, error) { return nil, nil }); !errors.Is(err, pdfpool.ErrCallerBusy) {
|
||||
t.Errorf("second job from the same caller: %v, want ErrCallerBusy", err)
|
||||
}
|
||||
// The same caller asking for the same document again joins the flight instead.
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := p.Do(context.Background(), "doc-1", "straylight", func(ctx context.Context) (any, error) { return nil, nil })
|
||||
done <- err
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Waiting("doc-1") == 1 })
|
||||
close(release)
|
||||
if err := <-done; err != nil {
|
||||
t.Errorf("joining its own flight: %v", err)
|
||||
}
|
||||
// Once the job is done the caller may start another.
|
||||
if _, err := p.Do(context.Background(), "doc-2", "straylight", func(ctx context.Context) (any, error) { return nil, nil }); err != nil {
|
||||
t.Errorf("after the first job ended: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrorsAndPanicsReachEverySharer(t *testing.T) {
|
||||
p := pdfpool.New(2, 4, time.Second)
|
||||
boom := errors.New("boom")
|
||||
if _, err := p.Do(context.Background(), "k", "c", func(ctx context.Context) (any, error) { return nil, boom }); !errors.Is(err, boom) {
|
||||
t.Errorf("fn error: %v", err)
|
||||
}
|
||||
_, err := p.Do(context.Background(), "k", "c", func(ctx context.Context) (any, error) { panic("kaboom") })
|
||||
if err == nil {
|
||||
t.Fatal("a panicking fn returned no error")
|
||||
}
|
||||
if p.Running() != 0 {
|
||||
t.Errorf("running = %d after a panic, want 0 (the slot leaked)", p.Running())
|
||||
}
|
||||
if _, err := p.Do(context.Background(), "k2", "c", func(ctx context.Context) (any, error) { return 1, nil }); err != nil {
|
||||
t.Errorf("the caller is still marked busy after its job panicked: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContextCancelWhileQueued(t *testing.T) {
|
||||
p := pdfpool.New(1, 2, 5*time.Second)
|
||||
release := make(chan struct{})
|
||||
defer close(release)
|
||||
go func() {
|
||||
_, _ = p.Do(context.Background(), "a", "ca", func(ctx context.Context) (any, error) { <-release; return nil, nil })
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Running() == 1 })
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
errc := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := p.Do(ctx, "b", "cb", func(ctx context.Context) (any, error) { return nil, nil })
|
||||
errc <- err
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Queued() == 1 })
|
||||
cancel()
|
||||
if err := <-errc; !errors.Is(err, context.Canceled) {
|
||||
t.Errorf("cancelled waiter: %v, want context.Canceled", err)
|
||||
}
|
||||
if p.Queued() != 0 {
|
||||
t.Errorf("queued = %d after cancel, want 0", p.Queued())
|
||||
}
|
||||
}
|
||||
|
||||
// The job's own context is not the first caller's: if the caller that started a shared flight
|
||||
// goes away, the sharers still get the result.
|
||||
func TestFlightOutlivesItsStarter(t *testing.T) {
|
||||
p := pdfpool.New(1, 2, 5*time.Second)
|
||||
release := make(chan struct{})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
jobCtxErr := make(chan error, 1)
|
||||
go func() {
|
||||
_, _ = p.Do(ctx, "doc", "first", func(jctx context.Context) (any, error) {
|
||||
<-release
|
||||
jobCtxErr <- jctx.Err()
|
||||
return "done", nil
|
||||
})
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Running() == 1 })
|
||||
res := make(chan any, 1)
|
||||
go func() {
|
||||
v, _ := p.Do(context.Background(), "doc", "second", nil)
|
||||
res <- v
|
||||
}()
|
||||
waitUntil(t, func() bool { return p.Waiting("doc") == 1 })
|
||||
cancel()
|
||||
close(release)
|
||||
if v := <-res; v != "done" {
|
||||
t.Errorf("sharer got %v after the starter left, want done", v)
|
||||
}
|
||||
if e := <-jobCtxErr; e != nil {
|
||||
t.Errorf("the job's context was cancelled with its starter: %v", e)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
// Package pdftest builds small, deterministic PDFs for tests: text pages in Helvetica, an optional
|
||||
// outline (bookmarks), and image-only ("scanned") pages. It exists so the P5a tests need no binary
|
||||
// fixtures from elsewhere. It is test support: nothing outside _test.go files imports it.
|
||||
package pdftest
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Outline is one bookmark: a title that points at a 1-based page, nested at Level (1 = top).
|
||||
// Build supports one level; Level is kept so tests can state what they expect back.
|
||||
type Outline struct {
|
||||
Title string
|
||||
Page int
|
||||
Level int
|
||||
}
|
||||
|
||||
// Doc describes a PDF to build. Pages holds each page's text; a page's lines are split on "\n"
|
||||
// and set one per line (long lines are not wrapped: callers keep them under ~90 characters).
|
||||
// Scanned pages (1-based) carry a small image and no text at all.
|
||||
type Doc struct {
|
||||
Title string
|
||||
Pages []string
|
||||
Outline []Outline
|
||||
Scanned []int
|
||||
}
|
||||
|
||||
// Build returns the PDF bytes for d. It panics on a Doc it cannot express (an outline entry
|
||||
// pointing past the last page), since that is a bug in the test that built it.
|
||||
func Build(d Doc) []byte {
|
||||
var b bytes.Buffer
|
||||
var offsets []int
|
||||
obj := func(body string) int {
|
||||
offsets = append(offsets, b.Len())
|
||||
n := len(offsets)
|
||||
fmt.Fprintf(&b, "%d 0 obj\n%s\nendobj\n", n, body)
|
||||
return n
|
||||
}
|
||||
b.WriteString("%PDF-1.4\n%\xe2\xe3\xcf\xd3\n")
|
||||
|
||||
// Object numbers are fixed by position: 1 catalog, 2 pages, 3 font, 4 image, 5 info, then
|
||||
// two per page (page, content), then the outline.
|
||||
nPages := len(d.Pages)
|
||||
firstPage := 6
|
||||
pageObj := func(i int) int { return firstPage + 2*i }
|
||||
outlineRoot := firstPage + 2*nPages
|
||||
scanned := map[int]bool{}
|
||||
for _, p := range d.Scanned {
|
||||
scanned[p] = true
|
||||
}
|
||||
|
||||
catalog := "<< /Type /Catalog /Pages 2 0 R"
|
||||
if len(d.Outline) > 0 {
|
||||
catalog += fmt.Sprintf(" /Outlines %d 0 R /PageMode /UseOutlines", outlineRoot)
|
||||
}
|
||||
obj(catalog + " >>")
|
||||
var kids []string
|
||||
for i := range d.Pages {
|
||||
kids = append(kids, fmt.Sprintf("%d 0 R", pageObj(i)))
|
||||
}
|
||||
obj(fmt.Sprintf("<< /Type /Pages /Kids [%s] /Count %d >>", strings.Join(kids, " "), nPages))
|
||||
obj("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>")
|
||||
img := []byte{0x00, 0x80, 0x80, 0x00} // 2x2 grey pixels
|
||||
obj(fmt.Sprintf("<< /Type /XObject /Subtype /Image /Width 2 /Height 2 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length %d >>\nstream\n%s\nendstream", len(img), img))
|
||||
obj(fmt.Sprintf("<< /Title (%s) /Producer (sift pdftest) >>", esc(d.Title)))
|
||||
|
||||
for i, text := range d.Pages {
|
||||
var content string
|
||||
if scanned[i+1] {
|
||||
content = "q 400 0 0 500 100 150 cm /Im1 Do Q"
|
||||
} else {
|
||||
var c strings.Builder
|
||||
c.WriteString("BT /F1 11 Tf 13 TL 60 760 Td\n")
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
fmt.Fprintf(&c, "(%s) Tj T*\n", esc(line))
|
||||
}
|
||||
c.WriteString("ET")
|
||||
content = c.String()
|
||||
}
|
||||
obj(fmt.Sprintf("<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 3 0 R >> /XObject << /Im1 4 0 R >> >> /Contents %d 0 R >>", pageObj(i)+1))
|
||||
obj(fmt.Sprintf("<< /Length %d >>\nstream\n%s\nendstream", len(content), content))
|
||||
}
|
||||
|
||||
if len(d.Outline) > 0 {
|
||||
first := outlineRoot + 1
|
||||
last := outlineRoot + len(d.Outline)
|
||||
obj(fmt.Sprintf("<< /Type /Outlines /First %d 0 R /Last %d 0 R /Count %d >>", first, last, len(d.Outline)))
|
||||
for i, o := range d.Outline {
|
||||
if o.Page < 1 || o.Page > nPages {
|
||||
panic(fmt.Sprintf("pdftest: outline %q points at page %d of %d", o.Title, o.Page, nPages))
|
||||
}
|
||||
n := first + i
|
||||
item := fmt.Sprintf("<< /Title (%s) /Parent %d 0 R /Dest [%d 0 R /XYZ null null null]", esc(o.Title), outlineRoot, pageObj(o.Page-1))
|
||||
if i > 0 {
|
||||
item += fmt.Sprintf(" /Prev %d 0 R", n-1)
|
||||
}
|
||||
if n < last {
|
||||
item += fmt.Sprintf(" /Next %d 0 R", n+1)
|
||||
}
|
||||
obj(item + " >>")
|
||||
}
|
||||
}
|
||||
|
||||
xref := b.Len()
|
||||
fmt.Fprintf(&b, "xref\n0 %d\n0000000000 65535 f \n", len(offsets)+1)
|
||||
for _, off := range offsets {
|
||||
fmt.Fprintf(&b, "%010d 00000 n \n", off)
|
||||
}
|
||||
fmt.Fprintf(&b, "trailer\n<< /Size %d /Root 1 0 R /Info 5 0 R >>\nstartxref\n%d\n%%%%EOF\n", len(offsets)+1, xref)
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
// Lorem returns n lines of distinct, searchable filler: "p<page> l<line> …", so a test can tell
|
||||
// which page a line came from.
|
||||
func Lorem(page, n int) string {
|
||||
words := []string{"alpha", "bravo", "charlie", "delta", "echo", "foxtrot", "golf", "hotel", "india", "juliet", "kilo", "lima"}
|
||||
var lines []string
|
||||
for i := 1; i <= n; i++ {
|
||||
var w []string
|
||||
for j := 0; j < 10; j++ {
|
||||
w = append(w, words[(page*7+i*3+j)%len(words)])
|
||||
}
|
||||
lines = append(lines, fmt.Sprintf("p%d l%d %s", page, i, strings.Join(w, " ")))
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
func esc(s string) string {
|
||||
r := strings.NewReplacer(`\`, `\\`, `(`, `\(`, `)`, `\)`)
|
||||
return r.Replace(s)
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package pdftext_test
|
||||
|
||||
// P5a task 02: Extract (the child's work), ChildMain (the `sift pdftext` entry point) and Runner
|
||||
// (the parent: exec, deadline, kill, parse).
|
||||
//
|
||||
// The Runner tests run the test binary itself as the child: TestMain turns into the child when
|
||||
// SIFT_PDFTEXT_CHILD=1 is set, exactly as `sift pdftext` would. The two other variables are test
|
||||
// hooks that live only here: SIFT_PDFTEXT_SLEEP delays the child (for the deadline test) and
|
||||
// SIFT_PDFTEXT_PIDFILE records its pid (to prove it is gone afterwards).
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdftest"
|
||||
"git.wntrmute.dev/kyle/sift/internal/pdftext"
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("SIFT_PDFTEXT_CHILD") == "1" {
|
||||
if p := os.Getenv("SIFT_PDFTEXT_PIDFILE"); p != "" {
|
||||
_ = os.WriteFile(p, []byte(strconv.Itoa(os.Getpid())), 0o600)
|
||||
}
|
||||
if d, err := time.ParseDuration(os.Getenv("SIFT_PDFTEXT_SLEEP")); err == nil {
|
||||
time.Sleep(d)
|
||||
}
|
||||
os.Exit(pdftext.ChildMain(os.Args[1:], os.Stdin, os.Stdout))
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func three() []byte {
|
||||
return pdftest.Build(pdftest.Doc{
|
||||
Title: "Three Pages",
|
||||
Pages: []string{
|
||||
"Chapter one begins here.\nSecond line (with parens).",
|
||||
"The quick brown fox is on page two.",
|
||||
pdftest.Lorem(3, 5),
|
||||
},
|
||||
Outline: []pdftest.Outline{{Title: "1 Intro", Page: 1, Level: 1}, {Title: "2 Fox", Page: 2, Level: 1}, {Title: "3 Filler", Page: 3, Level: 1}},
|
||||
})
|
||||
}
|
||||
|
||||
type line struct {
|
||||
Meta *pdftext.Meta `json:"meta"`
|
||||
Page int `json:"page"`
|
||||
Text *string `json:"text"`
|
||||
Error string `json:"error"`
|
||||
}
|
||||
|
||||
func lines(t *testing.T, out []byte) []line {
|
||||
t.Helper()
|
||||
var ls []line
|
||||
sc := bufio.NewScanner(bytes.NewReader(out))
|
||||
sc.Buffer(make([]byte, 1<<20), 64<<20)
|
||||
for sc.Scan() {
|
||||
var l line
|
||||
if err := json.Unmarshal(sc.Bytes(), &l); err != nil {
|
||||
t.Fatalf("output line is not JSON: %q (%v)", sc.Text(), err)
|
||||
}
|
||||
ls = append(ls, l)
|
||||
}
|
||||
return ls
|
||||
}
|
||||
|
||||
func TestExtractWritesMetaThenPages(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := pdftext.Extract(three(), 2000, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ls := lines(t, out.Bytes())
|
||||
if len(ls) != 4 || ls[0].Meta == nil {
|
||||
t.Fatalf("got %d lines, first meta=%v; want a meta line then 3 page lines:\n%s", len(ls), ls[0].Meta, out.String())
|
||||
}
|
||||
m := ls[0].Meta
|
||||
if m.Pages != 3 || m.Title != "Three Pages" {
|
||||
t.Errorf("meta = %+v, want 3 pages titled Three Pages", *m)
|
||||
}
|
||||
want := []pdftext.OutlineItem{{Title: "1 Intro", Page: 1, Level: 1}, {Title: "2 Fox", Page: 2, Level: 1}, {Title: "3 Filler", Page: 3, Level: 1}}
|
||||
if len(m.Outline) != 3 {
|
||||
t.Fatalf("outline = %+v, want %+v", m.Outline, want)
|
||||
}
|
||||
for i := range want {
|
||||
if m.Outline[i] != want[i] {
|
||||
t.Errorf("outline[%d] = %+v, want %+v", i, m.Outline[i], want[i])
|
||||
}
|
||||
}
|
||||
for i, l := range ls[1:] {
|
||||
if l.Page != i+1 || l.Text == nil || l.Error != "" {
|
||||
t.Errorf("line %d = %+v, want page %d with text", i+1, l, i+1)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(*ls[1].Text, "Chapter one begins here.") || !strings.Contains(*ls[2].Text, "quick brown fox") || !strings.Contains(*ls[3].Text, "p3 l5") {
|
||||
t.Errorf("page texts wrong: %q / %q / %q", *ls[1].Text, *ls[2].Text, *ls[3].Text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractPageCap(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := pdftext.Extract(three(), 2, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ls := lines(t, out.Bytes())
|
||||
if len(ls) != 3 || ls[0].Meta.Pages != 3 || ls[2].Page != 2 {
|
||||
t.Errorf("maxPages 2: %d lines, meta %+v; want meta (pages 3) and pages 1-2", len(ls), ls[0].Meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractNoOutline(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
if err := pdftext.Extract(pdftest.Build(pdftest.Doc{Pages: []string{"a", "b"}}), 10, &out); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ls := lines(t, out.Bytes())
|
||||
if ls[0].Meta.Outline == nil || len(ls[0].Meta.Outline) != 0 {
|
||||
t.Errorf("outline = %#v, want an empty non-nil slice (JSON [])", ls[0].Meta.Outline)
|
||||
}
|
||||
if !strings.Contains(out.String(), `"outline":[]`) {
|
||||
t.Errorf("meta line must carry \"outline\":[]: %s", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractErrors(t *testing.T) {
|
||||
enc, err := os.ReadFile(filepath.Join("..", "..", "tests", "testdata", "pdf", "encrypted-user.pdf"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := pdftext.Extract(enc, 10, &bytes.Buffer{}); !errors.Is(err, pdftext.ErrEncrypted) {
|
||||
t.Errorf("user-password pdf: %v, want ErrEncrypted", err)
|
||||
}
|
||||
owner, err := os.ReadFile(filepath.Join("..", "..", "tests", "testdata", "pdf", "encrypted-owner.pdf"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out bytes.Buffer
|
||||
if err := pdftext.Extract(owner, 10, &out); err != nil || !strings.Contains(out.String(), "quick brown fox") {
|
||||
t.Errorf("owner-password-only pdf: err %v; want its text", err)
|
||||
}
|
||||
if err := pdftext.Extract([]byte("not a pdf at all"), 10, &bytes.Buffer{}); !errors.Is(err, pdftext.ErrUnreadable) {
|
||||
t.Errorf("garbage: %v, want ErrUnreadable", err)
|
||||
}
|
||||
// A truncated file either yields what it can or ErrUnreadable; it never panics.
|
||||
b := three()
|
||||
_ = pdftext.Extract(b[:len(b)*6/10], 10, &bytes.Buffer{})
|
||||
}
|
||||
|
||||
func TestChildMainExitCodes(t *testing.T) {
|
||||
enc, _ := os.ReadFile(filepath.Join("..", "..", "tests", "testdata", "pdf", "encrypted-user.pdf"))
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
in []byte
|
||||
code int
|
||||
}{
|
||||
{"ok", three(), 0},
|
||||
{"encrypted", enc, 3},
|
||||
{"unreadable", []byte("nope"), 4},
|
||||
} {
|
||||
var out bytes.Buffer
|
||||
if code := pdftext.ChildMain([]string{"-max-pages", "5"}, bytes.NewReader(tc.in), &out); code != tc.code {
|
||||
t.Errorf("%s: exit %d, want %d", tc.name, code, tc.code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func runner(env ...string) *pdftext.Runner {
|
||||
return &pdftext.Runner{Exe: os.Args[0], Env: append([]string{"SIFT_PDFTEXT_CHILD=1"}, env...), Deadline: 20 * time.Second}
|
||||
}
|
||||
|
||||
func TestRunnerRun(t *testing.T) {
|
||||
doc, err := runner().Run(context.Background(), three())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if doc.Pages != 3 || doc.Title != "Three Pages" || len(doc.Outline) != 3 || len(doc.Text) != 3 {
|
||||
t.Fatalf("doc = %+v", doc)
|
||||
}
|
||||
if !strings.Contains(doc.Text[1], "quick brown fox") {
|
||||
t.Errorf("page 2 = %q", doc.Text[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerErrors(t *testing.T) {
|
||||
enc, _ := os.ReadFile(filepath.Join("..", "..", "tests", "testdata", "pdf", "encrypted-user.pdf"))
|
||||
if _, err := runner().Run(context.Background(), enc); !errors.Is(err, pdftext.ErrEncrypted) {
|
||||
t.Errorf("encrypted: %v, want ErrEncrypted", err)
|
||||
}
|
||||
if _, err := runner().Run(context.Background(), []byte("nope")); !errors.Is(err, pdftext.ErrUnreadable) {
|
||||
t.Errorf("garbage: %v, want ErrUnreadable", err)
|
||||
}
|
||||
scanned := pdftest.Build(pdftest.Doc{Pages: []string{"", "", ""}, Scanned: []int{1, 2, 3}})
|
||||
if _, err := runner().Run(context.Background(), scanned); !errors.Is(err, pdftext.ErrNoText) {
|
||||
t.Errorf("scanned: %v, want ErrNoText", err)
|
||||
}
|
||||
r := runner()
|
||||
r.MaxText = 100 // bytes of child output
|
||||
if _, err := r.Run(context.Background(), three()); !errors.Is(err, pdftext.ErrTooMuchText) {
|
||||
t.Errorf("output over MaxText: %v, want ErrTooMuchText", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The child is killed at the deadline and reaped: no process is left behind.
|
||||
func TestRunnerDeadlineKillsTheChild(t *testing.T) {
|
||||
pidfile := filepath.Join(t.TempDir(), "pid")
|
||||
r := runner("SIFT_PDFTEXT_SLEEP=30s", "SIFT_PDFTEXT_PIDFILE="+pidfile)
|
||||
r.Deadline = 300 * time.Millisecond
|
||||
start := time.Now()
|
||||
_, err := r.Run(context.Background(), three())
|
||||
if !errors.Is(err, pdftext.ErrTimeout) {
|
||||
t.Fatalf("err %v, want ErrTimeout", err)
|
||||
}
|
||||
if d := time.Since(start); d > 3*time.Second {
|
||||
t.Errorf("Run returned after %v; the 300 ms deadline was not enforced", d)
|
||||
}
|
||||
b, err := os.ReadFile(pidfile)
|
||||
if err != nil {
|
||||
t.Fatalf("child never wrote its pid: %v", err)
|
||||
}
|
||||
pid, _ := strconv.Atoi(string(b))
|
||||
if err := syscall.Kill(pid, 0); !errors.Is(err, syscall.ESRCH) {
|
||||
t.Errorf("child %d still exists after Run returned (kill -0: %v)", pid, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The caller's context cancels the child too.
|
||||
func TestRunnerContextCancel(t *testing.T) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err := runner("SIFT_PDFTEXT_SLEEP=30s").Run(ctx, three())
|
||||
if err == nil {
|
||||
t.Fatal("cancelled run succeeded")
|
||||
}
|
||||
}
|
||||
|
||||
// Runner passes only Env to the child: nothing of the parent's environment leaks in.
|
||||
func TestRunnerEnvIsExplicit(t *testing.T) {
|
||||
t.Setenv("SIFT_PDFTEXT_SLEEP", "30s") // in the parent only
|
||||
r := runner()
|
||||
r.Deadline = 5 * time.Second
|
||||
if _, err := r.Run(context.Background(), three()); err != nil {
|
||||
t.Errorf("parent env leaked into the child (it slept): %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
# PDF fixtures
|
||||
|
||||
Only the two encrypted files are binary; every other PDF the tests use is built at test time by
|
||||
`internal/pdftest`.
|
||||
|
||||
- `encrypted-user.pdf`: the `pdftest` three-page document, AES-256, user password `userpw`,
|
||||
owner password `ownerpw`. Made with
|
||||
`qpdf --encrypt userpw ownerpw 256 -- three.pdf encrypted-user.pdf`.
|
||||
- `encrypted-owner.pdf`: the same document, AES-256 with an empty user password
|
||||
(opens without one), owner password `ownerpw`:
|
||||
`qpdf --encrypt "" ownerpw 256 -- three.pdf encrypted-owner.pdf`.
|
||||
|
||||
Both contain the phrases "Chapter one begins here." (page 1) and
|
||||
"The quick brown fox is on page two." (page 2).
|
||||
Binary file not shown.
@@ -0,0 +1,81 @@
|
||||
%PDF-1.7
|
||||
%¿÷¢þ
|
||||
1 0 obj
|
||||
<< /Extensions << /ADBE << /BaseVersion /1.7 /ExtensionLevel 8 >> >> /Outlines 3 0 R /PageMode /UseOutlines /Pages 4 0 R /Type /Catalog >>
|
||||
endobj
|
||||
2 0 obj
|
||||
<< /Producer <43b5e97c0a7de15bc0f4dbcc052b5f80f21e0e3bede0eda33137092879ea433d> /Title <29f3ced7effab532ee871ab3ebc38a6869b6b7379f2fa12e139f3e2a1d1ee664> >>
|
||||
endobj
|
||||
3 0 obj
|
||||
<< /Count 3 /First 5 0 R /Last 6 0 R /Type /Outlines >>
|
||||
endobj
|
||||
4 0 obj
|
||||
<< /Count 3 /Kids [ 7 0 R 8 0 R 9 0 R ] /Type /Pages >>
|
||||
endobj
|
||||
5 0 obj
|
||||
<< /Dest [ 7 0 R /XYZ null null null ] /Next 10 0 R /Parent 3 0 R /Title <d2951c68344226cca87b8f66af448a05c4ba3c4baf7c7863293f2328ce407d09> >>
|
||||
endobj
|
||||
6 0 obj
|
||||
<< /Dest [ 9 0 R /XYZ null null null ] /Parent 3 0 R /Prev 10 0 R /Title <13b9a70c4f34e06e182eb823c2cb79a1f82c91f6c69ee89bf2093cd61d451407> >>
|
||||
endobj
|
||||
7 0 obj
|
||||
<< /Contents 11 0 R /MediaBox [ 0 0 612 792 ] /Parent 4 0 R /Resources << /Font << /F1 12 0 R >> /XObject << /Im1 13 0 R >> >> /Type /Page >>
|
||||
endobj
|
||||
8 0 obj
|
||||
<< /Contents 14 0 R /MediaBox [ 0 0 612 792 ] /Parent 4 0 R /Resources << /Font << /F1 12 0 R >> /XObject << /Im1 13 0 R >> >> /Type /Page >>
|
||||
endobj
|
||||
9 0 obj
|
||||
<< /Contents 15 0 R /MediaBox [ 0 0 612 792 ] /Parent 4 0 R /Resources << /Font << /F1 12 0 R >> /XObject << /Im1 13 0 R >> >> /Type /Page >>
|
||||
endobj
|
||||
10 0 obj
|
||||
<< /Dest [ 8 0 R /XYZ null null null ] /Next 6 0 R /Parent 3 0 R /Prev 5 0 R /Title <b734ddb0e84e981224c172818d645644e95ae16c94cf45d02ef870d181f97422> >>
|
||||
endobj
|
||||
11 0 obj
|
||||
<< /Length 128 /Filter /FlateDecode >>
|
||||
stream
|
||||
jl<ħ|Ýkj>Kþ"æ"-‚'agcYÓ“¡Áù ‹ä¾Îç„Â^›™h9 Àu¢ò4DY>€ä‡ÝhSkvJxŠý␍÷À'‡ÿù/טò£ Ð #¼Æo=ÖÛ:³1áÇ¢hÇôªÀè<Â|Hü“J‹xYz4�ÿÄendstream
|
||||
endobj
|
||||
12 0 obj
|
||||
<< /BaseFont /Helvetica /Encoding /WinAnsiEncoding /Subtype /Type1 /Type /Font >>
|
||||
endobj
|
||||
13 0 obj
|
||||
<< /BitsPerComponent 8 /ColorSpace /DeviceGray /Height 2 /Subtype /Image /Type /XObject /Width 2 /Length 32 /Filter /FlateDecode >>
|
||||
stream
|
||||
Ë™"€_œ¼Ë57Ó”Ì 60Ë%œ1´V\Ó$mG&0FåMendstream
|
||||
endobj
|
||||
14 0 obj
|
||||
<< /Length 112 /Filter /FlateDecode >>
|
||||
stream
|
||||
,#c¾}jè¹Ájw �)æ�„›d7Éwp¨íi»gvÍñÎâH»6oA•ï´vD}–Ð9·G`1wÆŽ×XŒÝ‰ž ÄrÜ›Vf:I¡)x9ÅéÌß“>��WãxŽÄ‡¡ã Š¤Ô«âÊ‘dendstream
|
||||
endobj
|
||||
15 0 obj
|
||||
<< /Length 160 /Filter /FlateDecode >>
|
||||
stream
|
||||
yx÷â°í%"(hß]�T~BùÞ]+$ík±Mû†e.ŒbÿPת®0ž;äòãæë%½Kû¾¤z‹C”¤² !̘£) (žœÛôŒáéÄ;/¬@yèºÚh¥Á qR|Ôcݹ²Ã((¼Õ{úc¿AÆeý ÏΧDíÚ¼)‡(íÄ! · #5u·ké#¹âW’<8pŽ÷I1XQ"¥G endstream
|
||||
endobj
|
||||
16 0 obj
|
||||
<< /CF << /StdCF << /AuthEvent /DocOpen /CFM /AESV3 /Length 32 >> >> /Filter /Standard /Length 256 /O <2a0820bbf93fff95a410525ae20d4abca153af1e3464706a4f83e49c8caa92e62e160b6871c75b9e47a6cd6dd15fe6ab> /OE <210b63a9a2305637af02dc6e89409285f61f82adaafa125e950042528164e2a3> /P -4 /Perms <23688a77e80aeb8b536511ce497c729e> /R 6 /StmF /StdCF /StrF /StdCF /U <98531c24f9d0bbabd16cc589be7618a59de91065126679a37b26b5d684cbcbc76d34bf68dc61eafe1681b4d5a21826d1> /UE <75ac70233a6a28bcd012febedaf3cccf6c25af218f3318a9fa9bc0f94a0e1b53> /V 5 >>
|
||||
endobj
|
||||
xref
|
||||
0 17
|
||||
0000000000 65535 f
|
||||
0000000015 00000 n
|
||||
0000000169 00000 n
|
||||
0000000341 00000 n
|
||||
0000000412 00000 n
|
||||
0000000483 00000 n
|
||||
0000000641 00000 n
|
||||
0000000799 00000 n
|
||||
0000000956 00000 n
|
||||
0000001113 00000 n
|
||||
0000001270 00000 n
|
||||
0000001440 00000 n
|
||||
0000001640 00000 n
|
||||
0000001738 00000 n
|
||||
0000001935 00000 n
|
||||
0000002119 00000 n
|
||||
0000002351 00000 n
|
||||
trailer << /Info 2 0 R /Root 1 0 R /Size 17 /ID [<39c444018179222865caf844f4fb7254><39c444018179222865caf844f4fb7254>] /Encrypt 16 0 R >>
|
||||
startxref
|
||||
2899
|
||||
%%EOF
|
||||
Reference in New Issue
Block a user