kyleandClaude Opus 5.5
aafc1d1c28
- pdftext/outline: resolve named destinations through the catalog /Dests and the /Names /Dests tree, resolve indirect /Dest, /A and /Title, and treat reader.Null (what Dict.Get returns for a missing key) as absent. Before this every LaTeX and Acrobat PDF had an empty outline. pdftest gains NamedDests; TestOutlineNamedDestinations. - api: pool() always goes through its Once (data race on concurrent first use); TestDefaultPoolConcurrentFirstUse. Revalidation of a stale cache entry counts against the caller/host limit. - pdftext/runner: output over MaxText kills the child at once instead of waiting for the deadline; TestRunnerTooMuchTextFailsFast. - fetch: Accept includes application/pdf;q=0.8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sift
LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it back cleaned, compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on orion, and Honcho for retrieval.
Status: P1 clean fetch (/v2/scrape, /v1/fetch, CLI, auth). See docs/PLAN.md and docs/P1.md.
Build
Makefile is the source of truth for flags (CGO_ENABLED=0, -trimpath, -s -w -X main.version=).
make build # bin/sift
make test # go test -race ./...
make lint # go vet, and staticcheck if installed
make install # PREFIX=/usr/local; example config in /etc/sift/
nix build # packages.sift, same ldflags
nix flake check # make test lint
The service does not assume Nix store paths. Non-Nix hosts: make install plus dist/sift.service.
PDF documents
A resource whose content type (or magic bytes) look like a PDF is extracted instead of cleaned.
Extraction runs in a child process — the sift pdftext subcommand of this binary, built on the
go-pdfkit/extract v0.3.0 engine, chosen by the bake-off in docs/P5a.md §3 — so a
malicious PDF can't take down the server: the child is isolated (empty environment, native syscalls
only), killed at a deadline, and its output capped.
Limits: a PDF may be 100 MB and take up to 120 s to download; extraction is capped at
60 s and 2,000 pages, with child output bounded to 64 MiB. Extraction runs under a pool
of 2 slots, 4 waiting, a 20 s wait cap — a caller that can't get a slot gets 503 busy,
and a caller with a job already in flight gets 429. A PDF already extracted is answered from the
cache for 24 h, then revalidated with a conditional GET (a 304 reuses the stored text).
Answers (through /v1/fetch or /v2/scrape):
- whole text when it fits the output cap, each page prefixed with
--- page N ---; - otherwise a map — the outline (with page numbers) and the first pages;
--pages 41-60for a range,--grep 'pattern'for matching lines with page context;--question "…"for a digest that, on a PDF, cites each fact as(p. N).
Scanned PDFs have no text layer and return no text layer (scanned PDF?); point them at an OCR
source rather than fixing them here.
Errors (all 422 unless noted — a PDF never yields a 500):
| cause | Status | Message | Hint | Retry-After |
|---|---|---|---|---|
no text layer (ErrNoText) |
422 | no text layer (scanned PDF?) |
ocr |
— |
encrypted (ErrEncrypted) |
422 | encrypted PDF (password required) |
— | — |
unreadable (ErrUnreadable) |
422 | unreadable PDF |
— | — |
timeout (ErrTimeout) |
422 | pdf extraction timed out |
— | — |
too much text (ErrTooMuchText) |
422 | pdf text over the output limit |
— | — |
pool full (ErrBusy) |
503 | pdf workers busy |
— | 10 |
caller busy (ErrCallerBusy) |
429 | busy |
— | — |
| anything else (e.g. a recovered panic) | 422 | unreadable PDF |
— | — |
Metrics (scraped at http://127.0.0.1:8890/metrics): sift_pdf_extract_total{outcome}
(ok, no_text, encrypted, unreadable, timeout, too_large, busy, or other),
sift_pdf_extract_seconds, sift_pdf_pages_total, sift_pdf_pool_wait_seconds, and
sift_pdf_cache_hits_total.