# sift

LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it back cleaned,
compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on
orion, and Honcho for retrieval.

Status: P1 clean fetch (`/v2/scrape`, `/v1/fetch`, CLI, auth). See [`docs/PLAN.md`](docs/PLAN.md) and [`docs/P1.md`](docs/P1.md).

## Build

Makefile is the source of truth for flags (`CGO_ENABLED=0`, `-trimpath`, `-s -w -X main.version=`).

```
make build          # bin/sift
make test           # go test -race ./...
make lint           # go vet, and staticcheck if installed
make install        # PREFIX=/usr/local; example config in /etc/sift/
nix build           # packages.sift, same ldflags
nix flake check     # make test lint
```

The service does not assume Nix store paths. Non-Nix hosts: `make install` plus `dist/sift.service`.

## PDF documents

A resource whose content type (or magic bytes) look like a PDF is extracted instead of cleaned.
Extraction runs in a **child process** — the `sift pdftext` subcommand of this binary, built on the
`go-pdfkit/extract` v0.3.0 engine, chosen by the bake-off in [`docs/P5a.md`](docs/P5a.md) §3 — so a
malicious PDF can't take down the server: the child is isolated (empty environment, native syscalls
only), killed at a deadline, and its output capped.

Limits: a PDF may be **100 MB** and take up to **120 s** to download; extraction is capped at
**60 s** and **2,000 pages**, with child output bounded to 64 MiB. Extraction runs under a **pool**
of **2 slots**, **4 waiting**, a **20 s** wait cap — a caller that can't get a slot gets `503 busy`,
and a caller with a job already in flight gets `429`. A PDF already extracted is answered from the
**cache** for **24 h**, then revalidated with a conditional GET (a `304` reuses the stored text).

Answers (through `/v1/fetch` or `/v2/scrape`):

- whole text when it fits the output cap, each page prefixed with `--- page N ---`;
- otherwise a **map** — the outline (with page numbers) and the first pages;
- `--pages 41-60` for a range, `--grep 'pattern'` for matching lines with page context;
- `--question "…"` for a **digest** that, on a PDF, cites each fact as `(p. N)`.

Scanned PDFs have no text layer and return `no text layer (scanned PDF?)`; point them at an OCR
source rather than fixing them here.

Errors (all `422` unless noted — a PDF never yields a `500`):

| cause | Status | Message | Hint | Retry-After |
|---|---|---|---|---|
| no text layer (`ErrNoText`) | 422 | `no text layer (scanned PDF?)` | `ocr` | — |
| encrypted (`ErrEncrypted`) | 422 | `encrypted PDF (password required)` | — | — |
| unreadable (`ErrUnreadable`) | 422 | `unreadable PDF` | — | — |
| timeout (`ErrTimeout`) | 422 | `pdf extraction timed out` | — | — |
| too much text (`ErrTooMuchText`) | 422 | `pdf text over the output limit` | — | — |
| pool full (`ErrBusy`) | 503 | `pdf workers busy` | — | 10 |
| caller busy (`ErrCallerBusy`) | 429 | `busy` | — | — |
| anything else (e.g. a recovered panic) | 422 | `unreadable PDF` | — | — |

Metrics (scraped at `http://127.0.0.1:8890/metrics`): `sift_pdf_extract_total{outcome}`
(`ok`, `no_text`, `encrypted`, `unreadable`, `timeout`, `too_large`, `busy`, or `other`),
`sift_pdf_extract_seconds`, `sift_pdf_pages_total`, `sift_pdf_pool_wait_seconds`, and
`sift_pdf_cache_hits_total`.