sift

LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it back cleaned, compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on orion, and Honcho for retrieval.

Status: implemented through P5a: P1 clean fetch (/v2/scrape, /v1/fetch, CLI, auth), P2 digest, fleet context and recover, P3 search, P2.7 Prometheus metrics, P5a PDF documents. The design is docs/PLAN.md; each phase has its own doc (docs/P1.md … docs/P5a.md).

API

All endpoints except /healthz take Authorization: Bearer <key> (per caller; see sift keys). The server binds 127.0.0.1:8890 and is published on the tailnet by tailscale serve at https://orion.scylla-hammerhead.ts.net:8443.

Endpoint	What
POST /v2/scrape	Firecrawl-compatible fetch (url, formats) — what Hermes web_extract calls
POST /v2/search	Firecrawl-compatible search (query, limit) — what Hermes web_search calls
POST /v1/fetch	native fetch: url, mode (clean/raw/digest), question, jq, context, recover, and for PDFs pages, grep
POST /v1/search	native search: query or queries, k, rerank, digest, domain/time/topic/exact_match/min_score
GET /metrics, /metrics.json	Prometheus text or JSON; open on loopback, bearer elsewhere
GET /healthz	liveness

Every outbound connection goes through the SSRF guard (internal/guard): private, tailnet and configured deny_prefixes addresses, and the host's own WAN address ([self_ip]), are refused with blocked: ….

CLI

sift serve  [-listen addr] [-config /etc/sift/config.toml] [-keys /var/lib/sift/keys.toml] [-deny-prefix CIDR]...
sift fetch  <url> [--question q] [--mode clean|raw|digest] [--jq filter] [--pages a-b] [--grep re]
                  [--no-context] [--recover] [--json]
sift extract <file.pdf> [--pages a-b] [--grep re] [--question q] [--mode clean|digest] [--json]
  sift search   "<query>" [--k n] [--json] [--include-domain host]... [--topic news] [--time-range month]
  sift batch    submit|status|get|cancel
  sift keys   add --caller name [--workspace ws] [--out file] | list | revoke --caller name
  sift version

sift fetch, sift extract and sift search are clients: they read url and key_file from client.toml ($SIFT_CLIENT, else $HERMES_HOME/sift/client.toml, else ~/.config/sift/client.toml); the key file must be mode 0600. sift extract uploads a local PDF to /v1/extract and prints the answer exactly as sift fetch does for a PDF URL — --pages, --grep and --question behave the same, and a file over 100 MiB is refused before the upload. The agent-facing guide is skill/SKILL.md. JSON search_queries (and sift_search_queries_*) counts each sub-query as its own upstream call, so a 4-query search is 4× one query.

Batch

sift batch runs several fetches as one job, so an agent can read a handful of URLs later instead of a loop of sift fetch. The four endpoints:

Endpoint	What
POST /v1/batch	{"targets":[…]} → 202 {"success","job_id","status","targets","reused"}
GET /v1/batch/{id}	summary; ?results=1 inlines each done target's /v1/fetch envelope
GET /v1/batch/{id}/{idx}	one target's result, as sift fetch --json
DELETE /v1/batch/{id}	cancel a queued job → 200 {"success","status"}

sift batch submit <targets.json|-> reads a JSON object with a non-empty targets array locally, or a bare array of targets or URL strings (each string becomes {"url": s}), then posts it. status prints the summary (or the body with ?results=1); get <job> <idx> prints one result; cancel stops a queued job.

A job runs serially (target i+1 never starts before i is stored) with one worker by default (see [batch] config) and the runner rotates jobs round-robin while several are live. Caps: 32 targets per job, 1 live job per caller, 8 live jobs in all. The posted body is capped at 256 KiB. A target that is blocked, times out, or errors is recorded and the job continues; a job whose targets are all done is done, all failed failed, mixed partial.

Idempotency: the same list from the same caller returns the same job while it is live or already has results (reused: true), so a retry does not start a second job. Each job carries a lease that a status read renews; a job whose lease has expired for more than the 24 h TTL is swept along with its targets. On restart, jobs in flight when the server died are marked failed so a retry re-submits them.

A status read is a summary (per-target status, never the content); ?results=1 inlines the full envelope for each done target, so it can be large — prefer get <job> <idx> for one result. Restricted callers see the same endpoints; their results are redacted the same way a restricted sift fetch is. Batch counters are sift_batch_jobs_total{outcome}, sift_batch_targets_total{outcome}, sift_batch_queue_depth, sift_batch_targets_inflight, sift_batch_job_seconds, sift_batch_target_seconds and sift_batch_result_bytes.

Configuration

/etc/sift/config.toml ($SIFT_CONFIG); every key is shown in dist/sift.toml.example. Paid-search keys (BRAVE_API_KEY, TAVILY_API_KEY) live on the box in /var/lib/sift/secret.env ($SIFT_SECRET), never in git or the Nix store. On NixOS use nixosModules.sift (services.sift), which writes the config and the hardened unit.

Build

Makefile is the source of truth for flags (CGO_ENABLED=0, -trimpath, -s -w -X main.version=).

make build          # bin/sift
make test           # go test -race ./...
make lint           # go vet, and staticcheck if installed
make install        # PREFIX=/usr/local; example config in /etc/sift/
nix build           # packages.sift, same ldflags
nix flake check     # make test lint

The service does not assume Nix store paths. Non-Nix hosts: make install plus dist/sift.service.

PDF documents

A resource whose content type (or magic bytes) look like a PDF is extracted instead of cleaned. Extraction runs in a child process — the sift pdftext subcommand of this binary, built on the go-pdfkit/extract v0.3.0 engine, chosen by the bake-off in docs/P5a.md §3 — so a malicious PDF can't take down the server: the child is isolated (empty environment, native syscalls only), killed at a deadline, and its output capped.

Limits: a PDF may be 100 MB and take up to 120 s to download; extraction is capped at 60 s and 2,000 pages, with child output bounded to 64 MiB. Extraction runs under a pool of 2 slots, 4 waiting, a 20 s wait cap — a caller that can't get a slot gets 503 busy, and a caller with a job already in flight gets 429. A PDF already extracted is answered from the cache for 24 h, then revalidated with a conditional GET (a 304 reuses the stored text).

Answers (through /v1/fetch or /v2/scrape):

• whole text when it fits the output cap, each page prefixed with --- page N ---;
• otherwise a map — the outline (with page numbers) and the first pages;
• --pages 41-60 for a range, --grep 'pattern' for matching lines with page context;
• --question "…" for a digest that, on a PDF, cites each fact as (p. N).

A range returns at most 100 pages; grep shows at most 100 matches and takes a pattern of at most 200 bytes. pages and grep on anything but a PDF are a 400. A question on a long document is answered first from BM25 (retrieval: "bm25") — lexical, before the document's chunks are embedded in the background — and hybrid on later questions, once those embeddings exist. A question over a document too long to answer even by retrieval (more than 5000 chunks) is refused with 422 document too long for question (N chunks); use grep or pages.

Scanned PDFs have no text layer and return no text layer (scanned PDF?); point them at an OCR source rather than fixing them here.

Errors (all 422 unless noted — a PDF never yields a 500):

cause	Status	Message	Hint	Retry-After
no text layer (ErrNoText)	422	no text layer (scanned PDF?)	ocr	—
encrypted (ErrEncrypted)	422	encrypted PDF (password required)	—	—
unreadable (ErrUnreadable)	422	unreadable PDF	—	—
timeout (ErrTimeout)	422	pdf extraction timed out	—	—
too much text (ErrTooMuchText)	422	pdf text over the output limit	—	—
pool full (ErrBusy)	503	pdf workers busy	—	10
caller busy (ErrCallerBusy)	429	busy	—	—
anything else (e.g. a recovered panic)	422	unreadable PDF	—	—

Metrics (scraped at http://127.0.0.1:8890/metrics): sift_pdf_extract_total{outcome} (ok, no_text, encrypted, unreadable, timeout, too_large, busy, or other), sift_pdf_extract_seconds, sift_pdf_pages_total, sift_pdf_pool_wait_seconds, and sift_pdf_cache_hits_total.