kyle
682b9b8044
S1: skill recipe now says result score is merged rank. Also send non-zero --min-score to the server (N1), sort include_domains (N2), and note that search_queries counts each sub-query (N3).
158 lines
9.0 KiB
Markdown
158 lines
9.0 KiB
Markdown
# sift
|
||
|
||
LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it back cleaned,
|
||
compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on
|
||
orion, and Honcho for retrieval.
|
||
|
||
Status: implemented through P5a: P1 clean fetch (`/v2/scrape`, `/v1/fetch`, CLI, auth), P2
|
||
digest, fleet context and recover, P3 search, P2.7 Prometheus metrics, P5a PDF documents. The design
|
||
is [`docs/PLAN.md`](docs/PLAN.md); each phase has its own doc (`docs/P1.md` … `docs/P5a.md`).
|
||
|
||
## API
|
||
|
||
All endpoints except `/healthz` take `Authorization: Bearer <key>` (per caller; see `sift keys`).
|
||
The server binds `127.0.0.1:8890` and is published on the tailnet by `tailscale serve` at
|
||
`https://orion.scylla-hammerhead.ts.net:8443`.
|
||
|
||
| Endpoint | What |
|
||
|---|---|
|
||
| `POST /v2/scrape` | Firecrawl-compatible fetch (`url`, `formats`) — what Hermes `web_extract` calls |
|
||
| `POST /v2/search` | Firecrawl-compatible search (`query`, `limit`) — what Hermes `web_search` calls |
|
||
| `POST /v1/fetch` | native fetch: `url`, `mode` (`clean`/`raw`/`digest`), `question`, `jq`, `context`, `recover`, and for PDFs `pages`, `grep` |
|
||
| `POST /v1/search` | native search: `query` or `queries`, `k`, `rerank`, `digest`, domain/time/topic/`exact_match`/`min_score` |
|
||
| `GET /metrics`, `/metrics.json` | Prometheus text or JSON; open on loopback, bearer elsewhere |
|
||
| `GET /healthz` | liveness |
|
||
|
||
Every outbound connection goes through the SSRF guard (`internal/guard`): private, tailnet and
|
||
configured `deny_prefixes` addresses, and the host's own WAN address (`[self_ip]`), are refused
|
||
with `blocked: …`.
|
||
|
||
## CLI
|
||
|
||
```
|
||
sift serve [-listen addr] [-config /etc/sift/config.toml] [-keys /var/lib/sift/keys.toml] [-deny-prefix CIDR]...
|
||
sift fetch <url> [--question q] [--mode clean|raw|digest] [--jq filter] [--pages a-b] [--grep re]
|
||
[--no-context] [--recover] [--json]
|
||
sift extract <file.pdf> [--pages a-b] [--grep re] [--question q] [--mode clean|digest] [--json]
|
||
sift search "<query>" [--k n] [--json] [--include-domain host]... [--topic news] [--time-range month]
|
||
sift batch submit|status|get|cancel
|
||
sift keys add --caller name [--workspace ws] [--out file] | list | revoke --caller name
|
||
sift version
|
||
```
|
||
|
||
`sift fetch`, `sift extract` and `sift search` are clients: they read `url` and `key_file` from
|
||
`client.toml` (`$SIFT_CLIENT`, else `$HERMES_HOME/sift/client.toml`, else `~/.config/sift/client.toml`);
|
||
the key file must be mode 0600. `sift extract` uploads a local PDF to `/v1/extract` and prints the
|
||
answer exactly as `sift fetch` does for a PDF URL — `--pages`, `--grep` and `--question` behave the
|
||
same, and a file over 100 MiB is refused before the upload. The agent-facing guide is
|
||
[`skill/SKILL.md`](skill/SKILL.md). JSON `search_queries` (and `sift_search_queries_*`) counts
|
||
each sub-query as its own upstream call, so a 4-query search is 4× one `query`.
|
||
|
||
## Batch
|
||
|
||
`sift batch` runs several fetches as one job, so an agent can read a handful of URLs later instead
|
||
of a loop of `sift fetch`. The four endpoints:
|
||
|
||
| Endpoint | What |
|
||
|---|---|
|
||
| `POST /v1/batch` | `{"targets":[…]}` → `202 {"success","job_id","status","targets","reused"}` |
|
||
| `GET /v1/batch/{id}` | summary; `?results=1` inlines each done target's `/v1/fetch` envelope |
|
||
| `GET /v1/batch/{id}/{idx}` | one target's result, as `sift fetch --json` |
|
||
| `DELETE /v1/batch/{id}` | cancel a queued job → `200 {"success","status"}` |
|
||
|
||
`sift batch submit <targets.json|->` reads a JSON object with a non-empty `targets` array locally,
|
||
or a bare array of targets or URL strings (each string becomes `{"url": s}`), then posts it. `status`
|
||
prints the summary (or the body with `?results=1`); `get <job> <idx>` prints one result; `cancel`
|
||
stops a queued job.
|
||
|
||
A job runs **serially** (target `i+1` never starts before `i` is stored) with **one worker by
|
||
default** (see `[batch]` config) and the runner rotates jobs **round-robin** while several are live.
|
||
Caps: **32 targets** per job, **1 live job per caller**, **8 live jobs** in all. The posted body is
|
||
capped at **256 KiB**. A target that is blocked, times out, or errors is recorded and the job
|
||
continues; a job whose targets are all done is `done`, all failed `failed`, mixed `partial`.
|
||
|
||
**Idempotency**: the same list from the same caller returns the same job while it is live or already
|
||
has results (`reused: true`), so a retry does not start a second job. Each job carries a **lease**
|
||
that a status read renews; a job whose lease has expired for more than the **24 h TTL** is swept
|
||
along with its targets. On restart, jobs in flight when the server died are marked failed so a retry
|
||
re-submits them.
|
||
|
||
A `status` read is a **summary** (per-target status, never the content); `?results=1` inlines the
|
||
full envelope for each done target, so it can be large — prefer `get <job> <idx>` for one result.
|
||
Restricted callers see the same endpoints; their results are **redacted** the same way a restricted
|
||
`sift fetch` is. Batch counters are `sift_batch_jobs_total{outcome}`,
|
||
`sift_batch_targets_total{outcome}`, `sift_batch_queue_depth`, `sift_batch_targets_inflight`,
|
||
`sift_batch_job_seconds`, `sift_batch_target_seconds` and `sift_batch_result_bytes`.
|
||
|
||
## Configuration
|
||
|
||
`/etc/sift/config.toml` (`$SIFT_CONFIG`); every key is shown in
|
||
[`dist/sift.toml.example`](dist/sift.toml.example). Paid-search keys (`BRAVE_API_KEY`,
|
||
`TAVILY_API_KEY`) live on the box in `/var/lib/sift/secret.env` (`$SIFT_SECRET`), never in git or
|
||
the Nix store. On NixOS use `nixosModules.sift` (`services.sift`), which writes the config and
|
||
the hardened unit.
|
||
|
||
## Build
|
||
|
||
Makefile is the source of truth for flags (`CGO_ENABLED=0`, `-trimpath`, `-s -w -X main.version=`).
|
||
|
||
```
|
||
make build # bin/sift
|
||
make test # go test -race ./...
|
||
make lint # go vet, and staticcheck if installed
|
||
make install # PREFIX=/usr/local; example config in /etc/sift/
|
||
nix build # packages.sift, same ldflags
|
||
nix flake check # make test lint
|
||
```
|
||
|
||
The service does not assume Nix store paths. Non-Nix hosts: `make install` plus `dist/sift.service`.
|
||
|
||
## PDF documents
|
||
|
||
A resource whose content type (or magic bytes) look like a PDF is extracted instead of cleaned.
|
||
Extraction runs in a **child process** — the `sift pdftext` subcommand of this binary, built on the
|
||
`go-pdfkit/extract` v0.3.0 engine, chosen by the bake-off in [`docs/P5a.md`](docs/P5a.md) §3 — so a
|
||
malicious PDF can't take down the server: the child is isolated (empty environment, native syscalls
|
||
only), killed at a deadline, and its output capped.
|
||
|
||
Limits: a PDF may be **100 MB** and take up to **120 s** to download; extraction is capped at
|
||
**60 s** and **2,000 pages**, with child output bounded to 64 MiB. Extraction runs under a **pool**
|
||
of **2 slots**, **4 waiting**, a **20 s** wait cap — a caller that can't get a slot gets `503 busy`,
|
||
and a caller with a job already in flight gets `429`. A PDF already extracted is answered from the
|
||
**cache** for **24 h**, then revalidated with a conditional GET (a `304` reuses the stored text).
|
||
|
||
Answers (through `/v1/fetch` or `/v2/scrape`):
|
||
|
||
- whole text when it fits the output cap, each page prefixed with `--- page N ---`;
|
||
- otherwise a **map** — the outline (with page numbers) and the first pages;
|
||
- `--pages 41-60` for a range, `--grep 'pattern'` for matching lines with page context;
|
||
- `--question "…"` for a **digest** that, on a PDF, cites each fact as `(p. N)`.
|
||
|
||
A range returns at most 100 pages; grep shows at most 100 matches and takes a pattern of at most
|
||
200 bytes. `pages` and `grep` on anything but a PDF are a `400`. A question on a long document is
|
||
answered first from BM25 (`retrieval: "bm25"`) — lexical, before the document's chunks are
|
||
embedded in the background — and hybrid on later questions, once those embeddings exist. A question
|
||
over a document too long to answer even by retrieval (more than 5000 chunks) is refused with
|
||
`422 document too long for question (N chunks); use grep or pages`.
|
||
|
||
Scanned PDFs have no text layer and return `no text layer (scanned PDF?)`; point them at an OCR
|
||
source rather than fixing them here.
|
||
|
||
Errors (all `422` unless noted — a PDF never yields a `500`):
|
||
|
||
| cause | Status | Message | Hint | Retry-After |
|
||
|---|---|---|---|---|
|
||
| no text layer (`ErrNoText`) | 422 | `no text layer (scanned PDF?)` | `ocr` | — |
|
||
| encrypted (`ErrEncrypted`) | 422 | `encrypted PDF (password required)` | — | — |
|
||
| unreadable (`ErrUnreadable`) | 422 | `unreadable PDF` | — | — |
|
||
| timeout (`ErrTimeout`) | 422 | `pdf extraction timed out` | — | — |
|
||
| too much text (`ErrTooMuchText`) | 422 | `pdf text over the output limit` | — | — |
|
||
| pool full (`ErrBusy`) | 503 | `pdf workers busy` | — | 10 |
|
||
| caller busy (`ErrCallerBusy`) | 429 | `busy` | — | — |
|
||
| anything else (e.g. a recovered panic) | 422 | `unreadable PDF` | — | — |
|
||
|
||
Metrics (scraped at `http://127.0.0.1:8890/metrics`): `sift_pdf_extract_total{outcome}`
|
||
(`ok`, `no_text`, `encrypted`, `unreadable`, `timeout`, `too_large`, `busy`, or `other`),
|
||
`sift_pdf_extract_seconds`, `sift_pdf_pages_total`, `sift_pdf_pool_wait_seconds`, and
|
||
`sift_pdf_cache_hits_total`.
|