kyleandornith-1.5-35b-a3b
3a3a470cd7
`sift extract <file.pdf>` stats the file (refusing over 100 MiB before the upload), streams it as the POST body with Content-Type application/pdf and the ContentLength set, and prints the /v1/extract answer exactly as `sift fetch` prints a PDF URL — same content output, same errors and exitFor codes. parseExtractOpts reuses splitAroundPositional; extractURL sends only the set options with mode=digest when a question is given. main.go gains the case and a help line; README and skill document the command. Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
120 lines
6.4 KiB
Markdown
120 lines
6.4 KiB
Markdown
# sift
|
|
|
|
LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it back cleaned,
|
|
compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on
|
|
orion, and Honcho for retrieval.
|
|
|
|
Status: implemented through P5a: P1 clean fetch (`/v2/scrape`, `/v1/fetch`, CLI, auth), P2
|
|
digest, fleet context and recover, P3 search, P2.7 Prometheus metrics, P5a PDF documents. The design
|
|
is [`docs/PLAN.md`](docs/PLAN.md); each phase has its own doc (`docs/P1.md` … `docs/P5a.md`).
|
|
|
|
## API
|
|
|
|
All endpoints except `/healthz` take `Authorization: Bearer <key>` (per caller; see `sift keys`).
|
|
The server binds `127.0.0.1:8890` and is published on the tailnet by `tailscale serve` at
|
|
`https://orion.scylla-hammerhead.ts.net:8443`.
|
|
|
|
| Endpoint | What |
|
|
|---|---|
|
|
| `POST /v2/scrape` | Firecrawl-compatible fetch (`url`, `formats`) — what Hermes `web_extract` calls |
|
|
| `POST /v2/search` | Firecrawl-compatible search (`query`, `limit`) — what Hermes `web_search` calls |
|
|
| `POST /v1/fetch` | native fetch: `url`, `mode` (`clean`/`raw`/`digest`), `question`, `jq`, `context`, `recover`, and for PDFs `pages`, `grep` |
|
|
| `POST /v1/search` | native search: `query`, `question`, `k`, `rerank`, `digest` |
|
|
| `GET /metrics`, `/metrics.json` | Prometheus text or JSON; open on loopback, bearer elsewhere |
|
|
| `GET /healthz` | liveness |
|
|
|
|
Every outbound connection goes through the SSRF guard (`internal/guard`): private, tailnet and
|
|
configured `deny_prefixes` addresses, and the host's own WAN address (`[self_ip]`), are refused
|
|
with `blocked: …`.
|
|
|
|
## CLI
|
|
|
|
```
|
|
sift serve [-listen addr] [-config /etc/sift/config.toml] [-keys /var/lib/sift/keys.toml] [-deny-prefix CIDR]...
|
|
sift fetch <url> [--question q] [--mode clean|raw|digest] [--jq filter] [--pages a-b] [--grep re]
|
|
[--no-context] [--recover] [--json]
|
|
sift extract <file.pdf> [--pages a-b] [--grep re] [--question q] [--mode clean|digest] [--json]
|
|
sift search "<query>" [--k n] [--json]
|
|
sift keys add --caller name [--workspace ws] [--out file] | list | revoke --caller name
|
|
sift version
|
|
```
|
|
|
|
`sift fetch`, `sift extract` and `sift search` are clients: they read `url` and `key_file` from
|
|
`client.toml` (`$SIFT_CLIENT`, else `$HERMES_HOME/sift/client.toml`, else `~/.config/sift/client.toml`);
|
|
the key file must be mode 0600. `sift extract` uploads a local PDF to `/v1/extract` and prints the
|
|
answer exactly as `sift fetch` does for a PDF URL — `--pages`, `--grep` and `--question` behave the
|
|
same, and a file over 100 MiB is refused before the upload. The agent-facing guide is
|
|
[`skill/SKILL.md`](skill/SKILL.md).
|
|
|
|
## Configuration
|
|
|
|
`/etc/sift/config.toml` (`$SIFT_CONFIG`); every key is shown in
|
|
[`dist/sift.toml.example`](dist/sift.toml.example). Paid-search keys (`BRAVE_API_KEY`,
|
|
`TAVILY_API_KEY`) live on the box in `/var/lib/sift/secret.env` (`$SIFT_SECRET`), never in git or
|
|
the Nix store. On NixOS use `nixosModules.sift` (`services.sift`), which writes the config and
|
|
the hardened unit.
|
|
|
|
## Build
|
|
|
|
Makefile is the source of truth for flags (`CGO_ENABLED=0`, `-trimpath`, `-s -w -X main.version=`).
|
|
|
|
```
|
|
make build # bin/sift
|
|
make test # go test -race ./...
|
|
make lint # go vet, and staticcheck if installed
|
|
make install # PREFIX=/usr/local; example config in /etc/sift/
|
|
nix build # packages.sift, same ldflags
|
|
nix flake check # make test lint
|
|
```
|
|
|
|
The service does not assume Nix store paths. Non-Nix hosts: `make install` plus `dist/sift.service`.
|
|
|
|
## PDF documents
|
|
|
|
A resource whose content type (or magic bytes) look like a PDF is extracted instead of cleaned.
|
|
Extraction runs in a **child process** — the `sift pdftext` subcommand of this binary, built on the
|
|
`go-pdfkit/extract` v0.3.0 engine, chosen by the bake-off in [`docs/P5a.md`](docs/P5a.md) §3 — so a
|
|
malicious PDF can't take down the server: the child is isolated (empty environment, native syscalls
|
|
only), killed at a deadline, and its output capped.
|
|
|
|
Limits: a PDF may be **100 MB** and take up to **120 s** to download; extraction is capped at
|
|
**60 s** and **2,000 pages**, with child output bounded to 64 MiB. Extraction runs under a **pool**
|
|
of **2 slots**, **4 waiting**, a **20 s** wait cap — a caller that can't get a slot gets `503 busy`,
|
|
and a caller with a job already in flight gets `429`. A PDF already extracted is answered from the
|
|
**cache** for **24 h**, then revalidated with a conditional GET (a `304` reuses the stored text).
|
|
|
|
Answers (through `/v1/fetch` or `/v2/scrape`):
|
|
|
|
- whole text when it fits the output cap, each page prefixed with `--- page N ---`;
|
|
- otherwise a **map** — the outline (with page numbers) and the first pages;
|
|
- `--pages 41-60` for a range, `--grep 'pattern'` for matching lines with page context;
|
|
- `--question "…"` for a **digest** that, on a PDF, cites each fact as `(p. N)`.
|
|
|
|
A range returns at most 100 pages; grep shows at most 100 matches and takes a pattern of at most
|
|
200 bytes. `pages` and `grep` on anything but a PDF are a `400`. A question on a long document is
|
|
answered first from BM25 (`retrieval: "bm25"`) — lexical, before the document's chunks are
|
|
embedded in the background — and hybrid on later questions, once those embeddings exist. A question
|
|
over a document too long to answer even by retrieval (more than 5000 chunks) is refused with
|
|
`422 document too long for question (N chunks); use grep or pages`.
|
|
|
|
Scanned PDFs have no text layer and return `no text layer (scanned PDF?)`; point them at an OCR
|
|
source rather than fixing them here.
|
|
|
|
Errors (all `422` unless noted — a PDF never yields a `500`):
|
|
|
|
| cause | Status | Message | Hint | Retry-After |
|
|
|---|---|---|---|---|
|
|
| no text layer (`ErrNoText`) | 422 | `no text layer (scanned PDF?)` | `ocr` | — |
|
|
| encrypted (`ErrEncrypted`) | 422 | `encrypted PDF (password required)` | — | — |
|
|
| unreadable (`ErrUnreadable`) | 422 | `unreadable PDF` | — | — |
|
|
| timeout (`ErrTimeout`) | 422 | `pdf extraction timed out` | — | — |
|
|
| too much text (`ErrTooMuchText`) | 422 | `pdf text over the output limit` | — | — |
|
|
| pool full (`ErrBusy`) | 503 | `pdf workers busy` | — | 10 |
|
|
| caller busy (`ErrCallerBusy`) | 429 | `busy` | — | — |
|
|
| anything else (e.g. a recovered panic) | 422 | `unreadable PDF` | — | — |
|
|
|
|
Metrics (scraped at `http://127.0.0.1:8890/metrics`): `sift_pdf_extract_total{outcome}`
|
|
(`ok`, `no_text`, `encrypted`, `unreadable`, `timeout`, `too_large`, `busy`, or `other`),
|
|
`sift_pdf_extract_seconds`, `sift_pdf_pages_total`, `sift_pdf_pool_wait_seconds`, and
|
|
`sift_pdf_cache_hits_total`.
|