docs: bring README, skill and dist up to date through P5a

README gains API, CLI and configuration sections and the PDF range/grep/question limits; the skill covers PDF busy and failure results; dist/sift.service matches the Nix unit (MemoryMax=1G, secret.env); the example config drops the unused [upstreams] block; stale task comments in api/pdf.go.

Gate overridden by the owner: TestGrepLimits (given test) exceeds its 1 s budget under -race on this host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-26 10:21:51 -07:00
co-authored by Claude Opus 5.5
parent 01c8d9daeb
commit c89281200a
5 changed files with 64 additions and 11 deletions
+50 -1
View File
@@ -4,7 +4,51 @@ LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it
compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on
orion, and Honcho for retrieval.
Status: P1 clean fetch (`/v2/scrape`, `/v1/fetch`, CLI, auth). See [`docs/PLAN.md`](docs/PLAN.md) and [`docs/P1.md`](docs/P1.md).
Status: implemented through P5a: P1 clean fetch (`/v2/scrape`, `/v1/fetch`, CLI, auth), P2
digest, fleet context and recover, P3 search, P2.7 Prometheus metrics, P5a PDF documents. The design
is [`docs/PLAN.md`](docs/PLAN.md); each phase has its own doc (`docs/P1.md` … `docs/P5a.md`).
## API
All endpoints except `/healthz` take `Authorization: Bearer <key>` (per caller; see `sift keys`).
The server binds `127.0.0.1:8890` and is published on the tailnet by `tailscale serve` at
`https://orion.scylla-hammerhead.ts.net:8443`.
| Endpoint | What |
|---|---|
| `POST /v2/scrape` | Firecrawl-compatible fetch (`url`, `formats`) — what Hermes `web_extract` calls |
| `POST /v2/search` | Firecrawl-compatible search (`query`, `limit`) — what Hermes `web_search` calls |
| `POST /v1/fetch` | native fetch: `url`, `mode` (`clean`/`raw`/`digest`), `question`, `jq`, `context`, `recover`, and for PDFs `pages`, `grep` |
| `POST /v1/search` | native search: `query`, `question`, `k`, `rerank`, `digest` |
| `GET /metrics`, `/metrics.json` | Prometheus text or JSON; open on loopback, bearer elsewhere |
| `GET /healthz` | liveness |
Every outbound connection goes through the SSRF guard (`internal/guard`): private, tailnet and
configured `deny_prefixes` addresses, and the host's own WAN address (`[self_ip]`), are refused
with `blocked: …`.
## CLI
```
sift serve [-listen addr] [-config /etc/sift/config.toml] [-keys /var/lib/sift/keys.toml] [-deny-prefix CIDR]...
sift fetch <url> [--question q] [--mode clean|raw|digest] [--jq filter] [--pages a-b] [--grep re]
[--no-context] [--recover] [--json]
sift search "<query>" [--k n] [--json]
sift keys add --caller name [--workspace ws] [--out file] | list | revoke --caller name
sift version
```
`sift fetch` and `sift search` are clients: they read `url` and `key_file` from `client.toml`
(`$SIFT_CLIENT`, else `$HERMES_HOME/sift/client.toml`, else `~/.config/sift/client.toml`); the key
file must be mode 0600. The agent-facing guide is [`skill/SKILL.md`](skill/SKILL.md).
## Configuration
`/etc/sift/config.toml` (`$SIFT_CONFIG`); every key is shown in
[`dist/sift.toml.example`](dist/sift.toml.example). Paid-search keys (`BRAVE_API_KEY`,
`TAVILY_API_KEY`) live on the box in `/var/lib/sift/secret.env` (`$SIFT_SECRET`), never in git or
the Nix store. On NixOS use `nixosModules.sift` (`services.sift`), which writes the config and
the hardened unit.
## Build
@@ -42,6 +86,11 @@ Answers (through `/v1/fetch` or `/v2/scrape`):
- `--pages 41-60` for a range, `--grep 'pattern'` for matching lines with page context;
- `--question "…"` for a **digest** that, on a PDF, cites each fact as `(p. N)`.
A range returns at most 100 pages; grep shows at most 100 matches and takes a pattern of at most
200 bytes. `pages` and `grep` on anything but a PDF are a `400`. A question over a document too long
to digest (more than 200 chunks) is refused with
`422 document too long for question (N chunks); use grep or pages`.
Scanned PDFs have no text layer and return `no text layer (scanned PDF?)`; point them at an OCR
source rather than fixing them here.