docs: P1 spec — clean fetch, auth, CLI, contract facts from firecrawl-py 4.17.0, talos wiring
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+195
@@ -0,0 +1,195 @@
|
||||
# Sift P1 — clean fetch, auth, CLI client, first host
|
||||
|
||||
Drafted 2026-09-22 by Claude. Implementer: straylight. Reviewer: Claude (contract + guard
|
||||
wiring). Orion deploy + talos wiring: talos. Builds on P0 (`099b949`, guard gate passed).
|
||||
|
||||
## 1. Scope
|
||||
|
||||
In: `POST /v2/scrape` (Firecrawl-compatible), `POST /v1/fetch` (modes `clean`/`raw`, `jq`),
|
||||
per-caller bearer auth, limits, structured errors, basic `/metrics`, the `sift` CLI client
|
||||
(`fetch`, `keys`), `skill/SKILL.md` installed on talos, orion deployment, talos wired.
|
||||
|
||||
Out (later phases): digest/`question`/Honcho context (P2), `--recover` (P2), `search` (P3),
|
||||
cache (P4). `/v1/fetch` must reject `mode=digest`, `question`, `context` with a clear
|
||||
`400 not implemented until P2`.
|
||||
|
||||
**Exit criteria** (post the evidence in the ~dev thread):
|
||||
1. `make test lint contract` green; `nix flake check` green.
|
||||
2. talos: a Hermes `web_extract` of a public HTML page and of a JSON API URL both return
|
||||
cleaned content via sift (show the orion log line with caller=`talos`).
|
||||
3. talos: `sift fetch http://100.95.252.120:8000/` and `sift fetch http://dixie:11434/` and a
|
||||
redirect-to-private URL are refused with `blocked: …`, from talos, against the live
|
||||
service.
|
||||
4. talos: `sift fetch <hf-api-url> --jq '<filter>'` works, and `tirith check` on that exact
|
||||
command returns `allow`.
|
||||
5. A `-deny-prefix` address is refused **end to end through `/v2/scrape`** (P0 carry-over).
|
||||
|
||||
## 2. Contract facts (verified 2026-09-22, not assumed)
|
||||
|
||||
Both fleet Hermes versions (talos 0.19.0, straylight 0.21.x) call self-hosted Firecrawl
|
||||
through **`firecrawl-py==4.17.0`**, lazily installed by Hermes (`tools/lazy_deps.py`,
|
||||
`search.firecrawl`) on first use. It is **not installed on any host yet** — the first
|
||||
`web_extract` after wiring triggers a pip install into Hermes's env. Pre-install it
|
||||
during wiring (`uv pip install --python <hermes python> firecrawl-py==4.17.0`) so the
|
||||
first call isn't a surprise network install.
|
||||
|
||||
**Request** (`firecrawl/v2/methods/scrape.py`), exactly what sift will receive:
|
||||
```json
|
||||
{"url": "https://example.com/x", "onlyMainContent": true, "skipTlsVerification": true,
|
||||
"removeBase64Images": true, "fastMode": false, "blockAds": true, "storeInCache": true,
|
||||
"maxAge": 14400000, "formats": ["markdown", "html"], "mobile": false,
|
||||
"origin": "python-sdk@4.17.0"}
|
||||
```
|
||||
- Decode leniently: ignore unknown fields.
|
||||
- **`skipTlsVerification: true` is sent by default and MUST be ignored.** Sift never
|
||||
disables TLS verification because a request asked it to. Add a test that proves a
|
||||
bad-cert upstream fails even with that field set.
|
||||
- Honour `formats` (`markdown`, `html`) and `onlyMainContent` (true → readability article;
|
||||
false → whole body converted). Ignore `mobile`, `blockAds`, `fastMode`,
|
||||
`removeBase64Images` (drop `data:` images always), `storeInCache`/`maxAge` (P4 cache hint).
|
||||
|
||||
**Auth**: SDK sends `Authorization: Bearer <FIRECRAWL_API_KEY>` when the key is set.
|
||||
|
||||
**Response**: SDK requires HTTP 2xx **and** `body.success == true`, then builds
|
||||
`Document(markdown, html, metadata)`; metadata keys are camelCase → snake_case
|
||||
normalised (`sourceURL` → `source_url`), extras allowed.
|
||||
|
||||
**Known limitation (0.19 and 0.21 SDK path)**: Hermes reads `metadata.get("sourceURL", url)`
|
||||
*after* the SDK renamed it to `source_url`, so Hermes cites the **requested** URL, not the
|
||||
final URL after redirects, and its own post-redirect SSRF re-check sees the original URL.
|
||||
Sift's server-side guard is unaffected. Document it in the skill; the native CLI returns
|
||||
`final_url` correctly.
|
||||
|
||||
**Errors → SDK behaviour** (`firecrawl/v2/utils/error_handler.py`): non-2xx raises with
|
||||
`body.error` in the message; **the SDK retries only on 502**. Therefore:
|
||||
|
||||
| Situation | HTTP | `error` |
|
||||
|---|---|---|
|
||||
| guard refusal | 403 | `blocked: private address 10.0.0.5` (surfaces as "Website Not Supported: … blocked: …") |
|
||||
| bad request / invalid URL / unsupported option | 400 | `invalid url` / `not implemented until P2` |
|
||||
| missing/unknown key | 401 | `unauthorized` |
|
||||
| upstream 403/429/451/503-with-challenge | 422 | `upstream 403 (likely bot-blocked); hint: browser` |
|
||||
| other upstream 4xx/5xx | 422 | `upstream 404` |
|
||||
| upstream timeout | 504 | `upstream timeout after 20s` |
|
||||
| body over cap | 413 | `too large (>5 MB decompressed)` |
|
||||
| unsupported content type | 415 | `unsupported content type application/pdf; hint: browser` |
|
||||
| sift internal error | 500 | `internal error` (details only in the server log) |
|
||||
|
||||
Never return 502 (SDK would retry the fetch). Every error body is
|
||||
`{"success": false, "error": "…"}`; add `"hint"` as a separate field too for the CLI.
|
||||
|
||||
## 3. Code layout (Go, add deps: go-readability, html-to-markdown/v2, gojq)
|
||||
|
||||
```
|
||||
internal/guard (P0) — unchanged API; Fetch uses it
|
||||
internal/fetch Fetcher{Guard *guard.Guard, UA string, Limits}; Fetch(ctx, rawURL) → Resource
|
||||
internal/clean HTML(Resource, onlyMain bool) → markdown+title; JSON(Resource, jq) ; Text
|
||||
internal/auth keys file load, bearer check, caller identity
|
||||
internal/api /healthz /metrics /v2/scrape /v1/fetch, error mapping (§2)
|
||||
cmd/sift serve | fetch | keys (one binary)
|
||||
```
|
||||
|
||||
**Single call path**: `fetch.Fetcher.Fetch` is the only code that performs outbound HTTP.
|
||||
It calls `Guard.CheckURL` then uses `Guard.Client()` built from **the same `*guard.Guard`
|
||||
that holds `DenyPrefixes`** (constructed once in `serve` and passed down — fixes P0's
|
||||
discarded instance at `cmd/sift/main.go:64`). Add a unit test with a fake guard that
|
||||
records every CheckURL/Control call to prove no path bypasses it.
|
||||
|
||||
**Fetch**: GET only; headers: `User-Agent` = config `user_agent` (default `sift/<ver>`),
|
||||
`Accept: text/html,application/xhtml+xml,application/json;q=0.9,text/plain;q=0.8,*/*;q=0.1`,
|
||||
`Accept-Encoding` left to Go (transparent gzip). No cookies (no jar), no auth, no Referer.
|
||||
Context timeout 20 s total. Read `resp.Body` through `io.LimitReader(body, max+1)`; the
|
||||
reader is already decompressed, so this is the decompressed cap (zip-bomb safe).
|
||||
|
||||
**Content handling** (by `Content-Type`, sniff only to reject a mismatch):
|
||||
- `text/html`, `application/xhtml+xml` → go-readability (with `onlyMainContent`) →
|
||||
html-to-markdown; keep links absolute (resolve against final URL); drop `script`,
|
||||
`style`, `noscript`, `iframe`, `data:` images. Title/byline/published date → metadata.
|
||||
If readability finds nothing, convert `<body>`.
|
||||
- `application/json`, `*+json` → validate; `jq` filter via gojq with a 2 s eval timeout and
|
||||
output cap; pretty-print. For `/v2/scrape`, JSON is returned as a fenced ```json block in
|
||||
`markdown` (Hermes only reads markdown/html).
|
||||
- `text/plain`, `text/markdown`, `text/csv` → as-is.
|
||||
- everything else → 415 with `hint: browser`.
|
||||
- Charset: decode per header/meta to UTF-8 (`golang.org/x/net/html/charset`).
|
||||
|
||||
**Limits** (config, these defaults): body 5 MB decompressed; output 200 k chars (truncate
|
||||
with a `warnings` entry and `metadata.sift.truncated: true`); redirects 5 (P0); total 20 s;
|
||||
global concurrency 8, per-(caller,host) 2 (semaphores; excess waits ≤ 5 s then 429
|
||||
`busy`); request body to sift ≤ 64 KB.
|
||||
|
||||
## 4. Auth and keys
|
||||
|
||||
- Keys file `/var/lib/sift/keys.toml` (0640 root:sift):
|
||||
`[[caller]] name = "talos" workspace = "hermes-fleet" key_sha256 = "…"`.
|
||||
Server compares `sha256(bearer)` with `subtle.ConstantTimeCompare`; reload on SIGHUP.
|
||||
- `sift keys add --caller talos --workspace hermes-fleet --keys /var/lib/sift/keys.toml
|
||||
--out /path/to/talos.key` generates 32 random bytes (base64url), writes the key **only**
|
||||
to `--out` (0600, refuses to overwrite), appends the hash. Nothing secret on stdout.
|
||||
`sift keys list` prints names/workspaces only. `sift keys revoke --caller talos`.
|
||||
- Distribution (talos does it, no key in chat/logs): generate on orion into a root-only
|
||||
temp file, `scp` it to the client, place it, delete the temp copy, `stat` both ends.
|
||||
- Client-side placement: `FIRECRAWL_API_KEY` in `~/.hermes/.env` (edit in place with
|
||||
`sed -i`/`hermes config`, mode stays 600 — see the 2026-09-22 `.env` 644 near-miss) and the
|
||||
same key file referenced by `~/.config/sift/client.toml` (0600).
|
||||
- Log caller name per request; never log the key or its hash.
|
||||
|
||||
## 5. CLI client
|
||||
|
||||
```
|
||||
sift fetch <url> [--mode clean|raw] [--jq FILTER] [--no-main] [--json]
|
||||
sift keys add|list|revoke … (server-side admin, on orion)
|
||||
sift serve … (the service)
|
||||
```
|
||||
- Config `~/.config/sift/client.toml`: `url = "https://orion.scylla-hammerhead.ts.net:8443"`,
|
||||
`key_file = "~/.config/sift/key"`. Refuse to run if the key file is group/world readable.
|
||||
- Default output: the content only (for agents); `--json`: full `/v1/fetch` envelope.
|
||||
Non-zero exit + `error`/`hint` on stderr for failures (exit 2 blocked, 3 upstream,
|
||||
4 unsupported, 5 sift unreachable) so the skill's decision table maps to exit codes.
|
||||
- No URL or key ever passed through a shell pipe; the CLI is the whole command.
|
||||
|
||||
## 6. Tests
|
||||
|
||||
- Unit: clean fixtures (HTML article, HTML no-article, charset, JSON + jq, text, oversize,
|
||||
gzip bomb), error mapping table (§2) one test per row, auth (good/bad/missing, constant
|
||||
time path), limits, `skipTlsVerification` ignored (httptest TLS server with an untrusted
|
||||
cert → failure).
|
||||
- Guard e2e through the API: `-deny-prefix` address → 403; `http://127.0.0.1`,
|
||||
`http://100.95.252.120:8000`, redirect-to-private → 403.
|
||||
- **Contract** (`make contract`): a Python harness (test-only; Hermes is Python) in a venv
|
||||
with `firecrawl-py==4.17.0` calls `Firecrawl(api_url=…, api_key=…).scrape(url,
|
||||
formats=[…])` against a locally started sift and asserts: Document parses; markdown
|
||||
present; title; error cases raise with sift's message. Upstream fixtures need a local
|
||||
HTTP server, which the guard blocks — so the fixture allowance is a **build-tag-only**
|
||||
flag: `//go:build sifttest` adds `-test-allow-prefix`. The release build must not contain
|
||||
it; add a test that `sift serve -h` of the release binary does not mention it.
|
||||
|
||||
## 7. Deployment (orion)
|
||||
|
||||
- `~/src/nixos`: add the sift flake input; `hw/orion` imports `sift.nixosModules.sift`:
|
||||
`services.sift = { enable = true; listen = "127.0.0.1:8890"; userAgent = null;
|
||||
denyPrefixes = [ "<Kyle's home WAN>/32" ]; keysFile = "/var/lib/sift/keys.toml"; };`
|
||||
— Kyle supplies the WAN address(es); they live in `hw/orion` only.
|
||||
- `tailscale-serve-sift` oneshot like `tailscale-serve-searx`, with `Restart=on-failure`
|
||||
and `after tailscaled + network-online` (lessons from 2026-09-22): `tailscale serve --bg
|
||||
--https=8443 http://127.0.0.1:8890`.
|
||||
- systemd hardening in the module (and `dist/sift.service`): `DynamicUser` or `sift`
|
||||
system user, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp`, `NoNewPrivileges`,
|
||||
`RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `CapabilityBoundingSet=`,
|
||||
`SystemCallFilter=@system-service`, `StateDirectory=sift`, `MemoryMax=512M`.
|
||||
- Deploy only via committed master + `rebuild-nixos` on orion; `git fetch` first
|
||||
(orion's clone was 91 commits stale on 2026-09-22). Don't run it from an SSH session
|
||||
you'll lose (a gateway reload killed a switch mid-activation on 2026-09-22) — use
|
||||
`systemd-run` or have Kyle run it.
|
||||
|
||||
## 8. Wiring talos (first host)
|
||||
|
||||
1. Key generated and placed (§4). Pre-install `firecrawl-py==4.17.0` into talos's Hermes env.
|
||||
2. `~/.hermes/.env` (in place, 600): `FIRECRAWL_API_URL=https://orion.scylla-hammerhead.ts.net:8443`,
|
||||
`FIRECRAWL_API_KEY=<key>`; `config.yaml`: `web.extract_backend: firecrawl`
|
||||
(0.19 supports it — `tools/web_tools.py` `_get_extract_backend`). `search_backend`
|
||||
stays `searxng`.
|
||||
3. Install `sift` binary (Nix profile or `make install`) + `~/.config/sift/client.toml`.
|
||||
4. Install the skill: `skill/SKILL.md` → `~/.hermes/skills/web/sift/SKILL.md`; add the
|
||||
known-limitation note (§2) to it.
|
||||
5. Kyle restarts talos's gateway. Then the exit-criteria evidence (§1).
|
||||
Reference in New Issue
Block a user