From 6cb206fb27f29762331e2007189e0fb590427167 Mon Sep 17 00:00:00 2001 From: Kyle Isom Date: Tue, 22 Sep 2026 21:38:29 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20P1=20spec=20=E2=80=94=20clean=20fetch,?= =?UTF-8?q?=20auth,=20CLI,=20contract=20facts=20from=20firecrawl-py=204.17?= =?UTF-8?q?.0,=20talos=20wiring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/P1.md | 195 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 docs/P1.md diff --git a/docs/P1.md b/docs/P1.md new file mode 100644 index 0000000..4b85a90 --- /dev/null +++ b/docs/P1.md @@ -0,0 +1,195 @@ +# Sift P1 — clean fetch, auth, CLI client, first host + +Drafted 2026-09-22 by Claude. Implementer: straylight. Reviewer: Claude (contract + guard +wiring). Orion deploy + talos wiring: talos. Builds on P0 (`099b949`, guard gate passed). + +## 1. Scope + +In: `POST /v2/scrape` (Firecrawl-compatible), `POST /v1/fetch` (modes `clean`/`raw`, `jq`), +per-caller bearer auth, limits, structured errors, basic `/metrics`, the `sift` CLI client +(`fetch`, `keys`), `skill/SKILL.md` installed on talos, orion deployment, talos wired. + +Out (later phases): digest/`question`/Honcho context (P2), `--recover` (P2), `search` (P3), +cache (P4). `/v1/fetch` must reject `mode=digest`, `question`, `context` with a clear +`400 not implemented until P2`. + +**Exit criteria** (post the evidence in the ~dev thread): +1. `make test lint contract` green; `nix flake check` green. +2. talos: a Hermes `web_extract` of a public HTML page and of a JSON API URL both return + cleaned content via sift (show the orion log line with caller=`talos`). +3. talos: `sift fetch http://100.95.252.120:8000/` and `sift fetch http://dixie:11434/` and a + redirect-to-private URL are refused with `blocked: …`, from talos, against the live + service. +4. talos: `sift fetch --jq ''` works, and `tirith check` on that exact + command returns `allow`. +5. A `-deny-prefix` address is refused **end to end through `/v2/scrape`** (P0 carry-over). + +## 2. Contract facts (verified 2026-09-22, not assumed) + +Both fleet Hermes versions (talos 0.19.0, straylight 0.21.x) call self-hosted Firecrawl +through **`firecrawl-py==4.17.0`**, lazily installed by Hermes (`tools/lazy_deps.py`, +`search.firecrawl`) on first use. It is **not installed on any host yet** — the first +`web_extract` after wiring triggers a pip install into Hermes's env. Pre-install it +during wiring (`uv pip install --python firecrawl-py==4.17.0`) so the +first call isn't a surprise network install. + +**Request** (`firecrawl/v2/methods/scrape.py`), exactly what sift will receive: +```json +{"url": "https://example.com/x", "onlyMainContent": true, "skipTlsVerification": true, + "removeBase64Images": true, "fastMode": false, "blockAds": true, "storeInCache": true, + "maxAge": 14400000, "formats": ["markdown", "html"], "mobile": false, + "origin": "python-sdk@4.17.0"} +``` +- Decode leniently: ignore unknown fields. +- **`skipTlsVerification: true` is sent by default and MUST be ignored.** Sift never + disables TLS verification because a request asked it to. Add a test that proves a + bad-cert upstream fails even with that field set. +- Honour `formats` (`markdown`, `html`) and `onlyMainContent` (true → readability article; + false → whole body converted). Ignore `mobile`, `blockAds`, `fastMode`, + `removeBase64Images` (drop `data:` images always), `storeInCache`/`maxAge` (P4 cache hint). + +**Auth**: SDK sends `Authorization: Bearer ` when the key is set. + +**Response**: SDK requires HTTP 2xx **and** `body.success == true`, then builds +`Document(markdown, html, metadata)`; metadata keys are camelCase → snake_case +normalised (`sourceURL` → `source_url`), extras allowed. + +**Known limitation (0.19 and 0.21 SDK path)**: Hermes reads `metadata.get("sourceURL", url)` +*after* the SDK renamed it to `source_url`, so Hermes cites the **requested** URL, not the +final URL after redirects, and its own post-redirect SSRF re-check sees the original URL. +Sift's server-side guard is unaffected. Document it in the skill; the native CLI returns +`final_url` correctly. + +**Errors → SDK behaviour** (`firecrawl/v2/utils/error_handler.py`): non-2xx raises with +`body.error` in the message; **the SDK retries only on 502**. Therefore: + +| Situation | HTTP | `error` | +|---|---|---| +| guard refusal | 403 | `blocked: private address 10.0.0.5` (surfaces as "Website Not Supported: … blocked: …") | +| bad request / invalid URL / unsupported option | 400 | `invalid url` / `not implemented until P2` | +| missing/unknown key | 401 | `unauthorized` | +| upstream 403/429/451/503-with-challenge | 422 | `upstream 403 (likely bot-blocked); hint: browser` | +| other upstream 4xx/5xx | 422 | `upstream 404` | +| upstream timeout | 504 | `upstream timeout after 20s` | +| body over cap | 413 | `too large (>5 MB decompressed)` | +| unsupported content type | 415 | `unsupported content type application/pdf; hint: browser` | +| sift internal error | 500 | `internal error` (details only in the server log) | + +Never return 502 (SDK would retry the fetch). Every error body is +`{"success": false, "error": "…"}`; add `"hint"` as a separate field too for the CLI. + +## 3. Code layout (Go, add deps: go-readability, html-to-markdown/v2, gojq) + +``` +internal/guard (P0) — unchanged API; Fetch uses it +internal/fetch Fetcher{Guard *guard.Guard, UA string, Limits}; Fetch(ctx, rawURL) → Resource +internal/clean HTML(Resource, onlyMain bool) → markdown+title; JSON(Resource, jq) ; Text +internal/auth keys file load, bearer check, caller identity +internal/api /healthz /metrics /v2/scrape /v1/fetch, error mapping (§2) +cmd/sift serve | fetch | keys (one binary) +``` + +**Single call path**: `fetch.Fetcher.Fetch` is the only code that performs outbound HTTP. +It calls `Guard.CheckURL` then uses `Guard.Client()` built from **the same `*guard.Guard` +that holds `DenyPrefixes`** (constructed once in `serve` and passed down — fixes P0's +discarded instance at `cmd/sift/main.go:64`). Add a unit test with a fake guard that +records every CheckURL/Control call to prove no path bypasses it. + +**Fetch**: GET only; headers: `User-Agent` = config `user_agent` (default `sift/`), +`Accept: text/html,application/xhtml+xml,application/json;q=0.9,text/plain;q=0.8,*/*;q=0.1`, +`Accept-Encoding` left to Go (transparent gzip). No cookies (no jar), no auth, no Referer. +Context timeout 20 s total. Read `resp.Body` through `io.LimitReader(body, max+1)`; the +reader is already decompressed, so this is the decompressed cap (zip-bomb safe). + +**Content handling** (by `Content-Type`, sniff only to reject a mismatch): +- `text/html`, `application/xhtml+xml` → go-readability (with `onlyMainContent`) → + html-to-markdown; keep links absolute (resolve against final URL); drop `script`, + `style`, `noscript`, `iframe`, `data:` images. Title/byline/published date → metadata. + If readability finds nothing, convert ``. +- `application/json`, `*+json` → validate; `jq` filter via gojq with a 2 s eval timeout and + output cap; pretty-print. For `/v2/scrape`, JSON is returned as a fenced ```json block in + `markdown` (Hermes only reads markdown/html). +- `text/plain`, `text/markdown`, `text/csv` → as-is. +- everything else → 415 with `hint: browser`. +- Charset: decode per header/meta to UTF-8 (`golang.org/x/net/html/charset`). + +**Limits** (config, these defaults): body 5 MB decompressed; output 200 k chars (truncate +with a `warnings` entry and `metadata.sift.truncated: true`); redirects 5 (P0); total 20 s; +global concurrency 8, per-(caller,host) 2 (semaphores; excess waits ≤ 5 s then 429 +`busy`); request body to sift ≤ 64 KB. + +## 4. Auth and keys + +- Keys file `/var/lib/sift/keys.toml` (0640 root:sift): + `[[caller]] name = "talos" workspace = "hermes-fleet" key_sha256 = "…"`. + Server compares `sha256(bearer)` with `subtle.ConstantTimeCompare`; reload on SIGHUP. +- `sift keys add --caller talos --workspace hermes-fleet --keys /var/lib/sift/keys.toml + --out /path/to/talos.key` generates 32 random bytes (base64url), writes the key **only** + to `--out` (0600, refuses to overwrite), appends the hash. Nothing secret on stdout. + `sift keys list` prints names/workspaces only. `sift keys revoke --caller talos`. +- Distribution (talos does it, no key in chat/logs): generate on orion into a root-only + temp file, `scp` it to the client, place it, delete the temp copy, `stat` both ends. +- Client-side placement: `FIRECRAWL_API_KEY` in `~/.hermes/.env` (edit in place with + `sed -i`/`hermes config`, mode stays 600 — see the 2026-09-22 `.env` 644 near-miss) and the + same key file referenced by `~/.config/sift/client.toml` (0600). +- Log caller name per request; never log the key or its hash. + +## 5. CLI client + +``` +sift fetch [--mode clean|raw] [--jq FILTER] [--no-main] [--json] +sift keys add|list|revoke … (server-side admin, on orion) +sift serve … (the service) +``` +- Config `~/.config/sift/client.toml`: `url = "https://orion.scylla-hammerhead.ts.net:8443"`, + `key_file = "~/.config/sift/key"`. Refuse to run if the key file is group/world readable. +- Default output: the content only (for agents); `--json`: full `/v1/fetch` envelope. + Non-zero exit + `error`/`hint` on stderr for failures (exit 2 blocked, 3 upstream, + 4 unsupported, 5 sift unreachable) so the skill's decision table maps to exit codes. +- No URL or key ever passed through a shell pipe; the CLI is the whole command. + +## 6. Tests + +- Unit: clean fixtures (HTML article, HTML no-article, charset, JSON + jq, text, oversize, + gzip bomb), error mapping table (§2) one test per row, auth (good/bad/missing, constant + time path), limits, `skipTlsVerification` ignored (httptest TLS server with an untrusted + cert → failure). +- Guard e2e through the API: `-deny-prefix` address → 403; `http://127.0.0.1`, + `http://100.95.252.120:8000`, redirect-to-private → 403. +- **Contract** (`make contract`): a Python harness (test-only; Hermes is Python) in a venv + with `firecrawl-py==4.17.0` calls `Firecrawl(api_url=…, api_key=…).scrape(url, + formats=[…])` against a locally started sift and asserts: Document parses; markdown + present; title; error cases raise with sift's message. Upstream fixtures need a local + HTTP server, which the guard blocks — so the fixture allowance is a **build-tag-only** + flag: `//go:build sifttest` adds `-test-allow-prefix`. The release build must not contain + it; add a test that `sift serve -h` of the release binary does not mention it. + +## 7. Deployment (orion) + +- `~/src/nixos`: add the sift flake input; `hw/orion` imports `sift.nixosModules.sift`: + `services.sift = { enable = true; listen = "127.0.0.1:8890"; userAgent = null; + denyPrefixes = [ "/32" ]; keysFile = "/var/lib/sift/keys.toml"; };` + — Kyle supplies the WAN address(es); they live in `hw/orion` only. +- `tailscale-serve-sift` oneshot like `tailscale-serve-searx`, with `Restart=on-failure` + and `after tailscaled + network-online` (lessons from 2026-09-22): `tailscale serve --bg + --https=8443 http://127.0.0.1:8890`. +- systemd hardening in the module (and `dist/sift.service`): `DynamicUser` or `sift` + system user, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp`, `NoNewPrivileges`, + `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `CapabilityBoundingSet=`, + `SystemCallFilter=@system-service`, `StateDirectory=sift`, `MemoryMax=512M`. +- Deploy only via committed master + `rebuild-nixos` on orion; `git fetch` first + (orion's clone was 91 commits stale on 2026-09-22). Don't run it from an SSH session + you'll lose (a gateway reload killed a switch mid-activation on 2026-09-22) — use + `systemd-run` or have Kyle run it. + +## 8. Wiring talos (first host) + +1. Key generated and placed (§4). Pre-install `firecrawl-py==4.17.0` into talos's Hermes env. +2. `~/.hermes/.env` (in place, 600): `FIRECRAWL_API_URL=https://orion.scylla-hammerhead.ts.net:8443`, + `FIRECRAWL_API_KEY=`; `config.yaml`: `web.extract_backend: firecrawl` + (0.19 supports it — `tools/web_tools.py` `_get_extract_backend`). `search_backend` + stays `searxng`. +3. Install `sift` binary (Nix profile or `make install`) + `~/.config/sift/client.toml`. +4. Install the skill: `skill/SKILL.md` → `~/.hermes/skills/web/sift/SKILL.md`; add the + known-limitation note (§2) to it. +5. Kyle restarts talos's gateway. Then the exit-criteria evidence (§1).