sift/docs/P1.md

Sift P1 — clean fetch, auth, CLI client, first host

Drafted 2026-09-22 by Claude. Implementer: straylight. Reviewer: Claude (contract + guard wiring). Orion deploy + talos wiring: talos. Builds on P0 (099b949, guard gate passed).

1. Scope

In: POST /v2/scrape (Firecrawl-compatible), POST /v1/fetch (modes clean/raw, jq), per-caller bearer auth, limits, structured errors, basic /metrics, the sift CLI client (fetch, keys), skill/SKILL.md installed on talos, orion deployment, talos wired.

Out (later phases): digest/question/Honcho context (P2), --recover (P2), search (P3), cache (P4). /v1/fetch must reject mode=digest, question, context with a clear 400 not implemented until P2.

Exit criteria (post the evidence in the ~dev thread):

1. make test lint contract green; nix flake check green.
2. talos: a Hermes web_extract of a public HTML page and of a JSON API URL both return cleaned content via sift (show the orion log line with caller=talos).
3. talos: sift fetch http://100.95.252.120:8000/ and sift fetch http://dixie:11434/ and a redirect-to-private URL are refused with blocked: …, from talos, against the live service.
4. talos: sift fetch <hf-api-url> --jq '<filter>' works, and tirith check on that exact command returns allow.
5. A -deny-prefix address is refused end to end through /v2/scrape (P0 carry-over).

2. Contract facts (verified 2026-09-22, not assumed)

Both fleet Hermes versions (talos 0.19.0, straylight 0.21.x) call self-hosted Firecrawl through firecrawl-py==4.17.0, lazily installed by Hermes (tools/lazy_deps.py, search.firecrawl) on first use. It is not installed on any host yet — the first web_extract after wiring triggers a pip install into Hermes's env. Pre-install it during wiring (uv pip install --python <hermes python> firecrawl-py==4.17.0) so the first call isn't a surprise network install.

Request (firecrawl/v2/methods/scrape.py), exactly what sift will receive:

{"url": "https://example.com/x", "onlyMainContent": true, "skipTlsVerification": true,
 "removeBase64Images": true, "fastMode": false, "blockAds": true, "storeInCache": true,
 "maxAge": 14400000, "formats": ["markdown", "html"], "mobile": false,
 "origin": "python-sdk@4.17.0"}

• Decode leniently: ignore unknown fields.
• skipTlsVerification: true is sent by default and MUST be ignored. Sift never disables TLS verification because a request asked it to. Add a test that proves a bad-cert upstream fails even with that field set.
• Honour formats (markdown, html) and onlyMainContent (true → readability article; false → whole body converted). Ignore mobile, blockAds, fastMode, removeBase64Images (drop data: images always), storeInCache/maxAge (P4 cache hint).

Auth: SDK sends Authorization: Bearer <FIRECRAWL_API_KEY> when the key is set.

Response: SDK requires HTTP 2xx and body.success == true, then builds Document(markdown, html, metadata); metadata keys are camelCase → snake_case normalised (sourceURL → source_url), extras allowed.

Known limitation (0.19 and 0.21 SDK path): Hermes reads metadata.get("sourceURL", url) after the SDK renamed it to source_url, so Hermes cites the requested URL, not the final URL after redirects, and its own post-redirect SSRF re-check sees the original URL. Sift's server-side guard is unaffected. Document it in the skill; the native CLI returns final_url correctly.

Errors → SDK behaviour (firecrawl/v2/utils/error_handler.py): non-2xx raises with body.error in the message; the SDK retries only on 502. Therefore:

Situation	HTTP	error
guard refusal	403	blocked: private address 10.0.0.5 (surfaces as "Website Not Supported: … blocked: …")
bad request / invalid URL / unsupported option	400	invalid url / not implemented until P2
missing/unknown key	401	unauthorized
upstream 403/429/451/503-with-challenge	422	upstream 403 (likely bot-blocked); hint: browser
other upstream 4xx/5xx	422	upstream 404
upstream timeout	504	upstream timeout after 20s
body over cap	413	too large (>5 MB decompressed)
unsupported content type	415	unsupported content type application/pdf; hint: browser
sift internal error	500	internal error (details only in the server log)

Never return 502 (SDK would retry the fetch). Every error body is {"success": false, "error": "…"}; add "hint" as a separate field too for the CLI.

3. Code layout (Go, add deps: go-readability, html-to-markdown/v2, gojq)

internal/guard     (P0) — unchanged API; Fetch uses it
internal/fetch     Fetcher{Guard *guard.Guard, UA string, Limits}; Fetch(ctx, rawURL) → Resource
internal/clean     HTML(Resource, onlyMain bool) → markdown+title; JSON(Resource, jq) ; Text
internal/auth      keys file load, bearer check, caller identity
internal/api       /healthz /metrics /v2/scrape /v1/fetch, error mapping (§2)
cmd/sift           serve | fetch | keys   (one binary)

Single call path: fetch.Fetcher.Fetch is the only code that performs outbound HTTP. It calls Guard.CheckURL then uses Guard.Client() built from the same *guard.Guard that holds DenyPrefixes (constructed once in serve and passed down — fixes P0's discarded instance at cmd/sift/main.go:64). Add a unit test with a fake guard that records every CheckURL/Control call to prove no path bypasses it.

Fetch: GET only; headers: User-Agent = config user_agent (default sift/<ver>), Accept: text/html,application/xhtml+xml,application/json;q=0.9,text/plain;q=0.8,*/*;q=0.1, Accept-Encoding left to Go (transparent gzip). No cookies (no jar), no auth, no Referer. Context timeout 20 s total. Read resp.Body through io.LimitReader(body, max+1); the reader is already decompressed, so this is the decompressed cap (zip-bomb safe).

Content handling (by Content-Type, sniff only to reject a mismatch):

• text/html, application/xhtml+xml → go-readability (with onlyMainContent) → html-to-markdown; keep links absolute (resolve against final URL); drop script, style, noscript, iframe, data: images. Title/byline/published date → metadata. If readability finds nothing, convert <body>.
• application/json, *+json → validate; jq filter via gojq with a 2 s eval timeout and output cap; pretty-print. For /v2/scrape, JSON is returned as a fenced ```json block in markdown (Hermes only reads markdown/html).
• text/plain, text/markdown, text/csv → as-is.
• everything else → 415 with hint: browser.
• Charset: decode per header/meta to UTF-8 (golang.org/x/net/html/charset).

Limits (config, these defaults): body 5 MB decompressed; output 200 k chars (truncate with a warnings entry and metadata.sift.truncated: true); redirects 5 (P0); total 20 s; global concurrency 8, per-(caller,host) 2 (semaphores; excess waits ≤ 5 s then 429 busy); request body to sift ≤ 64 KB.

4. Auth and keys

• Keys file /var/lib/sift/keys.toml (0640 root:sift): [[caller]] name = "talos" workspace = "hermes-fleet" key_sha256 = "…". Server compares sha256(bearer) with subtle.ConstantTimeCompare; reload on SIGHUP.
• sift keys add --caller talos --workspace hermes-fleet --keys /var/lib/sift/keys.toml --out /path/to/talos.key generates 32 random bytes (base64url), writes the key only to --out (0600, refuses to overwrite), appends the hash. Nothing secret on stdout. sift keys list prints names/workspaces only. sift keys revoke --caller talos.
• Distribution (talos does it, no key in chat/logs): generate on orion into a root-only temp file, scp it to the client, place it, delete the temp copy, stat both ends.
• Client-side placement: FIRECRAWL_API_KEY in ~/.hermes/.env (edit in place with sed -i/hermes config, mode stays 600 — see the 2026-09-22 .env 644 near-miss) and the same key file referenced by ~/.config/sift/client.toml (0600).
• Log caller name per request; never log the key or its hash.

5. CLI client

sift fetch <url> [--mode clean|raw] [--jq FILTER] [--no-main] [--json]
sift keys add|list|revoke …        (server-side admin, on orion)
sift serve …                       (the service)

• Config ~/.config/sift/client.toml: url = "https://orion.scylla-hammerhead.ts.net:8443", key_file = "~/.config/sift/key". Refuse to run if the key file is group/world readable.
• Default output: the content only (for agents); --json: full /v1/fetch envelope. Non-zero exit + error/hint on stderr for failures (exit 2 blocked, 3 upstream, 4 unsupported, 5 sift unreachable) so the skill's decision table maps to exit codes.
• No URL or key ever passed through a shell pipe; the CLI is the whole command.

6. Tests

• Unit: clean fixtures (HTML article, HTML no-article, charset, JSON + jq, text, oversize, gzip bomb), error mapping table (§2) one test per row, auth (good/bad/missing, constant time path), limits, skipTlsVerification ignored (httptest TLS server with an untrusted cert → failure).
• Guard e2e through the API: -deny-prefix address → 403; http://127.0.0.1, http://100.95.252.120:8000, redirect-to-private → 403.
• Contract (make contract): a Python harness (test-only; Hermes is Python) in a venv with firecrawl-py==4.17.0 calls Firecrawl(api_url=…, api_key=…).scrape(url, formats=[…]) against a locally started sift and asserts: Document parses; markdown present; title; error cases raise with sift's message. Upstream fixtures need a local HTTP server, which the guard blocks — so the fixture allowance is a build-tag-only flag: //go:build sifttest adds -test-allow-prefix. The release build must not contain it; add a test that sift serve -h of the release binary does not mention it.

7. Deployment (orion)

• ~/src/nixos: add the sift flake input; hw/orion imports sift.nixosModules.sift: services.sift = { enable = true; listen = "127.0.0.1:8890"; userAgent = null; denyPrefixes = [ "<Kyle's home WAN>/32" ]; keysFile = "/var/lib/sift/keys.toml"; }; — Kyle supplies the WAN address(es); they live in hw/orion only.
• tailscale-serve-sift oneshot like tailscale-serve-searx, with Restart=on-failure and after tailscaled + network-online (lessons from 2026-09-22): tailscale serve --bg --https=8443 http://127.0.0.1:8890.
• systemd hardening in the module (and dist/sift.service): DynamicUser or sift system user, ProtectSystem=strict, ProtectHome=yes, PrivateTmp, NoNewPrivileges, RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX, CapabilityBoundingSet=, SystemCallFilter=@system-service, StateDirectory=sift, MemoryMax=512M.
• Deploy only via committed master + rebuild-nixos on orion; git fetch first (orion's clone was 91 commits stale on 2026-09-22). Don't run it from an SSH session you'll lose (a gateway reload killed a switch mid-activation on 2026-09-22) — use systemd-run or have Kyle run it.

8. Wiring talos (first host)

1. Key generated and placed (§4). Pre-install firecrawl-py==4.17.0 into talos's Hermes env.
2. ~/.hermes/.env (in place, 600): FIRECRAWL_API_URL=https://orion.scylla-hammerhead.ts.net:8443, FIRECRAWL_API_KEY=<key>; config.yaml: web.extract_backend: firecrawl (0.19 supports it — tools/web_tools.py _get_extract_backend). search_backend stays searxng.
3. Install sift binary (Nix profile or make install) + ~/.config/sift/client.toml.
4. Install the skill: skill/SKILL.md → ~/.hermes/skills/web/sift/SKILL.md; add the known-limitation note (§2) to it.
5. Kyle restarts talos's gateway. Then the exit-criteria evidence (§1).

9. P1.1 — dynamic self-IP deny (Kyle, 2026-09-22)

The home WAN address is residential DSL and changes. A static /32 in denyPrefixes goes stale silently (and the old address gets reassigned to a stranger). Add self-address discovery to sift itself so the hairpin-NAT block follows the real address:

• Config self_ip block: sources = ["ddns:<hostname>", "trace:https://1.1.1.1/cdn-cgi/trace"], interval = "10m", keep_previous = "24h".
  • ddns:<host> — resolve the router's DDNS name (preferred if one exists, e.g. MikroTik IP Cloud …sn.mynetname.net); no outbound HTTP needed.
  • trace:<url> — GET through the guarded fetcher, parse the ip= line (Cloudflare trace format). A result that is not a global unicast address is ignored.
• Discovered addresses (v4 /32, v6 /64) are added to the dial-time deny set alongside denyPrefixes; previous addresses are kept for keep_previous so a change mid-request can't open a window.
• Discovery runs before the listener opens (bounded wait, e.g. 10 s); on failure sift starts with the static denyPrefixes plus the last persisted value (/var/lib/sift/self_ip.json) and logs WARN; it never serves with an empty self set if a previous value exists.
• Swap the deny set atomically (atomic.Pointer); /healthz reports self_ip age and source; /metrics counts discovery failures.
• Tests: injected resolver/trace returning A then B → both denied until B + keep_previous; non-global answer ignored; discovery failure keeps the persisted value.

Until P1.1 lands, deploy with the current address as a static entry (denyPrefixes = [ "99.73.161.224/32" ] in hw/orion).