# Sift P1 — clean fetch, auth, CLI client, first host

Drafted 2026-09-22 by Claude. Implementer: straylight. Reviewer: Claude (contract + guard
wiring). Orion deploy + talos wiring: talos. Builds on P0 (`099b949`, guard gate passed).

## 1. Scope

In: `POST /v2/scrape` (Firecrawl-compatible), `POST /v1/fetch` (modes `clean`/`raw`, `jq`),
per-caller bearer auth, limits, structured errors, basic `/metrics`, the `sift` CLI client
(`fetch`, `keys`), `skill/SKILL.md` installed on talos, orion deployment, talos wired.

Out (later phases): digest/`question`/Honcho context (P2), `--recover` (P2), `search` (P3),
cache (P4). `/v1/fetch` must reject `mode=digest`, `question`, `context` with a clear
`400 not implemented until P2`.

**Exit criteria** (post the evidence in the ~dev thread):
1. `make test lint contract` green; `nix flake check` green.
2. talos: a Hermes `web_extract` of a public HTML page and of a JSON API URL both return
   cleaned content via sift (show the orion log line with caller=`talos`).
3. talos: `sift fetch http://100.95.252.120:8000/` and `sift fetch http://dixie:11434/` and a
   redirect-to-private URL are refused with `blocked: …`, from talos, against the live
   service.
4. talos: `sift fetch <hf-api-url> --jq '<filter>'` works, and `tirith check` on that exact
   command returns `allow`.
5. A `-deny-prefix` address is refused **end to end through `/v2/scrape`** (P0 carry-over).

## 2. Contract facts (verified 2026-09-22, not assumed)

Both fleet Hermes versions (talos 0.19.0, straylight 0.21.x) call self-hosted Firecrawl
through **`firecrawl-py==4.17.0`**, lazily installed by Hermes (`tools/lazy_deps.py`,
`search.firecrawl`) on first use. It is **not installed on any host yet** — the first
`web_extract` after wiring triggers a pip install into Hermes's env. Pre-install it
during wiring (`uv pip install --python <hermes python> firecrawl-py==4.17.0`) so the
first call isn't a surprise network install.

**Request** (`firecrawl/v2/methods/scrape.py`), exactly what sift will receive:
```json
{"url": "https://example.com/x", "onlyMainContent": true, "skipTlsVerification": true,
 "removeBase64Images": true, "fastMode": false, "blockAds": true, "storeInCache": true,
 "maxAge": 14400000, "formats": ["markdown", "html"], "mobile": false,
 "origin": "python-sdk@4.17.0"}
```
- Decode leniently: ignore unknown fields.
- **`skipTlsVerification: true` is sent by default and MUST be ignored.** Sift never
  disables TLS verification because a request asked it to. Add a test that proves a
  bad-cert upstream fails even with that field set.
- Honour `formats` (`markdown`, `html`) and `onlyMainContent` (true → readability article;
  false → whole body converted). Ignore `mobile`, `blockAds`, `fastMode`,
  `removeBase64Images` (drop `data:` images always), `storeInCache`/`maxAge` (P4 cache hint).

**Auth**: SDK sends `Authorization: Bearer <FIRECRAWL_API_KEY>` when the key is set.

**Response**: SDK requires HTTP 2xx **and** `body.success == true`, then builds
`Document(markdown, html, metadata)`; metadata keys are camelCase → snake_case
normalised (`sourceURL` → `source_url`), extras allowed.

**Known limitation (0.19 and 0.21 SDK path)**: Hermes reads `metadata.get("sourceURL", url)`
*after* the SDK renamed it to `source_url`, so Hermes cites the **requested** URL, not the
final URL after redirects, and its own post-redirect SSRF re-check sees the original URL.
Sift's server-side guard is unaffected. Document it in the skill; the native CLI returns
`final_url` correctly.

**Errors → SDK behaviour** (`firecrawl/v2/utils/error_handler.py`): non-2xx raises with
`body.error` in the message; **the SDK retries only on 502**. Therefore:

| Situation | HTTP | `error` |
|---|---|---|
| guard refusal | 403 | `blocked: private address 10.0.0.5` (surfaces as "Website Not Supported: … blocked: …") |
| bad request / invalid URL / unsupported option | 400 | `invalid url` / `not implemented until P2` |
| missing/unknown key | 401 | `unauthorized` |
| upstream 403/429/451/503-with-challenge | 422 | `upstream 403 (likely bot-blocked); hint: browser` |
| other upstream 4xx/5xx | 422 | `upstream 404` |
| upstream timeout | 504 | `upstream timeout after 20s` |
| body over cap | 413 | `too large (>5 MB decompressed)` |
| unsupported content type | 415 | `unsupported content type application/pdf; hint: browser` |
| sift internal error | 500 | `internal error` (details only in the server log) |

Never return 502 (SDK would retry the fetch). Every error body is
`{"success": false, "error": "…"}`; add `"hint"` as a separate field too for the CLI.

## 3. Code layout (Go, add deps: go-readability, html-to-markdown/v2, gojq)

```
internal/guard     (P0) — unchanged API; Fetch uses it
internal/fetch     Fetcher{Guard *guard.Guard, UA string, Limits}; Fetch(ctx, rawURL) → Resource
internal/clean     HTML(Resource, onlyMain bool) → markdown+title; JSON(Resource, jq) ; Text
internal/auth      keys file load, bearer check, caller identity
internal/api       /healthz /metrics /v2/scrape /v1/fetch, error mapping (§2)
cmd/sift           serve | fetch | keys   (one binary)
```

**Single call path**: `fetch.Fetcher.Fetch` is the only code that performs outbound HTTP.
It calls `Guard.CheckURL` then uses `Guard.Client()` built from **the same `*guard.Guard`
that holds `DenyPrefixes`** (constructed once in `serve` and passed down — fixes P0's
discarded instance at `cmd/sift/main.go:64`). Add a unit test with a fake guard that
records every CheckURL/Control call to prove no path bypasses it.

**Fetch**: GET only; headers: `User-Agent` = config `user_agent` (default `sift/<ver>`),
`Accept: text/html,application/xhtml+xml,application/json;q=0.9,text/plain;q=0.8,*/*;q=0.1`,
`Accept-Encoding` left to Go (transparent gzip). No cookies (no jar), no auth, no Referer.
Context timeout 20 s total. Read `resp.Body` through `io.LimitReader(body, max+1)`; the
reader is already decompressed, so this is the decompressed cap (zip-bomb safe).

**Content handling** (by `Content-Type`, sniff only to reject a mismatch):
- `text/html`, `application/xhtml+xml` → go-readability (with `onlyMainContent`) →
  html-to-markdown; keep links absolute (resolve against final URL); drop `script`,
  `style`, `noscript`, `iframe`, `data:` images. Title/byline/published date → metadata.
  If readability finds nothing, convert `<body>`.
- `application/json`, `*+json` → validate; `jq` filter via gojq with a 2 s eval timeout and
  output cap; pretty-print. For `/v2/scrape`, JSON is returned as a fenced ```json block in
  `markdown` (Hermes only reads markdown/html).
- `text/plain`, `text/markdown`, `text/csv` → as-is.
- everything else → 415 with `hint: browser`.
- Charset: decode per header/meta to UTF-8 (`golang.org/x/net/html/charset`).

**Limits** (config, these defaults): body 5 MB decompressed; output 200 k chars (truncate
with a `warnings` entry and `metadata.sift.truncated: true`); redirects 5 (P0); total 20 s;
global concurrency 8, per-(caller,host) 2 (semaphores; excess waits ≤ 5 s then 429
`busy`); request body to sift ≤ 64 KB.

## 4. Auth and keys

- Keys file `/var/lib/sift/keys.toml` (0640 root:sift):
  `[[caller]] name = "talos"  workspace = "hermes-fleet"  key_sha256 = "…"`.
  Server compares `sha256(bearer)` with `subtle.ConstantTimeCompare`; reload on SIGHUP.
- `sift keys add --caller talos --workspace hermes-fleet --keys /var/lib/sift/keys.toml
  --out /path/to/talos.key` generates 32 random bytes (base64url), writes the key **only**
  to `--out` (0600, refuses to overwrite), appends the hash. Nothing secret on stdout.
  `sift keys list` prints names/workspaces only. `sift keys revoke --caller talos`.
- Distribution (talos does it, no key in chat/logs): generate on orion into a root-only
  temp file, `scp` it to the client, place it, delete the temp copy, `stat` both ends.
- Client-side placement: `FIRECRAWL_API_KEY` in `~/.hermes/.env` (edit in place with
  `sed -i`/`hermes config`, mode stays 600 — see the 2026-09-22 `.env` 644 near-miss) and the
  same key file referenced by `~/.config/sift/client.toml` (0600).
- Log caller name per request; never log the key or its hash.

## 5. CLI client

```
sift fetch <url> [--mode clean|raw] [--jq FILTER] [--no-main] [--json]
sift keys add|list|revoke …        (server-side admin, on orion)
sift serve …                       (the service)
```
- Config `~/.config/sift/client.toml`: `url = "https://orion.scylla-hammerhead.ts.net:8443"`,
  `key_file = "~/.config/sift/key"`. Refuse to run if the key file is group/world readable.
- Default output: the content only (for agents); `--json`: full `/v1/fetch` envelope.
  Non-zero exit + `error`/`hint` on stderr for failures (exit 2 blocked, 3 upstream,
  4 unsupported, 5 sift unreachable) so the skill's decision table maps to exit codes.
- No URL or key ever passed through a shell pipe; the CLI is the whole command.

## 6. Tests

- Unit: clean fixtures (HTML article, HTML no-article, charset, JSON + jq, text, oversize,
  gzip bomb), error mapping table (§2) one test per row, auth (good/bad/missing, constant
  time path), limits, `skipTlsVerification` ignored (httptest TLS server with an untrusted
  cert → failure).
- Guard e2e through the API: `-deny-prefix` address → 403; `http://127.0.0.1`,
  `http://100.95.252.120:8000`, redirect-to-private → 403.
- **Contract** (`make contract`): a Python harness (test-only; Hermes is Python) in a venv
  with `firecrawl-py==4.17.0` calls `Firecrawl(api_url=…, api_key=…).scrape(url,
  formats=[…])` against a locally started sift and asserts: Document parses; markdown
  present; title; error cases raise with sift's message. Upstream fixtures need a local
  HTTP server, which the guard blocks — so the fixture allowance is a **build-tag-only**
  flag: `//go:build sifttest` adds `-test-allow-prefix`. The release build must not contain
  it; add a test that `sift serve -h` of the release binary does not mention it.

## 7. Deployment (orion)

- `~/src/nixos`: add the sift flake input; `hw/orion` imports `sift.nixosModules.sift`:
  `services.sift = { enable = true; listen = "127.0.0.1:8890"; userAgent = null;
  denyPrefixes = [ "<Kyle's home WAN>/32" ]; keysFile = "/var/lib/sift/keys.toml"; };`
  — Kyle supplies the WAN address(es); they live in `hw/orion` only.
- `tailscale-serve-sift` oneshot like `tailscale-serve-searx`, with `Restart=on-failure`
  and `after tailscaled + network-online` (lessons from 2026-09-22): `tailscale serve --bg
  --https=8443 http://127.0.0.1:8890`.
- systemd hardening in the module (and `dist/sift.service`): `DynamicUser` or `sift`
  system user, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp`, `NoNewPrivileges`,
  `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `CapabilityBoundingSet=`,
  `SystemCallFilter=@system-service`, `StateDirectory=sift`, `MemoryMax=512M`.
- Deploy only via committed master + `rebuild-nixos` on orion; `git fetch` first
  (orion's clone was 91 commits stale on 2026-09-22). Don't run it from an SSH session
  you'll lose (a gateway reload killed a switch mid-activation on 2026-09-22) — use
  `systemd-run` or have Kyle run it.

## 8. Wiring talos (first host)

1. Key generated and placed (§4). Pre-install `firecrawl-py==4.17.0` into talos's Hermes env.
2. `~/.hermes/.env` (in place, 600): `FIRECRAWL_API_URL=https://orion.scylla-hammerhead.ts.net:8443`,
   `FIRECRAWL_API_KEY=<key>`; `config.yaml`: `web.extract_backend: firecrawl`
   (0.19 supports it — `tools/web_tools.py` `_get_extract_backend`). `search_backend`
   stays `searxng`.
3. Install `sift` binary (Nix profile or `make install`) + `~/.config/sift/client.toml`.
4. Install the skill: `skill/SKILL.md` → `~/.hermes/skills/web/sift/SKILL.md`; add the
   known-limitation note (§2) to it.
5. Kyle restarts talos's gateway. Then the exit-criteria evidence (§1).

## 9. P1.1 — dynamic self-IP deny (Kyle, 2026-09-22)

The home WAN address is residential DSL and changes. A static `/32` in `denyPrefixes` goes
stale silently (and the old address gets reassigned to a stranger). Add **self-address
discovery** to sift itself so the hairpin-NAT block follows the real address:

- Config `self_ip` block:
  `sources = ["ddns:<hostname>", "trace:https://1.1.1.1/cdn-cgi/trace"]`,
  `interval = "10m"`, `keep_previous = "24h"`.
  - `ddns:<host>` — resolve the router's DDNS name (preferred if one exists, e.g. MikroTik
    IP Cloud `…sn.mynetname.net`); no outbound HTTP needed.
  - `trace:<url>` — GET through the **guarded** fetcher, parse the `ip=` line (Cloudflare
    trace format). A result that is not a global unicast address is ignored.
- Discovered addresses (v4 /32, v6 /64) are added to the dial-time deny set alongside
  `denyPrefixes`; previous addresses are kept for `keep_previous` so a change mid-request
  can't open a window.
- Discovery runs **before** the listener opens (bounded wait, e.g. 10 s); on failure sift
  starts with the static `denyPrefixes` plus the last persisted value
  (`/var/lib/sift/self_ip.json`) and logs WARN; it never serves with an *empty* self set if a
  previous value exists.
- Swap the deny set atomically (`atomic.Pointer`); `/healthz` reports `self_ip` age and
  source; `/metrics` counts discovery failures.
- Tests: injected resolver/trace returning A then B → both denied until B + keep_previous;
  non-global answer ignored; discovery failure keeps the persisted value.

Until P1.1 lands, deploy with the current address as a static entry
(`denyPrefixes = [ "99.73.161.224/32" ]` in `hw/orion`).