kyleandClaude Opus 5.5
d404657820
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
224 lines
13 KiB
Markdown
224 lines
13 KiB
Markdown
# Sift P1 — clean fetch, auth, CLI client, first host
|
|
|
|
Drafted 2026-09-22 by Claude. Implementer: straylight. Reviewer: Claude (contract + guard
|
|
wiring). Orion deploy + talos wiring: talos. Builds on P0 (`099b949`, guard gate passed).
|
|
|
|
## 1. Scope
|
|
|
|
In: `POST /v2/scrape` (Firecrawl-compatible), `POST /v1/fetch` (modes `clean`/`raw`, `jq`),
|
|
per-caller bearer auth, limits, structured errors, basic `/metrics`, the `sift` CLI client
|
|
(`fetch`, `keys`), `skill/SKILL.md` installed on talos, orion deployment, talos wired.
|
|
|
|
Out (later phases): digest/`question`/Honcho context (P2), `--recover` (P2), `search` (P3),
|
|
cache (P4). `/v1/fetch` must reject `mode=digest`, `question`, `context` with a clear
|
|
`400 not implemented until P2`.
|
|
|
|
**Exit criteria** (post the evidence in the ~dev thread):
|
|
1. `make test lint contract` green; `nix flake check` green.
|
|
2. talos: a Hermes `web_extract` of a public HTML page and of a JSON API URL both return
|
|
cleaned content via sift (show the orion log line with caller=`talos`).
|
|
3. talos: `sift fetch http://100.95.252.120:8000/` and `sift fetch http://dixie:11434/` and a
|
|
redirect-to-private URL are refused with `blocked: …`, from talos, against the live
|
|
service.
|
|
4. talos: `sift fetch <hf-api-url> --jq '<filter>'` works, and `tirith check` on that exact
|
|
command returns `allow`.
|
|
5. A `-deny-prefix` address is refused **end to end through `/v2/scrape`** (P0 carry-over).
|
|
|
|
## 2. Contract facts (verified 2026-09-22, not assumed)
|
|
|
|
Both fleet Hermes versions (talos 0.19.0, straylight 0.21.x) call self-hosted Firecrawl
|
|
through **`firecrawl-py==4.17.0`**, lazily installed by Hermes (`tools/lazy_deps.py`,
|
|
`search.firecrawl`) on first use. It is **not installed on any host yet** — the first
|
|
`web_extract` after wiring triggers a pip install into Hermes's env. Pre-install it
|
|
during wiring (`uv pip install --python <hermes python> firecrawl-py==4.17.0`) so the
|
|
first call isn't a surprise network install.
|
|
|
|
**Request** (`firecrawl/v2/methods/scrape.py`), exactly what sift will receive:
|
|
```json
|
|
{"url": "https://example.com/x", "onlyMainContent": true, "skipTlsVerification": true,
|
|
"removeBase64Images": true, "fastMode": false, "blockAds": true, "storeInCache": true,
|
|
"maxAge": 14400000, "formats": ["markdown", "html"], "mobile": false,
|
|
"origin": "python-sdk@4.17.0"}
|
|
```
|
|
- Decode leniently: ignore unknown fields.
|
|
- **`skipTlsVerification: true` is sent by default and MUST be ignored.** Sift never
|
|
disables TLS verification because a request asked it to. Add a test that proves a
|
|
bad-cert upstream fails even with that field set.
|
|
- Honour `formats` (`markdown`, `html`) and `onlyMainContent` (true → readability article;
|
|
false → whole body converted). Ignore `mobile`, `blockAds`, `fastMode`,
|
|
`removeBase64Images` (drop `data:` images always), `storeInCache`/`maxAge` (P4 cache hint).
|
|
|
|
**Auth**: SDK sends `Authorization: Bearer <FIRECRAWL_API_KEY>` when the key is set.
|
|
|
|
**Response**: SDK requires HTTP 2xx **and** `body.success == true`, then builds
|
|
`Document(markdown, html, metadata)`; metadata keys are camelCase → snake_case
|
|
normalised (`sourceURL` → `source_url`), extras allowed.
|
|
|
|
**Known limitation (0.19 and 0.21 SDK path)**: Hermes reads `metadata.get("sourceURL", url)`
|
|
*after* the SDK renamed it to `source_url`, so Hermes cites the **requested** URL, not the
|
|
final URL after redirects, and its own post-redirect SSRF re-check sees the original URL.
|
|
Sift's server-side guard is unaffected. Document it in the skill; the native CLI returns
|
|
`final_url` correctly.
|
|
|
|
**Errors → SDK behaviour** (`firecrawl/v2/utils/error_handler.py`): non-2xx raises with
|
|
`body.error` in the message; **the SDK retries only on 502**. Therefore:
|
|
|
|
| Situation | HTTP | `error` |
|
|
|---|---|---|
|
|
| guard refusal | 403 | `blocked: private address 10.0.0.5` (surfaces as "Website Not Supported: … blocked: …") |
|
|
| bad request / invalid URL / unsupported option | 400 | `invalid url` / `not implemented until P2` |
|
|
| missing/unknown key | 401 | `unauthorized` |
|
|
| upstream 403/429/451/503-with-challenge | 422 | `upstream 403 (likely bot-blocked); hint: browser` |
|
|
| other upstream 4xx/5xx | 422 | `upstream 404` |
|
|
| upstream timeout | 504 | `upstream timeout after 20s` |
|
|
| body over cap | 413 | `too large (>5 MB decompressed)` |
|
|
| unsupported content type | 415 | `unsupported content type application/pdf; hint: browser` |
|
|
| sift internal error | 500 | `internal error` (details only in the server log) |
|
|
|
|
Never return 502 (SDK would retry the fetch). Every error body is
|
|
`{"success": false, "error": "…"}`; add `"hint"` as a separate field too for the CLI.
|
|
|
|
## 3. Code layout (Go, add deps: go-readability, html-to-markdown/v2, gojq)
|
|
|
|
```
|
|
internal/guard (P0) — unchanged API; Fetch uses it
|
|
internal/fetch Fetcher{Guard *guard.Guard, UA string, Limits}; Fetch(ctx, rawURL) → Resource
|
|
internal/clean HTML(Resource, onlyMain bool) → markdown+title; JSON(Resource, jq) ; Text
|
|
internal/auth keys file load, bearer check, caller identity
|
|
internal/api /healthz /metrics /v2/scrape /v1/fetch, error mapping (§2)
|
|
cmd/sift serve | fetch | keys (one binary)
|
|
```
|
|
|
|
**Single call path**: `fetch.Fetcher.Fetch` is the only code that performs outbound HTTP.
|
|
It calls `Guard.CheckURL` then uses `Guard.Client()` built from **the same `*guard.Guard`
|
|
that holds `DenyPrefixes`** (constructed once in `serve` and passed down — fixes P0's
|
|
discarded instance at `cmd/sift/main.go:64`). Add a unit test with a fake guard that
|
|
records every CheckURL/Control call to prove no path bypasses it.
|
|
|
|
**Fetch**: GET only; headers: `User-Agent` = config `user_agent` (default `sift/<ver>`),
|
|
`Accept: text/html,application/xhtml+xml,application/json;q=0.9,text/plain;q=0.8,*/*;q=0.1`,
|
|
`Accept-Encoding` left to Go (transparent gzip). No cookies (no jar), no auth, no Referer.
|
|
Context timeout 20 s total. Read `resp.Body` through `io.LimitReader(body, max+1)`; the
|
|
reader is already decompressed, so this is the decompressed cap (zip-bomb safe).
|
|
|
|
**Content handling** (by `Content-Type`, sniff only to reject a mismatch):
|
|
- `text/html`, `application/xhtml+xml` → go-readability (with `onlyMainContent`) →
|
|
html-to-markdown; keep links absolute (resolve against final URL); drop `script`,
|
|
`style`, `noscript`, `iframe`, `data:` images. Title/byline/published date → metadata.
|
|
If readability finds nothing, convert `<body>`.
|
|
- `application/json`, `*+json` → validate; `jq` filter via gojq with a 2 s eval timeout and
|
|
output cap; pretty-print. For `/v2/scrape`, JSON is returned as a fenced ```json block in
|
|
`markdown` (Hermes only reads markdown/html).
|
|
- `text/plain`, `text/markdown`, `text/csv` → as-is.
|
|
- everything else → 415 with `hint: browser`.
|
|
- Charset: decode per header/meta to UTF-8 (`golang.org/x/net/html/charset`).
|
|
|
|
**Limits** (config, these defaults): body 5 MB decompressed; output 200 k chars (truncate
|
|
with a `warnings` entry and `metadata.sift.truncated: true`); redirects 5 (P0); total 20 s;
|
|
global concurrency 8, per-(caller,host) 2 (semaphores; excess waits ≤ 5 s then 429
|
|
`busy`); request body to sift ≤ 64 KB.
|
|
|
|
## 4. Auth and keys
|
|
|
|
- Keys file `/var/lib/sift/keys.toml` (0640 root:sift):
|
|
`[[caller]] name = "talos" workspace = "hermes-fleet" key_sha256 = "…"`.
|
|
Server compares `sha256(bearer)` with `subtle.ConstantTimeCompare`; reload on SIGHUP.
|
|
- `sift keys add --caller talos --workspace hermes-fleet --keys /var/lib/sift/keys.toml
|
|
--out /path/to/talos.key` generates 32 random bytes (base64url), writes the key **only**
|
|
to `--out` (0600, refuses to overwrite), appends the hash. Nothing secret on stdout.
|
|
`sift keys list` prints names/workspaces only. `sift keys revoke --caller talos`.
|
|
- Distribution (talos does it, no key in chat/logs): generate on orion into a root-only
|
|
temp file, `scp` it to the client, place it, delete the temp copy, `stat` both ends.
|
|
- Client-side placement: `FIRECRAWL_API_KEY` in `~/.hermes/.env` (edit in place with
|
|
`sed -i`/`hermes config`, mode stays 600 — see the 2026-09-22 `.env` 644 near-miss) and the
|
|
same key file referenced by `~/.config/sift/client.toml` (0600).
|
|
- Log caller name per request; never log the key or its hash.
|
|
|
|
## 5. CLI client
|
|
|
|
```
|
|
sift fetch <url> [--mode clean|raw] [--jq FILTER] [--no-main] [--json]
|
|
sift keys add|list|revoke … (server-side admin, on orion)
|
|
sift serve … (the service)
|
|
```
|
|
- Config `~/.config/sift/client.toml`: `url = "https://orion.scylla-hammerhead.ts.net:8443"`,
|
|
`key_file = "~/.config/sift/key"`. Refuse to run if the key file is group/world readable.
|
|
- Default output: the content only (for agents); `--json`: full `/v1/fetch` envelope.
|
|
Non-zero exit + `error`/`hint` on stderr for failures (exit 2 blocked, 3 upstream,
|
|
4 unsupported, 5 sift unreachable) so the skill's decision table maps to exit codes.
|
|
- No URL or key ever passed through a shell pipe; the CLI is the whole command.
|
|
|
|
## 6. Tests
|
|
|
|
- Unit: clean fixtures (HTML article, HTML no-article, charset, JSON + jq, text, oversize,
|
|
gzip bomb), error mapping table (§2) one test per row, auth (good/bad/missing, constant
|
|
time path), limits, `skipTlsVerification` ignored (httptest TLS server with an untrusted
|
|
cert → failure).
|
|
- Guard e2e through the API: `-deny-prefix` address → 403; `http://127.0.0.1`,
|
|
`http://100.95.252.120:8000`, redirect-to-private → 403.
|
|
- **Contract** (`make contract`): a Python harness (test-only; Hermes is Python) in a venv
|
|
with `firecrawl-py==4.17.0` calls `Firecrawl(api_url=…, api_key=…).scrape(url,
|
|
formats=[…])` against a locally started sift and asserts: Document parses; markdown
|
|
present; title; error cases raise with sift's message. Upstream fixtures need a local
|
|
HTTP server, which the guard blocks — so the fixture allowance is a **build-tag-only**
|
|
flag: `//go:build sifttest` adds `-test-allow-prefix`. The release build must not contain
|
|
it; add a test that `sift serve -h` of the release binary does not mention it.
|
|
|
|
## 7. Deployment (orion)
|
|
|
|
- `~/src/nixos`: add the sift flake input; `hw/orion` imports `sift.nixosModules.sift`:
|
|
`services.sift = { enable = true; listen = "127.0.0.1:8890"; userAgent = null;
|
|
denyPrefixes = [ "<Kyle's home WAN>/32" ]; keysFile = "/var/lib/sift/keys.toml"; };`
|
|
— Kyle supplies the WAN address(es); they live in `hw/orion` only.
|
|
- `tailscale-serve-sift` oneshot like `tailscale-serve-searx`, with `Restart=on-failure`
|
|
and `after tailscaled + network-online` (lessons from 2026-09-22): `tailscale serve --bg
|
|
--https=8443 http://127.0.0.1:8890`.
|
|
- systemd hardening in the module (and `dist/sift.service`): `DynamicUser` or `sift`
|
|
system user, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp`, `NoNewPrivileges`,
|
|
`RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `CapabilityBoundingSet=`,
|
|
`SystemCallFilter=@system-service`, `StateDirectory=sift`, `MemoryMax=512M`.
|
|
- Deploy only via committed master + `rebuild-nixos` on orion; `git fetch` first
|
|
(orion's clone was 91 commits stale on 2026-09-22). Don't run it from an SSH session
|
|
you'll lose (a gateway reload killed a switch mid-activation on 2026-09-22) — use
|
|
`systemd-run` or have Kyle run it.
|
|
|
|
## 8. Wiring talos (first host)
|
|
|
|
1. Key generated and placed (§4). Pre-install `firecrawl-py==4.17.0` into talos's Hermes env.
|
|
2. `~/.hermes/.env` (in place, 600): `FIRECRAWL_API_URL=https://orion.scylla-hammerhead.ts.net:8443`,
|
|
`FIRECRAWL_API_KEY=<key>`; `config.yaml`: `web.extract_backend: firecrawl`
|
|
(0.19 supports it — `tools/web_tools.py` `_get_extract_backend`). `search_backend`
|
|
stays `searxng`.
|
|
3. Install `sift` binary (Nix profile or `make install`) + `~/.config/sift/client.toml`.
|
|
4. Install the skill: `skill/SKILL.md` → `~/.hermes/skills/web/sift/SKILL.md`; add the
|
|
known-limitation note (§2) to it.
|
|
5. Kyle restarts talos's gateway. Then the exit-criteria evidence (§1).
|
|
|
|
## 9. P1.1 — dynamic self-IP deny (Kyle, 2026-09-22)
|
|
|
|
The home WAN address is residential DSL and changes. A static `/32` in `denyPrefixes` goes
|
|
stale silently (and the old address gets reassigned to a stranger). Add **self-address
|
|
discovery** to sift itself so the hairpin-NAT block follows the real address:
|
|
|
|
- Config `self_ip` block:
|
|
`sources = ["ddns:<hostname>", "trace:https://1.1.1.1/cdn-cgi/trace"]`,
|
|
`interval = "10m"`, `keep_previous = "24h"`.
|
|
- `ddns:<host>` — resolve the router's DDNS name (preferred if one exists, e.g. MikroTik
|
|
IP Cloud `…sn.mynetname.net`); no outbound HTTP needed.
|
|
- `trace:<url>` — GET through the **guarded** fetcher, parse the `ip=` line (Cloudflare
|
|
trace format). A result that is not a global unicast address is ignored.
|
|
- Discovered addresses (v4 /32, v6 /64) are added to the dial-time deny set alongside
|
|
`denyPrefixes`; previous addresses are kept for `keep_previous` so a change mid-request
|
|
can't open a window.
|
|
- Discovery runs **before** the listener opens (bounded wait, e.g. 10 s); on failure sift
|
|
starts with the static `denyPrefixes` plus the last persisted value
|
|
(`/var/lib/sift/self_ip.json`) and logs WARN; it never serves with an *empty* self set if a
|
|
previous value exists.
|
|
- Swap the deny set atomically (`atomic.Pointer`); `/healthz` reports `self_ip` age and
|
|
source; `/metrics` counts discovery failures.
|
|
- Tests: injected resolver/trace returning A then B → both denied until B + keep_previous;
|
|
non-global answer ignored; discovery failure keeps the persisted value.
|
|
|
|
Until P1.1 lands, deploy with the current address as a static entry
|
|
(`denyPrefixes = [ "99.73.161.224/32" ]` in `hw/orion`).
|