skill: sift as the explicit first step for fetch/search; plan §4.3 CLI client, --recover, UA policy

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:25:57 -07:00
parent 1e915ce715
commit 7d4f090a12
2 changed files with 127 additions and 1 deletions
@@ -141,6 +141,45 @@ service.)
 `GET /healthz` (deps: searxng, dixie, honcho reachability), `GET /metrics` (counts, cache hit
 rate, guard blocks, token-savings ratio, per-caller usage).

+## 4.3 CLI client and the agent skill (Kyle, 2026-09-22)
+
+Agents use sift through a **skill** (`skill/SKILL.md`, installed to
+`~/.hermes/skills/web/sift/` on each host) that makes sift the explicit first step for any
+fetch or search. The skill drives a **`sift` CLI client** — the same Go binary, client
+subcommands — installed on every Hermes host:
+
+```
+sift fetch <url> [--question Q] [--mode digest|clean|raw] [--jq FILTER] [--recover] [--no-context] [--json]
+sift search <query> [--question Q] [--k N] [--json]
+```
+
+Why a CLI and not only the Firecrawl-compatible backend: it exposes the native features
+(question-focused digest, jq, recover, context) on every Hermes version without a Hermes
+plugin (no Python), and it is scanner-clean — verified 2026-09-22 with tirith:
+`sift fetch "https://…" --question "…"`, `sift fetch <hf-api-url> --jq '…'` and
+`sift search "…"` all return `allow` with no findings, where the equivalent
+`curl -A … | python3` is a HIGH `pipe_to_interpreter` block. Client config
+(`~/.config/sift/client.toml`: server URL + this host's key, mode 0600) — never put the key
+on the command line.
+
+**Motivating example** (titan, 2026-09-22): working page reads on most sites, but on a
+bot-walled site it improvised `curl -A "Mozilla/5.0 … Chrome/131" … | python3` → HIGH
+approval prompt. Hence:
+
+- **Blocked results are structured**: `{"error": "upstream 403 (likely bot-blocked)",
+  "hint": "recover" | "browser"}` so the agent's next step is defined.
+- **`--recover`**: sift runs the existing `blocked-page-recovery` ladder **server-side**
+  (Wayback `available` API → archive.today domain rotation → API/RSS/sitemap pivot on the
+  same host), with that skill's fake-success detection, returning provenance
+  `snapshot` + timestamp. Every hop still goes through the SSRF guard. This moves the
+  ladder's `curl` calls out of agent shells.
+- **User-Agent policy — Kyle to decide**: (a) honest `sift/<ver>` UA only, bot-walled sites
+  go to recover/browser; or (b) one retry with a browser UA. (b) removes more prompts but is
+  UA spoofing.
+
+Phase placement: CLI `fetch --mode clean` + skill in **P1**; digest/context flags in P2;
+`search` in P3; `--recover` in P2 (it is what removes the titan-style prompts).
+
 ## 5. Fetch pipeline (the security-critical part)

 Threat model: the URL is attacker-influenced (prompt-injected pages tell agents to fetch
@@ -231,7 +270,7 @@ tests + one live `web_extract`).
 | Phase | Scope | Done when |
 |---|---|---|
 | **P0** skeleton | repo, flake, NixOS module, Makefile (+ `dist/sift.service`), `/healthz`, URL guard + exhaustive unit tests (table of private/tailnet/rebinding/redirect cases incl. `http://100.95.252.120:8000`, `http://dixie:11434`, `http://[fd7a:115c:a1e0::1]`, decimal/octal IPs, redirect-to-private) | tests green; Claude security review of the guard passes |
-| **P1** clean fetch | `/v2/scrape` (html/json/text → clean), limits, auth, logging; deployed on orion; talos wired | `web_extract` works on talos; contract tests pass for 0.19 and 0.21; guard blocks verified live from talos |
+| **P1** clean fetch | `/v2/scrape` (html/json/text → clean), limits, auth, logging; `sift` CLI client (`fetch --mode clean`, `--jq`) + `skill/SKILL.md` installed on hosts; deployed on orion; talos wired | `web_extract` works on talos; contract tests pass for 0.19 and 0.21; guard blocks verified live from talos |
 | **P2** digest + context | `/v1/fetch` modes, Dixie digest (thinking on vs off measured), Honcho search context, token metrics | on 50 real fetches: median `returned/raw` ≤ 0.2 with spot-checked quality; p90 latency ≤ 30 s with digest; no guardian-star access possible |
 | **P3** search | `/v2/search` + `/v1/search`: SearXNG + Brave LLM Context, dedupe, honcho-embed rerank | search quality spot-check vs SearXNG alone; cost per query known |
 | **P4** cache + provenance | SQLite cache with TTL; optional provenance (if Kyle approves) | cache hit rate reported; repeat fetches served locally |