184 lines
9.7 KiB
Markdown
184 lines
9.7 KiB
Markdown
# Sift P6 — asynchronous batch fetch
|
|
|
|
Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer:
|
|
Claude (when it frees up this evening). Implementer: Kyle's call.
|
|
|
|
## 1. Scope
|
|
|
|
In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the
|
|
same per-target options as `/v1/fetch`) and gets a job ID back immediately; it polls for per-target
|
|
results later. One job drains **serially** — one target at a time — in the background.
|
|
|
|
Out:
|
|
|
|
- Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are
|
|
already expected to take a long time, not for interactive fetches.
|
|
- Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
|
|
- Firecrawl compatibility. This is a native-only feature; no Hermes plugin.
|
|
|
|
The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a
|
|
page, through the same SSRF guard and the same per-caller/host gates.
|
|
|
|
## 2. Why agents need this (2026-09-26)
|
|
|
|
`docs/PLAN.md` §13 (Kyle): "an agent that has collected a series of targets and wants them fetched
|
|
without holding a request open per target." Today that means a loop of `sift fetch` calls — one
|
|
request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole
|
|
list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the
|
|
`dcache` SQLite store, the `global`/`per` concurrency gates, and the `doFetch` pipeline.
|
|
|
|
## 3. Endpoints and wire shapes
|
|
|
|
| Method | Path | Result |
|
|
|---|---|---|
|
|
| `POST /v1/batch` | submit a list of targets | `202` created/reused, `400` malformed, `422` refused up front or over a cap, `429` caller/jobs busy |
|
|
| `GET /v1/batch/{id}` | status + per-target results (inline) | `200`, `401`/`404` |
|
|
| `DELETE /v1/batch/{id}` | cancel | `200`, `401`/`404` |
|
|
|
|
`POST /v1/batch` request — each target is the `/v1/fetch` field set minus the outer URL:
|
|
|
|
```json
|
|
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
|
|
"jq": "…", "context": true, "recover": false}, …]}
|
|
```
|
|
|
|
Response `202`:
|
|
```json
|
|
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
|
|
```
|
|
|
|
`GET /v1/batch/{id}`:
|
|
```json
|
|
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
|
|
"created_at": "…", "targets": [
|
|
{"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
|
|
{"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
|
|
]}
|
|
```
|
|
|
|
Results are inline regardless of job status, once a target is finished. A `pending` target has
|
|
`status: "pending"` and no `result`/`error`. `DELETE` → `200 {"success": true, "status":
|
|
"cancelled"}`.
|
|
|
|
Job status vocabulary: `queued → running → done | partial | failed | cancelled`. Target status:
|
|
`pending → running → done | failed`.
|
|
|
|
## 4. Storage (`dcache`)
|
|
|
|
Two tables, added to `internal/dcache` (which owns the single SQLite connection):
|
|
|
|
- `batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done,
|
|
created_at, lease, nbytes)`
|
|
- `batch_target(job_id, idx, url, options, status, result, error, hint)` — `result` is the JSON
|
|
envelope, `NULL` until the target finishes.
|
|
|
|
Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, **not** by
|
|
the content cache's LRU. `EvictBatchOldest` evicts whole jobs oldest-`lease`-first; a
|
|
`queued`/`running` job is never byte-evicted. `MarkInFlightFailed(reason, now)` is the restart hook.
|
|
|
|
## 5. The runner (`internal/batch`)
|
|
|
|
A new package with the job lifecycle and the serial state machine. Persistence is the `*dcache.DB`;
|
|
the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.
|
|
|
|
- **Idempotency (in v1).** A job has a random opaque `id` plus a derived `idem_key =
|
|
SHA256(caller + "\x00" + canonical target list)`. Resubmitting the same (caller, targets) while
|
|
the job's lease has not expired returns the existing job (`reused: true`) instead of a new one.
|
|
Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
|
|
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status
|
|
(`queued`/`running`), not lease. Exceeding a cap is a typed error the API maps to `422`/`429`.
|
|
- **Serial.** Targets drain in `idx` order; target `i+1` does not start until `i` is written.
|
|
- **Failure isolation.** One target's error is stored per-target and the job continues; the job
|
|
status is `done` (all ok), `failed` (all failed), or `partial` (mixed).
|
|
- **Cancel.** `DELETE` flips the job `cancelled`, stops before the next target, and lets the
|
|
in-flight target finish and store. Completed targets stay readable until the lease lapses.
|
|
|
|
## 6. Scheduling and fairness
|
|
|
|
The worker runs in the background, and every target still goes through the normal `acquire` gate
|
|
(8 global, 2 per-host). On `429 busy` the worker **waits and retries** instead of failing — nobody
|
|
is holding a socket open. Interactive requests are served by HTTP handlers that grab slots
|
|
immediately, so they win by construction; the batch is lower priority for free.
|
|
|
|
## 7. Retention and TTL
|
|
|
|
A job carries a single sliding **lease**, bumped on submit, on every target completion, and on
|
|
every status/collect read. All targets therefore share one expiry (the last completion's), and a
|
|
slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease
|
|
is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease
|
|
lapses; `queued`/`running` jobs are marked `failed` (`restart`) on startup and never resumed.
|
|
|
|
## 8. Restricted callers
|
|
|
|
Compartmentalized-access systems: their usage is redacted as much as possible. The existing
|
|
`logHost`/`logReason`/`logErr` machinery already blanks host, reason and error detail for
|
|
`PermitRestricted` callers, and the digest/embed caches partition by caller class — the batch
|
|
inherits all of it by passing the caller through. Two additions: batch log lines never carry a
|
|
target URL or hostname for **any** caller, and `GET`/`DELETE` enforce job ownership, returning
|
|
`404` (not `403`) for a job owned by another caller so a caller cannot probe for its existence.
|
|
The `idem_key` is stored as a hash and never logged alongside its target list.
|
|
|
|
## 9. Caps and metrics (sized empirically)
|
|
|
|
The caps are starting values, instrumented so they can be re-sized from observed data:
|
|
|
|
- `sift_batch_jobs_total{outcome}` — `created|reused|done|partial|failed|cancelled|restart`
|
|
- `sift_batch_targets_total{outcome}` — `ok|blocked|timeout|busy|upstream_4xx|upstream_5xx|
|
|
unsupported|too_long|cancel`
|
|
- `sift_batch_queue_depth`, `sift_batch_targets_inflight` (gauges)
|
|
- `sift_batch_job_seconds`, `sift_batch_target_seconds`, `sift_batch_result_bytes` (histograms)
|
|
|
|
Both `outcome` counters fold any value outside their known set to `other`, so a raw string never
|
|
becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).
|
|
|
|
## 10. CLI and skill
|
|
|
|
- `sift batch submit <targets.json>` (`-` reads stdin; prints `job_id` and `reused`).
|
|
- `sift batch status <job_id>` (prints the full status body).
|
|
- `sift batch cancel <job_id>`.
|
|
- `skill/SKILL.md`: use `sift batch submit` for a list of targets instead of a loop of
|
|
`sift fetch`; `sift batch status` to collect.
|
|
- `[batch]` config block: `targets_per_job`, `jobs_per_caller`, `jobs_total`, `ttl`, `max_bytes`.
|
|
The `/v1/batch` body cap (256 KiB, vs the 64 KiB global) is a constant, not config.
|
|
|
|
## 11. Refactor of existing code (explicit)
|
|
|
|
`v1fetch` today inlines the whole per-target pipeline (guard → `doFetch` → `recover` →
|
|
render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is
|
|
extracted into one shared method, `func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target)
|
|
(batch.Result, error)`, that both `v1fetch` and the runner's `FetchFunc` call. `v1fetch` keeps its
|
|
exact response shape and behavior — its existing tests must pass unchanged. This is the one change
|
|
to working code in P6 and is called out for reviewer attention.
|
|
|
|
## 12. Acceptance (contract) tests
|
|
|
|
Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the
|
|
intended reasons. See `docs/plans/p6/_files/`.
|
|
|
|
1. `dcache`: create/read round trip, idempotency lookup (lease expiry + caller scope), live-job
|
|
counts, ordered targets, exactly-once `targets_done`/lease bump, sweep, byte-eviction
|
|
(oldest-first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
|
|
2. `batch` runner: serial order (peak in-flight 1), failure isolation (`partial`), idempotent
|
|
resubmit (`reused`, no double-drain), the three caps, cancel (in-flight stored, later targets
|
|
`pending`), panic recovery.
|
|
3. `api`: submit `202` + idempotent resubmit; a guard-pre-check rejection (non-80/443 port) is
|
|
`422` at submit; 33 targets `422`; a private-address target (passes pre-check, blocked at
|
|
connect) is a per-target `failed`, aggregate `partial`; ownership (`GET` by another caller →
|
|
`404`); cancel → `cancelled`.
|
|
4. `metrics`: exact series names, label values, gauge values, histogram counts, and raw-outcome
|
|
folding to `other`.
|
|
5. `cmd/sift`: `parseBatchArgs` verb/arg pairs and error cases.
|
|
|
|
## 13. Rollout
|
|
|
|
1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the `v1fetch` refactor,
|
|
and the caps/lease semantics.
|
|
2. Orion pin bump with the `[batch]` config block; no `MemoryMax` change.
|
|
3. Skill update ships with the pin bump.
|
|
4. Watch `sift_batch_*` and tirith for a week; the success measure is that a research thread's
|
|
one-request-per-URL fetch loop collapses into a single batch submit.
|
|
|
|
## 14. Changes after the plan was written
|
|
|
|
(none yet)
|