P6 plan: asynchronous batch fetch endpoint — phase spec, task plan, given tests
Co-Authored-By: DeepSeek V4 Pro <noreply@deepseek.com>
This commit is contained in:
DeepSeek V4 Pro
+183
@@ -0,0 +1,183 @@
|
||||
# Sift P6 — asynchronous batch fetch
|
||||
|
||||
Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer:
|
||||
Claude (when it frees up this evening). Implementer: Kyle's call.
|
||||
|
||||
## 1. Scope
|
||||
|
||||
In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the
|
||||
same per-target options as `/v1/fetch`) and gets a job ID back immediately; it polls for per-target
|
||||
results later. One job drains **serially** — one target at a time — in the background.
|
||||
|
||||
Out:
|
||||
|
||||
- Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are
|
||||
already expected to take a long time, not for interactive fetches.
|
||||
- Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
|
||||
- Firecrawl compatibility. This is a native-only feature; no Hermes plugin.
|
||||
|
||||
The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a
|
||||
page, through the same SSRF guard and the same per-caller/host gates.
|
||||
|
||||
## 2. Why agents need this (2026-09-26)
|
||||
|
||||
`docs/PLAN.md` §13 (Kyle): "an agent that has collected a series of targets and wants them fetched
|
||||
without holding a request open per target." Today that means a loop of `sift fetch` calls — one
|
||||
request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole
|
||||
list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the
|
||||
`dcache` SQLite store, the `global`/`per` concurrency gates, and the `doFetch` pipeline.
|
||||
|
||||
## 3. Endpoints and wire shapes
|
||||
|
||||
| Method | Path | Result |
|
||||
|---|---|---|
|
||||
| `POST /v1/batch` | submit a list of targets | `202` created/reused, `400` malformed, `422` refused up front or over a cap, `429` caller/jobs busy |
|
||||
| `GET /v1/batch/{id}` | status + per-target results (inline) | `200`, `401`/`404` |
|
||||
| `DELETE /v1/batch/{id}` | cancel | `200`, `401`/`404` |
|
||||
|
||||
`POST /v1/batch` request — each target is the `/v1/fetch` field set minus the outer URL:
|
||||
|
||||
```json
|
||||
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
|
||||
"jq": "…", "context": true, "recover": false}, …]}
|
||||
```
|
||||
|
||||
Response `202`:
|
||||
```json
|
||||
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
|
||||
```
|
||||
|
||||
`GET /v1/batch/{id}`:
|
||||
```json
|
||||
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
|
||||
"created_at": "…", "targets": [
|
||||
{"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
|
||||
{"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
|
||||
]}
|
||||
```
|
||||
|
||||
Results are inline regardless of job status, once a target is finished. A `pending` target has
|
||||
`status: "pending"` and no `result`/`error`. `DELETE` → `200 {"success": true, "status":
|
||||
"cancelled"}`.
|
||||
|
||||
Job status vocabulary: `queued → running → done | partial | failed | cancelled`. Target status:
|
||||
`pending → running → done | failed`.
|
||||
|
||||
## 4. Storage (`dcache`)
|
||||
|
||||
Two tables, added to `internal/dcache` (which owns the single SQLite connection):
|
||||
|
||||
- `batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done,
|
||||
created_at, lease, nbytes)`
|
||||
- `batch_target(job_id, idx, url, options, status, result, error, hint)` — `result` is the JSON
|
||||
envelope, `NULL` until the target finishes.
|
||||
|
||||
Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, **not** by
|
||||
the content cache's LRU. `EvictBatchOldest` evicts whole jobs oldest-`lease`-first; a
|
||||
`queued`/`running` job is never byte-evicted. `MarkInFlightFailed(reason, now)` is the restart hook.
|
||||
|
||||
## 5. The runner (`internal/batch`)
|
||||
|
||||
A new package with the job lifecycle and the serial state machine. Persistence is the `*dcache.DB`;
|
||||
the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.
|
||||
|
||||
- **Idempotency (in v1).** A job has a random opaque `id` plus a derived `idem_key =
|
||||
SHA256(caller + "\x00" + canonical target list)`. Resubmitting the same (caller, targets) while
|
||||
the job's lease has not expired returns the existing job (`reused: true`) instead of a new one.
|
||||
Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
|
||||
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status
|
||||
(`queued`/`running`), not lease. Exceeding a cap is a typed error the API maps to `422`/`429`.
|
||||
- **Serial.** Targets drain in `idx` order; target `i+1` does not start until `i` is written.
|
||||
- **Failure isolation.** One target's error is stored per-target and the job continues; the job
|
||||
status is `done` (all ok), `failed` (all failed), or `partial` (mixed).
|
||||
- **Cancel.** `DELETE` flips the job `cancelled`, stops before the next target, and lets the
|
||||
in-flight target finish and store. Completed targets stay readable until the lease lapses.
|
||||
|
||||
## 6. Scheduling and fairness
|
||||
|
||||
The worker runs in the background, and every target still goes through the normal `acquire` gate
|
||||
(8 global, 2 per-host). On `429 busy` the worker **waits and retries** instead of failing — nobody
|
||||
is holding a socket open. Interactive requests are served by HTTP handlers that grab slots
|
||||
immediately, so they win by construction; the batch is lower priority for free.
|
||||
|
||||
## 7. Retention and TTL
|
||||
|
||||
A job carries a single sliding **lease**, bumped on submit, on every target completion, and on
|
||||
every status/collect read. All targets therefore share one expiry (the last completion's), and a
|
||||
slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease
|
||||
is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease
|
||||
lapses; `queued`/`running` jobs are marked `failed` (`restart`) on startup and never resumed.
|
||||
|
||||
## 8. Restricted callers
|
||||
|
||||
Compartmentalized-access systems: their usage is redacted as much as possible. The existing
|
||||
`logHost`/`logReason`/`logErr` machinery already blanks host, reason and error detail for
|
||||
`PermitRestricted` callers, and the digest/embed caches partition by caller class — the batch
|
||||
inherits all of it by passing the caller through. Two additions: batch log lines never carry a
|
||||
target URL or hostname for **any** caller, and `GET`/`DELETE` enforce job ownership, returning
|
||||
`404` (not `403`) for a job owned by another caller so a caller cannot probe for its existence.
|
||||
The `idem_key` is stored as a hash and never logged alongside its target list.
|
||||
|
||||
## 9. Caps and metrics (sized empirically)
|
||||
|
||||
The caps are starting values, instrumented so they can be re-sized from observed data:
|
||||
|
||||
- `sift_batch_jobs_total{outcome}` — `created|reused|done|partial|failed|cancelled|restart`
|
||||
- `sift_batch_targets_total{outcome}` — `ok|blocked|timeout|busy|upstream_4xx|upstream_5xx|
|
||||
unsupported|too_long|cancel`
|
||||
- `sift_batch_queue_depth`, `sift_batch_targets_inflight` (gauges)
|
||||
- `sift_batch_job_seconds`, `sift_batch_target_seconds`, `sift_batch_result_bytes` (histograms)
|
||||
|
||||
Both `outcome` counters fold any value outside their known set to `other`, so a raw string never
|
||||
becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).
|
||||
|
||||
## 10. CLI and skill
|
||||
|
||||
- `sift batch submit <targets.json>` (`-` reads stdin; prints `job_id` and `reused`).
|
||||
- `sift batch status <job_id>` (prints the full status body).
|
||||
- `sift batch cancel <job_id>`.
|
||||
- `skill/SKILL.md`: use `sift batch submit` for a list of targets instead of a loop of
|
||||
`sift fetch`; `sift batch status` to collect.
|
||||
- `[batch]` config block: `targets_per_job`, `jobs_per_caller`, `jobs_total`, `ttl`, `max_bytes`.
|
||||
The `/v1/batch` body cap (256 KiB, vs the 64 KiB global) is a constant, not config.
|
||||
|
||||
## 11. Refactor of existing code (explicit)
|
||||
|
||||
`v1fetch` today inlines the whole per-target pipeline (guard → `doFetch` → `recover` →
|
||||
render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is
|
||||
extracted into one shared method, `func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target)
|
||||
(batch.Result, error)`, that both `v1fetch` and the runner's `FetchFunc` call. `v1fetch` keeps its
|
||||
exact response shape and behavior — its existing tests must pass unchanged. This is the one change
|
||||
to working code in P6 and is called out for reviewer attention.
|
||||
|
||||
## 12. Acceptance (contract) tests
|
||||
|
||||
Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the
|
||||
intended reasons. See `docs/plans/p6/_files/`.
|
||||
|
||||
1. `dcache`: create/read round trip, idempotency lookup (lease expiry + caller scope), live-job
|
||||
counts, ordered targets, exactly-once `targets_done`/lease bump, sweep, byte-eviction
|
||||
(oldest-first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
|
||||
2. `batch` runner: serial order (peak in-flight 1), failure isolation (`partial`), idempotent
|
||||
resubmit (`reused`, no double-drain), the three caps, cancel (in-flight stored, later targets
|
||||
`pending`), panic recovery.
|
||||
3. `api`: submit `202` + idempotent resubmit; a guard-pre-check rejection (non-80/443 port) is
|
||||
`422` at submit; 33 targets `422`; a private-address target (passes pre-check, blocked at
|
||||
connect) is a per-target `failed`, aggregate `partial`; ownership (`GET` by another caller →
|
||||
`404`); cancel → `cancelled`.
|
||||
4. `metrics`: exact series names, label values, gauge values, histogram counts, and raw-outcome
|
||||
folding to `other`.
|
||||
5. `cmd/sift`: `parseBatchArgs` verb/arg pairs and error cases.
|
||||
|
||||
## 13. Rollout
|
||||
|
||||
1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the `v1fetch` refactor,
|
||||
and the caps/lease semantics.
|
||||
2. Orion pin bump with the `[batch]` config block; no `MemoryMax` change.
|
||||
3. Skill update ships with the pin bump.
|
||||
4. Watch `sift_batch_*` and tirith for a week; the success measure is that a research thread's
|
||||
one-request-per-URL fetch loop collapses into a single batch submit.
|
||||
|
||||
## 14. Changes after the plan was written
|
||||
|
||||
(none yet)
|
||||
Reference in New Issue
Block a user