# Sift P6 — asynchronous batch fetch

Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer:
Claude (when it frees up this evening). Implementer: Kyle's call.

## 1. Scope

In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the
same per-target options as `/v1/fetch`) and gets a job ID back immediately; it polls for per-target
results later. One job drains **serially** — one target at a time — in the background.

Out:

- Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are
  already expected to take a long time, not for interactive fetches.
- Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
- Firecrawl compatibility. This is a native-only feature; no Hermes plugin.

The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a
page, through the same SSRF guard and the same per-caller/host gates.

## 2. Why agents need this (2026-09-26)

`docs/PLAN.md` §13 (Kyle): "an agent that has collected a series of targets and wants them fetched
without holding a request open per target." Today that means a loop of `sift fetch` calls — one
request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole
list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the
`dcache` SQLite store, the `global`/`per` concurrency gates, and the `doFetch` pipeline.

## 3. Endpoints and wire shapes

| Method | Path | Result |
|---|---|---|
| `POST /v1/batch` | submit a list of targets | `202` created/reused, `400` malformed, `422` refused up front or over a cap, `429` caller/jobs busy |
| `GET /v1/batch/{id}` | status + per-target results (inline) | `200`, `401`/`404` |
| `DELETE /v1/batch/{id}` | cancel | `200`, `401`/`404` |

`POST /v1/batch` request — each target is the `/v1/fetch` field set minus the outer URL:

```json
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
              "jq": "…", "context": true, "recover": false}, …]}
```

Response `202`:
```json
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
```

`GET /v1/batch/{id}`:
```json
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
 "created_at": "…", "targets": [
   {"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
   {"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
 ]}
```

Results are inline regardless of job status, once a target is finished. A `pending` target has
`status: "pending"` and no `result`/`error`. `DELETE` → `200 {"success": true, "status":
"cancelled"}`.

Job status vocabulary: `queued → running → done | partial | failed | cancelled`. Target status:
`pending → running → done | failed`.

## 4. Storage (`dcache`)

Two tables, added to `internal/dcache` (which owns the single SQLite connection):

- `batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done,
  created_at, lease, nbytes)`
- `batch_target(job_id, idx, url, options, status, result, error, hint)` — `result` is the JSON
  envelope, `NULL` until the target finishes.

Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, **not** by
the content cache's LRU. `EvictBatchOldest` evicts whole jobs oldest-`lease`-first; a
`queued`/`running` job is never byte-evicted. `MarkInFlightFailed(reason, now)` is the restart hook.

## 5. The runner (`internal/batch`)

A new package with the job lifecycle and the serial state machine. Persistence is the `*dcache.DB`;
the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.

- **Idempotency (in v1).** A job has a random opaque `id` plus a derived `idem_key =
  SHA256(caller + "\x00" + canonical target list)`. Resubmitting the same (caller, targets) while
  the job's lease has not expired returns the existing job (`reused: true`) instead of a new one.
  Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status
  (`queued`/`running`), not lease. Exceeding a cap is a typed error the API maps to `422`/`429`.
- **Serial.** Targets drain in `idx` order; target `i+1` does not start until `i` is written.
- **Failure isolation.** One target's error is stored per-target and the job continues; the job
  status is `done` (all ok), `failed` (all failed), or `partial` (mixed).
- **Cancel.** `DELETE` flips the job `cancelled`, stops before the next target, and lets the
  in-flight target finish and store. Completed targets stay readable until the lease lapses.

## 6. Scheduling and fairness

The worker runs in the background, and every target still goes through the normal `acquire` gate
(8 global, 2 per-host). On `429 busy` the worker **waits and retries** instead of failing — nobody
is holding a socket open. Interactive requests are served by HTTP handlers that grab slots
immediately, so they win by construction; the batch is lower priority for free.

## 7. Retention and TTL

A job carries a single sliding **lease**, bumped on submit, on every target completion, and on
every status/collect read. All targets therefore share one expiry (the last completion's), and a
slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease
is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease
lapses; `queued`/`running` jobs are marked `failed` (`restart`) on startup and never resumed.

## 8. Restricted callers

Compartmentalized-access systems: their usage is redacted as much as possible. The existing
`logHost`/`logReason`/`logErr` machinery already blanks host, reason and error detail for
`PermitRestricted` callers, and the digest/embed caches partition by caller class — the batch
inherits all of it by passing the caller through. Two additions: batch log lines never carry a
target URL or hostname for **any** caller, and `GET`/`DELETE` enforce job ownership, returning
`404` (not `403`) for a job owned by another caller so a caller cannot probe for its existence.
The `idem_key` is stored as a hash and never logged alongside its target list.

## 9. Caps and metrics (sized empirically)

The caps are starting values, instrumented so they can be re-sized from observed data:

- `sift_batch_jobs_total{outcome}` — `created|reused|done|partial|failed|cancelled|restart`
- `sift_batch_targets_total{outcome}` — `ok|blocked|timeout|busy|upstream_4xx|upstream_5xx|
  unsupported|too_long|cancel`
- `sift_batch_queue_depth`, `sift_batch_targets_inflight` (gauges)
- `sift_batch_job_seconds`, `sift_batch_target_seconds`, `sift_batch_result_bytes` (histograms)

Both `outcome` counters fold any value outside their known set to `other`, so a raw string never
becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).

## 10. CLI and skill

- `sift batch submit <targets.json>` (`-` reads stdin; prints `job_id` and `reused`).
- `sift batch status <job_id>` (prints the full status body).
- `sift batch cancel <job_id>`.
- `skill/SKILL.md`: use `sift batch submit` for a list of targets instead of a loop of
  `sift fetch`; `sift batch status` to collect.
- `[batch]` config block: `targets_per_job`, `jobs_per_caller`, `jobs_total`, `ttl`, `max_bytes`.
  The `/v1/batch` body cap (256 KiB, vs the 64 KiB global) is a constant, not config.

## 11. Refactor of existing code (explicit)

`v1fetch` today inlines the whole per-target pipeline (guard → `doFetch` → `recover` →
render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is
extracted into one shared method, `func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target)
(batch.Result, error)`, that both `v1fetch` and the runner's `FetchFunc` call. `v1fetch` keeps its
exact response shape and behavior — its existing tests must pass unchanged. This is the one change
to working code in P6 and is called out for reviewer attention.

## 12. Acceptance (contract) tests

Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the
intended reasons. See `docs/plans/p6/_files/`.

1. `dcache`: create/read round trip, idempotency lookup (lease expiry + caller scope), live-job
   counts, ordered targets, exactly-once `targets_done`/lease bump, sweep, byte-eviction
   (oldest-first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
2. `batch` runner: serial order (peak in-flight 1), failure isolation (`partial`), idempotent
   resubmit (`reused`, no double-drain), the three caps, cancel (in-flight stored, later targets
   `pending`), panic recovery.
3. `api`: submit `202` + idempotent resubmit; a guard-pre-check rejection (non-80/443 port) is
   `422` at submit; 33 targets `422`; a private-address target (passes pre-check, blocked at
   connect) is a per-target `failed`, aggregate `partial`; ownership (`GET` by another caller →
   `404`); cancel → `cancelled`.
4. `metrics`: exact series names, label values, gauge values, histogram counts, and raw-outcome
   folding to `other`.
5. `cmd/sift`: `parseBatchArgs` verb/arg pairs and error cases.

## 13. Rollout

1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the `v1fetch` refactor,
   and the caps/lease semantics.
2. Orion pin bump with the `[batch]` config block; no `MemoryMax` change.
3. Skill update ships with the pin bump.
4. Watch `sift_batch_*` and tirith for a week; the success measure is that a research thread's
   one-request-per-URL fetch loop collapses into a single batch submit.

## 14. Changes after the plan was written

(none yet)