# P6 implementation plan: asynchronous batch fetch

> **For the implementing model:** do not work from this file. The owner gives you one task file
> at a time. This file is the index for the owner and the reviewer.

**Goal:** an agent that has collected a series of targets (URLs) and wants them fetched without
holding a request open per target submits them as one batch, gets a job ID back immediately, and
polls for per-target results later. A batch is **deliberate and slow** — serial, one target at a
time — so it is the tool for research threads, not for interactive fetches.

Decisions (settled with the owner, 2026-09-26):

- **Serial.** One target drains at a time per job, in the background. Each target still goes
  through the normal `acquire` gate (8 global / 2 per-host); on `429 busy` the worker waits and
  retries rather than failing, because nobody is holding a socket open. Interactive requests win
  by construction: they are served by HTTP handlers that grab slots immediately.
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8; per-target result bounded by
  the output cap. All instrumented (`sift_batch_*`) so the numbers can be re-sized empirically.
- **Idempotency in v1.** A job has a random opaque `id` plus a derived `idem_key`; resubmitting
  the same (caller, targets) while the job is live returns the existing job instead of a new one.
- **Storage.** `batch_job` + `batch_target` tables in the existing `dcache` SQLite.
- **Retention / TTL.** A job carries a single sliding **lease**, bumped on submit, on every target
  completion, and on every status/collect read. All targets therefore share one expiry, and a slow
  consumer's reads keep a finished batch alive. GC deletes a job (and its targets) once its lease
  is older than the TTL. Batch storage is a coherent whole: it is bounded by the lease GC plus a
  separate byte budget (oldest-lease jobs evicted whole), not by the content cache's LRU.
- **Restart.** Completed results survive restart and stay readable until the lease lapses.
  `queued`/`running` jobs are marked `failed` (reason `restart`) on startup; never resumed.
- **Restricted callers.** Compartmentalized-access systems: their usage is redacted as much as
  possible (host, error reason, error detail — the existing `logHost`/`logReason`/`logErr`
  machinery), and no other caller may read their jobs or results. The worker passes the caller
  through, so this happens automatically; batch log lines never carry a target URL or hostname.

| Task | Package | Given |
|---|---|---|
| 01-batchstore | `internal/dcache` | `dcache/batch_test.go` |
| 02-runner | `internal/batch` | `batch/runner_test.go` |
| 03-api | `internal/api` | `api/batch_test.go` |
| 04-metrics | `internal/metrics` | `metrics/batch_test.go` |
| 05-cli-docs | `cmd/sift`, `README.md`, `skill/SKILL.md`, config | `cmd/sift/batch_test.go` |

**How this plan was made:** design worked through with the owner from `docs/PLAN.md` §13 (future
work) on top of the P5 pool (`internal/pdfpool`), cache (`internal/dcache`), and gate
(`global`/`per` in `internal/api`) infrastructure. Acceptance tests first: the given tests are
authored against this spec and compiled against a panic-only skeleton so they fail for the
intended reasons before each task is implemented.