sift/docs/plans/p6

P6 implementation plan: asynchronous batch fetch

For the implementing model: do not work from this file. The owner gives you one task file at a time. This file is the index for the owner and the reviewer.

Goal: an agent that has collected a series of targets (URLs) and wants them fetched without holding a request open per target submits them as one batch, gets a job ID back immediately, and polls for per-target results later. A batch is deliberate and slow — serial, one target at a time — so it is the tool for research threads, not for interactive fetches.

Decisions (settled with the owner, 2026-09-26):

• Serial. One target drains at a time per job, in the background. Each target still goes through the normal acquire gate (8 global / 2 per-host); on 429 busy the worker waits and retries rather than failing, because nobody is holding a socket open. Interactive requests win by construction: they are served by HTTP handlers that grab slots immediately.
• Caps. targets/job 32; live jobs/caller 1; live jobs total 8; per-target result bounded by the output cap. All instrumented (sift_batch_*) so the numbers can be re-sized empirically.
• Idempotency in v1. A job has a random opaque id plus a derived idem_key; resubmitting the same (caller, targets) while the job is live returns the existing job instead of a new one.
• Storage. batch_job + batch_target tables in the existing dcache SQLite.
• Retention / TTL. A job carries a single sliding lease, bumped on submit, on every target completion, and on every status/collect read. All targets therefore share one expiry, and a slow consumer's reads keep a finished batch alive. GC deletes a job (and its targets) once its lease is older than the TTL. Batch storage is a coherent whole: it is bounded by the lease GC plus a separate byte budget (oldest-lease jobs evicted whole), not by the content cache's LRU.
• Restart. Completed results survive restart and stay readable until the lease lapses. queued/running jobs are marked failed (reason restart) on startup; never resumed.
• Restricted callers. Compartmentalized-access systems: their usage is redacted as much as possible (host, error reason, error detail — the existing logHost/logReason/logErr machinery), and no other caller may read their jobs or results. The worker passes the caller through, so this happens automatically; batch log lines never carry a target URL or hostname.

Task	Package	Given
01-batchstore	internal/dcache	dcache/batch_test.go
02-runner	internal/batch	batch/runner_test.go
03-api	internal/api	api/batch_test.go
04-metrics	internal/metrics	metrics/batch_test.go
05-cli-docs	cmd/sift, README.md, skill/SKILL.md, config	cmd/sift/batch_test.go

How this plan was made: design worked through with the owner from docs/PLAN.md §13 (future work) on top of the P5 pool (internal/pdfpool), cache (internal/dcache), and gate (global/per in internal/api) infrastructure. Acceptance tests first: the given tests are authored against this spec and compiled against a panic-only skeleton so they fail for the intended reasons before each task is implemented.