sift/docs/plans/p6/README.md

P6 implementation plan: asynchronous batch fetch

For the implementing model: do not work from this file. The owner gives you one task file at a time. This file is the index for the owner and the reviewer.

Goal: an agent that has collected a series of targets (URLs) and wants them fetched without holding a request open per target submits them as one batch, gets a job ID back immediately, and polls for per-target results later. A batch is deliberate and slow — serial, one target at a time — so it is the tool for research threads, not for interactive fetches.

Decisions (settled with the owner, 2026-09-26; revised after review the same evening, see docs/P6.md §14):

• Serial per job, few workers. One worker by default (at most 2) drains all live jobs, round-robin, one target at a time. The batch therefore holds at most 1–2 of the 8 global fetch slots, so interactive requests are never starved. A target refused busy (fetch gate or PDF pool) is retried after a pause, for up to 10 minutes, then failed.
• The caller travels with the job. The runner keeps each job's *auth.Caller in memory and hands it to every fetch, so per-caller gates, the PDF pool, the digest cache class and restricted-caller redaction all apply. Nothing about the caller is persisted beyond its name and class: jobs never resume after a restart.
• Caps. targets/job 32; live jobs/caller 1; live jobs total 8; body 256 KiB. All instrumented (sift_batch_*).
• Idempotency in v1. The same (caller, target list) returns the existing job while it is live or has results (queued/running/done/partial) and its lease holds. A failed or cancelled job is not reused.
• Storage. batch_job + batch_target tables in the existing dcache SQLite, with its own byte budget (64 MiB), outside the content cache's LRU.
• Retention / TTL. One sliding lease per job, bumped on submit, on every target write, and on every owner read (the API calls Touch; store reads never write). GC deletes a job when its lease is older than the TTL (24 h).
• Reads. GET /v1/batch/{id} is a summary (statuses, errors). A result is read one at a time (GET /v1/batch/{id}/{idx}), or all at once with ?results=1. 32 inline results can be 6 MB.
• Restart. queued/running jobs are marked failed (restart) on startup; never resumed.
• Restricted callers. Their fetches go through the same fetchTarget with the real caller, so the existing redaction applies. No caller can see another's job (404, never 403). Batch log lines never carry a target URL or hostname.

Task	Package	Given
01-batchstore	internal/dcache	dcache/batch_test.go
02-runner-core	internal/batch (types, Submit, Status, Touch, Cancel; no workers)	batch/runner_core_test.go
03-runner-workers	internal/batch (workers, retry, cancel in flight, hooks)	batch/runner_test.go
04-metrics	internal/metrics	metrics/batch_test.go
05-fetchtarget	internal/api (the one refactor of working code)	api/fetchtarget_test.go
06-batch-wiring	internal/api	api/batchrun_test.go
07-api-endpoints	internal/api	api/batch_test.go
08-serve-config	internal/config, cmd/sift/serve.go	config/batch_test.go
09-cli-docs	cmd/sift, README, skill	cmd_sift/batch_test.go

guard.PreCheck and its test are on master (owner commit): AGENTS.md forbids the implementer to edit internal/guard/, the security boundary.

Given files live under _files/<package>/ (cmd_sift is cmd/sift); each task says where to copy its file. Every given test was compiled against a panic-only skeleton of the whole plan and fails there on panic: todo, not on a compile error.

How this plan was made: design worked through with the owner from docs/PLAN.md §13 (future work) on top of the P5 pool (internal/pdfpool), cache (internal/dcache), and gate (global/per in internal/api) infrastructure. Acceptance tests first: the given tests are authored against this spec and compiled against a panic-only skeleton so they fail for the intended reasons before each task is implemented.

Changes during the run

• Task 02 (runner), first attempt, 20:39–21:08. Model and task size, in about equal parts.
  • The session wrote the whole runner out as ~700 lines of reasoning in its reply, never in a file. OpenCode compacted the context, and the model then emitted a malformed tool call (</tool_call>) and ended with no commit and no file.
  • It had also looked for the given test at a wrong path. The task said _files/… relative to the plan; that is my ambiguity.
  • The owner changes:
    • task 02 split into 02-runner-core (no goroutines) and 03-runner-workers, with the given test split to match (runner_core_test.go + runner_test.go); later tasks renumbered 04–09;
    • every copy path is spelled from the repo root (docs/plans/p6/_files/…);
    • AGENTS.md rule 6: write code into files early, in small steps;
    • guard.PreCheck moved to an owner commit on master. The old task 05 (now 06) told the implementer to edit internal/guard/, which AGENTS.md forbids: my contradiction, found while fixing the above.