sift/docs/plans/p6/README.md

# P6 implementation plan: asynchronous batch fetch

> **For the implementing model:** do not work from this file. The owner gives you one task file
> at a time. This file is the index for the owner and the reviewer.

**Goal:** an agent that has collected a series of targets (URLs) and wants them fetched without
holding a request open per target submits them as one batch, gets a job ID back immediately, and
polls for per-target results later. A batch is **deliberate and slow** — serial, one target at a
time — so it is the tool for research threads, not for interactive fetches.

Decisions (settled with the owner, 2026-09-26; revised after review the same evening, see
`docs/P6.md` §14):

- **Serial per job, few workers.** One worker by default (at most 2) drains all live jobs,
  round-robin, one target at a time. The batch therefore holds at most 1–2 of the 8 global fetch
  slots, so interactive requests are never starved. A target refused busy (fetch gate or PDF
  pool) is retried after a pause, for up to 10 minutes, then failed.
- **The caller travels with the job.** The runner keeps each job's `*auth.Caller` in memory and
  hands it to every fetch, so per-caller gates, the PDF pool, the digest cache class and
  restricted-caller redaction all apply. Nothing about the caller is persisted beyond its name
  and class: jobs never resume after a restart.
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8; body 256 KiB. All instrumented
  (`sift_batch_*`).
- **Idempotency in v1.** The same (caller, target list) returns the existing job while it is live or
  has results (`queued`/`running`/`done`/`partial`) and its lease holds. A failed or cancelled job
  is not reused.
- **Storage.** `batch_job` + `batch_target` tables in the existing `dcache` SQLite, with its own
  byte budget (64 MiB), outside the content cache's LRU.
- **Retention / TTL.** One sliding lease per job, bumped on submit, on every target write, and on
  every owner read (the API calls `Touch`; store reads never write). GC deletes a job when its
  lease is older than the TTL (24 h).
- **Reads.** `GET /v1/batch/{id}` is a summary (statuses, errors). A result is read one at a time
  (`GET /v1/batch/{id}/{idx}`), or all at once with `?results=1`. 32 inline results can be 6 MB.
- **Restart.** `queued`/`running` jobs are marked `failed` (`restart`) on startup; never resumed.
- **Restricted callers.** Their fetches go through the same `fetchTarget` with the real caller,
  so the existing redaction applies. No caller can see another's job (404, never 403). Batch
  log lines never carry a target URL or hostname.

| Task | Package | Given |
|---|---|---|
| 01-batchstore | `internal/dcache` | `dcache/batch_test.go` |
| 02-runner-core | `internal/batch` (types, Submit, Status, Touch, Cancel; no workers) | `batch/runner_core_test.go` |
| 03-runner-workers | `internal/batch` (workers, retry, cancel in flight, hooks) | `batch/runner_test.go` |
| 04-metrics | `internal/metrics` | `metrics/batch_test.go` |
| 05-fetchtarget | `internal/api` (the one refactor of working code) | `api/fetchtarget_test.go` |
| 06-batch-wiring | `internal/api` | `api/batchrun_test.go` |
| 07-api-endpoints | `internal/api` | `api/batch_test.go` |
| 08-serve-config | `internal/config`, `cmd/sift/serve.go` | `config/batch_test.go` |
| 09-cli-docs | `cmd/sift`, README, skill | `cmd_sift/batch_test.go` |

`guard.PreCheck` and its test are on master (owner commit): `AGENTS.md` forbids the implementer
to edit `internal/guard/`, the security boundary.

Given files live under `_files/<package>/` (`cmd_sift` is `cmd/sift`); each task says where to
copy its file. Every given test was compiled against a panic-only skeleton of the whole plan and
fails there on `panic: todo`, not on a compile error.

**How this plan was made:** design worked through with the owner from `docs/PLAN.md` §13 (future
work) on top of the P5 pool (`internal/pdfpool`), cache (`internal/dcache`), and gate
(`global`/`per` in `internal/api`) infrastructure. Acceptance tests first: the given tests are
authored against this spec and compiled against a panic-only skeleton so they fail for the
intended reasons before each task is implemented.

## Changes during the run

- **Task 02 (runner), first attempt, 20:39–21:08. Model and task size, in about equal parts.**
  - The session wrote the whole runner out as ~700 lines of reasoning in its reply, never in a
    file. OpenCode compacted the context, and the model then emitted a malformed tool call
    (`</tool_call>`) and ended with no commit and no file.
  - It had also looked for the given test at a wrong path. The task said `_files/…` relative
    to the plan; that is my ambiguity.
  - The owner changes:
    - task 02 split into **02-runner-core** (no goroutines) and **03-runner-workers**, with the
      given test split to match (`runner_core_test.go` + `runner_test.go`); later tasks
      renumbered 04–09;
    - every copy path is spelled from the repo root (`docs/plans/p6/_files/…`);
    - `AGENTS.md` rule 6: write code into files early, in small steps;
    - `guard.PreCheck` moved to an owner commit on master. The old task 05 (now 06) told the
      implementer to edit `internal/guard/`, which `AGENTS.md` forbids: my contradiction, found
      while fixing the above.
- **Task 03 (workers), first attempt, 21:31–21:48.** Same mode again: the model deliberated over
  the concurrency design in one reply, which ended mid-sentence. That was probably the 32k
  output cap, and a reply without a tool call ends `opencode run`. No file was written.
  `AGENTS.md` rule 6 alone did not prevent it. The owner changes: task 03 gains a **Design**
  section that decides the concurrency (state, the worker loop, Cancel, Close), so the task is
  transcription, not design. Task 02 (core) passed first time (`e2ddb2e`).