47 lines
3.2 KiB
Markdown
47 lines
3.2 KiB
Markdown
# P6 implementation plan: asynchronous batch fetch
|
|
|
|
> **For the implementing model:** do not work from this file. The owner gives you one task file
|
|
> at a time. This file is the index for the owner and the reviewer.
|
|
|
|
**Goal:** an agent that has collected a series of targets (URLs) and wants them fetched without
|
|
holding a request open per target submits them as one batch, gets a job ID back immediately, and
|
|
polls for per-target results later. A batch is **deliberate and slow** — serial, one target at a
|
|
time — so it is the tool for research threads, not for interactive fetches.
|
|
|
|
Decisions (settled with the owner, 2026-09-26):
|
|
|
|
- **Serial.** One target drains at a time per job, in the background. Each target still goes
|
|
through the normal `acquire` gate (8 global / 2 per-host); on `429 busy` the worker waits and
|
|
retries rather than failing, because nobody is holding a socket open. Interactive requests win
|
|
by construction: they are served by HTTP handlers that grab slots immediately.
|
|
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8; per-target result bounded by
|
|
the output cap. All instrumented (`sift_batch_*`) so the numbers can be re-sized empirically.
|
|
- **Idempotency in v1.** A job has a random opaque `id` plus a derived `idem_key`; resubmitting
|
|
the same (caller, targets) while the job is live returns the existing job instead of a new one.
|
|
- **Storage.** `batch_job` + `batch_target` tables in the existing `dcache` SQLite.
|
|
- **Retention / TTL.** A job carries a single sliding **lease**, bumped on submit, on every target
|
|
completion, and on every status/collect read. All targets therefore share one expiry, and a slow
|
|
consumer's reads keep a finished batch alive. GC deletes a job (and its targets) once its lease
|
|
is older than the TTL. Batch storage is a coherent whole: it is bounded by the lease GC plus a
|
|
separate byte budget (oldest-lease jobs evicted whole), not by the content cache's LRU.
|
|
- **Restart.** Completed results survive restart and stay readable until the lease lapses.
|
|
`queued`/`running` jobs are marked `failed` (reason `restart`) on startup; never resumed.
|
|
- **Restricted callers.** Compartmentalized-access systems: their usage is redacted as much as
|
|
possible (host, error reason, error detail — the existing `logHost`/`logReason`/`logErr`
|
|
machinery), and no other caller may read their jobs or results. The worker passes the caller
|
|
through, so this happens automatically; batch log lines never carry a target URL or hostname.
|
|
|
|
| Task | Package | Given |
|
|
|---|---|---|
|
|
| 01-batchstore | `internal/dcache` | `dcache/batch_test.go` |
|
|
| 02-runner | `internal/batch` | `batch/runner_test.go` |
|
|
| 03-api | `internal/api` | `api/batch_test.go` |
|
|
| 04-metrics | `internal/metrics` | `metrics/batch_test.go` |
|
|
| 05-cli-docs | `cmd/sift`, `README.md`, `skill/SKILL.md`, config | `cmd/sift/batch_test.go` |
|
|
|
|
**How this plan was made:** design worked through with the owner from `docs/PLAN.md` §13 (future
|
|
work) on top of the P5 pool (`internal/pdfpool`), cache (`internal/dcache`), and gate
|
|
(`global`/`per` in `internal/api`) infrastructure. Acceptance tests first: the given tests are
|
|
authored against this spec and compiled against a panic-only skeleton so they fail for the
|
|
intended reasons before each task is implemented.
|