kyleandClaude Opus 5.5
103ec3d05a
Task 02 (the whole runner) ended at 21:08 with no file and no commit: Ornith wrote the
implementation out in its reply (~700 lines), OpenCode compacted the session, and the model
then emitted a malformed tool call. It had also looked for the given test at a wrong path
("_files/…" relative to the plan: my ambiguity).
- 02-runner → 02-runner-core (types, Submit, Status, Touch, Cancel; no goroutines) and
03-runner-workers; the given test split to match (runner_core_test.go gains three tests:
queued storage, cancel while queued, Status never moves the lease); later tasks renumbered
04-09 and every cross-reference checked.
- Copy paths spelled from the repo root.
- guard.PreCheck (+ its test) is committed here by the owner: the old task 05 told the
implementer to edit internal/guard/, which AGENTS.md forbids. My contradiction.
- AGENTS.md rule 6: write code into files early, in small steps.
Both runner test files compile against a panic-only skeleton (the core file also alone) and
fail there on panic: todo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
82 lines
5.2 KiB
Markdown
82 lines
5.2 KiB
Markdown
# P6 implementation plan: asynchronous batch fetch
|
||
|
||
> **For the implementing model:** do not work from this file. The owner gives you one task file
|
||
> at a time. This file is the index for the owner and the reviewer.
|
||
|
||
**Goal:** an agent that has collected a series of targets (URLs) and wants them fetched without
|
||
holding a request open per target submits them as one batch, gets a job ID back immediately, and
|
||
polls for per-target results later. A batch is **deliberate and slow** — serial, one target at a
|
||
time — so it is the tool for research threads, not for interactive fetches.
|
||
|
||
Decisions (settled with the owner, 2026-09-26; revised after review the same evening, see
|
||
`docs/P6.md` §14):
|
||
|
||
- **Serial per job, few workers.** One worker by default (at most 2) drains all live jobs,
|
||
round-robin, one target at a time. The batch therefore holds at most 1–2 of the 8 global fetch
|
||
slots, so interactive requests are never starved. A target refused busy (fetch gate or PDF
|
||
pool) is retried after a pause, for up to 10 minutes, then failed.
|
||
- **The caller travels with the job.** The runner keeps each job's `*auth.Caller` in memory and
|
||
hands it to every fetch, so per-caller gates, the PDF pool, the digest cache class and
|
||
restricted-caller redaction all apply. Nothing about the caller is persisted beyond its name
|
||
and class: jobs never resume after a restart.
|
||
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8; body 256 KiB. All instrumented
|
||
(`sift_batch_*`).
|
||
- **Idempotency in v1.** The same (caller, target list) returns the existing job while it is live or
|
||
has results (`queued`/`running`/`done`/`partial`) and its lease holds. A failed or cancelled job
|
||
is not reused.
|
||
- **Storage.** `batch_job` + `batch_target` tables in the existing `dcache` SQLite, with its own
|
||
byte budget (64 MiB), outside the content cache's LRU.
|
||
- **Retention / TTL.** One sliding lease per job, bumped on submit, on every target write, and on
|
||
every owner read (the API calls `Touch`; store reads never write). GC deletes a job when its
|
||
lease is older than the TTL (24 h).
|
||
- **Reads.** `GET /v1/batch/{id}` is a summary (statuses, errors). A result is read one at a time
|
||
(`GET /v1/batch/{id}/{idx}`), or all at once with `?results=1`. 32 inline results can be 6 MB.
|
||
- **Restart.** `queued`/`running` jobs are marked `failed` (`restart`) on startup; never resumed.
|
||
- **Restricted callers.** Their fetches go through the same `fetchTarget` with the real caller,
|
||
so the existing redaction applies. No caller can see another's job (404, never 403). Batch
|
||
log lines never carry a target URL or hostname.
|
||
|
||
| Task | Package | Given |
|
||
|---|---|---|
|
||
| 01-batchstore | `internal/dcache` | `dcache/batch_test.go` |
|
||
| 02-runner-core | `internal/batch` (types, Submit, Status, Touch, Cancel; no workers) | `batch/runner_core_test.go` |
|
||
| 03-runner-workers | `internal/batch` (workers, retry, cancel in flight, hooks) | `batch/runner_test.go` |
|
||
| 04-metrics | `internal/metrics` | `metrics/batch_test.go` |
|
||
| 05-fetchtarget | `internal/api` (the one refactor of working code) | `api/fetchtarget_test.go` |
|
||
| 06-batch-wiring | `internal/api` | `api/batchrun_test.go` |
|
||
| 07-api-endpoints | `internal/api` | `api/batch_test.go` |
|
||
| 08-serve-config | `internal/config`, `cmd/sift/serve.go` | `config/batch_test.go` |
|
||
| 09-cli-docs | `cmd/sift`, README, skill | `cmd_sift/batch_test.go` |
|
||
|
||
`guard.PreCheck` and its test are on master (owner commit): `AGENTS.md` forbids the implementer
|
||
to edit `internal/guard/`, the security boundary.
|
||
|
||
Given files live under `_files/<package>/` (`cmd_sift` is `cmd/sift`); each task says where to
|
||
copy its file. Every given test was compiled against a panic-only skeleton of the whole plan and
|
||
fails there on `panic: todo`, not on a compile error.
|
||
|
||
**How this plan was made:** design worked through with the owner from `docs/PLAN.md` §13 (future
|
||
work) on top of the P5 pool (`internal/pdfpool`), cache (`internal/dcache`), and gate
|
||
(`global`/`per` in `internal/api`) infrastructure. Acceptance tests first: the given tests are
|
||
authored against this spec and compiled against a panic-only skeleton so they fail for the
|
||
intended reasons before each task is implemented.
|
||
|
||
## Changes during the run
|
||
|
||
- **Task 02 (runner), first attempt, 20:39–21:08. Model and task size, in about equal parts.**
|
||
- The session wrote the whole runner out as ~700 lines of reasoning in its reply, never in a
|
||
file. OpenCode compacted the context, and the model then emitted a malformed tool call
|
||
(`</tool_call>`) and ended with no commit and no file.
|
||
- It had also looked for the given test at a wrong path. The task said `_files/…` relative
|
||
to the plan; that is my ambiguity.
|
||
- The owner changes:
|
||
- task 02 split into **02-runner-core** (no goroutines) and **03-runner-workers**, with the
|
||
given test split to match (`runner_core_test.go` + `runner_test.go`); later tasks
|
||
renumbered 04–09;
|
||
- every copy path is spelled from the repo root (`docs/plans/p6/_files/…`);
|
||
- `AGENTS.md` rule 6: write code into files early, in small steps;
|
||
- `guard.PreCheck` moved to an owner commit on master. The old task 05 (now 06) told the
|
||
implementer to edit `internal/guard/`, which `AGENTS.md` forbids: my contradiction, found
|
||
while fixing the above.
|
||
|