Owner review of p6 (Ornith tasks 01-09), with an independent reviewer pass whose findings I
confirmed against the code. Every fix has a regression test that fails on the old code.
internal/batch
- Close let the workers keep draining every live job under the cancelled context, failing
every target; they now stop at once, store nothing on a failure during shutdown, and leave
the job live for MarkInFlightFailed("restart").
- A job cancelled while its in-flight target came back busy stayed in the rotation forever
(depth gauge stuck, job held in memory); it now leaves. The task 03 design had this gap: mine.
- Submit's idempotency lookup, caps and create ran unlocked: 20 concurrent submits by one
caller made 8 jobs. Serialised with submitMu.
- A failed target insert left a queued job no worker would run (holding the caller's cap);
it is marked failed.
- A second concurrent cancel returned true and counted twice.
- Config: zero OR negative values take the defaults (negative Workers panicked the
WaitGroup; a negative TTL or byte budget would sweep or evict everything).
- Retry timing used the pre-fetch time (a 5 s busy answer made RetryWait moot).
- The rotation cursor is kept within [0, len(live)] (it was absolute and drifted on removal).
- The unused in-memory callers map is gone (Submit racing Close wrote to a nil map).
internal/dcache
- PutBatchTargets, SweepBatch and eviction are transactional; the sweep never removes a
queued or running job.
internal/api
- Every /v1/batch endpoint answers 503 when batch is off; POST panicked on the nil runner.
- GET /{id}/{idx} bounds idx by the rows actually returned.
- batchFetch copies a *fetch.Error instead of mutating one the PDF pool may share.
internal/config
- ParseSize no longer overflows to a negative size.
cmd/sift
- sift batch treated 202 Accepted (submit) as an error and exited 1 with no message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The HTTP surface over s.Batch. New internal/api/batch.go: batchSubmit (POST)
decodes {"targets":[{url,mode,question,pages,grep,jq,context,recover}]} flat as
/v1/fetch does, 400 "invalid batch" on bad json / empty list / empty url / bad
mode, then PreCheck per target -> 422 "blocked: target <idx>: <reason>" with no
job, then Submit mapping ErrTooManyTargets->422, ErrCallerBusy->429 Retry-After
30, ErrBusy->429 Retry-After 60, ErrNoTargets->400, and 202
{success,job_id,status,targets,reused}. batchStatus (GET), batchResult
(GET {id}/{idx}) and batchCancel (DELETE) share ownership: unknown or not
c.Name -> 404, never 403; status is a summary (no inline results unless
?results=1), idx validated 0..targets_total-1 else 404, cancel of a terminal
job -> 409 else 200 {success,status:cancelled}. api.go scoped to rule 2:
batchMaxBody (256 KiB), the four routes in New, and ServeHTTP applying the
per-path cap once and collapsing every batch path to a fixed metrics label so a
job id never becomes a label. Each handler logs one batch line, never a URL,
hostname or idem_key.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Wire the batch runner to fetchTarget and feed the sift_batch_* metrics.
New internal/api/batch.go: NewBatchRunner overwrites cfg.Hooks (Job, Target,
JobSeconds, Depth, Inflight → s.Prom) and sets the runner's FetchFunc to
batchFetch, which tags the context as a batch fetch, runs under requestDeadline,
and maps fetchTarget's errors to what the runner retries (429/503 →
ErrRetryLater) and what the metrics classify (batchOutcome). Added the Batch
field to Server (nil → task 07 answers 503 "batch disabled") and, in
extractPDF, charged the PDF pool to c.Name+"/batch" under the batch mark so a
batch never takes a caller's interactive slot while the single-flight key is
unchanged. No HTTP endpoints.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
kyleandClaude Opus 5.5
