P6 plan, owner review folded in: caller travels with the job, 1–2 round-robin workers, busy retry, summary reads, local-only tests, 8 tasks
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
+171
-87
@@ -1,126 +1,210 @@
|
||||
# 02-runner — serial job runner in `internal/batch`
|
||||
# P6 task 02: the batch runner in `internal/batch`
|
||||
|
||||
Add a new package `internal/batch`: the job lifecycle and the serial state machine. It knows how
|
||||
to create a job, drain its targets one at a time, write each target's outcome, bump the lease,
|
||||
handle cancellation, and enforce the caps and idempotency. It knows **nothing** about HTTP, the
|
||||
guard, or the real fetch pipeline: the per-target work is an injected function, and persistence is
|
||||
the `*dcache.DB` from task 01.
|
||||
**Branch:** `p6` (`git switch p6`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `batch: store-backed job runner — serial per job, round-robin workers, caps, idempotency, cancel, retry`
|
||||
|
||||
## Context
|
||||
## Goal
|
||||
|
||||
`internal/pdfpool` is the wrong shape (in-memory, coalescing, forgets results). The runner is the
|
||||
opposite: store-backed, persistent, one job = many distinct targets, results written to the store
|
||||
as they complete. It is deliberately serial.
|
||||
A new package `internal/batch`: the job lifecycle. It creates a job, lets a small pool of
|
||||
workers drain its targets one at a time, writes each outcome to the store, handles cancellation,
|
||||
retries a target whose fetch was only busy, and enforces the caps and idempotency.
|
||||
|
||||
## Public surface (exact names — the given test compiles against them)
|
||||
It knows **nothing** about HTTP, the guard, or the real fetch pipeline: the per-target work is an
|
||||
injected `FetchFunc`, and persistence is the `*dcache.DB` from task 01.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `_files/batch/runner_test.go` → `internal/batch/runner_test.go`
|
||||
- Create: `internal/batch/batch.go` (types, `New`, `Submit`, `Cancel`, `Status`, `Touch`, `Close`),
|
||||
`internal/batch/worker.go` (the workers and the drain)
|
||||
- Modify: `docs/implementer-log.md`
|
||||
|
||||
## What already exists (do not re-read these packages; this is their API)
|
||||
|
||||
```go
|
||||
// internal/dcache (task 01). Every time argument is unix seconds.
|
||||
type BatchJob struct{ ID, Caller, CallerClass, IDemKey, Status, Error string; TargetsTotal, TargetsDone int; CreatedAt, Lease, NBytes int64 }
|
||||
type BatchTarget struct{ JobID string; Idx int; URL, Options, Status string; Result, Error, Hint *string }
|
||||
func (d *DB) CreateBatchJob(j BatchJob) error
|
||||
func (d *DB) BatchJob(id string) (BatchJob, bool) // a pure read
|
||||
func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool) // queued/running/done/partial only
|
||||
func (d *DB) LiveBatchJobs(caller string) int // queued + running
|
||||
func (d *DB) LiveBatchJobsTotal() int
|
||||
func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
|
||||
func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool) // in idx order
|
||||
func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
|
||||
func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
|
||||
func (d *DB) TouchBatchJob(id string, now int64) error
|
||||
func (d *DB) SweepBatch(now int64, ttl time.Duration) int
|
||||
func (d *DB) EvictBatchOldest(maxBytes int64) int
|
||||
const ClassFleet, ClassRestricted = "fleet", "restricted"
|
||||
|
||||
// internal/auth
|
||||
type Caller struct{ Name, Workspace, KeySHA256 string }
|
||||
```
|
||||
|
||||
## Interfaces (exact: the given test compiles against them)
|
||||
|
||||
```go
|
||||
package batch
|
||||
|
||||
type Options struct {
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Question string `json:"question,omitempty"`
|
||||
Pages string `json:"pages,omitempty"`
|
||||
Grep string `json:"grep,omitempty"`
|
||||
JQ string `json:"jq,omitempty"`
|
||||
Context bool `json:"context,omitempty"`
|
||||
Recover bool `json:"recover,omitempty"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
Question string `json:"question,omitempty"`
|
||||
Pages string `json:"pages,omitempty"`
|
||||
Grep string `json:"grep,omitempty"`
|
||||
JQ string `json:"jq,omitempty"`
|
||||
Context *bool `json:"context,omitempty"` // nil = /v1/fetch's default (on when there is a question)
|
||||
Recover bool `json:"recover,omitempty"`
|
||||
}
|
||||
|
||||
type Target struct {
|
||||
URL string `json:"url"`
|
||||
Options Options `json:"options,omitempty"`
|
||||
URL string `json:"url"`
|
||||
Options Options `json:"options"`
|
||||
}
|
||||
|
||||
// Result is a completed target's outcome. The runner JSON-encodes it into the
|
||||
// store's target.result column and decodes it back in Status.
|
||||
// Result is a finished target: the /v1/fetch envelope's content, and every other envelope field
|
||||
// in Meta. It is stored JSON-encoded in the target's result column.
|
||||
type Result struct {
|
||||
Content string `json:"content"`
|
||||
Meta map[string]any `json:"meta,omitempty"`
|
||||
Content string `json:"content"`
|
||||
Meta map[string]any `json:"meta,omitempty"`
|
||||
}
|
||||
|
||||
type FetchFunc func(ctx context.Context, t Target) (Result, error)
|
||||
// TargetError is a failure with an agent-facing hint. Status is the HTTP status the same
|
||||
// failure gets from /v1/fetch; the runner ignores it (the metrics hook reads it). Any other
|
||||
// error is stored with its Error() text and no hint.
|
||||
type TargetError struct {
|
||||
Status int
|
||||
Msg, Hint string
|
||||
}
|
||||
|
||||
func (e *TargetError) Error() string { return e.Msg }
|
||||
|
||||
// FetchFunc does one target for the caller that submitted the job.
|
||||
type FetchFunc func(ctx context.Context, c *auth.Caller, t Target) (Result, error)
|
||||
|
||||
// Status views returned by Status (mapped from the store rows).
|
||||
type Job struct {
|
||||
ID string
|
||||
Status string
|
||||
Error string
|
||||
TargetsTotal int
|
||||
TargetsDone int
|
||||
ID string
|
||||
Caller string
|
||||
Status string // queued | running | done | partial | failed | cancelled
|
||||
Error string
|
||||
TargetsTotal int
|
||||
TargetsDone int
|
||||
CreatedAt int64 // unix seconds
|
||||
}
|
||||
|
||||
type TargetStatus struct {
|
||||
Idx int
|
||||
URL string
|
||||
Status string
|
||||
Result *Result
|
||||
Error string
|
||||
Hint string
|
||||
Idx int
|
||||
URL string
|
||||
Status string // pending | running | done | failed
|
||||
Result *Result
|
||||
Error string
|
||||
Hint string
|
||||
}
|
||||
|
||||
// Hooks observe the runner for metrics. Any nil field is skipped.
|
||||
type Hooks struct {
|
||||
Job func(outcome string) // created, reused, done, partial, failed, cancelled
|
||||
Target func(err error, d time.Duration, resultBytes int) // once per stored target; err nil on success
|
||||
JobSeconds func(d time.Duration) // submit → terminal, once per job
|
||||
Depth func(live int) // live jobs, after every change
|
||||
Inflight func(n int) // targets being fetched, after every change
|
||||
}
|
||||
|
||||
type Config struct {
|
||||
TTL time.Duration // 0 → 24h
|
||||
TargetsPerJob int // 0 → 32
|
||||
JobsPerCaller int // 0 → 1
|
||||
JobsTotal int // 0 → 8
|
||||
TTL time.Duration // 0 → 24h
|
||||
TargetsPerJob int // 0 → 32
|
||||
JobsPerCaller int // 0 → 1
|
||||
JobsTotal int // 0 → 8
|
||||
Workers int // 0 → 1; values above 2 are clamped to 2
|
||||
MaxBytes int64 // 0 → 64 MiB
|
||||
RetryWait time.Duration // 0 → 2s: the pause before a busy target is tried again
|
||||
RetryFor time.Duration // 0 → 10m: how long one target may keep coming back busy
|
||||
Hooks Hooks
|
||||
}
|
||||
|
||||
var (
|
||||
ErrTooManyTargets = errors.New("too many targets")
|
||||
ErrCallerBusy = errors.New("busy")
|
||||
ErrBusy = errors.New("jobs busy")
|
||||
ErrNoTargets = errors.New("no targets")
|
||||
ErrTooManyTargets = errors.New("too many targets")
|
||||
ErrCallerBusy = errors.New("busy")
|
||||
ErrBusy = errors.New("jobs busy")
|
||||
ErrRetryLater = errors.New("retry later") // a FetchFunc wraps this when the target was only busy
|
||||
)
|
||||
|
||||
func New(db *dcache.DB, fetch FetchFunc, cfg Config, log *slog.Logger) *Runner
|
||||
func (r *Runner) Submit(ctx context.Context, caller, callerClass string, targets []Target) (id string, reused bool, err error)
|
||||
func New(db *dcache.DB, fetch FetchFunc, cfg Config, log *slog.Logger) *Runner // starts the workers
|
||||
func (r *Runner) Submit(ctx context.Context, c *auth.Caller, callerClass string, targets []Target) (id string, reused bool, err error)
|
||||
func (r *Runner) Cancel(id string) bool
|
||||
func (r *Runner) Status(id string) (Job, []TargetStatus, bool)
|
||||
func (r *Runner) Status(id string) (Job, []TargetStatus, bool) // a pure read
|
||||
func (r *Runner) Touch(id string) // bumps the lease to now
|
||||
func (r *Runner) Close() // stops the workers; waits for an in-flight fetch to return
|
||||
```
|
||||
|
||||
The runner depends on `*dcache.DB` (task 01) for persistence; the `FetchFunc` is the only injected
|
||||
behavior. Zero `Config` fields fall back to the defaults above.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Idempotency first.** `Submit` derives `idem_key = SHA256(caller + "\x00" + canonical(targets))`
|
||||
where `canonical` is `json.Marshal` of the decoded targets (so field order and whitespace don't
|
||||
matter). If a job whose lease has not expired (`BatchJobByIDem`) exists for the same
|
||||
`(caller, idem_key)` — any status — return it with `reused=true` and do **not** create a new job
|
||||
or enforce the caps. Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
|
||||
2. **Caps second.** targets/job > `TargetsPerJob` → `ErrTooManyTargets`; a live (`queued`/`running`)
|
||||
job already exists for the caller → `ErrCallerBusy`; live jobs total >= `JobsTotal` → `ErrBusy`.
|
||||
"Live" is by status, not lease — read from `LiveBatchJobs`/`LiveBatchJobsTotal`. The typed errors
|
||||
map to `422`/`429` in the API layer.
|
||||
3. **Submit transitions to `running` synchronously** (before the drain goroutine starts) so a
|
||||
caller-busy check against a just-submitted job is deterministic. The job is created `queued`
|
||||
then immediately `running`; `queued` is only ever persisted as the crash-window state that
|
||||
`MarkInFlightFailed` reaps.
|
||||
4. **Serial drain.** The worker iterates targets in `idx` order, one at a time, and does not start
|
||||
target `i+1` until `i` is written to the store. Each target's `FetchFunc` result or error is
|
||||
written immediately (JSON-encoded); a target failure does not stop the job.
|
||||
5. **Lease on completion.** Each target write bumps the job lease (task 01); the first and last
|
||||
target therefore share one expiry.
|
||||
6. **Cancellation.** `Cancel` flips the job to `cancelled` and signals the worker; the worker
|
||||
finishes the in-flight target, stores it, and stops before the next. A `cancelled` job's
|
||||
completed targets stay readable until the lease lapses. `Cancel` returns false for an unknown id
|
||||
or a job already in a terminal state.
|
||||
7. **Terminal statuses.** When the last target is written: all `done` → `done`; all `failed` →
|
||||
`failed`; a mix → `partial`. Set exactly once.
|
||||
8. **No restart concept in the runner.** It is created fresh each process start; `main` (task 03)
|
||||
calls `dcache.MarkInFlightFailed("restart", now)` before constructing the runner. Completed
|
||||
targets on a restart-killed job stay readable; the job is never resumed.
|
||||
9. **No panic on input.** Library code never panics: bad ids, missing targets, a `FetchFunc` panic
|
||||
is recovered into a target error and the job continues. The drain runs under
|
||||
`context.WithoutCancel` of a background context; only the per-job cancellation stops it.
|
||||
1. **Submit, in this order:**
|
||||
- `len(targets) == 0` → `ErrNoTargets`.
|
||||
- **Idempotency.** `idem_key` = hex SHA-256 of `c.Name + "\x00" + json.Marshal(targets)`.
|
||||
If `BatchJobByIDem(c.Name, key, now, TTL)` finds a job, return its id with `reused = true`:
|
||||
no new job, no caps, `Hooks.Job("reused")`.
|
||||
- **Caps:** more than `TargetsPerJob` → `ErrTooManyTargets`; `LiveBatchJobs(c.Name) >= JobsPerCaller`
|
||||
→ `ErrCallerBusy`; `LiveBatchJobsTotal() >= JobsTotal` → `ErrBusy`.
|
||||
- Store the job **`queued`** (random 16-byte hex id, `CreatedAt` = `Lease` = now) and its targets
|
||||
(`pending`, `Options` = the JSON of `t.Options`), **before returning**. Keep `c` in memory
|
||||
for this job: jobs never resume after a restart, so the caller is never persisted.
|
||||
- `Hooks.Job("created")`; `SweepBatch(now, TTL)`; wake a worker.
|
||||
2. **Workers.** `Workers` goroutines share one rotation of live jobs in submit order. A worker
|
||||
takes the next job in the rotation that has a pending target **and no target in flight**. It
|
||||
sets the job to `running` on its first target, fetches that one target, stores it, and moves
|
||||
on to the next job in the rotation. So:
|
||||
- each job is serial: its target `i+1` never starts before `i` is stored;
|
||||
- with one worker and two live jobs, their targets alternate (`a1 b1 a2 b2 …`);
|
||||
- a job that a worker has not reached yet stays `queued`.
|
||||
3. **One target:**
|
||||
- Call the `FetchFunc` with the job's caller, under a context that `Close` cancels. Recover
|
||||
a panic into the error `internal error` (never echo the panic value).
|
||||
- If the error wraps `ErrRetryLater`, the target stays `pending`: the worker goes on with the
|
||||
rotation, and the target is tried again after `RetryWait`. A target that has been coming
|
||||
back busy for longer than `RetryFor` is stored `failed` with the error's text.
|
||||
- Otherwise store it: success → `done` with the JSON of the `Result`; error → `failed`, with
|
||||
`TargetError`'s `Msg`/`Hint` or `err.Error()` and no hint.
|
||||
- Then call `Hooks.Target(err, time, len(result JSON))`, then `EvictBatchOldest(MaxBytes)`.
|
||||
4. **Terminal status.** When a job's last target is stored: all `done` → `done`; all `failed` →
|
||||
`failed`; a mix → `partial`. `SetBatchJobStatus`, then `Hooks.Job(status)` and
|
||||
`Hooks.JobSeconds(now − created)`, exactly once per job.
|
||||
5. **Cancel(id)** returns false for an unknown id or a job already terminal. Otherwise it sets the job
|
||||
`cancelled` at once. The in-flight target, if any, finishes and is stored; no further target
|
||||
starts; later targets stay `pending`. Storing that last target must not overwrite `cancelled`.
|
||||
`Hooks.Job("cancelled")` once.
|
||||
6. **Status(id)** maps the store rows; `Result` is decoded from the result column. It writes
|
||||
nothing. **Touch(id)** is `TouchBatchJob(id, now)`; the API calls it after an owner's read.
|
||||
7. **Hooks.Depth** gets the live-job count and **Hooks.Inflight** the in-flight count whenever
|
||||
either changes.
|
||||
8. Library code never panics; a store error is logged (`Warn`, with the job id, never a URL) and
|
||||
the job goes on. Log lines carry the job id, counts and statuses, **never a URL or host**.
|
||||
|
||||
## Given
|
||||
## Facts
|
||||
|
||||
Copy `_files/batch/runner_test.go` → `internal/batch/runner_test.go`. It uses an in-memory `dcache`
|
||||
plus a `FetchFunc` that records call order and can fail or panic. Asserts serial order (peak
|
||||
in-flight 1), failure isolation (`partial`), idempotent resubmit (`reused`, no double-drain), the
|
||||
three caps, cancel (in-flight stored, later targets `pending`), and panic recovery.
|
||||
- The given test uses an in-memory `dcache` and `FetchFunc`s that hold on channels, fail, return
|
||||
`ErrRetryLater`, or panic. It calls `Close` before the store closes.
|
||||
- `TestRoundRobin` holds the first target until the second job is submitted, then expects
|
||||
`a1 b1 a2 b2 a3 b3` from one worker.
|
||||
|
||||
## Gate
|
||||
## Steps
|
||||
|
||||
`make gate` green; `go test ./internal/batch/` green under `-race`.
|
||||
- [ ] **1.** `git switch p6`; copy the given test. **2. See it fail** (does not compile).
|
||||
- [ ] **3.** `batch.go`, **4.** `worker.go`.
|
||||
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/batch/` → `ok`. **6.** `make gate`.
|
||||
- [ ] **7.** Row `p6/02-runner`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/batch docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given test passes under `-race -count=3`; `make gate` ok; the given file is byte-identical.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A given test's timing can only be met by sleeping longer than `RetryWait` inside a worker.
|
||||
|
||||
Reference in New Issue
Block a user