sift/docs/plans/p6/02-runner.md

02-runner — serial job runner in internal/batch

Add a new package internal/batch: the job lifecycle and the serial state machine. It knows how to create a job, drain its targets one at a time, write each target's outcome, bump the lease, handle cancellation, and enforce the caps and idempotency. It knows nothing about HTTP, the guard, or the real fetch pipeline: the per-target work is an injected function, and persistence is the *dcache.DB from task 01.

Context

internal/pdfpool is the wrong shape (in-memory, coalescing, forgets results). The runner is the opposite: store-backed, persistent, one job = many distinct targets, results written to the store as they complete. It is deliberately serial.

Public surface (exact names — the given test compiles against them)

package batch

type Options struct {
    Mode     string `json:"mode,omitempty"`
    Question string `json:"question,omitempty"`
    Pages    string `json:"pages,omitempty"`
    Grep     string `json:"grep,omitempty"`
    JQ       string `json:"jq,omitempty"`
    Context  bool   `json:"context,omitempty"`
    Recover  bool   `json:"recover,omitempty"`
}

type Target struct {
    URL     string  `json:"url"`
    Options Options `json:"options,omitempty"`
}

// Result is a completed target's outcome. The runner JSON-encodes it into the
// store's target.result column and decodes it back in Status.
type Result struct {
    Content string         `json:"content"`
    Meta    map[string]any `json:"meta,omitempty"`
}

type FetchFunc func(ctx context.Context, t Target) (Result, error)

// Status views returned by Status (mapped from the store rows).
type Job struct {
    ID           string
    Status       string
    Error        string
    TargetsTotal int
    TargetsDone  int
}

type TargetStatus struct {
    Idx    int
    URL    string
    Status string
    Result *Result
    Error  string
    Hint   string
}

type Config struct {
    TTL           time.Duration // 0 → 24h
    TargetsPerJob int           // 0 → 32
    JobsPerCaller int           // 0 → 1
    JobsTotal     int           // 0 → 8
}

var (
    ErrTooManyTargets = errors.New("too many targets")
    ErrCallerBusy     = errors.New("busy")
    ErrBusy           = errors.New("jobs busy")
)

func New(db *dcache.DB, fetch FetchFunc, cfg Config, log *slog.Logger) *Runner
func (r *Runner) Submit(ctx context.Context, caller, callerClass string, targets []Target) (id string, reused bool, err error)
func (r *Runner) Cancel(id string) bool
func (r *Runner) Status(id string) (Job, []TargetStatus, bool)

The runner depends on *dcache.DB (task 01) for persistence; the FetchFunc is the only injected behavior. Zero Config fields fall back to the defaults above.

Rules

1. Idempotency first. Submit derives idem_key = SHA256(caller + "\x00" + canonical(targets)) where canonical is json.Marshal of the decoded targets (so field order and whitespace don't matter). If a job whose lease has not expired (BatchJobByIDem) exists for the same (caller, idem_key) — any status — return it with reused=true and do not create a new job or enforce the caps. Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
2. Caps second. targets/job > TargetsPerJob → ErrTooManyTargets; a live (queued/running) job already exists for the caller → ErrCallerBusy; live jobs total >= JobsTotal → ErrBusy. "Live" is by status, not lease — read from LiveBatchJobs/LiveBatchJobsTotal. The typed errors map to 422/429 in the API layer.
3. Submit transitions to running synchronously (before the drain goroutine starts) so a caller-busy check against a just-submitted job is deterministic. The job is created queued then immediately running; queued is only ever persisted as the crash-window state that MarkInFlightFailed reaps.
4. Serial drain. The worker iterates targets in idx order, one at a time, and does not start target i+1 until i is written to the store. Each target's FetchFunc result or error is written immediately (JSON-encoded); a target failure does not stop the job.
5. Lease on completion. Each target write bumps the job lease (task 01); the first and last target therefore share one expiry.
6. Cancellation. Cancel flips the job to cancelled and signals the worker; the worker finishes the in-flight target, stores it, and stops before the next. A cancelled job's completed targets stay readable until the lease lapses. Cancel returns false for an unknown id or a job already in a terminal state.
7. Terminal statuses. When the last target is written: all done → done; all failed → failed; a mix → partial. Set exactly once.
8. No restart concept in the runner. It is created fresh each process start; main (task 03) calls dcache.MarkInFlightFailed("restart", now) before constructing the runner. Completed targets on a restart-killed job stay readable; the job is never resumed.
9. No panic on input. Library code never panics: bad ids, missing targets, a FetchFunc panic is recovered into a target error and the job continues. The drain runs under context.WithoutCancel of a background context; only the per-job cancellation stops it.

Given

Copy _files/batch/runner_test.go → internal/batch/runner_test.go. It uses an in-memory dcache plus a FetchFunc that records call order and can fail or panic. Asserts serial order (peak in-flight 1), failure isolation (partial), idempotent resubmit (reused, no double-drain), the three caps, cancel (in-flight stored, later targets pending), and panic recovery.

Gate

make gate green; go test ./internal/batch/ green under -race.