# P6 task 03: the batch runner's workers

**Branch:** `p6` (`git switch p6`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `batch: workers — round-robin, serial per job, busy retry, cancel in flight, hooks`

## Goal

Task 02 wrote the runner's types, `Submit`, `Status`, `Touch` and `Cancel`; every job stays
`queued`. This task adds the workers that drain jobs: one target at a time per job, round-robin
across jobs, with a retry for a target that was only busy.

## Files

- Copy: `docs/plans/p6/_files/batch/runner_test.go` → `internal/batch/runner_test.go`
- Create: `internal/batch/worker.go` (the workers, the rotation, one target, the terminal status)
- Modify: `internal/batch/batch.go` (`New` starts the workers; `Close` stops them; `Submit` wakes
  one; `Cancel` handles a job with a target in flight), `docs/implementer-log.md`

**Work in small steps, in files.** Write `worker.go` with the worker loop and a stub for one
target, and make it compile. Then fill it in, running the tests as you go. Do not write the
implementation out in your reply before putting it in a file.

## What already exists (task 02, in `internal/batch/batch.go`)

All the types in the task 02 file: `Options`, `Target`, `Result`, `TargetError`, `FetchFunc`,
`Job`, `TargetStatus`, `Hooks`, `Config`, the errors (including `ErrRetryLater`), `Runner`,
`New`, `Submit`, `Status`, `Touch`, `Cancel`, `Close`. Read `batch.go` for how the runner keeps
its jobs and each job's caller; the dcache API it uses is listed in the task 02 file's
"What already exists".

## Rules

1. **Workers.** `Workers` goroutines share one rotation of live jobs in submit order. A worker
   takes the next job in the rotation that has a pending target **and no target in flight**. It
   sets the job to `running` on its first target, fetches that one target, stores it, and moves
   on to the next job in the rotation. So:
   - each job is serial: its target `i+1` never starts before `i` is stored;
   - with one worker and two live jobs, their targets alternate (`a1 b1 a2 b2 …`);
   - a job that a worker has not reached yet stays `queued`.
2. **One target:**
   - Call the `FetchFunc` with the job's caller, under a context that `Close` cancels. Recover
     a panic into the error `internal error` (never echo the panic value).
   - If the error wraps `ErrRetryLater`, the target stays `pending`: the worker goes on with the
     rotation, and the target is tried again after `RetryWait`. A target that has been coming
     back busy for longer than `RetryFor` is stored `failed` with the error's text.
   - Otherwise store it: success → `done` with the JSON of the `Result`; error → `failed`, with
     `TargetError`'s `Msg`/`Hint` or `err.Error()` and no hint.
   - Then call `Hooks.Target(err, time, len(result JSON))`, then `EvictBatchOldest(MaxBytes)`.
3. **Terminal status.** When a job's last target is stored: all `done` → `done`; all `failed` →
   `failed`; a mix → `partial`. `SetBatchJobStatus`, then `Hooks.Job(status)` and
   `Hooks.JobSeconds(now − created)`, exactly once per job.
4. **Cancel(id)** returns false for an unknown id or a job already terminal. Otherwise it sets the job
   `cancelled` at once. The in-flight target, if any, finishes and is stored; no further target
   starts; later targets stay `pending`. Storing that last target must not overwrite `cancelled`.
   `Hooks.Job("cancelled")` once.
5. **Hooks.Depth** gets the live-job count and **Hooks.Inflight** the in-flight count whenever
   either changes.

## Facts

- The given test uses `FetchFunc`s that hold on channels, fail, return `ErrRetryLater`, or panic.
  It calls `Close` before the store closes.
- `TestRoundRobin` holds the first target until the second job is submitted, then expects
  `a1 b1 a2 b2 a3 b3` from one worker.
- Task 02's tests (`runner_core_test.go`) must still pass: with workers, a job there may
  already be `running` or `done` where it was `queued`; those tests allow for that.

## Steps

- [ ] **1.** `git switch p6`; copy the given test. **2. See it fail** (the jobs never finish).
- [ ] **3.** `worker.go`. **4.** The `batch.go` changes.
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/batch/` → `ok`. **6.** `make gate`.
- [ ] **7.** Row `p6/03-runner-workers`; commit.

```sh
git add internal/batch docs/implementer-log.md
git commit -F .state/commit-msg.txt
```

## Done when

- Both given tests pass under `-race -count=3`; `make gate` ok; the given files are byte-identical.

## Stop and report if

- A given test's timing can only be met by sleeping longer than `RetryWait` inside a worker.