P6 plan, owner review folded in: caller travels with the job, 1–2 round-robin workers, busy retry, summary reads, local-only tests, 8 tasks

Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
  caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
  busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
  APIs it uses; every given test compiles against a panic-only skeleton and fails on it.

P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-26 17:55:50 -07:00
co-authored by Claude Opus 5.5
parent 63e70e19fc
commit 218eaa8b53
20 changed files with 1854 additions and 617 deletions
+42 -20
View File
@@ -1,8 +1,11 @@
# 01-batchstore — batch tables, lease, GC, byte budget in `internal/dcache`
# P6 task 01: batch tables, lease, GC, byte budget in `internal/dcache`
**Branch:** `p6` (`git switch -c p6 master` if it does not exist, else `git switch p6`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `dcache: batch job and target tables with a sliding lease, sweep and byte budget`
Add the batch job store to the existing `internal/dcache` SQLite database. No HTTP, no worker —
this task is storage only. `dcache` owns the single SQLite connection, so the batch tables and
their accessors live here; the runner (task 02) and API (task 03) call them.
their accessors live here; the runner (task 02) and API (task 05) call them.
## Context
@@ -43,7 +46,7 @@ CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);
```
`result`/`error`/`hint` are `NULL` until the target finishes. The `result` JSON is the
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (task 03 enforces the cap;
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (the render step enforces the cap;
the store just stores what it is given and counts its bytes into `nbytes`).
## Public surface (exact names — the given test compiles against them)
@@ -96,38 +99,57 @@ caller decides the clock, so the given test asserts exact lease values.
## Rules
1. **Sliding lease.** `CreateBatchJob` stores the job as given (the caller sets `Lease`/`CreatedAt`).
`UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` bump the job
lease to their `now`. `BatchJob` (a read) also bumps the lease to the current wall-clock time so
a slow consumer's reads keep the job alive.
`UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` set the job
lease to their `now`. **Reads never write:** `BatchJob`, `BatchTargets` and the counts change
nothing, so the given test can assert exact lease values after reading. A consumer's read keeps
a job alive because the API calls `TouchBatchJob` after it (task 05), not because the read does.
2. **No partial-job reads leak a competing caller.** Every accessor is keyed by `job_id` only; the
caller filter is the API layer's job (task 03). The store stores what it is told and returns
caller filter is the API layer's job (task 05). The store stores what it is told and returns
what is asked.
3. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
3. **Idempotency lookup.** `BatchJobByIDem(caller, idemKey, now, ttl)` returns the id of a job with
that `(caller, idem_key)` whose `lease + ttl > now` **and whose status is `queued`, `running`,
`done` or `partial`**. A `failed` job (including one killed by a restart) or a `cancelled` one
is never returned: resubmitting the same list after either starts a new job. If several match,
return the newest by `created_at`.
4. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
exactly once per target that transitions *into* `done`/`failed` (a re-write of the same target
does not double-count), bumps the job lease to `now`, and adds `len(result)` to `nbytes` when a
result is written.
4. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
returns the number of jobs deleted. It is the only thing that removes a finished job by time.
5. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
5. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
returns the number of jobs deleted. It is the only thing that removes a finished job by time; the runner calls it (task 02).
6. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
is `<= maxBytes`, and returns the number evicted. A job in `queued`/`running` is never
byte-evicted (only lease-GC, cancel, or restart removes an active job). It runs after a target
result is written and is separate from the content-cache `evict()`.
6. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
byte-evicted (only lease-GC, cancel, or restart removes an active job). The runner (task 02) calls it
after each target result is written; it is separate from the content-cache `evict()`.
7. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
`error=reason`, bumps their lease to `now`, and returns the count. This is the restart
reconciliation hook; `main` calls it on startup (task 03).
7. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
reconciliation hook; `serve` calls it on startup (task 06).
8. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
the content `evict()` union; batch is a coherent unit with its own budget.
8. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
9. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
count a miss in the appropriate `*M` counter. Reuse the existing `ClassFleet`/`ClassRestricted`
constants for `caller_class`.
## Given
Copy `_files/dcache/batch_test.go` → `internal/dcache/batch_test.go`. It opens an in-memory `DB`
and asserts the create/read round trip, `BatchJobByIDem` lease expiry + caller scoping, live-job
and asserts the create/read round trip, `BatchJobByIDem` lease expiry, caller scoping and the
status filter (rule 3), live-job
counts, ordered targets, the exactly-once `targets_done`/lease-bump rules, `SweepBatch`,
`EvictBatchOldest` (oldest-first, never a live job), `MarkInFlightFailed`, and `BatchNbytes`.
## Gate
## Steps
`make gate` green; `go test ./internal/dcache/` green under `-race`.
- [ ] **1.** Branch; copy the given test. **2. See it fail** (`go test ./internal/dcache/` does not compile).
- [ ] **3.** Schema in `Open`, the types, the methods, in a new `internal/dcache/batch.go`.
- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/dcache/` → `ok` (every existing test too).
- [ ] **5.** `make gate`. **6.** Row `p6/01-batchstore` in `docs/implementer-log.md`; commit.
```sh
git add internal/dcache docs/implementer-log.md
git commit -F .state/commit-msg.txt
```
## Done when
- The given and existing dcache tests pass under `-race -count=3`; `make gate` ok; the given file is byte-identical.