P6 plan, owner review folded in: caller travels with the job, 1–2 round-robin workers, busy retry, summary reads, local-only tests, 8 tasks
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,8 +1,11 @@
|
||||
# 01-batchstore — batch tables, lease, GC, byte budget in `internal/dcache`
|
||||
# P6 task 01: batch tables, lease, GC, byte budget in `internal/dcache`
|
||||
|
||||
**Branch:** `p6` (`git switch -c p6 master` if it does not exist, else `git switch p6`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `dcache: batch job and target tables with a sliding lease, sweep and byte budget`
|
||||
|
||||
Add the batch job store to the existing `internal/dcache` SQLite database. No HTTP, no worker —
|
||||
this task is storage only. `dcache` owns the single SQLite connection, so the batch tables and
|
||||
their accessors live here; the runner (task 02) and API (task 03) call them.
|
||||
their accessors live here; the runner (task 02) and API (task 05) call them.
|
||||
|
||||
## Context
|
||||
|
||||
@@ -43,7 +46,7 @@ CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);
|
||||
```
|
||||
|
||||
`result`/`error`/`hint` are `NULL` until the target finishes. The `result` JSON is the
|
||||
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (task 03 enforces the cap;
|
||||
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (the render step enforces the cap;
|
||||
the store just stores what it is given and counts its bytes into `nbytes`).
|
||||
|
||||
## Public surface (exact names — the given test compiles against them)
|
||||
@@ -96,38 +99,57 @@ caller decides the clock, so the given test asserts exact lease values.
|
||||
## Rules
|
||||
|
||||
1. **Sliding lease.** `CreateBatchJob` stores the job as given (the caller sets `Lease`/`CreatedAt`).
|
||||
`UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` bump the job
|
||||
lease to their `now`. `BatchJob` (a read) also bumps the lease to the current wall-clock time so
|
||||
a slow consumer's reads keep the job alive.
|
||||
`UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` set the job
|
||||
lease to their `now`. **Reads never write:** `BatchJob`, `BatchTargets` and the counts change
|
||||
nothing, so the given test can assert exact lease values after reading. A consumer's read keeps
|
||||
a job alive because the API calls `TouchBatchJob` after it (task 05), not because the read does.
|
||||
2. **No partial-job reads leak a competing caller.** Every accessor is keyed by `job_id` only; the
|
||||
caller filter is the API layer's job (task 03). The store stores what it is told and returns
|
||||
caller filter is the API layer's job (task 05). The store stores what it is told and returns
|
||||
what is asked.
|
||||
3. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
|
||||
3. **Idempotency lookup.** `BatchJobByIDem(caller, idemKey, now, ttl)` returns the id of a job with
|
||||
that `(caller, idem_key)` whose `lease + ttl > now` **and whose status is `queued`, `running`,
|
||||
`done` or `partial`**. A `failed` job (including one killed by a restart) or a `cancelled` one
|
||||
is never returned: resubmitting the same list after either starts a new job. If several match,
|
||||
return the newest by `created_at`.
|
||||
4. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
|
||||
exactly once per target that transitions *into* `done`/`failed` (a re-write of the same target
|
||||
does not double-count), bumps the job lease to `now`, and adds `len(result)` to `nbytes` when a
|
||||
result is written.
|
||||
4. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
|
||||
returns the number of jobs deleted. It is the only thing that removes a finished job by time.
|
||||
5. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
|
||||
5. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
|
||||
returns the number of jobs deleted. It is the only thing that removes a finished job by time; the runner calls it (task 02).
|
||||
6. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
|
||||
is `<= maxBytes`, and returns the number evicted. A job in `queued`/`running` is never
|
||||
byte-evicted (only lease-GC, cancel, or restart removes an active job). It runs after a target
|
||||
result is written and is separate from the content-cache `evict()`.
|
||||
6. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
|
||||
byte-evicted (only lease-GC, cancel, or restart removes an active job). The runner (task 02) calls it
|
||||
after each target result is written; it is separate from the content-cache `evict()`.
|
||||
7. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
|
||||
`error=reason`, bumps their lease to `now`, and returns the count. This is the restart
|
||||
reconciliation hook; `main` calls it on startup (task 03).
|
||||
7. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
|
||||
reconciliation hook; `serve` calls it on startup (task 06).
|
||||
8. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
|
||||
the content `evict()` union; batch is a coherent unit with its own budget.
|
||||
8. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
|
||||
9. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
|
||||
count a miss in the appropriate `*M` counter. Reuse the existing `ClassFleet`/`ClassRestricted`
|
||||
constants for `caller_class`.
|
||||
|
||||
## Given
|
||||
|
||||
Copy `_files/dcache/batch_test.go` → `internal/dcache/batch_test.go`. It opens an in-memory `DB`
|
||||
and asserts the create/read round trip, `BatchJobByIDem` lease expiry + caller scoping, live-job
|
||||
and asserts the create/read round trip, `BatchJobByIDem` lease expiry, caller scoping and the
|
||||
status filter (rule 3), live-job
|
||||
counts, ordered targets, the exactly-once `targets_done`/lease-bump rules, `SweepBatch`,
|
||||
`EvictBatchOldest` (oldest-first, never a live job), `MarkInFlightFailed`, and `BatchNbytes`.
|
||||
|
||||
## Gate
|
||||
## Steps
|
||||
|
||||
`make gate` green; `go test ./internal/dcache/` green under `-race`.
|
||||
- [ ] **1.** Branch; copy the given test. **2. See it fail** (`go test ./internal/dcache/` does not compile).
|
||||
- [ ] **3.** Schema in `Open`, the types, the methods, in a new `internal/dcache/batch.go`.
|
||||
- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/dcache/` → `ok` (every existing test too).
|
||||
- [ ] **5.** `make gate`. **6.** Row `p6/01-batchstore` in `docs/implementer-log.md`; commit.
|
||||
|
||||
```sh
|
||||
git add internal/dcache docs/implementer-log.md
|
||||
git commit -F .state/commit-msg.txt
|
||||
```
|
||||
|
||||
## Done when
|
||||
|
||||
- The given and existing dcache tests pass under `-race -count=3`; `make gate` ok; the given file is byte-identical.
|
||||
|
||||
Reference in New Issue
Block a user