P6 plan: asynchronous batch fetch endpoint — phase spec, task plan, given tests

Co-Authored-By: DeepSeek V4 Pro <noreply@deepseek.com>
2026-09-26 12:09:38 -07:00
parent b1da18d49e
commit 63e70e19fc
12 changed files with 1564 additions and 0 deletions
@@ -0,0 +1,133 @@
+# 01-batchstore — batch tables, lease, GC, byte budget in `internal/dcache`
+
+Add the batch job store to the existing `internal/dcache` SQLite database. No HTTP, no worker —
+this task is storage only. `dcache` owns the single SQLite connection, so the batch tables and
+their accessors live here; the runner (task 02) and API (task 03) call them.
+
+## Context
+
+`dcache.DB` already opens one SQLite connection (`SetMaxOpenConns(1)`, WAL) in `Open`, owns
+`maxBytes`, `evict()`, `Size()`, and `Stats()`. The batch store is a sibling, not a member of the
+content-addressed cache: a batch job is an atomic unit (a row plus its target rows) and must be
+evicted whole by lease, never row-by-row by the content LRU.
+
+## Schema (add to `Open`)
+
+```sql
+CREATE TABLE IF NOT EXISTS batch_job (
+    id            TEXT PRIMARY KEY,
+    caller        TEXT NOT NULL,
+    caller_class  TEXT NOT NULL,          -- 'fleet' | 'restricted'
+    idem_key      TEXT NOT NULL,
+    status        TEXT NOT NULL,          -- queued|running|done|partial|failed|cancelled
+    error         TEXT NOT NULL DEFAULT '',
+    targets_total INTEGER NOT NULL,
+    targets_done  INTEGER NOT NULL DEFAULT 0,
+    created_at    INTEGER NOT NULL,       -- unix seconds
+    lease         INTEGER NOT NULL,       -- unix seconds; bumped on submit/completion/read
+    nbytes        INTEGER NOT NULL DEFAULT 0  -- sum of stored per-target result bytes
+);
+CREATE TABLE IF NOT EXISTS batch_target (
+    job_id  TEXT NOT NULL,
+    idx     INTEGER NOT NULL,             -- order within the job, 0..n-1
+    url     TEXT NOT NULL,
+    options TEXT NOT NULL,                -- JSON: the per-target options
+    status  TEXT NOT NULL,                -- pending|running|done|failed
+    result  TEXT,                         -- JSON result envelope, or NULL
+    error   TEXT,                         -- error message, or NULL
+    hint    TEXT,                         -- error hint, or NULL
+    PRIMARY KEY (job_id, idx)
+);
+CREATE INDEX IF NOT EXISTS batch_job_idem  ON batch_job(caller, idem_key);
+CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);
+```
+
+`result`/`error`/`hint` are `NULL` until the target finishes. The `result` JSON is the
+`/v1/fetch`-shaped envelope, bounded by the output cap at write time (task 03 enforces the cap;
+the store just stores what it is given and counts its bytes into `nbytes`).
+
+## Public surface (exact names — the given test compiles against them)
+
+```go
+type BatchJob struct {
+    ID           string
+    Caller       string
+    CallerClass  string
+    IDemKey      string
+    Status       string
+    Error        string
+    TargetsTotal int
+    TargetsDone  int
+    CreatedAt    int64
+    Lease        int64
+    NBytes       int64
+}
+
+type BatchTarget struct {
+    JobID   string
+    Idx     int
+    URL     string
+    Options string
+    Status  string
+    Result  *string
+    Error   *string
+    Hint    *string
+}
+
+func (d *DB) CreateBatchJob(j BatchJob) error
+func (d *DB) BatchJob(id string) (BatchJob, bool)
+func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool)
+func (d *DB) LiveBatchJobs(caller string) int
+func (d *DB) LiveBatchJobsTotal() int
+func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
+func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool)
+func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
+func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
+func (d *DB) TouchBatchJob(id string, now int64) error
+func (d *DB) SweepBatch(now int64, ttl time.Duration) int
+func (d *DB) EvictBatchOldest(maxBytes int64) int
+func (d *DB) MarkInFlightFailed(reason string, now int64) int
+func (d *DB) BatchNbytes() int64
+```
+
+Every method that touches time takes an explicit `now` (unix seconds) — the store is pure and the
+caller decides the clock, so the given test asserts exact lease values.
+
+## Rules
+
+1. **Sliding lease.** `CreateBatchJob` stores the job as given (the caller sets `Lease`/`CreatedAt`).
+   `UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` bump the job
+   lease to their `now`. `BatchJob` (a read) also bumps the lease to the current wall-clock time so
+   a slow consumer's reads keep the job alive.
+2. **No partial-job reads leak a competing caller.** Every accessor is keyed by `job_id` only; the
+   caller filter is the API layer's job (task 03). The store stores what it is told and returns
+   what is asked.
+3. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
+   exactly once per target that transitions *into* `done`/`failed` (a re-write of the same target
+   does not double-count), bumps the job lease to `now`, and adds `len(result)` to `nbytes` when a
+   result is written.
+4. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
+   returns the number of jobs deleted. It is the only thing that removes a finished job by time.
+5. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
+   is `<= maxBytes`, and returns the number evicted. A job in `queued`/`running` is never
+   byte-evicted (only lease-GC, cancel, or restart removes an active job). It runs after a target
+   result is written and is separate from the content-cache `evict()`.
+6. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
+   `error=reason`, bumps their lease to `now`, and returns the count. This is the restart
+   reconciliation hook; `main` calls it on startup (task 03).
+7. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
+   the content `evict()` union; batch is a coherent unit with its own budget.
+8. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
+   count a miss in the appropriate `*M` counter. Reuse the existing `ClassFleet`/`ClassRestricted`
+   constants for `caller_class`.
+
+## Given
+
+Copy `_files/dcache/batch_test.go` → `internal/dcache/batch_test.go`. It opens an in-memory `DB`
+and asserts the create/read round trip, `BatchJobByIDem` lease expiry + caller scoping, live-job
+counts, ordered targets, the exactly-once `targets_done`/lease-bump rules, `SweepBatch`,
+`EvictBatchOldest` (oldest-first, never a live job), `MarkInFlightFailed`, and `BatchNbytes`.
+
+## Gate
+
+`make gate` green; `go test ./internal/dcache/` green under `-race`.