sift/docs/plans/p6/01-batchstore.md

01-batchstore — batch tables, lease, GC, byte budget in internal/dcache

Add the batch job store to the existing internal/dcache SQLite database. No HTTP, no worker — this task is storage only. dcache owns the single SQLite connection, so the batch tables and their accessors live here; the runner (task 02) and API (task 03) call them.

Context

dcache.DB already opens one SQLite connection (SetMaxOpenConns(1), WAL) in Open, owns maxBytes, evict(), Size(), and Stats(). The batch store is a sibling, not a member of the content-addressed cache: a batch job is an atomic unit (a row plus its target rows) and must be evicted whole by lease, never row-by-row by the content LRU.

Schema (add to Open)

CREATE TABLE IF NOT EXISTS batch_job (
    id            TEXT PRIMARY KEY,
    caller        TEXT NOT NULL,
    caller_class  TEXT NOT NULL,          -- 'fleet' | 'restricted'
    idem_key      TEXT NOT NULL,
    status        TEXT NOT NULL,          -- queued|running|done|partial|failed|cancelled
    error         TEXT NOT NULL DEFAULT '',
    targets_total INTEGER NOT NULL,
    targets_done  INTEGER NOT NULL DEFAULT 0,
    created_at    INTEGER NOT NULL,       -- unix seconds
    lease         INTEGER NOT NULL,       -- unix seconds; bumped on submit/completion/read
    nbytes        INTEGER NOT NULL DEFAULT 0  -- sum of stored per-target result bytes
);
CREATE TABLE IF NOT EXISTS batch_target (
    job_id  TEXT NOT NULL,
    idx     INTEGER NOT NULL,             -- order within the job, 0..n-1
    url     TEXT NOT NULL,
    options TEXT NOT NULL,                -- JSON: the per-target options
    status  TEXT NOT NULL,                -- pending|running|done|failed
    result  TEXT,                         -- JSON result envelope, or NULL
    error   TEXT,                         -- error message, or NULL
    hint    TEXT,                         -- error hint, or NULL
    PRIMARY KEY (job_id, idx)
);
CREATE INDEX IF NOT EXISTS batch_job_idem  ON batch_job(caller, idem_key);
CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);

result/error/hint are NULL until the target finishes. The result JSON is the /v1/fetch-shaped envelope, bounded by the output cap at write time (task 03 enforces the cap; the store just stores what it is given and counts its bytes into nbytes).

Public surface (exact names — the given test compiles against them)

type BatchJob struct {
    ID           string
    Caller       string
    CallerClass  string
    IDemKey      string
    Status       string
    Error        string
    TargetsTotal int
    TargetsDone  int
    CreatedAt    int64
    Lease        int64
    NBytes       int64
}

type BatchTarget struct {
    JobID   string
    Idx     int
    URL     string
    Options string
    Status  string
    Result  *string
    Error   *string
    Hint    *string
}

func (d *DB) CreateBatchJob(j BatchJob) error
func (d *DB) BatchJob(id string) (BatchJob, bool)
func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool)
func (d *DB) LiveBatchJobs(caller string) int
func (d *DB) LiveBatchJobsTotal() int
func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool)
func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
func (d *DB) TouchBatchJob(id string, now int64) error
func (d *DB) SweepBatch(now int64, ttl time.Duration) int
func (d *DB) EvictBatchOldest(maxBytes int64) int
func (d *DB) MarkInFlightFailed(reason string, now int64) int
func (d *DB) BatchNbytes() int64

Every method that touches time takes an explicit now (unix seconds) — the store is pure and the caller decides the clock, so the given test asserts exact lease values.

Rules

1. Sliding lease. CreateBatchJob stores the job as given (the caller sets Lease/CreatedAt). UpdateBatchTarget, SetBatchJobStatus, TouchBatchJob, and MarkInFlightFailed bump the job lease to their now. BatchJob (a read) also bumps the lease to the current wall-clock time so a slow consumer's reads keep the job alive.
2. No partial-job reads leak a competing caller. Every accessor is keyed by job_id only; the caller filter is the API layer's job (task 03). The store stores what it is told and returns what is asked.
3. BatchTargets returns targets in idx order. UpdateBatchTarget increments targets_done exactly once per target that transitions into done/failed (a re-write of the same target does not double-count), bumps the job lease to now, and adds len(result) to nbytes when a result is written.
4. SweepBatch(now, ttl) deletes every job whose lease + ttl <= now, and its target rows, and returns the number of jobs deleted. It is the only thing that removes a finished job by time.
5. EvictBatchOldest(maxBytes) evicts whole jobs (oldest lease first) until BatchNbytes() is <= maxBytes, and returns the number evicted. A job in queued/running is never byte-evicted (only lease-GC, cancel, or restart removes an active job). It runs after a target result is written and is separate from the content-cache evict().
6. MarkInFlightFailed(reason, now) marks every queued/running job failed with error=reason, bumps their lease to now, and returns the count. This is the restart reconciliation hook; main calls it on startup (task 03).
7. BatchNbytes() sums nbytes over batch_job. Batch bytes are not included in Size() or the content evict() union; batch is a coherent unit with its own budget.
8. Accessors follow the existing dcache style: return (value, bool) or error, never panic, count a miss in the appropriate *M counter. Reuse the existing ClassFleet/ClassRestricted constants for caller_class.

Given

Copy _files/dcache/batch_test.go → internal/dcache/batch_test.go. It opens an in-memory DB and asserts the create/read round trip, BatchJobByIDem lease expiry + caller scoping, live-job counts, ordered targets, the exactly-once targets_done/lease-bump rules, SweepBatch, EvictBatchOldest (oldest-first, never a live job), MarkInFlightFailed, and BatchNbytes.

Gate

make gate green; go test ./internal/dcache/ green under -race.