sift/docs/plans/p6/01-batchstore.md

P6 task 01: batch tables, lease, GC, byte budget in internal/dcache

Branch: p6 (git switch -c p6 master if it does not exist, else git switch p6; git status --short must be empty, otherwise stop) Commit subject: dcache: batch job and target tables with a sliding lease, sweep and byte budget

Add the batch job store to the existing internal/dcache SQLite database. No HTTP, no worker — this task is storage only. dcache owns the single SQLite connection, so the batch tables and their accessors live here; the runner (task 02) and the API (tasks 06–07) call them.

Context

dcache.DB already opens one SQLite connection (SetMaxOpenConns(1), WAL) in Open, owns maxBytes, evict(), Size(), and Stats(). The batch store is a sibling, not a member of the content-addressed cache: a batch job is an atomic unit (a row plus its target rows) and must be evicted whole by lease, never row-by-row by the content LRU.

Schema (add to Open)

CREATE TABLE IF NOT EXISTS batch_job (
    id            TEXT PRIMARY KEY,
    caller        TEXT NOT NULL,
    caller_class  TEXT NOT NULL,          -- 'fleet' | 'restricted'
    idem_key      TEXT NOT NULL,
    status        TEXT NOT NULL,          -- queued|running|done|partial|failed|cancelled
    error         TEXT NOT NULL DEFAULT '',
    targets_total INTEGER NOT NULL,
    targets_done  INTEGER NOT NULL DEFAULT 0,
    created_at    INTEGER NOT NULL,       -- unix seconds
    lease         INTEGER NOT NULL,       -- unix seconds; bumped on submit/completion/read
    nbytes        INTEGER NOT NULL DEFAULT 0  -- sum of stored per-target result bytes
);
CREATE TABLE IF NOT EXISTS batch_target (
    job_id  TEXT NOT NULL,
    idx     INTEGER NOT NULL,             -- order within the job, 0..n-1
    url     TEXT NOT NULL,
    options TEXT NOT NULL,                -- JSON: the per-target options
    status  TEXT NOT NULL,                -- pending|running|done|failed
    result  TEXT,                         -- JSON result envelope, or NULL
    error   TEXT,                         -- error message, or NULL
    hint    TEXT,                         -- error hint, or NULL
    PRIMARY KEY (job_id, idx)
);
CREATE INDEX IF NOT EXISTS batch_job_idem  ON batch_job(caller, idem_key);
CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);

result/error/hint are NULL until the target finishes. The result JSON is the /v1/fetch-shaped envelope, bounded by the output cap at write time (the render step enforces the cap; the store just stores what it is given and counts its bytes into nbytes).

Public surface (exact names — the given test compiles against them)

type BatchJob struct {
    ID           string
    Caller       string
    CallerClass  string
    IDemKey      string
    Status       string
    Error        string
    TargetsTotal int
    TargetsDone  int
    CreatedAt    int64
    Lease        int64
    NBytes       int64
}

type BatchTarget struct {
    JobID   string
    Idx     int
    URL     string
    Options string
    Status  string
    Result  *string
    Error   *string
    Hint    *string
}

func (d *DB) CreateBatchJob(j BatchJob) error
func (d *DB) BatchJob(id string) (BatchJob, bool)
func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool)
func (d *DB) LiveBatchJobs(caller string) int
func (d *DB) LiveBatchJobsTotal() int
func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool)
func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
func (d *DB) TouchBatchJob(id string, now int64) error
func (d *DB) SweepBatch(now int64, ttl time.Duration) int
func (d *DB) EvictBatchOldest(maxBytes int64) int
func (d *DB) MarkInFlightFailed(reason string, now int64) int
func (d *DB) BatchNbytes() int64

Every method that touches time takes an explicit now (unix seconds) — the store is pure and the caller decides the clock, so the given test asserts exact lease values.

Rules

1. Sliding lease. CreateBatchJob stores the job as given (the caller sets Lease/CreatedAt). UpdateBatchTarget, SetBatchJobStatus, TouchBatchJob, and MarkInFlightFailed set the job lease to their now. Reads never write: BatchJob, BatchTargets and the counts change nothing, so the given test can assert exact lease values after reading. A consumer's read keeps a job alive because the API calls TouchBatchJob after it (task 07), not because the read does.
2. No partial-job reads leak a competing caller. Every accessor is keyed by job_id only; the caller filter is the API layer's job (task 07). The store stores what it is told and returns what is asked.
3. Idempotency lookup. BatchJobByIDem(caller, idemKey, now, ttl) returns the id of a job with that (caller, idem_key) whose lease + ttl > now and whose status is queued, running, done or partial. A failed job (including one killed by a restart) or a cancelled one is never returned: resubmitting the same list after either starts a new job. If several match, return the newest by created_at.
4. BatchTargets returns targets in idx order. UpdateBatchTarget increments targets_done exactly once per target that transitions into done/failed (a re-write of the same target does not double-count), bumps the job lease to now, and adds len(result) to nbytes when a result is written.
5. SweepBatch(now, ttl) deletes every job whose lease + ttl <= now, and its target rows, and returns the number of jobs deleted. It is the only thing that removes a finished job by time; the runner calls it (task 02).
6. EvictBatchOldest(maxBytes) evicts whole jobs (oldest lease first) until BatchNbytes() is <= maxBytes, and returns the number evicted. A job in queued/running is never byte-evicted (only lease-GC, cancel, or restart removes an active job). The runner (task 02) calls it after each target result is written; it is separate from the content-cache evict().
7. MarkInFlightFailed(reason, now) marks every queued/running job failed with error=reason, bumps their lease to now, and returns the count. This is the restart reconciliation hook; serve calls it on startup (task 08).
8. BatchNbytes() sums nbytes over batch_job. Batch bytes are not included in Size() or the content evict() union; batch is a coherent unit with its own budget.
9. Accessors follow the existing dcache style: return (value, bool) or error, never panic, count a miss in the appropriate *M counter. Reuse the existing ClassFleet/ClassRestricted constants for caller_class.

Given

Copy docs/plans/p6/_files/dcache/batch_test.go → internal/dcache/batch_test.go. It opens an in-memory DB and asserts the create/read round trip, BatchJobByIDem lease expiry, caller scoping and the status filter (rule 3), live-job counts, ordered targets, the exactly-once targets_done/lease-bump rules, SweepBatch, EvictBatchOldest (oldest-first, never a live job), MarkInFlightFailed, and BatchNbytes.

Steps

• 1. Branch; copy the given test. 2. See it fail (go test ./internal/dcache/ does not compile).
• 3. Schema in Open, the types, the methods, in a new internal/dcache/batch.go.
• 4. gofmt -w; go test -race -count=3 ./internal/dcache/ → ok (every existing test too).
• 5. make gate. 6. Row p6/01-batchstore in docs/implementer-log.md; commit.

git add internal/dcache docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The given and existing dcache tests pass under -race -count=3; make gate ok; the given file is byte-identical.