sift/docs/plans/p6/01-batchstore.md

# P6 task 01: batch tables, lease, GC, byte budget in `internal/dcache`

**Branch:** `p6` (`git switch -c p6 master` if it does not exist, else `git switch p6`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `dcache: batch job and target tables with a sliding lease, sweep and byte budget`

Add the batch job store to the existing `internal/dcache` SQLite database. No HTTP, no worker —
this task is storage only. `dcache` owns the single SQLite connection, so the batch tables and
their accessors live here; the runner (task 02) and the API (tasks 06–07) call them.

## Context

`dcache.DB` already opens one SQLite connection (`SetMaxOpenConns(1)`, WAL) in `Open`, owns
`maxBytes`, `evict()`, `Size()`, and `Stats()`. The batch store is a sibling, not a member of the
content-addressed cache: a batch job is an atomic unit (a row plus its target rows) and must be
evicted whole by lease, never row-by-row by the content LRU.

## Schema (add to `Open`)

```sql
CREATE TABLE IF NOT EXISTS batch_job (
    id            TEXT PRIMARY KEY,
    caller        TEXT NOT NULL,
    caller_class  TEXT NOT NULL,          -- 'fleet' | 'restricted'
    idem_key      TEXT NOT NULL,
    status        TEXT NOT NULL,          -- queued|running|done|partial|failed|cancelled
    error         TEXT NOT NULL DEFAULT '',
    targets_total INTEGER NOT NULL,
    targets_done  INTEGER NOT NULL DEFAULT 0,
    created_at    INTEGER NOT NULL,       -- unix seconds
    lease         INTEGER NOT NULL,       -- unix seconds; bumped on submit/completion/read
    nbytes        INTEGER NOT NULL DEFAULT 0  -- sum of stored per-target result bytes
);
CREATE TABLE IF NOT EXISTS batch_target (
    job_id  TEXT NOT NULL,
    idx     INTEGER NOT NULL,             -- order within the job, 0..n-1
    url     TEXT NOT NULL,
    options TEXT NOT NULL,                -- JSON: the per-target options
    status  TEXT NOT NULL,                -- pending|running|done|failed
    result  TEXT,                         -- JSON result envelope, or NULL
    error   TEXT,                         -- error message, or NULL
    hint    TEXT,                         -- error hint, or NULL
    PRIMARY KEY (job_id, idx)
);
CREATE INDEX IF NOT EXISTS batch_job_idem  ON batch_job(caller, idem_key);
CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);
```

`result`/`error`/`hint` are `NULL` until the target finishes. The `result` JSON is the
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (the render step enforces the cap;
the store just stores what it is given and counts its bytes into `nbytes`).

## Public surface (exact names — the given test compiles against them)

```go
type BatchJob struct {
    ID           string
    Caller       string
    CallerClass  string
    IDemKey      string
    Status       string
    Error        string
    TargetsTotal int
    TargetsDone  int
    CreatedAt    int64
    Lease        int64
    NBytes       int64
}

type BatchTarget struct {
    JobID   string
    Idx     int
    URL     string
    Options string
    Status  string
    Result  *string
    Error   *string
    Hint    *string
}

func (d *DB) CreateBatchJob(j BatchJob) error
func (d *DB) BatchJob(id string) (BatchJob, bool)
func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool)
func (d *DB) LiveBatchJobs(caller string) int
func (d *DB) LiveBatchJobsTotal() int
func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool)
func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
func (d *DB) TouchBatchJob(id string, now int64) error
func (d *DB) SweepBatch(now int64, ttl time.Duration) int
func (d *DB) EvictBatchOldest(maxBytes int64) int
func (d *DB) MarkInFlightFailed(reason string, now int64) int
func (d *DB) BatchNbytes() int64
```

Every method that touches time takes an explicit `now` (unix seconds) — the store is pure and the
caller decides the clock, so the given test asserts exact lease values.

## Rules

1. **Sliding lease.** `CreateBatchJob` stores the job as given (the caller sets `Lease`/`CreatedAt`).
   `UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` set the job
   lease to their `now`. **Reads never write:** `BatchJob`, `BatchTargets` and the counts change
   nothing, so the given test can assert exact lease values after reading. A consumer's read keeps
   a job alive because the API calls `TouchBatchJob` after it (task 07), not because the read does.
2. **No partial-job reads leak a competing caller.** Every accessor is keyed by `job_id` only; the
   caller filter is the API layer's job (task 07). The store stores what it is told and returns
   what is asked.
3. **Idempotency lookup.** `BatchJobByIDem(caller, idemKey, now, ttl)` returns the id of a job with
   that `(caller, idem_key)` whose `lease + ttl > now` **and whose status is `queued`, `running`,
   `done` or `partial`**. A `failed` job (including one killed by a restart) or a `cancelled` one
   is never returned: resubmitting the same list after either starts a new job. If several match,
   return the newest by `created_at`.
4. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
   exactly once per target that transitions *into* `done`/`failed` (a re-write of the same target
   does not double-count), bumps the job lease to `now`, and adds `len(result)` to `nbytes` when a
   result is written.
5. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
   returns the number of jobs deleted. It is the only thing that removes a finished job by time; the runner calls it (task 02).
6. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
   is `<= maxBytes`, and returns the number evicted. A job in `queued`/`running` is never
   byte-evicted (only lease-GC, cancel, or restart removes an active job). The runner (task 02) calls it
   after each target result is written; it is separate from the content-cache `evict()`.
7. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
   `error=reason`, bumps their lease to `now`, and returns the count. This is the restart
   reconciliation hook; `serve` calls it on startup (task 08).
8. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
   the content `evict()` union; batch is a coherent unit with its own budget.
9. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
   count a miss in the appropriate `*M` counter. Reuse the existing `ClassFleet`/`ClassRestricted`
   constants for `caller_class`.

## Given

Copy `docs/plans/p6/_files/dcache/batch_test.go` → `internal/dcache/batch_test.go`. It opens an in-memory `DB`
and asserts the create/read round trip, `BatchJobByIDem` lease expiry, caller scoping and the
status filter (rule 3), live-job
counts, ordered targets, the exactly-once `targets_done`/lease-bump rules, `SweepBatch`,
`EvictBatchOldest` (oldest-first, never a live job), `MarkInFlightFailed`, and `BatchNbytes`.

## Steps

- [ ] **1.** Branch; copy the given test. **2. See it fail** (`go test ./internal/dcache/` does not compile).
- [ ] **3.** Schema in `Open`, the types, the methods, in a new `internal/dcache/batch.go`.
- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/dcache/` → `ok` (every existing test too).
- [ ] **5.** `make gate`. **6.** Row `p6/01-batchstore` in `docs/implementer-log.md`; commit.

```sh
git add internal/dcache docs/implementer-log.md
git commit -F .state/commit-msg.txt
```

## Done when

- The given and existing dcache tests pass under `-race -count=3`; `make gate` ok; the given file is byte-identical.