134 lines
6.4 KiB
Markdown
134 lines
6.4 KiB
Markdown
# 01-batchstore — batch tables, lease, GC, byte budget in `internal/dcache`
|
|
|
|
Add the batch job store to the existing `internal/dcache` SQLite database. No HTTP, no worker —
|
|
this task is storage only. `dcache` owns the single SQLite connection, so the batch tables and
|
|
their accessors live here; the runner (task 02) and API (task 03) call them.
|
|
|
|
## Context
|
|
|
|
`dcache.DB` already opens one SQLite connection (`SetMaxOpenConns(1)`, WAL) in `Open`, owns
|
|
`maxBytes`, `evict()`, `Size()`, and `Stats()`. The batch store is a sibling, not a member of the
|
|
content-addressed cache: a batch job is an atomic unit (a row plus its target rows) and must be
|
|
evicted whole by lease, never row-by-row by the content LRU.
|
|
|
|
## Schema (add to `Open`)
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS batch_job (
|
|
id TEXT PRIMARY KEY,
|
|
caller TEXT NOT NULL,
|
|
caller_class TEXT NOT NULL, -- 'fleet' | 'restricted'
|
|
idem_key TEXT NOT NULL,
|
|
status TEXT NOT NULL, -- queued|running|done|partial|failed|cancelled
|
|
error TEXT NOT NULL DEFAULT '',
|
|
targets_total INTEGER NOT NULL,
|
|
targets_done INTEGER NOT NULL DEFAULT 0,
|
|
created_at INTEGER NOT NULL, -- unix seconds
|
|
lease INTEGER NOT NULL, -- unix seconds; bumped on submit/completion/read
|
|
nbytes INTEGER NOT NULL DEFAULT 0 -- sum of stored per-target result bytes
|
|
);
|
|
CREATE TABLE IF NOT EXISTS batch_target (
|
|
job_id TEXT NOT NULL,
|
|
idx INTEGER NOT NULL, -- order within the job, 0..n-1
|
|
url TEXT NOT NULL,
|
|
options TEXT NOT NULL, -- JSON: the per-target options
|
|
status TEXT NOT NULL, -- pending|running|done|failed
|
|
result TEXT, -- JSON result envelope, or NULL
|
|
error TEXT, -- error message, or NULL
|
|
hint TEXT, -- error hint, or NULL
|
|
PRIMARY KEY (job_id, idx)
|
|
);
|
|
CREATE INDEX IF NOT EXISTS batch_job_idem ON batch_job(caller, idem_key);
|
|
CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);
|
|
```
|
|
|
|
`result`/`error`/`hint` are `NULL` until the target finishes. The `result` JSON is the
|
|
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (task 03 enforces the cap;
|
|
the store just stores what it is given and counts its bytes into `nbytes`).
|
|
|
|
## Public surface (exact names — the given test compiles against them)
|
|
|
|
```go
|
|
type BatchJob struct {
|
|
ID string
|
|
Caller string
|
|
CallerClass string
|
|
IDemKey string
|
|
Status string
|
|
Error string
|
|
TargetsTotal int
|
|
TargetsDone int
|
|
CreatedAt int64
|
|
Lease int64
|
|
NBytes int64
|
|
}
|
|
|
|
type BatchTarget struct {
|
|
JobID string
|
|
Idx int
|
|
URL string
|
|
Options string
|
|
Status string
|
|
Result *string
|
|
Error *string
|
|
Hint *string
|
|
}
|
|
|
|
func (d *DB) CreateBatchJob(j BatchJob) error
|
|
func (d *DB) BatchJob(id string) (BatchJob, bool)
|
|
func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool)
|
|
func (d *DB) LiveBatchJobs(caller string) int
|
|
func (d *DB) LiveBatchJobsTotal() int
|
|
func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
|
|
func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool)
|
|
func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
|
|
func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
|
|
func (d *DB) TouchBatchJob(id string, now int64) error
|
|
func (d *DB) SweepBatch(now int64, ttl time.Duration) int
|
|
func (d *DB) EvictBatchOldest(maxBytes int64) int
|
|
func (d *DB) MarkInFlightFailed(reason string, now int64) int
|
|
func (d *DB) BatchNbytes() int64
|
|
```
|
|
|
|
Every method that touches time takes an explicit `now` (unix seconds) — the store is pure and the
|
|
caller decides the clock, so the given test asserts exact lease values.
|
|
|
|
## Rules
|
|
|
|
1. **Sliding lease.** `CreateBatchJob` stores the job as given (the caller sets `Lease`/`CreatedAt`).
|
|
`UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` bump the job
|
|
lease to their `now`. `BatchJob` (a read) also bumps the lease to the current wall-clock time so
|
|
a slow consumer's reads keep the job alive.
|
|
2. **No partial-job reads leak a competing caller.** Every accessor is keyed by `job_id` only; the
|
|
caller filter is the API layer's job (task 03). The store stores what it is told and returns
|
|
what is asked.
|
|
3. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
|
|
exactly once per target that transitions *into* `done`/`failed` (a re-write of the same target
|
|
does not double-count), bumps the job lease to `now`, and adds `len(result)` to `nbytes` when a
|
|
result is written.
|
|
4. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
|
|
returns the number of jobs deleted. It is the only thing that removes a finished job by time.
|
|
5. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
|
|
is `<= maxBytes`, and returns the number evicted. A job in `queued`/`running` is never
|
|
byte-evicted (only lease-GC, cancel, or restart removes an active job). It runs after a target
|
|
result is written and is separate from the content-cache `evict()`.
|
|
6. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
|
|
`error=reason`, bumps their lease to `now`, and returns the count. This is the restart
|
|
reconciliation hook; `main` calls it on startup (task 03).
|
|
7. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
|
|
the content `evict()` union; batch is a coherent unit with its own budget.
|
|
8. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
|
|
count a miss in the appropriate `*M` counter. Reuse the existing `ClassFleet`/`ClassRestricted`
|
|
constants for `caller_class`.
|
|
|
|
## Given
|
|
|
|
Copy `_files/dcache/batch_test.go` → `internal/dcache/batch_test.go`. It opens an in-memory `DB`
|
|
and asserts the create/read round trip, `BatchJobByIDem` lease expiry + caller scoping, live-job
|
|
counts, ordered targets, the exactly-once `targets_done`/lease-bump rules, `SweepBatch`,
|
|
`EvictBatchOldest` (oldest-first, never a live job), `MarkInFlightFailed`, and `BatchNbytes`.
|
|
|
|
## Gate
|
|
|
|
`make gate` green; `go test ./internal/dcache/` green under `-race`.
|