sift/docs/plans/p6/07-serve-config.md

P6 task 07: the [batch] config and serve wiring

Branch: p6 (git switch p6; git status --short must be empty, otherwise stop) Commit subject: serve: [batch] config; open the store, reap jobs killed by a restart, start the runner

Goal

sift serve runs the batch runner from [batch] config. On startup it marks jobs that a restart killed as failed (they are never resumed). On shutdown it stops the runner before closing the store.

Files

• Copy: _files/config/batch_test.go → internal/config/batch_test.go
• Modify: internal/config/config.go (the Batch type and field, ParseSize), cmd/sift/serve.go (rule 3), dist/sift.toml.example (rule 4), docs/implementer-log.md

What already exists

// internal/batch (task 02)
type Config struct{ TTL time.Duration; TargetsPerJob, JobsPerCaller, JobsTotal, Workers int; MaxBytes int64; RetryWait, RetryFor time.Duration; Hooks Hooks }
func (r *Runner) Close()

// internal/api (task 05)
func (s *Server) NewBatchRunner(db *dcache.DB, cfg batch.Config) *batch.Runner
	Batch *batch.Runner // in Server
// s.Prom.ObserveBatchJob(outcome string) (task 03)

// internal/dcache (task 01)
func (d *DB) MarkInFlightFailed(reason string, now int64) int

// internal/config
func ParseDuration(s string, def time.Duration) time.Duration

Interfaces added here

package config

type Batch struct {
	Disabled      bool   `toml:"disabled"`
	TargetsPerJob int    `toml:"targets_per_job"`
	JobsPerCaller int    `toml:"jobs_per_caller"`
	JobsTotal     int    `toml:"jobs_total"`
	Workers       int    `toml:"workers"`
	TTL           string `toml:"ttl"`
	MaxBytes      string `toml:"max_bytes"`
}

// in File, after Search:
	Batch Batch `toml:"batch"`

// ParseSize parses a positive byte count: a plain integer, or one with KiB, MiB or GiB.
// Anything else, zero or negative returns def.
func ParseSize(s string, def int64) int64

Rules

1. Batch is on by default: no [batch] block means the runner with its own defaults. disabled = true turns it off, and the endpoints then answer 503.
2. ParseSize trims spaces and accepts only the binary suffixes KiB, MiB, GiB, or none (bytes).
3. serve.go:
   • Open the dcache store when batch is on, as well as in today's cases: same path, same call. If the store cannot be opened, log Warn and run without batch.
   • Then, before building the runner: n := cache.MarkInFlightFailed("restart", now), and call srv.Prom.ObserveBatchJob("restart") n times. Log batch reaped with n if n > 0.
   • srv.Batch = srv.NewBatchRunner(cache, batch.Config{...}), with the config fields: TTL via ParseDuration(…, 0) and MaxBytes via ParseSize(…, 0). The runner's defaults fill the zeros.
   • On shutdown (both branches of the final select), srv.Batch.Close() before the process exits, when it is non-nil.
4. dist/sift.toml.example: a commented [batch] block with the defaults, in the style of the file:

   # [batch]
   # disabled = false
   # targets_per_job = 32
   # jobs_per_caller = 1
   # jobs_total = 8
   # workers = 1          # at most 2: each worker fetches one target at a time
   # ttl = "24h"          # a job's results live this long after its last read or write
   # max_bytes = "64MiB"  # stored results; the oldest finished jobs go first
   

Steps

• 1. git switch p6; copy the given test. 2. See it fail.
• 3. config.go. 4. serve.go. 5. The example config.
• 6. gofmt -w; go test -race -count=3 ./internal/config/ ./cmd/sift/ → ok. 7. make gate.
• 8. Row p6/07-serve-config; commit.

git add internal/config cmd/sift/serve.go dist/sift.toml.example docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The given test and every existing config and cmd test pass under -race -count=3; make gate ok; the given file is byte-identical.