kyleandClaude Opus 5.5
218eaa8b53
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
102 lines
3.9 KiB
Markdown
102 lines
3.9 KiB
Markdown
# P6 task 07: the `[batch]` config and `serve` wiring
|
|
|
|
**Branch:** `p6` (`git switch p6`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `serve: [batch] config; open the store, reap jobs killed by a restart, start the runner`
|
|
|
|
## Goal
|
|
|
|
`sift serve` runs the batch runner from `[batch]` config. On startup it marks jobs that a restart
|
|
killed as failed (they are never resumed). On shutdown it stops the runner before closing the store.
|
|
|
|
## Files
|
|
|
|
- Copy: `_files/config/batch_test.go` → `internal/config/batch_test.go`
|
|
- Modify: `internal/config/config.go` (the `Batch` type and field, `ParseSize`),
|
|
`cmd/sift/serve.go` (rule 3), `dist/sift.toml.example` (rule 4), `docs/implementer-log.md`
|
|
|
|
## What already exists
|
|
|
|
```go
|
|
// internal/batch (task 02)
|
|
type Config struct{ TTL time.Duration; TargetsPerJob, JobsPerCaller, JobsTotal, Workers int; MaxBytes int64; RetryWait, RetryFor time.Duration; Hooks Hooks }
|
|
func (r *Runner) Close()
|
|
|
|
// internal/api (task 05)
|
|
func (s *Server) NewBatchRunner(db *dcache.DB, cfg batch.Config) *batch.Runner
|
|
Batch *batch.Runner // in Server
|
|
// s.Prom.ObserveBatchJob(outcome string) (task 03)
|
|
|
|
// internal/dcache (task 01)
|
|
func (d *DB) MarkInFlightFailed(reason string, now int64) int
|
|
|
|
// internal/config
|
|
func ParseDuration(s string, def time.Duration) time.Duration
|
|
```
|
|
|
|
## Interfaces added here
|
|
|
|
```go
|
|
package config
|
|
|
|
type Batch struct {
|
|
Disabled bool `toml:"disabled"`
|
|
TargetsPerJob int `toml:"targets_per_job"`
|
|
JobsPerCaller int `toml:"jobs_per_caller"`
|
|
JobsTotal int `toml:"jobs_total"`
|
|
Workers int `toml:"workers"`
|
|
TTL string `toml:"ttl"`
|
|
MaxBytes string `toml:"max_bytes"`
|
|
}
|
|
|
|
// in File, after Search:
|
|
Batch Batch `toml:"batch"`
|
|
|
|
// ParseSize parses a positive byte count: a plain integer, or one with KiB, MiB or GiB.
|
|
// Anything else, zero or negative returns def.
|
|
func ParseSize(s string, def int64) int64
|
|
```
|
|
|
|
## Rules
|
|
|
|
1. Batch is **on by default**: no `[batch]` block means the runner with its own defaults.
|
|
`disabled = true` turns it off, and the endpoints then answer 503.
|
|
2. `ParseSize` trims spaces and accepts only the binary suffixes `KiB`, `MiB`, `GiB`, or none (bytes).
|
|
3. **`serve.go`:**
|
|
- Open the `dcache` store when batch is on, as well as in today's cases: same path, same call.
|
|
If the store cannot be opened, log `Warn` and run without batch.
|
|
- Then, **before** building the runner: `n := cache.MarkInFlightFailed("restart", now)`, and call
|
|
`srv.Prom.ObserveBatchJob("restart")` `n` times. Log `batch reaped` with `n` if `n > 0`.
|
|
- `srv.Batch = srv.NewBatchRunner(cache, batch.Config{...})`, with the config fields: `TTL` via
|
|
`ParseDuration(…, 0)` and `MaxBytes` via `ParseSize(…, 0)`. The runner's defaults fill the zeros.
|
|
- On shutdown (both branches of the final `select`), `srv.Batch.Close()` before the process
|
|
exits, when it is non-nil.
|
|
4. **`dist/sift.toml.example`:** a commented `[batch]` block with the defaults, in the style of the
|
|
file:
|
|
```toml
|
|
# [batch]
|
|
# disabled = false
|
|
# targets_per_job = 32
|
|
# jobs_per_caller = 1
|
|
# jobs_total = 8
|
|
# workers = 1 # at most 2: each worker fetches one target at a time
|
|
# ttl = "24h" # a job's results live this long after its last read or write
|
|
# max_bytes = "64MiB" # stored results; the oldest finished jobs go first
|
|
```
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch p6`; copy the given test. **2. See it fail.**
|
|
- [ ] **3.** `config.go`. **4.** `serve.go`. **5.** The example config.
|
|
- [ ] **6.** `gofmt -w`; `go test -race -count=3 ./internal/config/ ./cmd/sift/` → `ok`. **7.** `make gate`.
|
|
- [ ] **8.** Row `p6/07-serve-config`; commit.
|
|
|
|
```sh
|
|
git add internal/config cmd/sift/serve.go dist/sift.toml.example docs/implementer-log.md
|
|
git commit -F .state/commit-msg.txt
|
|
```
|
|
|
|
## Done when
|
|
|
|
- The given test and every existing config and cmd test pass under `-race -count=3`; `make gate`
|
|
ok; the given file is byte-identical.
|