kyleandClaude Opus 5.5
218eaa8b53
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
113 lines
5.7 KiB
Markdown
113 lines
5.7 KiB
Markdown
# P6 task 06: the `/v1/batch` endpoints
|
|
|
|
**Branch:** `p6` (`git switch p6`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `api: /v1/batch — submit, status summary, one result, cancel`
|
|
|
|
## Goal
|
|
|
|
The HTTP surface over `s.Batch` (task 05): submit, a status summary, one target's result, and
|
|
cancel, with the submit-time guard pre-check, the caps as HTTP statuses, and ownership.
|
|
|
|
## Files
|
|
|
|
- Copy: `_files/api/batch_test.go` → `internal/api/batch_test.go`
|
|
- Modify: `internal/api/batch.go` (the four handlers), `internal/api/api.go` (only the routes in
|
|
`New` and `ServeHTTP`, rule 2), `docs/implementer-log.md`
|
|
|
|
## What already exists
|
|
|
|
```go
|
|
// internal/batch (task 02)
|
|
type Target struct{ URL string; Options Options }
|
|
type Options struct{ Mode, Question, Pages, Grep, JQ string; Context *bool; Recover bool }
|
|
type Result struct{ Content string; Meta map[string]any }
|
|
type Job struct{ ID, Caller, Status, Error string; TargetsTotal, TargetsDone int; CreatedAt int64 }
|
|
type TargetStatus struct{ Idx int; URL, Status string; Result *Result; Error, Hint string }
|
|
var ErrNoTargets, ErrTooManyTargets, ErrCallerBusy, ErrBusy error
|
|
func (r *Runner) Submit(ctx context.Context, c *auth.Caller, callerClass string, targets []Target) (string, bool, error)
|
|
func (r *Runner) Cancel(id string) bool
|
|
func (r *Runner) Status(id string) (Job, []TargetStatus, bool) // a pure read
|
|
func (r *Runner) Touch(id string) // bumps the lease to now
|
|
|
|
// internal/guard (task 05)
|
|
func (g *Guard) PreCheck(raw string) error // "blocked: …" errors
|
|
|
|
// internal/api
|
|
Batch *batch.Runner // in Server (task 05); nil → 503 "batch disabled"
|
|
func (s *Server) caller(w http.ResponseWriter, r *http.Request) *auth.Caller // nil after writing 401/403
|
|
func (s *Server) callerClass(c *auth.Caller) string
|
|
// maxReqBody (64 KiB), writeErr(w, status, msg, hint), writeJSON(w, status, v), s.Prom.Observe(path, d, code)
|
|
```
|
|
|
|
Add `const batchMaxBody = 256 << 10`.
|
|
|
|
## Rules
|
|
|
|
1. **Routes** (in `New`): `POST /v1/batch`, `GET /v1/batch/{id}`, `GET /v1/batch/{id}/{idx}`,
|
|
`DELETE /v1/batch/{id}`. Each starts with `s.caller(w, r)`.
|
|
2. **`ServeHTTP`:**
|
|
- The body cap is `batchMaxBody` for `/v1/batch` and `maxReqBody` for every other path. The cap
|
|
is applied once, here: a handler cannot raise a cap already set.
|
|
- Batch paths are observed with a fixed label: `/v1/batch` or `/v1/batch/{id}`. A job id must
|
|
never become a label value.
|
|
3. **Submit** (`POST /v1/batch`), checked in this order:
|
|
- The body is `{"targets": [{"url", "mode", "question", "pages", "grep", "jq", "context",
|
|
"recover"}, …]}`, each target flat with `/v1/fetch`'s field names. Bad JSON, a missing or
|
|
empty list, an empty `url`, or a `mode` other than empty, `clean`, `raw` or `digest` →
|
|
`400 invalid batch: …`.
|
|
- `s.Guard.PreCheck(url)` for every target. The first failure → `422` with the message
|
|
`blocked: target <idx>: <reason>`, and no job is created.
|
|
- `s.Batch.Submit(r.Context(), c, s.callerClass(c), targets)`:
|
|
- `ErrTooManyTargets` → `422 too many targets (max 32)` (the configured cap);
|
|
- `ErrCallerBusy` → `429 busy: a batch of yours is still running`, `Retry-After: 30`;
|
|
- `ErrBusy` → `429 batch jobs busy`, `Retry-After: 60`;
|
|
- `ErrNoTargets` → `400`.
|
|
- `202 {"success": true, "job_id", "status", "targets": n, "reused"}`. `status` is `"queued"`
|
|
for a new job, and the job's current status for a reused one.
|
|
4. **Ownership:** a job whose `Caller` is not `c.Name`, or an unknown id, is `404 not found` for
|
|
every GET and DELETE. It is never `403`: a caller cannot probe for another caller's jobs.
|
|
5. **Status** (`GET /v1/batch/{id}`): after the ownership check, `s.Batch.Touch(id)`, then
|
|
`200 {"job_id", "status", "error" (only when set), "targets_total", "targets_done",
|
|
"created_at" (RFC 3339, UTC), "targets": [{"idx", "url", "status", "error"?, "hint"?}]}`.
|
|
- This is a summary: **no results**, because 32 inline results can be 6 MB, too much for one
|
|
agent tool call.
|
|
- With `?results=1`, each `done` target also carries `"result"`: the envelope, i.e. `Meta`
|
|
plus `"content"`.
|
|
6. **One result** (`GET /v1/batch/{id}/{idx}`): after ownership and `Touch`, `200 {"job_id", "idx",
|
|
"url", "status", "result"?, "error"?, "hint"?}`. An `idx` that is not a number in
|
|
`0..targets_total-1` → `404`. **Cancel** (`DELETE`): after ownership, a terminal job →
|
|
`409 job already finished`; otherwise `Cancel` → `200 {"success": true, "status": "cancelled"}`.
|
|
7. **Logging:** each batch handler logs one line: `batch` with `caller`, `job_id`, `targets` and
|
|
`status` (or `reused`). **Never a target URL or hostname, for any caller**, and never the
|
|
`idem_key`. The per-target `fetch` line inside `fetchTarget` is unchanged: it is redacted
|
|
for restricted callers exactly as for `/v1/fetch`.
|
|
|
|
|
|
## Facts
|
|
|
|
- `newBatchServer`, `pageServer` and `fakeDNS` come from `batchrun_test.go` (task 05). Targets
|
|
are local `httptest` servers, some held on a channel, or `internal.test` (resolves to `10.0.0.5`).
|
|
- `TestBatchBodyCap` posts 32 targets with questions, about 66 KiB: over `maxReqBody`, under
|
|
`batchMaxBody`.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch p6`; copy the given test. **2. See it fail.**
|
|
- [ ] **3.** The handlers. **4.** Routes and `ServeHTTP`.
|
|
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/api/` → `ok`. **6.** `make gate`.
|
|
- [ ] **7.** Row `p6/06-api-endpoints`; commit.
|
|
|
|
```sh
|
|
git add internal/api docs/implementer-log.md
|
|
git commit -F .state/commit-msg.txt
|
|
```
|
|
|
|
## Done when
|
|
|
|
- The given test and every existing api test pass under `-race -count=3`; `make gate` ok; the
|
|
given file is byte-identical.
|
|
|
|
## Stop and report if
|
|
|
|
- Any given test needs a URL that is not local, or a change to an existing test.
|