sift/docs/plans/p6/03-api.md

03-api — /v1/batch endpoints in internal/api

Wire the batch runner into the API server. This is the HTTP surface and the only place the guard, auth, caps-as-HTTP-statuses, and restricted-caller redaction meet the runner. New code goes in a new file internal/api/batch.go; do not grow api.go.

Endpoints

Method	Path	Result
POST /v1/batch	submit a list of targets	202 job created/reused, 400 malformed, 422 a target refused up front or over a cap, 429 caller/jobs busy
GET /v1/batch/{id}	status + per-target results	200 (results inline), 401/404
DELETE /v1/batch/{id}	cancel	200 cancelled, 401/404

Register them in New and add /v1/batch, /v1/batch/ to the ServeHTTP fast-path switch (no requestDeadline — submit/poll are quick; the worker is not under any request context).

Request/response shapes (exact JSON — the given test compares these)

POST /v1/batch:

{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
              "jq": "…", "context": true, "recover": false}, …]}

Each target carries the same fields as /v1/fetch minus the outer url semantics (each target is its own URL). A targets list is required and non-empty; otherwise 400.

Response 202:

{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}

GET /v1/batch/{id}:

{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
 "created_at": "…", "targets": [
   {"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
   {"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
 ]}

result is the /v1/fetch-shaped envelope (content, mode, untrusted, source, warnings, sizes, timings, and pdf/digest/context when present), written by the runner's FetchFunc adapter. A failed target carries error and optional hint instead of result. Results are inline regardless of status, once a target is finished; a pending target has status: "pending" and no result/error.

DELETE /v1/batch/{id} → 200 {"success": true, "status": "cancelled"}.

Refactor of existing code (explicit — this is the one task that edits working code)

v1fetch today inlines the whole per-target pipeline: guard → doFetch → recover → render/digest → context → envelope. The batch worker needs the same pipeline for one target. Do not copy-paste that pipeline into the batch adapter — extract it into one shared method that both v1fetch and the runner's FetchFunc call:

func (s *Server) fetchTarget(ctx context.Context, c *auth.Caller, t batch.Target) (batch.Result, error)

v1fetch is refactored to call it, keeping its exact response shape and behavior — the existing v1fetch tests must pass unchanged. The runner is wired as a new field s.Batch *batch.Runner, and its FetchFunc is a closure over s.fetchTarget bound to the authenticated caller. This refactor is called out explicitly for reviewer attention: flag any behavioral drift between the two callers. The per-target result is capped by the output cap before it is stored.

Rules

1. Guard pre-check at submit. Run guard.ParseURL (the no-resolution pre-check) on every target URL before accepting. If any target fails, refuse the whole submit with 422 and the offending index + reason (blocked: …). This is the "refused up front" gate — the connect-time guard (private IP, rebinding) cannot run until fetch time and surfaces as a per-target failed, not a submit refusal.
2. Body cap. The global MaxBytesReader caps bodies at maxReqBody (64 KiB); /v1/batch gets its own larger limit (256 KiB) so 32 targets with questions fit. Apply it in the handler.
3. Caps → statuses. Map the runner's typed errors: ErrTooManyTargets → 422; ErrCallerBusy and ErrBusy → 429 with a Retry-After when sensible. Idempotent reuse returns 202 with reused: true.
4. Ownership. GET/DELETE resolve the job by id, then require job.caller == c.Name; else 404 (not 403 — do not reveal that another caller's job exists). The job id is the random opaque token; idem_key is never returned or logged.
5. Restricted redaction. Pass the caller (not just its name) into the runner so each target's fetch goes through doFetch with the real *auth.Caller; the existing logHost/logReason/ logErr then blank host, reason, and error detail automatically. The batch handler's own log line logs caller, job_id, target count, and status — never a target URL or hostname, for any caller.
6. The adapter. The runner's FetchFunc is the shared s.fetchTarget (see the refactor section above), bound to the authenticated *auth.Caller. It wraps doFetch + recover + digest + context + envelope, keyed to one batch.Target, returning batch.Result (whose Meta carries the envelope's source/digest/pdf/context/warnings/sizes/timings). The per-target result is capped by the output cap before it is stored.
7. Job status on the wire uses the runner's vocabulary (queued|running|done|partial|failed| cancelled); unknown id → 404. All three endpoints require a bearer key via the existing caller helper (MetricsOnly → 403).

Given

Copy _files/api/batch_test.go → internal/api/batch_test.go. It builds a Server with s.Batch wired over an in-memory dcache (pinning the s.Batch *batch.Runner field and the s.fetchTarget adapter), then drives the endpoints. Asserts, at least:

• submit → 202 with a job_id; resubmit of the same body → same id, reused: true;
• a target the guard pre-check rejects (non-80/443 port) → 422 and no job created;
• 33 targets → 422;
• a private-address target (passes pre-check, blocked at connect) → per-target failed with error containing blocked, job aggregate partial;
• GET by another caller's key → 404; DELETE by the owner → 200, then GET → cancelled;

Gate

make gate green; go test ./internal/api/ green under -race.