P6 plan: asynchronous batch fetch endpoint — phase spec, task plan, given tests
Co-Authored-By: DeepSeek V4 Pro <noreply@deepseek.com>
This commit is contained in:
DeepSeek V4 Pro
@@ -0,0 +1,110 @@
|
||||
# 03-api — `/v1/batch` endpoints in `internal/api`
|
||||
|
||||
Wire the batch runner into the API server. This is the HTTP surface and the only place the guard,
|
||||
auth, caps-as-HTTP-statuses, and restricted-caller redaction meet the runner. New code goes in a
|
||||
new file `internal/api/batch.go`; do **not** grow `api.go`.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Path | Result |
|
||||
|---|---|---|
|
||||
| `POST /v1/batch` | submit a list of targets | `202` job created/reused, `400` malformed, `422` a target refused up front or over a cap, `429` caller/jobs busy |
|
||||
| `GET /v1/batch/{id}` | status + per-target results | `200` (results inline), `401`/`404` |
|
||||
| `DELETE /v1/batch/{id}` | cancel | `200` cancelled, `401`/`404` |
|
||||
|
||||
Register them in `New` and add `/v1/batch`, `/v1/batch/` to the `ServeHTTP` fast-path switch
|
||||
(no `requestDeadline` — submit/poll are quick; the worker is not under any request context).
|
||||
|
||||
## Request/response shapes (exact JSON — the given test compares these)
|
||||
|
||||
`POST /v1/batch`:
|
||||
```json
|
||||
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
|
||||
"jq": "…", "context": true, "recover": false}, …]}
|
||||
```
|
||||
Each target carries the same fields as `/v1/fetch` minus the outer `url` semantics (each target is
|
||||
its own URL). A `targets` list is required and non-empty; otherwise `400`.
|
||||
|
||||
Response `202`:
|
||||
```json
|
||||
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
|
||||
```
|
||||
|
||||
`GET /v1/batch/{id}`:
|
||||
```json
|
||||
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
|
||||
"created_at": "…", "targets": [
|
||||
{"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
|
||||
{"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
|
||||
]}
|
||||
```
|
||||
`result` is the `/v1/fetch`-shaped envelope (content, mode, untrusted, source, warnings, sizes,
|
||||
timings, and pdf/digest/context when present), written by the runner's `FetchFunc` adapter. A
|
||||
`failed` target carries `error` and optional `hint` instead of `result`. Results are inline
|
||||
regardless of status, once a target is finished; a pending target has `status: "pending"` and no
|
||||
`result`/`error`.
|
||||
|
||||
`DELETE /v1/batch/{id}` → `200 {"success": true, "status": "cancelled"}`.
|
||||
|
||||
## Refactor of existing code (explicit — this is the one task that edits working code)
|
||||
|
||||
`v1fetch` today inlines the whole per-target pipeline: guard → `doFetch` → `recover` →
|
||||
render/digest → context → envelope. The batch worker needs the *same* pipeline for one target. Do
|
||||
**not** copy-paste that pipeline into the batch adapter — extract it into one shared method that
|
||||
both `v1fetch` and the runner's `FetchFunc` call:
|
||||
|
||||
```go
|
||||
func (s *Server) fetchTarget(ctx context.Context, c *auth.Caller, t batch.Target) (batch.Result, error)
|
||||
```
|
||||
|
||||
`v1fetch` is refactored to call it, keeping its exact response shape and behavior — the existing
|
||||
`v1fetch` tests must pass unchanged. The runner is wired as a new field `s.Batch *batch.Runner`,
|
||||
and its `FetchFunc` is a closure over `s.fetchTarget` bound to the authenticated caller. This
|
||||
refactor is called out explicitly for reviewer attention: flag any behavioral drift between the
|
||||
two callers. The per-target result is capped by the output cap before it is stored.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **Guard pre-check at submit.** Run `guard.ParseURL` (the no-resolution pre-check) on every
|
||||
target URL before accepting. If any target fails, refuse the **whole** submit with `422` and
|
||||
the offending index + reason (`blocked: …`). This is the "refused up front" gate — the
|
||||
connect-time guard (private IP, rebinding) cannot run until fetch time and surfaces as a
|
||||
per-target `failed`, not a submit refusal.
|
||||
2. **Body cap.** The global `MaxBytesReader` caps bodies at `maxReqBody` (64 KiB); `/v1/batch`
|
||||
gets its own larger limit (256 KiB) so 32 targets with questions fit. Apply it in the handler.
|
||||
3. **Caps → statuses.** Map the runner's typed errors: `ErrTooManyTargets` → `422`; `ErrCallerBusy`
|
||||
and `ErrBusy` → `429` with a `Retry-After` when sensible. Idempotent reuse returns `202` with
|
||||
`reused: true`.
|
||||
4. **Ownership.** `GET`/`DELETE` resolve the job by id, then require `job.caller == c.Name`; else
|
||||
`404` (not `403` — do not reveal that another caller's job exists). The job id is the random
|
||||
opaque token; `idem_key` is never returned or logged.
|
||||
5. **Restricted redaction.** Pass the caller (not just its name) into the runner so each target's
|
||||
fetch goes through `doFetch` with the real `*auth.Caller`; the existing `logHost`/`logReason`/
|
||||
`logErr` then blank host, reason, and error detail automatically. The batch handler's own log
|
||||
line logs `caller`, `job_id`, target count, and status — **never** a target URL or hostname, for
|
||||
any caller.
|
||||
6. **The adapter.** The runner's `FetchFunc` is the shared `s.fetchTarget` (see the refactor
|
||||
section above), bound to the authenticated `*auth.Caller`. It wraps `doFetch` + `recover` +
|
||||
digest + context + envelope, keyed to one `batch.Target`, returning `batch.Result` (whose `Meta`
|
||||
carries the envelope's source/digest/pdf/context/warnings/sizes/timings). The per-target result
|
||||
is capped by the output cap before it is stored.
|
||||
7. **Job status on the wire** uses the runner's vocabulary (`queued|running|done|partial|failed|
|
||||
cancelled`); unknown id → `404`. All three endpoints require a bearer key via the existing
|
||||
`caller` helper (`MetricsOnly` → `403`).
|
||||
|
||||
## Given
|
||||
|
||||
Copy `_files/api/batch_test.go` → `internal/api/batch_test.go`. It builds a Server with `s.Batch`
|
||||
wired over an in-memory `dcache` (pinning the `s.Batch *batch.Runner` field and the
|
||||
`s.fetchTarget` adapter), then drives the endpoints. Asserts, at least:
|
||||
|
||||
- submit → `202` with a `job_id`; resubmit of the same body → same id, `reused: true`;
|
||||
- a target the guard pre-check rejects (non-80/443 port) → `422` and no job created;
|
||||
- 33 targets → `422`;
|
||||
- a private-address target (passes pre-check, blocked at connect) → per-target `failed` with
|
||||
`error` containing `blocked`, job aggregate `partial`;
|
||||
- `GET` by another caller's key → `404`; `DELETE` by the owner → `200`, then `GET` → `cancelled`;
|
||||
|
||||
## Gate
|
||||
|
||||
`make gate` green; `go test ./internal/api/` green under `-race`.
|
||||
Reference in New Issue
Block a user