kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

P6 plan, owner review folded in: caller travels with the job, 1–2 round-robin workers, busy retry, summary reads, local-only tests, 8 tasks

Browse Source 
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
  caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
  busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
  APIs it uses; every given test compiles against a panic-only skeleton and fails on it.

P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-26 17:55:50 -07:00
co-authored by Claude Opus 5.5
parent 63e70e19fc
commit 218eaa8b53
20 changed files with 1854 additions and 617 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/P6.md
+108 -31
View File
@@ -32,8 +32,11 @@ list, walk away, and collect once. It builds on the P5 infrastructure that alrea
| Method | Path | Result |
|---|---|---|
| `POST /v1/batch` | submit a list of targets | `202` created/reused, `400` malformed, `422` refused up front or over a cap, `429` caller/jobs busy |
| `GET /v1/batch/{id}` | status + per-target results (inline) | `200`, `401`/`404` |
| `DELETE /v1/batch/{id}` | cancel | `200`, `401`/`404` |
| `GET /v1/batch/{id}` | status summary; `?results=1` inlines results | `200`, `401`/`404` |
| `GET /v1/batch/{id}/{idx}` | one target's status and result | `200`, `401`/`404` |
| `DELETE /v1/batch/{id}` | cancel | `200`, `409` already finished, `401`/`404` |
With `[batch] disabled = true` every batch endpoint answers `503 batch disabled`.
`POST /v1/batch` request — each target is the `/v1/fetch` field set minus the outer URL:
@@ -47,17 +50,20 @@ Response `202`:
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
```
`GET /v1/batch/{id}`:
`GET /v1/batch/{id}` (a summary: no results, since 32 inline results can be ~6 MB in one agent
tool call):
```json
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
"created_at": "…", "targets": [
{"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
{"job_id": "…", "status": "partial", "targets_total": 3, "targets_done": 3,
"created_at": "2026-09-27T01:02:03Z", "targets": [
{"idx": 0, "url": "…", "status": "done"},
{"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
]}
```
Results are inline regardless of job status, once a target is finished. A `pending` target has
`status: "pending"` and no `result`/`error`. `DELETE` → `200 {"success": true, "status":
`?results=1` adds `"result"` (the `/v1/fetch` envelope) to every `done` target.
`GET /v1/batch/{id}/{idx}` returns `{"job_id", "idx", "url", "status", "result"?, "error"?,
"hint"?}`. A `pending` target has `status: "pending"` and no `result`/`error`. A `202` for a new
job says `"status": "queued"`; a reused one says the job's current status. `DELETE` → `200 {"success": true, "status":
"cancelled"}`.
Job status vocabulary: `queued → running → done | partial | failed | cancelled`. Target status:
@@ -87,7 +93,15 @@ the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.
Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status
(`queued`/`running`), not lease. Exceeding a cap is a typed error the API maps to `422`/`429`.
- **The caller.** `Submit` takes the `*auth.Caller`; the runner keeps it in memory with the job
and passes it to every `FetchFunc` call (`func(ctx, *auth.Caller, Target)`), so the fetch runs
under the caller's gates, pool slot, cache class and redaction. Only its name and class are
stored.
- **Workers.** `workers` (default 1, at most 2) share one round-robin over the live jobs, one
target at a time. A job is `queued` until a worker first takes it, then `running`.
- **Serial.** Targets drain in `idx` order; target `i+1` does not start until `i` is written.
- **Busy.** A `FetchFunc` error wrapping `ErrRetryLater` (the API maps fetch `429`/`503` to it)
leaves the target pending; it is tried again after 2 s, for up to 10 min, then failed.
- **Failure isolation.** One target's error is stored per-target and the job continues; the job
status is `done` (all ok), `failed` (all failed), or `partial` (mixed).
- **Cancel.** `DELETE` flips the job `cancelled`, stops before the next target, and lets the
@@ -95,15 +109,26 @@ the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.
## 6. Scheduling and fairness
The worker runs in the background, and every target still goes through the normal `acquire` gate
(8 global, 2 per-host). On `429 busy` the worker **waits and retries** instead of failing — nobody
is holding a socket open. Interactive requests are served by HTTP handlers that grab slots
immediately, so they win by construction; the batch is lower priority for free.
The workers run in the background, and every target still goes through the normal `acquire` gate
(8 global, 2 per-host). With 1–2 workers the batch holds at most 1–2 of the 8 global slots, so
interactive requests always have room. (The first draft let 8 live jobs each run a target at
once, which could fill the whole gate.) On `429 busy` the worker **waits and retries** instead of
failing: nobody is holding a socket open.
Two shared resources get the same treatment:
- **The PDF pool.** A batch extraction uses the pool caller name `<caller>/batch`, so it never
takes the caller's own interactive slot. The single-flight key is unchanged, so a batch and an
interactive request for the same PDF still share one child.
- **The digest gate.** A batch question queues on it like any other digest. With one worker
that is at most one batch digest at a time.
Each batch fetch runs under the same 210 s deadline as a `/v1/fetch` request.
## 7. Retention and TTL
A job carries a single sliding **lease**, bumped on submit, on every target completion, and on
every status/collect read. All targets therefore share one expiry (the last completion's), and a
every read by its owner (the API calls `Touch` after the ownership check; the store's reads
never write). All targets therefore share one expiry (the last completion's), and a
slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease
is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease
lapses; `queued`/`running` jobs are marked `failed` (`restart`) on startup and never resumed.
@@ -134,11 +159,13 @@ becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).
## 10. CLI and skill
- `sift batch submit <targets.json>` (`-` reads stdin; prints `job_id` and `reused`).
- `sift batch status <job_id>` (prints the full status body).
- `sift batch status [--results] <job_id>` (the summary; `--results` inlines every result).
- `sift batch get <job_id> <idx>` (one result).
- `sift batch cancel <job_id>`.
- `skill/SKILL.md`: use `sift batch submit` for a list of targets instead of a loop of
`sift fetch`; `sift batch status` to collect.
- `[batch]` config block: `targets_per_job`, `jobs_per_caller`, `jobs_total`, `ttl`, `max_bytes`.
- `[batch]` config block: `disabled`, `targets_per_job`, `jobs_per_caller`, `jobs_total`,
`workers`, `ttl`, `max_bytes`.
The `/v1/batch` body cap (256 KiB, vs the 64 KiB global) is a constant, not config.
## 11. Refactor of existing code (explicit)
@@ -146,28 +173,39 @@ becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).
`v1fetch` today inlines the whole per-target pipeline (guard → `doFetch` → `recover` →
render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is
extracted into one shared method, `func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target)
(batch.Result, error)`, that both `v1fetch` and the runner's `FetchFunc` call. `v1fetch` keeps its
(batch.Result, error)`, that both `v1fetch` and the runner's adapter (`batchFetch`) call. It is its
own task (04), done before any batch endpoint exists, so the diff is a pure move. `v1fetch` keeps its
exact response shape and behavior — its existing tests must pass unchanged. This is the one change
to working code in P6 and is called out for reviewer attention.
## 12. Acceptance (contract) tests
Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the
intended reasons. See `docs/plans/p6/_files/`.
intended reasons. See `docs/plans/p6/_files/` and the task table in `docs/plans/p6/README.md`.
1. `dcache`: create/read round trip, idempotency lookup (lease expiry + caller scope), live-job
counts, ordered targets, exactly-once `targets_done`/lease bump, sweep, byte-eviction
(oldest-first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
2. `batch` runner: serial order (peak in-flight 1), failure isolation (`partial`), idempotent
resubmit (`reused`, no double-drain), the three caps, cancel (in-flight stored, later targets
`pending`), panic recovery.
3. `api`: submit `202` + idempotent resubmit; a guard-pre-check rejection (non-80/443 port) is
`422` at submit; 33 targets `422`; a private-address target (passes pre-check, blocked at
connect) is a per-target `failed`, aggregate `partial`; ownership (`GET` by another caller →
`404`); cancel → `cancelled`.
4. `metrics`: exact series names, label values, gauge values, histogram counts, and raw-outcome
folding to `other`.
5. `cmd/sift`: `parseBatchArgs` verb/arg pairs and error cases.
1. `dcache`: round trip, idempotency lookup (lease expiry, caller scope, status filter), live
counts, ordered targets, exactly-once `targets_done` and lease bumps, sweep, byte eviction
(oldest first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
2. `batch` runner:
- serial order (peak in flight 1), with the caller passed to every fetch;
- round-robin between jobs, and `queued` → `running`;
- failure isolation (`partial`, `failed`);
- idempotent resubmit, including after `done`, but a new job after a cancel;
- all three caps and `ErrNoTargets`; cancel (the in-flight target stored, later ones `pending`);
- busy retry and its limit; panic recovery (the panic value not echoed); the hooks.
3. `metrics`: series names, labels, folding to `other`.
4. `api` `fetchTarget`: the same content and envelope as `/v1/fetch`; its errors as `*fetch.Error`.
5. `api` wiring: the runner through `fetchTarget`, a private-resolving name blocked per target,
metrics through the hooks, outcome classes, busy → `ErrRetryLater`.
6. `guard.PreCheck`: no resolution; the port, scheme, userinfo and literal rules; the allow list.
7. `api` endpoints:
- submit `202` and idempotent resubmit;
- `400`/`422` refusals (the guard pre-check naming the index, 33 targets);
- the 256 KiB body; the per-target `blocked` failure giving `partial`;
- summary, one result, `?results=1`;
- ownership `404` (GET, one result, DELETE); cancel and `409`;
- no job id in `/metrics`.
8. `config`: the `[batch]` block, `ParseSize`. `cmd/sift`: `parseBatchArgs`.
## 13. Rollout
@@ -180,4 +218,43 @@ intended reasons. See `docs/plans/p6/_files/`.
## 14. Changes after the plan was written
(none yet)
**2026-09-26 evening, owner review (Claude), at Kyle's direction ("fold them into the plan").**
The design stands; these were implementation-blocking or would have hurt agents:
1. **The caller was lost.** `FetchFunc` had no caller and the runner was built once, so the given
api test (`batch.New(db, s.fetchTarget, …)`) could not compile against the specified
`fetchTarget(ctx, *auth.Caller, Target)`. Made to compile, it would have fetched every target
with no caller: no per-caller gates, pool limit, cache class or redaction. Now `Submit` takes
the `*auth.Caller` and `FetchFunc` receives it (§5).
2. **Task 01 rule 1 contradicted its test.** Store reads no longer bump the lease; the API calls
`Touch` after an owner's read (§7).
3. **Batch could starve interactive fetches.** 8 live jobs × 1 target filled the 8-slot global
gate. The busy-retry of §6 was in no task. Now there are 1–2 round-robin workers, busy-retry
is in the runner, a batch PDF uses its own pool name, and a per-target deadline applies (§5, §6).
4. **Flaky, networked API tests.** Two tests fetched `https://example.com/`, and the cancel test
raced the fetch. Every api test is now local, using held `httptest` servers and a fake DNS
name for the private-address case.
5. **A resubmit could return a dead job.** A failed or cancelled job was reused for 24 h, and
polling it kept extending that. Now only `queued`/`running`/`done`/`partial` jobs are reused.
6. **Reads were too large.** `GET` inlined every result (up to ~6 MB). It is now a summary, with
`GET …/{idx}` and `?results=1` (§3). The CLI gains `get` and `status --results`.
7. **Body cap.** The global 64 KiB `MaxBytesReader` runs in `ServeHTTP`, so a handler cannot
raise it. `ServeHTTP` now picks 256 KiB for `/v1/batch`. Request metrics use a fixed
`/v1/batch/{id}` label, so a job id never becomes a label value.
8. **Submit pre-check.** `guard.ParseURL` rejects the test's own loopback port and knows nothing
of the allow list. New `Guard.PreCheck` is `CheckURL` without resolution (given verbatim in
task 05). A private IP literal is now refused at submit; a *name* that resolves privately
still fails per target at fetch time.
9. **Task shape.** 5 tasks → 8, one package each where possible. The old task 03 (refactor +
endpoints + main) is now 04 (refactor only), 05 (runner wiring), 06 (endpoints) and 07
(serve/config). Every task restates the APIs it uses from earlier tasks. Every given test
compiles against a panic-only skeleton and fails there on `panic: todo`.
10. **Also fixed:**
- the `ErrBusy` (jobs total) cap now has a test;
- the `202` status is `queued`, consistent with the workers;
- the `restart` outcome is counted by `serve`;
- `Options.Context` is `*bool`, keeping `/v1/fetch`'s default.
**Sequencing.** P6 starts after P5c merges. P5b/P5c change `v1fetch` (the digest meta gains
`retrieval`/`embedded`), which is the code task 04 moves. Recheck the api given tests against
a skeleton built on post-P5c master before the run.