P6 plan: asynchronous batch fetch endpoint — phase spec, task plan, given tests
Co-Authored-By: DeepSeek V4 Pro <noreply@deepseek.com>
This commit is contained in:
DeepSeek V4 Pro
+183
@@ -0,0 +1,183 @@
|
|||||||
|
# Sift P6 — asynchronous batch fetch
|
||||||
|
|
||||||
|
Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer:
|
||||||
|
Claude (when it frees up this evening). Implementer: Kyle's call.
|
||||||
|
|
||||||
|
## 1. Scope
|
||||||
|
|
||||||
|
In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the
|
||||||
|
same per-target options as `/v1/fetch`) and gets a job ID back immediately; it polls for per-target
|
||||||
|
results later. One job drains **serially** — one target at a time — in the background.
|
||||||
|
|
||||||
|
Out:
|
||||||
|
|
||||||
|
- Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are
|
||||||
|
already expected to take a long time, not for interactive fetches.
|
||||||
|
- Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
|
||||||
|
- Firecrawl compatibility. This is a native-only feature; no Hermes plugin.
|
||||||
|
|
||||||
|
The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a
|
||||||
|
page, through the same SSRF guard and the same per-caller/host gates.
|
||||||
|
|
||||||
|
## 2. Why agents need this (2026-09-26)
|
||||||
|
|
||||||
|
`docs/PLAN.md` §13 (Kyle): "an agent that has collected a series of targets and wants them fetched
|
||||||
|
without holding a request open per target." Today that means a loop of `sift fetch` calls — one
|
||||||
|
request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole
|
||||||
|
list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the
|
||||||
|
`dcache` SQLite store, the `global`/`per` concurrency gates, and the `doFetch` pipeline.
|
||||||
|
|
||||||
|
## 3. Endpoints and wire shapes
|
||||||
|
|
||||||
|
| Method | Path | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| `POST /v1/batch` | submit a list of targets | `202` created/reused, `400` malformed, `422` refused up front or over a cap, `429` caller/jobs busy |
|
||||||
|
| `GET /v1/batch/{id}` | status + per-target results (inline) | `200`, `401`/`404` |
|
||||||
|
| `DELETE /v1/batch/{id}` | cancel | `200`, `401`/`404` |
|
||||||
|
|
||||||
|
`POST /v1/batch` request — each target is the `/v1/fetch` field set minus the outer URL:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
|
||||||
|
"jq": "…", "context": true, "recover": false}, …]}
|
||||||
|
```
|
||||||
|
|
||||||
|
Response `202`:
|
||||||
|
```json
|
||||||
|
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
|
||||||
|
```
|
||||||
|
|
||||||
|
`GET /v1/batch/{id}`:
|
||||||
|
```json
|
||||||
|
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
|
||||||
|
"created_at": "…", "targets": [
|
||||||
|
{"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
|
||||||
|
{"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
|
||||||
|
]}
|
||||||
|
```
|
||||||
|
|
||||||
|
Results are inline regardless of job status, once a target is finished. A `pending` target has
|
||||||
|
`status: "pending"` and no `result`/`error`. `DELETE` → `200 {"success": true, "status":
|
||||||
|
"cancelled"}`.
|
||||||
|
|
||||||
|
Job status vocabulary: `queued → running → done | partial | failed | cancelled`. Target status:
|
||||||
|
`pending → running → done | failed`.
|
||||||
|
|
||||||
|
## 4. Storage (`dcache`)
|
||||||
|
|
||||||
|
Two tables, added to `internal/dcache` (which owns the single SQLite connection):
|
||||||
|
|
||||||
|
- `batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done,
|
||||||
|
created_at, lease, nbytes)`
|
||||||
|
- `batch_target(job_id, idx, url, options, status, result, error, hint)` — `result` is the JSON
|
||||||
|
envelope, `NULL` until the target finishes.
|
||||||
|
|
||||||
|
Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, **not** by
|
||||||
|
the content cache's LRU. `EvictBatchOldest` evicts whole jobs oldest-`lease`-first; a
|
||||||
|
`queued`/`running` job is never byte-evicted. `MarkInFlightFailed(reason, now)` is the restart hook.
|
||||||
|
|
||||||
|
## 5. The runner (`internal/batch`)
|
||||||
|
|
||||||
|
A new package with the job lifecycle and the serial state machine. Persistence is the `*dcache.DB`;
|
||||||
|
the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.
|
||||||
|
|
||||||
|
- **Idempotency (in v1).** A job has a random opaque `id` plus a derived `idem_key =
|
||||||
|
SHA256(caller + "\x00" + canonical target list)`. Resubmitting the same (caller, targets) while
|
||||||
|
the job's lease has not expired returns the existing job (`reused: true`) instead of a new one.
|
||||||
|
Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
|
||||||
|
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status
|
||||||
|
(`queued`/`running`), not lease. Exceeding a cap is a typed error the API maps to `422`/`429`.
|
||||||
|
- **Serial.** Targets drain in `idx` order; target `i+1` does not start until `i` is written.
|
||||||
|
- **Failure isolation.** One target's error is stored per-target and the job continues; the job
|
||||||
|
status is `done` (all ok), `failed` (all failed), or `partial` (mixed).
|
||||||
|
- **Cancel.** `DELETE` flips the job `cancelled`, stops before the next target, and lets the
|
||||||
|
in-flight target finish and store. Completed targets stay readable until the lease lapses.
|
||||||
|
|
||||||
|
## 6. Scheduling and fairness
|
||||||
|
|
||||||
|
The worker runs in the background, and every target still goes through the normal `acquire` gate
|
||||||
|
(8 global, 2 per-host). On `429 busy` the worker **waits and retries** instead of failing — nobody
|
||||||
|
is holding a socket open. Interactive requests are served by HTTP handlers that grab slots
|
||||||
|
immediately, so they win by construction; the batch is lower priority for free.
|
||||||
|
|
||||||
|
## 7. Retention and TTL
|
||||||
|
|
||||||
|
A job carries a single sliding **lease**, bumped on submit, on every target completion, and on
|
||||||
|
every status/collect read. All targets therefore share one expiry (the last completion's), and a
|
||||||
|
slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease
|
||||||
|
is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease
|
||||||
|
lapses; `queued`/`running` jobs are marked `failed` (`restart`) on startup and never resumed.
|
||||||
|
|
||||||
|
## 8. Restricted callers
|
||||||
|
|
||||||
|
Compartmentalized-access systems: their usage is redacted as much as possible. The existing
|
||||||
|
`logHost`/`logReason`/`logErr` machinery already blanks host, reason and error detail for
|
||||||
|
`PermitRestricted` callers, and the digest/embed caches partition by caller class — the batch
|
||||||
|
inherits all of it by passing the caller through. Two additions: batch log lines never carry a
|
||||||
|
target URL or hostname for **any** caller, and `GET`/`DELETE` enforce job ownership, returning
|
||||||
|
`404` (not `403`) for a job owned by another caller so a caller cannot probe for its existence.
|
||||||
|
The `idem_key` is stored as a hash and never logged alongside its target list.
|
||||||
|
|
||||||
|
## 9. Caps and metrics (sized empirically)
|
||||||
|
|
||||||
|
The caps are starting values, instrumented so they can be re-sized from observed data:
|
||||||
|
|
||||||
|
- `sift_batch_jobs_total{outcome}` — `created|reused|done|partial|failed|cancelled|restart`
|
||||||
|
- `sift_batch_targets_total{outcome}` — `ok|blocked|timeout|busy|upstream_4xx|upstream_5xx|
|
||||||
|
unsupported|too_long|cancel`
|
||||||
|
- `sift_batch_queue_depth`, `sift_batch_targets_inflight` (gauges)
|
||||||
|
- `sift_batch_job_seconds`, `sift_batch_target_seconds`, `sift_batch_result_bytes` (histograms)
|
||||||
|
|
||||||
|
Both `outcome` counters fold any value outside their known set to `other`, so a raw string never
|
||||||
|
becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).
|
||||||
|
|
||||||
|
## 10. CLI and skill
|
||||||
|
|
||||||
|
- `sift batch submit <targets.json>` (`-` reads stdin; prints `job_id` and `reused`).
|
||||||
|
- `sift batch status <job_id>` (prints the full status body).
|
||||||
|
- `sift batch cancel <job_id>`.
|
||||||
|
- `skill/SKILL.md`: use `sift batch submit` for a list of targets instead of a loop of
|
||||||
|
`sift fetch`; `sift batch status` to collect.
|
||||||
|
- `[batch]` config block: `targets_per_job`, `jobs_per_caller`, `jobs_total`, `ttl`, `max_bytes`.
|
||||||
|
The `/v1/batch` body cap (256 KiB, vs the 64 KiB global) is a constant, not config.
|
||||||
|
|
||||||
|
## 11. Refactor of existing code (explicit)
|
||||||
|
|
||||||
|
`v1fetch` today inlines the whole per-target pipeline (guard → `doFetch` → `recover` →
|
||||||
|
render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is
|
||||||
|
extracted into one shared method, `func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target)
|
||||||
|
(batch.Result, error)`, that both `v1fetch` and the runner's `FetchFunc` call. `v1fetch` keeps its
|
||||||
|
exact response shape and behavior — its existing tests must pass unchanged. This is the one change
|
||||||
|
to working code in P6 and is called out for reviewer attention.
|
||||||
|
|
||||||
|
## 12. Acceptance (contract) tests
|
||||||
|
|
||||||
|
Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the
|
||||||
|
intended reasons. See `docs/plans/p6/_files/`.
|
||||||
|
|
||||||
|
1. `dcache`: create/read round trip, idempotency lookup (lease expiry + caller scope), live-job
|
||||||
|
counts, ordered targets, exactly-once `targets_done`/lease bump, sweep, byte-eviction
|
||||||
|
(oldest-first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
|
||||||
|
2. `batch` runner: serial order (peak in-flight 1), failure isolation (`partial`), idempotent
|
||||||
|
resubmit (`reused`, no double-drain), the three caps, cancel (in-flight stored, later targets
|
||||||
|
`pending`), panic recovery.
|
||||||
|
3. `api`: submit `202` + idempotent resubmit; a guard-pre-check rejection (non-80/443 port) is
|
||||||
|
`422` at submit; 33 targets `422`; a private-address target (passes pre-check, blocked at
|
||||||
|
connect) is a per-target `failed`, aggregate `partial`; ownership (`GET` by another caller →
|
||||||
|
`404`); cancel → `cancelled`.
|
||||||
|
4. `metrics`: exact series names, label values, gauge values, histogram counts, and raw-outcome
|
||||||
|
folding to `other`.
|
||||||
|
5. `cmd/sift`: `parseBatchArgs` verb/arg pairs and error cases.
|
||||||
|
|
||||||
|
## 13. Rollout
|
||||||
|
|
||||||
|
1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the `v1fetch` refactor,
|
||||||
|
and the caps/lease semantics.
|
||||||
|
2. Orion pin bump with the `[batch]` config block; no `MemoryMax` change.
|
||||||
|
3. Skill update ships with the pin bump.
|
||||||
|
4. Watch `sift_batch_*` and tirith for a week; the success measure is that a research thread's
|
||||||
|
one-request-per-URL fetch loop collapses into a single batch submit.
|
||||||
|
|
||||||
|
## 14. Changes after the plan was written
|
||||||
|
|
||||||
|
(none yet)
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
# 01-batchstore — batch tables, lease, GC, byte budget in `internal/dcache`
|
||||||
|
|
||||||
|
Add the batch job store to the existing `internal/dcache` SQLite database. No HTTP, no worker —
|
||||||
|
this task is storage only. `dcache` owns the single SQLite connection, so the batch tables and
|
||||||
|
their accessors live here; the runner (task 02) and API (task 03) call them.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`dcache.DB` already opens one SQLite connection (`SetMaxOpenConns(1)`, WAL) in `Open`, owns
|
||||||
|
`maxBytes`, `evict()`, `Size()`, and `Stats()`. The batch store is a sibling, not a member of the
|
||||||
|
content-addressed cache: a batch job is an atomic unit (a row plus its target rows) and must be
|
||||||
|
evicted whole by lease, never row-by-row by the content LRU.
|
||||||
|
|
||||||
|
## Schema (add to `Open`)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS batch_job (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
caller TEXT NOT NULL,
|
||||||
|
caller_class TEXT NOT NULL, -- 'fleet' | 'restricted'
|
||||||
|
idem_key TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL, -- queued|running|done|partial|failed|cancelled
|
||||||
|
error TEXT NOT NULL DEFAULT '',
|
||||||
|
targets_total INTEGER NOT NULL,
|
||||||
|
targets_done INTEGER NOT NULL DEFAULT 0,
|
||||||
|
created_at INTEGER NOT NULL, -- unix seconds
|
||||||
|
lease INTEGER NOT NULL, -- unix seconds; bumped on submit/completion/read
|
||||||
|
nbytes INTEGER NOT NULL DEFAULT 0 -- sum of stored per-target result bytes
|
||||||
|
);
|
||||||
|
CREATE TABLE IF NOT EXISTS batch_target (
|
||||||
|
job_id TEXT NOT NULL,
|
||||||
|
idx INTEGER NOT NULL, -- order within the job, 0..n-1
|
||||||
|
url TEXT NOT NULL,
|
||||||
|
options TEXT NOT NULL, -- JSON: the per-target options
|
||||||
|
status TEXT NOT NULL, -- pending|running|done|failed
|
||||||
|
result TEXT, -- JSON result envelope, or NULL
|
||||||
|
error TEXT, -- error message, or NULL
|
||||||
|
hint TEXT, -- error hint, or NULL
|
||||||
|
PRIMARY KEY (job_id, idx)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS batch_job_idem ON batch_job(caller, idem_key);
|
||||||
|
CREATE INDEX IF NOT EXISTS batch_job_lease ON batch_job(lease);
|
||||||
|
```
|
||||||
|
|
||||||
|
`result`/`error`/`hint` are `NULL` until the target finishes. The `result` JSON is the
|
||||||
|
`/v1/fetch`-shaped envelope, bounded by the output cap at write time (task 03 enforces the cap;
|
||||||
|
the store just stores what it is given and counts its bytes into `nbytes`).
|
||||||
|
|
||||||
|
## Public surface (exact names — the given test compiles against them)
|
||||||
|
|
||||||
|
```go
|
||||||
|
type BatchJob struct {
|
||||||
|
ID string
|
||||||
|
Caller string
|
||||||
|
CallerClass string
|
||||||
|
IDemKey string
|
||||||
|
Status string
|
||||||
|
Error string
|
||||||
|
TargetsTotal int
|
||||||
|
TargetsDone int
|
||||||
|
CreatedAt int64
|
||||||
|
Lease int64
|
||||||
|
NBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type BatchTarget struct {
|
||||||
|
JobID string
|
||||||
|
Idx int
|
||||||
|
URL string
|
||||||
|
Options string
|
||||||
|
Status string
|
||||||
|
Result *string
|
||||||
|
Error *string
|
||||||
|
Hint *string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *DB) CreateBatchJob(j BatchJob) error
|
||||||
|
func (d *DB) BatchJob(id string) (BatchJob, bool)
|
||||||
|
func (d *DB) BatchJobByIDem(caller, idemKey string, now int64, ttl time.Duration) (string, bool)
|
||||||
|
func (d *DB) LiveBatchJobs(caller string) int
|
||||||
|
func (d *DB) LiveBatchJobsTotal() int
|
||||||
|
func (d *DB) PutBatchTargets(jobID string, targets []BatchTarget) error
|
||||||
|
func (d *DB) BatchTargets(jobID string) ([]BatchTarget, bool)
|
||||||
|
func (d *DB) UpdateBatchTarget(jobID string, idx int, now int64, status string, result, errMsg, hint *string) error
|
||||||
|
func (d *DB) SetBatchJobStatus(id, status, errMsg string, now int64) error
|
||||||
|
func (d *DB) TouchBatchJob(id string, now int64) error
|
||||||
|
func (d *DB) SweepBatch(now int64, ttl time.Duration) int
|
||||||
|
func (d *DB) EvictBatchOldest(maxBytes int64) int
|
||||||
|
func (d *DB) MarkInFlightFailed(reason string, now int64) int
|
||||||
|
func (d *DB) BatchNbytes() int64
|
||||||
|
```
|
||||||
|
|
||||||
|
Every method that touches time takes an explicit `now` (unix seconds) — the store is pure and the
|
||||||
|
caller decides the clock, so the given test asserts exact lease values.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **Sliding lease.** `CreateBatchJob` stores the job as given (the caller sets `Lease`/`CreatedAt`).
|
||||||
|
`UpdateBatchTarget`, `SetBatchJobStatus`, `TouchBatchJob`, and `MarkInFlightFailed` bump the job
|
||||||
|
lease to their `now`. `BatchJob` (a read) also bumps the lease to the current wall-clock time so
|
||||||
|
a slow consumer's reads keep the job alive.
|
||||||
|
2. **No partial-job reads leak a competing caller.** Every accessor is keyed by `job_id` only; the
|
||||||
|
caller filter is the API layer's job (task 03). The store stores what it is told and returns
|
||||||
|
what is asked.
|
||||||
|
3. `BatchTargets` returns targets in `idx` order. `UpdateBatchTarget` increments `targets_done`
|
||||||
|
exactly once per target that transitions *into* `done`/`failed` (a re-write of the same target
|
||||||
|
does not double-count), bumps the job lease to `now`, and adds `len(result)` to `nbytes` when a
|
||||||
|
result is written.
|
||||||
|
4. `SweepBatch(now, ttl)` deletes every job whose `lease + ttl <= now`, and its target rows, and
|
||||||
|
returns the number of jobs deleted. It is the only thing that removes a finished job by time.
|
||||||
|
5. `EvictBatchOldest(maxBytes)` evicts **whole jobs** (oldest `lease` first) until `BatchNbytes()`
|
||||||
|
is `<= maxBytes`, and returns the number evicted. A job in `queued`/`running` is never
|
||||||
|
byte-evicted (only lease-GC, cancel, or restart removes an active job). It runs after a target
|
||||||
|
result is written and is separate from the content-cache `evict()`.
|
||||||
|
6. `MarkInFlightFailed(reason, now)` marks every `queued`/`running` job `failed` with
|
||||||
|
`error=reason`, bumps their lease to `now`, and returns the count. This is the restart
|
||||||
|
reconciliation hook; `main` calls it on startup (task 03).
|
||||||
|
7. `BatchNbytes()` sums `nbytes` over `batch_job`. Batch bytes are **not** included in `Size()` or
|
||||||
|
the content `evict()` union; batch is a coherent unit with its own budget.
|
||||||
|
8. Accessors follow the existing `dcache` style: return `(value, bool)` or `error`, never panic,
|
||||||
|
count a miss in the appropriate `*M` counter. Reuse the existing `ClassFleet`/`ClassRestricted`
|
||||||
|
constants for `caller_class`.
|
||||||
|
|
||||||
|
## Given
|
||||||
|
|
||||||
|
Copy `_files/dcache/batch_test.go` → `internal/dcache/batch_test.go`. It opens an in-memory `DB`
|
||||||
|
and asserts the create/read round trip, `BatchJobByIDem` lease expiry + caller scoping, live-job
|
||||||
|
counts, ordered targets, the exactly-once `targets_done`/lease-bump rules, `SweepBatch`,
|
||||||
|
`EvictBatchOldest` (oldest-first, never a live job), `MarkInFlightFailed`, and `BatchNbytes`.
|
||||||
|
|
||||||
|
## Gate
|
||||||
|
|
||||||
|
`make gate` green; `go test ./internal/dcache/` green under `-race`.
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
# 02-runner — serial job runner in `internal/batch`
|
||||||
|
|
||||||
|
Add a new package `internal/batch`: the job lifecycle and the serial state machine. It knows how
|
||||||
|
to create a job, drain its targets one at a time, write each target's outcome, bump the lease,
|
||||||
|
handle cancellation, and enforce the caps and idempotency. It knows **nothing** about HTTP, the
|
||||||
|
guard, or the real fetch pipeline: the per-target work is an injected function, and persistence is
|
||||||
|
the `*dcache.DB` from task 01.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`internal/pdfpool` is the wrong shape (in-memory, coalescing, forgets results). The runner is the
|
||||||
|
opposite: store-backed, persistent, one job = many distinct targets, results written to the store
|
||||||
|
as they complete. It is deliberately serial.
|
||||||
|
|
||||||
|
## Public surface (exact names — the given test compiles against them)
|
||||||
|
|
||||||
|
```go
|
||||||
|
package batch
|
||||||
|
|
||||||
|
type Options struct {
|
||||||
|
Mode string `json:"mode,omitempty"`
|
||||||
|
Question string `json:"question,omitempty"`
|
||||||
|
Pages string `json:"pages,omitempty"`
|
||||||
|
Grep string `json:"grep,omitempty"`
|
||||||
|
JQ string `json:"jq,omitempty"`
|
||||||
|
Context bool `json:"context,omitempty"`
|
||||||
|
Recover bool `json:"recover,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type Target struct {
|
||||||
|
URL string `json:"url"`
|
||||||
|
Options Options `json:"options,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Result is a completed target's outcome. The runner JSON-encodes it into the
|
||||||
|
// store's target.result column and decodes it back in Status.
|
||||||
|
type Result struct {
|
||||||
|
Content string `json:"content"`
|
||||||
|
Meta map[string]any `json:"meta,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FetchFunc func(ctx context.Context, t Target) (Result, error)
|
||||||
|
|
||||||
|
// Status views returned by Status (mapped from the store rows).
|
||||||
|
type Job struct {
|
||||||
|
ID string
|
||||||
|
Status string
|
||||||
|
Error string
|
||||||
|
TargetsTotal int
|
||||||
|
TargetsDone int
|
||||||
|
}
|
||||||
|
|
||||||
|
type TargetStatus struct {
|
||||||
|
Idx int
|
||||||
|
URL string
|
||||||
|
Status string
|
||||||
|
Result *Result
|
||||||
|
Error string
|
||||||
|
Hint string
|
||||||
|
}
|
||||||
|
|
||||||
|
type Config struct {
|
||||||
|
TTL time.Duration // 0 → 24h
|
||||||
|
TargetsPerJob int // 0 → 32
|
||||||
|
JobsPerCaller int // 0 → 1
|
||||||
|
JobsTotal int // 0 → 8
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrTooManyTargets = errors.New("too many targets")
|
||||||
|
ErrCallerBusy = errors.New("busy")
|
||||||
|
ErrBusy = errors.New("jobs busy")
|
||||||
|
)
|
||||||
|
|
||||||
|
func New(db *dcache.DB, fetch FetchFunc, cfg Config, log *slog.Logger) *Runner
|
||||||
|
func (r *Runner) Submit(ctx context.Context, caller, callerClass string, targets []Target) (id string, reused bool, err error)
|
||||||
|
func (r *Runner) Cancel(id string) bool
|
||||||
|
func (r *Runner) Status(id string) (Job, []TargetStatus, bool)
|
||||||
|
```
|
||||||
|
|
||||||
|
The runner depends on `*dcache.DB` (task 01) for persistence; the `FetchFunc` is the only injected
|
||||||
|
behavior. Zero `Config` fields fall back to the defaults above.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **Idempotency first.** `Submit` derives `idem_key = SHA256(caller + "\x00" + canonical(targets))`
|
||||||
|
where `canonical` is `json.Marshal` of the decoded targets (so field order and whitespace don't
|
||||||
|
matter). If a job whose lease has not expired (`BatchJobByIDem`) exists for the same
|
||||||
|
`(caller, idem_key)` — any status — return it with `reused=true` and do **not** create a new job
|
||||||
|
or enforce the caps. Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
|
||||||
|
2. **Caps second.** targets/job > `TargetsPerJob` → `ErrTooManyTargets`; a live (`queued`/`running`)
|
||||||
|
job already exists for the caller → `ErrCallerBusy`; live jobs total >= `JobsTotal` → `ErrBusy`.
|
||||||
|
"Live" is by status, not lease — read from `LiveBatchJobs`/`LiveBatchJobsTotal`. The typed errors
|
||||||
|
map to `422`/`429` in the API layer.
|
||||||
|
3. **Submit transitions to `running` synchronously** (before the drain goroutine starts) so a
|
||||||
|
caller-busy check against a just-submitted job is deterministic. The job is created `queued`
|
||||||
|
then immediately `running`; `queued` is only ever persisted as the crash-window state that
|
||||||
|
`MarkInFlightFailed` reaps.
|
||||||
|
4. **Serial drain.** The worker iterates targets in `idx` order, one at a time, and does not start
|
||||||
|
target `i+1` until `i` is written to the store. Each target's `FetchFunc` result or error is
|
||||||
|
written immediately (JSON-encoded); a target failure does not stop the job.
|
||||||
|
5. **Lease on completion.** Each target write bumps the job lease (task 01); the first and last
|
||||||
|
target therefore share one expiry.
|
||||||
|
6. **Cancellation.** `Cancel` flips the job to `cancelled` and signals the worker; the worker
|
||||||
|
finishes the in-flight target, stores it, and stops before the next. A `cancelled` job's
|
||||||
|
completed targets stay readable until the lease lapses. `Cancel` returns false for an unknown id
|
||||||
|
or a job already in a terminal state.
|
||||||
|
7. **Terminal statuses.** When the last target is written: all `done` → `done`; all `failed` →
|
||||||
|
`failed`; a mix → `partial`. Set exactly once.
|
||||||
|
8. **No restart concept in the runner.** It is created fresh each process start; `main` (task 03)
|
||||||
|
calls `dcache.MarkInFlightFailed("restart", now)` before constructing the runner. Completed
|
||||||
|
targets on a restart-killed job stay readable; the job is never resumed.
|
||||||
|
9. **No panic on input.** Library code never panics: bad ids, missing targets, a `FetchFunc` panic
|
||||||
|
is recovered into a target error and the job continues. The drain runs under
|
||||||
|
`context.WithoutCancel` of a background context; only the per-job cancellation stops it.
|
||||||
|
|
||||||
|
## Given
|
||||||
|
|
||||||
|
Copy `_files/batch/runner_test.go` → `internal/batch/runner_test.go`. It uses an in-memory `dcache`
|
||||||
|
plus a `FetchFunc` that records call order and can fail or panic. Asserts serial order (peak
|
||||||
|
in-flight 1), failure isolation (`partial`), idempotent resubmit (`reused`, no double-drain), the
|
||||||
|
three caps, cancel (in-flight stored, later targets `pending`), and panic recovery.
|
||||||
|
|
||||||
|
## Gate
|
||||||
|
|
||||||
|
`make gate` green; `go test ./internal/batch/` green under `-race`.
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
# 03-api — `/v1/batch` endpoints in `internal/api`
|
||||||
|
|
||||||
|
Wire the batch runner into the API server. This is the HTTP surface and the only place the guard,
|
||||||
|
auth, caps-as-HTTP-statuses, and restricted-caller redaction meet the runner. New code goes in a
|
||||||
|
new file `internal/api/batch.go`; do **not** grow `api.go`.
|
||||||
|
|
||||||
|
## Endpoints
|
||||||
|
|
||||||
|
| Method | Path | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| `POST /v1/batch` | submit a list of targets | `202` job created/reused, `400` malformed, `422` a target refused up front or over a cap, `429` caller/jobs busy |
|
||||||
|
| `GET /v1/batch/{id}` | status + per-target results | `200` (results inline), `401`/`404` |
|
||||||
|
| `DELETE /v1/batch/{id}` | cancel | `200` cancelled, `401`/`404` |
|
||||||
|
|
||||||
|
Register them in `New` and add `/v1/batch`, `/v1/batch/` to the `ServeHTTP` fast-path switch
|
||||||
|
(no `requestDeadline` — submit/poll are quick; the worker is not under any request context).
|
||||||
|
|
||||||
|
## Request/response shapes (exact JSON — the given test compares these)
|
||||||
|
|
||||||
|
`POST /v1/batch`:
|
||||||
|
```json
|
||||||
|
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
|
||||||
|
"jq": "…", "context": true, "recover": false}, …]}
|
||||||
|
```
|
||||||
|
Each target carries the same fields as `/v1/fetch` minus the outer `url` semantics (each target is
|
||||||
|
its own URL). A `targets` list is required and non-empty; otherwise `400`.
|
||||||
|
|
||||||
|
Response `202`:
|
||||||
|
```json
|
||||||
|
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
|
||||||
|
```
|
||||||
|
|
||||||
|
`GET /v1/batch/{id}`:
|
||||||
|
```json
|
||||||
|
{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
|
||||||
|
"created_at": "…", "targets": [
|
||||||
|
{"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
|
||||||
|
{"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
|
||||||
|
]}
|
||||||
|
```
|
||||||
|
`result` is the `/v1/fetch`-shaped envelope (content, mode, untrusted, source, warnings, sizes,
|
||||||
|
timings, and pdf/digest/context when present), written by the runner's `FetchFunc` adapter. A
|
||||||
|
`failed` target carries `error` and optional `hint` instead of `result`. Results are inline
|
||||||
|
regardless of status, once a target is finished; a pending target has `status: "pending"` and no
|
||||||
|
`result`/`error`.
|
||||||
|
|
||||||
|
`DELETE /v1/batch/{id}` → `200 {"success": true, "status": "cancelled"}`.
|
||||||
|
|
||||||
|
## Refactor of existing code (explicit — this is the one task that edits working code)
|
||||||
|
|
||||||
|
`v1fetch` today inlines the whole per-target pipeline: guard → `doFetch` → `recover` →
|
||||||
|
render/digest → context → envelope. The batch worker needs the *same* pipeline for one target. Do
|
||||||
|
**not** copy-paste that pipeline into the batch adapter — extract it into one shared method that
|
||||||
|
both `v1fetch` and the runner's `FetchFunc` call:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (s *Server) fetchTarget(ctx context.Context, c *auth.Caller, t batch.Target) (batch.Result, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
`v1fetch` is refactored to call it, keeping its exact response shape and behavior — the existing
|
||||||
|
`v1fetch` tests must pass unchanged. The runner is wired as a new field `s.Batch *batch.Runner`,
|
||||||
|
and its `FetchFunc` is a closure over `s.fetchTarget` bound to the authenticated caller. This
|
||||||
|
refactor is called out explicitly for reviewer attention: flag any behavioral drift between the
|
||||||
|
two callers. The per-target result is capped by the output cap before it is stored.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **Guard pre-check at submit.** Run `guard.ParseURL` (the no-resolution pre-check) on every
|
||||||
|
target URL before accepting. If any target fails, refuse the **whole** submit with `422` and
|
||||||
|
the offending index + reason (`blocked: …`). This is the "refused up front" gate — the
|
||||||
|
connect-time guard (private IP, rebinding) cannot run until fetch time and surfaces as a
|
||||||
|
per-target `failed`, not a submit refusal.
|
||||||
|
2. **Body cap.** The global `MaxBytesReader` caps bodies at `maxReqBody` (64 KiB); `/v1/batch`
|
||||||
|
gets its own larger limit (256 KiB) so 32 targets with questions fit. Apply it in the handler.
|
||||||
|
3. **Caps → statuses.** Map the runner's typed errors: `ErrTooManyTargets` → `422`; `ErrCallerBusy`
|
||||||
|
and `ErrBusy` → `429` with a `Retry-After` when sensible. Idempotent reuse returns `202` with
|
||||||
|
`reused: true`.
|
||||||
|
4. **Ownership.** `GET`/`DELETE` resolve the job by id, then require `job.caller == c.Name`; else
|
||||||
|
`404` (not `403` — do not reveal that another caller's job exists). The job id is the random
|
||||||
|
opaque token; `idem_key` is never returned or logged.
|
||||||
|
5. **Restricted redaction.** Pass the caller (not just its name) into the runner so each target's
|
||||||
|
fetch goes through `doFetch` with the real `*auth.Caller`; the existing `logHost`/`logReason`/
|
||||||
|
`logErr` then blank host, reason, and error detail automatically. The batch handler's own log
|
||||||
|
line logs `caller`, `job_id`, target count, and status — **never** a target URL or hostname, for
|
||||||
|
any caller.
|
||||||
|
6. **The adapter.** The runner's `FetchFunc` is the shared `s.fetchTarget` (see the refactor
|
||||||
|
section above), bound to the authenticated `*auth.Caller`. It wraps `doFetch` + `recover` +
|
||||||
|
digest + context + envelope, keyed to one `batch.Target`, returning `batch.Result` (whose `Meta`
|
||||||
|
carries the envelope's source/digest/pdf/context/warnings/sizes/timings). The per-target result
|
||||||
|
is capped by the output cap before it is stored.
|
||||||
|
7. **Job status on the wire** uses the runner's vocabulary (`queued|running|done|partial|failed|
|
||||||
|
cancelled`); unknown id → `404`. All three endpoints require a bearer key via the existing
|
||||||
|
`caller` helper (`MetricsOnly` → `403`).
|
||||||
|
|
||||||
|
## Given
|
||||||
|
|
||||||
|
Copy `_files/api/batch_test.go` → `internal/api/batch_test.go`. It builds a Server with `s.Batch`
|
||||||
|
wired over an in-memory `dcache` (pinning the `s.Batch *batch.Runner` field and the
|
||||||
|
`s.fetchTarget` adapter), then drives the endpoints. Asserts, at least:
|
||||||
|
|
||||||
|
- submit → `202` with a `job_id`; resubmit of the same body → same id, `reused: true`;
|
||||||
|
- a target the guard pre-check rejects (non-80/443 port) → `422` and no job created;
|
||||||
|
- 33 targets → `422`;
|
||||||
|
- a private-address target (passes pre-check, blocked at connect) → per-target `failed` with
|
||||||
|
`error` containing `blocked`, job aggregate `partial`;
|
||||||
|
- `GET` by another caller's key → `404`; `DELETE` by the owner → `200`, then `GET` → `cancelled`;
|
||||||
|
|
||||||
|
## Gate
|
||||||
|
|
||||||
|
`make gate` green; `go test ./internal/api/` green under `-race`.
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# 04-metrics — `sift_batch_*` series in `internal/metrics`
|
||||||
|
|
||||||
|
Add the batch series and wire them from the runner and API, so the caps (32 targets, 1 job/caller,
|
||||||
|
8 jobs total) and the byte budget can be re-sized from observed data.
|
||||||
|
|
||||||
|
## Series (names and label values exact — the given test asserts these)
|
||||||
|
|
||||||
|
| Metric | Type | Labels / meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `sift_batch_jobs_total` | counter | `outcome` ∈ `created`, `reused`, `done`, `partial`, `failed`, `cancelled`, `restart` |
|
||||||
|
| `sift_batch_targets_total` | counter | `outcome` ∈ `ok`, `blocked`, `timeout`, `busy`, `upstream_4xx`, `upstream_5xx`, `unsupported`, `too_long`, `cancel` |
|
||||||
|
| `sift_batch_queue_depth` | gauge | live jobs (`queued`+`running`) |
|
||||||
|
| `sift_batch_targets_inflight` | gauge | targets currently being fetched |
|
||||||
|
| `sift_batch_job_seconds` | histogram | wall time from submit to terminal status |
|
||||||
|
| `sift_batch_target_seconds` | histogram | wall time per target |
|
||||||
|
| `sift_batch_result_bytes` | histogram | stored bytes per target result |
|
||||||
|
|
||||||
|
`outcome` labels on `sift_batch_targets_total` are derived from the fetch error classification the
|
||||||
|
same way `coarseReason` does (`blocked` for a guard block, `busy` for `429`, `timeout` for `504`,
|
||||||
|
`unsupported` for `415`, `too_long` for the question-too-long `422`, `upstream_4xx`/`upstream_5xx`
|
||||||
|
otherwise; `ok` on success; `cancel` when the target was skipped by a cancel).
|
||||||
|
|
||||||
|
## Public surface (exact names — the given test compiles against them)
|
||||||
|
|
||||||
|
```go
|
||||||
|
// fields added to metrics.Set (registered in New)
|
||||||
|
BatchJobsTotal *prometheus.CounterVec // label "outcome"
|
||||||
|
BatchTargetsTotal *prometheus.CounterVec // label "outcome"
|
||||||
|
BatchQueueDepth prometheus.Gauge
|
||||||
|
BatchTargetsInflight prometheus.Gauge
|
||||||
|
BatchJobSeconds prometheus.Histogram
|
||||||
|
BatchTargetSeconds prometheus.Histogram
|
||||||
|
BatchResultBytes prometheus.Histogram
|
||||||
|
|
||||||
|
// methods (nil-guarded, like ObservePDF)
|
||||||
|
func (s *Set) ObserveBatchJob(outcome string)
|
||||||
|
func (s *Set) ObserveBatchTarget(outcome string)
|
||||||
|
func (s *Set) SetBatchQueueDepth(n int)
|
||||||
|
func (s *Set) SetBatchTargetsInflight(n int)
|
||||||
|
func (s *Set) ObserveBatchJobSeconds(d time.Duration)
|
||||||
|
func (s *Set) ObserveBatchTargetSeconds(d time.Duration)
|
||||||
|
func (s *Set) ObserveBatchResultBytes(n int)
|
||||||
|
```
|
||||||
|
|
||||||
|
Both `outcome` counters fold any value outside their known set to `other` — the same "a raw string
|
||||||
|
never becomes a label" guard as `ObservePDF`. Job outcomes: `created, reused, done, partial,
|
||||||
|
failed, cancelled, restart`. Target outcomes: `ok, blocked, timeout, busy, upstream_4xx,
|
||||||
|
upstream_5xx, unsupported, too_long, cancel`.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. Follow the existing `internal/metrics` pattern: a new `internal/metrics/batch.go` that adds the
|
||||||
|
series to `metrics.Set`, with `Observe…` methods mirroring `ObservePDF`/`ObserveDigest`.
|
||||||
|
2. `sift_batch_jobs_total{outcome}` is incremented exactly once per job, at its terminal
|
||||||
|
transition (or `created`/`reused` at submit, `restart` at reconciliation).
|
||||||
|
3. `sift_batch_targets_total{outcome}` is incremented exactly once per target.
|
||||||
|
4. Gauges reflect the runner's live state: `queue_depth` = live job count; `targets_inflight` = the
|
||||||
|
one in-flight target across jobs (0 or 1 while a job is running, summed over jobs).
|
||||||
|
5. Histograms record durations/bytes regardless of outcome; `result_bytes` is the stored target
|
||||||
|
size, which is what sizes the byte budget.
|
||||||
|
6. The metrics are exposed through the existing `/metrics` and `/metrics.json` paths — no new
|
||||||
|
endpoint. The API server's `Snapshot`/`writeMetricsJSON` do not need to enumerate them (the
|
||||||
|
Prometheus text handler covers them), but the given test reads the `metrics.Set` values directly.
|
||||||
|
|
||||||
|
## Given
|
||||||
|
|
||||||
|
Copy `_files/metrics/batch_test.go` → `internal/metrics/batch_test.go`. It drives the `Observe…`
|
||||||
|
methods, serves the `/metrics` handler (as `pdf_test.go` does), and asserts the exact series names,
|
||||||
|
label values, gauge values, histogram counts, and that a raw outcome string is folded to `other`.
|
||||||
|
|
||||||
|
## Gate
|
||||||
|
|
||||||
|
`make gate` green; `go test ./internal/metrics/` green under `-race`.
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# 05-cli-docs — `sift batch`, skill, README, config
|
||||||
|
|
||||||
|
Expose the batch endpoint from the CLI and document it for agents. No Hermes plugin — the CLI is
|
||||||
|
the agent surface, same rationale as `sift fetch`/`sift search`.
|
||||||
|
|
||||||
|
## CLI
|
||||||
|
|
||||||
|
Add a `sift batch` subcommand with three verbs. It reads `url` and `key_file` from `client.toml`
|
||||||
|
exactly as `fetch`/`search` do, and never puts a key on the command line.
|
||||||
|
|
||||||
|
```
|
||||||
|
sift batch submit <targets.json> # '-' reads stdin; prints the job_id (and reused) as JSON
|
||||||
|
sift batch status <job_id> # prints status + per-target results as JSON
|
||||||
|
sift batch cancel <job_id> # prints the cancelled status as JSON
|
||||||
|
```
|
||||||
|
|
||||||
|
`submit` takes the same JSON as `POST /v1/batch` (a `{"targets":[…]}` object) and prints
|
||||||
|
`{"job_id": …, "reused": …}`. `status` prints the full `GET /v1/batch/{id}` body. All three
|
||||||
|
propagate the server's error envelope and exit non-zero on a non-2xx.
|
||||||
|
|
||||||
|
The dispatch/validation is a pure function pinned by the given test:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func parseBatchArgs(args []string) (verb, arg string, err error)
|
||||||
|
```
|
||||||
|
|
||||||
|
`verb` is `submit`/`status`/`cancel`; `arg` is the targets file (submit, `-` for stdin) or the
|
||||||
|
`job_id` (status/cancel). No verb, a missing arg, an extra arg, or an unknown verb is an error.
|
||||||
|
|
||||||
|
## Config
|
||||||
|
|
||||||
|
Add a `[batch]` block to `internal/config` (`File`), with:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[batch]
|
||||||
|
targets_per_job = 32
|
||||||
|
jobs_per_caller = 1
|
||||||
|
jobs_total = 8
|
||||||
|
ttl = "24h"
|
||||||
|
max_bytes = "64MiB"
|
||||||
|
```
|
||||||
|
|
||||||
|
The runner and API read these instead of hard-coding; the defaults match the settled caps. The
|
||||||
|
body-cap knob for `/v1/batch` (256 KiB) is a constant, not config.
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- `README.md`: add a short "Batch" section (the three endpoints, the serial/background model, caps,
|
||||||
|
idempotency, lease/TTL semantics, restricted-caller redaction, and the `sift_batch_*` metrics).
|
||||||
|
- `skill/SKILL.md`: add a batch section telling agents to use `sift batch submit` for a list of
|
||||||
|
targets rather than a loop of `sift fetch`, and to `sift batch status` to collect.
|
||||||
|
- `dist/sift.toml.example`: add the `[batch]` block with its defaults and a comment.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. The CLI reuses the existing client plumbing (`client.toml`, bearer header, JSON envelope
|
||||||
|
handling) — no new HTTP client.
|
||||||
|
2. `submit` validates its input locally (a non-empty `targets` array) before calling the server.
|
||||||
|
3. `status`/`cancel` require exactly one `job_id` argument.
|
||||||
|
4. Keep each verb a small function; do not grow `main.go` — add `cmd/sift/batchcmd.go`.
|
||||||
|
|
||||||
|
## Given
|
||||||
|
|
||||||
|
Copy `_files/cmd_sift/batch_test.go` → `cmd/sift/batch_test.go`. It pins `parseBatchArgs` — the
|
||||||
|
valid verb+arg pairs (including `-` as a stdin target file) and the error cases (no verb, missing
|
||||||
|
arg, extra arg, unknown verb).
|
||||||
|
|
||||||
|
## Gate
|
||||||
|
|
||||||
|
`make gate` green; `go test ./cmd/sift/` green under `-race`; `go test ./internal/config/` green.
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# P6 implementation plan: asynchronous batch fetch
|
||||||
|
|
||||||
|
> **For the implementing model:** do not work from this file. The owner gives you one task file
|
||||||
|
> at a time. This file is the index for the owner and the reviewer.
|
||||||
|
|
||||||
|
**Goal:** an agent that has collected a series of targets (URLs) and wants them fetched without
|
||||||
|
holding a request open per target submits them as one batch, gets a job ID back immediately, and
|
||||||
|
polls for per-target results later. A batch is **deliberate and slow** — serial, one target at a
|
||||||
|
time — so it is the tool for research threads, not for interactive fetches.
|
||||||
|
|
||||||
|
Decisions (settled with the owner, 2026-09-26):
|
||||||
|
|
||||||
|
- **Serial.** One target drains at a time per job, in the background. Each target still goes
|
||||||
|
through the normal `acquire` gate (8 global / 2 per-host); on `429 busy` the worker waits and
|
||||||
|
retries rather than failing, because nobody is holding a socket open. Interactive requests win
|
||||||
|
by construction: they are served by HTTP handlers that grab slots immediately.
|
||||||
|
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8; per-target result bounded by
|
||||||
|
the output cap. All instrumented (`sift_batch_*`) so the numbers can be re-sized empirically.
|
||||||
|
- **Idempotency in v1.** A job has a random opaque `id` plus a derived `idem_key`; resubmitting
|
||||||
|
the same (caller, targets) while the job is live returns the existing job instead of a new one.
|
||||||
|
- **Storage.** `batch_job` + `batch_target` tables in the existing `dcache` SQLite.
|
||||||
|
- **Retention / TTL.** A job carries a single sliding **lease**, bumped on submit, on every target
|
||||||
|
completion, and on every status/collect read. All targets therefore share one expiry, and a slow
|
||||||
|
consumer's reads keep a finished batch alive. GC deletes a job (and its targets) once its lease
|
||||||
|
is older than the TTL. Batch storage is a coherent whole: it is bounded by the lease GC plus a
|
||||||
|
separate byte budget (oldest-lease jobs evicted whole), not by the content cache's LRU.
|
||||||
|
- **Restart.** Completed results survive restart and stay readable until the lease lapses.
|
||||||
|
`queued`/`running` jobs are marked `failed` (reason `restart`) on startup; never resumed.
|
||||||
|
- **Restricted callers.** Compartmentalized-access systems: their usage is redacted as much as
|
||||||
|
possible (host, error reason, error detail — the existing `logHost`/`logReason`/`logErr`
|
||||||
|
machinery), and no other caller may read their jobs or results. The worker passes the caller
|
||||||
|
through, so this happens automatically; batch log lines never carry a target URL or hostname.
|
||||||
|
|
||||||
|
| Task | Package | Given |
|
||||||
|
|---|---|---|
|
||||||
|
| 01-batchstore | `internal/dcache` | `dcache/batch_test.go` |
|
||||||
|
| 02-runner | `internal/batch` | `batch/runner_test.go` |
|
||||||
|
| 03-api | `internal/api` | `api/batch_test.go` |
|
||||||
|
| 04-metrics | `internal/metrics` | `metrics/batch_test.go` |
|
||||||
|
| 05-cli-docs | `cmd/sift`, `README.md`, `skill/SKILL.md`, config | `cmd/sift/batch_test.go` |
|
||||||
|
|
||||||
|
**How this plan was made:** design worked through with the owner from `docs/PLAN.md` §13 (future
|
||||||
|
work) on top of the P5 pool (`internal/pdfpool`), cache (`internal/dcache`), and gate
|
||||||
|
(`global`/`per` in `internal/api`) infrastructure. Acceptance tests first: the given tests are
|
||||||
|
authored against this spec and compiled against a panic-only skeleton so they fail for the
|
||||||
|
intended reasons before each task is implemented.
|
||||||
@@ -0,0 +1,254 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"net/netip"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.wntrmute.dev/kyle/sift/internal/auth"
|
||||||
|
"git.wntrmute.dev/kyle/sift/internal/batch"
|
||||||
|
"git.wntrmute.dev/kyle/sift/internal/dcache"
|
||||||
|
"git.wntrmute.dev/kyle/sift/internal/fetch"
|
||||||
|
"git.wntrmute.dev/kyle/sift/internal/guard"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newBatchServer builds a Server with the batch runner wired over an in-memory
|
||||||
|
// dcache, exactly as serve does in production. callers are the caller names to
|
||||||
|
// provision (each with a bearer key).
|
||||||
|
func newBatchServer(t *testing.T, g *guard.Guard, callers ...string) (*Server, map[string]string) {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
keysPath := filepath.Join(dir, "keys.toml")
|
||||||
|
keys := map[string]string{}
|
||||||
|
for _, name := range callers {
|
||||||
|
out := filepath.Join(dir, name)
|
||||||
|
if err := auth.Add(keysPath, name, "hermes-fleet", out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
b, err := os.ReadFile(out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
keys[name] = strings.TrimSpace(string(b))
|
||||||
|
}
|
||||||
|
store, err := auth.Load(keysPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if g == nil {
|
||||||
|
g = &guard.Guard{}
|
||||||
|
}
|
||||||
|
f := &fetch.Fetcher{Guard: g, UA: "sift-test"}
|
||||||
|
s := New("test", g, f, store, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
|
db, err := dcache.Open("", 1<<20, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
s.Batch = batch.New(db, s.fetchTarget, batch.Config{}, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
|
return s, keys
|
||||||
|
}
|
||||||
|
|
||||||
|
func submitBatch(t *testing.T, s *Server, key string, targets []map[string]string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
return postJSON(t, s, "/v1/batch", key, map[string]any{"targets": targets})
|
||||||
|
}
|
||||||
|
|
||||||
|
func getBatch(t *testing.T, s *Server, key, id string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/v1/batch/"+id, nil)
|
||||||
|
req.Header.Set("Authorization", "Bearer "+key)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(rec, req)
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
func batchStatus(t *testing.T, s *Server, key, id string) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
rec := getBatch(t, s, key, id)
|
||||||
|
var env map[string]any
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &env); err != nil {
|
||||||
|
t.Fatalf("status decode: %v (%s)", err, rec.Body.String())
|
||||||
|
}
|
||||||
|
return env
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitBatchTerminal polls until the job reaches a terminal status.
|
||||||
|
func waitBatchTerminal(t *testing.T, s *Server, key, id string) map[string]any {
|
||||||
|
t.Helper()
|
||||||
|
deadline := time.Now().Add(3 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
env := batchStatus(t, s, key, id)
|
||||||
|
if st, _ := env["status"].(string); st == "done" || st == "partial" || st == "failed" || st == "cancelled" {
|
||||||
|
return env
|
||||||
|
}
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatal("batch never reached a terminal status")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchSubmitAndCollect(t *testing.T) {
|
||||||
|
g := &guard.Guard{AllowPrefixes: []netip.Prefix{netip.MustParsePrefix("127.0.0.0/8")}}
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||||
|
_, _ = w.Write([]byte(`<html><head><title>T</title></head><body><article><p>hello batch</p></article></body></html>`))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
s, keys := newBatchServer(t, g, "test")
|
||||||
|
rec := submitBatch(t, s, keys["test"], []map[string]string{{"url": ts.URL}, {"url": ts.URL + "/2"}})
|
||||||
|
if rec.Code != 202 {
|
||||||
|
t.Fatalf("submit %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
var sub map[string]any
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &sub); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if sub["success"] != true {
|
||||||
|
t.Fatalf("success %v", sub["success"])
|
||||||
|
}
|
||||||
|
id, _ := sub["job_id"].(string)
|
||||||
|
if id == "" {
|
||||||
|
t.Fatal("missing job_id")
|
||||||
|
}
|
||||||
|
if reused, _ := sub["reused"].(bool); reused {
|
||||||
|
t.Fatal("first submit must not be reused")
|
||||||
|
}
|
||||||
|
env := waitBatchTerminal(t, s, keys["test"], id)
|
||||||
|
if env["status"] != "done" {
|
||||||
|
t.Fatalf("status %v", env["status"])
|
||||||
|
}
|
||||||
|
targets, _ := env["targets"].([]any)
|
||||||
|
if len(targets) != 2 {
|
||||||
|
t.Fatalf("targets %d", len(targets))
|
||||||
|
}
|
||||||
|
first := targets[0].(map[string]any)
|
||||||
|
if first["status"] != "done" {
|
||||||
|
t.Fatalf("target0 %v", first["status"])
|
||||||
|
}
|
||||||
|
res, _ := first["result"].(map[string]any)
|
||||||
|
if res == nil || !strings.Contains(res["content"].(string), "hello batch") {
|
||||||
|
t.Fatalf("target0 result %v", res)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchIdempotentResubmit(t *testing.T) {
|
||||||
|
g := &guard.Guard{AllowPrefixes: []netip.Prefix{netip.MustParsePrefix("127.0.0.0/8")}}
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
_, _ = w.Write([]byte("ok"))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
s, keys := newBatchServer(t, g, "test")
|
||||||
|
targets := []map[string]string{{"url": ts.URL}}
|
||||||
|
first := submitBatch(t, s, keys["test"], targets)
|
||||||
|
second := submitBatch(t, s, keys["test"], targets)
|
||||||
|
var a, b map[string]any
|
||||||
|
_ = json.Unmarshal(first.Body.Bytes(), &a)
|
||||||
|
_ = json.Unmarshal(second.Body.Bytes(), &b)
|
||||||
|
if a["job_id"] != b["job_id"] {
|
||||||
|
t.Fatalf("ids differ: %v vs %v", a["job_id"], b["job_id"])
|
||||||
|
}
|
||||||
|
if reused, _ := b["reused"].(bool); !reused {
|
||||||
|
t.Fatal("resubmit must be reused=true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchGuardRefusedUpFront(t *testing.T) {
|
||||||
|
s, keys := newBatchServer(t, nil, "test")
|
||||||
|
rec := submitBatch(t, s, keys["test"], []map[string]string{{"url": "http://example.com:8080/"}})
|
||||||
|
if rec.Code != 422 {
|
||||||
|
t.Fatalf("want 422, got %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), "blocked") {
|
||||||
|
t.Fatalf("body %s", rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchTooManyTargets(t *testing.T) {
|
||||||
|
s, keys := newBatchServer(t, nil, "test")
|
||||||
|
var targets []map[string]string
|
||||||
|
for i := 0; i < 33; i++ {
|
||||||
|
targets = append(targets, map[string]string{"url": "https://example.com/"})
|
||||||
|
}
|
||||||
|
rec := submitBatch(t, s, keys["test"], targets)
|
||||||
|
if rec.Code != 422 {
|
||||||
|
t.Fatalf("want 422, got %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchPartialFailure(t *testing.T) {
|
||||||
|
g := &guard.Guard{AllowPrefixes: []netip.Prefix{netip.MustParsePrefix("127.0.0.0/8")}}
|
||||||
|
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/plain")
|
||||||
|
_, _ = w.Write([]byte("ok"))
|
||||||
|
}))
|
||||||
|
defer ts.Close()
|
||||||
|
s, keys := newBatchServer(t, g, "test")
|
||||||
|
// target 1 is a private address: passes the submit pre-check (valid URL) but
|
||||||
|
// is blocked by the connect-time guard, surfacing as a per-target failure.
|
||||||
|
rec := submitBatch(t, s, keys["test"], []map[string]string{{"url": ts.URL}, {"url": "http://10.0.0.5/"}})
|
||||||
|
if rec.Code != 202 {
|
||||||
|
t.Fatalf("submit %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
var sub map[string]any
|
||||||
|
_ = json.Unmarshal(rec.Body.Bytes(), &sub)
|
||||||
|
id := sub["job_id"].(string)
|
||||||
|
env := waitBatchTerminal(t, s, keys["test"], id)
|
||||||
|
if env["status"] != "partial" {
|
||||||
|
t.Fatalf("status %v", env["status"])
|
||||||
|
}
|
||||||
|
targets := env["targets"].([]any)
|
||||||
|
second := targets[1].(map[string]any)
|
||||||
|
if second["status"] != "failed" {
|
||||||
|
t.Fatalf("target1 status %v", second["status"])
|
||||||
|
}
|
||||||
|
if err, _ := second["error"].(string); !strings.Contains(err, "blocked") {
|
||||||
|
t.Fatalf("target1 error %q", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchOwnership404(t *testing.T) {
|
||||||
|
s, keys := newBatchServer(t, nil, "test", "other")
|
||||||
|
rec := submitBatch(t, s, keys["test"], []map[string]string{{"url": "https://example.com/"}})
|
||||||
|
if rec.Code != 202 {
|
||||||
|
t.Fatalf("submit %d", rec.Code)
|
||||||
|
}
|
||||||
|
var sub map[string]any
|
||||||
|
_ = json.Unmarshal(rec.Body.Bytes(), &sub)
|
||||||
|
id := sub["job_id"].(string)
|
||||||
|
if got := getBatch(t, s, keys["other"], id); got.Code != 404 {
|
||||||
|
t.Fatalf("other caller GET: %d (must not reveal the job)", got.Code)
|
||||||
|
}
|
||||||
|
if got := getBatch(t, s, keys["test"], id); got.Code != 200 {
|
||||||
|
t.Fatalf("owner GET: %d", got.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchCancel(t *testing.T) {
|
||||||
|
g := &guard.Guard{AllowPrefixes: []netip.Prefix{netip.MustParsePrefix("127.0.0.0/8")}}
|
||||||
|
s, keys := newBatchServer(t, g, "test")
|
||||||
|
rec := submitBatch(t, s, keys["test"], []map[string]string{{"url": "https://example.com/"}})
|
||||||
|
var sub map[string]any
|
||||||
|
_ = json.Unmarshal(rec.Body.Bytes(), &sub)
|
||||||
|
id := sub["job_id"].(string)
|
||||||
|
req := httptest.NewRequest(http.MethodDelete, "/v1/batch/"+id, nil)
|
||||||
|
req.Header.Set("Authorization", "Bearer "+keys["test"])
|
||||||
|
drec := httptest.NewRecorder()
|
||||||
|
s.ServeHTTP(drec, req)
|
||||||
|
if drec.Code != 200 {
|
||||||
|
t.Fatalf("cancel %d %s", drec.Code, drec.Body.String())
|
||||||
|
}
|
||||||
|
env := waitBatchTerminal(t, s, keys["test"], id)
|
||||||
|
if env["status"] != "cancelled" {
|
||||||
|
t.Fatalf("status %v", env["status"])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,234 @@
|
|||||||
|
package batch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.wntrmute.dev/kyle/sift/internal/dcache"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newRunner(t *testing.T, fetch FetchFunc, cfg Config) *Runner {
|
||||||
|
t.Helper()
|
||||||
|
db, err := dcache.Open("", 1<<20, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
return New(db, fetch, cfg, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitStatus(t *testing.T, r *Runner, id, want string) {
|
||||||
|
t.Helper()
|
||||||
|
deadline := time.Now().Add(3 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
j, _, ok := r.Status(id)
|
||||||
|
if ok && j.Status == want {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
}
|
||||||
|
j, _, _ := r.Status(id)
|
||||||
|
t.Fatalf("job %s never reached %q (now %q)", id, want, j.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSerialOrder(t *testing.T) {
|
||||||
|
var mu sync.Mutex
|
||||||
|
var calls []string
|
||||||
|
inflight, peak := 0, 0
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) {
|
||||||
|
mu.Lock()
|
||||||
|
inflight++
|
||||||
|
if inflight > peak {
|
||||||
|
peak = inflight
|
||||||
|
}
|
||||||
|
calls = append(calls, tgt.URL)
|
||||||
|
mu.Unlock()
|
||||||
|
time.Sleep(time.Millisecond)
|
||||||
|
mu.Lock()
|
||||||
|
inflight--
|
||||||
|
mu.Unlock()
|
||||||
|
return Result{Content: "ok " + tgt.URL}, nil
|
||||||
|
}
|
||||||
|
r := newRunner(t, fetch, Config{})
|
||||||
|
id, reused, err := r.Submit(context.Background(), "alice", "fleet", []Target{
|
||||||
|
{URL: "https://a.example/"}, {URL: "https://b.example/"}, {URL: "https://c.example/"},
|
||||||
|
})
|
||||||
|
if err != nil || reused {
|
||||||
|
t.Fatalf("submit err=%v reused=%v", err, reused)
|
||||||
|
}
|
||||||
|
waitStatus(t, r, id, "done")
|
||||||
|
mu.Lock()
|
||||||
|
order := append([]string(nil), calls...)
|
||||||
|
p := peak
|
||||||
|
mu.Unlock()
|
||||||
|
if len(order) != 3 || order[0] != "https://a.example/" || order[1] != "https://b.example/" || order[2] != "https://c.example/" {
|
||||||
|
t.Fatalf("order %v", order)
|
||||||
|
}
|
||||||
|
if p != 1 {
|
||||||
|
t.Fatalf("peak inflight %d, want 1 (serial)", p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailureIsolation(t *testing.T) {
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) {
|
||||||
|
if tgt.URL == "https://b.example/" {
|
||||||
|
return Result{}, errors.New("upstream 500")
|
||||||
|
}
|
||||||
|
return Result{Content: "ok " + tgt.URL}, nil
|
||||||
|
}
|
||||||
|
r := newRunner(t, fetch, Config{})
|
||||||
|
id, _, err := r.Submit(context.Background(), "alice", "fleet", []Target{
|
||||||
|
{URL: "https://a.example/"}, {URL: "https://b.example/"}, {URL: "https://c.example/"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
waitStatus(t, r, id, "partial")
|
||||||
|
j, ts, ok := r.Status(id)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("no job")
|
||||||
|
}
|
||||||
|
if j.Status != "partial" || j.TargetsDone != 3 {
|
||||||
|
t.Fatalf("job %+v", j)
|
||||||
|
}
|
||||||
|
if ts[0].Status != "done" || ts[1].Status != "failed" || ts[2].Status != "done" {
|
||||||
|
t.Fatalf("statuses %+v", ts)
|
||||||
|
}
|
||||||
|
if ts[1].Error == "" {
|
||||||
|
t.Fatal("failed target missing error")
|
||||||
|
}
|
||||||
|
if ts[0].Result == nil || ts[0].Result.Content != "ok https://a.example/" {
|
||||||
|
t.Fatalf("result %+v", ts[0].Result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIdempotentResubmit(t *testing.T) {
|
||||||
|
var mu sync.Mutex
|
||||||
|
n := 0
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) {
|
||||||
|
mu.Lock()
|
||||||
|
n++
|
||||||
|
mu.Unlock()
|
||||||
|
return Result{Content: "ok"}, nil
|
||||||
|
}
|
||||||
|
r := newRunner(t, fetch, Config{})
|
||||||
|
targets := []Target{{URL: "https://a.example/"}, {URL: "https://b.example/"}}
|
||||||
|
id1, reused1, err := r.Submit(context.Background(), "alice", "fleet", targets)
|
||||||
|
if err != nil || reused1 {
|
||||||
|
t.Fatalf("first submit err=%v reused=%v", err, reused1)
|
||||||
|
}
|
||||||
|
id2, reused2, err := r.Submit(context.Background(), "alice", "fleet", targets)
|
||||||
|
if err != nil || !reused2 || id2 != id1 {
|
||||||
|
t.Fatalf("resubmit id=%q reused=%v err=%v", id2, reused2, err)
|
||||||
|
}
|
||||||
|
waitStatus(t, r, id1, "done")
|
||||||
|
mu.Lock()
|
||||||
|
calls := n
|
||||||
|
mu.Unlock()
|
||||||
|
if calls != 2 {
|
||||||
|
t.Fatalf("fetched %d times, want 2 (idempotent)", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTooManyTargets(t *testing.T) {
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) { return Result{Content: "ok"}, nil }
|
||||||
|
r := newRunner(t, fetch, Config{TargetsPerJob: 2})
|
||||||
|
three := []Target{{URL: "https://a.example/"}, {URL: "https://b.example/"}, {URL: "https://c.example/"}}
|
||||||
|
if _, _, err := r.Submit(context.Background(), "alice", "fleet", three); !errors.Is(err, ErrTooManyTargets) {
|
||||||
|
t.Fatalf("want ErrTooManyTargets, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCallerBusy(t *testing.T) {
|
||||||
|
release := make(chan struct{})
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) {
|
||||||
|
<-release // hold the job in the running state
|
||||||
|
return Result{Content: "ok"}, nil
|
||||||
|
}
|
||||||
|
r := newRunner(t, fetch, Config{})
|
||||||
|
id, _, err := r.Submit(context.Background(), "alice", "fleet", []Target{{URL: "https://a.example/"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
waitStatus(t, r, id, "running")
|
||||||
|
if _, _, err := r.Submit(context.Background(), "alice", "fleet", []Target{{URL: "https://b.example/"}}); !errors.Is(err, ErrCallerBusy) {
|
||||||
|
t.Fatalf("want ErrCallerBusy, got %v", err)
|
||||||
|
}
|
||||||
|
close(release)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCancel(t *testing.T) {
|
||||||
|
started := make(chan struct{}, 1)
|
||||||
|
release := make(chan struct{})
|
||||||
|
var mu sync.Mutex
|
||||||
|
var calls []string
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) {
|
||||||
|
mu.Lock()
|
||||||
|
calls = append(calls, tgt.URL)
|
||||||
|
mu.Unlock()
|
||||||
|
select {
|
||||||
|
case started <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
<-release
|
||||||
|
return Result{Content: "ok"}, nil
|
||||||
|
}
|
||||||
|
r := newRunner(t, fetch, Config{})
|
||||||
|
id, _, err := r.Submit(context.Background(), "alice", "fleet", []Target{
|
||||||
|
{URL: "https://a.example/"}, {URL: "https://b.example/"}, {URL: "https://c.example/"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
<-started // first target is in flight
|
||||||
|
if !r.Cancel(id) {
|
||||||
|
t.Fatal("cancel returned false")
|
||||||
|
}
|
||||||
|
close(release) // let the in-flight target finish
|
||||||
|
waitStatus(t, r, id, "cancelled")
|
||||||
|
mu.Lock()
|
||||||
|
n := len(calls)
|
||||||
|
mu.Unlock()
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("cancelled after %d fetches, want 1", n)
|
||||||
|
}
|
||||||
|
_, ts, ok := r.Status(id)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("no job")
|
||||||
|
}
|
||||||
|
if ts[0].Status != "done" {
|
||||||
|
t.Fatalf("in-flight target should be stored done: %+v", ts[0])
|
||||||
|
}
|
||||||
|
if ts[1].Status != "pending" || ts[2].Status != "pending" {
|
||||||
|
t.Fatalf("later targets must stay pending: %+v", ts[1:])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPanicRecovered(t *testing.T) {
|
||||||
|
fetch := func(_ context.Context, tgt Target) (Result, error) {
|
||||||
|
if tgt.URL == "https://b.example/" {
|
||||||
|
panic("boom")
|
||||||
|
}
|
||||||
|
return Result{Content: "ok"}, nil
|
||||||
|
}
|
||||||
|
r := newRunner(t, fetch, Config{})
|
||||||
|
id, _, err := r.Submit(context.Background(), "alice", "fleet", []Target{
|
||||||
|
{URL: "https://a.example/"}, {URL: "https://b.example/"}, {URL: "https://c.example/"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
waitStatus(t, r, id, "partial")
|
||||||
|
_, ts, _ := r.Status(id)
|
||||||
|
if ts[0].Status != "done" || ts[1].Status != "failed" || ts[2].Status != "done" {
|
||||||
|
t.Fatalf("statuses %+v", ts)
|
||||||
|
}
|
||||||
|
if ts[1].Error == "" {
|
||||||
|
t.Fatal("panic target missing error")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestParseBatchArgs(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
args []string
|
||||||
|
verb string
|
||||||
|
arg string
|
||||||
|
ok bool
|
||||||
|
}{
|
||||||
|
{[]string{"submit", "targets.json"}, "submit", "targets.json", true},
|
||||||
|
{[]string{"submit", "-"}, "submit", "-", true},
|
||||||
|
{[]string{"status", "job-123"}, "status", "job-123", true},
|
||||||
|
{[]string{"cancel", "job-123"}, "cancel", "job-123", true},
|
||||||
|
{[]string{}, "", "", false},
|
||||||
|
{[]string{"status"}, "", "", false},
|
||||||
|
{[]string{"status", "a", "b"}, "", "", false},
|
||||||
|
{[]string{"bogus", "x"}, "", "", false},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
verb, arg, err := parseBatchArgs(c.args)
|
||||||
|
if c.ok {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("%v: unexpected error %v", c.args, err)
|
||||||
|
}
|
||||||
|
if verb != c.verb || arg != c.arg {
|
||||||
|
t.Fatalf("%v: got verb=%q arg=%q", c.args, verb, arg)
|
||||||
|
}
|
||||||
|
} else if err == nil {
|
||||||
|
t.Fatalf("%v: want error", c.args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,240 @@
|
|||||||
|
package dcache
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newBatchDB(t *testing.T) *DB {
|
||||||
|
t.Helper()
|
||||||
|
db, err := Open("", 1<<20, slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
|
return db
|
||||||
|
}
|
||||||
|
|
||||||
|
func strp(s string) *string { return &s }
|
||||||
|
|
||||||
|
func TestBatchJobRoundTrip(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
j := BatchJob{
|
||||||
|
ID: "j1", Caller: "alice", CallerClass: ClassFleet, IDemKey: "k1",
|
||||||
|
Status: "queued", TargetsTotal: 3, CreatedAt: 100, Lease: 100,
|
||||||
|
}
|
||||||
|
if err := db.CreateBatchJob(j); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got, ok := db.BatchJob("j1")
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("missing job")
|
||||||
|
}
|
||||||
|
if got.ID != j.ID || got.Caller != "alice" || got.CallerClass != ClassFleet ||
|
||||||
|
got.Status != "queued" || got.TargetsTotal != 3 || got.Lease != 100 || got.CreatedAt != 100 {
|
||||||
|
t.Fatalf("round trip mismatch: %+v", got)
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchJob("nope"); ok {
|
||||||
|
t.Fatal("unknown id must be ok=false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchIDemLookup(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: "j1", Caller: "a", CallerClass: ClassFleet, IDemKey: "k", Status: "running", TargetsTotal: 1, CreatedAt: 100, Lease: 1000}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if id, ok := db.BatchJobByIDem("a", "k", 2000, time.Hour); !ok || id != "j1" {
|
||||||
|
t.Fatalf("live idem: %q %v", id, ok)
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchJobByIDem("a", "k", 5000, time.Hour); ok {
|
||||||
|
t.Fatal("lease-expired job must not match")
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchJobByIDem("b", "k", 2000, time.Hour); ok {
|
||||||
|
t.Fatal("other caller must not match")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchLiveCounts(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
mk := func(id, caller, status string) {
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: id, Caller: caller, CallerClass: ClassFleet, IDemKey: id, Status: status, TargetsTotal: 1, CreatedAt: 1, Lease: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mk("j1", "a", "queued")
|
||||||
|
mk("j2", "a", "running")
|
||||||
|
mk("j3", "a", "done")
|
||||||
|
mk("j4", "b", "running")
|
||||||
|
if n := db.LiveBatchJobs("a"); n != 2 {
|
||||||
|
t.Fatalf("caller a live=%d want 2", n)
|
||||||
|
}
|
||||||
|
if n := db.LiveBatchJobsTotal(); n != 3 {
|
||||||
|
t.Fatalf("total live=%d want 3", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchTargetsOrderedAndUpdate(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
must := func(err error) {
|
||||||
|
t.Helper()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
must(db.CreateBatchJob(BatchJob{ID: "j1", Caller: "a", CallerClass: ClassFleet, IDemKey: "k", Status: "running", TargetsTotal: 3, CreatedAt: 100, Lease: 100}))
|
||||||
|
must(db.PutBatchTargets("j1", []BatchTarget{
|
||||||
|
{JobID: "j1", Idx: 0, URL: "https://a.example/", Options: `{"mode":"clean"}`, Status: "pending"},
|
||||||
|
{JobID: "j1", Idx: 1, URL: "https://b.example/", Status: "pending"},
|
||||||
|
{JobID: "j1", Idx: 2, URL: "https://c.example/", Status: "pending"},
|
||||||
|
}))
|
||||||
|
got, ok := db.BatchTargets("j1")
|
||||||
|
if !ok || len(got) != 3 {
|
||||||
|
t.Fatalf("targets len=%d ok=%v", len(got), ok)
|
||||||
|
}
|
||||||
|
for i := range got {
|
||||||
|
if got[i].Idx != i {
|
||||||
|
t.Fatalf("order broken: got[%d].Idx=%d", i, got[i].Idx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A done write bumps the lease to now and counts the target once.
|
||||||
|
res := strp(`{"content":"ok"}`)
|
||||||
|
must(db.UpdateBatchTarget("j1", 0, 200, "done", res, nil, nil))
|
||||||
|
j, _ := db.BatchJob("j1")
|
||||||
|
if j.Lease != 200 || j.TargetsDone != 1 {
|
||||||
|
t.Fatalf("after first write lease=%d done=%d", j.Lease, j.TargetsDone)
|
||||||
|
}
|
||||||
|
// Re-writing the same target (still a done transition) must not double-count.
|
||||||
|
must(db.UpdateBatchTarget("j1", 0, 201, "done", res, nil, nil))
|
||||||
|
j, _ = db.BatchJob("j1")
|
||||||
|
if j.TargetsDone != 1 {
|
||||||
|
t.Fatalf("double-counted: done=%d", j.TargetsDone)
|
||||||
|
}
|
||||||
|
// A failed write counts too.
|
||||||
|
must(db.UpdateBatchTarget("j1", 1, 202, "failed", nil, strp("upstream 500"), strp("browser")))
|
||||||
|
j, _ = db.BatchJob("j1")
|
||||||
|
if j.TargetsDone != 2 || j.Lease != 202 {
|
||||||
|
t.Fatalf("after failed write lease=%d done=%d", j.Lease, j.TargetsDone)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBatchTouchAndStatus(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: "j1", Caller: "a", CallerClass: ClassFleet, IDemKey: "k", Status: "running", TargetsTotal: 1, CreatedAt: 100, Lease: 100}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.TouchBatchJob("j1", 500); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if j, _ := db.BatchJob("j1"); j.Lease != 500 {
|
||||||
|
t.Fatalf("touch lease=%d want 500", j.Lease)
|
||||||
|
}
|
||||||
|
if err := db.SetBatchJobStatus("j1", "cancelled", "", 600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if j, _ := db.BatchJob("j1"); j.Status != "cancelled" || j.Lease != 600 {
|
||||||
|
t.Fatalf("status=%q lease=%d", j.Status, j.Lease)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSweepBatch(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
mk := func(id string, lease int64) {
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: id, Caller: "a", CallerClass: ClassFleet, IDemKey: id, Status: "done", TargetsTotal: 1, CreatedAt: 1, Lease: lease}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.PutBatchTargets(id, []BatchTarget{{JobID: id, Idx: 0, URL: "https://x.example/", Status: "done", Result: strp("x")}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mk("old", 100)
|
||||||
|
mk("new", 1000)
|
||||||
|
n := db.SweepBatch(300, 100*time.Second) // old: 100+100=200<=300 expired; new: 1100>300 kept
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("swept %d want 1", n)
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchJob("old"); ok {
|
||||||
|
t.Fatal("old job must be gone")
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchJob("new"); !ok {
|
||||||
|
t.Fatal("new job must remain")
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchTargets("old"); ok {
|
||||||
|
t.Fatal("old targets must be gone")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEvictBatchOldest(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
done := func(id string, lease int64, size int) {
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: id, Caller: "a", CallerClass: ClassFleet, IDemKey: id, Status: "done", TargetsTotal: 1, CreatedAt: 1, Lease: lease}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.PutBatchTargets(id, []BatchTarget{{JobID: id, Idx: 0, URL: "https://x.example/", Status: "pending"}}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.UpdateBatchTarget(id, 0, lease, "done", strp(strings.Repeat("x", size)), nil, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
done("old1", 100, 100)
|
||||||
|
done("old2", 200, 200)
|
||||||
|
done("new", 300, 300)
|
||||||
|
|
||||||
|
// A running job with one completed target must never be byte-evicted.
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: "run", Caller: "a", CallerClass: ClassFleet, IDemKey: "run", Status: "running", TargetsTotal: 2, TargetsDone: 1, CreatedAt: 1, Lease: 400}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.PutBatchTargets("run", []BatchTarget{
|
||||||
|
{JobID: "run", Idx: 0, URL: "https://x.example/", Status: "pending"},
|
||||||
|
{JobID: "run", Idx: 1, URL: "https://y.example/", Status: "pending"},
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := db.UpdateBatchTarget("run", 0, 400, "done", strp(strings.Repeat("x", 500)), nil, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// total nbytes = 100+200+300+500 = 1100. Evict oldest done until <= 250:
|
||||||
|
// old1(100), old2(200), new(300) all evicted (oldest first); run stays (live).
|
||||||
|
db.EvictBatchOldest(250)
|
||||||
|
for _, id := range []string{"old1", "old2", "new"} {
|
||||||
|
if _, ok := db.BatchJob(id); ok {
|
||||||
|
t.Fatalf("%s should be byte-evicted", id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, ok := db.BatchJob("run"); !ok {
|
||||||
|
t.Fatal("running job must not be byte-evicted")
|
||||||
|
}
|
||||||
|
if n := db.BatchNbytes(); n != 500 {
|
||||||
|
t.Fatalf("BatchNbytes=%d want 500", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMarkInFlightFailed(t *testing.T) {
|
||||||
|
db := newBatchDB(t)
|
||||||
|
mk := func(id, status string) {
|
||||||
|
if err := db.CreateBatchJob(BatchJob{ID: id, Caller: "a", CallerClass: ClassFleet, IDemKey: id, Status: status, TargetsTotal: 1, CreatedAt: 1, Lease: 1}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mk("q", "queued")
|
||||||
|
mk("r", "running")
|
||||||
|
mk("d", "done")
|
||||||
|
if n := db.MarkInFlightFailed("restart", 999); n != 2 {
|
||||||
|
t.Fatalf("marked %d want 2", n)
|
||||||
|
}
|
||||||
|
if j, _ := db.BatchJob("q"); j.Status != "failed" || j.Error != "restart" || j.Lease != 999 {
|
||||||
|
t.Fatalf("q %+v", j)
|
||||||
|
}
|
||||||
|
if j, _ := db.BatchJob("r"); j.Status != "failed" || j.Error != "restart" {
|
||||||
|
t.Fatalf("r %+v", j)
|
||||||
|
}
|
||||||
|
if j, _ := db.BatchJob("d"); j.Status != "done" {
|
||||||
|
t.Fatalf("done job changed: %+v", j)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
package metrics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type batchSrc struct{}
|
||||||
|
|
||||||
|
func (batchSrc) Snapshot() Snapshot { return Snapshot{} }
|
||||||
|
|
||||||
|
func TestBatchSeries(t *testing.T) {
|
||||||
|
s := New(batchSrc{})
|
||||||
|
s.ObserveBatchJob("created")
|
||||||
|
s.ObserveBatchJob("done")
|
||||||
|
s.ObserveBatchJob("done")
|
||||||
|
s.ObserveBatchJob("reused")
|
||||||
|
s.ObserveBatchJob("some raw string") // folded to "other", never a label
|
||||||
|
s.ObserveBatchTarget("ok")
|
||||||
|
s.ObserveBatchTarget("ok")
|
||||||
|
s.ObserveBatchTarget("blocked")
|
||||||
|
s.ObserveBatchTarget("upstream_5xx")
|
||||||
|
s.ObserveBatchTarget("leak me") // folded to "other"
|
||||||
|
s.SetBatchQueueDepth(3)
|
||||||
|
s.SetBatchTargetsInflight(1)
|
||||||
|
s.ObserveBatchJobSeconds(4 * time.Second)
|
||||||
|
s.ObserveBatchTargetSeconds(1500 * time.Millisecond)
|
||||||
|
s.ObserveBatchResultBytes(4096)
|
||||||
|
|
||||||
|
req := httptest.NewRequest("GET", "/metrics", nil)
|
||||||
|
req.RemoteAddr, req.Host = "127.0.0.1:9", "127.0.0.1:8890"
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
s.Handler().ServeHTTP(rec, req)
|
||||||
|
body, _ := io.ReadAll(rec.Body)
|
||||||
|
out := string(body)
|
||||||
|
for _, want := range []string{
|
||||||
|
`sift_batch_jobs_total{outcome="created"} 1`,
|
||||||
|
`sift_batch_jobs_total{outcome="done"} 2`,
|
||||||
|
`sift_batch_jobs_total{outcome="reused"} 1`,
|
||||||
|
`sift_batch_jobs_total{outcome="other"} 1`,
|
||||||
|
`sift_batch_targets_total{outcome="ok"} 2`,
|
||||||
|
`sift_batch_targets_total{outcome="blocked"} 1`,
|
||||||
|
`sift_batch_targets_total{outcome="upstream_5xx"} 1`,
|
||||||
|
`sift_batch_targets_total{outcome="other"} 1`,
|
||||||
|
`sift_batch_queue_depth 3`,
|
||||||
|
`sift_batch_targets_inflight 1`,
|
||||||
|
`sift_batch_job_seconds_count 1`,
|
||||||
|
`sift_batch_target_seconds_count 1`,
|
||||||
|
`sift_batch_result_bytes_count 1`,
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Errorf("metrics lack %q", want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "raw") || strings.Contains(out, "leak") {
|
||||||
|
t.Error("a raw outcome string leaked into a label")
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user