sift/docs/P6.md

Sift P6 — asynchronous batch fetch

Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer: Claude (when it frees up this evening). Implementer: Kyle's call.

1. Scope

In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the same per-target options as /v1/fetch) and gets a job ID back immediately; it polls for per-target results later. One job drains serially — one target at a time — in the background.

Out:

• Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are already expected to take a long time, not for interactive fetches.
• Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
• Firecrawl compatibility. This is a native-only feature; no Hermes plugin.

The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a page, through the same SSRF guard and the same per-caller/host gates.

2. Why agents need this (2026-09-26)

docs/PLAN.md §13 (Kyle): "an agent that has collected a series of targets and wants them fetched without holding a request open per target." Today that means a loop of sift fetch calls — one request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the dcache SQLite store, the global/per concurrency gates, and the doFetch pipeline.

3. Endpoints and wire shapes

Method	Path	Result
POST /v1/batch	submit a list of targets	202 created/reused, 400 malformed, 422 refused up front or over a cap, 429 caller/jobs busy
GET /v1/batch/{id}	status summary; ?results=1 inlines results	200, 401/404
GET /v1/batch/{id}/{idx}	one target's status and result	200, 401/404
DELETE /v1/batch/{id}	cancel	200, 409 already finished, 401/404

With [batch] disabled = true every batch endpoint answers 503 batch disabled.

POST /v1/batch request — each target is the /v1/fetch field set minus the outer URL:

{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
              "jq": "…", "context": true, "recover": false}, …]}

Response 202:

{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}

GET /v1/batch/{id} (a summary: no results, since 32 inline results can be ~6 MB in one agent tool call):

{"job_id": "…", "status": "partial", "targets_total": 3, "targets_done": 3,
 "created_at": "2026-09-27T01:02:03Z", "targets": [
   {"idx": 0, "url": "…", "status": "done"},
   {"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
 ]}

?results=1 adds "result" (the /v1/fetch envelope) to every done target. GET /v1/batch/{id}/{idx} returns {"job_id", "idx", "url", "status", "result"?, "error"?, "hint"?}. A pending target has status: "pending" and no result/error. A 202 for a new job says "status": "queued"; a reused one says the job's current status. DELETE → 200 {"success": true, "status": "cancelled"}.

Job status vocabulary: queued → running → done | partial | failed | cancelled. Target status: pending → running → done | failed.

4. Storage (dcache)

Two tables, added to internal/dcache (which owns the single SQLite connection):

• batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done, created_at, lease, nbytes)
• batch_target(job_id, idx, url, options, status, result, error, hint) — result is the JSON envelope, NULL until the target finishes.

Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, not by the content cache's LRU. EvictBatchOldest evicts whole jobs oldest-lease-first; a queued/running job is never byte-evicted. MarkInFlightFailed(reason, now) is the restart hook.

5. The runner (internal/batch)

A new package with the job lifecycle and the serial state machine. Persistence is the *dcache.DB; the only injected behavior is a FetchFunc func(ctx, Target) (Result, error).

• Idempotency (in v1). A job has a random opaque id plus a derived idem_key = SHA256(caller + "\x00" + canonical target list). Resubmitting the same (caller, targets) while the job's lease has not expired returns the existing job (reused: true) instead of a new one. Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
• Caps. targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status (queued/running), not lease. Exceeding a cap is a typed error the API maps to 422/429.
• The caller. Submit takes the *auth.Caller; the runner keeps it in memory with the job and passes it to every FetchFunc call (func(ctx, *auth.Caller, Target)), so the fetch runs under the caller's gates, pool slot, cache class and redaction. Only its name and class are stored.
• Workers. workers (default 1, at most 2) share one round-robin over the live jobs, one target at a time. A job is queued until a worker first takes it, then running.
• Serial. Targets drain in idx order; target i+1 does not start until i is written.
• Busy. A FetchFunc error wrapping ErrRetryLater (the API maps fetch 429/503 to it) leaves the target pending; it is tried again after 2 s, for up to 10 min, then failed.
• Failure isolation. One target's error is stored per-target and the job continues; the job status is done (all ok), failed (all failed), or partial (mixed).
• Cancel. DELETE flips the job cancelled, stops before the next target, and lets the in-flight target finish and store. Completed targets stay readable until the lease lapses.

6. Scheduling and fairness

The workers run in the background, and every target still goes through the normal acquire gate (8 global, 2 per-host). With 1–2 workers the batch holds at most 1–2 of the 8 global slots, so interactive requests always have room. (The first draft let 8 live jobs each run a target at once, which could fill the whole gate.) On 429 busy the worker waits and retries instead of failing: nobody is holding a socket open.

Two shared resources get the same treatment:

• The PDF pool. A batch extraction uses the pool caller name <caller>/batch, so it never takes the caller's own interactive slot. The single-flight key is unchanged, so a batch and an interactive request for the same PDF still share one child.
• The digest gate. A batch question queues on it like any other digest. With one worker that is at most one batch digest at a time.

Each batch fetch runs under the same 210 s deadline as a /v1/fetch request.

7. Retention and TTL

A job carries a single sliding lease, bumped on submit, on every target completion, and on every read by its owner (the API calls Touch after the ownership check; the store's reads never write). All targets therefore share one expiry (the last completion's), and a slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease lapses; queued/running jobs are marked failed (restart) on startup and never resumed.

8. Restricted callers

Compartmentalized-access systems: their usage is redacted as much as possible. The existing logHost/logReason/logErr machinery already blanks host, reason and error detail for PermitRestricted callers, and the digest/embed caches partition by caller class — the batch inherits all of it by passing the caller through. Two additions: batch log lines never carry a target URL or hostname for any caller, and GET/DELETE enforce job ownership, returning 404 (not 403) for a job owned by another caller so a caller cannot probe for its existence. The idem_key is stored as a hash and never logged alongside its target list.

9. Caps and metrics (sized empirically)

The caps are starting values, instrumented so they can be re-sized from observed data:

• sift_batch_jobs_total{outcome} — created|reused|done|partial|failed|cancelled|restart
• sift_batch_targets_total{outcome} — ok|blocked|timeout|busy|upstream_4xx|upstream_5xx| unsupported|too_long|cancel
• sift_batch_queue_depth, sift_batch_targets_inflight (gauges)
• sift_batch_job_seconds, sift_batch_target_seconds, sift_batch_result_bytes (histograms)

Both outcome counters fold any value outside their known set to other, so a raw string never becomes a label. result_bytes is what sizes the byte budget (64 MiB).

10. CLI and skill

• sift batch submit <targets.json> (- reads stdin; prints job_id and reused).
• sift batch status [--results] <job_id> (the summary; --results inlines every result).
• sift batch get <job_id> <idx> (one result).
• sift batch cancel <job_id>.
• skill/SKILL.md: use sift batch submit for a list of targets instead of a loop of sift fetch; sift batch status to collect.
• [batch] config block: disabled, targets_per_job, jobs_per_caller, jobs_total, workers, ttl, max_bytes. The /v1/batch body cap (256 KiB, vs the 64 KiB global) is a constant, not config.

11. Refactor of existing code (explicit)

v1fetch today inlines the whole per-target pipeline (guard → doFetch → recover → render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is extracted into one shared method, func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target) (batch.Result, error), that both v1fetch and the runner's adapter (batchFetch) call. It is its own task (04), done before any batch endpoint exists, so the diff is a pure move. v1fetch keeps its exact response shape and behavior — its existing tests must pass unchanged. This is the one change to working code in P6 and is called out for reviewer attention.

12. Acceptance (contract) tests

Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the intended reasons. See docs/plans/p6/_files/ and the task table in docs/plans/p6/README.md.

1. dcache: round trip, idempotency lookup (lease expiry, caller scope, status filter), live counts, ordered targets, exactly-once targets_done and lease bumps, sweep, byte eviction (oldest first, never a live job), MarkInFlightFailed, BatchNbytes.
2. batch runner:
   • serial order (peak in flight 1), with the caller passed to every fetch;
   • round-robin between jobs, and queued → running;
   • failure isolation (partial, failed);
   • idempotent resubmit, including after done, but a new job after a cancel;
   • all three caps and ErrNoTargets; cancel (the in-flight target stored, later ones pending);
   • busy retry and its limit; panic recovery (the panic value not echoed); the hooks.
3. metrics: series names, labels, folding to other.
4. api fetchTarget: the same content and envelope as /v1/fetch; its errors as *fetch.Error.
5. api wiring: the runner through fetchTarget, a private-resolving name blocked per target, metrics through the hooks, outcome classes, busy → ErrRetryLater.
6. guard.PreCheck: no resolution; the port, scheme, userinfo and literal rules; the allow list.
7. api endpoints:
   • submit 202 and idempotent resubmit;
   • 400/422 refusals (the guard pre-check naming the index, 33 targets);
   • the 256 KiB body; the per-target blocked failure giving partial;
   • summary, one result, ?results=1;
   • ownership 404 (GET, one result, DELETE); cancel and 409;
   • no job id in /metrics.
8. config: the [batch] block, ParseSize. cmd/sift: parseBatchArgs.

13. Rollout

1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the v1fetch refactor, and the caps/lease semantics.
2. Orion pin bump with the [batch] config block; no MemoryMax change.
3. Skill update ships with the pin bump.
4. Watch sift_batch_* and tirith for a week; the success measure is that a research thread's one-request-per-URL fetch loop collapses into a single batch submit.

14. Changes after the plan was written

2026-09-26 evening, owner review (Claude), at Kyle's direction ("fold them into the plan"). The design stands; these were implementation-blocking or would have hurt agents:

1. The caller was lost. FetchFunc had no caller and the runner was built once, so the given api test (batch.New(db, s.fetchTarget, …)) could not compile against the specified fetchTarget(ctx, *auth.Caller, Target). Made to compile, it would have fetched every target with no caller: no per-caller gates, pool limit, cache class or redaction. Now Submit takes the *auth.Caller and FetchFunc receives it (§5).
2. Task 01 rule 1 contradicted its test. Store reads no longer bump the lease; the API calls Touch after an owner's read (§7).
3. Batch could starve interactive fetches. 8 live jobs × 1 target filled the 8-slot global gate. The busy-retry of §6 was in no task. Now there are 1–2 round-robin workers, busy-retry is in the runner, a batch PDF uses its own pool name, and a per-target deadline applies (§5, §6).
4. Flaky, networked API tests. Two tests fetched https://example.com/, and the cancel test raced the fetch. Every api test is now local, using held httptest servers and a fake DNS name for the private-address case.
5. A resubmit could return a dead job. A failed or cancelled job was reused for 24 h, and polling it kept extending that. Now only queued/running/done/partial jobs are reused.
6. Reads were too large. GET inlined every result (up to ~6 MB). It is now a summary, with GET …/{idx} and ?results=1 (§3). The CLI gains get and status --results.
7. Body cap. The global 64 KiB MaxBytesReader runs in ServeHTTP, so a handler cannot raise it. ServeHTTP now picks 256 KiB for /v1/batch. Request metrics use a fixed /v1/batch/{id} label, so a job id never becomes a label value.
8. Submit pre-check. guard.ParseURL rejects the test's own loopback port and knows nothing of the allow list. New Guard.PreCheck is CheckURL without resolution (an owner commit on master, since AGENTS.md keeps the implementer out of internal/guard/). A private IP literal is now refused at submit; a name that resolves privately still fails per target at fetch time.
9. Task shape. 5 tasks → 9 (the last split came during the run, below), one package each where possible: 01 dcache, 02 runner core, 03 runner workers, 04 metrics, 05 the fetchTarget refactor alone, 06 runner wiring, 07 endpoints, 08 serve/config, 09 CLI/docs. Every task restates the APIs it uses from earlier tasks. Every given test compiles against a panic-only skeleton and fails there on panic: todo.
10. Also fixed:
    • the ErrBusy (jobs total) cap now has a test;
    • the 202 status is queued, consistent with the workers;
    • the restart outcome is counted by serve;
    • Options.Context is *bool, keeping /v1/fetch's default.

Sequencing. P6 starts after P5c merges. P5b/P5c change v1fetch (the digest meta gains retrieval/embedded), which is the code task 05 moves. Recheck the api given tests against a skeleton built on post-P5c master before the run.

During the run (2026-09-26 21:08). Task 02 (the whole runner) ended with no file: the model wrote the implementation out in its reply, the session was compacted, and it then emitted a malformed tool call. The runner is now two tasks (02 core, 03 workers), each with its own given test. AGENTS.md gains a rule to write code into files early. Copy paths are spelled from the repo root. Details in docs/plans/p6/README.md, "Changes during the run".