sift/docs/P6.md

Sift P6 — asynchronous batch fetch

Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer: Claude (when it frees up this evening). Implementer: Kyle's call.

1. Scope

In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the same per-target options as /v1/fetch) and gets a job ID back immediately; it polls for per-target results later. One job drains serially — one target at a time — in the background.

Out:

• Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are already expected to take a long time, not for interactive fetches.
• Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
• Firecrawl compatibility. This is a native-only feature; no Hermes plugin.

The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a page, through the same SSRF guard and the same per-caller/host gates.

2. Why agents need this (2026-09-26)

docs/PLAN.md §13 (Kyle): "an agent that has collected a series of targets and wants them fetched without holding a request open per target." Today that means a loop of sift fetch calls — one request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the dcache SQLite store, the global/per concurrency gates, and the doFetch pipeline.

3. Endpoints and wire shapes

Method	Path	Result
POST /v1/batch	submit a list of targets	202 created/reused, 400 malformed, 422 refused up front or over a cap, 429 caller/jobs busy
GET /v1/batch/{id}	status + per-target results (inline)	200, 401/404
DELETE /v1/batch/{id}	cancel	200, 401/404

POST /v1/batch request — each target is the /v1/fetch field set minus the outer URL:

{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
              "jq": "…", "context": true, "recover": false}, …]}

Response 202:

{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}

GET /v1/batch/{id}:

{"job_id": "…", "status": "done", "targets_total": 3, "targets_done": 3,
 "created_at": "…", "targets": [
   {"idx": 0, "url": "…", "status": "done", "result": { …/v1/fetch envelope… }},
   {"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
 ]}

Results are inline regardless of job status, once a target is finished. A pending target has status: "pending" and no result/error. DELETE → 200 {"success": true, "status": "cancelled"}.

Job status vocabulary: queued → running → done | partial | failed | cancelled. Target status: pending → running → done | failed.

4. Storage (dcache)

Two tables, added to internal/dcache (which owns the single SQLite connection):

• batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done, created_at, lease, nbytes)
• batch_target(job_id, idx, url, options, status, result, error, hint) — result is the JSON envelope, NULL until the target finishes.

Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, not by the content cache's LRU. EvictBatchOldest evicts whole jobs oldest-lease-first; a queued/running job is never byte-evicted. MarkInFlightFailed(reason, now) is the restart hook.

5. The runner (internal/batch)

A new package with the job lifecycle and the serial state machine. Persistence is the *dcache.DB; the only injected behavior is a FetchFunc func(ctx, Target) (Result, error).

• Idempotency (in v1). A job has a random opaque id plus a derived idem_key = SHA256(caller + "\x00" + canonical target list). Resubmitting the same (caller, targets) while the job's lease has not expired returns the existing job (reused: true) instead of a new one. Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
• Caps. targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status (queued/running), not lease. Exceeding a cap is a typed error the API maps to 422/429.
• Serial. Targets drain in idx order; target i+1 does not start until i is written.
• Failure isolation. One target's error is stored per-target and the job continues; the job status is done (all ok), failed (all failed), or partial (mixed).
• Cancel. DELETE flips the job cancelled, stops before the next target, and lets the in-flight target finish and store. Completed targets stay readable until the lease lapses.

6. Scheduling and fairness

The worker runs in the background, and every target still goes through the normal acquire gate (8 global, 2 per-host). On 429 busy the worker waits and retries instead of failing — nobody is holding a socket open. Interactive requests are served by HTTP handlers that grab slots immediately, so they win by construction; the batch is lower priority for free.

7. Retention and TTL

A job carries a single sliding lease, bumped on submit, on every target completion, and on every status/collect read. All targets therefore share one expiry (the last completion's), and a slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease lapses; queued/running jobs are marked failed (restart) on startup and never resumed.

8. Restricted callers

Compartmentalized-access systems: their usage is redacted as much as possible. The existing logHost/logReason/logErr machinery already blanks host, reason and error detail for PermitRestricted callers, and the digest/embed caches partition by caller class — the batch inherits all of it by passing the caller through. Two additions: batch log lines never carry a target URL or hostname for any caller, and GET/DELETE enforce job ownership, returning 404 (not 403) for a job owned by another caller so a caller cannot probe for its existence. The idem_key is stored as a hash and never logged alongside its target list.

9. Caps and metrics (sized empirically)

The caps are starting values, instrumented so they can be re-sized from observed data:

• sift_batch_jobs_total{outcome} — created|reused|done|partial|failed|cancelled|restart
• sift_batch_targets_total{outcome} — ok|blocked|timeout|busy|upstream_4xx|upstream_5xx| unsupported|too_long|cancel
• sift_batch_queue_depth, sift_batch_targets_inflight (gauges)
• sift_batch_job_seconds, sift_batch_target_seconds, sift_batch_result_bytes (histograms)

Both outcome counters fold any value outside their known set to other, so a raw string never becomes a label. result_bytes is what sizes the byte budget (64 MiB).

10. CLI and skill

• sift batch submit <targets.json> (- reads stdin; prints job_id and reused).
• sift batch status <job_id> (prints the full status body).
• sift batch cancel <job_id>.
• skill/SKILL.md: use sift batch submit for a list of targets instead of a loop of sift fetch; sift batch status to collect.
• [batch] config block: targets_per_job, jobs_per_caller, jobs_total, ttl, max_bytes. The /v1/batch body cap (256 KiB, vs the 64 KiB global) is a constant, not config.

11. Refactor of existing code (explicit)

v1fetch today inlines the whole per-target pipeline (guard → doFetch → recover → render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is extracted into one shared method, func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target) (batch.Result, error), that both v1fetch and the runner's FetchFunc call. v1fetch keeps its exact response shape and behavior — its existing tests must pass unchanged. This is the one change to working code in P6 and is called out for reviewer attention.

12. Acceptance (contract) tests

Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the intended reasons. See docs/plans/p6/_files/.

1. dcache: create/read round trip, idempotency lookup (lease expiry + caller scope), live-job counts, ordered targets, exactly-once targets_done/lease bump, sweep, byte-eviction (oldest-first, never a live job), MarkInFlightFailed, BatchNbytes.
2. batch runner: serial order (peak in-flight 1), failure isolation (partial), idempotent resubmit (reused, no double-drain), the three caps, cancel (in-flight stored, later targets pending), panic recovery.
3. api: submit 202 + idempotent resubmit; a guard-pre-check rejection (non-80/443 port) is 422 at submit; 33 targets 422; a private-address target (passes pre-check, blocked at connect) is a per-target failed, aggregate partial; ownership (GET by another caller → 404); cancel → cancelled.
4. metrics: exact series names, label values, gauge values, histogram counts, and raw-outcome folding to other.
5. cmd/sift: parseBatchArgs verb/arg pairs and error cases.

13. Rollout

1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the v1fetch refactor, and the caps/lease semantics.
2. Orion pin bump with the [batch] config block; no MemoryMax change.
3. Skill update ships with the pin bump.
4. Watch sift_batch_* and tirith for a week; the success measure is that a research thread's one-request-per-URL fetch loop collapses into a single batch submit.

14. Changes after the plan was written

(none yet)