sift/docs/P6.md

# Sift P6 — asynchronous batch fetch

Drafted 2026-09-26 by straylight (Hermes agent) from the design discussion with Kyle. Reviewer:
Claude (when it frees up this evening). Implementer: Kyle's call.

## 1. Scope

In: an asynchronous batch fetch endpoint. An agent submits a list of targets (URLs, each with the
same per-target options as `/v1/fetch`) and gets a job ID back immediately; it polls for per-target
results later. One job drains **serially** — one target at a time — in the background.

Out:

- Parallel fan-out. A batch is deliberately slow; it is the tool for research threads that are
  already expected to take a long time, not for interactive fetches.
- Crawling, retry-with-backoff at the target level, and anything that requires a headless browser.
- Firecrawl compatibility. This is a native-only feature; no Hermes plugin.

The guard, auth and restricted-caller rules do not change: each target is fetched exactly like a
page, through the same SSRF guard and the same per-caller/host gates.

## 2. Why agents need this (2026-09-26)

`docs/PLAN.md` §13 (Kyle): "an agent that has collected a series of targets and wants them fetched
without holding a request open per target." Today that means a loop of `sift fetch` calls — one
request held open per URL, each its own tool call and gate. A batch lets the agent submit the whole
list, walk away, and collect once. It builds on the P5 infrastructure that already exists: the
`dcache` SQLite store, the `global`/`per` concurrency gates, and the `doFetch` pipeline.

## 3. Endpoints and wire shapes

| Method | Path | Result |
|---|---|---|
| `POST /v1/batch` | submit a list of targets | `202` created/reused, `400` malformed, `422` refused up front or over a cap, `429` caller/jobs busy |
| `GET /v1/batch/{id}` | status summary; `?results=1` inlines results | `200`, `401`/`404` |
| `GET /v1/batch/{id}/{idx}` | one target's status and result | `200`, `401`/`404` |
| `DELETE /v1/batch/{id}` | cancel | `200`, `409` already finished, `401`/`404` |

With `[batch] disabled = true` every batch endpoint answers `503 batch disabled`.

`POST /v1/batch` request — each target is the `/v1/fetch` field set minus the outer URL:

```json
{"targets": [{"url": "…", "mode": "clean", "question": "…", "pages": "…", "grep": "…",
              "jq": "…", "context": true, "recover": false}, …]}
```

Response `202`:
```json
{"success": true, "job_id": "…", "status": "queued", "targets": 3, "reused": false}
```

`GET /v1/batch/{id}` (a summary: no results, since 32 inline results can be ~6 MB in one agent
tool call):
```json
{"job_id": "…", "status": "partial", "targets_total": 3, "targets_done": 3,
 "created_at": "2026-09-27T01:02:03Z", "targets": [
   {"idx": 0, "url": "…", "status": "done"},
   {"idx": 1, "url": "…", "status": "failed", "error": "blocked: private address …", "hint": "browser"}
 ]}
```

`?results=1` adds `"result"` (the `/v1/fetch` envelope) to every `done` target.
`GET /v1/batch/{id}/{idx}` returns `{"job_id", "idx", "url", "status", "result"?, "error"?,
"hint"?}`. A `pending` target has `status: "pending"` and no `result`/`error`. A `202` for a new
job says `"status": "queued"`; a reused one says the job's current status. `DELETE` → `200 {"success": true, "status":
"cancelled"}`.

Job status vocabulary: `queued → running → done | partial | failed | cancelled`. Target status:
`pending → running → done | failed`.

## 4. Storage (`dcache`)

Two tables, added to `internal/dcache` (which owns the single SQLite connection):

- `batch_job(id, caller, caller_class, idem_key, status, error, targets_total, targets_done,
  created_at, lease, nbytes)`
- `batch_target(job_id, idx, url, options, status, result, error, hint)` — `result` is the JSON
  envelope, `NULL` until the target finishes.

Batch storage is a coherent whole: it is bounded by its own lease GC and a byte budget, **not** by
the content cache's LRU. `EvictBatchOldest` evicts whole jobs oldest-`lease`-first; a
`queued`/`running` job is never byte-evicted. `MarkInFlightFailed(reason, now)` is the restart hook.

## 5. The runner (`internal/batch`)

A new package with the job lifecycle and the serial state machine. Persistence is the `*dcache.DB`;
the only injected behavior is a `FetchFunc func(ctx, Target) (Result, error)`.

- **Idempotency (in v1).** A job has a random opaque `id` plus a derived `idem_key =
  SHA256(caller + "\x00" + canonical target list)`. Resubmitting the same (caller, targets) while
  the job's lease has not expired returns the existing job (`reused: true`) instead of a new one.
  Idempotency is lease-scoped, so it is "reuse within TTL, fresh after".
- **Caps.** targets/job 32; live jobs/caller 1; live jobs total 8. "Live" is by status
  (`queued`/`running`), not lease. Exceeding a cap is a typed error the API maps to `422`/`429`.
- **The caller.** `Submit` takes the `*auth.Caller`; the runner keeps it in memory with the job
  and passes it to every `FetchFunc` call (`func(ctx, *auth.Caller, Target)`), so the fetch runs
  under the caller's gates, pool slot, cache class and redaction. Only its name and class are
  stored.
- **Workers.** `workers` (default 1, at most 2) share one round-robin over the live jobs, one
  target at a time. A job is `queued` until a worker first takes it, then `running`.
- **Serial.** Targets drain in `idx` order; target `i+1` does not start until `i` is written.
- **Busy.** A `FetchFunc` error wrapping `ErrRetryLater` (the API maps fetch `429`/`503` to it)
  leaves the target pending; it is tried again after 2 s, for up to 10 min, then failed.
- **Failure isolation.** One target's error is stored per-target and the job continues; the job
  status is `done` (all ok), `failed` (all failed), or `partial` (mixed).
- **Cancel.** `DELETE` flips the job `cancelled`, stops before the next target, and lets the
  in-flight target finish and store. Completed targets stay readable until the lease lapses.

## 6. Scheduling and fairness

The workers run in the background, and every target still goes through the normal `acquire` gate
(8 global, 2 per-host). With 1–2 workers the batch holds at most 1–2 of the 8 global slots, so
interactive requests always have room. (The first draft let 8 live jobs each run a target at
once, which could fill the whole gate.) On `429 busy` the worker **waits and retries** instead of
failing: nobody is holding a socket open.

Two shared resources get the same treatment:
- **The PDF pool.** A batch extraction uses the pool caller name `<caller>/batch`, so it never
  takes the caller's own interactive slot. The single-flight key is unchanged, so a batch and an
  interactive request for the same PDF still share one child.
- **The digest gate.** A batch question queues on it like any other digest. With one worker
  that is at most one batch digest at a time.

Each batch fetch runs under the same 210 s deadline as a `/v1/fetch` request.

## 7. Retention and TTL

A job carries a single sliding **lease**, bumped on submit, on every target completion, and on
every read by its owner (the API calls `Touch` after the ownership check; the store's reads
never write). All targets therefore share one expiry (the last completion's), and a
slow consumer's reads keep a finished batch alive. GC deletes a job and its targets once the lease
is older than the TTL (24 h). Completed results survive a restart and stay readable until the lease
lapses; `queued`/`running` jobs are marked `failed` (`restart`) on startup and never resumed.

## 8. Restricted callers

Compartmentalized-access systems: their usage is redacted as much as possible. The existing
`logHost`/`logReason`/`logErr` machinery already blanks host, reason and error detail for
`PermitRestricted` callers, and the digest/embed caches partition by caller class — the batch
inherits all of it by passing the caller through. Two additions: batch log lines never carry a
target URL or hostname for **any** caller, and `GET`/`DELETE` enforce job ownership, returning
`404` (not `403`) for a job owned by another caller so a caller cannot probe for its existence.
The `idem_key` is stored as a hash and never logged alongside its target list.

## 9. Caps and metrics (sized empirically)

The caps are starting values, instrumented so they can be re-sized from observed data:

- `sift_batch_jobs_total{outcome}` — `created|reused|done|partial|failed|cancelled|restart`
- `sift_batch_targets_total{outcome}` — `ok|blocked|timeout|busy|upstream_4xx|upstream_5xx|
  unsupported|too_long|cancel`
- `sift_batch_queue_depth`, `sift_batch_targets_inflight` (gauges)
- `sift_batch_job_seconds`, `sift_batch_target_seconds`, `sift_batch_result_bytes` (histograms)

Both `outcome` counters fold any value outside their known set to `other`, so a raw string never
becomes a label. `result_bytes` is what sizes the byte budget (64 MiB).

## 10. CLI and skill

- `sift batch submit <targets.json>` (`-` reads stdin; prints `job_id` and `reused`).
- `sift batch status [--results] <job_id>` (the summary; `--results` inlines every result).
- `sift batch get <job_id> <idx>` (one result).
- `sift batch cancel <job_id>`.
- `skill/SKILL.md`: use `sift batch submit` for a list of targets instead of a loop of
  `sift fetch`; `sift batch status` to collect.
- `[batch]` config block: `disabled`, `targets_per_job`, `jobs_per_caller`, `jobs_total`,
  `workers`, `ttl`, `max_bytes`.
  The `/v1/batch` body cap (256 KiB, vs the 64 KiB global) is a constant, not config.

## 11. Refactor of existing code (explicit)

`v1fetch` today inlines the whole per-target pipeline (guard → `doFetch` → `recover` →
render/digest → context → envelope). The batch worker needs the same pipeline for one target. It is
extracted into one shared method, `func (s *Server) fetchTarget(ctx, *auth.Caller, batch.Target)
(batch.Result, error)`, that both `v1fetch` and the runner's adapter (`batchFetch`) call. It is its
own task (04), done before any batch endpoint exists, so the diff is a pure move. `v1fetch` keeps its
exact response shape and behavior — its existing tests must pass unchanged. This is the one change
to working code in P6 and is called out for reviewer attention.

## 12. Acceptance (contract) tests

Acceptance-first, as in P5a: given tests compiled against a panic-only skeleton, failing for the
intended reasons. See `docs/plans/p6/_files/` and the task table in `docs/plans/p6/README.md`.

1. `dcache`: round trip, idempotency lookup (lease expiry, caller scope, status filter), live
   counts, ordered targets, exactly-once `targets_done` and lease bumps, sweep, byte eviction
   (oldest first, never a live job), `MarkInFlightFailed`, `BatchNbytes`.
2. `batch` runner:
   - serial order (peak in flight 1), with the caller passed to every fetch;
   - round-robin between jobs, and `queued` → `running`;
   - failure isolation (`partial`, `failed`);
   - idempotent resubmit, including after `done`, but a new job after a cancel;
   - all three caps and `ErrNoTargets`; cancel (the in-flight target stored, later ones `pending`);
   - busy retry and its limit; panic recovery (the panic value not echoed); the hooks.
3. `metrics`: series names, labels, folding to `other`.
4. `api` `fetchTarget`: the same content and envelope as `/v1/fetch`; its errors as `*fetch.Error`.
5. `api` wiring: the runner through `fetchTarget`, a private-resolving name blocked per target,
   metrics through the hooks, outcome classes, busy → `ErrRetryLater`.
6. `guard.PreCheck`: no resolution; the port, scheme, userinfo and literal rules; the allow list.
7. `api` endpoints:
   - submit `202` and idempotent resubmit;
   - `400`/`422` refusals (the guard pre-check naming the index, 33 targets);
   - the 256 KiB body; the per-target `blocked` failure giving `partial`;
   - summary, one result, `?results=1`;
   - ownership `404` (GET, one result, DELETE); cancel and `409`;
   - no job id in `/metrics`.
8. `config`: the `[batch]` block, `ParseSize`. `cmd/sift`: `parseBatchArgs`.

## 13. Rollout

1. Implementer builds P6 on a branch; the reviewer (Claude) checks §11, the `v1fetch` refactor,
   and the caps/lease semantics.
2. Orion pin bump with the `[batch]` config block; no `MemoryMax` change.
3. Skill update ships with the pin bump.
4. Watch `sift_batch_*` and tirith for a week; the success measure is that a research thread's
   one-request-per-URL fetch loop collapses into a single batch submit.

## 14. Changes after the plan was written

**2026-09-26 evening, owner review (Claude), at Kyle's direction ("fold them into the plan").**
The design stands; these were implementation-blocking or would have hurt agents:

1. **The caller was lost.** `FetchFunc` had no caller and the runner was built once, so the given
   api test (`batch.New(db, s.fetchTarget, …)`) could not compile against the specified
   `fetchTarget(ctx, *auth.Caller, Target)`. Made to compile, it would have fetched every target
   with no caller: no per-caller gates, pool limit, cache class or redaction. Now `Submit` takes
   the `*auth.Caller` and `FetchFunc` receives it (§5).
2. **Task 01 rule 1 contradicted its test.** Store reads no longer bump the lease; the API calls
   `Touch` after an owner's read (§7).
3. **Batch could starve interactive fetches.** 8 live jobs × 1 target filled the 8-slot global
   gate. The busy-retry of §6 was in no task. Now there are 1–2 round-robin workers, busy-retry
   is in the runner, a batch PDF uses its own pool name, and a per-target deadline applies (§5, §6).
4. **Flaky, networked API tests.** Two tests fetched `https://example.com/`, and the cancel test
   raced the fetch. Every api test is now local, using held `httptest` servers and a fake DNS
   name for the private-address case.
5. **A resubmit could return a dead job.** A failed or cancelled job was reused for 24 h, and
   polling it kept extending that. Now only `queued`/`running`/`done`/`partial` jobs are reused.
6. **Reads were too large.** `GET` inlined every result (up to ~6 MB). It is now a summary, with
   `GET …/{idx}` and `?results=1` (§3). The CLI gains `get` and `status --results`.
7. **Body cap.** The global 64 KiB `MaxBytesReader` runs in `ServeHTTP`, so a handler cannot
   raise it. `ServeHTTP` now picks 256 KiB for `/v1/batch`. Request metrics use a fixed
   `/v1/batch/{id}` label, so a job id never becomes a label value.
8. **Submit pre-check.** `guard.ParseURL` rejects the test's own loopback port and knows nothing
   of the allow list. New `Guard.PreCheck` is `CheckURL` without resolution (an owner commit on
   master, since `AGENTS.md` keeps the implementer out of `internal/guard/`). A private IP literal is now refused at submit; a *name* that resolves privately
   still fails per target at fetch time.
9. **Task shape.** 5 tasks → 9 (the last split came during the run, below), one package each where possible:
   01 dcache, 02 runner core, 03 runner workers, 04 metrics, 05 the `fetchTarget` refactor alone,
   06 runner wiring, 07 endpoints, 08 serve/config, 09 CLI/docs. Every task restates the APIs it
   uses from earlier tasks. Every given test compiles against a panic-only skeleton and fails
   there on `panic: todo`.
10. **Also fixed:**
    - the `ErrBusy` (jobs total) cap now has a test;
    - the `202` status is `queued`, consistent with the workers;
    - the `restart` outcome is counted by `serve`;
    - `Options.Context` is `*bool`, keeping `/v1/fetch`'s default.

**Sequencing.** P6 starts after P5c merges. P5b/P5c change `v1fetch` (the digest meta gains
`retrieval`/`embedded`), which is the code task 05 moves. Recheck the api given tests against
a skeleton built on post-P5c master before the run.

**During the run (2026-09-26 21:08).** Task 02 (the whole runner) ended with no file: the model
wrote the implementation out in its reply, the session was compacted, and it then emitted a
malformed tool call. The runner is now two tasks (02 core, 03 workers), each with its own given
test. `AGENTS.md` gains a rule to write code into files early. Copy paths are spelled from the
repo root. Details in `docs/plans/p6/README.md`, "Changes during the run".