kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cli: sift batch submit|status|get|cancel; README and skill

Browse Source 
Add `sift batch` so agents reach the batch endpoint through the CLI like
`fetch` and `search`. `batchcmd.go` has `parseBatchArgs` (verb plus the
unchanged args after it; submit takes one arg, status a job id optionally
preceded by `--results`, get a job id plus a non-negative-integer idx, cancel
a job id, else a usage error), `batchMain` dispatch, one small function per
verb, and `batchDo` (bearer header, decode the envelope, print it on stdout,
on non-2xx the error to stderr and `exitFor`). `submitMain` reads the file or
stdin and validates locally: a JSON object with a non-empty `targets` array,
or a bare array of targets or URL strings wrapped into `{"targets": [...]}`.

README gains a "Batch" section (endpoints, serial-per-job/one-worker/
round-robin model, caps, idempotency, lease/TTL/restart, summary-vs-results,
redaction, metrics); SKILL.md gains a "Batch" section.

Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
This commit is contained in:
kyle
2026-09-26 23:13:48 -07:00
co-authored by ornith-1.5-35b-a3b
parent 484c33f986
commit be48ecb550
6 changed files with 317 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+40 -3
View File
@@ -34,9 +34,10 @@ sift serve [-listen addr] [-config /etc/sift/config.toml] [-keys /var/lib/sift/
sift fetch <url> [--question q] [--mode clean|raw|digest] [--jq filter] [--pages a-b] [--grep re]
[--no-context] [--recover] [--json]
sift extract <file.pdf> [--pages a-b] [--grep re] [--question q] [--mode clean|digest] [--json]
sift search "<query>" [--k n] [--json]
sift keys add --caller name [--workspace ws] [--out file] | list | revoke --caller name
sift version
sift search "<query>" [--k n] [--json]
sift batch submit|status|get|cancel
sift keys add --caller name [--workspace ws] [--out file] | list | revoke --caller name
sift version
```
`sift fetch`, `sift extract` and `sift search` are clients: they read `url` and `key_file` from
@@ -46,6 +47,42 @@ answer exactly as `sift fetch` does for a PDF URL — `--pages`, `--grep` and `-
same, and a file over 100 MiB is refused before the upload. The agent-facing guide is
[`skill/SKILL.md`](skill/SKILL.md).
## Batch
`sift batch` runs several fetches as one job, so an agent can read a handful of URLs later instead
of a loop of `sift fetch`. The four endpoints:
| Endpoint | What |
|---|---|
| `POST /v1/batch` | `{"targets":[…]}` → `202 {"success","job_id","status","targets","reused"}` |
| `GET /v1/batch/{id}` | summary; `?results=1` inlines each done target's `/v1/fetch` envelope |
| `GET /v1/batch/{id}/{idx}` | one target's result, as `sift fetch --json` |
| `DELETE /v1/batch/{id}` | cancel a queued job → `200 {"success","status"}` |
`sift batch submit <targets.json|->` reads a JSON object with a non-empty `targets` array locally,
or a bare array of targets or URL strings (each string becomes `{"url": s}`), then posts it. `status`
prints the summary (or the body with `?results=1`); `get <job> <idx>` prints one result; `cancel`
stops a queued job.
A job runs **serially** (target `i+1` never starts before `i` is stored) with **one worker by
default** (see `[batch]` config) and the runner rotates jobs **round-robin** while several are live.
Caps: **32 targets** per job, **1 live job per caller**, **8 live jobs** in all. The posted body is
capped at **256 KiB**. A target that is blocked, times out, or errors is recorded and the job
continues; a job whose targets are all done is `done`, all failed `failed`, mixed `partial`.
**Idempotency**: the same list from the same caller returns the same job while it is live or already
has results (`reused: true`), so a retry does not start a second job. Each job carries a **lease**
that a status read renews; a job whose lease has expired for more than the **24 h TTL** is swept
along with its targets. On restart, jobs in flight when the server died are marked failed so a retry
re-submits them.
A `status` read is a **summary** (per-target status, never the content); `?results=1` inlines the
full envelope for each done target, so it can be large — prefer `get <job> <idx>` for one result.
Restricted callers see the same endpoints; their results are **redacted** the same way a restricted
`sift fetch` is. Batch counters are `sift_batch_jobs_total{outcome}`,
`sift_batch_targets_total{outcome}`, `sift_batch_queue_depth`, `sift_batch_targets_inflight`,
`sift_batch_job_seconds`, `sift_batch_target_seconds` and `sift_batch_result_bytes`.
## Configuration
`/etc/sift/config.toml` (`$SIFT_CONFIG`); every key is shown in