Add `sift batch` so agents reach the batch endpoint through the CLI like
`fetch` and `search`. `batchcmd.go` has `parseBatchArgs` (verb plus the
unchanged args after it; submit takes one arg, status a job id optionally
preceded by `--results`, get a job id plus a non-negative-integer idx, cancel
a job id, else a usage error), `batchMain` dispatch, one small function per
verb, and `batchDo` (bearer header, decode the envelope, print it on stdout,
on non-2xx the error to stderr and `exitFor`). `submitMain` reads the file or
stdin and validates locally: a JSON object with a non-empty `targets` array,
or a bare array of targets or URL strings wrapped into `{"targets": [...]}`.
README gains a "Batch" section (endpoints, serial-per-job/one-worker/
round-robin model, caps, idempotency, lease/TTL/restart, summary-vs-results,
redaction, metrics); SKILL.md gains a "Batch" section.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add the [batch] config block (Batch type + File.Batch) and ParseSize (binary
KiB/MiB/GiB byte counts) in internal/config. serve now opens the dcache store
when batch is on, marks the queued/running jobs a restart killed as failed
(never resumed) and observes them, builds the runner from the config with the
runner's defaults filling zeros, and closes the runner on both shutdown paths.
Batch is on by default; disabled = true leaves srv.Batch nil so the endpoints
answer 503.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The HTTP surface over s.Batch. New internal/api/batch.go: batchSubmit (POST)
decodes {"targets":[{url,mode,question,pages,grep,jq,context,recover}]} flat as
/v1/fetch does, 400 "invalid batch" on bad json / empty list / empty url / bad
mode, then PreCheck per target -> 422 "blocked: target <idx>: <reason>" with no
job, then Submit mapping ErrTooManyTargets->422, ErrCallerBusy->429 Retry-After
30, ErrBusy->429 Retry-After 60, ErrNoTargets->400, and 202
{success,job_id,status,targets,reused}. batchStatus (GET), batchResult
(GET {id}/{idx}) and batchCancel (DELETE) share ownership: unknown or not
c.Name -> 404, never 403; status is a summary (no inline results unless
?results=1), idx validated 0..targets_total-1 else 404, cancel of a terminal
job -> 409 else 200 {success,status:cancelled}. api.go scoped to rule 2:
batchMaxBody (256 KiB), the four routes in New, and ServeHTTP applying the
per-path cap once and collapsing every batch path to a fixed metrics label so a
job id never becomes a label. Each handler logs one batch line, never a URL,
hostname or idem_key.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Wire the batch runner to fetchTarget and feed the sift_batch_* metrics.
New internal/api/batch.go: NewBatchRunner overwrites cfg.Hooks (Job, Target,
JobSeconds, Depth, Inflight → s.Prom) and sets the runner's FetchFunc to
batchFetch, which tags the context as a batch fetch, runs under requestDeadline,
and maps fetchTarget's errors to what the runner retries (429/503 →
ErrRetryLater) and what the metrics classify (batchOutcome). Added the Batch
field to Server (nil → task 07 answers 503 "batch disabled") and, in
extractPDF, charged the PDF pool to c.Name+"/batch" under the batch mark so a
batch never takes a caller's interactive slot while the single-flight key is
unchanged. No HTTP endpoints.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Move v1fetch's inline per-target pipeline into a new internal/api/target.go
method, `fetchTarget(ctx, c, t batch.Target) (batch.Result, error)`, so the
batch worker can drive the same work. The body moves unchanged in order:
question truncation, mode default/check, wantCtx, doFetch, the recover path,
the PDF branch (render, pages/grep errors, question-chunk limit), the
clean/raw branch, digest, context search, the fetch log line and the
envelope. `content` is returned as Result.Content; every other envelope key
goes in Result.Meta, never content. v1fetch keeps reqN/caller/decode/400,
builds a batch.Target from the request, calls fetchTarget, writeFetchErr on
error and writeJSON(w, 200, env) on success where env is a copy of Meta plus
"content". doFetch takes context.Context instead of *http.Request; scrape
passes r.Context(). No new behaviour, fields or JSON keys.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
New internal/metrics/batch.go adds the batch series to metrics.Set, registered
in New: BatchJobsTotal and BatchTargetsTotal counters (label "outcome", any
value outside the known set folded to "other" so a raw string never becomes a
label), BatchQueueDepth and BatchTargetsInflight gauges, BatchJobSeconds and
BatchTargetSeconds histograms (the existing request buckets) and
BatchResultBytes (1K,4K,16K,64K,256K,1M). The Observe…/Set… methods are
nil-receiver safe like ObservePDF and flow through the existing /metrics
handler; no new endpoint or Snapshot change.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Start cfg.Workers goroutines in New that drain one rotation of live jobs in
submit order, each taking the next job with a pending target and none in flight,
so targets alternate across jobs and each job stays serial. One target per fetch
under the Close-cancelled context: panic recovers into ErrInternal (never
echoed), ErrRetryLater retries after RetryWait (bounded by RetryFor), otherwise
the target is stored done/failed and the hooks fire. Terminal status is set once
per job; cancelling a running job stores the in-flight target without overwriting
cancelled and leaves later targets pending.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
New package internal/batch holding the runner core with no workers yet (task 03
adds them; until then every job stays queued, so a queued job counts as live for
the caps). batch.go defines the public surface: Options/Target/Result/TargetError,
FetchFunc, Job/TargetStatus, Hooks and Config with its defaults (TTL 24h, 32
targets/job, 1 job/caller, 8 total, worker clamped to 2, 64 MiB, 2 s / 10 m retry).
Submit runs in the given order: empty → ErrNoTargets; idempotency via hex SHA-256
of `c.Name + NUL + json.Marshal(targets)` (a live BatchJobByIDem hit returns the id
reused, touching no caps); then TargetsPerJob / JobsPerCaller / JobsTotal; then it
stores the job queued (random 16-byte hex id, CreatedAt = Lease = now) and its
targets (pending, Options = json of t.Options) before returning. The submitting
caller is held in memory keyed by job id, never persisted (jobs never resume after
a restart). Status maps the rows and decodes Result from the result column without
writing; Touch moves the lease; Cancel sets a queued job cancelled at once and
leaves its targets pending; Close is idempotent. Store errors are Warn-logged with
the job id, never a URL or host.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Task 02 (the whole runner) ended at 21:08 with no file and no commit: Ornith wrote the
implementation out in its reply (~700 lines), OpenCode compacted the session, and the model
then emitted a malformed tool call. It had also looked for the given test at a wrong path
("_files/…" relative to the plan: my ambiguity).
- 02-runner → 02-runner-core (types, Submit, Status, Touch, Cancel; no goroutines) and
03-runner-workers; the given test split to match (runner_core_test.go gains three tests:
queued storage, cancel while queued, Status never moves the lease); later tasks renumbered
04-09 and every cross-reference checked.
- Copy paths spelled from the repo root.
- guard.PreCheck (+ its test) is committed here by the owner: the old task 05 told the
implementer to edit internal/guard/, which AGENTS.md forbids. My contradiction.
- AGENTS.md rule 6: write code into files early, in small steps.
Both runner test files compile against a panic-only skeleton (the core file also alone) and
fail there on panic: todo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the batch job store to internal/dcache (storage only; no HTTP, no worker).
New internal/dcache/batch.go holds the BatchJob/BatchTarget types and all
accessors; Open gains the batch_job/batch_target tables and indices, and DB
gains BatchH/BatchM counters.
Sliding lease: writes (UpdateBatchTarget, SetBatchJobStatus, TouchBatchJob,
MarkInFlightFailed) bump the job lease to now; reads never write. BatchJobByIDem
returns the newest (caller, idem_key) job with lease+ttl>now and status
queued/running/done/partial. UpdateBatchTarget counts targets_done exactly once
per target entering done/failed and adds len(result) to nbytes. SweepBatch
removes by lease; EvictBatchOldest evicts whole non-live jobs oldest-lease-first;
MarkInFlightFailed reconciles in-flight jobs on restart. Batch nbytes are kept out
of Size() and the content evict union.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Found in the P5b prod check: "how many PIO state machines in total?" on the RP2040 datasheet
came back 4 (bm25 and hybrid); the answer, 8, is a terse bullet on p. 9 that --grep finds at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner review of p5c (tasks 01-02 by Ornith):
- extract.go: an upload (up to 100 MiB) is read whole before the PDF pool's limits apply, so
any number of concurrent uploads could buffer at once in a 1 GiB service. At most 2 bodies
are read at a time now; a third waits up to 5 s, then 429 busy with Retry-After 10.
- extract.go: `warnings` was null when empty; it is [] as in /v1/fetch.
- extractcmd.go: extractURL replaced the client base URL's path; it now appends /v1/extract
like fetch and search do.
Regression tests in extract_review_test.go (api and cmd/sift).
End-to-end (local serve -> Dixie; sift extract with SIFT_CLIENT): Llama 2 paper, 13.6 MB /
77 pages: --pages 1 ok, repeat served from the content-hash cache, --grep 2 hits, --question
answered with page citations in 16.8 s. Log lines carry bytes/pages/mode/cached only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sift extract <file.pdf>` stats the file (refusing over 100 MiB before the
upload), streams it as the POST body with Content-Type application/pdf and the
ContentLength set, and prints the /v1/extract answer exactly as `sift fetch`
prints a PDF URL — same content output, same errors and exitFor codes.
parseExtractOpts reuses splitAroundPositional; extractURL sends only the set
options with mode=digest when a question is given. main.go gains the case and a
help line; README and skill document the command.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add POST /v1/extract: auth the caller, read the PDF body, hash it, and answer
it on the PDF fetch path exactly as /v1/fetch answers the same PDF by URL. The
hash is the content-addressed cache key ("upload:"+hash under the caller class),
so identical content is extracted once with no TTL or revalidation. The api.go
diff is scoped to the rule-1 changes: the DefaultPDFMaxUpload const, the
PDFMaxUpload Server field, the route, and the two ServeHTTP lines.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Owner review of p5b (tasks 01-03 by Ornith):
- background.go: `seen` was never cleared, so a document whose background embedding failed
(Dixie hiccup) or was abandoned could never be queued again: every later question on it stayed
BM25 for the life of the process. Dedupe now covers only queued and in-progress documents.
- background.go: acquireGate gave up after one retry and dropped the document; the task said to
keep retrying once a second. It now waits (background work has no deadline).
- retrieval.go/background.go: a vector the server left out is not cached as an empty vector.
- bm25.go: Scores on a nil *BM25 dereferenced it; returns nil.
Regression tests in review_test.go; both fail on the old code.
End-to-end against Dixie (local serve): Karpathy's thesis, 95 pages / 51 chunks: bm25 in 19 s,
hybrid (51 embedded) after the background; RP2040 datasheet, 642 pages / 284 chunks (a 422
before P5b): bm25 in 13.5 s, all 284 embedded within 90 s, hybrid answer correct with its quote.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The digest now handles long documents itself (task 02), so the API's refusal
limit is only a safety cap: DefaultPDFMaxQuestionChunks 200 -> 5000 (the 422
message text is unchanged). In v1fetch the digest-meta object gains "retrieval"
and "embedded" next to chunks_used/chunks_total, only when the digest ran
without a fallback. The PDF section explains a long-doc question is answered
first from BM25 while its chunks embed in the background and hybrid on later
questions; skill/SKILL.md gains one sentence. Given tests byte-identical.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Cached chunk embeddings change what a long document answers with: BM25 first,
embedding uncached chunks (or the whole document) in the background so the next
question on the same doc scores hybrid. selectText now returns its mode and the
number of chunks that had a vector at answer-build time; the digest cache stores
only hybrid answers, never a provisional bm25 one. The background worker batches
through a Dixie gate and WaitBackground polls for true idle.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
SplitPages is Split's budget/line splitting plus page awareness: a "--- page N ---"
line sets the current page and a chunk that starts after the first marker but not on
a marker line is prefixed "--- page N (cont.) ---\n\n" for the page its first line
belongs to, so every chunk names its page. BM25 (k1=1.2, b=0.75) ranks chunks
lexically with counted term frequencies and ln(1+(N-df+0.5)/(df+0.5)) IDF.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Found by Ornith in the p5b/02 session (it measured the document); my test's fault, not the task's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check distinguishes linear time from catastrophic backtracking, which would never
finish on a 1 MB input, so a generous bound loses nothing. Reported by talos.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README gains API, CLI and configuration sections and the PDF range/grep/question limits; the skill covers PDF busy and failure results; dist/sift.service matches the Nix unit (MemoryMax=1G, secret.env); the example config drops the unused [upstreams] block; stale task comments in api/pdf.go.
Gate overridden by the owner: TestGrepLimits (given test) exceeds its 1 s budget under -race on this host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- pdftext/outline: resolve named destinations through the catalog /Dests and the /Names /Dests
tree, resolve indirect /Dest, /A and /Title, and treat reader.Null (what Dict.Get returns for
a missing key) as absent. Before this every LaTeX and Acrobat PDF had an empty outline.
pdftest gains NamedDests; TestOutlineNamedDestinations.
- api: pool() always goes through its Once (data race on concurrent first use);
TestDefaultPoolConcurrentFirstUse. Revalidation of a stale cache entry counts against the
caller/host limit.
- pdftext/runner: output over MaxText kills the child at once instead of waiting for the
deadline; TestRunnerTooMuchTextFailsFast.
- fetch: Accept includes application/pdf;q=0.8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Digest systemPrompt gains a page-cite instruction and PromptVersion is 3.
Add the sift_pdf_* metric series (extract_total{outcome}, extract_seconds,
pages_total, pool_wait_seconds, cache_hits_total) to internal/metrics and wire
ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go. Refactor
cmd/sift/fetchcmd.go to a fetchOpts struct with parseFetchOpts and fetchBody,
adding --pages/--grep. Update skill/SKILL.md, set nix MemoryMax to 1G, and add a
PDF documents section to the README.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Answer extracted PDFs from the document cache: fresh for PDFTTL, then a
conditional GET (304 reuses the stored text, a 200 is extracted and stored
without a second GET). Store after extraction with the response hash, ETag,
fetch time and extracted pages. Route pages/grep through pdfdoc.Options with
400s for bad ranges and non-PDFs, and refuse a question when the document
exceeds PDFMaxQuestionChunks with a pointer to grep and pages.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Route "pdf" through clean.Kind in doFetch/cleanFetched: extract under the pool
(keyed by caller class + document hash, one per caller), map the five pdftext
errors plus a pool panic to 422, ErrBusy to 503+Retry-After 10, ErrCallerBusy to
429, and answer with the whole text or the map. Adds RetryAfter to fetch.Error
and the matching header. No cache, pages or grep — those are task 07.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Pure render function on a *pdftext.Doc with no I/O: the whole text with
page markers when it fits, otherwise a map (title, outline with page
numbers, leading pages and how to ask for more), a page range, or grep
results with page numbers. Every length is in runes.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Generic in-process scheduler for the expensive PDF-extraction child process.
A pool runs at most `slots` jobs, keeps at most `queue` waiting, and refuses a
waiter past `wait`. Do() applies six rules in order: join an existing running
or queued job (no slot, allowed while the caller holds another, fn may be nil);
refuse a caller that already has a job in flight; take a free slot, else queue
(first-come-first-served, else ErrBusy); run fn under context.WithoutCancel so
the flight outlives a caller that leaves; broadcast the result to every sharer
with no caching and a recovered panic returned as an error. Running counts only
jobs holding a slot; Queued counts the line; Waiting(key) counts joiners.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add pdf_doc/pdf_page tables plus PDFEntry, PDFKey, PutPDF (one transaction,
old pages deleted first), GetPDF (page-ordered, updates last_hit) and TouchPDF
(304 revalidation moves only fetched_at). PDF documents count toward Size() and
are evicted whole by oldest use; PDFH/PDFM counters added. evict() now breaks
last_hit ties by rowid so same-second inserts evict in insertion order.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The parser is pure Go but a goroutine cannot be killed, so extraction runs in
a child process: the binary re-executed as `sift pdftext`, reading the PDF on
stdin and writing JSON Lines on stdout, killed at a deadline by the parent.
pdftext.go holds the types, the five errors, Extract (reader.Open with
ErrWrongPassword -> ErrEncrypted, per-page recover, trailer /Info /Title, and
rule-3 UTF-16BE/Latin-1 text decode) and ChildMain (soft 768 MiB memory limit,
-no env read, no network, exit codes 0/3/4/1). outline.go walks /Outlines ->
/First -> /Next with /First children at level+1, resolving /Dest and /GoTo /A
dests to 1-based pages, bounded at 10,000 items and never visiting an object
twice. runner.go is the Runner: env-isolated child (nil env is an empty
environment), 60 s deadline, 64 MiB output cap, 200 MiB stdin cap, stderr
capped at 4 KiB, exit-code to error mapping, and Doc parsing with ErrNoText.
cmd/sift/main.go adds the unlisted `pdftext` case. tools.go is removed; the
go-pdfkit modules stay vendored.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Recognise PDFs in clean.Kind (declared types and %PDF- sniffing on
octet-stream/bodyless bodies), give PDF responses the 100 MB body cap and
120 s deadline while web pages keep 5 MB / 20 s, and add FetchIf, which
sends If-None-Match / If-Modified-Since and maps a 304 to a Resource with
Status 304 and an empty Body. Fetch and FetchIf share one doFetch path for
guard, client, limits and error mapping.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
kyleandornith-1.5-35b-a3b
