Wire the batch runner to fetchTarget and feed the sift_batch_* metrics.
New internal/api/batch.go: NewBatchRunner overwrites cfg.Hooks (Job, Target,
JobSeconds, Depth, Inflight → s.Prom) and sets the runner's FetchFunc to
batchFetch, which tags the context as a batch fetch, runs under requestDeadline,
and maps fetchTarget's errors to what the runner retries (429/503 →
ErrRetryLater) and what the metrics classify (batchOutcome). Added the Batch
field to Server (nil → task 07 answers 503 "batch disabled") and, in
extractPDF, charged the PDF pool to c.Name+"/batch" under the batch mark so a
batch never takes a caller's interactive slot while the single-flight key is
unchanged. No HTTP endpoints.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The digest now handles long documents itself (task 02), so the API's refusal
limit is only a safety cap: DefaultPDFMaxQuestionChunks 200 -> 5000 (the 422
message text is unchanged). In v1fetch the digest-meta object gains "retrieval"
and "embedded" next to chunks_used/chunks_total, only when the digest ran
without a fallback. The PDF section explains a long-doc question is answered
first from BM25 while its chunks embed in the background and hybrid on later
questions; skill/SKILL.md gains one sentence. Given tests byte-identical.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
README gains API, CLI and configuration sections and the PDF range/grep/question limits; the skill covers PDF busy and failure results; dist/sift.service matches the Nix unit (MemoryMax=1G, secret.env); the example config drops the unused [upstreams] block; stale task comments in api/pdf.go.
Gate overridden by the owner: TestGrepLimits (given test) exceeds its 1 s budget under -race on this host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- pdftext/outline: resolve named destinations through the catalog /Dests and the /Names /Dests
tree, resolve indirect /Dest, /A and /Title, and treat reader.Null (what Dict.Get returns for
a missing key) as absent. Before this every LaTeX and Acrobat PDF had an empty outline.
pdftest gains NamedDests; TestOutlineNamedDestinations.
- api: pool() always goes through its Once (data race on concurrent first use);
TestDefaultPoolConcurrentFirstUse. Revalidation of a stale cache entry counts against the
caller/host limit.
- pdftext/runner: output over MaxText kills the child at once instead of waiting for the
deadline; TestRunnerTooMuchTextFailsFast.
- fetch: Accept includes application/pdf;q=0.8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Digest systemPrompt gains a page-cite instruction and PromptVersion is 3.
Add the sift_pdf_* metric series (extract_total{outcome}, extract_seconds,
pages_total, pool_wait_seconds, cache_hits_total) to internal/metrics and wire
ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go. Refactor
cmd/sift/fetchcmd.go to a fetchOpts struct with parseFetchOpts and fetchBody,
adding --pages/--grep. Update skill/SKILL.md, set nix MemoryMax to 1G, and add a
PDF documents section to the README.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Answer extracted PDFs from the document cache: fresh for PDFTTL, then a
conditional GET (304 reuses the stored text, a 200 is extracted and stored
without a second GET). Store after extraction with the response hash, ETag,
fetch time and extracted pages. Route pages/grep through pdfdoc.Options with
400s for bad ranges and non-PDFs, and refuse a question when the document
exceeds PDFMaxQuestionChunks with a pointer to grep and pages.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Route "pdf" through clean.Kind in doFetch/cleanFetched: extract under the pool
(keyed by caller class + document hash, one per caller), map the five pdftext
errors plus a pool panic to 422, ErrBusy to 503+Retry-After 10, ErrCallerBusy to
429, and answer with the whole text or the map. Adds RetryAfter to fetch.Error
and the matching header. No cache, pages or grep — those are task 07.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
kyleandornith-1.5-35b-a3b
