Wire the batch runner to fetchTarget and feed the sift_batch_* metrics.
New internal/api/batch.go: NewBatchRunner overwrites cfg.Hooks (Job, Target,
JobSeconds, Depth, Inflight → s.Prom) and sets the runner's FetchFunc to
batchFetch, which tags the context as a batch fetch, runs under requestDeadline,
and maps fetchTarget's errors to what the runner retries (429/503 →
ErrRetryLater) and what the metrics classify (batchOutcome). Added the Batch
field to Server (nil → task 07 answers 503 "batch disabled") and, in
extractPDF, charged the PDF pool to c.Name+"/batch" under the batch mark so a
batch never takes a caller's interactive slot while the single-flight key is
unchanged. No HTTP endpoints.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Move v1fetch's inline per-target pipeline into a new internal/api/target.go
method, `fetchTarget(ctx, c, t batch.Target) (batch.Result, error)`, so the
batch worker can drive the same work. The body moves unchanged in order:
question truncation, mode default/check, wantCtx, doFetch, the recover path,
the PDF branch (render, pages/grep errors, question-chunk limit), the
clean/raw branch, digest, context search, the fetch log line and the
envelope. `content` is returned as Result.Content; every other envelope key
goes in Result.Meta, never content. v1fetch keeps reqN/caller/decode/400,
builds a batch.Target from the request, calls fetchTarget, writeFetchErr on
error and writeJSON(w, 200, env) on success where env is a copy of Meta plus
"content". doFetch takes context.Context instead of *http.Request; scrape
passes r.Context(). No new behaviour, fields or JSON keys.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
New internal/metrics/batch.go adds the batch series to metrics.Set, registered
in New: BatchJobsTotal and BatchTargetsTotal counters (label "outcome", any
value outside the known set folded to "other" so a raw string never becomes a
label), BatchQueueDepth and BatchTargetsInflight gauges, BatchJobSeconds and
BatchTargetSeconds histograms (the existing request buckets) and
BatchResultBytes (1K,4K,16K,64K,256K,1M). The Observe…/Set… methods are
nil-receiver safe like ObservePDF and flow through the existing /metrics
handler; no new endpoint or Snapshot change.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Start cfg.Workers goroutines in New that drain one rotation of live jobs in
submit order, each taking the next job with a pending target and none in flight,
so targets alternate across jobs and each job stays serial. One target per fetch
under the Close-cancelled context: panic recovers into ErrInternal (never
echoed), ErrRetryLater retries after RetryWait (bounded by RetryFor), otherwise
the target is stored done/failed and the hooks fire. Terminal status is set once
per job; cancelling a running job stores the in-flight target without overwriting
cancelled and leaves later targets pending.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
New package internal/batch holding the runner core with no workers yet (task 03
adds them; until then every job stays queued, so a queued job counts as live for
the caps). batch.go defines the public surface: Options/Target/Result/TargetError,
FetchFunc, Job/TargetStatus, Hooks and Config with its defaults (TTL 24h, 32
targets/job, 1 job/caller, 8 total, worker clamped to 2, 64 MiB, 2 s / 10 m retry).
Submit runs in the given order: empty → ErrNoTargets; idempotency via hex SHA-256
of `c.Name + NUL + json.Marshal(targets)` (a live BatchJobByIDem hit returns the id
reused, touching no caps); then TargetsPerJob / JobsPerCaller / JobsTotal; then it
stores the job queued (random 16-byte hex id, CreatedAt = Lease = now) and its
targets (pending, Options = json of t.Options) before returning. The submitting
caller is held in memory keyed by job id, never persisted (jobs never resume after
a restart). Status maps the rows and decodes Result from the result column without
writing; Touch moves the lease; Cancel sets a queued job cancelled at once and
leaves its targets pending; Close is idempotent. Store errors are Warn-logged with
the job id, never a URL or host.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Task 02 (the whole runner) ended at 21:08 with no file and no commit: Ornith wrote the
implementation out in its reply (~700 lines), OpenCode compacted the session, and the model
then emitted a malformed tool call. It had also looked for the given test at a wrong path
("_files/…" relative to the plan: my ambiguity).
- 02-runner → 02-runner-core (types, Submit, Status, Touch, Cancel; no goroutines) and
03-runner-workers; the given test split to match (runner_core_test.go gains three tests:
queued storage, cancel while queued, Status never moves the lease); later tasks renumbered
04-09 and every cross-reference checked.
- Copy paths spelled from the repo root.
- guard.PreCheck (+ its test) is committed here by the owner: the old task 05 told the
implementer to edit internal/guard/, which AGENTS.md forbids. My contradiction.
- AGENTS.md rule 6: write code into files early, in small steps.
Both runner test files compile against a panic-only skeleton (the core file also alone) and
fail there on panic: todo.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the batch job store to internal/dcache (storage only; no HTTP, no worker).
New internal/dcache/batch.go holds the BatchJob/BatchTarget types and all
accessors; Open gains the batch_job/batch_target tables and indices, and DB
gains BatchH/BatchM counters.
Sliding lease: writes (UpdateBatchTarget, SetBatchJobStatus, TouchBatchJob,
MarkInFlightFailed) bump the job lease to now; reads never write. BatchJobByIDem
returns the newest (caller, idem_key) job with lease+ttl>now and status
queued/running/done/partial. UpdateBatchTarget counts targets_done exactly once
per target entering done/failed and adds len(result) to nbytes. SweepBatch
removes by lease; EvictBatchOldest evicts whole non-live jobs oldest-lease-first;
MarkInFlightFailed reconciles in-flight jobs on restart. Batch nbytes are kept out
of Size() and the content evict union.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
`sift extract <file.pdf>` stats the file (refusing over 100 MiB before the
upload), streams it as the POST body with Content-Type application/pdf and the
ContentLength set, and prints the /v1/extract answer exactly as `sift fetch`
prints a PDF URL — same content output, same errors and exitFor codes.
parseExtractOpts reuses splitAroundPositional; extractURL sends only the set
options with mode=digest when a question is given. main.go gains the case and a
help line; README and skill document the command.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add POST /v1/extract: auth the caller, read the PDF body, hash it, and answer
it on the PDF fetch path exactly as /v1/fetch answers the same PDF by URL. The
hash is the content-addressed cache key ("upload:"+hash under the caller class),
so identical content is extracted once with no TTL or revalidation. The api.go
diff is scoped to the rule-1 changes: the DefaultPDFMaxUpload const, the
PDFMaxUpload Server field, the route, and the two ServeHTTP lines.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The digest now handles long documents itself (task 02), so the API's refusal
limit is only a safety cap: DefaultPDFMaxQuestionChunks 200 -> 5000 (the 422
message text is unchanged). In v1fetch the digest-meta object gains "retrieval"
and "embedded" next to chunks_used/chunks_total, only when the digest ran
without a fallback. The PDF section explains a long-doc question is answered
first from BM25 while its chunks embed in the background and hybrid on later
questions; skill/SKILL.md gains one sentence. Given tests byte-identical.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Cached chunk embeddings change what a long document answers with: BM25 first,
embedding uncached chunks (or the whole document) in the background so the next
question on the same doc scores hybrid. selectText now returns its mode and the
number of chunks that had a vector at answer-build time; the digest cache stores
only hybrid answers, never a provisional bm25 one. The background worker batches
through a Dixie gate and WaitBackground polls for true idle.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
SplitPages is Split's budget/line splitting plus page awareness: a "--- page N ---"
line sets the current page and a chunk that starts after the first marker but not on
a marker line is prefixed "--- page N (cont.) ---\n\n" for the page its first line
belongs to, so every chunk names its page. BM25 (k1=1.2, b=0.75) ranks chunks
lexically with counted term frequencies and ln(1+(N-df+0.5)/(df+0.5)) IDF.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Found by Ornith in the p5b/02 session (it measured the document); my test's fault, not the task's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check distinguishes linear time from catastrophic backtracking, which would never
finish on a 1 MB input, so a generous bound loses nothing. Reported by talos.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pdftext/outline: resolve named destinations through the catalog /Dests and the /Names /Dests
tree, resolve indirect /Dest, /A and /Title, and treat reader.Null (what Dict.Get returns for
a missing key) as absent. Before this every LaTeX and Acrobat PDF had an empty outline.
pdftest gains NamedDests; TestOutlineNamedDestinations.
- api: pool() always goes through its Once (data race on concurrent first use);
TestDefaultPoolConcurrentFirstUse. Revalidation of a stale cache entry counts against the
caller/host limit.
- pdftext/runner: output over MaxText kills the child at once instead of waiting for the
deadline; TestRunnerTooMuchTextFailsFast.
- fetch: Accept includes application/pdf;q=0.8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Digest systemPrompt gains a page-cite instruction and PromptVersion is 3.
Add the sift_pdf_* metric series (extract_total{outcome}, extract_seconds,
pages_total, pool_wait_seconds, cache_hits_total) to internal/metrics and wire
ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go. Refactor
cmd/sift/fetchcmd.go to a fetchOpts struct with parseFetchOpts and fetchBody,
adding --pages/--grep. Update skill/SKILL.md, set nix MemoryMax to 1G, and add a
PDF documents section to the README.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Answer extracted PDFs from the document cache: fresh for PDFTTL, then a
conditional GET (304 reuses the stored text, a 200 is extracted and stored
without a second GET). Store after extraction with the response hash, ETag,
fetch time and extracted pages. Route pages/grep through pdfdoc.Options with
400s for bad ranges and non-PDFs, and refuse a question when the document
exceeds PDFMaxQuestionChunks with a pointer to grep and pages.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Route "pdf" through clean.Kind in doFetch/cleanFetched: extract under the pool
(keyed by caller class + document hash, one per caller), map the five pdftext
errors plus a pool panic to 422, ErrBusy to 503+Retry-After 10, ErrCallerBusy to
429, and answer with the whole text or the map. Adds RetryAfter to fetch.Error
and the matching header. No cache, pages or grep — those are task 07.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Pure render function on a *pdftext.Doc with no I/O: the whole text with
page markers when it fits, otherwise a map (title, outline with page
numbers, leading pages and how to ask for more), a page range, or grep
results with page numbers. Every length is in runes.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Generic in-process scheduler for the expensive PDF-extraction child process.
A pool runs at most `slots` jobs, keeps at most `queue` waiting, and refuses a
waiter past `wait`. Do() applies six rules in order: join an existing running
or queued job (no slot, allowed while the caller holds another, fn may be nil);
refuse a caller that already has a job in flight; take a free slot, else queue
(first-come-first-served, else ErrBusy); run fn under context.WithoutCancel so
the flight outlives a caller that leaves; broadcast the result to every sharer
with no caching and a recovered panic returned as an error. Running counts only
jobs holding a slot; Queued counts the line; Waiting(key) counts joiners.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add pdf_doc/pdf_page tables plus PDFEntry, PDFKey, PutPDF (one transaction,
old pages deleted first), GetPDF (page-ordered, updates last_hit) and TouchPDF
(304 revalidation moves only fetched_at). PDF documents count toward Size() and
are evicted whole by oldest use; PDFH/PDFM counters added. evict() now breaks
last_hit ties by rowid so same-second inserts evict in insertion order.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The parser is pure Go but a goroutine cannot be killed, so extraction runs in
a child process: the binary re-executed as `sift pdftext`, reading the PDF on
stdin and writing JSON Lines on stdout, killed at a deadline by the parent.
pdftext.go holds the types, the five errors, Extract (reader.Open with
ErrWrongPassword -> ErrEncrypted, per-page recover, trailer /Info /Title, and
rule-3 UTF-16BE/Latin-1 text decode) and ChildMain (soft 768 MiB memory limit,
-no env read, no network, exit codes 0/3/4/1). outline.go walks /Outlines ->
/First -> /Next with /First children at level+1, resolving /Dest and /GoTo /A
dests to 1-based pages, bounded at 10,000 items and never visiting an object
twice. runner.go is the Runner: env-isolated child (nil env is an empty
environment), 60 s deadline, 64 MiB output cap, 200 MiB stdin cap, stderr
capped at 4 KiB, exit-code to error mapping, and Doc parsing with ErrNoText.
cmd/sift/main.go adds the unlisted `pdftext` case. tools.go is removed; the
go-pdfkit modules stay vendored.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Recognise PDFs in clean.Kind (declared types and %PDF- sniffing on
octet-stream/bodyless bodies), give PDF responses the 100 MB body cap and
120 s deadline while web pages keep 5 MB / 20 s, and add FetchIf, which
sends If-None-Match / If-Modified-Since and maps a 304 to a Resource with
Status 304 and an empty Body. Fetch and FetchIf share one doFetch path for
guard, client, limits and error mapping.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables,
04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI,
skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the
intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted
fixtures.
Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via
tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for
the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard;
TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and
writeTimeout 120→240 s for the PDF path's worst case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Default GET /metrics is Prometheus text; JSON stays at /metrics.json
and ?format=json. :8443 needs the metrics caller bearer; loopback
stays open. Restricted callers have no per-caller series and no
tavily:restricted gauge.
Live llm/context hits live in grounding.generic plus a URL-keyed
sources map, not a sources array. Fan-in uses reciprocal-rank fusion
instead of cosine. no_rerank keeps that source's own order as a prefix.
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
CheckMappings now logs WARN for each permit_restricted name missing
from keys.toml, so a missing straylight-gs key is visible at start
and on reload instead of passing silently.
kyleandornith-1.5-35b-a3b
