Owner review of p5b (tasks 01-03 by Ornith):
- background.go: `seen` was never cleared, so a document whose background embedding failed
(Dixie hiccup) or was abandoned could never be queued again: every later question on it stayed
BM25 for the life of the process. Dedupe now covers only queued and in-progress documents.
- background.go: acquireGate gave up after one retry and dropped the document; the task said to
keep retrying once a second. It now waits (background work has no deadline).
- retrieval.go/background.go: a vector the server left out is not cached as an empty vector.
- bm25.go: Scores on a nil *BM25 dereferenced it; returns nil.
Regression tests in review_test.go; both fail on the old code.
End-to-end against Dixie (local serve): Karpathy's thesis, 95 pages / 51 chunks: bm25 in 19 s,
hybrid (51 embedded) after the background; RP2040 datasheet, 642 pages / 284 chunks (a 422
before P5b): bm25 in 13.5 s, all 284 embedded within 90 s, hybrid answer correct with its quote.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The digest now handles long documents itself (task 02), so the API's refusal
limit is only a safety cap: DefaultPDFMaxQuestionChunks 200 -> 5000 (the 422
message text is unchanged). In v1fetch the digest-meta object gains "retrieval"
and "embedded" next to chunks_used/chunks_total, only when the digest ran
without a fallback. The PDF section explains a long-doc question is answered
first from BM25 while its chunks embed in the background and hybrid on later
questions; skill/SKILL.md gains one sentence. Given tests byte-identical.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Cached chunk embeddings change what a long document answers with: BM25 first,
embedding uncached chunks (or the whole document) in the background so the next
question on the same doc scores hybrid. selectText now returns its mode and the
number of chunks that had a vector at answer-build time; the digest cache stores
only hybrid answers, never a provisional bm25 one. The background worker batches
through a Dixie gate and WaitBackground polls for true idle.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
SplitPages is Split's budget/line splitting plus page awareness: a "--- page N ---"
line sets the current page and a chunk that starts after the first marker but not on
a marker line is prefixed "--- page N (cont.) ---\n\n" for the page its first line
belongs to, so every chunk names its page. BM25 (k1=1.2, b=0.75) ranks chunks
lexically with counted term frequencies and ln(1+(N-df+0.5)/(df+0.5)) IDF.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Found by Ornith in the p5b/02 session (it measured the document); my test's fault, not the task's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check distinguishes linear time from catastrophic backtracking, which would never
finish on a 1 MB input, so a generous bound loses nothing. Reported by talos.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README gains API, CLI and configuration sections and the PDF range/grep/question limits; the skill covers PDF busy and failure results; dist/sift.service matches the Nix unit (MemoryMax=1G, secret.env); the example config drops the unused [upstreams] block; stale task comments in api/pdf.go.
Gate overridden by the owner: TestGrepLimits (given test) exceeds its 1 s budget under -race on this host.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- pdftext/outline: resolve named destinations through the catalog /Dests and the /Names /Dests
tree, resolve indirect /Dest, /A and /Title, and treat reader.Null (what Dict.Get returns for
a missing key) as absent. Before this every LaTeX and Acrobat PDF had an empty outline.
pdftest gains NamedDests; TestOutlineNamedDestinations.
- api: pool() always goes through its Once (data race on concurrent first use);
TestDefaultPoolConcurrentFirstUse. Revalidation of a stale cache entry counts against the
caller/host limit.
- pdftext/runner: output over MaxText kills the child at once instead of waiting for the
deadline; TestRunnerTooMuchTextFailsFast.
- fetch: Accept includes application/pdf;q=0.8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Digest systemPrompt gains a page-cite instruction and PromptVersion is 3.
Add the sift_pdf_* metric series (extract_total{outcome}, extract_seconds,
pages_total, pool_wait_seconds, cache_hits_total) to internal/metrics and wire
ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go. Refactor
cmd/sift/fetchcmd.go to a fetchOpts struct with parseFetchOpts and fetchBody,
adding --pages/--grep. Update skill/SKILL.md, set nix MemoryMax to 1G, and add a
PDF documents section to the README.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Answer extracted PDFs from the document cache: fresh for PDFTTL, then a
conditional GET (304 reuses the stored text, a 200 is extracted and stored
without a second GET). Store after extraction with the response hash, ETag,
fetch time and extracted pages. Route pages/grep through pdfdoc.Options with
400s for bad ranges and non-PDFs, and refuse a question when the document
exceeds PDFMaxQuestionChunks with a pointer to grep and pages.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Route "pdf" through clean.Kind in doFetch/cleanFetched: extract under the pool
(keyed by caller class + document hash, one per caller), map the five pdftext
errors plus a pool panic to 422, ErrBusy to 503+Retry-After 10, ErrCallerBusy to
429, and answer with the whole text or the map. Adds RetryAfter to fetch.Error
and the matching header. No cache, pages or grep — those are task 07.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Pure render function on a *pdftext.Doc with no I/O: the whole text with
page markers when it fits, otherwise a map (title, outline with page
numbers, leading pages and how to ask for more), a page range, or grep
results with page numbers. Every length is in runes.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Generic in-process scheduler for the expensive PDF-extraction child process.
A pool runs at most `slots` jobs, keeps at most `queue` waiting, and refuses a
waiter past `wait`. Do() applies six rules in order: join an existing running
or queued job (no slot, allowed while the caller holds another, fn may be nil);
refuse a caller that already has a job in flight; take a free slot, else queue
(first-come-first-served, else ErrBusy); run fn under context.WithoutCancel so
the flight outlives a caller that leaves; broadcast the result to every sharer
with no caching and a recovered panic returned as an error. Running counts only
jobs holding a slot; Queued counts the line; Waiting(key) counts joiners.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add pdf_doc/pdf_page tables plus PDFEntry, PDFKey, PutPDF (one transaction,
old pages deleted first), GetPDF (page-ordered, updates last_hit) and TouchPDF
(304 revalidation moves only fetched_at). PDF documents count toward Size() and
are evicted whole by oldest use; PDFH/PDFM counters added. evict() now breaks
last_hit ties by rowid so same-second inserts evict in insertion order.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The parser is pure Go but a goroutine cannot be killed, so extraction runs in
a child process: the binary re-executed as `sift pdftext`, reading the PDF on
stdin and writing JSON Lines on stdout, killed at a deadline by the parent.
pdftext.go holds the types, the five errors, Extract (reader.Open with
ErrWrongPassword -> ErrEncrypted, per-page recover, trailer /Info /Title, and
rule-3 UTF-16BE/Latin-1 text decode) and ChildMain (soft 768 MiB memory limit,
-no env read, no network, exit codes 0/3/4/1). outline.go walks /Outlines ->
/First -> /Next with /First children at level+1, resolving /Dest and /GoTo /A
dests to 1-based pages, bounded at 10,000 items and never visiting an object
twice. runner.go is the Runner: env-isolated child (nil env is an empty
environment), 60 s deadline, 64 MiB output cap, 200 MiB stdin cap, stderr
capped at 4 KiB, exit-code to error mapping, and Doc parsing with ErrNoText.
cmd/sift/main.go adds the unlisted `pdftext` case. tools.go is removed; the
go-pdfkit modules stay vendored.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Recognise PDFs in clean.Kind (declared types and %PDF- sniffing on
octet-stream/bodyless bodies), give PDF responses the 100 MB body cap and
120 s deadline while web pages keep 5 MB / 20 s, and add FetchIf, which
sends If-None-Match / If-Modified-Since and maps a 304 to a Resource with
Status 304 and an empty Body. Fetch and FetchIf share one doFetch path for
guard, client, limits and error mapping.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables,
04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI,
skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the
intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted
fixtures.
Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via
tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for
the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard;
TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and
writeTimeout 120→240 s for the PDF path's worst case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Default GET /metrics is Prometheus text; JSON stays at /metrics.json
and ?format=json. :8443 needs the metrics caller bearer; loopback
stays open. Restricted callers have no per-caller series and no
tavily:restricted gauge.
Live llm/context hits live in grounding.generic plus a URL-keyed
sources map, not a sources array. Fan-in uses reciprocal-rank fusion
instead of cosine. no_rerank keeps that source's own order as a prefix.
Exempt title from the vendor-token check. Truncate snippets to 160
chars and clip all four packs to the same per-query k. run.py fails
cached replies with a rerun message instead of a config mismatch.
run.py records one named config at a time. Sift dumps fail if
sizes.sources_ok does not match --config. Hermes Tavily uses the
live-tool request shape, not sift. Raw dumps stay gitignored.
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
A restricted profile shares the Unix user with the fleet. If its
$HERMES_HOME/sift/client.toml is missing, returning ~/.config/sift
would silently use the fleet key. Non-default HERMES_HOME now pins
the profile path so loadClient errors instead.
A 200 with a bad Tavily body is tavily: error, not an empty success
that would cache for the hour. Upstream clients stay in the source
list if DNS is down at boot and retry the pin on Search; /healthz
lists them as unresolved until a pin lands.
CheckMappings now logs WARN for each permit_restricted name missing
from keys.toml, so a missing straylight-gs key is visible at start
and on reload instead of passing silently.
POST /v1/search and /v2/search fan in SearXNG, Brave LLM Context, and
Tavily from config-fixed endpoints. Public sources refuse private A
records; SearXNG is a loopback-only client, not the fetch guard.
Cache keys include caller class. Restricted callers search like the
fleet; logs omit query, hosts, URLs, and upstream errors. Degraded
results are not stored. Tavily sends search_depth=basic and drops
answer. Missing keys surface in /healthz.
Default-on sources stay off until the bake-off. Query harvest is not
in this commit.
Digest, chunk-embed, and question-embed keys include fleet|restricted so
a fleet caller cannot observe a restricted cache hit. Digest.Run takes
the class from permit_restricted.
A permit_restricted caller whose keys.toml workspace is not
guardian-star is refused at Search (no Honcho request) and flagged at
serve start and keys reload.
docs: P3 restricted callers search like the fleet; cache/logs isolate.
Guard messages embed the hostname (resolve <host>, no addresses for
<host>). Restricted callers now log a coarse reason class and err=-.
Test dumps slog records and forbids the hostname anywhere in them.
docs: P3 review — fail-closed search, cache class, public dialer
Scrape, fetch, refused, fetch-failed, and internal lines redact the
host for callers in context.permit_restricted. final_url stays in the
response. Test captures slog records, not the journal.
guardian-star context is allowed only when the authenticated caller is in
permit_restricted and mapped to that workspace. Fleet callers remapped to it
still refuse. CLI client.toml: SIFT_CLIENT, then $HERMES_HOME/sift, then ~/.config.
sift search is P3 and not built — keep web_search until then.
Recover ladder is Wayback then archive.today. digest.fallback
means clean text capped at ~6k tokens.
Question-mode selection always keeps the lead chunk, then fills
the budget by 0.7 cosine + 0.3 term overlap. prompt_version 2.
too-large embed retry no longer halves the question (index 0).
Link-heavy chunks estimated under EmbedMaxTok still overflow
Dixie's 2048-token embed slot. Cap every embed input at 3000
runes. A 500 whose body contains 'too large' halves inputs and
retries once. Fallback logs the truncated upstream body.
Embed requests send dixie.embed_model (default honcho-embed).
Wayback accepts http://web.archive.org, rewrites to https, skips
non-200 closest.status. Memento-Datetime is parsed as an HTTP date
and future/garbage values become empty snapshot_at. keys add writes
via temp+fchown/chmod/rename so root does not clobber keys.toml.
Digest fallback is capped at the 6k-token budget.
WriteTimeout 30s was cutting digest mid-flight (queue+fetch+digest)
and tailscale then 502'd. 120s write, 90s handler deadline.
sift fetch URL --question now parses; extra positionals error.
Neutralise </document and <document in page text. Counters are
atomic.Uint64. Workspace names are [A-Za-z0-9_-] and PathEscaped;
foreign workspace_id rows are dropped. Wayback closest.url must be
https://web.archive.org. archive.today snapshot_at is Memento-Datetime
or null.
Dixie is capped (max_inflight 2, queue_depth 8, queue_wait 20s) and
never queued on llama-server. Streaming chat, thinking off, 45s
fallback to clean. Digest and chunk-embedding caches are
content-addressed SQLite (Honcho context is not cached). Recover
uses the guarded Fetcher. guardian-star is refused in code.
Discover WAN via ddns:<host> then trace:<url> before listen. Persist to
self_ip.json. Previous addresses stay denied for keep_previous. Failure
keeps the last known set. /healthz reports age and source.
jq results are bounded (10k items / output cap) and return 400 when
exceeded. 401/403 log caller, host, and reason. make contract exits 1
unless firecrawl-py is present (SIFT_CONTRACT_SKIP_SDK=1 to skip).
Single Fetcher.Fetch on the Guard that holds DenyPrefixes. skipTlsVerification
is ignored. No 502. keys add writes the secret only to --out. Release binary
has no -test-allow-prefix.
Block 6to4/Teredo/NAT64-local/site-local/benchmarking/reserved
ranges from the IANA registries (families, not decoded embeds).
Guard.DenyPrefixes / -deny-prefix covers hairpin WAN /32s; home
addresses stay on-box, not in git. Control is tcp/tcp4/tcp6 only.
Go service with /healthz, SSRF guard (parse + resolve-all + Control),
exhaustive tests, NixOS module, and Makefile as source of truth for
build flags. nix flake check runs make test lint.
kyleandClaude Opus 5.5
