Native include/exclude domains, time_range, topic, exact_match, min_score,
and up to 4 queries fused with RRF. Honesty notes stay off the Warn/fail
path. Tavily keeps include_answer false and sets auto_parameters false.
Cache key is canonical JSON. CLI flags and skill recipe included.
Owner review of p6 (Ornith tasks 01-09), with an independent reviewer pass whose findings I
confirmed against the code. Every fix has a regression test that fails on the old code.
internal/batch
- Close let the workers keep draining every live job under the cancelled context, failing
every target; they now stop at once, store nothing on a failure during shutdown, and leave
the job live for MarkInFlightFailed("restart").
- A job cancelled while its in-flight target came back busy stayed in the rotation forever
(depth gauge stuck, job held in memory); it now leaves. The task 03 design had this gap: mine.
- Submit's idempotency lookup, caps and create ran unlocked: 20 concurrent submits by one
caller made 8 jobs. Serialised with submitMu.
- A failed target insert left a queued job no worker would run (holding the caller's cap);
it is marked failed.
- A second concurrent cancel returned true and counted twice.
- Config: zero OR negative values take the defaults (negative Workers panicked the
WaitGroup; a negative TTL or byte budget would sweep or evict everything).
- Retry timing used the pre-fetch time (a 5 s busy answer made RetryWait moot).
- The rotation cursor is kept within [0, len(live)] (it was absolute and drifted on removal).
- The unused in-memory callers map is gone (Submit racing Close wrote to a nil map).
internal/dcache
- PutBatchTargets, SweepBatch and eviction are transactional; the sweep never removes a
queued or running job.
internal/api
- Every /v1/batch endpoint answers 503 when batch is off; POST panicked on the nil runner.
- GET /{id}/{idx} bounds idx by the rows actually returned.
- batchFetch copies a *fetch.Error instead of mutating one the PDF pool may share.
internal/config
- ParseSize no longer overflows to a negative size.
cmd/sift
- sift batch treated 202 Accepted (submit) as an error and exited 1 with no message.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `sift batch` so agents reach the batch endpoint through the CLI like
`fetch` and `search`. `batchcmd.go` has `parseBatchArgs` (verb plus the
unchanged args after it; submit takes one arg, status a job id optionally
preceded by `--results`, get a job id plus a non-negative-integer idx, cancel
a job id, else a usage error), `batchMain` dispatch, one small function per
verb, and `batchDo` (bearer header, decode the envelope, print it on stdout,
on non-2xx the error to stderr and `exitFor`). `submitMain` reads the file or
stdin and validates locally: a JSON object with a non-empty `targets` array,
or a bare array of targets or URL strings wrapped into `{"targets": [...]}`.
README gains a "Batch" section (endpoints, serial-per-job/one-worker/
round-robin model, caps, idempotency, lease/TTL/restart, summary-vs-results,
redaction, metrics); SKILL.md gains a "Batch" section.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Add the [batch] config block (Batch type + File.Batch) and ParseSize (binary
KiB/MiB/GiB byte counts) in internal/config. serve now opens the dcache store
when batch is on, marks the queued/running jobs a restart killed as failed
(never resumed) and observes them, builds the runner from the config with the
runner's defaults filling zeros, and closes the runner on both shutdown paths.
Batch is on by default; disabled = true leaves srv.Batch nil so the endpoints
answer 503.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Owner review of p5c (tasks 01-02 by Ornith):
- extract.go: an upload (up to 100 MiB) is read whole before the PDF pool's limits apply, so
any number of concurrent uploads could buffer at once in a 1 GiB service. At most 2 bodies
are read at a time now; a third waits up to 5 s, then 429 busy with Retry-After 10.
- extract.go: `warnings` was null when empty; it is [] as in /v1/fetch.
- extractcmd.go: extractURL replaced the client base URL's path; it now appends /v1/extract
like fetch and search do.
Regression tests in extract_review_test.go (api and cmd/sift).
End-to-end (local serve -> Dixie; sift extract with SIFT_CLIENT): Llama 2 paper, 13.6 MB /
77 pages: --pages 1 ok, repeat served from the content-hash cache, --grep 2 hits, --question
answered with page citations in 16.8 s. Log lines carry bytes/pages/mode/cached only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`sift extract <file.pdf>` stats the file (refusing over 100 MiB before the
upload), streams it as the POST body with Content-Type application/pdf and the
ContentLength set, and prints the /v1/extract answer exactly as `sift fetch`
prints a PDF URL — same content output, same errors and exitFor codes.
parseExtractOpts reuses splitAroundPositional; extractURL sends only the set
options with mode=digest when a question is given. main.go gains the case and a
help line; README and skill document the command.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Digest systemPrompt gains a page-cite instruction and PromptVersion is 3.
Add the sift_pdf_* metric series (extract_total{outcome}, extract_seconds,
pages_total, pool_wait_seconds, cache_hits_total) to internal/metrics and wire
ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go. Refactor
cmd/sift/fetchcmd.go to a fetchOpts struct with parseFetchOpts and fetchBody,
adding --pages/--grep. Update skill/SKILL.md, set nix MemoryMax to 1G, and add a
PDF documents section to the README.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
The parser is pure Go but a goroutine cannot be killed, so extraction runs in
a child process: the binary re-executed as `sift pdftext`, reading the PDF on
stdin and writing JSON Lines on stdout, killed at a deadline by the parent.
pdftext.go holds the types, the five errors, Extract (reader.Open with
ErrWrongPassword -> ErrEncrypted, per-page recover, trailer /Info /Title, and
rule-3 UTF-16BE/Latin-1 text decode) and ChildMain (soft 768 MiB memory limit,
-no env read, no network, exit codes 0/3/4/1). outline.go walks /Outlines ->
/First -> /Next with /First children at level+1, resolving /Dest and /GoTo /A
dests to 1-based pages, bounded at 10,000 items and never visiting an object
twice. runner.go is the Runner: env-isolated child (nil env is an empty
environment), 60 s deadline, 64 MiB output cap, 200 MiB stdin cap, stderr
capped at 4 KiB, exit-code to error mapping, and Doc parsing with ErrNoText.
cmd/sift/main.go adds the unlisted `pdftext` case. tools.go is removed; the
go-pdfkit modules stay vendored.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Default GET /metrics is Prometheus text; JSON stays at /metrics.json
and ?format=json. :8443 needs the metrics caller bearer; loopback
stays open. Restricted callers have no per-caller series and no
tavily:restricted gauge.
A restricted profile shares the Unix user with the fleet. If its
$HERMES_HOME/sift/client.toml is missing, returning ~/.config/sift
would silently use the fleet key. Non-default HERMES_HOME now pins
the profile path so loadClient errors instead.
POST /v1/search and /v2/search fan in SearXNG, Brave LLM Context, and
Tavily from config-fixed endpoints. Public sources refuse private A
records; SearXNG is a loopback-only client, not the fetch guard.
Cache keys include caller class. Restricted callers search like the
fleet; logs omit query, hosts, URLs, and upstream errors. Degraded
results are not stored. Tavily sends search_depth=basic and drops
answer. Missing keys surface in /healthz.
Default-on sources stay off until the bake-off. Query harvest is not
in this commit.
Digest, chunk-embed, and question-embed keys include fleet|restricted so
a fleet caller cannot observe a restricted cache hit. Digest.Run takes
the class from permit_restricted.
A permit_restricted caller whose keys.toml workspace is not
guardian-star is refused at Search (no Honcho request) and flagged at
serve start and keys reload.
docs: P3 restricted callers search like the fleet; cache/logs isolate.
Scrape, fetch, refused, fetch-failed, and internal lines redact the
host for callers in context.permit_restricted. final_url stays in the
response. Test captures slog records, not the journal.
guardian-star context is allowed only when the authenticated caller is in
permit_restricted and mapped to that workspace. Fleet callers remapped to it
still refuse. CLI client.toml: SIFT_CLIENT, then $HERMES_HOME/sift, then ~/.config.
Embed requests send dixie.embed_model (default honcho-embed).
Wayback accepts http://web.archive.org, rewrites to https, skips
non-200 closest.status. Memento-Datetime is parsed as an HTTP date
and future/garbage values become empty snapshot_at. keys add writes
via temp+fchown/chmod/rename so root does not clobber keys.toml.
Digest fallback is capped at the 6k-token budget.
WriteTimeout 30s was cutting digest mid-flight (queue+fetch+digest)
and tailscale then 502'd. 120s write, 90s handler deadline.
sift fetch URL --question now parses; extra positionals error.
Dixie is capped (max_inflight 2, queue_depth 8, queue_wait 20s) and
never queued on llama-server. Streaming chat, thinking off, 45s
fallback to clean. Digest and chunk-embedding caches are
content-addressed SQLite (Honcho context is not cached). Recover
uses the guarded Fetcher. guardian-star is refused in code.
Discover WAN via ddns:<host> then trace:<url> before listen. Persist to
self_ip.json. Previous addresses stay denied for keep_previous. Failure
keeps the last known set. /healthz reports age and source.
Single Fetcher.Fetch on the Guard that holds DenyPrefixes. skipTlsVerification
is ignored. No 502. keys add writes the secret only to --out. Release binary
has no -test-allow-prefix.
Block 6to4/Teredo/NAT64-local/site-local/benchmarking/reserved
ranges from the IANA registries (families, not decoded embeds).
Guard.DenyPrefixes / -deny-prefix covers hairpin WAN /32s; home
addresses stay on-box, not in git. Control is tcp/tcp4/tcp6 only.
Go service with /healthz, SSRF guard (parse + resolve-all + Control),
exhaustive tests, NixOS module, and Makefile as source of truth for
build flags. nix flake check runs make test lint.
kyle
