The parser is pure Go but a goroutine cannot be killed, so extraction runs in
a child process: the binary re-executed as `sift pdftext`, reading the PDF on
stdin and writing JSON Lines on stdout, killed at a deadline by the parent.
pdftext.go holds the types, the five errors, Extract (reader.Open with
ErrWrongPassword -> ErrEncrypted, per-page recover, trailer /Info /Title, and
rule-3 UTF-16BE/Latin-1 text decode) and ChildMain (soft 768 MiB memory limit,
-no env read, no network, exit codes 0/3/4/1). outline.go walks /Outlines ->
/First -> /Next with /First children at level+1, resolving /Dest and /GoTo /A
dests to 1-based pages, bounded at 10,000 items and never visiting an object
twice. runner.go is the Runner: env-isolated child (nil env is an empty
environment), 60 s deadline, 64 MiB output cap, 200 MiB stdin cap, stderr
capped at 4 KiB, exit-code to error mapping, and Doc parsing with ErrNoText.
cmd/sift/main.go adds the unlisted `pdftext` case. tools.go is removed; the
go-pdfkit modules stay vendored.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
Recognise PDFs in clean.Kind (declared types and %PDF- sniffing on
octet-stream/bodyless bodies), give PDF responses the 100 MB body cap and
120 s deadline while web pages keep 5 MB / 20 s, and add FetchIf, which
sends If-None-Match / If-Modified-Since and maps a 304 to a Resource with
Status 304 and an empty Body. Fetch and FetchIf share one doFetch path for
guard, client, limits and error mapping.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables,
04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI,
skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the
intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted
fixtures.
Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via
tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for
the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard;
TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and
writeTimeout 120→240 s for the PDF path's worst case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Default GET /metrics is Prometheus text; JSON stays at /metrics.json
and ?format=json. :8443 needs the metrics caller bearer; loopback
stays open. Restricted callers have no per-caller series and no
tavily:restricted gauge.
Live llm/context hits live in grounding.generic plus a URL-keyed
sources map, not a sources array. Fan-in uses reciprocal-rank fusion
instead of cosine. no_rerank keeps that source's own order as a prefix.
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
CheckMappings now logs WARN for each permit_restricted name missing
from keys.toml, so a missing straylight-gs key is visible at start
and on reload instead of passing silently.
POST /v1/search and /v2/search fan in SearXNG, Brave LLM Context, and
Tavily from config-fixed endpoints. Public sources refuse private A
records; SearXNG is a loopback-only client, not the fetch guard.
Cache keys include caller class. Restricted callers search like the
fleet; logs omit query, hosts, URLs, and upstream errors. Degraded
results are not stored. Tavily sends search_depth=basic and drops
answer. Missing keys surface in /healthz.
Default-on sources stay off until the bake-off. Query harvest is not
in this commit.
Digest, chunk-embed, and question-embed keys include fleet|restricted so
a fleet caller cannot observe a restricted cache hit. Digest.Run takes
the class from permit_restricted.
A permit_restricted caller whose keys.toml workspace is not
guardian-star is refused at Search (no Honcho request) and flagged at
serve start and keys reload.
docs: P3 restricted callers search like the fleet; cache/logs isolate.
Guard messages embed the hostname (resolve <host>, no addresses for
<host>). Restricted callers now log a coarse reason class and err=-.
Test dumps slog records and forbids the hostname anywhere in them.
docs: P3 review — fail-closed search, cache class, public dialer
Scrape, fetch, refused, fetch-failed, and internal lines redact the
host for callers in context.permit_restricted. final_url stays in the
response. Test captures slog records, not the journal.
guardian-star context is allowed only when the authenticated caller is in
permit_restricted and mapped to that workspace. Fleet callers remapped to it
still refuse. CLI client.toml: SIFT_CLIENT, then $HERMES_HOME/sift, then ~/.config.
kyleandornith-1.5-35b-a3b
