docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables,
04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI,
skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the
intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted
fixtures.
Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via
tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for
the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard;
TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and
writeTimeout 120→240 s for the PDF path's worst case.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Default GET /metrics is Prometheus text; JSON stays at /metrics.json
and ?format=json. :8443 needs the metrics caller bearer; loopback
stays open. Restricted callers have no per-caller series and no
tavily:restricted gauge.
Live llm/context hits live in grounding.generic plus a URL-keyed
sources map, not a sources array. Fan-in uses reciprocal-rank fusion
instead of cosine. no_rerank keeps that source's own order as a prefix.
Exempt title from the vendor-token check. Truncate snippets to 160
chars and clip all four packs to the same per-query k. run.py fails
cached replies with a rerun message instead of a config mismatch.
run.py records one named config at a time. Sift dumps fail if
sizes.sources_ok does not match --config. Hermes Tavily uses the
live-tool request shape, not sift. Raw dumps stay gitignored.
Rewrite A-D to [{rank, url, title, snippet}] only. Fail closed if
searxng/brave/tavily/hermes appear outside url or snippet. Self-test
and a 28-query fake-dump dry run pass.
A restricted profile shares the Unix user with the fleet. If its
$HERMES_HOME/sift/client.toml is missing, returning ~/.config/sift
would silently use the fleet key. Non-default HERMES_HOME now pins
the profile path so loadClient errors instead.
A 200 with a bad Tavily body is tavily: error, not an empty success
that would cache for the hour. Upstream clients stay in the source
list if DNS is down at boot and retry the pin on Search; /healthz
lists them as unresolved until a pin lands.
CheckMappings now logs WARN for each permit_restricted name missing
from keys.toml, so a missing straylight-gs key is visible at start
and on reload instead of passing silently.
POST /v1/search and /v2/search fan in SearXNG, Brave LLM Context, and
Tavily from config-fixed endpoints. Public sources refuse private A
records; SearXNG is a loopback-only client, not the fetch guard.
Cache keys include caller class. Restricted callers search like the
fleet; logs omit query, hosts, URLs, and upstream errors. Degraded
results are not stored. Tavily sends search_depth=basic and drops
answer. Missing keys surface in /healthz.
Default-on sources stay off until the bake-off. Query harvest is not
in this commit.
Digest, chunk-embed, and question-embed keys include fleet|restricted so
a fleet caller cannot observe a restricted cache hit. Digest.Run takes
the class from permit_restricted.
A permit_restricted caller whose keys.toml workspace is not
guardian-star is refused at Search (no Honcho request) and flagged at
serve start and keys reload.
docs: P3 restricted callers search like the fleet; cache/logs isolate.
Guard messages embed the hostname (resolve <host>, no addresses for
<host>). Restricted callers now log a coarse reason class and err=-.
Test dumps slog records and forbids the hostname anywhere in them.
docs: P3 review — fail-closed search, cache class, public dialer
Scrape, fetch, refused, fetch-failed, and internal lines redact the
host for callers in context.permit_restricted. final_url stays in the
response. Test captures slog records, not the journal.
guardian-star context is allowed only when the authenticated caller is in
permit_restricted and mapped to that workspace. Fleet callers remapped to it
still refuse. CLI client.toml: SIFT_CLIENT, then $HERMES_HOME/sift, then ~/.config.
sift search is P3 and not built — keep web_search until then.
Recover ladder is Wayback then archive.today. digest.fallback
means clean text capped at ~6k tokens.
Question-mode selection always keeps the lead chunk, then fills
the budget by 0.7 cosine + 0.3 term overlap. prompt_version 2.
too-large embed retry no longer halves the question (index 0).
Link-heavy chunks estimated under EmbedMaxTok still overflow
Dixie's 2048-token embed slot. Cap every embed input at 3000
runes. A 500 whose body contains 'too large' halves inputs and
retries once. Fallback logs the truncated upstream body.
Embed requests send dixie.embed_model (default honcho-embed).
Wayback accepts http://web.archive.org, rewrites to https, skips
non-200 closest.status. Memento-Datetime is parsed as an HTTP date
and future/garbage values become empty snapshot_at. keys add writes
via temp+fchown/chmod/rename so root does not clobber keys.toml.
Digest fallback is capped at the 6k-token budget.
WriteTimeout 30s was cutting digest mid-flight (queue+fetch+digest)
and tailscale then 502'd. 120s write, 90s handler deadline.
sift fetch URL --question now parses; extra positionals error.
Neutralise </document and <document in page text. Counters are
atomic.Uint64. Workspace names are [A-Za-z0-9_-] and PathEscaped;
foreign workspace_id rows are dropped. Wayback closest.url must be
https://web.archive.org. archive.today snapshot_at is Memento-Datetime
or null.
Dixie is capped (max_inflight 2, queue_depth 8, queue_wait 20s) and
never queued on llama-server. Streaming chat, thinking off, 45s
fallback to clean. Digest and chunk-embedding caches are
content-addressed SQLite (Honcho context is not cached). Recover
uses the guarded Fetcher. guardian-star is refused in code.
Discover WAN via ddns:<host> then trace:<url> before listen. Persist to
self_ip.json. Previous addresses stay denied for keep_previous. Failure
keeps the last known set. /healthz reports age and source.
jq results are bounded (10k items / output cap) and return 400 when
exceeded. 401/403 log caller, host, and reason. make contract exits 1
unless firecrawl-py is present (SIFT_CONTRACT_SKIP_SDK=1 to skip).
Single Fetcher.Fetch on the Guard that holds DenyPrefixes. skipTlsVerification
is ignored. No 502. keys add writes the secret only to --out. Release binary
has no -test-allow-prefix.
Block 6to4/Teredo/NAT64-local/site-local/benchmarking/reserved
ranges from the IANA registries (families, not decoded embeds).
Guard.DenyPrefixes / -deny-prefix covers hairpin WAN /32s; home
addresses stay on-box, not in git. Control is tcp/tcp4/tcp6 only.
Go service with /healthz, SSRF guard (parse + resolve-all + Control),
exhaustive tests, NixOS module, and Makefile as source of truth for
build flags. nix flake check runs make test lint.
kyleandClaude Opus 5.5