Commit Graph
28 Commits
Author SHA1 Message Date
kyle 85113f9c0b p2.5: log host=- for permit_restricted callers
Scrape, fetch, refused, fetch-failed, and internal lines redact the
host for callers in context.permit_restricted. final_url stays in the
response. Test captures slog records, not the journal.
2026-09-23 11:40:45 -07:00
kyle e36da19f57 p2.4: restricted workspace for permitted callers
guardian-star context is allowed only when the authenticated caller is in
permit_restricted and mapped to that workspace. Fleet callers remapped to it
still refuse. CLI client.toml: SIFT_CLIENT, then $HERMES_HOME/sift, then ~/.config.
2026-09-23 08:33:58 -07:00
kyle 14eb7808ce docs: skill matches shipped P2 (no search, recover, fallback cap)
sift search is P3 and not built — keep web_search until then.
Recover ladder is Wayback then archive.today. digest.fallback
means clean text capped at ~6k tokens.
2026-09-23 06:00:50 -07:00
kyle 767c12c1af p2.3: pin chunk 0, hybrid score, keep question on retry
Question-mode selection always keeps the lead chunk, then fills
the budget by 0.7 cosine + 0.3 term overlap. prompt_version 2.
too-large embed retry no longer halves the question (index 0).
2026-09-23 03:18:29 -07:00
kyle 5dbc7e4094 p2.2: cap embed inputs at 3000 runes, retry too-large 500
Link-heavy chunks estimated under EmbedMaxTok still overflow
Dixie's 2048-token embed slot. Cap every embed input at 3000
runes. A 500 whose body contains 'too large' halves inputs and
retries once. Fallback logs the truncated upstream body.
2026-09-23 03:03:11 -07:00
kyle 16a7d678d3 p2.1: honcho-embed, wayback http/204, memento, keys owner
Embed requests send dixie.embed_model (default honcho-embed).
Wayback accepts http://web.archive.org, rewrites to https, skips
non-200 closest.status. Memento-Datetime is parsed as an HTTP date
and future/garbage values become empty snapshot_at. keys add writes
via temp+fchown/chmod/rename so root does not clobber keys.toml.
Digest fallback is capped at the 6k-token budget.
2026-09-23 02:41:46 -07:00
kyle 2db001e0d7 p2: 120s write timeout and flags after the URL
WriteTimeout 30s was cutting digest mid-flight (queue+fetch+digest)
and tailscale then 502'd. 120s write, 90s handler deadline.
sift fetch URL --question now parses; extra positionals error.
2026-09-23 00:28:59 -07:00
kyle a992216ec0 p2: widen document fence for space and zero-width
Strip U+200B–U+200D, U+2060, U+FEFF before fencing. Match
(?i)<\s*/?\s*document so </ document> cannot sit outside the frame.
2026-09-23 00:09:14 -07:00
kyle ff87a04f94 p2: fence document tags, atomic fleetctx, harden workspace
Neutralise </document and <document in page text. Counters are
atomic.Uint64. Workspace names are [A-Za-z0-9_-] and PathEscaped;
foreign workspace_id rows are dropped. Wayback closest.url must be
https://web.archive.org. archive.today snapshot_at is Memento-Datetime
or null.
2026-09-23 00:01:04 -07:00
kyle 2f9d3d97e6 p2: digest with in-sift FIFO, SQLite caches, Honcho context, recover
Dixie is capped (max_inflight 2, queue_depth 8, queue_wait 20s) and
never queued on llama-server. Streaming chat, thinking off, 45s
fallback to clean. Digest and chunk-embedding caches are
content-addressed SQLite (Honcho context is not cached). Recover
uses the guarded Fetcher. guardian-star is refused in code.
2026-09-22 23:54:49 -07:00
kyleandClaude Opus 5.5 505e113b64 docs: P2 scope line matches §2a caches
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 23:47:24 -07:00
kyleandClaude Opus 5.5 9f8d9f4f13 docs: P2 — bounded in-sift queue; digest + chunk-embedding caches pulled forward
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 23:47:09 -07:00
kyleandClaude Opus 5.5 cb0cb83fb9 docs: P2 spec — digest (bounded, streaming, thinking off), Honcho context (guardian-star refused), recover ladder
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 23:43:40 -07:00
kyle 0598c8985b p1.1: note IPv4-only self-IP assumption
Trace via 1.1.1.1 never learns a delegated v6 prefix.
2026-09-22 23:01:19 -07:00
kyle 447842f860 p1.1: dynamic self-IP deny (DDNS/trace, keep previous, atomic swap)
Discover WAN via ddns:<host> then trace:<url> before listen. Persist to
self_ip.json. Previous addresses stay denied for keep_previous. Failure
keeps the last known set. /healthz reports age and source.
2026-09-22 22:59:08 -07:00
kyle dca635fb62 p1: vendorHash = null when vendor/ is committed
buildGoModule refuses a hashed fetch if vendor/ exists. Keep vendor/;
the check path already uses GOPROXY=off GOFLAGS=-mod=vendor.
2026-09-22 22:27:24 -07:00
kyle fac1462e7b p1: ExecStart must invoke sift serve
P1 split the binary into subcommands. Without `serve`, the unit exits 2
on `unknown command`.
2026-09-22 22:11:28 -07:00
kyleandClaude Opus 5.5 d404657820 docs: P1.1 dynamic self-IP deny (DDNS/trace discovery, keep previous, atomic swap)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 22:06:20 -07:00
kyle 8e3a6cd339 p1: cap jq output, log refusals, fail contract without SDK
jq results are bounded (10k items / output cap) and return 400 when
exceeded. 401/403 log caller, host, and reason. make contract exits 1
unless firecrawl-py is present (SIFT_CONTRACT_SKIP_SDK=1 to skip).
2026-09-22 22:02:38 -07:00
kyle 9444a35322 p1: /v2/scrape, /v1/fetch, auth, CLI, contract tests
Single Fetcher.Fetch on the Guard that holds DenyPrefixes. skipTlsVerification
is ignored. No 502. keys add writes the secret only to --out. Release binary
has no -test-allow-prefix.
2026-09-22 21:54:01 -07:00
kyleandClaude Opus 5.5 6cb206fb27 docs: P1 spec — clean fetch, auth, CLI, contract facts from firecrawl-py 4.17.0, talos wiring
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:38:29 -07:00
kyle 099b94979c p0: IANA special-purpose deny table, extra deny prefixes
Block 6to4/Teredo/NAT64-local/site-local/benchmarking/reserved
ranges from the IANA registries (families, not decoded embeds).
Guard.DenyPrefixes / -deny-prefix covers hairpin WAN /32s; home
addresses stay on-box, not in git. Control is tcp/tcp4/tcp6 only.
2026-09-22 21:33:52 -07:00
kyleandClaude Opus 5.5 3b41d6ffbc plan: UA decided — default sift/<ver>, config override, measure bot-blocks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:28:04 -07:00
kyle 2607d56e4c p0: skeleton, dialer-Control URL guard, flake, Makefile
Go service with /healthz, SSRF guard (parse + resolve-all + Control),
exhaustive tests, NixOS module, and Makefile as source of truth for
build flags. nix flake check runs make test lint.
2026-09-22 21:26:13 -07:00
kyleandClaude Opus 5.5 7d4f090a12 skill: sift as the explicit first step for fetch/search; plan §4.3 CLI client, --recover, UA policy
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:25:57 -07:00
kyleandClaude Opus 5.5 1e915ce715 plan: Makefile for non-Nix hosts (source of truth for build flags)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:15:29 -07:00
kyleandClaude Opus 5.5 e9aea1829e plan: Go instead of Python (Kyle's preference); dialer-Control SSRF check, gojq
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:05:59 -07:00
kyleandClaude Opus 5.5 94876864e5 sift: implementation plan
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 21:00:00 -07:00