1c57845ebd174729709c09131ee1843cbe3f7f48
CanonicalURL is the dedupe key only. Hits keep the first source's original URL, including a trailing slash. Dedupe has the same rule.
sift
LLM-oriented fetch and search for the Hermes fleet: fetch a resource and get it back cleaned, compact, and with useful context attached — using local models (Dixie), SearXNG and Brave on orion, and Honcho for retrieval.
Status: P1 clean fetch (/v2/scrape, /v1/fetch, CLI, auth). See docs/PLAN.md and docs/P1.md.
Build
Makefile is the source of truth for flags (CGO_ENABLED=0, -trimpath, -s -w -X main.version=).
make build # bin/sift
make test # go test -race ./...
make lint # go vet, and staticcheck if installed
make install # PREFIX=/usr/local; example config in /etc/sift/
nix build # packages.sift, same ldflags
nix flake check # make test lint
The service does not assume Nix store paths. Non-Nix hosts: make install plus dist/sift.service.
Languages
Go
93.1%
Python
4.7%
Nix
1.4%
Shell
0.5%
Makefile
0.3%