sift/AGENTS.md

AGENTS.md

sift is the fleet's fetch and search service for LLM agents: it fetches a URL through an SSRF guard, cleans the result (HTML → Markdown, JSON, text, and from P5a PDF), optionally digests it with a small model, and serves Hermes agents through a Firecrawl-compatible API. It is written in Go and runs on orion under a hardened systemd unit. You are implementing it one task at a time.

How you work

1. The owner gives you one task file, docs/plans/<plan>/NN-name.md. Read it fully. Do that task and nothing else. Do not start the next task.
2. Do the steps in order. Where a step shows a command and its expected output, run it and compare.
3. Do not read other task files, docs/PLAN.md or other phase docs unless the task tells you to. The task file quotes what you need.
4. If something in the task is impossible, contradictory, or fails twice in the same way, stop. Do not improvise, do not change a test, do not weaken a check. Add your row to docs/implementer-log.md with status stopped, say what you tried and what happened, commit only that file, and tell the owner.
5. Work only from files inside this repository. Never read another checkout, and never search the file system for code. If you are stuck, stop as in point 4.
6. Write code into files early and in small steps. Start each new file with its types and signatures, make it compile, then fill in one function at a time and run the tests as you go. Never write a whole implementation out in your reply before putting it in a file: a long session is compacted, and what was only in the reply is lost (P6 task 02, 2026-09-26).

Files you must never edit

• docs/plans/, docs/PLAN.md, docs/P*.md, AGENTS.md, tools/run-plan.sh
• internal/guard/ and internal/auth/ — the SSRF guard and the key store are the security boundary; no task changes them.
• go.mod, go.sum, vendor/ — every dependency is already vendored. Never add a module and never run go get or go mod tidy/vendor; the build is offline.
• nix/ and flake.nix, unless the task names the exact change.
• Anything a task told you to copy from docs/plans/**/_files/ — tests, fixtures, internal/pdftest — in this plan or any earlier one. If a copied test fails, your code is wrong. If a copied test can no longer be right because the task changes what it tested, that is the owner's error: stop and report it.

Code rules

• Go 1.26 (go.mod). Standard library plus what is already in vendor/.
• No new source file over 400 lines. Do not grow internal/api/api.go: new API code goes in the new file the task names.
• Library code (internal/...) never panics on input: no panic, no indexing that can go out of range on data from a file, a request, a PDF or a peer. Check lengths. main may exit with a message.
• Errors are values: return them, wrap with fmt.Errorf("...: %w", err). A check that cannot do its job fails; it does not return "ok".
• Never log, print or store a request or response body, a document's text or its title. Log lines carry caller, host, sizes, counts, durations and outcomes. For a restricted caller (PermitRestricted) use the existing helpers (logHost, restrictedLog, logErr): they already blank what must not be written.
• Anything read from the network, a child process or a file is bounded (io.LimitReader, deadlines).
• Exported names, field names and JSON tags are exactly as the task gives them; the tests compile against them.
• Comments say why, not what. gofmt decides layout.

Lessons from earlier work (crossbar, same owner, same process)

• A type assertion on a value from outside your package uses the two-value form and handles false.
• When a rule says "every" or "everywhere", finish by listing each place it applies and checking them one by one.
• Never end a turn by describing what you are about to do. Do it, then report.
• Work only inside this repository. Scratch programs under /tmp or anywhere else are refused by the sandbox, and a refused tool call is not a reason to end the turn: write the experiment as a _test.go file inside the repository (delete it before committing), or reason it out.
• Never change when production code releases, flushes or records something just to make a given test's timing pass. If a given test seems to check a value before the code could settle it, that is the owner's bug: stop and report it.

The gate

make gate must pass before a task is done:

make gate    # gofmt check, go vet, go test -race ./..., staticcheck if installed

It runs offline against vendor/. Run gofmt -w on the files you touched first. go test ./internal/<pkg>/ runs one package.

Git

• Work on the branch the task names. One task is one commit.
• Stage only the paths the task lists: git add <path> .... Never git add -A or git add ..
• Never push, amend, rebase, reset, or switch to another branch.
• Commit message: the subject line the task gives, a blank line, a short body if useful, then the Co-Authored-By: trailer line the owner's prompt gives you, exactly, as the last line.
• A message with more than one line goes in .state/commit-msg.txt (inside the repository and ignored by git; /tmp is refused) and is committed with git commit -F .state/commit-msg.txt.

The implementer log

Before you commit, add one row to the table in docs/implementer-log.md and include the file in the commit:

| <plan>/<task> | <date> | done or stopped | <sessions> | gate pass/fail | deviations | what you did |

Be honest: the log is how the owner judges the process. A deviation you log is a finding; one you hide is a defect.