sift/AGENTS.md

# AGENTS.md

sift is the fleet's fetch and search service for LLM agents: it fetches a URL through an SSRF
guard, cleans the result (HTML → Markdown, JSON, text, and from P5a PDF), optionally digests it
with a small model, and serves Hermes agents through a Firecrawl-compatible API. It is written in
Go and runs on orion under a hardened systemd unit. You are implementing it one task at a time.

## How you work

1. The owner gives you one task file, `docs/plans/<plan>/NN-name.md`. Read it fully. Do that task
   and nothing else. Do not start the next task.
2. Do the steps in order. Where a step shows a command and its expected output, run it and compare.
3. Do not read other task files, `docs/PLAN.md` or other phase docs unless the task tells you to.
   The task file quotes what you need.
4. If something in the task is impossible, contradictory, or fails twice in the same way, **stop**.
   Do not improvise, do not change a test, do not weaken a check. Add your row to
   `docs/implementer-log.md` with status `stopped`, say what you tried and what happened, commit
   only that file, and tell the owner.
5. Work only from files inside this repository. Never read another checkout, and never search the
   file system for code. If you are stuck, stop as in point 4.

## Files you must never edit

- `docs/plans/`, `docs/PLAN.md`, `docs/P*.md`, `AGENTS.md`, `tools/run-plan.sh`
- `internal/guard/` and `internal/auth/` — the SSRF guard and the key store are the security
  boundary; no task changes them.
- `go.mod`, `go.sum`, `vendor/` — every dependency is already vendored. Never add a module and
  never run `go get` or `go mod tidy`/`vendor`; the build is offline.
- `nix/` and `flake.nix`, unless the task names the exact change.
- Anything a task told you to copy from `docs/plans/**/_files/` — tests, fixtures,
  `internal/pdftest` — in this plan **or any earlier one**. If a copied test fails, your code is
  wrong. If a copied test can no longer be right because the task changes what it tested, that
  is the owner's error: stop and report it.

## Code rules

- Go 1.26 (`go.mod`). Standard library plus what is already in `vendor/`.
- No **new** source file over 400 lines. Do not grow `internal/api/api.go`: new API code goes in
  the new file the task names.
- Library code (`internal/...`) never panics on input: no `panic`, no indexing that can go out of
  range on data from a file, a request, a PDF or a peer. Check lengths. `main` may exit with a
  message.
- Errors are values: return them, wrap with `fmt.Errorf("...: %w", err)`. A check that cannot do
  its job fails; it does not return "ok".
- **Never log, print or store a request or response body, a document's text or its title.** Log
  lines carry caller, host, sizes, counts, durations and outcomes. For a restricted caller
  (`PermitRestricted`) use the existing helpers (`logHost`, `restrictedLog`, `logErr`): they
  already blank what must not be written.
- Anything read from the network, a child process or a file is bounded (`io.LimitReader`,
  deadlines).
- Exported names, field names and JSON tags are exactly as the task gives them; the tests compile
  against them.
- Comments say why, not what. `gofmt` decides layout.

## Lessons from earlier work (crossbar, same owner, same process)

- A type assertion on a value from outside your package uses the two-value form and handles
  `false`.
- When a rule says "every" or "everywhere", finish by listing each place it applies and checking
  them one by one.
- Never end a turn by describing what you are about to do. Do it, then report.
- Work only inside this repository. Scratch programs under `/tmp` or anywhere else are refused by
  the sandbox, and **a refused tool call is not a reason to end the turn**: write the experiment
  as a `_test.go` file inside the repository (delete it before committing), or reason it out.
- Never change when production code releases, flushes or records something just to make a given
  test's timing pass. If a given test seems to check a value before the code could settle it,
  that is the owner's bug: stop and report it.

## The gate

`make gate` must pass before a task is done:

```
make gate    # gofmt check, go vet, go test -race ./..., staticcheck if installed
```

It runs offline against `vendor/`. Run `gofmt -w` on the files you touched first. `go test
./internal/<pkg>/` runs one package.

## Git

- Work on the branch the task names. One task is one commit.
- Stage only the paths the task lists: `git add <path> ...`. Never `git add -A` or `git add .`.
- Never push, amend, rebase, reset, or switch to another branch.
- Commit message: the subject line the task gives, a blank line, a short body if useful, then the
  `Co-Authored-By:` trailer line the owner's prompt gives you, exactly, as the last line.
- A message with more than one line goes in `.state/commit-msg.txt` (inside the repository and
  ignored by git; `/tmp` is refused) and is committed with `git commit -F .state/commit-msg.txt`.

## The implementer log

Before you commit, add one row to the table in `docs/implementer-log.md` and include the file in
the commit:

`| <plan>/<task> | <date> | done or stopped | <sessions> | gate pass/fail | deviations | what you did |`

Be honest: the log is how the owner judges the process. A deviation you log is a finding; one
you hide is a defect.