sift/docs/plans/p5a/01-kind-fetch.md

# P5a task 01: recognise PDFs; give them a larger body cap, a longer deadline, and revalidation

**Branch:** `p5a` (`git switch -c p5a master` if it does not exist, else `git switch p5a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `clean, fetch: recognise PDFs; 100 MB / 120 s for PDFs; FetchIf revalidates`

## Goal

Today every response that is not HTML, JSON or text is `415 unsupported content type`. After this
task `clean.Kind` names PDFs, and the fetcher lets a PDF be as large as 100 MB and take 120 s
while a web page keeps 5 MB and 20 s. `FetchIf` sends `If-None-Match` / `If-Modified-Since` so a
cached document can be revalidated with a 304. Nothing uses these yet (task 06 does).

## Files

- Copy: `internal/clean/kind_test.go`, `internal/fetch/pdf_test.go`
- Create: `internal/fetch/pdf.go` (the PDF rule and `FetchIf` live here; keep `fetch.go`'s
  changes to what the new file needs)
- Modify: `internal/clean/clean.go`, `internal/fetch/fetch.go`, `docs/implementer-log.md`

## Interfaces

```go
package clean

// Kind classifies a fetched body: "html", "json", "text", "pdf" or "other".
func Kind(contentType string, body []byte) string

package fetch

const (
	DefaultPDFMaxBody = 100 << 20
	DefaultPDFTimeout = 120 * time.Second
)

// in Fetcher:
	PDFMaxBody int64         // 0 = DefaultPDFMaxBody
	PDFTimeout time.Duration // 0 = DefaultPDFTimeout

// FetchIf is Fetch with validators: a non-empty etag is sent as If-None-Match and a non-empty
// lastModified as If-Modified-Since. A 304 answer returns a Resource with Status 304, an empty
// Body and a nil error. With both empty it is exactly Fetch.
func (f *Fetcher) FetchIf(ctx context.Context, rawURL, etag, lastModified string) (*Resource, error)
```

## Rules

1. **`Kind`** (the media type is lower-cased and its parameters dropped, as `MediaKind` does):
   - `application/pdf` or `application/x-pdf` → `"pdf"`, whatever the body.
   - `text/html`, `application/xhtml+xml` → `"html"`; `application/json` or `*+json` → `"json"`;
     `text/plain`, `text/markdown`, `text/csv` → `"text"` (as today).
   - `application/octet-stream` or `binary/octet-stream` → `"pdf"` if `%PDF-` occurs within the
     first 1024 bytes of the body, else `"other"`.
   - empty Content-Type → `"pdf"` if `%PDF-` occurs within the first 1024 bytes, else today's
     `Sniff` (html / json / text).
   - anything else → `"other"`.
   `MediaKind` also returns `"pdf"` for the two PDF types; its other answers do not change.
2. **Fetch limits.** A response is a PDF when its Content-Type is a PDF type, or when it is empty
   or octet-stream and the body starts (within 1024 bytes) with `%PDF-`. For a PDF the body cap
   is `PDFMaxBody` and the deadline is `PDFTimeout`, both counted from the start of the request;
   for anything else they stay `MaxBody` and `Timeout`. Over the cap is the existing
   `413 too large (>N MB decompressed)` with N from the cap that applied.
3. **How to get the deadline right.** `Fetch` today makes one context with `Timeout`, and the
   shared `http.Client` has `Timeout` set too — both would cut a PDF off at 20 s. Make the
   client's `Timeout` at least the PDF deadline (or unset it and rely on contexts), give the
   request a context with the **PDF** deadline, and cancel it at the ordinary deadline unless the
   response has been recognised as a PDF by then (`time.AfterFunc` + a flag you set once the
   headers, or the first bytes of an octet-stream body, say PDF). Read the body through
   `io.LimitReader` with the cap that applies (+1 to detect overflow), as today.
4. `FetchIf` goes through the same guard, client, limits and error mapping as `Fetch`; only the
   two request headers differ. Do not copy `Fetch`: factor the shared part.

## Facts

- The tests reach an `httptest` server on 127.0.0.1 through a `guard.Guard` with
  `AllowPrefixes: 127.0.0.0/8`; the guard itself is not yours to change.
- `internal/fetch` has no other tests; `internal/api` tests exercise `Fetch` through the API and
  must stay green.

## Steps

- [ ] **1.** Branch as above; copy the two given tests.
- [ ] **2. See them fail:** `go test -count=1 ./internal/clean/ ./internal/fetch/` → `Kind`,
  `FetchIf`, `DefaultPDFMaxBody` undefined.
- [ ] **3.** `clean.go`. **4.** `fetch/pdf.go` and the smallest changes to `fetch.go`.
- [ ] **5.** `gofmt -w`; `go test -race -count=3 ./internal/clean/ ./internal/fetch/ ./internal/api/` → `ok`.
- [ ] **6.** `make gate`. **7.** Row `p5a/01-kind-fetch`; commit.

```sh
git add internal/clean internal/fetch docs/implementer-log.md
git commit -F .state/commit-msg.txt
```

## Done when

- The given tests and every earlier test pass under `-race -count=3`; `make gate` prints
  `gate: ok`; given files byte-identical.

## Stop and report if

- A given test needs a change to `internal/guard`.