sift/docs/plans/p5a/01-kind-fetch.md

P5a task 01: recognise PDFs; give them a larger body cap, a longer deadline, and revalidation

Branch: p5a (git switch -c p5a master if it does not exist, else git switch p5a; git status --short must be empty, otherwise stop) Commit subject: clean, fetch: recognise PDFs; 100 MB / 120 s for PDFs; FetchIf revalidates

Goal

Today every response that is not HTML, JSON or text is 415 unsupported content type. After this task clean.Kind names PDFs, and the fetcher lets a PDF be as large as 100 MB and take 120 s while a web page keeps 5 MB and 20 s. FetchIf sends If-None-Match / If-Modified-Since so a cached document can be revalidated with a 304. Nothing uses these yet (task 06 does).

Files

• Copy: internal/clean/kind_test.go, internal/fetch/pdf_test.go
• Create: internal/fetch/pdf.go (the PDF rule and FetchIf live here; keep fetch.go's changes to what the new file needs)
• Modify: internal/clean/clean.go, internal/fetch/fetch.go, docs/implementer-log.md

Interfaces

package clean

// Kind classifies a fetched body: "html", "json", "text", "pdf" or "other".
func Kind(contentType string, body []byte) string

package fetch

const (
	DefaultPDFMaxBody = 100 << 20
	DefaultPDFTimeout = 120 * time.Second
)

// in Fetcher:
	PDFMaxBody int64         // 0 = DefaultPDFMaxBody
	PDFTimeout time.Duration // 0 = DefaultPDFTimeout

// FetchIf is Fetch with validators: a non-empty etag is sent as If-None-Match and a non-empty
// lastModified as If-Modified-Since. A 304 answer returns a Resource with Status 304, an empty
// Body and a nil error. With both empty it is exactly Fetch.
func (f *Fetcher) FetchIf(ctx context.Context, rawURL, etag, lastModified string) (*Resource, error)

Rules

1. Kind (the media type is lower-cased and its parameters dropped, as MediaKind does):
   • application/pdf or application/x-pdf → "pdf", whatever the body.
   • text/html, application/xhtml+xml → "html"; application/json or *+json → "json"; text/plain, text/markdown, text/csv → "text" (as today).
   • application/octet-stream or binary/octet-stream → "pdf" if %PDF- occurs within the first 1024 bytes of the body, else "other".
   • empty Content-Type → "pdf" if %PDF- occurs within the first 1024 bytes, else today's Sniff (html / json / text).
   • anything else → "other". MediaKind also returns "pdf" for the two PDF types; its other answers do not change.
2. Fetch limits. A response is a PDF when its Content-Type is a PDF type, or when it is empty or octet-stream and the body starts (within 1024 bytes) with %PDF-. For a PDF the body cap is PDFMaxBody and the deadline is PDFTimeout, both counted from the start of the request; for anything else they stay MaxBody and Timeout. Over the cap is the existing 413 too large (>N MB decompressed) with N from the cap that applied.
3. How to get the deadline right. Fetch today makes one context with Timeout, and the shared http.Client has Timeout set too — both would cut a PDF off at 20 s. Make the client's Timeout at least the PDF deadline (or unset it and rely on contexts), give the request a context with the PDF deadline, and cancel it at the ordinary deadline unless the response has been recognised as a PDF by then (time.AfterFunc + a flag you set once the headers, or the first bytes of an octet-stream body, say PDF). Read the body through io.LimitReader with the cap that applies (+1 to detect overflow), as today.
4. FetchIf goes through the same guard, client, limits and error mapping as Fetch; only the two request headers differ. Do not copy Fetch: factor the shared part.

Facts

• The tests reach an httptest server on 127.0.0.1 through a guard.Guard with AllowPrefixes: 127.0.0.0/8; the guard itself is not yours to change.
• internal/fetch has no other tests; internal/api tests exercise Fetch through the API and must stay green.

Steps

• 1. Branch as above; copy the two given tests.
• 2. See them fail: go test -count=1 ./internal/clean/ ./internal/fetch/ → Kind, FetchIf, DefaultPDFMaxBody undefined.
• 3. clean.go. 4. fetch/pdf.go and the smallest changes to fetch.go.
• 5. gofmt -w; go test -race -count=3 ./internal/clean/ ./internal/fetch/ ./internal/api/ → ok.
• 6. make gate. 7. Row p5a/01-kind-fetch; commit.

git add internal/clean internal/fetch docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The given tests and every earlier test pass under -race -count=3; make gate prints gate: ok; given files byte-identical.

Stop and report if

• A given test needs a change to internal/guard.