api: PDF cache with revalidation; pages and grep; refuse questions on very long documents

Answer extracted PDFs from the document cache: fresh for PDFTTL, then a
conditional GET (304 reuses the stored text, a 200 is extracted and stored
without a second GET). Store after extraction with the response hash, ETag,
fetch time and extracted pages. Route pages/grep through pdfdoc.Options with
400s for bad ranges and non-PDFs, and refuse a question when the document
exceeds PDFMaxQuestionChunks with a pointer to grep and pages.

Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>

docs/implementer-log.md

@@ -10,3 +10,4 @@ One row per task session that ends in a commit. Newest last.
 | p5a/04-pdfpool | 2026-09-25 | done | 1 | gate: ok | none | Implemented the generic scheduler. `Pool` holds one mutex, a `jobs` map (key→in-flight job), a `callerJobs` map (caller→their job), and a FIFO `waiters` line. `Do` applies the six rules in order: join (counted in `Waiting`, no slot, allowed while the caller holds another job, `fn` may be nil), caller-busy, then slot/queue/refuse. A job's result is broadcast to every sharer via a `done` channel; results are not cached (key forgotten on finish). `fn` runs under `context.WithoutCancel` so the flight outlives a caller that leaves. Panic in `fn` is recovered into an error, freed the slot and caller. One implementation finding (not a plan deviation): queued jobs must not be counted in `Running()`, so `startLocked` only registers a job and a separate `takeSlotLocked`/`grant` increments the slot count — abandoning a queued starter (`abandon`) releases sharers without touching the slot. Given test byte-identical. |
 | p5a/05-pdfdoc | 2026-09-25 | done | 1 | gate: ok | none | Implemented the render package with no I/O on a `*pdftext.Doc`. `Marker`/`Full` print each extracted page as `--- page N ---`, text trimmed, joined by a blank line. `Render` dispatches grep → page range → whole. Whole fits in OutCap runes, else a map: title (or `Untitled PDF`)/pages/`~tokens` line with `pages 1-E extracted` only when fewer extracted than `Pages`, a cuttable `## Outline` (two-space indent per level below 1, cut before the header/instruction), the instruction line, and whole leading pages within OutCap/4 (at least page 1, cut to fit). Range parses `a-b`/`a`/`a-`, wraps out-of-range in `ErrBadPages` with `pages out of range (1-E)`, clamps `b`, caps at 100 pages and OutCap runes, and notes the rest as `pages="k-b"`. Grep is RE2 with smart-case `(?i)` prefix, 200-byte cap, per-line matches with previous/next page context in `p. P-`/`p. P:` form, first 100 shown, total counted; RE2 keeps the `(a+)+$` test linear. Every length is in runes. Given test byte-identical; make gate ok. |
 | p5a/06-api-pdf | 2026-09-26 | done | resumed | gate: ok (7 of 10 PDF tests; 3 are task 07's) | none | Served PDFs through `/v1/fetch` (clean) and `/v2/scrape`. Added `RetryAfter int` to `fetch.Error` and a `Retry-After` header line guarded by `> 0`. Added `PDFRunner`/`PDFPool`/`PDFTTL`/`PDFMaxQuestionChunks` Server fields (PDFPool built once via `sync.Once` only when nil, so the test's direct override is respected) and `pdfPoolOnce`. Routed `clean.Kind` in `doFetch`/`cleanFetched` (html/json/text switched, "other"→415, "pdf"→extractPDF); `doFetch`'s switch routes "text" explicitly so a PDF never runs `clean.Text` first. New `internal/api/pdf.go`: `DefaultPDFTTL`/`DefaultPDFMaxQuestionChunks` (task 07), `runner()`/`pool()` defaults, `extractPDF` (pool key `callerClass|SHA256`, two-value assert, `fmt.Errorf("pdf: bad result %T")`), `pdfError` (422 for the five pdftext errors + panic default, 503+RetryAfter 10 for ErrBusy, 429 for ErrCallerBusy), `renderPDF` (`pdfdoc.Render`, whole text or map). `doFetch` returns the 4th `doc *pdftext.Doc`; digest uses `pdfdoc.Full(doc)` (computed once) as content, clean uses `renderPDF.Content`, envelope `pdf` object has pages/title/outline/extracted/first/last/map/grep_hits/note, raw mode→400. Scrape markdown=`renderPDF.Content`, html="", `metadata.numPages`=doc.Pages. 7 of 10 given tests pass under -race -count=3; the 3 failures (`TestPDFPagesAndGrep`, `TestPDFCache`, `TestPDFQuestionTooLong`) are task 07 tests that reference fields not in this task, exactly as the task doc (lines 145, 157-158) predicts; nothing else fails. Given files byte-identical. |
+| p5a/07-api-cache-pages | 2026-09-26 | done | 1 | gate: ok | none | Implemented the PDF document cache with revalidation in `doFetch`: `pdfFromCache` (miss / fresh within `PDFTTL` / stale → `FetchIf` with 304→`TouchPDF`+answer-fresh / 200→return response with ok=false and no second GET) and `storePDF` after `extractPDF` (`ContentHash=res.SHA256`, ETag/Last-Modified from headers, FetchedAt=now, Meta=JSON of `doc.Meta`, Pages=`doc.Text`). `pages`/`grep` passed into `pdfdoc.Options` in `renderPDF`; `ErrBadPages`/`ErrBadGrep`→400 with the error's own text, non-PDF→400 "pages and grep apply to PDFs only" before the doc block. Question limit in digest mode: `n=len(digest.Split(pdfdoc.Full(doc), ChunkTokens))`, `n>PDFMaxQuestionChunks`→422 "document too long for question (n chunks); use grep or pages" before any digest call. Envelope `cached` set true for cache answers via the synthetic cache res's empty body (`len(res.Body)==0`); the one `doFetch` change beyond the rule-1 call was reusing that 200 response instead of a second `Fetch`. Added `fmt` import to api.go. Given file byte-identical; make gate ok. |

internal/api/api.go

@@ -6,6 +6,7 @@ import (
 	"encoding/hex"
 	"encoding/json"
 	"errors"
+	"fmt"
 	"io"
 	"log/slog"
 	"net/http"
@@ -381,7 +382,7 @@ func (s *Server) scrape(w http.ResponseWriter, r *http.Request) {
 	md, htmlOut := cl.Markdown, ""
 	pages := 0
 	if doc != nil {
-		rp, err := s.renderPDF(doc)
+		rp, err := s.renderPDF(doc, pdfdoc.Options{OutCap: s.OutCap})
 		if err != nil {
 			s.writeFetchErr(w, c, req.URL, err)
 			return
@@ -472,6 +473,11 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) {
 	}
 	fetchMS := time.Since(t0).Milliseconds()
 
+	if doc == nil && (req.Pages != "" || req.Grep != "") {
+		s.writeFetchErr(w, c, req.URL, &fetch.Error{Status: 400, Msg: "pages and grep apply to PDFs only"})
+		return
+	}
+
 	var docLen, pdfPages int
 	var content string
 	var pdfMap map[string]any
@@ -481,8 +487,12 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) {
 			writeErr(w, 400, "raw mode does not apply to PDFs", "")
 			return
 		}
-		rp, err := s.renderPDF(doc)
+		rp, err := s.renderPDF(doc, pdfdoc.Options{Pages: req.Pages, Grep: req.Grep, OutCap: s.OutCap})
 		if err != nil {
+			if errors.Is(err, pdfdoc.ErrBadPages) || errors.Is(err, pdfdoc.ErrBadGrep) {
+				s.writeFetchErr(w, c, req.URL, &fetch.Error{Status: 400, Msg: err.Error()})
+				return
+			}
 			s.writeFetchErr(w, c, req.URL, err)
 			return
 		}
@@ -526,6 +536,17 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) {
 	var digMeta map[string]any
 	var digestMS int64
 	if mode == "digest" {
+		if doc != nil {
+			n := len(digest.Split(pdfdoc.Full(doc), digest.ChunkTokens))
+			limit := s.PDFMaxQuestionChunks
+			if limit == 0 {
+				limit = DefaultPDFMaxQuestionChunks
+			}
+			if n > limit {
+				s.writeFetchErr(w, c, req.URL, &fetch.Error{Status: 422, Msg: fmt.Sprintf("document too long for question (%d chunks); use grep or pages", n)})
+				return
+			}
+		}
 		if s.Digest == nil {
 			warn = append(warn, "digest fell back to clean: unset")
 			mode = "clean"
@@ -590,7 +611,7 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) {
 		"fetched_at":   res.FetchedAt.Format(time.RFC3339),
 		"hash":         res.SHA256(),
 		"bytes":        len(res.Body),
-		"cached":       false,
+		"cached":       len(res.Body) == 0,
 		"kind":         "live",
 	}
 	if snap != nil {
@@ -673,15 +694,24 @@ func (s *Server) cleanFetched(res *fetch.Resource, jq string, onlyMain bool) (*f
 }
 
 func (s *Server) doFetch(r *http.Request, c *auth.Caller, rawURL, jq string, onlyMain bool) (*fetch.Resource, *clean.Result, *pdftext.Doc, error) {
+	res, hitDoc, ok, err := s.pdfFromCache(r.Context(), c, rawURL)
+	if err != nil {
+		return nil, nil, nil, err
+	}
+	if ok {
+		return res, &clean.Result{Kind: "pdf", Title: hitDoc.Title}, hitDoc, nil
+	}
+	if res == nil {
 		if err := s.acquire(c.Name, rawURL); err != nil {
 			return nil, nil, nil, err
 		}
 		defer s.release(c.Name, rawURL)
 
-	res, err := s.Fetch.Fetch(r.Context(), rawURL)
+		res, err = s.Fetch.Fetch(r.Context(), rawURL)
 		if err != nil {
 			return nil, nil, nil, err
 		}
+	}
 	kind := clean.Kind(res.ContentType, res.Body)
 	if kind == "other" {
 		ct := res.ContentType
@@ -713,6 +743,7 @@ func (s *Server) doFetch(r *http.Request, c *auth.Caller, rawURL, jq string, onl
 		if err != nil {
 			return nil, nil, nil, err
 		}
+		s.storePDF(dcache.PDFKey(rawURL, s.callerClass(c)), res, doc)
 		cl = &clean.Result{Kind: "pdf", Title: doc.Title}
 	}
 	return res, cl, doc, nil

internal/api/pdf.go

@@ -2,11 +2,14 @@ package api
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"fmt"
+	"net/http"
 	"time"
 
 	"git.wntrmute.dev/kyle/sift/internal/auth"
+	"git.wntrmute.dev/kyle/sift/internal/dcache"
 	"git.wntrmute.dev/kyle/sift/internal/fetch"
 	"git.wntrmute.dev/kyle/sift/internal/pdfdoc"
 	"git.wntrmute.dev/kyle/sift/internal/pdfpool"
@@ -80,7 +83,77 @@ func pdfError(err error) error {
 }
 
 // renderPDF draws the document for an agent: the whole text when it fits the
-// output cap, otherwise the map.
-func (s *Server) renderPDF(doc *pdftext.Doc) (pdfdoc.Rendered, error) {
-	return pdfdoc.Render(doc, pdfdoc.Options{OutCap: s.OutCap})
+// output cap, otherwise the map, a page range, or grep results.
+func (s *Server) renderPDF(doc *pdftext.Doc, opts pdfdoc.Options) (pdfdoc.Rendered, error) {
+	return pdfdoc.Render(doc, opts)
+}
+
+// pdfFromCache answers a PDF from the document cache before any fetch: a fresh
+// entry directly, a stale one after a conditional GET (a 304 reuses the stored
+// text), or a miss. A 200 from the revalidation is returned with ok=false so the
+// caller extracts it; a nil cache is always a miss.
+func (s *Server) pdfFromCache(ctx context.Context, c *auth.Caller, rawURL string) (*fetch.Resource, *pdftext.Doc, bool, error) {
+	cache := s.cache()
+	if cache == nil {
+		return nil, nil, false, nil
+	}
+	key := dcache.PDFKey(rawURL, s.callerClass(c))
+	e, ok := cache.GetPDF(key)
+	if !ok {
+		return nil, nil, false, nil
+	}
+	ttl := s.PDFTTL
+	if ttl == 0 {
+		ttl = DefaultPDFTTL
+	}
+	if time.Since(e.FetchedAt) < ttl {
+		res, doc := s.pdfFromEntry(rawURL, e)
+		return res, doc, true, nil
+	}
+	res, err := s.Fetch.FetchIf(ctx, rawURL, e.ETag, e.LastModified)
+	if err != nil {
+		return nil, nil, false, err
+	}
+	if res.Status == http.StatusNotModified {
+		cache.TouchPDF(key, time.Now().UTC())
+		res, doc := s.pdfFromEntry(rawURL, e)
+		return res, doc, true, nil
+	}
+	return res, nil, false, nil
+}
+
+// pdfFromEntry rebuilds the synthetic response and doc for a cached entry: the
+// request's URL, a 200 PDF with the stored fetch time and hash, and no body.
+func (s *Server) pdfFromEntry(rawURL string, e dcache.PDFEntry) (*fetch.Resource, *pdftext.Doc) {
+	var doc pdftext.Doc
+	_ = json.Unmarshal(e.Meta, &doc.Meta)
+	doc.Text = e.Pages
+	res := &fetch.Resource{
+		URL:         rawURL,
+		FinalURL:    rawURL,
+		Status:      200,
+		ContentType: "application/pdf",
+		FetchedAt:   e.FetchedAt,
+		Hash:        e.ContentHash,
+	}
+	return res, &doc
+}
+
+// storePDF writes an extracted PDF into the document cache for its TTL: its
+// hash, the response's validators, the metadata as JSON and the per-page text.
+// A nil cache is a no-op.
+func (s *Server) storePDF(key string, res *fetch.Resource, doc *pdftext.Doc) {
+	cache := s.cache()
+	if cache == nil {
+		return
+	}
+	meta, _ := json.Marshal(doc.Meta)
+	cache.PutPDF(key, dcache.PDFEntry{
+		ContentHash:  res.SHA256(),
+		ETag:         res.Header.Get("ETag"),
+		LastModified: res.Header.Get("Last-Modified"),
+		FetchedAt:    time.Now().UTC(),
+		Meta:         meta,
+		Pages:        doc.Text,
+	})
 }