From 09f738bcac7bc3c8a45aad8f5476a86bd2e9a1fd Mon Sep 17 00:00:00 2001 From: Kyle Isom Date: Sat, 26 Sep 2026 04:23:21 -0700 Subject: [PATCH] api: PDF cache with revalidation; pages and grep; refuse questions on very long documents Answer extracted PDFs from the document cache: fresh for PDFTTL, then a conditional GET (304 reuses the stored text, a 200 is extracted and stored without a second GET). Store after extraction with the response hash, ETag, fetch time and extracted pages. Route pages/grep through pdfdoc.Options with 400s for bad ranges and non-PDFs, and refuse a question when the document exceeds PDFMaxQuestionChunks with a pointer to grep and pages. Co-Authored-By: ornith-1.5-35b-a3b --- docs/implementer-log.md | 1 + internal/api/api.go | 49 ++++++++++++++++++++----- internal/api/pdf.go | 79 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 117 insertions(+), 12 deletions(-) diff --git a/docs/implementer-log.md b/docs/implementer-log.md index b709470..f422c11 100644 --- a/docs/implementer-log.md +++ b/docs/implementer-log.md @@ -10,3 +10,4 @@ One row per task session that ends in a commit. Newest last. | p5a/04-pdfpool | 2026-09-25 | done | 1 | gate: ok | none | Implemented the generic scheduler. `Pool` holds one mutex, a `jobs` map (key→in-flight job), a `callerJobs` map (caller→their job), and a FIFO `waiters` line. `Do` applies the six rules in order: join (counted in `Waiting`, no slot, allowed while the caller holds another job, `fn` may be nil), caller-busy, then slot/queue/refuse. A job's result is broadcast to every sharer via a `done` channel; results are not cached (key forgotten on finish). `fn` runs under `context.WithoutCancel` so the flight outlives a caller that leaves. Panic in `fn` is recovered into an error, freed the slot and caller. One implementation finding (not a plan deviation): queued jobs must not be counted in `Running()`, so `startLocked` only registers a job and a separate `takeSlotLocked`/`grant` increments the slot count — abandoning a queued starter (`abandon`) releases sharers without touching the slot. Given test byte-identical. | | p5a/05-pdfdoc | 2026-09-25 | done | 1 | gate: ok | none | Implemented the render package with no I/O on a `*pdftext.Doc`. `Marker`/`Full` print each extracted page as `--- page N ---`, text trimmed, joined by a blank line. `Render` dispatches grep → page range → whole. Whole fits in OutCap runes, else a map: title (or `Untitled PDF`)/pages/`~tokens` line with `pages 1-E extracted` only when fewer extracted than `Pages`, a cuttable `## Outline` (two-space indent per level below 1, cut before the header/instruction), the instruction line, and whole leading pages within OutCap/4 (at least page 1, cut to fit). Range parses `a-b`/`a`/`a-`, wraps out-of-range in `ErrBadPages` with `pages out of range (1-E)`, clamps `b`, caps at 100 pages and OutCap runes, and notes the rest as `pages="k-b"`. Grep is RE2 with smart-case `(?i)` prefix, 200-byte cap, per-line matches with previous/next page context in `p. P-`/`p. P:` form, first 100 shown, total counted; RE2 keeps the `(a+)+$` test linear. Every length is in runes. Given test byte-identical; make gate ok. | | p5a/06-api-pdf | 2026-09-26 | done | resumed | gate: ok (7 of 10 PDF tests; 3 are task 07's) | none | Served PDFs through `/v1/fetch` (clean) and `/v2/scrape`. Added `RetryAfter int` to `fetch.Error` and a `Retry-After` header line guarded by `> 0`. Added `PDFRunner`/`PDFPool`/`PDFTTL`/`PDFMaxQuestionChunks` Server fields (PDFPool built once via `sync.Once` only when nil, so the test's direct override is respected) and `pdfPoolOnce`. Routed `clean.Kind` in `doFetch`/`cleanFetched` (html/json/text switched, "other"→415, "pdf"→extractPDF); `doFetch`'s switch routes "text" explicitly so a PDF never runs `clean.Text` first. New `internal/api/pdf.go`: `DefaultPDFTTL`/`DefaultPDFMaxQuestionChunks` (task 07), `runner()`/`pool()` defaults, `extractPDF` (pool key `callerClass|SHA256`, two-value assert, `fmt.Errorf("pdf: bad result %T")`), `pdfError` (422 for the five pdftext errors + panic default, 503+RetryAfter 10 for ErrBusy, 429 for ErrCallerBusy), `renderPDF` (`pdfdoc.Render`, whole text or map). `doFetch` returns the 4th `doc *pdftext.Doc`; digest uses `pdfdoc.Full(doc)` (computed once) as content, clean uses `renderPDF.Content`, envelope `pdf` object has pages/title/outline/extracted/first/last/map/grep_hits/note, raw mode→400. Scrape markdown=`renderPDF.Content`, html="", `metadata.numPages`=doc.Pages. 7 of 10 given tests pass under -race -count=3; the 3 failures (`TestPDFPagesAndGrep`, `TestPDFCache`, `TestPDFQuestionTooLong`) are task 07 tests that reference fields not in this task, exactly as the task doc (lines 145, 157-158) predicts; nothing else fails. Given files byte-identical. | +| p5a/07-api-cache-pages | 2026-09-26 | done | 1 | gate: ok | none | Implemented the PDF document cache with revalidation in `doFetch`: `pdfFromCache` (miss / fresh within `PDFTTL` / stale → `FetchIf` with 304→`TouchPDF`+answer-fresh / 200→return response with ok=false and no second GET) and `storePDF` after `extractPDF` (`ContentHash=res.SHA256`, ETag/Last-Modified from headers, FetchedAt=now, Meta=JSON of `doc.Meta`, Pages=`doc.Text`). `pages`/`grep` passed into `pdfdoc.Options` in `renderPDF`; `ErrBadPages`/`ErrBadGrep`→400 with the error's own text, non-PDF→400 "pages and grep apply to PDFs only" before the doc block. Question limit in digest mode: `n=len(digest.Split(pdfdoc.Full(doc), ChunkTokens))`, `n>PDFMaxQuestionChunks`→422 "document too long for question (n chunks); use grep or pages" before any digest call. Envelope `cached` set true for cache answers via the synthetic cache res's empty body (`len(res.Body)==0`); the one `doFetch` change beyond the rule-1 call was reusing that 200 response instead of a second `Fetch`. Added `fmt` import to api.go. Given file byte-identical; make gate ok. | diff --git a/internal/api/api.go b/internal/api/api.go index 953759d..800f775 100644 --- a/internal/api/api.go +++ b/internal/api/api.go @@ -6,6 +6,7 @@ import ( "encoding/hex" "encoding/json" "errors" + "fmt" "io" "log/slog" "net/http" @@ -381,7 +382,7 @@ func (s *Server) scrape(w http.ResponseWriter, r *http.Request) { md, htmlOut := cl.Markdown, "" pages := 0 if doc != nil { - rp, err := s.renderPDF(doc) + rp, err := s.renderPDF(doc, pdfdoc.Options{OutCap: s.OutCap}) if err != nil { s.writeFetchErr(w, c, req.URL, err) return @@ -472,6 +473,11 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) { } fetchMS := time.Since(t0).Milliseconds() + if doc == nil && (req.Pages != "" || req.Grep != "") { + s.writeFetchErr(w, c, req.URL, &fetch.Error{Status: 400, Msg: "pages and grep apply to PDFs only"}) + return + } + var docLen, pdfPages int var content string var pdfMap map[string]any @@ -481,8 +487,12 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) { writeErr(w, 400, "raw mode does not apply to PDFs", "") return } - rp, err := s.renderPDF(doc) + rp, err := s.renderPDF(doc, pdfdoc.Options{Pages: req.Pages, Grep: req.Grep, OutCap: s.OutCap}) if err != nil { + if errors.Is(err, pdfdoc.ErrBadPages) || errors.Is(err, pdfdoc.ErrBadGrep) { + s.writeFetchErr(w, c, req.URL, &fetch.Error{Status: 400, Msg: err.Error()}) + return + } s.writeFetchErr(w, c, req.URL, err) return } @@ -526,6 +536,17 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) { var digMeta map[string]any var digestMS int64 if mode == "digest" { + if doc != nil { + n := len(digest.Split(pdfdoc.Full(doc), digest.ChunkTokens)) + limit := s.PDFMaxQuestionChunks + if limit == 0 { + limit = DefaultPDFMaxQuestionChunks + } + if n > limit { + s.writeFetchErr(w, c, req.URL, &fetch.Error{Status: 422, Msg: fmt.Sprintf("document too long for question (%d chunks); use grep or pages", n)}) + return + } + } if s.Digest == nil { warn = append(warn, "digest fell back to clean: unset") mode = "clean" @@ -590,7 +611,7 @@ func (s *Server) v1fetch(w http.ResponseWriter, r *http.Request) { "fetched_at": res.FetchedAt.Format(time.RFC3339), "hash": res.SHA256(), "bytes": len(res.Body), - "cached": false, + "cached": len(res.Body) == 0, "kind": "live", } if snap != nil { @@ -673,15 +694,24 @@ func (s *Server) cleanFetched(res *fetch.Resource, jq string, onlyMain bool) (*f } func (s *Server) doFetch(r *http.Request, c *auth.Caller, rawURL, jq string, onlyMain bool) (*fetch.Resource, *clean.Result, *pdftext.Doc, error) { - if err := s.acquire(c.Name, rawURL); err != nil { - return nil, nil, nil, err - } - defer s.release(c.Name, rawURL) - - res, err := s.Fetch.Fetch(r.Context(), rawURL) + res, hitDoc, ok, err := s.pdfFromCache(r.Context(), c, rawURL) if err != nil { return nil, nil, nil, err } + if ok { + return res, &clean.Result{Kind: "pdf", Title: hitDoc.Title}, hitDoc, nil + } + if res == nil { + if err := s.acquire(c.Name, rawURL); err != nil { + return nil, nil, nil, err + } + defer s.release(c.Name, rawURL) + + res, err = s.Fetch.Fetch(r.Context(), rawURL) + if err != nil { + return nil, nil, nil, err + } + } kind := clean.Kind(res.ContentType, res.Body) if kind == "other" { ct := res.ContentType @@ -713,6 +743,7 @@ func (s *Server) doFetch(r *http.Request, c *auth.Caller, rawURL, jq string, onl if err != nil { return nil, nil, nil, err } + s.storePDF(dcache.PDFKey(rawURL, s.callerClass(c)), res, doc) cl = &clean.Result{Kind: "pdf", Title: doc.Title} } return res, cl, doc, nil diff --git a/internal/api/pdf.go b/internal/api/pdf.go index 7efa38a..d830892 100644 --- a/internal/api/pdf.go +++ b/internal/api/pdf.go @@ -2,11 +2,14 @@ package api import ( "context" + "encoding/json" "errors" "fmt" + "net/http" "time" "git.wntrmute.dev/kyle/sift/internal/auth" + "git.wntrmute.dev/kyle/sift/internal/dcache" "git.wntrmute.dev/kyle/sift/internal/fetch" "git.wntrmute.dev/kyle/sift/internal/pdfdoc" "git.wntrmute.dev/kyle/sift/internal/pdfpool" @@ -80,7 +83,77 @@ func pdfError(err error) error { } // renderPDF draws the document for an agent: the whole text when it fits the -// output cap, otherwise the map. -func (s *Server) renderPDF(doc *pdftext.Doc) (pdfdoc.Rendered, error) { - return pdfdoc.Render(doc, pdfdoc.Options{OutCap: s.OutCap}) +// output cap, otherwise the map, a page range, or grep results. +func (s *Server) renderPDF(doc *pdftext.Doc, opts pdfdoc.Options) (pdfdoc.Rendered, error) { + return pdfdoc.Render(doc, opts) +} + +// pdfFromCache answers a PDF from the document cache before any fetch: a fresh +// entry directly, a stale one after a conditional GET (a 304 reuses the stored +// text), or a miss. A 200 from the revalidation is returned with ok=false so the +// caller extracts it; a nil cache is always a miss. +func (s *Server) pdfFromCache(ctx context.Context, c *auth.Caller, rawURL string) (*fetch.Resource, *pdftext.Doc, bool, error) { + cache := s.cache() + if cache == nil { + return nil, nil, false, nil + } + key := dcache.PDFKey(rawURL, s.callerClass(c)) + e, ok := cache.GetPDF(key) + if !ok { + return nil, nil, false, nil + } + ttl := s.PDFTTL + if ttl == 0 { + ttl = DefaultPDFTTL + } + if time.Since(e.FetchedAt) < ttl { + res, doc := s.pdfFromEntry(rawURL, e) + return res, doc, true, nil + } + res, err := s.Fetch.FetchIf(ctx, rawURL, e.ETag, e.LastModified) + if err != nil { + return nil, nil, false, err + } + if res.Status == http.StatusNotModified { + cache.TouchPDF(key, time.Now().UTC()) + res, doc := s.pdfFromEntry(rawURL, e) + return res, doc, true, nil + } + return res, nil, false, nil +} + +// pdfFromEntry rebuilds the synthetic response and doc for a cached entry: the +// request's URL, a 200 PDF with the stored fetch time and hash, and no body. +func (s *Server) pdfFromEntry(rawURL string, e dcache.PDFEntry) (*fetch.Resource, *pdftext.Doc) { + var doc pdftext.Doc + _ = json.Unmarshal(e.Meta, &doc.Meta) + doc.Text = e.Pages + res := &fetch.Resource{ + URL: rawURL, + FinalURL: rawURL, + Status: 200, + ContentType: "application/pdf", + FetchedAt: e.FetchedAt, + Hash: e.ContentHash, + } + return res, &doc +} + +// storePDF writes an extracted PDF into the document cache for its TTL: its +// hash, the response's validators, the metadata as JSON and the per-page text. +// A nil cache is a no-op. +func (s *Server) storePDF(key string, res *fetch.Resource, doc *pdftext.Doc) { + cache := s.cache() + if cache == nil { + return + } + meta, _ := json.Marshal(doc.Meta) + cache.PutPDF(key, dcache.PDFEntry{ + ContentHash: res.SHA256(), + ETag: res.Header.Get("ETag"), + LastModified: res.Header.Get("Last-Modified"), + FetchedAt: time.Now().UTC(), + Meta: meta, + Pages: doc.Text, + }) }