sift/docs/plans/p5a/07-api-cache-pages.md

P5a task 07: the PDF document cache, pages, grep, and the question limit

Branch: p5a (git switch p5a; git status --short must be empty, otherwise stop) Commit subject: api: PDF cache with revalidation; pages and grep; refuse questions on very long documents

Goal

Task 06 serves PDFs but fetches and extracts every time. After this task, a PDF already extracted is answered from the cache: fresh for 24 h, then revalidated with a conditional GET. An agent can ask for pages or grep. A question on a document too long for the digest is refused with a pointer to those two.

Files

• Copy: nothing (the given internal/api/pdf_test.go came in task 06)
• Modify: internal/api/pdf.go (all new logic), internal/api/api.go (only the call in rule 1 and the two request fields' use in rule 3), docs/implementer-log.md

What already exists

• Task 06's extractPDF, pdfError, renderPDF, the Server fields (PDFTTL, PDFMaxQuestionChunks), the v1Req fields (Pages, Grep), and DefaultPDFTTL / DefaultPDFMaxQuestionChunks.
• internal/dcache:

  type PDFEntry struct{ ContentHash, ETag, LastModified string; FetchedAt time.Time; Meta []byte; Pages []string }
  func PDFKey(url, callerClass string) string
  func (d *DB) PutPDF(key string, e PDFEntry)
  func (d *DB) GetPDF(key string) (PDFEntry, bool)
  func (d *DB) TouchPDF(key string, fetchedAt time.Time)
  

  A nil *DB is a no-op.
• internal/fetch: func (f *Fetcher) FetchIf(ctx, rawURL, etag, lastModified string) (*Resource, error). A 304 returns Status 304, an empty body and a nil error.
• internal/digest: Split(text string, maxTok int) []string and ChunkTokens.

Rules

1. Cache first, in doFetch, before acquire and s.Fetch.Fetch: call s.pdfFromCache(ctx, c, rawURL) (res *fetch.Resource, doc *pdftext.Doc, ok bool, err error). The key is dcache.PDFKey(rawURL, s.callerClass(c)); the cache is s.cache() (may be nil).
   • Miss: ok == false; go on as today.
   • Fresh (time.Since(e.FetchedAt) < ttl, where ttl is s.PDFTTL or DefaultPDFTTL): build the doc from the entry (unmarshal Meta into doc.Meta; doc.Text = e.Pages), and a synthetic res: URL and FinalURL = rawURL, Status 200, ContentType "application/pdf", FetchedAt = e.FetchedAt, Hash = e.ContentHash, no body. No network call and no child.
   • Stale: s.Fetch.FetchIf(ctx, rawURL, e.ETag, e.LastModified).
     • 304 → TouchPDF(key, now), then answer as fresh.
     • 200 → return that response with ok == false, so doFetch goes on without a second GET: the new document is extracted and stored.
     • An error → return it.
   • The source in the /v1/fetch envelope says "cached": true for an answer from the cache (fresh or after a 304).
2. Store after extraction: after extractPDF succeeds, PutPDF under the URL key with:
   • ContentHash: res.SHA256();
   • ETag and Last-Modified from the response headers;
   • FetchedAt: now;
   • Meta: the JSON of doc.Meta;
   • Pages: doc.Text.
3. pages and grep (/v1/fetch):
   • Pass req.Pages and req.Grep into pdfdoc.Options in renderPDF.
   • pdfdoc.ErrBadPages or pdfdoc.ErrBadGrep (use errors.Is) → 400, with the error's own text as the message.
   • pages or grep on a non-PDF → 400 pages and grep apply to PDFs only.
4. Question limit: in mode digest with a doc, compute n := len(digest.Split(pdfdoc.Full(doc), digest.ChunkTokens)). If n exceeds s.PDFMaxQuestionChunks (or its default), answer 422 document too long for question (<n> chunks); use grep or pages before any digest call.

Steps

• 1. git switch p5a. 2. See the three tests fail: go test -count=1 -run 'TestPDFPagesAndGrep|TestPDFCache|TestPDFQuestionTooLong' ./internal/api/.
• 3. pdfFromCache and the store (rules 1–2), 4. pages and grep, 5. the question limit.
• 6. gofmt -w; go test -race -count=3 ./internal/api/ → ok (all ten PDF tests and every existing test).
• 7. make gate → gate: ok. 8. Row p5a/07-api-cache-pages; commit.

git add internal/api docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• go test -race -count=3 ./internal/api/ is ok; make gate ok; the given file is still byte-identical.