90 lines
4.4 KiB
Markdown
90 lines
4.4 KiB
Markdown
# P5a task 07: the PDF document cache, `pages`, `grep`, and the question limit
|
||
|
||
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
||
**Commit subject:** `api: PDF cache with revalidation; pages and grep; refuse questions on very long documents`
|
||
|
||
## Goal
|
||
|
||
Task 06 serves PDFs but fetches and extracts every time. After this task, a PDF already
|
||
extracted is answered from the cache: fresh for 24 h, then revalidated with a conditional GET.
|
||
An agent can ask for `pages` or `grep`. A question on a document too long for the digest is
|
||
refused with a pointer to those two.
|
||
|
||
## Files
|
||
|
||
- Copy: nothing (the given `internal/api/pdf_test.go` came in task 06)
|
||
- Modify: `internal/api/pdf.go` (all new logic), `internal/api/api.go` (only the call in rule 1
|
||
and the two request fields' use in rule 3), `docs/implementer-log.md`
|
||
|
||
## What already exists
|
||
|
||
- Task 06's `extractPDF`, `pdfError`, `renderPDF`, the `Server` fields (`PDFTTL`,
|
||
`PDFMaxQuestionChunks`), the `v1Req` fields (`Pages`, `Grep`), and `DefaultPDFTTL` /
|
||
`DefaultPDFMaxQuestionChunks`.
|
||
- `internal/dcache`:
|
||
```go
|
||
type PDFEntry struct{ ContentHash, ETag, LastModified string; FetchedAt time.Time; Meta []byte; Pages []string }
|
||
func PDFKey(url, callerClass string) string
|
||
func (d *DB) PutPDF(key string, e PDFEntry)
|
||
func (d *DB) GetPDF(key string) (PDFEntry, bool)
|
||
func (d *DB) TouchPDF(key string, fetchedAt time.Time)
|
||
```
|
||
A nil `*DB` is a no-op.
|
||
- `internal/fetch`: `func (f *Fetcher) FetchIf(ctx, rawURL, etag, lastModified string) (*Resource, error)`.
|
||
A 304 returns `Status 304`, an empty body and a nil error.
|
||
- `internal/digest`: `Split(text string, maxTok int) []string` and `ChunkTokens`.
|
||
|
||
## Rules
|
||
|
||
1. **Cache first, in `doFetch`**, before `acquire` and `s.Fetch.Fetch`: call
|
||
`s.pdfFromCache(ctx, c, rawURL) (res *fetch.Resource, doc *pdftext.Doc, ok bool, err error)`.
|
||
The key is `dcache.PDFKey(rawURL, s.callerClass(c))`; the cache is `s.cache()` (may be nil).
|
||
- **Miss:** `ok == false`; go on as today.
|
||
- **Fresh** (`time.Since(e.FetchedAt) < ttl`, where `ttl` is `s.PDFTTL` or `DefaultPDFTTL`):
|
||
build the doc from the entry (unmarshal `Meta` into `doc.Meta`; `doc.Text = e.Pages`), and a
|
||
synthetic `res`: `URL` and `FinalURL` = rawURL, `Status` 200,
|
||
`ContentType "application/pdf"`, `FetchedAt` = `e.FetchedAt`, `Hash` = `e.ContentHash`, no
|
||
body. No network call and no child.
|
||
- **Stale:** `s.Fetch.FetchIf(ctx, rawURL, e.ETag, e.LastModified)`.
|
||
- 304 → `TouchPDF(key, now)`, then answer as fresh.
|
||
- 200 → return that response with `ok == false`, so `doFetch` goes on without a second
|
||
GET: the new document is extracted and stored.
|
||
- An error → return it.
|
||
- The source in the `/v1/fetch` envelope says `"cached": true` for an answer from the cache
|
||
(fresh or after a 304).
|
||
2. **Store after extraction:** after `extractPDF` succeeds, `PutPDF` under the URL key with:
|
||
- `ContentHash: res.SHA256()`;
|
||
- `ETag` and `Last-Modified` from the response headers;
|
||
- `FetchedAt: now`;
|
||
- `Meta`: the JSON of `doc.Meta`;
|
||
- `Pages: doc.Text`.
|
||
3. **`pages` and `grep` (`/v1/fetch`):**
|
||
- Pass `req.Pages` and `req.Grep` into `pdfdoc.Options` in `renderPDF`.
|
||
- `pdfdoc.ErrBadPages` or `pdfdoc.ErrBadGrep` (use `errors.Is`) → 400, with the error's own
|
||
text as the message.
|
||
- `pages` or `grep` on a non-PDF → `400 pages and grep apply to PDFs only`.
|
||
4. **Question limit:** in mode `digest` with a doc, compute
|
||
`n := len(digest.Split(pdfdoc.Full(doc), digest.ChunkTokens))`. If `n` exceeds
|
||
`s.PDFMaxQuestionChunks` (or its default), answer
|
||
`422 document too long for question (<n> chunks); use grep or pages` **before** any digest
|
||
call.
|
||
|
||
## Steps
|
||
|
||
- [ ] **1.** `git switch p5a`. **2. See the three tests fail:**
|
||
`go test -count=1 -run 'TestPDFPagesAndGrep|TestPDFCache|TestPDFQuestionTooLong' ./internal/api/`.
|
||
- [ ] **3.** `pdfFromCache` and the store (rules 1–2), **4.** pages and grep, **5.** the question limit.
|
||
- [ ] **6.** `gofmt -w`; `go test -race -count=3 ./internal/api/` → `ok` (all ten PDF tests and
|
||
every existing test).
|
||
- [ ] **7.** `make gate` → `gate: ok`. **8.** Row `p5a/07-api-cache-pages`; commit.
|
||
|
||
```sh
|
||
git add internal/api docs/implementer-log.md
|
||
git commit -F .state/commit-msg.txt
|
||
```
|
||
|
||
## Done when
|
||
|
||
- `go test -race -count=3 ./internal/api/` is ok; `make gate` ok; the given file is still
|
||
byte-identical.
|