sift/docs/plans/p5a/03-dcache.md

P5a task 03: a document cache for PDFs

Branch: p5a (git switch p5a; git status --short must be empty, otherwise stop) Commit subject: dcache: PDF document cache (per URL and caller class, whole-document eviction)

Goal

Extracting a thesis takes a second; downloading it can take a minute. Once a document is extracted, later requests for other pages, a grep, or a question must not fetch or extract it again. This task adds two tables to the existing SQLite cache and four functions. Freshness (the 24 h TTL and revalidation) is decided by the caller in task 06, not here.

Files

• Copy: internal/dcache/pdf_test.go
• Create: internal/dcache/pdf.go
• Modify: internal/dcache/dcache.go (the two CREATE TABLEs, Size, evict), docs/implementer-log.md

Interfaces

package dcache

// PDFEntry is one cached PDF: how to revalidate it, its metadata (the pdftext.Meta JSON, opaque
// here) and the text of each extracted page (Pages[i] is page i+1).
type PDFEntry struct {
	ContentHash  string
	ETag         string
	LastModified string
	FetchedAt    time.Time
	Meta         []byte
	Pages        []string
}

// PDFKey is the cache key for a document: a hash of the URL and the caller class, like the
// other keys here (the URL never appears in the clear).
func PDFKey(url, callerClass string) string

func (d *DB) PutPDF(key string, e PDFEntry)            // replaces any entry under key, pages included
func (d *DB) GetPDF(key string) (PDFEntry, bool)
func (d *DB) TouchPDF(key string, fetchedAt time.Time) // after a 304: only FetchedAt moves; a miss is a no-op

Rules

1. Tables:
   • pdf_doc(k TEXT PRIMARY KEY, content_hash, etag, last_modified, fetched_at INTEGER, meta BLOB, last_hit INTEGER, nbytes INTEGER)
   • pdf_page(k TEXT, page INTEGER, text TEXT, PRIMARY KEY (k, page)) nbytes is the document's size (meta plus every page's text) and counts toward Size().
2. PutPDF writes the document and its pages in one transaction, deleting the old pages first, then calls evict. GetPDF updates last_hit, and returns the pages in page order; an empty page keeps its place.
3. evict treats a PDF document like any other entry, by last_hit, and removes it whole: its pdf_doc row and all its pdf_page rows together. A cache never holds half a document.
4. Hit and miss counters: add PDFH, PDFM beside the others (the metrics task reads them later).
5. A nil *DB is a no-op everywhere, as for the existing methods; a SQL error is logged at Warn (never with the key's text or page text) and behaves like a miss.

Facts

• DigestKey, EmbedKey and SearchKey show how keys are hashed (sha256 of the parts joined with a separator). Follow the same pattern.
• The existing evict picks the oldest last_hit across the three existing tables; extend that union with pdf_doc.

Steps

• 1. git switch p5a; copy the given test. 2. See it fail (undefined names).
• 3. dcache.go schema; pdf.go; Size and evict.
• 4. gofmt -w; go test -race -count=3 ./internal/dcache/ → ok (the existing dcache tests too).
• 5. make gate. 6. Row p5a/03-dcache; commit.

git add internal/dcache docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The given and existing dcache tests pass under -race -count=3; make gate ok; given file byte-identical.