sift/docs/plans/p5a/06-api.md

P5a task 06: PDFs through /v1/fetch and /v2/scrape

Branch: p5a (git switch p5a; git status --short must be empty, otherwise stop) Commit subject: api: serve PDFs — child extraction via the pool, document cache, map, pages, grep

Goal

Wire tasks 01–05 into the API. A PDF URL fetched through /v1/fetch or /v2/scrape is:

1. extracted once in the child, under the pool;
2. cached per page;
3. answered with the whole text, the map, a page range, or grep results.

Later requests for the same document are served from the cache, revalidated after 24 h with a conditional GET. question digests the document's text, page markers included, unless the document is too long.

Files

• Copy: internal/api/pdf_test.go
• Create: internal/api/pdf.go — everything PDF lives here
• Modify: internal/api/api.go — only the hooks listed in rule 7. api.go is already 840 lines; do not move code into it.
• Modify: docs/implementer-log.md

Interfaces

package api

const (
	DefaultPDFTTL               = 24 * time.Hour
	DefaultPDFMaxQuestionChunks = 200
)

// in Server:
	PDFRunner            *pdftext.Runner // nil = &pdftext.Runner{} (the child is /proc/self/exe pdftext)
	PDFPool              *pdfpool.Pool   // nil = pdfpool.New(2, 4, 20*time.Second)
	PDFTTL               time.Duration   // 0 = DefaultPDFTTL
	PDFMaxQuestionChunks int             // 0 = DefaultPDFMaxQuestionChunks

// in v1Req:
	Pages string `json:"pages"`
	Grep  string `json:"grep"`

Defaults are applied where the field is read: a zero Server from New must work. Build the default pool once, with a sync.Once — never one per request.

Rules

1. Cache first.

   • Before fetching, look up dcache.PDFKey(rawURL, s.callerClass(c)) in s.cache() (which may be nil: then there is no cache).
   • Fresh (now − FetchedAt < TTL): build the *pdftext.Doc from the entry (unmarshal Meta; Text = Pages). Answer with no network call and no child. The source is marked "cached": true.
   • Stale: s.Fetch.FetchIf(ctx, rawURL, e.ETag, e.LastModified).
     • 304 → TouchPDF(key, now) and answer from the entry, with no child.
     • 200 → go on as for a new document.
   • Not cached: the usual s.Fetch.Fetch, under the existing acquire/release.

2. A PDF is clean.Kind(res.ContentType, res.Body) == "pdf". Every other kind goes through today's path unchanged. Use clean.Kind in place of the MediaKind/Sniff pair in both doFetch and cleanFetched. "other" stays 415.

3. Extraction.

   • Call s.PDFPool.Do(ctx, class+"|"+res.SHA256(), c.Name, fn), where fn runs s.PDFRunner.Run(jobCtx, res.Body).
   • The key is the content hash, so identical bytes are extracted once however many callers or URLs ask. The key includes the caller class, so a restricted caller never shares a job with the fleet.
   • Then PutPDF under the URL key with: the content hash; the response's ETag and Last-Modified; FetchedAt = now; Meta = the JSON of doc.Meta; Pages = doc.Text.

4. Errors, all through the existing writeErr shape ({"success":false,"error":…,"hint":…}):

   cause	status	error	hint
   pdftext.ErrNoText	422	no text layer (scanned PDF?)	ocr
   pdftext.ErrEncrypted	422	encrypted PDF (password required)
   pdftext.ErrUnreadable	422	unreadable PDF
   pdftext.ErrTimeout	422	pdf extraction timed out
   pdftext.ErrTooMuchText	422	pdf text over the output limit
   pdfpool.ErrBusy	503	pdf workers busy, header Retry-After: 10
   pdfpool.ErrCallerBusy	429	busy
   pdfdoc.ErrBadPages / ErrBadGrep	400	the error's text
   pages or grep on a non-PDF	400	pages and grep apply to PDFs only
   question on a document over PDFMaxQuestionChunks chunks	422	document too long for question (N chunks); use grep or pages

   A PDF path never answers 500 for anything the document did.

5. /v1/fetch.

   • Mode clean: content is pdfdoc.Render(doc, {Pages, Grep, OutCap: s.OutCap}).Content.
   • The envelope gains:

     "pdf": {"pages", "title", "outline", "extracted", "first", "last", "map", "grep_hits", "note"}
     

     extracted is len(doc.Text); outline is doc.Outline; the rest come from Rendered.
   • Mode digest: the text is pdfdoc.Full(doc). First count len(digest.Split(full, digest.ChunkTokens)); if it is over PDFMaxQuestionChunks, answer the 422 above before any model call. Otherwise run the digest as for any page, with full as the content.
   • Mode raw on a PDF: 400 raw mode does not apply to PDFs.
   • sizes.raw_tokens is the rune count of Full(doc) / 4 for a PDF (there is no body on a cache hit).

6. /v2/scrape.

   • markdown is Render(doc, {OutCap: s.OutCap}).Content; html is "".
   • metadata.title is doc.Title; metadata.numPages is doc.Pages; metadata.contentType is the response's (application/pdf on a cache hit).

7. api.go hooks (the only edits allowed there):

   • the four Server fields and two v1Req fields;
   • the Kind switch in doFetch/cleanFetched, now returning the doc;
   • one call into pdf.go in v1fetch and in scrape;
   • Retry-After in writeFetchErr for a 503.

   doFetch and cleanFetched may gain a result, doc *pdftext.Doc (nil unless a PDF).

8. Logging:

   • Keep today's fetch/scrape log lines and add pdf_pages (a count) to them.
   • Never log the title, the outline or any text.
   • A restricted caller's lines go through the existing helpers (host blanked); the given test reads every log record.

Facts

• The tests set s.Digest = &digest.Engine{Cache: db} only to give s.cache() a database: no model is configured.
• The child in the tests is the test binary (TestMain in the given file).
• setupAPI, postJSON and recHandler come from api_test.go in the same package.
• requestDeadline (210 s) and writeTimeout (240 s) were raised by the owner for this path: download ≤ 120 s + queue ≤ 20 s + extraction ≤ 60 s.

Steps

• 1. git switch p5a; copy the given test. 2. See it fail (undefined fields).
• 3. pdf.go (cache, extraction, errors, render), then the api.go hooks.
• 4. gofmt -w; go test -race -count=3 ./internal/api/ → ok: the new tests and every existing api test (search, scrape, restricted logging, write timeout).
• 5. make gate. 6. Row p5a/06-api; commit.

git add internal/api docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• go test -race -count=3 ./internal/api/ is ok; make gate ok; given file byte-identical; api.go grew by the hooks only (the owner will diff it).

Stop and report if

• A rule here contradicts an existing api test.
• The pool's default wait (20 s) makes a test take longer than the test allows. The given tests set their own pool where timing matters.