sift/docs/plans/p5a/06-api-pdf.md

P5a task 06: the PDF path through /v1/fetch and /v2/scrape

Branch: p5a (git switch p5a; git status --short must be empty, otherwise stop) Commit subject: api: serve PDFs — extraction in the child under the pool, full text or map, errors

This task is half of the original task 06, which proved too large for one session. Task 07 adds the cache, pages, grep and the question limit. Do only what is here.

Goal

A PDF URL fetched through /v1/fetch (mode clean) or /v2/scrape is extracted in the child process under the pool and answered with the whole text (page markers) or the map. Its failures get the errors below. No cache in this task (every request fetches and extracts); no pages or grep.

Files

• Copy: internal/api/pdf_test.go (the whole given file; 7 of its 10 tests must pass after this task, listed in Done when)
• Create: internal/api/pdf.go: every function named below lives here
• Modify: internal/api/api.go, only the hooks in rule 6; internal/fetch/fetch.go, only the RetryAfter field in rule 5; docs/implementer-log.md

What already exists (do not re-read these packages; this is their API)

// internal/clean
func Kind(contentType string, body []byte) string // "html" | "json" | "text" | "pdf" | "other"

// internal/pdftext
type OutlineItem struct{ Title string; Page, Level int }
type Meta struct{ Pages int; Title string; Outline []OutlineItem }
type Doc struct{ Meta; Text []string; Failed []int } // Text[i] = page i+1
var ErrEncrypted, ErrUnreadable, ErrNoText, ErrTimeout, ErrTooMuchText error
type Runner struct{ Exe string; Args, Env []string; Deadline time.Duration; MaxText int64; MaxPages int }
func (r *Runner) Run(ctx context.Context, pdf []byte) (*Doc, error) // a zero Runner runs /proc/self/exe pdftext

// internal/pdfpool
var ErrBusy, ErrCallerBusy error
func New(slots, queue int, wait time.Duration) *Pool
func (p *Pool) Do(ctx context.Context, key, caller string, fn func(context.Context) (any, error)) (any, error)

// internal/pdfdoc
type Options struct{ Pages, Grep string; OutCap int }
type Rendered struct{ Content string; Map bool; First, Last, GrepHits int; Note string }
func Render(doc *pdftext.Doc, opt Options) (Rendered, error)
func Full(doc *pdftext.Doc) string

Interfaces added here

package api

const (
	DefaultPDFTTL               = 24 * time.Hour // used by task 07; declare it now
	DefaultPDFMaxQuestionChunks = 200            // used by task 07; declare it now
)

// in Server (after Prom):
	PDFRunner            *pdftext.Runner // nil = &pdftext.Runner{}
	PDFPool              *pdfpool.Pool   // nil = pdfpool.New(2, 4, 20*time.Second), built once (sync.Once)
	PDFTTL               time.Duration   // task 07
	PDFMaxQuestionChunks int             // task 07

// in v1Req:
	Pages string `json:"pages"` // task 07
	Grep  string `json:"grep"`  // task 07

Rules

1. doFetch and cleanFetched gain a fourth result, doc *pdftext.Doc, which is nil unless the resource is a PDF. Each uses clean.Kind(res.ContentType, res.Body) in place of today's MediaKind/Sniff pair.

   • "other" stays the 415 exactly as today.
   • "pdf" calls s.extractPDF(ctx, c, res); on success it returns cl = &clean.Result{Kind: "pdf", Title: doc.Title} and the doc.
   • cleanFetched has no caller at hand for the pool. For a PDF it returns the 415 as before: the recover path (archive snapshots) does not serve PDFs.

2. func (s *Server) extractPDF(ctx context.Context, c *auth.Caller, res *fetch.Resource) (*pdftext.Doc, error)

   • Calls s.pool().Do(ctx, s.callerClass(c)+"|"+res.SHA256(), c.Name, fn), where fn returns s.runner().Run(jobCtx, res.Body).
   • pool() and runner() apply the defaults.
   • Type-assert the result with the two-value form.

3. func pdfError(err error) error maps every PDF-path error to a *fetch.Error:

   cause	Status	Msg	Hint	RetryAfter
   pdftext.ErrNoText	422	no text layer (scanned PDF?)	ocr
   pdftext.ErrEncrypted	422	encrypted PDF (password required)
   pdftext.ErrUnreadable (also when wrapped)	422	unreadable PDF
   pdftext.ErrTimeout	422	pdf extraction timed out
   pdftext.ErrTooMuchText	422	pdf text over the output limit
   pdfpool.ErrBusy	503	pdf workers busy		10
   pdfpool.ErrCallerBusy	429	busy
   anything else (a panic recovered by the pool)	422	unreadable PDF

   Use errors.Is. A PDF never produces a 500.

4. Rendering (pdf.go), func (s *Server) renderPDF(doc *pdftext.Doc) (pdfdoc.Rendered, error): pdfdoc.Render(doc, pdfdoc.Options{OutCap: s.OutCap}).

   • /v1/fetch, mode clean with a doc:
     • content is Rendered.Content;
     • the envelope gains "pdf": {"pages": doc.Pages, "title": doc.Title, "outline": doc.Outline, "extracted": len(doc.Text), "first", "last", "map", "grep_hits", "note"}, the last five from Rendered;
     • sizes.raw_tokens is the rune count of pdfdoc.Full(doc) / 4.
   • Mode raw with a doc: 400 raw mode does not apply to PDFs.
   • Mode digest with a doc: run the digest exactly as today but with pdfdoc.Full(doc) as the content. Task 07 adds the length limit before it.
   • /v2/scrape with a doc:
     • markdown is Rendered.Content; html is "";
     • metadata.title is doc.Title;
     • metadata.numPages is doc.Pages;
     • metadata.contentType is the response's.

5. internal/fetch/fetch.go: add RetryAfter int to fetch.Error (seconds, 0 = none). writeFetchErr sets the Retry-After header when it is non-zero, before writeErr. This is the only change allowed in the fetch package.

6. api.go hooks, and nothing else there:

   • the Server and v1Req fields;
   • the two Kind switches (rule 1);
   • the new result at the three call sites of doFetch/cleanFetched;
   • one if doc != nil { … } block in v1fetch and in scrape that calls into pdf.go;
   • the Retry-After line in writeFetchErr;
   • pdf_pages (a count) added to the existing fetch and scrape log lines.

   Never log a title, an outline or any text. Restricted callers go through the existing helpers.

Facts

• The given test's child is the test binary itself (its TestMain turns into pdftext when SIFT_PDFTEXT_CHILD=1). The rig builds s.PDFRunner with that environment; you never start a child yourself outside Runner.Run.
• setupAPI, postJSON and recHandler come from api_test.go in the same package.
• s.cache() may be nil; this task never uses it.

Steps

• 1. git switch p5a; copy the given test.
• 2. See it fail: go test -count=1 ./internal/api/ fails to compile (undefined fields).
• 3. fetch.Error.RetryAfter. 4. pdf.go. 5. The api.go hooks.
• 6. gofmt -w; then go test -race -count=3 -run 'TestPDFCleanFetch|TestPDFMap|TestPDFErrors|TestPDFTimeout|TestPDFBusy|TestPDFRestrictedCaller|TestPDFScrape' ./internal/api/ → ok, and go test -race -count=1 ./internal/api/ -run 'Test[^P]' (every existing api test) → ok.
• 7. Row p5a/06-api-pdf. Its Gate column says pass (7 of 10 PDF tests; 3 are task 07's). Commit.

git add internal/api internal/fetch/fetch.go docs/implementer-log.md
git commit -F .state/commit-msg.txt

Done when

• The seven named tests pass under -race -count=3, and every non-PDF api test passes.
• The three remaining given tests fail only on pages, grep, the cache or the question limit: TestPDFPagesAndGrep, TestPDFCache, TestPDFQuestionTooLong. That is task 07. make gate is expected red on exactly those three and nothing else. Say so in the log row.
• The given file is byte-identical.

Stop and report if

• Any of the seven fails for a reason that needs the cache, pages or grep.