P5b and P5c plans: long-document retrieval (BM25 first, background embedding) and /v1/extract + sift extract; given tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 11:01:18 -07:00
parent 1311cf38d4
commit 49d96e197b
14 changed files with 1464 additions and 0 deletions
@@ -0,0 +1,43 @@
+# P5b task 03: the API answers long-document questions and says how
+
+**Branch:** `p5b` (`git switch p5b`; `git status --short` must be empty, otherwise stop)
+**Commit subject:** `api: questions on long PDFs are answered (retrieval in the digest meta); README and skill`
+
+## Goal
+
+With task 02, the digest handles long documents itself. The API's 200-chunk refusal becomes a
+5000-chunk safety cap (task 02's `MaxDocChunks`), and the `/v1/fetch` envelope's `digest`
+object says how the answer was built.
+
+## Files
+
+- Copy: `internal/api/pdf_retrieval_test.go`, and the **replacement** `internal/api/pdf_test.go`
+  (P5a's copy asserted `DefaultPDFMaxQuestionChunks == 200`; now 5000 — the only change; the
+  P5b copy is the protected one from now on)
+- Modify: `internal/api/pdf.go` (the constant), `internal/api/api.go` (the two digest-meta keys
+  only), `README.md`, `skill/SKILL.md`, `docs/implementer-log.md`
+
+## Rules
+
+1. `DefaultPDFMaxQuestionChunks = 5000`. The 422 message is unchanged.
+2. In `v1fetch`, when the digest ran without a fallback, `digMeta` gains
+   `"retrieval": dr.Retrieval` and `"embedded": dr.Embedded`, next to `chunks_used` and
+   `chunks_total`.
+3. **`README.md`:** in the PDF section, a question on a long document is answered first from BM25 (`retrieval: "bm25"`) while its chunks are embedded in the background, and is hybrid on later questions. Also mention the 5000-chunk cap.
+4. **`skill/SKILL.md`:** one sentence: `--question` works on long PDFs; the first answer may be `bm25` (lexical); ask again later for a sharper one.
+
+## Steps
+
+- [ ] **1.** `git switch p5b`; copy the two given files. **2. See them fail.**
+- [ ] **3.** The constant, the two meta keys, README, skill.
+- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/api/` → `ok`. **5.** `make gate`.
+- [ ] **6.** Row `p5b/03-api`; commit.
+
+```sh
+git add internal/api README.md skill/SKILL.md docs/implementer-log.md
+git commit -F .state/commit-msg.txt
+```
+
+## Done when
+
+- All tests pass; `make gate` ok; given files byte-identical.